Now boasting over a billion members, LinkedIn operates on a global scale. However, just over a year ago, as the company embarked on constructing its latest public feature involving collaborative articles, the challenge emerged of harnessing the then-emerging generative AI to expedite the sharing of advice within the world’s largest professional network.
This four-month journey of design and development unveiled numerous sociotechnical challenges in utilizing GenAI, and two members of the team exclusively shared these insights with The New Stack. Gain insights from Shweta Patira, Director of Creator Engineering and project lead, and Lakshman Somasundaram, Director of Product Management for the Moonshots team. This knowledge equips you to embark on building with generative AI, keeping both the AI and human users in mind, all at an extensive scale.
LinkedIn Aims to Harness GenAI for Stimulating Conversations
“It begins with the recognition that everyone faces challenges at work every single day. It could be something like, ‘I wish I could get promoted, and I don’t know how to go about it,’ or ‘I need to conduct an interview, and I’m not sure how to approach it.’ Each person has their own workplace challenges on a daily basis,” shared Somasundaram with The New Stack. Up until now, the conventional approach to finding answers involved seeking advice from connections or individuals who have experienced and successfully resolved similar issues.
However, these inquiries are rarely binary, and there isn’t a one-size-fits-all solution. Conducting a technical interview or navigating the path to a promotion involves nuanced considerations, influenced by various contexts and biases. Seeking diverse perspectives enhances the likelihood of obtaining more informed answers.
Unfortunately, not everyone has access to an extensive network. Somasundaram explained, “This means that individuals who often access the best solutions to these questions are those with large networks. They can tap into multiple people in their networks to gather advice, perspectives, and opinions on how to address specific problems.” In contrast, he continued, “Most of the world does not have large networks. Most people lack a significant number of contacts to consult for advice and diverse perspectives.”
With over a billion LinkedIn members collectively possessing over 10 billion years of work experience, the company is uniquely positioned to capitalize on the social network dynamics and leverage its members to share expertise on a large scale.
However, the challenge arises when workers have limited time to allocate for tasks beyond their primary responsibilities. Searching for questions and providing responses on LinkedIn may not be part of their job description.
On the other hand, the creation process for AI faces fewer limitations in the current landscape. According to Somasundaram, generative AI enables a more straightforward critique of its creations. Within collaborative articles, LinkedIn has utilized the power of AI to generate starter articles, inviting global experts on those topics to contribute their insights and comments.
The generative AI has been trained on the 40,000 skills within LinkedIn profiles, allowing it to ask questions and suggest subtopics within those questions. This approach allows identified experts to respond without the need for the original inquirer to post the question directly.
When the decision was made to build LinkedIn’s first major generative AI feature in October 2022, the team faced unique challenges specific to the GenAI space. The subsequent experience provided valuable insights for the stories engineering team, shedding light on the intricacies of working with generative AI.”
GenAI Sociotechnical Challenge #1: Prompt Engineering
Certainly, LinkedIn, along with GitHub (the creator of Copilot) and OpenAI (the creator of ChatGPT), is under the ownership of Microsoft. However, this does not imply immediate insider access. Patira clarified to The New Stack that their access was limited to ChatGPT 3.5, which launched around the same time as their project, but they did not have access to 3.5 at scale. They predominantly relied on its predecessor, GPT-3, to develop and deploy this feature.
The nascent stage of generative AI at that time led to the realization that they not only needed to create large language model (LLM) prompts but also had to construct much of the GenAI scaffolding, including program-to-workflow management and tooling, from the ground up while building the product. According to Patira, they quickly recognized that generative AI was not, and still is not, at a stage where it can work entirely autonomously. GPT-3 fell short, and human evaluation of generative AI remained crucial. The LinkedIn stories team followed a pattern of human evaluation to assess the quality of these articles.
They would initiate the process with a prompt like, “What are common causes of fear of public speaking?” Once responses were received, they aimed to develop these responses into starter articles in batches and at scale. To achieve this, they enlisted their editorial team to evaluate the quality based on relevance, accuracy, and the absence of red flags.
At this stage, the GenAI results were either approved or rejected before proceeding to further refinement, such as requesting the GenAI to enhance the writing’s clarity. This iterative process was performed repeatedly, leading to the realization that tools were necessary. Spreadsheet-based approaches were deemed inefficient, prompting them to concurrently develop tools while building the product. This tooling was essential to ensure scalability, batch processing, human evaluation, and integration of trust classifiers into the workflow.
GenAI Sociotechnical Challenge #2: Trust
However, what does trust signify in the context of interactions with robots?
“At LinkedIn, trust is an integral aspect of every product we develop,” explained Patira. “Throughout the entire process of generating these articles, from inception to distributing answers, at every juncture, we employ what we refer to as trust classifiers. These are proactive defenses, and we adopt a Swiss cheese model because we acknowledge that a singular defense is insufficient. Thus, at each stage, we implement these defenses.”
Patira clarified that this goes beyond generative AI to encompass AI in general, emphasizing the importance of distinguishing between dissent and harassment.
“While we encourage constructive debate, we aim to avoid toxicity,” she continued. “We utilize classifiers, which are essentially AI models, to scrutinize both human contributions and generative AI-generated starter articles at each step. They assess for toxic content, harassment, and bullying.”
The necessity of human-in-the-loop intervention prompted the team to rapidly expand to support the collaborative articles project at scale. Approximately 12 sociotechnical teams, each comprising four or five members, were formed to address the outlined challenges and sub-challenges.
“Throughout the entire journey, we discovered that AI is still more reliable than humans,” reflected Patira. As they sought to eliminate hate speech and spam, they implemented proactive defenses early on. “Humans can be unpredictable, whereas AI is more predictable. In our product, this translated into the implementation of numerous trust guardrails to scrutinize AI. Surprisingly, we found that we required more trust guardrails for humans than for AI.”
Offering guidance to those working with generative AI, Patira emphasized the need for additional measures to ensure that conversations on LinkedIn remain healthy, primarily driven by humans rather than AI.
GenAI Sociotechnical Challenge #3: Expert Identification
LinkedIn stands as the world’s largest professional network, creating a challenge in distinguishing genuine experts from those merely projecting expertise, a hurdle existing even before the feature’s March launch this year, given the platform’s billion-plus members.
“LinkedIn excels in this aspect due to our rich Caleygraph,” noted Patira, encompassing individual job histories, skills, skill endorsements, and proficiency tests. “This provides a unique advantage in determining whether someone is a true expert in a particular area.”
Leveraging publicly available profile data, LinkedIn employs AI to identify the top 10% among the 40,000 skills, categorized into approximately 1,000 expert topics at present. Patira explained, “We will rank items for you based on what we consider most relevant.”
While there is an incentive for users to respond in areas where they aim to be recognized as experts, as evidenced by the potential “Top Voice” badge on their profile, Patira emphasized transparency. Contributions won’t be hidden unless they are genuinely low quality or violate professional community policies.
In this initial phase of collaborative articles, “experts” undergo manual vetting. LinkedIn members engage with stories using reactions similar to posts, influencing both the AI suggestion algorithm and the human evaluators in the loop. The feature also incorporates a violation reporting system akin to other LinkedIn functionalities.
GenAI Sociotechnical Challenge #4: Distribution
Initially, the platform welcomes experts, followed by those who eventually become experts through collaborative stories.
“For instance, if you’re an expert in podcasting and have shared your stories, we aim to deliver this content to individuals genuinely interested in learning more about public speaking or those seeking relevant answers. We refer to these individuals as Knowledge Seekers, and our goal is to reach them where they are,” explained Patira. “Therefore, distribution poses another significant technological challenge.”
Given that LinkedIn users are human, the instinctual response to a question is often a Google search. LinkedIn is strategically working to rank on Google features like “People also ask.” Moreover, recommendations for expert collaborations appear in users’ LinkedIn feeds and email notifications, with immediate notifications for those who have requested updates on a specific user.
“Our aim is to reshape people’s habits in terms of when and where they seek these answers,” Patira stated. “We aspire to bring this content to them in the places they already frequent.”
GenAI Sociotechnical Challenge #5: At Scale
“These challenges are not overly complex on a smaller scale; they simply become more formidable with large-scale operations,” remarked Patira. LinkedIn has identified approximately 40 million potential experts, received millions of questions through a combination of prompt engineering and generative AI, and manages millions of jobs in the backend, in addition to expert evaluation for identification at scale.
Patira elaborated on the prompt workflow, stating, “We employ workflows where we input data into queues, retrieve them from Kafka queues, transfer them to another part of the workflow, make online calls to GPT, obtain responses, and store them — all in an end-to-end process.”
The subsequent workflow revolves around expert identification, involving backend jobs executed offline every few hours to analyze data for expert prospecting. Patira explained the process of evaluating the current expert list, identifying potential experts, detecting changes, and then updating the data, with most of the operations occurring in the backend.
Key components like expert identification and prompt workflow management are primarily conducted offline. Online operations are reserved for timely notifications on members’ feeds, ensuring users receive notifications when relevant content is shared.
The LinkedIn collaborative stories team continually works on optimizing and minimizing the reliance on online data processing, emphasizing a balance with offline and nearline processing. Nearline data serves as an intermediate type, not as fast as real-time data but quicker than retrieving offline data. The management of feed and notifications, being the two significant online systems, requires careful scalability management.
“We heavily leverage our LinkedIn Graph on both of these,” Patira mentioned, referring to LinkedIn’s Economic Graph. The scale is anticipated to grow further, with the implementation of Chat GPT-4 expected to expedite growth.
“In the eight months since our launch, there has been significant acceleration, especially in the last month and a half,” noted Somasundaram. This includes a global expansion in English and recent launches in French, Spanish, Portuguese, and German. They have also unveiled a substantial desktop redesign.
Future plans include allowing members to pose questions. Both emphasized that the goal of these articles is not to rely solely on AI text but to leverage generative AI to kickstart professional, human-led conversations.
