LinkedIn engineering blog

Loading
Rebuilding Data Lineage at LinkedIn: From Storage to Compute
LinkedIn rebuilt its data lineage platform to capture lineage from executed query plans across more than 15 compute and data-movement systems. The design routes events through Kafka into an append-only log, stores the graph in MySQL with a breadth-first traversal engine, and attaches type, confidence, and granularity to every edge. The post reports a production graph of roughly 6 million datasets with P99 traversal latency near 200 ms.
Building View-Agnostic Analytics for Modern iOS UI Systems
LinkedIn describes a view-agnostic analytics design for iOS apps that use flattened layout engines, where logical containers exist only as virtual nodes rather than UIViews. The post introduces a measurable node contract covering identity, frame, logical parent, and attachment, along with a hybrid parent-chain walk, lifecycle rules for attach and detach, and a two-path design. It also covers the silent failure modes this creates and the reliability practices used to catch them.
Profile Relevance: Boosting Feed Relevance with LLMs at LinkedIn
LinkedIn's engineering team describes Profile Relevance, an unsupervised LLM-embedding retrieval system that matches members and posts by professional context to surface Unconnected Content. The post compares Rewrite and Non-Rewrite embedding architectures, covers an LLM-driven AutoEval framework, and reports production lessons on incremental profile updates, horizontal versus vertical GPU scaling, and geographic augmentation.
Responsible AI Principles in Practice
LinkedIn describes how it applies its Responsible AI principles across product development, covering privacy and data-leakage testing, red-teaming, prompt guardrails, counterfactual and fairness testing, transparency, and governance. The post stays at the level of principles and policy, naming techniques without explaining how they work or what they found.
How we scale deployments with a custom GitOps engine
LinkedIn describes how it scaled Kubernetes deployments across 250+ clusters, moving from a per-cluster Argo CD setup to a custom GitOps engine built around per-deployment ephemeral pods, PUT-based apply, and first-class drift detection. The post details specific Argo CD failure modes such as missing observedGeneration, managedFields truncation, and noisy-neighbor watch contention, and reports a drop in infra failures and median failed-deployment detection time from 16.1 to 2.45 minutes.
From Member Tap to Backend: How LinkedIn Traced the Member Jou...
LinkedIn describes how it rebuilt its telemetry foundation around OpenTelemetry to connect client-side analytics on iOS, Android, and web with backend distributed traces. The post covers a pageForestId-based identifier scheme, head and client/server sampling strategies, a Kafka and ClickHouse ingestion pipeline, and practical lessons on batching, app suspension, and PII safeguards. Two production investigation case studies illustrate the approach.
Rebuilding LinkedIn's Tracking Infrastructure: Scaling, Migrat...
LinkedIn engineers describe rebuilding their high-volume tracking event pipeline, moving from Java 8 and Play Framework to a gRPC-based service with workload isolation, tagged 10% shadow-topic comparison for safe migration, and inline Avro encoding and zstd compression that improved throughput. The post also covers an overload detector that connects delayed Kafka callback failures to request admission. Reported results include a roughly 89% reduction in inbound network usage and migration of...
Scaling Spark and Flink Shuffle at LinkedIn with Apache Celeborn
LinkedIn describes how it moved its Spark and Flink shuffle layer from Magnet's push-based local-disk model to a disaggregated Apache Celeborn service. The post covers Raft-based control-plane HA, network-aware replication, disruption-aware slot selection, NVMe and 100 Gbps networking, ZSTD compression tuning, and reports measured gains such as roughly 10% lower compute cost and fetch failures falling from about 750 to 20 per day.
Inside LinkedIn's data context layer: curating trustworthy context for humans and agents
LinkedIn describes a shared data context layer that combines AI-generated dataset profiles with owner-curated guidance on meaning, usage, joins, filters, and limitations. The system uses query-log analysis, masked samples, Git-based review, embeddings, MCP tools, and progressive-disclosure indexes to help humans and agents select trustworthy data, improving acceptable-answer rates and reducing response times.
Announcing Our LinkedIn-Cornell 2026 Grant Recipients
LinkedIn announces the 2026 recipients of its Cornell Bowers research grants, supporting projects in AI agents, LLMs, text diffusion, database efficiency, privacy, safety, and recommendation systems. The post also summarizes the partnership’s five-year research impact.
Building LinkedIn Premium All‑in‑One: Unified AI‑Powered Workf...
LinkedIn engineers describe the architecture behind Premium All-in-One, including LLM-based target-audience inference, activity signals with time decay, embedding and index-based prospect retrieval, and targeted post scoring. The post also covers privacy guardrails, experimentation, staged rollouts, and lessons from unifying systems across multiple product lines.
High-Signal AI Code Review That Adapts to Your Codebase at Scale
LinkedIn describes how it built a customized multi-agent AI code review system that scales across thousands of repositories while producing high-signal, actionable feedback. The post explains the orchestration, deduplication, customization framework, and evaluation methods behind the system, including its acceptance rates, infrastructure, and plans to extend review beyond comments and into production incident learning.