Databricks' Compute Platform team describes how fixed request-rate limits failed to protect long-running cluster-creation workflows, and how adaptive concurrency limits with slot-based tracking, bounded queues, and fair scheduling let them raise default cluster-creation rate limits by 7.5×. The post covers retry storms, distinguishing system degradation from tenant overload, and reported outcomes including over 99.99% fewer rate-limit errors.
Data & Databases
Databases and the pipelines feeding them — Postgres, MySQL, Redis and DynamoDB, plus Spark, ETL, and warehouse and lakehouse architecture. Expect query plans, migration stories, and the occasional hard-won lesson about what stops working at a billion rows.
Loading
Datadog describes how its metrics aggregation system evolved across three generations, from static partition ownership with local RocksDB state, to dynamically assigned consumers with state snapshots in blob storage, to a best-effort streaming layer where durable reconciliation of partial aggregates is delegated to the timeseries storage engine. The post explains the recovery, scaling, and cost trade-offs behind each design and why the final architecture lets the streaming layer resume at the...
Lyft describes how Smooth Cruiser, its driver safety scoring feature, measures driving signals relative to local road conditions, regional peer groups, and a rolling window of recent rides. The post also covers the data platform behind the score, including ML-based harsh event detection, an Apache Airflow ETL pipeline, and DynamoDB for low-latency score retrieval, along with transparency features that show drivers the rides behind their scores.
A DuckDB User Group workshop built a pipeline over 2.1 million Amazon appliance reviews to find products with falling ratings. DuckDB SQL selects the largest rating drops and the most helpful recent complaints, and only those 40 reviews go to an LLM for theme summarization. The post also covers running the job on Hugging Face and deploying the workflow as a Gradio app with API endpoints.
Heroku has released heroku data:pg:migrate, a guided CLI workflow for moving Postgres Classic databases to Heroku Postgres Advanced. The post explains the two-phase design: an online preparation phase, then an offline snapshot copy that blocks writes, restores constraints and indexes, and validates the result. It also covers rollback behavior, rehearsing on a fork to measure downtime, and upgrading the Postgres major version during migration.
Explains how Databricks Lakebase Postgres uses a tiered architecture so that object storage almost never sits on the query read path. Production stats show 99.46% of page reads hit compute-node caches, with pageservers serving most of the rest from local NVMe, and object storage handling only about 1 in 28 million accesses. The post also covers how image and delta layers keep pageserver replay chains short.
Pinterest Engineering describes rebuilding Guardian, its real-time abuse and risk platform, as Guardian V2 on modern data infrastructure. The post covers the move from a monolithic V1 to separated analytics, signal evaluation, and replay paths using StarRocks, FlinkSQL, and Iceberg, plus stream splitting that cut indexed column scale from O(20K) to O(K).
LinkedIn rebuilt its data lineage platform to capture lineage from executed query plans across more than 15 compute and data-movement systems. The design routes events through Kafka into an append-only log, stores the graph in MySQL with a breadth-first traversal engine, and attaches type, confidence, and granularity to every edge. The post reports a production graph of roughly 6 million datasets with P99 traversal latency near 200 ms.
The post explains how LiveRamp's identity resolution runs as a Databricks packaged clean room, where the provider ships code and the customer runs it against its own governed data. It covers the architecture, including read-only OpenSharing access, serverless compute in a central clean room, the separate customer-only and shared output schemas, and how the design avoids exporting PII. It also contrasts packaged and approval-based clean rooms.
NVIDIA's Ian Buck used SC26 to unveil Nemotron for Science, an open model and chemistry dataset for running research agents locally, along with the Rubin HPC GPU and expanded Vera CPU platform. The talk covers agent examples in GROMACS repair and WRF porting, plus quantum error-correction work including magic state cultivation and a CUDA-Q decoding algorithm. Most results are presenter claims without published methodology.
GitHub's October 2026 availability report covers eight incidents, including Actions job-assignment failures caused by a cloud networking disruption, a migration-driven database overload, a write outage from a misrouted database leader, and a Copilot credential failure. It also details ongoing Azure capacity expansion and reliability work, with each incident broken down by root cause, response, and planned mitigations.
Cloudflare introduces Spend Management in beta, adding opt-in spending caps for Workers and Durable Objects. The post explains the two-stage metering pipeline, the account-partitioned Usage Engine, and the Rating Engine that turns usage into spend. It also describes how caps reach the global network asynchronously through Quicksilver, keeping rating out of the request path.
Whatnot's reliability platform team describes how their Redis-backed concurrency limiter in front of the main API became a scaling bottleneck, with hot keys and every request hitting primaries. They replaced exact synchronous counting with an eventually consistent distributed semaphore that reads from replicas, batches local deltas every 100 ms, shards windows across nodes, and drops TTLs. The post quantifies results (hottest primary near 50% CPU on 8 shards versus over 90% before) and openly...
PlanetScale describes its open-source Heroku Migrator, which uses trigger-based Bucardo replication to move multi-terabyte Postgres databases off standard Heroku plans lacking logical replication. The post details the discovery and Liftoff assessment workflow, and three large-migration failures (COPY restarts from zero, silent network drops, and a delta_check stall) with concrete fixes such as split index rebuilds, TCP keepalive tuning, and a txntime index on track tables.
This second installment in UiPath's Spark series explains how they built real-time alerting over EventHub streams supporting aggregated conditions, multiple window sizes, and many alert types. It details why they chose flatMapGroupsWithState() over built-in window() and withWatermark(), using composite window-keyed grouping, processing-time timeouts, and memory-optimized GroupState schemas that cut state memory from 12GB to 4GB. It also covers decoupling Snowflake writes from EventHub writes to...
Atlassian describes how Jira Search slowed down when customers accumulated nearly 6 million release versions, because resolving version functions like earliestUnreleasedVersion() loaded all versions into memory. The team reduced P99 resolution time from about 18 seconds to 92 milliseconds by resolving only the project-scoped versions, and cut unscoped SQL fetch times from over 30 seconds to under 700 milliseconds by chunking large ID lists to keep Postgres on its primary-key index.
Databricks describes a custom OpenSharing server, deployed as a Databricks App, that exposes AWS HealthLake FHIR data as a shared Unity Catalog catalog without export pipelines. The server uses Lake Formation credentials to resolve Iceberg snapshots and returns pre-signed S3 file references, with Databricks compute reading the data directly and AWS trusting the app via OIDC federation. Governance and Genie One natural-language querying are also covered.
AWS Emulator is a free, open-source local emulator written in Rust that lets developers run AWS CLI, SDK, and coding-agent workloads against a local HTTPS endpoint. The post explains its SigV4 enforcement, embedded mode for Rust test suites, and a fidelity-testing method that checks responses against real AWS. It also notes limits such as the lack of IAM policy evaluation.
Meta describes a vendor-neutral accelerator plugin architecture in lldb-server that lets LLDB debug applications spanning CPUs and GPUs. The design models the CPU and GPU as separate debugger targets coordinated by a plugin that uses internal breakpoints and structured protocol actions, avoiding a forced unified process model. The post notes that most of the common framework has been upstreamed to LLVM, with a mock plugin enabling hardware-independent testing.
Yelp describes how it moved about 1000 Cassandra pods and roughly a petabyte of data from self-managed Kubernetes clusters to Amazon EKS by reattaching EBS volumes to new pods rather than streaming data between nodes. The post covers the rolling PV transplant procedure, using skipReconciliation to stop two operators from fighting over the same ring, and a phantom VolumeAttachment failure that hit about 5% of pod migrations. It also explains how the approach made rollback a simple pointer swap...