Data & Databases

Databases and the pipelines feeding them — Postgres, MySQL, Redis and DynamoDB, plus Spark, ETL, and warehouse and lakehouse architecture. Expect query plans, migration stories, and the occasional hard-won lesson about what stops working at a billion rows.

Loading
How we raised Databricks cluster rate limits by 7.5×
Databricks' Compute Platform team describes how fixed request-rate limits failed to protect long-running cluster-creation workflows, and how adaptive concurrency limits with slot-based tracking, bounded queues, and fair scheduling let them raise default cluster-creation rate limits by 7.5×. The post covers retry storms, distinguishing system degradation from tenant overload, and reported outcomes including over 99.99% fewer rate-limit errors.
How we scaled metrics aggregation by moving durable state into storage
Datadog describes how its metrics aggregation system evolved across three generations, from static partition ownership with local RocksDB state, to dynamically assigned consumers with state snapshots in blob storage, to a best-effort streaming layer where durable reconciliation of partial aggregates is delegated to the timeseries storage engine. The post explains the recovery, scaling, and cost trade-offs behind each design and why the final architecture lets the streaming layer resume at the...
Smooth Cruiser at Lyft: Built for Safety and Transparency
Lyft describes how Smooth Cruiser, its driver safety scoring feature, measures driving signals relative to local road conditions, regional peer groups, and a rolling window of recent rides. The post also covers the data platform behind the score, including ML-based harsh event detection, an Apache Airflow ETL pipeline, and DynamoDB for low-latency score retrieval, along with transparency features that show drivers the rides behind their scores.
DuckDB User Group Workshop on AI-Powered Review Analysis with Hugging Face
A DuckDB User Group workshop built a pipeline over 2.1 million Amazon appliance reviews to find products with falling ratings. DuckDB SQL selects the largest rating drops and the most helpful recent complaints, and only those 40 reviews go to an LLM for theme summarization. The post also covers running the job on Hugging Face and deploying the workflow as a Gradio app with API endpoints.
Introducing Snapshot Migration to Heroku Postgres Advanced
Heroku has released heroku data:pg:migrate, a guided CLI workflow for moving Postgres Classic databases to Heroku Postgres Advanced. The post explains the two-phase design: an online preparation phase, then an offline snapshot copy that blocks writes, restores constraints and indexes, and validates the result. It also covers rollback behavior, rehearsing on a fork to measure downtime, and upgrading the Postgres major version during migration.
Built on object storage, served from cache
Explains how Databricks Lakebase Postgres uses a tiered architecture so that object storage almost never sits on the query read path. Production stats show 99.46% of page reads hit compute-node caches, with pageservers serving most of the rest from local NVMe, and object storage handling only about 1 in 28 million accesses. The post also covers how image and delta layers keep pageserver replay chains short.
Rebuilding Data Lineage at LinkedIn: From Storage to Compute
LinkedIn rebuilt its data lineage platform to capture lineage from executed query plans across more than 15 compute and data-movement systems. The design routes events through Kafka into an append-only log, stores the graph in MySQL with a breadth-first traversal engine, and attaches type, confidence, and granularity to every edge. The post reports a production graph of roughly 6 million datasets with P99 traversal latency near 200 ms.
How packaged clean rooms bring RampID to your lakehouse
The post explains how LiveRamp's identity resolution runs as a Databricks packaged clean room, where the provider ships code and the customer runs it against its own governed data. It covers the architecture, including read-only OpenSharing access, serverless compute in a central clean room, the separate customer-only and shared output schemas, and how the design avoids exporting PII. It also contrasts packaged and approval-based clean rooms.
Ian Buck Kicks Off NVIDIA’s Week at SC26 as Science Enters the Age of Agents
NVIDIA's Ian Buck used SC26 to unveil Nemotron for Science, an open model and chemistry dataset for running research agents locally, along with the Rubin HPC GPU and expanded Vera CPU platform. The talk covers agent examples in GROMACS repair and WRF porting, plus quantum error-correction work including magic state cultivation and a CUDA-Q decoding algorithm. Most results are presenter claims without published methodology.
GitHub availability report: October 2026
GitHub's October 2026 availability report covers eight incidents, including Actions job-assignment failures caused by a cloud networking disruption, a migration-driven database overload, a write outage from a misrouted database leader, and a Copilot credential failure. It also details ongoing Azure capacity expansion and reliability work, with each incident broken down by root cause, response, and planned mitigations.
Introducing Spend Management: Spending limits and alerts for Cloudflare
Cloudflare introduces Spend Management in beta, adding opt-in spending caps for Workers and Durable Objects. The post explains the two-stage metering pipeline, the account-partitioned Usage Engine, and the Rating Engine that turns usage into spend. It also describes how caps reach the global network asynchronously through Quicksilver, keeping rating out of the request path.
Don’t let overload protection be the enemy of scaling
Whatnot's reliability platform team describes how their Redis-backed concurrency limiter in front of the main API became a scaling bottleneck, with hot keys and every request hitting primaries. They replaced exact synchronous counting with an eventually consistent distributed semaphore that reads from replicas, batches local deltas every 100 ms, shards windows across nodes, and drops TTLs. The post quantifies results (hottest primary near 50% CPU on 8 shards versus over 90% before) and openly...
How we migrate TBs from Heroku without logical replication
PlanetScale describes its open-source Heroku Migrator, which uses trigger-based Bucardo replication to move multi-terabyte Postgres databases off standard Heroku plans lacking logical replication. The post details the discovery and Liftoff assessment workflow, and three large-migration failures (COPY restarts from zero, silent network drops, and a delta_check stall) with concrete fixes such as split index rebuilds, TCP keepalive tuning, and a txntime index on track tables.
Real-time Alert in Spark
This second installment in UiPath's Spark series explains how they built real-time alerting over EventHub streams supporting aggregated conditions, multiple window sizes, and many alert types. It details why they chose flatMapGroupsWithState() over built-in window() and withWatermark(), using composite window-keyed grouping, processing-time timeouts, and memory-optimized GroupState schemas that cut state memory from 12GB to 4GB. It also covers decoupling Snowflake writes from EventHub writes to...
Scaling Jira to ~6M Versions
Atlassian describes how Jira Search slowed down when customers accumulated nearly 6 million release versions, because resolving version functions like earliestUnreleasedVersion() loaded all versions into memory. The team reduced P99 resolution time from about 18 seconds to 92 milliseconds by resolving only the project-scoped versions, and cut unscoped SQL fetch times from over 30 seconds to under 700 milliseconds by chunking large ID lists to keep Postgres on its primary-key index.
Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One
Databricks describes a custom OpenSharing server, deployed as a Databricks App, that exposes AWS HealthLake FHIR data as a shared Unity Catalog catalog without export pipelines. The server uses Lake Formation credentials to resolve Iceberg snapshots and returns pre-signed S3 file references, with Databricks compute reading the data directly and AWS trusting the app via OIDC federation. Governance and Genie One natural-language querying are also covered.
Test your applications against AWS locally with AWS Emulator
AWS Emulator is a free, open-source local emulator written in Rust that lets developers run AWS CLI, SDK, and coding-agent workloads against a local HTTPS endpoint. The post explains its SigV4 enforcement, embedded mode for Rust test suites, and a fidelity-testing method that checks responses against real AWS. It also notes limits such as the lack of IAM policy evaluation.
Extending LLDB to Debug Applications That Span CPUs and GPUs
Meta describes a vendor-neutral accelerator plugin architecture in lldb-server that lets LLDB debug applications spanning CPUs and GPUs. The design models the CPU and GPU as separate debugger targets coordinated by a plugin that uses internal breakpoints and structured protocol actions, avoiding a forced unified process model. The post notes that most of the common framework has been upstreamed to LLVM, with a mock plugin enabling hardware-independent testing.
Zero-Copy Migration: How Yelp Moved Its Cassandra Fleet Between Kubernetes Clusters
Yelp describes how it moved about 1000 Cassandra pods and roughly a petabyte of data from self-managed Kubernetes clusters to Amazon EKS by reattaching EBS volumes to new pods rather than streaming data between nodes. The post covers the rolling PV transplant procedure, using skipReconciliation to stop two operators from fighting over the same ring, and a phantom VolumeAttachment failure that hit about 5% of pod migrations. It also explains how the approach made rollback a simple pointer swap...