DevOps & Cloud

Shipping and running software: Kubernetes and containers, CI/CD pipelines, Terraform and infrastructure-as-code, and the observability stack that tells you when any of it has gone wrong. Heavy on AWS, Azure and GCP, and on SRE practice — capacity, reliability, and the postmortem afterwards.

Loading
How we raised Databricks cluster rate limits by 7.5×
Databricks' Compute Platform team describes how fixed request-rate limits failed to protect long-running cluster-creation workflows, and how adaptive concurrency limits with slot-based tracking, bounded queues, and fair scheduling let them raise default cluster-creation rate limits by 7.5×. The post covers retry storms, distinguishing system degradation from tenant overload, and reported outcomes including over 99.99% fewer rate-limit errors.
How we scaled metrics aggregation by moving durable state into storage
Datadog describes how its metrics aggregation system evolved across three generations, from static partition ownership with local RocksDB state, to dynamically assigned consumers with state snapshots in blob storage, to a best-effort streaming layer where durable reconciliation of partial aggregates is delegated to the timeseries storage engine. The post explains the recovery, scaling, and cost trade-offs behind each design and why the final architecture lets the streaming layer resume at the...
Introducing eve 1.0
Vercel announces eve 1.0, an open-source framework for production AI agents, and details the engineering behind it. Median hosted turn latency fell from 3.04s to 414ms through a single-orchestrator workflow design and WebSocket event writes, and estimated workflow cost for a 50-turn session dropped about 76% by storing only changed conversation data. The post also covers OpenTelemetry-based tracing, multi-tenant agent configuration, and dynamic tool loading.
Using a "chaos loop" to harden the Claude Docs launch: 60 server errors across billions of requests
Vivek Nair describes how a team hardened the Claude Docs backend before its September 2026 launch by running Claude in a long-lived loop against a chaos-testing staging cluster. Key lessons include using copy-on-write to fix per-keystroke memory growth, which cut projected pods from over 500 to about 50, and choosing Rust with minimal dependencies so telemetry traces back to owned code. The launch saw 60 server errors across 2 billion requests with no rollbacks.
How AI Slashed 600-Test-Failure Triage From Days to 90 Minutes
Salesforce engineers describe building AI-Assisted Test Failure Buddy, a Python and FastMCP tool exposing eight MCP tools to Claude Code and Cursor, to triage over 600 test failures caused by retiring the SOAP API login mechanism. The post explains how deterministic code handles API calls, batching, signature matching, and reassignment while AI interprets unmatched stack traces and ambiguous ownership, reducing estimated two to three days of triage to about 90 minutes.
Quad9’s withdrawal from France: A canary in the coal mine
Cloudflare argues that Quad9's withdrawal from France, after a court imposed daily penalties for not blocking domains via its public DNS resolver, shows the harms of mandating DNS-level blocking. The post explains why domain-level DNS blocking is easy to bypass, causes collateral damage to legitimate content on the same domain, and fails silently for users. It warns that forcing public resolvers out of markets weakens the open Internet.
Built on object storage, served from cache
Explains how Databricks Lakebase Postgres uses a tiered architecture so that object storage almost never sits on the query read path. Production stats show 99.46% of page reads hit compute-node caches, with pageservers serving most of the rest from local NVMe, and object storage handling only about 1 in 28 million accesses. The post also covers how image and delta layers keep pageserver replay chains short.
Rebuilding Data Lineage at LinkedIn: From Storage to Compute
LinkedIn rebuilt its data lineage platform to capture lineage from executed query plans across more than 15 compute and data-movement systems. The design routes events through Kafka into an append-only log, stores the graph in MySQL with a breadth-first traversal engine, and attaches type, confidence, and granularity to every edge. The post reports a production graph of roughly 6 million datasets with P99 traversal latency near 200 ms.
Scheduling the removal of the kube-proxy IPVS backend
Dan Winship of Red Hat explains why the Kubernetes kube-proxy ipvs backend is being deprecated and scheduled for removal in Kubernetes 1.40, covering its original data-plane and control-plane performance motivations and its later compatibility, batch-update, and scheduler limitations. The post recommends migrating to the nftables mode, or to the substantially improved iptables mode on older clusters, and gives concrete configuration guidance for kubeadm clusters.
From Predictive to Generative – How Michelangelo Accelerates Uber’s AI Journey
Uber's engineer write-up traces the evolution of its centralized ML platform, Michelangelo, across three phases from 2016 to the present. It details the shift from tree models to deep learning, including the move from Spark to Ray-based trainers, elastic Horovod, and Triton model serving, along with the platform changes that followed. It also covers model quality tiering, Canvas for code-driven pipelines, and the Gen AI Gateway for LLM access.
How the World’s Most Advanced AI Infrastructure Supply Chain Invents the Future
NVIDIA's operations leader Deb Shoquist describes how the company builds supplier relationships to scale AI infrastructure production. The post walks through a GB200 compute board test-handler throughput failure, the cross-team response that followed, and a traceability process that cut chip-to-system lead time from five weeks to two. It is written by NVIDIA about its own manufacturing operations.
Netmap caching: keeping Tailscale up on iffy connections
Tailscale introduces netmap caching, a client feature that saves a copy of the tailnet network map to disk so nodes start faster and can still reach peers when the coordination server is unavailable. The post explains the netmap contents, how only changed chunks are written, and how DISCO keys are exchanged through encrypted DERP-relayed WireGuard messages to re-establish direct connections without the coordination server. It also notes the staleness tradeoff over time.
Agents are users, so your software needs evals
The post argues that software used by AI agents needs evals, meaning repeated, graded runs rather than single tests, and lays out an agentic testing pyramid. It walks through building an end-to-end eval with Harbor on Vercel Sandbox and AI Gateway, including choosing between pass@k and pass^k, designing tasks and verifiers, and analyzing traces to find failure causes.
How the Claude Code team thinks about verification
The post explains how the Claude Code team uses end-to-end, property, and differential tests to let coding agents verify their work with evidence. It also describes the practices used to keep verifiers honest, including a commit gate, a reliability checklist tied to diffs, and evaluations that measure review recall against past incidents.
Throwing Out The Phonebook
Fly.io describes replacing its 6PN private network's fleet-wide phonebook, a BPF map of every migrated Machine kept synchronized via Corrosion, with a lazy design. VXLAN unknown-address events trigger on-demand route lookups by s6pnforwarderd, leaving a table of 100 to 200 routes per host. The post covers fwmark-gated fail-closed filtering, the tradeoff of a roughly one second first-connection latency, and a dual-running rollout across the fleet.
GitHub availability report: October 2026
GitHub's October 2026 availability report covers eight incidents, including Actions job-assignment failures caused by a cloud networking disruption, a migration-driven database overload, a write outage from a misrouted database leader, and a Copilot credential failure. It also details ongoing Azure capacity expansion and reliability work, with each incident broken down by root cause, response, and planned mitigations.
Cuttlefish on Mac: Running Android’s Virtual Device Framework Natively on Apple Silicon
Meta's Reality Labs ported Android's Cuttlefish virtual device framework to macOS on Apple silicon, consolidating a split Goldfish and Cuttlefish setup into one high-fidelity platform. The post details the technical changes: making QEMU's vhost-user path portable beyond Linux, increasing memory-region tracking slots in the HVF backend, adding a native Cocoa display, and handling JIT entitlements and code signing.
Don’t let overload protection be the enemy of scaling
Whatnot's reliability platform team describes how their Redis-backed concurrency limiter in front of the main API became a scaling bottleneck, with hot keys and every request hitting primaries. They replaced exact synchronous counting with an eventually consistent distributed semaphore that reads from replicas, batches local deltas every 100 ms, shards windows across nodes, and drops TTLs. The post quantifies results (hottest primary near 50% CPU on 8 shards versus over 90% before) and openly...
Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One
Databricks describes a custom OpenSharing server, deployed as a Databricks App, that exposes AWS HealthLake FHIR data as a shared Unity Catalog catalog without export pipelines. The server uses Lake Formation credentials to resolve Iceberg snapshots and returns pre-signed S3 file references, with Databricks compute reading the data directly and AWS trusting the app via OIDC federation. Governance and Genie One natural-language querying are also covered.