Databricks engineering blog

Loading
How we raised Databricks cluster rate limits by 7.5×
Databricks' Compute Platform team describes how fixed request-rate limits failed to protect long-running cluster-creation workflows, and how adaptive concurrency limits with slot-based tracking, bounded queues, and fair scheduling let them raise default cluster-creation rate limits by 7.5×. The post covers retry storms, distinguishing system degradation from tenant overload, and reported outcomes including over 99.99% fewer rate-limit errors.
The new Azure Databricks on Azure Government is now in Beta
Databricks announces that Azure Databricks on Azure Government has entered Beta, bringing Unity Catalog, Databricks SQL, and serverless compute to U.S. government regions with a Compliance Security Profile enabled by default. The post outlines the roadmap for AI features and FedRAMP Class D and IL5 authorization expected in early 2027, and gives migration guidance for teams on the legacy environment.
How packaged clean rooms bring RampID to your lakehouse
The post explains how LiveRamp's identity resolution runs as a Databricks packaged clean room, where the provider ships code and the customer runs it against its own governed data. It covers the architecture, including read-only OpenSharing access, serverless compute in a central clean room, the separate customer-only and shared output schemas, and how the design avoids exporting PII. It also contrasts packaged and approval-based clean rooms.
Built on object storage, served from cache
Explains how Databricks Lakebase Postgres uses a tiered architecture so that object storage almost never sits on the query read path. Production stats show 99.46% of page reads hit compute-node caches, with pageservers serving most of the rest from local NVMe, and object storage handling only about 1 in 28 million accesses. The post also covers how image and delta layers keep pageserver replay chains short.
Azure Databricks sessions at Microsoft Ignite 2026
Azure Databricks announces its sponsorship and session lineup at Microsoft Ignite 2026, covering governed AI, agents, model fine-tuning with NVIDIA Nemotron, and Genie integrations in Microsoft Teams and Copilot Cowork. The post is primarily a conference logistics and event promotion piece with booth locations, executive meeting details, and session schedules, offering little technical depth beyond session titles.
Branch, sync, serve: What our Lakebase Hackathon taught us about building on Lakebase
Databricks recaps its Lakebase hackathon, where eleven customer projects in two tracks used database branching, synced tables, and real-time serving. Highlights include Lyft's branch-based agent testing, which resolved 10 of 11 support backlogs versus 2 with up-front planning, and sub-five-second preview environments built on zero-copy branches. The post shows how teams use Lakebase to replace search indexes, staging databases, and hand-built sync jobs.
Teach Genie One your workflows: The ultimate guide for skills in Genie One
Databricks explains how to build reusable workflow skills for Genie One by mapping a task, defining its steps, checks and output format, and testing it against a golden reference. It also covers when to separate strict rules from judgment, how to chain skills together, and how to govern and schedule them through Unity Gateway.
Clean data was never enough: How IFS Copperleaf taught AI its business with Databricks Genie One
Databricks describes how IFS Copperleaf built IONA, a company-wide AI assistant on Databricks Genie One, that answers questions over raw data from nearly 30 systems while business teams define metrics through governed Skills. The post highlights a model-only-what-is-high-stakes approach, owner-approved definitions, and reported adoption results of a 70% activation rate and cost recovery within four months.
Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One
Databricks describes a custom OpenSharing server, deployed as a Databricks App, that exposes AWS HealthLake FHIR data as a shared Unity Catalog catalog without export pipelines. The server uses Lake Formation credentials to resolve Iceberg snapshots and returns pre-signed S3 file references, with Databricks compute reading the data directly and AWS trusting the app via OIDC federation. Governance and Genie One natural-language querying are also covered.
Tabular foundation models on Databricks AI Runtime
The post explains how tabular foundation models (TFMs) use in-context learning and row-compression architectures such as TabPFNv2 and TabICLv1, and compares their licensing and GPU inference costs. It then describes how Databricks AI Runtime addresses GPU provisioning, observability, and data-governance challenges for TFM experimentation and fine-tuning.
Introducing the Databricks AI Index: The state of enterprise AI tokenomics
Databricks introduces the AI Index, a public resource combining anonymized enterprise usage data from Unity Gateway with coding-agent benchmarks. The post reports that open-weight model traffic rose from 7.3% to 30.9% and that cost per completed task can diverge sharply from token price, with one model costing about 32x more than another at equal success rates. It also describes the benchmark methodology, including hidden tests and publicly listed prices.
Build a real-time next-best-offer recommender using Databricks Feature Views
The post walks through a Databricks Solution Accelerator that uses Feature Views to define batch and Kafka streaming features once for both training and serving, reducing train/serve skew. It reports benchmark results for a real-time next-best-offer recommender, including latency, feature freshness, and throughput, along with a concurrency sizing formula and the measurement limitations.
Databricks is a Leader and positioned highest in execution in the 2026 Gartner® Magic Quadrant™ for Data Lakehouse Platforms
Databricks reports its Leader placement with the highest execution score in Gartner's 2026 Magic Quadrant for Data Lakehouse Platforms and its Visionary placement in the Cloud Operational Database Management Systems quadrant. The post describes the lakebase concept, a transactional database built on open lake storage, and the LTAP architecture, which aims to unify operational and analytical processing without copying data through pipelines. Most of the content is analyst recognition and...
Apache Spark vs Apache Flink: which should you use?
The post compares Apache Spark and Apache Flink for streaming workloads, arguing that Spark Real-Time Mode now reaches millisecond latency on the same engine used for batch, SQL and ML. It reviews a vendor-run benchmark at 10,000 records per second with p99 results for stateless, aggregation and join workloads, and notes that the benchmark pairs Spark's worst run against Flink's best. It also lists cases where Flink still fits better, such as very large keyed state, complex event processing and...
What is Real-Time Mode in Apache Spark?
Explains Real-Time Mode, a Spark Structured Streaming trigger that processes records continuously for millisecond end-to-end latency by replacing micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle. The post covers configuration requirements, supported sources, sinks and stateful operators, published p99 latency benchmarks, and guidance on when to choose it over micro-batch or Apache Flink.
OpenAI and Databricks: Enterprise AI on your governed data
Databricks and OpenAI describe integrations that let enterprises use GPT models, Codex, and ChatGPT on governed Databricks data. The post covers the Genie Ontology context layer, Unity Gateway routing and budgeting for Codex, and deploying agents with Model Serving and Databricks Apps without managing a separate API key. Most of the content is a feature and partnership overview with limited implementation depth.
Eclipse Aerospace simplifies flight-data ingestion with Databricks
Eclipse Aerospace replaced a multi-stage Lambda and Avro ingestion pipeline with a PySpark custom data source that wraps a Rust decoder and passes records to Spark through Apache Arrow without copying. The write-up covers the trade-offs among UDFs, Scala decoders, and intermediary formats, plus reported latency, storage, and observability gains.
Genie in Slack is now Generally Available: Governed AI cowork, right where work happens
Databricks announces general availability of the Genie app for Slack, which lets users DM Genie or @mention it in channels to get answers grounded in Unity Catalog data and connected sources. Answers run under the asker's permissions, and MCP connectors can make changes only after human approval. Admins configure per-channel routing to Genie One or specific Genie Agents, and Private Link is supported through context-based ingress policies.
Dispelling LTAP myths
Databricks explains why LTAP, the Lakebase design that treats the lake as the durable tier for Postgres data, is not CDC replication. It argues that tiering differs from replication, that one table can have a lossless heap and Parquet encoding, and that an LSN-based merge gives consistent analytical reads. It also contrasts logical replication costs such as WAL reads, replication slots, and DDL handling with page-based transcoding.