DuckDB engineering blog

Loading
DuckDB User Group Workshop on AI-Powered Review Analysis with Hugging Face
A DuckDB User Group workshop built a pipeline over 2.1 million Amazon appliance reviews to find products with falling ratings. DuckDB SQL selects the largest rating drops and the most helpful recent complaints, and only those 40 reviews go to an LLM for theme summarization. The post also covers running the job on Hugging Face and deploying the workflow as a Gradio app with API endpoints.
Local Analytics Doesn’t Need a Database Server
The post describes how DuckDB, run in the browser via WebAssembly or embedded in desktop apps, enables analytics without a database server. It walks through the SQLRooms toolkit, a dynamic vector-tiling SQL pipeline for large geospatial datasets, and the local-first design behind AI-assisted analysis.
Ducks in a Column
This post explains how DuckDB's columnar storage shapes query performance, covering querying Parquet and CSV files in place, column-level SQL such as EXCLUDE, REPLACE and COLUMNS, aggregation shortcuts, window functions, and ASOF joins. It also gives practical guidance on writing well-organized Parquet through partitioning, sort order, narrow types, and checking query plans with EXPLAIN ANALYZE.
How JetBrains Built Cross-Database Queries into DataGrip with DuckDB
DataGrip, JetBrains' database IDE, enables cross-database joins by embedding DuckDB and connecting it through custom table functions rather than building its own query engine. The post explains why DuckDB beat Apache Calcite, a PSI-based approach, and H2, and outlines planned work on optimizations and file-based sources.
Bringing Local Analytics to the Quant's Desktop
Ryan Hamilton, founder of TimeStored, describes how DuckDB became the embedded analytics layer in his QStudio and Pulse tools for smaller quant teams. QStudio caches expensive remote query results locally as Parquet, and the post notes Java client integration issues that were resolved by rebuilding the Appender. The article also covers DuckDB's role in his PeachQ project, an open-source Q implementation.
From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB
Data engineer Matt Martin traces his path from Excel and SQL Server reporting to big data tools like Spark, then to DuckDB and DuckLake. The post contrasts Iceberg's file-based metadata overhead with DuckLake's SQL-backed catalog and previews DuckDB v2.0 alpha features, including expression-based ATTACH and asynchronous S3 I/O.
Gems of DuckDB 2.0.0
This post from the DuckDB team highlights smaller features in the DuckDB 2.0.0 LTS release, including a Python appender, CLI improvements such as .manual and .history, and SQL additions like SECURE VIEW, setval, session-variable parameter defaults, and PARTITION_PATH for custom partition layouts. It includes code examples and explains the security and write-path behavior of several features. Most of the content is a feature tour of the company's own release.
Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”
DuckDB 2.0.0 LTS adds a polished Quack client-server protocol with a new CONNECT statement, asynchronous I/O for Parquet, CSV and native storage, and a fully integrated shredded VARIANT type. It also makes a PEG parser the default in place of the YACC-generated one, adds triggers that can maintain materialized views incrementally, and removes the ICU dependency. The storage format is no longer readable by older versions, so the post recommends testing on a shadow system before upgrading...
From JSON Strings to Typed Columns
DuckDB's team explains how to turn a JSON text column already stored in a table into typed columns directly in SQL, without writing it to a file first. The post covers inferring a structure with json_group_structure, overriding types with json_merge_patch, and expanding the result with json_transform and unnest, including recursive unnesting with keep_parent_names to avoid name collisions and handling of values that fail conversion.
Benchmarking DuckDB v2.0-dev on NVIDIA Vera
The DuckDB team benchmarks a preview build of DuckDB on NVIDIA's Vera arm64 CPU against an Intel Xeon 6 x86 baseline using the full TPC-H workload at scale factor 1,000. The post documents the hardware setup, thread tuning, numactl pinning, and reproducible scripts, reporting a median QphH@Size advantage of roughly 46% for Vera and attributing it to memory bandwidth, low latency, and large caches for hash tables.
What Is Inside an Iceberg Table?
A walkthrough of how an Apache Iceberg table is structured on disk, from the catalog-held metadata pointer through the JSON metadata file, Avro manifest list and manifests, down to Parquet data and position delete files. The post uses DuckDB SQL against a local Iceberg REST fixture to inspect snapshots, column bounds, partition stats, and merge-on-read deletes, and explains when this inspection helps diagnose slow or unexpected queries.
Agent Mode in the DuckDB CLI
DuckDB’s v2.0 CLI adds an agent mode designed for AI coding agents, replacing padded tables with compact Markdown, making truncation explicit, returning structured errors, and stopping runaway queries early. An evaluation on 132 TPC-H runs found a 59% reduction in output tokens while preserving correct answers.
A DuckDB Database with No Data in It
DuckDB can serve as a compact, shareable catalog containing only view definitions over remote Parquet data. The post demonstrates how to build and host such a database, attach it read-only, preserve stable schemas, and hide changes to the underlying data layout from consumers.
Importing Data using Java Table Functions
The authors compare MongoDB-to-DuckDB ingestion strategies in Java, including JDBC batch inserts, the Appender API, and custom table functions. They show how vectorized output, parallel execution, and disabling insertion-order preservation reduce import overhead, ultimately delivering a robust pure-Java pipeline with near-native performance.
Faster String Aggregations with Dimension Tables
The post explains how encoding repeated, long strings as sorted, narrow integer keys in dimension tables can make DuckDB aggregations faster and more memory-efficient. It covers schema construction, query patterns, ENUM alternatives, maintenance trade-offs, benchmarking, and cases where encoding does not help.
Jev and DuckDB: Plain-English Conditions in SQL
The post explains how Jev brings typed, plain-English judgments to DuckDB, enabling SQL filtering, classification, ranking, and scoring over semi-structured data. It compares community extensions, batching and caching strategies, performance trade-offs, costs, and privacy considerations.
Announcing DuckDB 1.5.6
DuckDB 1.5.6 is a patch release containing correctness fixes, crash and recovery fixes, security hardening, and C API improvements. The post also previews DuckDB 2.0 performance on Windows, reporting a sixfold speedup on a TPC-H benchmark through compiler and allocator changes.
DuckDB and Hugging Face: Querying Datasets Directly
The post explains how DuckDB’s hf:// protocol queries Hugging Face datasets remotely without first downloading them. It demonstrates globbing files, pinning revisions, accessing private datasets, filtering into local Parquet files, and joining Hub data with local tables.
DuckDB Now Ships inside dbt v2
The post explains how dbt v2 integrates DuckDB through its Rust-based Fusion engine, eliminating the separate adapter installation. It covers setup, DuckLake and Iceberg catalogs, querying dbt metadata as Parquet, column-level lineage, pinned DuckDB versions, and migration from dbt v1.
Persistent Databases in the Browser with DuckDB-Wasm and OPFS
The post explains how to use DuckDB-Wasm with the browser’s Origin Private File System (OPFS) for persistent databases and data files. It covers version pitfalls, WAL and checkpoint durability, automatic versus manual file handling, caching remote data, and exporting databases or Parquet files.