A DuckDB User Group workshop built a pipeline over 2.1 million Amazon appliance reviews to find products with falling ratings. DuckDB SQL selects the largest rating drops and the most helpful recent complaints, and only those 40 reviews go to an LLM for theme summarization. The post also covers running the job on Hugging Face and deploying the workflow as a Gradio app with API endpoints.
DuckDB engineering blog
Loading
The post describes how DuckDB, run in the browser via WebAssembly or embedded in desktop apps, enables analytics without a database server. It walks through the SQLRooms toolkit, a dynamic vector-tiling SQL pipeline for large geospatial datasets, and the local-first design behind AI-assisted analysis.
This post explains how DuckDB's columnar storage shapes query performance, covering querying Parquet and CSV files in place, column-level SQL such as EXCLUDE, REPLACE and COLUMNS, aggregation shortcuts, window functions, and ASOF joins. It also gives practical guidance on writing well-organized Parquet through partitioning, sort order, narrow types, and checking query plans with EXPLAIN ANALYZE.
DataGrip, JetBrains' database IDE, enables cross-database joins by embedding DuckDB and connecting it through custom table functions rather than building its own query engine. The post explains why DuckDB beat Apache Calcite, a PSI-based approach, and H2, and outlines planned work on optimizations and file-based sources.
Ryan Hamilton, founder of TimeStored, describes how DuckDB became the embedded analytics layer in his QStudio and Pulse tools for smaller quant teams. QStudio caches expensive remote query results locally as Parquet, and the post notes Java client integration issues that were resolved by rebuilding the Appender. The article also covers DuckDB's role in his PeachQ project, an open-source Q implementation.
Data engineer Matt Martin traces his path from Excel and SQL Server reporting to big data tools like Spark, then to DuckDB and DuckLake. The post contrasts Iceberg's file-based metadata overhead with DuckLake's SQL-backed catalog and previews DuckDB v2.0 alpha features, including expression-based ATTACH and asynchronous S3 I/O.
This post from the DuckDB team highlights smaller features in the DuckDB 2.0.0 LTS release, including a Python appender, CLI improvements such as .manual and .history, and SQL additions like SECURE VIEW, setval, session-variable parameter defaults, and PARTITION_PATH for custom partition layouts. It includes code examples and explains the security and write-path behavior of several features. Most of the content is a feature tour of the company's own release.
DuckDB 2.0.0 LTS adds a polished Quack client-server protocol with a new CONNECT statement, asynchronous I/O for Parquet, CSV and native storage, and a fully integrated shredded VARIANT type. It also makes a PEG parser the default in place of the YACC-generated one, adds triggers that can maintain materialized views incrementally, and removes the ICU dependency. The storage format is no longer readable by older versions, so the post recommends testing on a shadow system before upgrading...
DuckDB's team explains how to turn a JSON text column already stored in a table into typed columns directly in SQL, without writing it to a file first. The post covers inferring a structure with json_group_structure, overriding types with json_merge_patch, and expanding the result with json_transform and unnest, including recursive unnesting with keep_parent_names to avoid name collisions and handling of values that fail conversion.
The DuckDB team benchmarks a preview build of DuckDB on NVIDIA's Vera arm64 CPU against an Intel Xeon 6 x86 baseline using the full TPC-H workload at scale factor 1,000. The post documents the hardware setup, thread tuning, numactl pinning, and reproducible scripts, reporting a median QphH@Size advantage of roughly 46% for Vera and attributing it to memory bandwidth, low latency, and large caches for hash tables.
A walkthrough of how an Apache Iceberg table is structured on disk, from the catalog-held metadata pointer through the JSON metadata file, Avro manifest list and manifests, down to Parquet data and position delete files. The post uses DuckDB SQL against a local Iceberg REST fixture to inspect snapshots, column bounds, partition stats, and merge-on-read deletes, and explains when this inspection helps diagnose slow or unexpected queries.
DuckDB’s v2.0 CLI adds an agent mode designed for AI coding agents, replacing padded tables with compact Markdown, making truncation explicit, returning structured errors, and stopping runaway queries early. An evaluation on 132 TPC-H runs found a 59% reduction in output tokens while preserving correct answers.
DuckDB can serve as a compact, shareable catalog containing only view definitions over remote Parquet data. The post demonstrates how to build and host such a database, attach it read-only, preserve stable schemas, and hide changes to the underlying data layout from consumers.
The authors compare MongoDB-to-DuckDB ingestion strategies in Java, including JDBC batch inserts, the Appender API, and custom table functions. They show how vectorized output, parallel execution, and disabling insertion-order preservation reduce import overhead, ultimately delivering a robust pure-Java pipeline with near-native performance.
The post explains how encoding repeated, long strings as sorted, narrow integer keys in dimension tables can make DuckDB aggregations faster and more memory-efficient. It covers schema construction, query patterns, ENUM alternatives, maintenance trade-offs, benchmarking, and cases where encoding does not help.
The post explains how Jev brings typed, plain-English judgments to DuckDB, enabling SQL filtering, classification, ranking, and scoring over semi-structured data. It compares community extensions, batching and caching strategies, performance trade-offs, costs, and privacy considerations.
DuckDB 1.5.6 is a patch release containing correctness fixes, crash and recovery fixes, security hardening, and C API improvements. The post also previews DuckDB 2.0 performance on Windows, reporting a sixfold speedup on a TPC-H benchmark through compiler and allocator changes.
The post explains how DuckDB’s hf:// protocol queries Hugging Face datasets remotely without first downloading them. It demonstrates globbing files, pinning revisions, accessing private datasets, filtering into local Parquet files, and joining Hub data with local tables.
The post explains how dbt v2 integrates DuckDB through its Rust-based Fusion engine, eliminating the separate adapter installation. It covers setup, DuckLake and Iceberg catalogs, querying dbt metadata as Parquet, column-level lineage, pinned DuckDB versions, and migration from dbt v1.
The post explains how to use DuckDB-Wasm with the browser’s Origin Private File System (OPFS) for persistent databases and data files. It covers version pitfalls, WAL and checkpoint durability, automatic versus manual file handling, caching remote data, and exporting databases or Parquet files.