PlanetScale Engineering engineering blog

Loading
How we migrate TBs from Heroku without logical replication
PlanetScale describes its open-source Heroku Migrator, which uses trigger-based Bucardo replication to move multi-terabyte Postgres databases off standard Heroku plans lacking logical replication. The post details the discovery and Liftoff assessment workflow, and three large-migration failures (COPY restarts from zero, silent network drops, and a delta_check stall) with concrete fixes such as split index rebuilds, TCP keepalive tuning, and a txntime index on track tables.
Designing a reliable unplanned failover
Shlomi Noach describes how Neki's control plane detects a failed Postgres primary during an unplanned failover. Instead of leases, it asks replicas about replication state via pg_stat_wal_receiver, and uses timeouts to catch frozen processes. It fences the old primary by altering primary_conninfo, then promotes the replica with the highest received WAL position to preserve acknowledged commits.
The hidden latency inside your database
The post compares network-attached and locally attached storage for cloud databases, tracing how EBS latency and IOPS limits shape performance. It explains how PlanetScale Metal uses local NVMe disks with cross-zone replication to keep durability while removing the network hop on reads and most writes.
Designing a planned switchover for minimum disruption
The post describes how Neki performs planned Postgres primary switchovers with minimal disruption, using preflight validation, query buffering at the routers, and a read-only transition on the old primary. It explains how a final WAL position is established without stopping Postgres, how synchronous replication is preserved through promotion, and how a timeout triggers a fast rollback.
What was the turning point for Postgres?
PlanetScale traces the turning points that made PostgreSQL the default database for new applications, from its 1990s origins as Postgres95 and its community governance to Heroku's role in cloud adoption, JSONB's response to document databases, and the MySQL ownership changes that opened the door. The post is mostly historical context rather than hands-on implementation detail, and it closes by noting horizontal sharding as an unsolved gap.
Shard design doesn't start from your tables
Rohit Nayak explains how to design shard groups in Neki, PlanetScale's sharded Postgres, starting from the queries and transactions that matter most rather than from individual tables. The post covers co-locating related tables on a shared shard key, replicating reference tables, using SQL expressions in shard indexes, and adding global secondary indexes for lookups that lack a shard key.
I don't trust ON DELETE CASCADE
The post argues that ON DELETE CASCADE can cause production outages in Postgres, because large cascading deletes hold locks that block migrations and writes, and can generate hundreds of gigabytes of WAL. It also covers unindexed foreign keys that trigger sequential scans, and logical replication gaps where cascades are skipped on subscribers or fail on publishers without replica identity. The author recommends soft deletes with batched background reaping and short, bounded transactions.
Sequences and other volatiles in Neki
Mats Rydberg explains how Neki, PlanetScale's sharded Postgres, evaluates stable and volatile expressions such as now() and nextval() in its distributed query planner. The post covers Postgres expression volatility, table-expression cardinality, and why sequence allocation is routed through a central authoritative shard. It also walks through router-only evaluation and the trade-offs of pushing work down to shards.
What gets TOASTed in Postgres
Ahmed Darwich explains how PostgreSQL TOAST decides when to compress or move variable-length values, including its 2,032-byte threshold, storage strategies, and chunking behavior. The post benchmarks pglz versus LZ4 and shows when compression improves storage and when uncompressed data is faster for previews or random access.
TIN v1.0.6: 2-3x faster, with support for stemming
PlanetScale's TIN v1.0.6 adds language-specific stemming for 18 languages, deterministic top-k tiebreakers, and substantial query-performance improvements. The post explains its indexing and BM25 scoring trade-offs and presents reproducible benchmarks against ParadeDB across x86-64 and ARM64 systems.
What today's software owes to Bronze Age clocks
Simeon Griggs explains how leaky and token buckets implement low-cost admission control, rate limiting work by capacity and time. The post applies these concepts to PostgreSQL, showing how query-plan estimates, resource budgets, burst limits, and expected failure states can protect databases from overload.
Designing Neki for performance
The post explains how Neki improves sharded PostgreSQL performance by preserving wire-format messages and decoding data lazily. It details composable representations for routing, limits, sorting, projection, joins, and aggregation, showing how byte-level reuse avoids unnecessary allocations, parsing, and serialization.
Handling hot shards
The post explains how tenant growth and changing access patterns can make an initially sensible shard key produce hot shards. It compares vertical scaling and tenant isolation with table-specific resharding, showing how declarative topology and online data movement can redistribute load without downtime.
When to choose x86-64 vs aarch64
Ahmed Darwich compares ARM and x86-64 cloud CPUs for PostgreSQL workloads, explaining vCPU differences, parallelism, clock speed, vector instructions, and extension compatibility. He outlines when each architecture is preferable and how to benchmark or migrate between them safely.
TIN Postgres search is faster, better, and cheaper
Patrick Reynolds demonstrates how TIN accelerates Postgres full-text search with immediate keystroke queries, wildcard autocomplete, fuzzy matching, stop-word support, and exact COUNT(*) results. The comparison reports 11ms ranking, 213,447 exact matches versus an estimated 3,400, and substantially lower infrastructure costs.
Anatomy of a (Postgres) search engine
The post explains how full-text search engines use inverted indexes, postings lists, positional and frequency data, BM25 scoring, and segmented storage. It then details the additional challenges of implementing these structures as a transactional, mutable Postgres index, including visibility, VACUUM, tombstones, merging, and query planning.
Blocking cutovers to save replication slots
Simeon Griggs explains how PostgreSQL logical replication slots can become invalid during a replica promotion, causing downstream CDC consumers to miss changes. The post details the failover, hot_standby_feedback, and sync_replication_slots settings, along with PlanetScale’s safeguards for blocking unsafe cutovers.
The architecture of Neki
PlanetScale explains Neki’s architecture for sharding ordinary PostgreSQL across a single client-facing endpoint. The post details its Router, Sidecar, Admin, Operator, topology, and Replicator components, including failover, distributed query execution, OID consistency, resharding, and online schema changes.
Introducing Lead: TIN-compatible full-text search for CI
PlanetScale introduces Lead, a deliberately slow, TIN-compatible PostgreSQL full-text search extension for CI, development, and staging. It preserves TINQL queries, tokenization, BM25 scoring, highlighting, transactional visibility, and durability while using full table scans, making it practical for small test datasets rather than production workloads.
Introducing TIN: full-text search for Postgres
PlanetScale introduces TIN, a full-text search index for Postgres supporting Boolean, phrase, fuzzy, wildcard, regex, COUNT, and BM25 queries. The post details its benchmark results and explains how ctid-based bitmap indexing, vectorization, MVCC handling, and segment merging deliver substantially higher performance than competing indexes.