Rootconf 2026 Annual Conference

Session title
Replacing gRPC in CockroachDB: Lessons from RPC migration in a Distributed SQL Database
One-line summary
This talk shares why and how CockroachDB, a distributed SQL database, migrated from the industry-standard gRPC protocol to a custom RPC based on DRPC and rolled it out safely.

What problem are you addressing?
In a distributed database, the internal RPC layer is on the hot path of nearly everything: Raft consensus, replication, and distributed SQL execution. CockroachDB used gRPC for this. However, during internal testing with newer gRPC releases, we observed that gRPC’s HTTP/2 framing complexity, allocation overhead and the resulting GC pressure caused performance regression in CockroachDB - observable in the form of lowered throughput (ops/sec) and allocations (B/op) in our CPU and heap profiles.

DRPC, a lightweight open-source RPC framework originally built by Storj, did much better in our benchmarks. However, it was minimal by design - it had no interceptors, no stream multiplexing (only one active stream per connection), no flow control, no compression, and only basic metadata, error-code and observability support. To adopt it into CockroachDB, we forked DRPC and rebuilt those features ourselves without impacting the performance that made us switch. We also changed the wire protocol under a live database: across mixed-version clusters during rolling upgrades, with no downtime.

This talk covers what this exercise cost us and what we got out of it. It ends with the measured results, including the places where the lighter protocol wasn’t a clear win.

Who is the intended audience?

• Distributed systems and database engineers
• Infrastructure teams evaluating RPC framework choices
• Platform engineers responsible for migrating critical infrastructure

Level
Intermediate

Practical takeaways

• How to tell if the RPC layer is worth replacing.
• A checklist of what gRPC gives for free (interceptors, multiplexing, flow control, compression, error semantics) and what it took us to rebuild each. Use it to decide between building, adopting or forking.
• A playbook for changing a wire protocol in production. A handshake for protocol negotiation, two stacks running side by side, version gates, and the metrics to watch during rollout.

What will you share?

1. Why leave gRPC?

• Profiling evidence: HTTP/2 framing, flow-control bookkeeping showed up as measurable CPU cost.
• Per-message allocations in gRPC’s framing and compression paths drove GC pressure.
• Alternatives we considered (tuning gRPC, a custom codec, other frameworks) and why we forked DRPC instead.

2. Costs of going with a “minimal” RPC protocol

• What upstream DRPC was missing: interceptors, multiplexing, flow control, compression, metadata and gRPC-compatible error codes.
• Parity groundwork:
  • Unary and streaming interceptors, so tracing, auth and metrics keep working.
  • Metadata bridging.
  • Mapping errors to gRPC status codes, so existing retry logic is unchanged.
  • Testing: A randomized test strategy that flips the protocol on each test run to surface differences.

3. Multiplexing and flow control: you can’t ship one without the other

• Running many concurrent streams over one connection, with per-stream write queues for fairness and zero-copy, pooled frames on the send path.
  What broke next: one slow consumer stalled every stream on the shared connection.
• The fix: credit-based per-stream windows, grant-on-consume accounting and a new window-update frame.

4. Compression: choosing by benchmark, not by default

• We benchmarked Snappy, S2, Zstd and minLZ on real CockroachDB traffic. minLZ-fastest won: about 2× faster decompression than Snappy, a slightly better ratio and lower CPU.

5. Changing the wire protocol under a live database

• A connection handshake for protocol negotiation, so new frame types and features roll out safely across mixed-version clusters.
  Running both RPC stacks side by side, with feature flags and version gates to prevent accidental enablement mid-upgrade, plus a rollback path.
• A staged rollout: test clusters, then staging with real workloads, then customer opt-in in public preview.
• Operator metrics that mattered during rollout: connection health, in-flight requests, flow-control stalls.
• Results that will include side-by-side dashboards showing cluster CPU, p50 RPC round-trip latency, allocated objects per transaction, total GC pause time with gRPC and the custom RPC.

How can this help other practitioners?

1. Profiling methodology to decide if your RPC layer is worth replacing (what pprof signals to look for, how to measure allocation vs. CPU cost) 2. The hidden cost of “lightweight” protocols: a minimal base saves CPU but shifts work onto you.
2. Multiplexing + flow control as a package: why you can’t ship one without the other, and the credit-based design pattern that worked for us. 4. Dual-protocol migration pattern: running two RPC frameworks side-by-side with feature flags and version gate.
3. Compression algorithm selection: benchmarking across real workloads, size thresholds, and the per-message vs. streaming allocation tradeoff. 6. Observability requirements for a protocol migration: which metrics actually matter during rollout.
4. Build vs. buy vs. fork: the decision framework we used, and how to think about ongoing maintenance cost of a forked RPC stack.

Why It Matters
Most RPC comparisons are microbenchmarks. This is a real production migration of a distributed database’s internal communication layer, including the honest accounting of what gRPC does well, what a lighter protocol costs you to replicate, and how to roll out a protocol change without downtime.

Current state
Public preview

Tags
#databases #rpc #distributedsystems #casestudy #migration #performance #grpc #scalability #infrastructure