Cloudflare's tokio-quiche: Elevating QUIC and HTTP/3 in Rust

Overview of tokio-quiche

Cloudflare has open sourced tokio-quiche, an asynchronous QUIC and HTTP/3 Rust library that wraps its battle-tested quiche implementation with the Tokio runtime. The library has been refined inside production systems such as Apple iCloud Private Relay, next generation Oxy based proxies, and WARP's MASQUE client, where it handles millions of HTTP/3 requests per second with low latency and high throughput. tokio-quiche targets Rust teams that want QUIC and HTTP/3 without writing their own UDP and event loop integration code.

From quiche to tokio-quiche

quiche is Cloudflare’s open source QUIC and HTTP/3 implementation written in Rust, designed as a low-level, sans-io library. It implements the QUIC transport state machine, including connection establishment, flow control, and stream multiplexing, while making no assumptions about how applications perform IO. To use quiche directly, integrators must open UDP sockets, send and receive datagrams, manage timers, and feed all packet data into quiche in the correct order. This design, while flexible, can be error-prone and time-consuming.

tokio-quiche packages this integration work into a reusable crate. It combines the sans-io QUIC or HTTP/3 implementation from quiche with the Tokio async runtime and offers an API that manages UDP sockets, packet routing, and calls into the quiche state machine.

Actor-based architecture on Tokio

Internally, tokio-quiche uses an actor model on top of Tokio. Actors are small tasks with local state that communicate through message passing over channels, making them ideal for sans-io protocol implementations that manage internal states and operate on message-like buffers.

The primary actor is the IO loop actor, which moves packets between quiche and the UDP socket. One of the key message types is an Incoming struct that describes received UDP packets. The async integration follows a fixed pattern: the IO loop awaits new messages, translates them into inputs for quiche, advances the QUIC state machine, and translates outputs into outbound packets for the socket.

For each UDP socket, tokio-quiche spawns two essential tasks: InboundPacketRouter, which owns the receiving half of the socket and routes inbound datagrams by destination connection ID, and IoWorker, the per connection IO loop that drives a single quiche Connection. This design encapsulates connection state inside each actor and keeps QUIC processing isolated from higher-level protocol code.

ApplicationOverQuic and H3Driver

As a transport protocol, QUIC can carry multiple application protocols, including HTTP/3, DNS over QUIC, and Media over QUIC. To avoid coupling tokio-quiche to a single protocol, the Cloudflare team exposes an ApplicationOverQuic trait. This trait abstracts over quiche methods and the underlying IO, providing higher-level events and hooks to the application that implements the protocol. For instance, the HTTP/3 debug and test client h3i uses a non-HTTP/3 implementation of ApplicationOverQuic.

On top of this trait, tokio-quiche features a dedicated HTTP/3-focused implementation named H3Driver, which connects quiche’s HTTP/3 module to the IO loop actor and converts raw HTTP/3 events into higher-level events with asynchronous body streams that suit application code. H3Driver is generic and exposes ServerH3Driver and ClientH3Driver variants, adding server-side and client-side behavior on top of the core driver. These components provide the building blocks for HTTP/3 servers and clients that align with Cloudflare’s internal infrastructure.

Production usage and roadmap

tokio-quiche has been in use at Cloudflare for several years before its public release. It powers Proxy B in Apple iCloud Private Relay, Oxy-based HTTP/3 servers, and the WARP MASQUE client, as well as the async version of h3i. In the WARP client, MASQUE-based tunnels built on tokio-quiche replace earlier WireGuard-based tunnels with QUIC-based tunnels. These systems run at Cloudflare-edge scale, demonstrating the integration's capacity to sustain millions of HTTP/3 requests per second in production.

Cloudflare sees tokio-quiche as a foundation rather than a complete HTTP/3 framework. The library exposes low-level protocol capabilities alongside example client and server event loops while leaving room for higher-level projects to implement opinionated HTTP servers, DNS over QUIC clients, MASQUE-based VPNs, and other QUIC applications. By releasing the crate, Cloudflare aims to lower the barrier for Rust teams to adopt QUIC, HTTP/3, and MASQUE, aligning external integrations with the same transport stack used in its edge services.

Key Takeaways

• tokio-quiche = quiche + Tokio: An async Rust library that integrates Cloudflare’s sans-io QUIC and HTTP/3 implementation with the Tokio runtime, eliminating the need for manual UDP and event loop plumbing.
• Actor-based architecture for QUIC connections: Utilizes an actor model on Tokio, featuring an InboundPacketRouter for routing UDP datagrams and IoWorker that drives a single quiche Connection, keeping transport state isolated and composable.
• ApplicationOverQuic abstraction: Protocol logic separation via the ApplicationOverQuic trait enables various QUIC-based protocols to be implemented on the same transport core.
• HTTP/3 via H3Driver, ServerH3Driver, and ClientH3Driver: Employs H3Driver alongside its variants to connect quiche’s HTTP/3 module to async Rust code, exposing HTTP/3 streams and bodies suited for typical Tokio-based services.