Design a Fully Streaming Voice Agent with Low Latency

Overview

In this tutorial, we build an end-to-end streaming voice agent that mirrors how modern low-latency conversational systems operate in real time. We simulate the complete pipeline, from chunked audio input and streaming speech recognition to incremental language model reasoning and streamed text-to-speech output, while explicitly tracking latency at every stage. By working with strict latency budgets and observing metrics such as time to first token and time to first audio, we focus on the practical engineering trade-offs that shape responsive voice-based user experiences.

Core Data Structures

We define the core data structures and state representations that allow us to track latency across the entire voice pipeline. We formalize timing signals for ASR, LLM, and TTS to ensure consistent measurement across all stages. We also establish a clear agent state machine that guides how the system transitions during a conversational turn.

Audio Input Simulation

We simulate real-time audio input by breaking speech into fixed-duration chunks that arrive asynchronously. We model realistic speaking rates and streaming behavior to mimic live microphone input. This stream serves as the foundation for testing downstream latency-sensitive components.

Streaming ASR Implementation

We implement a streaming ASR module that produces partial transcriptions before emitting a final result. By progressively revealing words, we reflect how modern ASR systems operate in real time. We also introduce silence-based finalization to approximate end-of-utterance detection.

Streaming LLM and TTS

In this step, we model a streaming language model and a streaming text-to-speech engine that work together. We generate responses token by token to capture time-to-first-token behavior, and then convert incremental text into audio chunks for early speech synthesis.

Orchestrating the Voice Agent

We orchestrate the complete voice agent by wiring audio input, ASR, LLM, and TTS into a single asynchronous flow. Precise timestamps are recorded at each transition to compute critical latency metrics. Each user turn is treated as an isolated experiment for systematic performance analysis.

Running the Demo

We run the entire system across multiple conversational turns to observe latency consistency. Utilizing aggressive latency budgets allows us to stress the pipeline under realistic constraints, validating responsiveness targets across interactions.

Conclusion

This tutorial demonstrated how to orchestrate a fully streaming voice agent as a unified asynchronous pipeline with clear stage boundaries and measurable performance guarantees. Combining partial ASR, token-level LLM streaming, and early-start TTS reduces perceived latency, establishing a strong foundation for real-world deployments. Check out the FULL CODES here.