From 24 to 216: A Systematic Performance Engineering Journey for LLM-based TTS

The challenge of serving real-time text-to-speech at scale is often framed as a model problem: bigger models, better architectures, or specialized hardware. Yet, the most impactful gains frequently emerge not from the model itself, but from the intricate dance of components surrounding it. SAIL's internal lab recently conducted a deep performance engineering study on a production-grade TTS pipeline, targeting the publicly available Orpheus-TTS deployment served via Baseten. Their objective was to characterize its performance envelope and systematically exceed it, providing a methodology applicable to any latency-sensitive system.

The results were striking. Starting from a baseline of approximately 24 concurrent real-time connections per H100 GPU, the team achieved a sustained concurrency of 216 connections on the same hardware while maintaining strict p99 latency and real-time factor (RTF) constraints. This represents a nearly 10x increase in effective throughput, achieved without modifying model architecture, retraining weights, or relying on specialized hardware. In practical terms, a production deployment provisioned with 100 H100 GPUs to serve 2,400 concurrent streams could be reduced to just 10 GPUs, each serving 240 streams, slashing annual accelerator spend from roughly $1.4 million to $140,000 while delivering identical service capacity.

The System Under Study

Modern text-to-speech systems, particularly those based on LLMs, consist of two primary modules. The first is the engine hosting the LLM (in this case, Orpheus-TTS, a fine-tuned variant of LLaMA 3.2 3B) which translates input text into audio features. The second is the codec decoder (the 19.8M-parameter SNAC decoder) which converts these predicted audio tokens back into a waveform. This two-stage pipeline is representative of many contemporary TTS architectures, making the optimizations broadly relevant.

Performance was evaluated using two critical metrics: Time to First Byte (TTFB), which measures the elapsed time between a request and the receipt of the first audio byte, and Real-Time Factor (RTF), the ratio of generation time to audio duration. An RTF below 1.0 indicates faster-than-real-time generation, essential for streaming scenarios. All testing was conducted on a production-grade node with an Intel Xeon Platinum 8481C CPU and an NVIDIA H100 SXM GPU.

The Optimization Philosophy: System-Level Focus

The team deliberately focused on system-level optimizations rather than model-level techniques like quantization or pruning. While model-centric improvements are well-documented, system-level effects—scheduling, resource allocation, CPU-GPU interaction, and pipeline coupling—are often the dominant factors in end-to-end performance but are less commonly documented. The approach was holistic, treating the entire inference pipeline as a single entity rather than isolating individual kernels or modules. This framing ensures the methodology remains applicable across diverse models and deployments.

The process was empirical and iterative: profile the system under load, identify the bottleneck, apply a targeted fix, and re-profile at the new operating point. This disciplined measurement cycle is critical in complex systems where bottlenecks shift as previous constraints are removed.

Baseline Analysis: Uncovering Hidden Stalls

The initial baseline assessment revealed a system struggling under load. At 16 concurrent connections, inter-token latency (ITL) was stable at ~6 ms. However, at 24 and 32 connections, severe ITL spikes emerged—more than a tenfold increase over the steady state. These spikes, coupled with an oscillating scheduler state, indicated intermittent stalls rather than sustained saturation. The vLLM engine telemetry suggested that the decoding stage was falling behind, creating back pressure that prevented new requests from entering execution.

For reference, Baseten's own deployment was measured to sustain 40 concurrent connections, establishing a practical benchmark. The baseline system, however, began degrading sharply beyond 24 connections, with p99 TTFB exceeding 2 seconds at 32 connections.

Optimization 1: Pinned Memory

Profiling with the PyTorch profiler identified a rare but severe stall originating in the sampling stage. A single operation, make_tensor_with_pad, occasionally expanded to nearly 78 ms, blocking the main thread. The root cause was an explicit, synchronous copy from pageable to pinned memory, a documented anti-pattern in PyTorch. By removing this explicit host-side pinning and allowing the CUDA driver to manage the transfer end-to-end, the stall was eliminated.

The impact was immediate. Engine step execution became uniform (~8 ms), and sustained concurrency increased from 24 to 48 connections per node. ITL spikes vanished, and p99 RTF remained below 1.0. This fix alone doubled the system's capacity, but it also exposed the next bottleneck: decode-side throughput.

Optimization 2: Two-Dimensional Dynamic Batching

With host-side synchronization resolved, the system was now limited by how efficiently decode work was amortized across concurrent requests. Profiling with NVIDIA Nsight revealed that issuing decode operations independently for each request created continuous background interference, fragmenting GPU execution and limiting utilization.

The solution was two-dimensional dynamic batching. Instead of processing decode work per request, the system batches operations across both audio chunks within a request and across concurrent requests. This consolidates decode work into short, dense execution windows, amortizing Python dispatch, scheduling, and kernel launch overhead. Combined with torch.compile for the decoding path, decode efficiency improved by over 50x.

Under a load of 128 concurrent connections (5.3x the baseline), the system maintained stable scheduler behavior, with mean TTFB dropping to ~700 ms and p99 under 800 ms. Decoding was no longer the critical path.

Optimization 3: Asynchronous Scheduling

Even with efficient batching, profiling at 128 connections revealed large idle gaps between forward passes. These gaps were caused by synchronous scheduling and output processing, which stalled the GPU while waiting for CPU-side work to complete. Enabling asynchronous scheduling in vLLM allowed scheduling and output processing to overlap with model execution, reducing idle time.

The step duration decreased from 7.8 ms to 6.1 ms, a 1.27x speedup. This translated to a supported concurrency of 192 connections (8.0x the baseline) while maintaining RTF < 1.0. The GPU utilization became nearly continuous, with only small synchronization barriers on the order of a few hundred microseconds.

Optimization 4: Penalty Refactors

Further profiling at high concurrency identified a final host-side bottleneck in the sampling path. The make_tensor_with_pad operation, now appearing again in a different context, was still executing synchronously on the CPU. This time, the issue was rooted in the data structure used to track output tokens: a list of lists of integers that required repeated conversion to dense tensors.

The fix involved refactoring the token tracking to use preallocated tensors, mirroring how the engine tracks other sequences. This eliminated the need for repeated list-to-tensor conversions. The change was integrated without introducing additional synchronization barriers, preserving the asynchronous execution model.

This refactor increased sustained concurrency from 192 to 216 connections (9.0x the baseline) while keeping RTF < 1.0. Token throughput rose to nearly 17k tokens/s, with TTFB distribution continuing to improve.

Optimization 5: Pipeline Tuning

With major structural bottlenecks removed, the final phase focused on incremental refinements. These included:

• More aggressive batching: Increasing the batch timeout by 3-4x without impacting per-request latency, improving batch density.
• Rebalancing torch.compile configurations: Prioritizing compilation for tail requests, which occur more frequently.
• Shape padding: Padding kernel inputs to fixed sizes to reduce the number of compiled graphs by up to 24x, enabling larger batch sizes.
• Optimizing CPU-to-GPU transfers: Performing a single large copy per batch instead of many small transfers.
• Eliminating unnecessary I/O: Replacing temporary file creation for WAV headers with direct in-memory generation.

These tweaks further reduced mean TTFB to 540 ms and p99 to 630 ms at 216 connections. The system also demonstrated flexibility: under fp8 quantization, it could sustain 200 connections with TTFB < 0.5 s, or 300 connections with RTF < 1.0, showing the pipeline's adaptability to different latency-throughput trade-offs.

Key Takeaways and Limitations

The journey from 24 to 216 concurrent connections illustrates several universal lessons in performance engineering:

1. Coupling matters: Tightly coupled systems mean a bottleneck in one module suppresses utilization elsewhere. Removing it unlocks latent capacity across the pipeline.
2. Iterative optimization is essential: Complex systems rarely have a single static bottleneck. Changes alter dynamics, requiring careful, step-by-step validation.
3. Tooling must match the question: High-level telemetry surfaces systemic issues; low-level traces pinpoint root causes. Mismatched tools obscure inefficiencies.
4. Simplicity is the greatest sophistication: The difficulty lies not in implementing the fix, but in correctly identifying the bottleneck within a noisy system.

The work has limitations. It was tuned for a specific load profile and hardware configuration (H100 SXM). Different request patterns or hardware may surface different bottlenecks. Strict RTF and TTFB constraints guided the optimization; changing these constraints would alter the optimization landscape.

Future Directions

While this study focused on system-level optimizations, model-level techniques like quantization, pruning, or speculative decoding could compound these gains. Speculative decoding, in particular, is promising but requires careful evaluation due to its overhead in latency-sensitive pipelines. The TTS setting, with its smooth acoustic feature continuity, may benefit from tolerance-based sampling thresholds combined with EAGLE-style speculative decoding.

Conclusion

SAIL's work demonstrates that significant performance remains unclaimed in production systems, often hidden in the interactions between components rather than in the models themselves. By systematically profiling, identifying, and removing bottlenecks, they achieved a 9x throughput increase without model changes or hardware upgrades. The methodology—iterative, measurement-driven, and holistic—provides a template for performance engineering across diverse latency-sensitive systems. The final code changes were often simple, but the journey to find them required disciplined reasoning and careful instrumentation. As the team notes, the hardest part is rarely the fix itself, but knowing where to look next.

For those interested in the technical details, the Orpheus-TTS model is available from Canopy Labs, and the SNAC decoder is an open-source implementation. The optimizations were applied within the vLLM framework, which now includes asynchronous scheduling by default. The full methodology can be adapted to any inference pipeline with similar structural characteristics.