Zhipu AI announced a new high‑speed variant of its GLM‑5.1 model that it says can generate 400 tokens per second, positioning it as the fastest LLM inference service among the major providers. The article examines the benchmark, the engineering tricks behind the speed, and the practical limits that remain for real‑time enterprise use.
Zhipu AI’s GLM‑5.1‑highspeed API claims 400 tokens/s – what the numbers really mean
Zhipu AI (operating internationally as Z.ai) rolled out an API called GLM‑5.1‑highspeed, a variant of its 175‑billion‑parameter GLM‑5.1 model. The company’s press release touts a throughput of 400 tokens per second, which it frames as a new global benchmark for large‑language‑model inference speed. Below we unpack the claim, look at how it compares with other services, and discuss the constraints that still matter for production workloads.
What is being claimed?
- Throughput: 400 tokens / s on a single API request.
- Latency: Implied sub‑second response for short prompts (the company cites “real‑time or near‑real‑time” generation).
- Target customers: Select enterprise accounts that need high‑volume, low‑latency text generation—e.g., automated content pipelines, coding assistants, and live chat systems.
- Marketing framing: “A full day of human writing in one minute” and “three days of a software engineer’s work in a coffee break.”
The announcement does not disclose the hardware configuration, batch size, or whether the speed figure is measured on a single GPU, a multi‑GPU server, or a distributed cluster. Those details are crucial for anyone trying to gauge the relevance of the benchmark to their own environment.
How does it compare to other providers?
| Provider | Model (size) | Reported throughput* | Typical hardware |
|---|---|---|---|
| Z.ai | GLM‑5.1‑highspeed (175B) | 400 t/s | Not disclosed (likely multi‑GPU) |
| OpenAI | GPT‑4o (2024) | ~150 t/s (single request) | A100‑40GB (single) |
| Anthropic | Claude‑3.5 Sonnet | ~120 t/s | A100‑40GB |
| Gemini‑1.5‑Flash | ~180 t/s (batch‑1) | TPU v4 pod slice | |
| Meta | LLaMA‑2‑70B | ~80 t/s (single A100) | A100‑40GB |
*Throughput numbers are taken from public benchmarks or vendor documentation; they are not directly comparable because they may be measured under different conditions (batch size, sequence length, precision). The GLM‑5.1‑highspeed figure sits well above the publicly known numbers for comparable models, but the lack of a common measurement methodology makes the claim less conclusive.
Where does the speed come from?
Zhipu AI does not publish a technical white‑paper for the high‑speed variant, but the usual suspects for accelerating inference at this scale are:
- Quantization to 4‑bit or 8‑bit integer – reduces memory bandwidth and allows more tokens to be processed per GPU cycle. Recent work (e.g., GPTQ, AWQ) shows that 4‑bit quantization can keep near‑full accuracy for many tasks.
- Tensor‑parallelism across multiple GPUs – splitting the model’s weight matrix across devices can increase raw FLOPs available for each token.
- Kernel fusion and custom CUDA kernels – eliminating unnecessary memory copies and merging attention, feed‑forward, and layer‑norm operations into a single kernel can shave milliseconds per token.
- Prefill‑decode optimization – separating the expensive “prefill” stage (processing the prompt) from the cheap “decode” stage (generating each new token) and caching KV‑states efficiently.
- Dynamic batching – grouping multiple user requests into a single batch when possible, which improves GPU utilization without increasing per‑token latency.
If Zhipu is employing a combination of these techniques, the 400 t/s figure is plausible on a well‑tuned multi‑GPU server. However, the claim that the speed is available to every customer via a public API is unlikely without a substantial backend investment.
Practical limitations
1. Latency vs. throughput
High throughput does not automatically translate to low latency for a single request. A system that can push 400 t/s on a large batch may still take several hundred milliseconds to return the first token for a single‑prompt query. Real‑time chat or code‑completion use cases often care more about the first‑token latency than the overall tokens‑per‑second metric.
2. Cost and availability
Running a 175‑billion‑parameter model at the speeds claimed typically requires a cluster of high‑end GPUs (e.g., eight A100‑80GB or comparable). The operating cost per million tokens can be substantially higher than for smaller, more efficient models. Zhipu AI’s announcement mentions “select enterprise customers” only, suggesting that the service will be priced at a premium and may have limited capacity.
3. Accuracy trade‑offs
Quantization and aggressive kernel optimizations can introduce small but measurable degradations in generation quality, especially for tasks that rely on subtle probability differences (e.g., code synthesis, nuanced reasoning). Without a benchmark that reports both speed and standard evaluation metrics (e.g., MMLU, HumanEval), it is hard to assess whether the speed gain comes at an unacceptable accuracy cost.
4. Ecosystem integration
Many enterprises already have pipelines built around OpenAI, Anthropic, or Azure OpenAI endpoints. Switching to a new provider involves integration work, data‑privacy assessments, and possibly re‑training of prompt‑engineering heuristics. The high‑speed API will be attractive only if the latency benefit is demonstrably superior for the target workload.
What does this mean for the market?
The announcement signals that Chinese LLM vendors are catching up in the performance‑optimisation arms race that has been dominated by the U.S. cloud players. If Zhipu can sustain a public, low‑latency endpoint at the advertised speed, it could push other providers to release similar high‑throughput variants (e.g., OpenAI’s “Turbo” line or Anthropic’s “Fast” mode).
Nevertheless, the headline number should be taken with caution. Real‑world deployments will need to evaluate both throughput and first‑token latency, while also measuring quality on the specific tasks they care about. Until Zhipu publishes a detailed technical report or an independent benchmark, the claim remains a marketing point rather than a proven engineering breakthrough.
Bottom line
- 400 tokens/s is an impressive figure for a 175‑billion‑parameter model, but the lack of hardware and measurement details makes it hard to verify.
- The speed likely relies on a mix of quantization, tensor‑parallelism, and custom kernels—techniques that are becoming standard across the industry.
- Enterprise users should focus on latency, cost, and quality trade‑offs rather than raw throughput when deciding whether to adopt the GLM‑5.1‑highspeed API.
- Expect other vendors to respond with their own high‑throughput offerings, which will ultimately benefit the community by driving more transparent benchmarking.
Further reading
- Zhipu AI’s official announcement: https://www.zhipu.ai/blog/glm5-1-highspeed
- Independent benchmark of LLM inference speed (GitHub): https://github.com/EleutherAI/lm-evaluation-harness
- Overview of quantization techniques for LLMs: https://arxiv.org/abs/2210.17323
Comments
Please log in or register to join the discussion