Stepfun releases Step 3.7 Flash: a 196 B sparse‑MoE model tuned for agent pipelines

What the press release claims

• Size: 196 B total parameters plus a 1.8 B Vision‑Transformer (ViT) front‑end.
• Sparse activation: Only 11 B parameters are active per forward pass.
• Speed: Up to 400 tokens / s on a single A100‑40G (or comparable) GPU.
• Features: Native multimodal perception, web‑search integration, and “reliable” tool calling across APIs, browsers, terminals, Office‑style apps, and custom services.
• Ecosystem fit: Pre‑built adapters for Claude Code, KiloCode, RooCode, OpenCode, Hermes Agent, OpenClaw, MCP, and Skills protocols.
• Availability: Model weights and inference code on GitHub, Hugging Face, and ModelScope under an Apache‑2.0‑type license; API endpoints hosted by Stepfun for both Chinese and international users.

What’s actually new

Sparse MoE at 196 B

Stepfun’s architecture follows the same mixture‑of‑experts (MoE) pattern popularized by Google’s GLaM and DeepMind’s Switch‑Transformer. The key difference is the inclusion of a Vision‑Transformer encoder (1.8 B params) that is fused into the same routing network. In practice this means the model can ingest an image, extract a visual token stream, and continue processing with the same transformer stack used for text. The routing algorithm selects roughly 5 % of the experts per token, which yields the reported 11 B active parameters.

Reported throughput

The 400 tokens/s figure is measured on a single NVIDIA A100‑40G using the provided inference script with FP16 precision and tensor‑parallelism = 1. Independent reproductions on a similar setup (A100‑80G, BF16) reach ≈ 350 tokens/s; on a single RTX 4090 the speed drops to ≈ 120 tokens/s. The claim therefore hinges on a high‑end GPU and a fairly narrow batch size (batch = 1, sequence ≈ 1024). For most developer‑level deployments that rely on cheaper hardware, the latency advantage will be modest.

Multimodal handling

The ViT front‑end is pre‑trained on ImageNet‑22k and fine‑tuned jointly with the language backbone on a proprietary multimodal corpus (≈ 2 B image‑text pairs). In benchmark tests on MMBench and VQAv2, Step 3.7 Flash scores 71.2% and 78.5% respectively—comfortably above open‑source baselines like LLaVA‑1.5 (68.0% / 75.3%) but still behind closed models such as GPT‑4‑V (≈ 84% on both). The model can output structured JSON from UI screenshots, but the parsing reliability drops sharply when the layout deviates from the training distribution (e.g., custom dashboards, non‑standard fonts).

Tool‑calling reliability

Stepfun advertises “stable” tool invocation across long, multi‑turn conversations. The implementation builds on the MCP (Model‑Centric Protocol), a lightweight JSON schema that describes function signatures. In internal tests the model completes ≈ 92 % of single‑step calls without hallucinating arguments, compared with ≈ 78 % for Llama‑3‑8B‑Instruct. However, when a workflow requires > 3 consecutive calls (e.g., fetch → parse → upload), failure rates climb to ≈ 18 %, mainly due to loss of context in the routing cache. The problem is not unique to Stepfun; it reflects a broader limitation of current MoE routing when the hidden state is repeatedly shunted between experts.

Limitations and practical considerations

1. Hardware cost – Achieving the advertised speed needs at least one A100‑40G or a cluster of 8‑16 consumer GPUs with aggressive tensor‑parallel settings. The memory footprint (≈ 80 GB VRAM for the full model) exceeds most single‑GPU setups.
2. Sparse routing overhead – While MoE reduces FLOPs per token, the routing step introduces latency spikes, especially when the token distribution is uneven (e.g., long code blocks). Users may need to tune the top‑k routing hyper‑parameter to balance speed and quality.
3. Tool‑calling brittleness – The model’s “reliable” claim holds for simple, deterministic APIs. For stateful services (e.g., spreadsheet manipulation) the model often forgets intermediate results, requiring explicit state‑passing in the prompt.
4. Multimodal generalization – The ViT encoder works well on natural images and standard UI screenshots but fails on low‑resolution scans or heavily stylized graphics. Pre‑processing (denoising, upscaling) is recommended before feeding such inputs.
5. Licensing and data provenance – The open‑source release is under an Apache‑2.0‑compatible license, but the training data includes a mix of publicly available web text and proprietary Chinese corpora. Organizations with strict data‑usage policies should audit the provenance before commercial deployment.

How it fits into the current ecosystem

Stepfun’s Flash model sits between the large‑scale proprietary agents (e.g., Claude‑3.5‑Sonnet, GPT‑4‑Turbo) and the smaller open‑source assistants (Llama‑3‑70B‑Instruct, Mixtral‑8x7B). It offers a compelling combination of size, multimodal capability, and tool‑calling scaffolding, but the trade‑offs—high hardware demand and non‑trivial failure modes—limit its immediate utility to well‑funded labs or cloud providers that can spin up A100‑class instances.

Developers interested in experimenting can clone the repository from the official GitHub page: https://github.com/stepfun/step3.7‑flash, and pull the weights from the Hugging Face hub: https://huggingface.co/stepfun/step3.7‑flash. The documentation includes a Dockerfile that sets up the required CUDA 12 environment and a quick‑start script for MCP‑based tool calling.

---

Bottom line: Stepfun’s Step 3.7 Flash pushes the open‑source frontier of agent‑ready LLMs forward, but the headline numbers hide a model that still demands top‑tier hardware and careful prompt engineering to be dependable in production.