Tencent Hy4 Preview: Full Specs and How It Stacks Up to GLM 5.3, Kimi K3
Tencent's open-weight Hy4 preview MoE model edges out GLM 5.3 and Kimi K3 in blind engineering evals. Specs, architecture, and benchmarks explained.
What is Tencent Hy4 preview?
Hy4 preview is Tencent’s newest open-weight flagship large language model, released under the Hunyuan (Hy) program with Apache 2.0 licensing. It’s a Mixture-of-Experts (MoE) model with 770 billion total parameters, only 49 billion of which activate per token. Tencent positions it as the current peak of its Hunyuan lineup, trained on more data with a longer context window than prior generations, and benchmarked directly against GLM 5.3 and Kimi K3 in blind human evaluations on real engineering work.
TL;DR
- 770B total parameters with 49B active per token puts Hy4 preview in the same MoE class as other frontier open models, balancing scale against inference cost.
- Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache lets the model reuse sparse attention indices across layers, a design borrowed and extended from DeepSeek and GLM research.
- A 1M token context window and a native multi-token prediction (MTP) layer for speculative decoding support both long-document work and faster generation.
- In blind evaluation across 203 engineering tasks, Hy4 preview scored 2.99 average versus GLM 5.3’s 2.92 and Kimi K3’s 2.94, with win rates of 46.8% and 51.2% respectively.
- The model ships in two forms, a full-precision release and an FP8-quantized version, both deployable through vLLM or SGLang with OpenAI-compatible APIs.
- Tencent frames this as an early preview, openly listing known issues like over-verification and unnecessarily long reasoning chains that it plans to fix in future updates.
How does Hy4 preview’s architecture work?
The backbone runs 78 transformer layers. The first layer uses a standard dense feed-forward network, and the remaining 77 replace that with MoE blocks. Each MoE layer holds 256 routed experts plus 1 shared expert, and every token activates the top 8 routed experts alongside the shared one. That routing pattern, common in modern MoE design, lets the model carry a huge total parameter count while keeping the actual compute per token much smaller (49B activated parameters versus 770B total).
On top of the MoE backbone, Tencent added a native MTP (multi-token prediction) layer with 10B total parameters and 0.7B activated, built specifically to support speculative decoding at inference time. This is a separate, smaller module that predicts multiple tokens ahead, which speeds up generation when paired with compatible serving frameworks.
The attention mechanism is where Hy4 preview borrows most visibly from other labs’ research. It uses Gated DeepSeek Sparse Attention (Gated DSA), a sparse attention variant with query compression down to 2048 dimensions and key-value compression to 512 dimensions. It also implements IndexCache, a technique for reusing sparse attention indices across layers rather than recomputing them at each layer, which cuts overhead. The indexer itself runs 32 heads at 128 dimensions each, selecting a top-k of 2048 tokens to attend to.
The residual pathway uses iHC (identity Hyper-Connections) across 4 residual streams, a technique meant to widen how information flows between layers beyond a single residual connection. Hidden size sits at 6144, vocabulary size at 120,832, and context length extends to 1 million tokens.
What benchmarks does Hy4 preview report against GLM 5.3 and Kimi K3?
Tencent’s headline comparison isn’t a standard multiple-choice benchmark suite. It’s a blind side-by-side human evaluation. The company had 163 internal experts rate model outputs across 203 real engineering tasks, comparing Hy4 preview against both GLM 5.3 and Kimi K3 without knowing which model produced which answer.
Against GLM 5.3, Hy4 preview scored an average of 2.99 versus GLM’s 2.92, with a win/tie/loss breakdown of 46.8% wins, 12.8% ties, and 40.4% losses.
Against Kimi K3, Hy4 preview again scored 2.99 versus Kimi’s 2.94, with 51.2% wins, 7.9% ties, and 40.9% losses.
Both results show Hy4 preview ahead, but the margins are narrow, and loss rates sit around 40% in both matchups. This isn’t a blowout. It’s a model that wins more often than it loses against two strong open rivals, on tasks that internal engineering teams actually care about, rather than a model that dominates outright.
Tencent also published a broader benchmark appendix covering additional evaluation categories, though the model card doesn’t break out individual scores for those in text form, presenting them instead as a chart. The company describes the overall jump from its previous generation as the largest it has measured to date, enough to place Hy4 preview at what it calls the open-source frontier.
What is Hy4 preview actually good at?
Tencent built training data in partnership with internal teams across four domains, and describes concrete gains in each:
Software engineering: better long-horizon planning, debugging, and verification, plus improved visual quality and interaction design in front-end code output.
Office and analysis work: converting scattered, messy source material across many files into finished documents, spreadsheets, and presentations, with more precision on data analysis, equations, and financial modeling.
Game development: turning a single prompt into a playable prototype and working across multiple turns inside game engines as a project evolves.
One coffee. One working app.
You bring the idea. Remy manages the project.
Scientific research: stronger reasoning on hard research questions, with stated progress in AI research tasks, molecular dynamics, condensed matter physics, and pure mathematics.
Tencent says it co-designed Hy4 preview alongside its own products, CodeBuddy and WorkBuddy, so gains measured internally are meant to translate into tools people already use rather than staying confined to benchmark charts.
Is Hy4 preview worth deploying right now?
That depends heavily on your hardware and your tolerance for a preview-stage model. At 770B total parameters, this is not something you run on a single GPU or even a single high-end workstation. Tencent’s deployment recipes assume multi-GPU setups: the vLLM example uses --tensor-parallel-size 8, meaning eight GPUs working together, and that’s using the FP8-quantized checkpoint rather than the full-precision one. Full-precision deployment would need substantially more memory across an even larger cluster.
The model card explicitly frames this as an early, unfinished release. Tencent lists known issues up front: the model spends longer than necessary reasoning through complex tasks, and it tends to over-verify its own work, essentially double-checking answers more than needed, which adds latency without proportional benefit. Tencent says this mirrors how it shipped Hy3 preview, an earlier version that improved substantially after public feedback, and that it expects the same iterative pattern here.
For teams already running open MoE models like GLM or Kimi in production, and with the infrastructure to serve a 770B-parameter model, Hy4 preview is a reasonable model to benchmark against your own workloads given the win rates reported above. For smaller teams or single-GPU setups, it’s not a practical option yet, at least not until smaller distilled or further-quantized variants appear.
Frequently Asked Questions
How many parameters does Hy4 preview have?
It has 770 billion total parameters, with 49 billion activated per token through its Mixture-of-Experts routing. A separate multi-token prediction module adds 10 billion total parameters (0.7 billion activated) for speculative decoding.
Is Hy4 preview open source?
Yes. Tencent released the full model weights and an FP8-quantized version under the Apache 2.0 license on Hugging Face, ModelScope, GitCode, and CNB, along with finetuning recipes and deployment guides.
How does Hy4 preview compare to GLM 5.3 and Kimi K3?
In blind evaluations across 203 engineering tasks rated by 163 internal experts, Hy4 preview beat GLM 5.3 (46.8% wins vs 40.4% losses) and beat Kimi K3 (51.2% wins vs 40.9% losses), with both matchups scoring close to even overall.
What hardware do you need to run Hy4 preview?
Tencent’s official deployment recipes for vLLM and SGLang use eight-GPU tensor-parallel configurations even with the FP8-quantized checkpoint, meaning this is a multi-GPU server deployment, not something suited to consumer hardware.
What is Gated DSA in Hy4 preview?
Gated DeepSeek Sparse Attention (Gated DSA) is the attention mechanism Hy4 preview uses, combining compressed query and key-value representations with a sparse indexer that selects a top-2048 subset of tokens to attend to, paired with IndexCache for reusing those indices across layers.
