Tencent Hy4 Preview: A 770B MoE Model That Edges Out GLM-5.3
Tencent's Hy4 preview is a 770B-parameter, 49B-active MoE model with 1M context that beat GLM-5.3 and Kimi K3 in blind evals.
What is Tencent Hy4 preview?
Hy4 preview is an open-weight Mixture-of-Experts (MoE) language model from the Tencent Hy Team (Hunyuan), released under Apache 2.0 on Hugging Face, ModelScope, GitCode, and CNB. It has 770 billion total parameters with only 49 billion active per token, a 1 million token context window, and new attention and residual designs aimed at long-horizon coding and agentic work. In blind internal evaluations, it narrowly outperformed GLM-5.3 and Kimi K3 on engineering tasks.
TL;DR
- Scale and sparsity: Hy4 preview packs 770B total parameters but activates only 49B per token across 78 layers, keeping inference costs closer to a much smaller dense model.
- Gated DSA attention: The model uses Gated DeepSeek Sparse Attention with IndexCache for cross-layer index reuse, a design lineage that borrows from both DeepSeek and GLM’s recent attention work.
- iHC residual pathway: Identity Hyper-Connections expand the residual stream to 4 parallel paths, giving the model more routes for information to flow between layers than a standard residual connection.
- 1M context window: The model supports up to 1 million tokens of context, putting it in the same long-context tier as other 2026-era frontier open models.
- Blind eval results: Across 203 engineering tasks judged by 163 internal experts, Hy4 preview scored 2.99 average versus GLM-5.3’s 2.92 and Kimi K3’s 2.94, with win rates around 47-51%.
- Ships in two formats: Both a full-precision Hy4-preview and an FP8-quantized Hy4-preview-FP8 are available, with day-one recipes for vLLM and SGLang.
- Self-described as unfinished: Tencent explicitly flags known issues, including over-long reasoning chains and excessive self-verification, and frames this release as an early checkpoint rather than a finished flagship.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
How big is Hy4 preview, really?
The headline number is 770 billion total parameters, but that figure is misleading if you’re thinking about compute cost. Hy4 preview is a sparse MoE model, so only 49 billion parameters activate for any given token. The architecture has 78 transformer layers: the first layer runs a standard dense feed-forward network, and the remaining 77 layers each contain 256 routed experts plus 1 shared expert. Every token routes to the top 8 experts plus the always-on shared expert.
On top of the main backbone, Tencent bundles a native Multi-Token Prediction (MTP) layer, roughly 10B parameters with 0.7B active, used for speculative decoding to speed up generation. Other specs from the model card: hidden size of 6144, vocabulary of 120,832 tokens, 64 attention heads, and an MoE intermediate size of 2048 against an FFN intermediate size of 18432 for the dense layer. This puts Hy4 preview in the same general weight class as other trillion-parameter-adjacent MoE releases, while keeping active compute manageable enough to run on multi-GPU inference setups rather than requiring a full supercomputer per query.
What makes the attention mechanism different?
Hy4 preview’s attention module is called Gated DSA, short for Gated DeepSeek Sparse Attention. It builds on the DeepSeek Sparse Attention approach (a technique for reducing the quadratic cost of attention by having the model select a sparse subset of relevant tokens rather than attending to everything). Hy4 preview adds a gating mechanism on top and pairs it with IndexCache, a method for reusing sparse attention indices across layers instead of recomputing them each time.
The model card lists concrete parameters for this: 64 attention heads, a query compression dimension of 2048, a key-value compression dimension of 512, 32 indexer heads with a head dimension of 128, and an indexer top-k of 2048 (meaning the sparse attention mechanism selects up to 2048 tokens to attend to per step, rather than the full context). Tencent credits this design lineage explicitly to both DeepSeek’s sparse attention research and GLM’s related attention work, which signals a degree of convergence in how the top open-weight labs are tackling the cost of very long context windows.
What is iHC and why does it matter?
The second architectural change is in the residual pathway. Standard transformers pass information between layers through a single residual stream, essentially one long corridor that every layer reads from and writes to. Hy4 preview instead uses identity Hyper-Connections (iHC), which expands this into 4 parallel residual streams.
The practical effect is that the model has more distinct channels for information to persist or recombine across layers, rather than everything being compressed through one path. This is a relatively new technique in large model design, and Hy4 preview’s use of it alongside Gated DSA suggests Tencent is trying to address two separate bottlenecks at once: attention efficiency over long sequences, and information flow across a very deep 78-layer stack.
How does Hy4 preview perform against GLM-5.3 and Kimi K3?
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
Tencent ran a blind side-by-side evaluation specifically targeting the kind of work its own product teams do. It recruited 163 internal experts, spanning software engineers, game developers, finance analysts, and security specialists, and had them rate model outputs across 203 real engineering tasks without knowing which model produced which answer.
The results: Hy4 preview scored an average of 2.99, compared to 2.92 for GLM-5.3 and 2.94 for Kimi K3. Broken down into head-to-head win rates, Hy4 preview beat GLM-5.3 in 46.8% of comparisons, tied in 12.8%, and lost in 40.4%. Against Kimi K3, it won 51.2%, tied 7.9%, and lost 40.9%. These are narrow margins, not blowouts. Hy4 preview is ahead on aggregate, but the loss rates in the low 40% range indicate the other two models remain highly competitive on a meaningful share of tasks.
Tencent frames this evaluation as evidence the model translates into real gains for the specific workflows it was trained against: long-horizon software engineering (understanding, planning, debugging, verifying), office and analysis tasks that convert messy multi-file context into documents, spreadsheets, and financial models, game development (prompt-to-prototype and iterative engine work), and scientific research questions spanning AI research, molecular dynamics, condensed matter physics, and pure mathematics.
Is Hy4 preview ready for production use?
Tencent is upfront that this is a preview, not a finished release. The model card lists known limitations directly: Hy4 preview tends to spend longer than necessary reasoning through complex tasks, and it has a tendency to over-verify its own work, both of which translate into slower and more expensive inference for a given task. Tencent says there is meaningful headroom left in both pre-training and post-training.
The company’s stated strategy mirrors what it did with the prior Hy3 preview release: ship early, collect feedback on what breaks, and iterate quickly. That approach reportedly made Hy3 substantially better between preview and final release, and Tencent says it plans to repeat the pattern with Hy4.
For teams evaluating it today, two deployment paths are documented: vLLM and SGLang, both with official prebuilt Docker images (vllm/vllm-openai:hy4-preview and lmsysorg/sglang:hy4-preview). The FP8-quantized version, Hy4-preview-FP8, is the variant used in Tencent’s own deployment recipes, targeting an 8-way tensor-parallel setup with the MTP layer enabled for speculative decoding. A full finetuning pipeline is also included, supporting both LLaMA-Factory and ms-swift workflows, plus Tencent’s AngelSlim toolkit for further quantization and compression.
Frequently Asked Questions
How many parameters does Hy4 preview have?
Hy4 preview has 770 billion total parameters, of which 49 billion are activated per token due to its Mixture-of-Experts architecture. A separate MTP module adds roughly 10B parameters (0.7B active) used for speculative decoding.
What is Gated DSA attention?
Gated DSA (Gated DeepSeek Sparse Attention) is Hy4 preview’s attention mechanism. It extends DeepSeek’s sparse attention approach with an added gating function and uses IndexCache to reuse sparse attention indices across layers instead of recalculating them at each layer.
How does Hy4 preview compare to GLM-5.3 and Kimi K3?
In a blind evaluation of 203 engineering tasks judged by 163 internal experts, Hy4 preview scored slightly higher on average (2.99) than GLM-5.3 (2.92) and Kimi K3 (2.94), with win rates of roughly 47% and 51% respectively against each model.
What context length does Hy4 preview support?
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Hy4 preview supports a context window of up to 1 million tokens.
Can I run Hy4 preview myself?
Yes. Weights for both the standard and FP8-quantized versions are open on Hugging Face, ModelScope, GitCode, and CNB under Apache 2.0. Tencent provides deployment recipes for vLLM and SGLang, along with finetuning and quantization tooling.
