Skip to main content
MindStudio
Pricing
BlogAbout
My Workspace
Hy4-Flash local setupTencent Hunyuan modelrun Hy4-Flash vLLM

Running Tencent's Hy4-Flash Locally: Hardware, Quantization, and Setup

How to deploy Tencent's Hy4-Flash MoE model locally: architecture, GGUF/FP8 quantization, and serving with vLLM or SGLang.

Edited by Luis Chavez-Mattos, Director of Product RSS
Running Tencent's Hy4-Flash Locally: Hardware, Quantization, and Setup

What is Tencent Hy4-Flash and why does it matter for local deployment?

Hy4-Flash is Tencent’s latest open-weight Hunyuan model, built as a cost-optimized Mixture-of-Experts (MoE) system with 220B total parameters but only about 12B active per token (11.3B text plus 0.43B vision). That gap between total and active parameters is the whole story for local deployment: you need enough storage and memory to hold the full weight set, but the compute cost per generated token stays close to what a much smaller dense model would require. Tencent released it alongside an FP8-quantized variant and a pretrained base model, all on Hugging Face, ModelScope, and GitCode, with GGUF conversions available separately for llama.cpp-style runtimes.

TL;DR

  • Hy4-Flash is a 220B-parameter MoE model with only 12B parameters active per token, including a 64B Engram component and a 0.85B speculative decoding drafter.
  • The model supports a 1M-token context window and takes both text and image input through a SigLIP2-based vision encoder.
  • Tencent ships three variants: the base instruct model, an FP8-quantized version, and a pretrained base checkpoint for fine-tuning.
  • Serving options include vLLM and SGLang for GPU clusters, plus a GGUF conversion for llama.cpp-based local inference on consumer hardware.
  • A built-in speculative decoding module called DFly2 reuses the main model’s KV cache to speed up long-context generation at low added cost.
  • Tencent reports decoding speeds past 150 tokens/sec when combining megakernels, 4-bit quantization, and DFly2 in their own inference stack.
  • The architecture’s KV cache is dramatically smaller than prior Hunyuan models (1/24 of Hy3, 1/6 of Hy4), which directly reduces memory pressure during long-context inference.
The AI software platform for companies that need regulatory compliance and security.
Patient record #20417Viewing as: Billing clerk
Name
Maria Alvarez
Date of birth
SSN
Diagnosis
Balance due
$412.80
✓ HIPAA✓ Role-based access✓ Audit log

How is Hy4-Flash architected, and why does it matter for hardware planning?

Hy4-Flash uses a 42-layer backbone: one dense feed-forward layer followed by 41 MoE layers. Each MoE layer routes tokens to 16 of 512 “routed” experts plus one shared expert, with experts operating in a compressed latent space (what Tencent calls LatentMoE) rather than full hidden-dimension space. This latent compression is a major reason the active-parameter count stays low relative to the 220B total.

Three architectural choices matter most for anyone planning hardware:

Hybrid attention. The model interleaves sparse attention (IndexPool, selecting up to 2,048 tokens via group-level selection) with sliding-window attention at a 1:3 ratio. Multi-head Latent Attention (MLA) compresses the KV cache further. Combined, this is why Tencent reports a KV cache just 1/24 the size of their earlier Hy3 model and 1/6 the size of Hy4, which matters enormously once you’re running long-context workloads where KV cache, not weights, becomes the memory bottleneck.

Engram (n-gram embedding). A 63.5B-parameter embedding table storing learned bigram and trigram representations, retrieved through hashed lookups at four specific layers. This is a large chunk of the total parameter count but adds negligible active compute, since it’s a lookup rather than a matrix multiply across the full network.

DFly2 drafter. A 5-layer, 0.85B-parameter speculative decoding module built into the model. It reuses the target model’s KV cache and sparse attention indices, so enabling speculative decoding for long-context generation doesn’t require a second full KV cache the way external draft models typically do.

For someone sizing a GPU setup, the practical takeaway is: storage and total VRAM must account for the full 220B parameters (plus the 63.5B Engram table), but the compute budget during generation behaves much closer to a 12B active-parameter dense model, especially once quantized.

What quantization and deployment options are available?

Tencent published three ready-to-use checkpoints:

  • Hy4-Flash: the standard instruct model, suited for GPU clusters running at full or near-full precision.
  • Hy4-Flash-FP8: an FP8-quantized instruct model, roughly halving memory footprint versus BF16/FP16 while retaining most quality, and the more realistic option for single-node multi-GPU deployment.
  • Hy4-Flash-base: the pretrained (non-instruct) checkpoint, intended for fine-tuning rather than chat use.

For running on consumer or prosumer hardware without a datacenter-class GPU cluster, a GGUF conversion is available (published by AngelSlim, linked from the model card). GGUF lets you run the model through llama.cpp or compatible runtimes with 4-bit, 5-bit, or other quantization levels, trading some quality for a much smaller memory footprint. Given the 220B total parameter count, even aggressive quantization still requires a substantial amount of combined RAM/VRAM, so this is not a model that fits on a single consumer GPU at full capability; it’s aimed at users with high-memory workstations or multi-GPU setups who want to avoid API costs or need offline/private inference.

How do you serve Hy4-Flash with vLLM or SGLang?

For production-style local or on-prem serving, Tencent points to two inference frameworks, both standard choices for MoE models at this scale:

✗ VIBE-CODED APP
Tangled. Half-built. Brittle.
✓ AN APP, MANAGED BY REMY
UIReact + Tailwind✓
APIValidated routes✓
DBPostgres + auth✓
DEPLOYProduction-ready✓
Architected. End to end.

Built like a system. Not vibe-coded.

Remy manages the project — every layer architected, not stitched together at the last second.

vLLM supports the model’s MoE routing and MLA attention natively once the necessary model definitions are merged in (the model card notes that Hugging Face Transformers support was pending a merge at release time, requiring installation from a specific pull request branch). vLLM’s continuous batching and paged attention make it the more throughput-oriented option for serving multiple concurrent users.

SGLang is the other supported path, generally favored for its scheduling efficiency with MoE architectures and structured generation features.

Both frameworks expose an OpenAI-compatible API once running, so existing tooling built against the OpenAI client library should work with minimal changes, just pointing the base URL at your local server.

For raw experimentation or single-request inference, Tencent’s own quickstart example uses Hugging Face Transformers directly via AutoModelForImageTextToText and AutoProcessor, which is simpler to set up but slower and less memory-efficient than vLLM or SGLang for serving multiple requests.

One setup detail worth flagging: the instruct models support a reasoning_effort parameter with two modes, "high" (thinking enabled, the default) and "no_think" (direct responses without extended reasoning). Recommended sampling settings are temperature=0.9 and top_p=1.0. Getting these parameters right matters more for MoE reasoning models than for typical dense chat models, since the thinking mode changes both output length and latency substantially.

Is Hy4-Flash worth running locally instead of using an API?

The honest answer depends on your workload and hardware budget. Tencent’s own benchmarking (an internal suite called WorkIndex, covering tasks like slide creation, document editing, and spreadsheet repair) shows Hy4-Flash scoring 42.1, ahead of the larger “Hy4 preview” model’s 39.7, while costing roughly 1/13th as much per task. In a blind evaluation against Claude Haiku 5.5 across 85 users and 259 tasks, Hy4-Flash scored 2.79 out of 4 on user satisfaction versus Haiku’s 2.81, with a median completion time of 6.7 minutes and a reported cost of RMB 0.26 per task on Tencent’s own infrastructure.

Those cost figures reflect Tencent’s hosted inference stack (megakernels, 4-bit quantization, DFly2 speculative decoding tuned for their hardware), not necessarily what you’ll achieve on your own GPUs. Running it locally makes most sense if you need data privacy, want to avoid per-token API billing at volume, or are building on top of the 1M-token context window for workloads an API provider might rate-limit or price aggressively. If you mainly need occasional access and don’t want to manage a multi-GPU deployment, using a hosted endpoint is simpler.

Tencent itself recommends Hy4-Flash for “everyday productivity tasks” like deep search, document processing, slide creation, simple frontend demos, and computer-use agents, while steering complex, long-horizon software engineering work toward the larger, non-Flash Hy4 model.

Frequently Asked Questions

How much VRAM does Hy4-Flash need to run locally?

The model card doesn’t publish an exact VRAM requirement, but with 220B total parameters (including the 63.5B Engram table), full-precision deployment requires a multi-GPU cluster. The FP8 variant roughly halves that footprint, and GGUF quantization (4-bit or lower) reduces it further, though a model this large will still need a high-memory workstation or multiple GPUs even when heavily quantized.

What’s the difference between Hy4-Flash and Hy4-Flash-FP8?

Hy4-Flash is the standard instruct checkpoint at full precision. Hy4-Flash-FP8 is the same model quantized to FP8, shrinking memory requirements substantially while aiming to preserve most of the output quality, making it the more practical choice for self-hosted serving on GPU clusters.

The enterprise platform for building software with AI.
Vendor Risk Portal✓ SOC 2✓ HIPAA✓ Audit log
VendorsApprovalsContractsReportsSettings
Active vendors1,284
In review37
High risk6
VendorTierRiskApprovalNEW
Northwind SupplyTier 1LowApproved
Acme LogisticsTier 2MediumApprove
Helios DataTier 1HighPending
Granite FreightTier 3LowApproved
Pinecrest LabsTier 2LowApprove
You → RemyAdd an approval step for new vendors

Can Hy4-Flash handle image inputs, or is it text-only?

It’s multimodal. A vision encoder adapted from SigLIP2-So400m-Patch16-Res512 feeds into the backbone through an MLP projector, so the model accepts both text and image input, though it only generates text output.

What is DFly2 and do I need to configure it manually?

DFly2 is a built-in 5-layer, 0.85B-parameter speculative decoding drafter that ships as part of the model, reusing the main model’s KV cache and sparse attention indices. It’s designed to accelerate long-context decoding at low added cost; specifics of enabling it depend on which inference framework (vLLM, SGLang, or Transformers) you’re using.

Is Hy4-Flash better than Tencent’s full Hy4 model?

On Tencent’s internal WorkIndex benchmark, Hy4-Flash actually scores slightly higher (42.1) than “Hy4 preview” (39.7) while costing far less per task, but the full Hy4 model scores higher still (51.4) and is recommended for complex, long-horizon tasks like extended software engineering where Flash’s lower active-parameter count becomes a limiting factor.

Editorial standards

Presented by MindStudio

No spam. Unsubscribe anytime.