Skip to main content
MindStudio
Pricing
BlogAbout
My Workspace
HP ZGX Furylocal LLM benchmarksNVFP4 quantization

HP ZGX Fury Benchmarked: Local LLM Speeds for DeepSeek, GLM, Qwen

HP ZGX Fury benchmarks for local inference: token speeds, concurrency, and VRAM math running DeepSeek, GLM 5.3 Flash, and Qwen models.

Edited by Luis Chavez-Mattos, Director of Product RSS
HP ZGX Fury Benchmarked: Local LLM Speeds for DeepSeek, GLM, Qwen

What is the HP ZGX Fury and what makes it different for local inference?

The HP ZGX Fury is a Blackwell based workstation built around Nvidia’s GB300 class architecture, pairing a Grace CPU with a Blackwell GPU. What sets it apart from smaller local-AI boxes is memory: 252 GB of high bandwidth memory (HBM) on the GPU itself, running at roughly 7.1 terabytes per second, plus 496 GB of unified memory attached to the Grace CPU at around 396 GB per second. The GPU can address both pools, but the HBM is about 18 times faster, so the practical question for any model isn’t just “does it fit” but “does it fit in HBM.”

That distinction matters because it’s the difference between a model that streams tokens quickly and one that technically loads but crawls. A creator who benchmarked the machine across DeepSeek, GLM, Qwen, Nemotron, and several smaller models used this HBM-versus-unified-memory split as the organizing idea behind every result, since it predicts speed before you even run a prompt.

TL;DR

  • HBM capacity is the real ceiling: the ZGX Fury’s 252 GB of HBM determines which models run fast, while anything that overflows into the 496 GB of slower unified memory takes a real speed hit.
  • Memory math is predictable: at roughly 4.5-bit quantization, a model’s footprint in gigabytes is about parameters in billions times 0.56, so a 100B model needs around 56 GB.
  • Mixture of experts models beat dense models of similar size: GLM 5.3 Flash only activates 18B of its 320B parameters per token, so it outruns a 65B dense model like Ripple despite having far more total weights.
  • Speculative decoding adds real headroom, pushing GLM 5.3 Flash from roughly 225 tokens per second for a single user up to about 283, but only while spare compute is available.
  • Concurrency turns memory-bound decode into a compute problem: 16 simultaneous GLM 5.3 Flash users each got close to 80 tokens per second, over 1,200 tokens per second combined.
  • Agent workloads need a higher per-user speed floor than chat, since agentic steps generate thousands of tokens repeatedly; the creator suggests targeting at least 80 tokens per second for agentic use.
  • Even aggressive 1-bit quantization can be usable: a 1-bit quant of DeepSeek 4.1 Flash ran at about 80 tokens per second with surprisingly solid output quality, though still not recommended for precision-heavy coding tasks.
App builders are for prototypes. Enterprises use Remy.
Prototype
v0
Replit
Glide
Framer
RemyProduction
  • Self-hosted
  • SOC 2
  • HIPAA
  • Audit log
Built into every app you make

How does memory math decide which models actually fit?

The baseline formula is simple: parameters times bits per weight, divided by 8, gives you the gigabyte footprint. Quantization formats labeled “4-bit” are often closer to 4.5-bit in practice, since some layers stay at higher precision to preserve quality. That shortcut (parameters in billions times roughly 0.56) gives a fast estimate: a 100B parameter model lands around 56 GB.

On the ZGX Fury, the benchmark run stacked several models at once to show how this adds up. GLM 5.3 Flash (320B parameters total) took about 180 GB. Adding a Nemotron 3 Nano model (19 GB), a Qwen 3.5-4B model, an EmbeddingGemma model, and a small fine-tuned 270M Gemma decision model pushed the total past 212 GB, still inside the 252 GB HBM budget.

But raw weight size isn’t the whole story. Running models also need headroom for the KV cache, the memory that tracks ongoing conversation state. That cache grows with context length and with the number of concurrent users, so any serious deployment plan has to subtract that buffer from available HBM before deciding how many models or users a system can serve at once.

Why do some models run faster than others of similar size?

Inference happens in two phases, and they behave differently. Prefill is when the model processes the incoming prompt, often thousands of tokens at once, which makes it compute-bound: the GPU is doing lots of math per weight loaded. Decode is when the model writes its answer one token at a time, and for each token it has to re-read its active weights from memory. That makes decode memory-bandwidth-bound, which is why bandwidth, not just raw compute, dominates perceived speed.

This explains why GB10-class machines (like the ZGX Nano) can have fine prefill performance but slow decode: their memory bandwidth tops out around 270 GB per second. The ZGX Fury’s 7.1 TB per second of HBM bandwidth is the reason it decodes dramatically faster.

You can estimate a theoretical ceiling yourself: bandwidth divided by active parameters. For GLM 5.3 Flash, with 18 billion active parameters (roughly 10 GB per token at this quantization), 7.1 TB per second of bandwidth implies a ceiling near 700 tokens per second. In practice, a single user only reaches about a third of that, around 225 tokens per second, once attention computation, KV cache reads, and kernel overhead are factored in.

Speculative decoding helps close some of that gap. A small draft model guesses several tokens ahead, and the larger model verifies them in a single pass, trading spare compute for fewer memory round-trips. With that enabled, GLM 5.3 Flash picked up to around 283 tokens per second, though that boost only holds as long as there’s spare compute; pile on too many concurrent users and the advantage disappears.

Does mixture-of-experts architecture matter more than model size?

Everyone else built a construction worker.
We built the contractor.

🦺
CODING AGENT
Types the code you tell it to.
One file at a time.
🧠
CONTRACTOR · REMY
Runs the entire build.
UI, API, database, deploy.

Yes, and the benchmark results make the point clearly. A 65B dense model (Ripple) reads all 65 billion parameters on every single token, capping its speed at around 200 tokens per second. GLM 5.3 Flash, despite having 320 billion total parameters, only activates about 18 billion of them per token because it’s a mixture-of-experts design, so it runs faster than the much smaller dense model.

This doesn’t make dense models obsolete. Dense architectures like Qwen’s 2.7B coding model remain popular for coding tasks specifically. But the broader pattern holds: switching to a larger mixture-of-experts model often delivers a speed and quality improvement over a smaller dense one, as long as the total weights still fit in fast memory. That caveat is the catch: MoE models need all their expert weights resident, even the ones not active on a given token, so total parameter count still governs the memory budget.

What token speed do agentic workflows actually need?

Chat and retrieval-augmented generation (RAG) use cases tolerate lower speeds because a human is reading along as text streams in; around 30 tokens per second feels fine. Agents are a different story. An agent might generate several thousand tokens of reasoning and code before calling a single tool, and it may repeat that cycle dozens of times to complete one task.

At 30 tokens per second, generating a 3,000-token step takes close to two minutes. Multiply that by 40 steps in an agentic rollout and the task takes over an hour. That’s the reasoning behind recommending a floor closer to 80 tokens per second for agent-heavy workloads, with the caveat that running multiple sub-agents in parallel multiplies the effective demand even further.

How many concurrent users can a system like this actually serve?

Batching changes the economics of decode. Serving 16 simultaneous users of GLM 5.3 Flash dropped individual throughput to just under 80 tokens per second each, but the combined output topped 1,200 tokens per second, because each weight read from memory now serves 16 requests instead of one. That effectively turns a memory-bandwidth problem back into a compute problem, which plays to the ZGX Fury’s listed 20 petaflops of compute.

Results vary heavily by model and by prompt length (the benchmark used 8,000-token prompts). DeepSeek 4.0 Flash served around 54 concurrent users with a floor near 40 tokens per second each. Nemotron 3 Nano, being smaller, scaled to roughly 188 concurrent users. For workloads that outgrow a single unit, pairing two of these systems over high-bandwidth networking was cited as a way to run larger models like DeepSeek 4.1 Flash at higher aggregate throughput, reportedly reaching 441 tokens per second for a single user or serving around 40 users at roughly 48 tokens per second each.

Frequently Asked Questions

What is NVFP4 quantization and why does it matter on Blackwell hardware?

NVFP4 stores each weight in four bits, grouped in blocks of 16 that share an 8-bit scale factor. Blackwell’s tensor cores can perform math directly in this format without an unpacking step that older GPU generations required, which is why NVFP4 quantized models run efficiently on the ZGX Fury.

Can the HP ZGX Fury run DeepSeek models fully offline?

Yes. The benchmark demonstrated running a full agent harness, including DeepSeek models, completely disconnected from the network, which matters for cybersecurity teams handling incidents or any workflow requiring total data privacy.

The enterprise platform for building software with AI.
You → Remy

Build a vendor-risk app for our 300 buyers.

SOC 2Audit logRole-based accessSelf-hosted

Why didn’t a 1-bit quantized model perform as poorly as expected?

Modern quantization toolchains don’t uniformly shrink every layer to the lowest bit depth. Some layers are kept at higher precision to preserve model behavior, which is why a 1-bit quant of DeepSeek 4.1 Flash still produced usable output at around 80 tokens per second, despite expectations that it would degrade heavily.

Is unified memory useless for running large language models?

No, but it’s significantly slower (around 396 GB per second versus 7.1 TB per second for HBM). Models that overflow into unified memory because they’re too large for HBM, like certain DeepSeek variants at NVFP4, will still run, just at reduced speed.

How do I estimate if a model will fit before downloading it?

Use the formula: parameters in billions times roughly 0.56 for a 4.5-bit quantization gives an approximate gigabyte footprint. Then add expected KV cache overhead based on context length and number of concurrent users before comparing against available HBM.

Editorial standards

Related Articles

Edge0-145B: A 145B Agent Model That Runs on One 24GB GPU

Edge0-145B is a sparse MoE agent model built to run locally on a single consumer GPU. Here's what the specs and quantization formats mean.

Edge0-145BNVFP4 quantizationlocal AI agent model

Edge0-145B Benchmarks: Terminal-Bench, GPQA, HLE, and IFBench Explained

Edge0-145B's NVFP4 build hits 86.5 on Terminal-Bench 2.1. Here's how it stacks up on GPQA-Diamond, HLE, and IFBench, and the NVFP4 vs Q3_K tradeoff.

Edge0-145B benchmarksTerminal-Bench 2.1 scoreEdge0 GPQA Diamond

Edge0-145B: How a 145B-Parameter Model Fits on One RTX 5090

Edge0-145B packs 145 billion parameters into 24GB VRAM using native 4-bit quantization and a sparse MoE design. Here's how it works.

Edge0-145BEdge0 NVFP4145B model consumer GPU

Sam Altman's RAM Comment: What He Actually Said

Sam Altman's viral clip on local inference and RAM, explained in full podcast context, plus why the backlash still has a real point.

Sam Altman RAMlocal inferenceOpenAI DRAM shortage

GLM 5.3 Flash: Ox Alpha Stealth Model Revealed by ZAI

ZAI confirms Ox Alpha was GLM 5.3 Flash, a 320B MoE model given MIT weights after serving 44 trillion tokens in stealth testing.

GLM 5.3 FlashOx Alpha revealedZAI GLM release

HP ZGX Fury Review: A Shared AI Workstation, Not a Desktop GPU

HP's ZGX Fury pairs Nvidia's GB300 Blackwell Ultra chip with 748GB unified memory to serve large open models to an entire team on-prem.

HP ZGX FuryZGX Fury specsDGX Station review

Presented by MindStudio

No spam. Unsubscribe anytime.