Running 120B+ LLMs Locally on the RTX Spark Laptop: Real Numbers
Benchmarks for running GPT-OSS-120B and Qwen locally on the RTX Spark laptop via Llama.cpp, WSL, and vLLM, with real tokens-per-second data.
What is the RTX Spark laptop and why does it matter for local AI?
The RTX Spark laptop, demonstrated here in the form of the MSI Prestige 16 AI Plus, is a Windows-on-ARM machine built around Nvidia’s N1X chip with 128GB of unified LPDDR5 memory. It’s positioned as a portable alternative to Nvidia’s DGX Spark desktop box, which uses similar memory capacity but lives on a desk and needs to stay plugged in. The pitch is simple: take the DGX Spark’s ability to run huge local models and put it in a laptop you can close and carry to a coffee shop. MSI advertises support for 120 billion parameter models, which independent testing confirms, and then some.
This matters because most “run big LLMs locally” conversations assume a desktop tower with a discrete GPU and a wall outlet. A laptop with 128GB of memory shared between CPU and GPU changes the calculus for anyone who wants to do model inference, fine-tuning experiments, or agent workflows without a cloud bill or a network dependency.
TL;DR
- The RTX Spark laptop ships with 128GB of unified LPDDR5 memory on an ARM-based N1X chip, with up to 96GB assignable to the GPU through Nvidia’s control panel.
- It runs GPT-OSS-120B at roughly 65 tokens per second in Llama.cpp on Windows, noticeably faster than the same model on a DGX Spark desktop box (around 50 tokens per second) in the same test.
- Mixture-of-experts models outperform dense models on this hardware: a 27B dense Qwen model only hits 13.4 tokens per second, while the 120B MoE model runs nearly 5x faster because fewer parameters activate per token.
- vLLM runs inside WSL (it doesn’t build natively on Windows) and crushes prompt processing, hitting 4,681 tokens per second versus about 1,800 for Llama.cpp, a 2.5x jump.
- For a single user, Llama.cpp beats vLLM on generation speed (65 vs 35 tokens per second on GPT-OSS-120B), but vLLM scales far better with concurrent users, reaching 127.5 tokens per second with eight simultaneous sessions.
- Unplugging the laptop barely changes GPU inference speed (65.4 vs 65.9 tokens per second on battery), though CPU-heavy compilation work does slow down slightly on battery.
- Over a sustained 20-minute load test with the GPU and all 20 CPU cores running, token generation held steady (around 50 to 52 tokens per second), suggesting no meaningful thermal throttling.
One coffee. One working app.
You bring the idea. Remy manages the project.
How many tokens per second does GPT-OSS-120B actually run at?
In testing, GPT-OSS-120B loaded in about 28 seconds and occupied roughly 61GB of the 96GB allocated to the GPU. Generation speed came in at 65 tokens per second on Windows using Llama.cpp, and almost identically at 65.1 tokens per second inside WSL. For comparison, the same model on the same version of Llama.cpp on a DGX Spark desktop produced about 50.6 tokens per second, a meaningful gap in the laptop’s favor.
The explanation traces back to memory bandwidth, which is the main bottleneck for token generation in large language models. The laptop measured about 143GB/s copying 8GB blocks on the GPU, compared to 113GB/s on the DGX Spark. Read bandwidth showed an even bigger gap: 309GB/s versus 234GB/s. Raw GPU compute (the part responsible for prompt processing and matrix multiplication) came out roughly even between the two devices, around 101 versus 97 in the benchmark used. So the laptop’s edge comes specifically from faster memory movement, not from a more powerful GPU core.
As context length grows, generation speed drops on both platforms, and WSL shows a slightly steeper decline than native Windows, though Windows consistently held the lead across the tests.
Why do dense models run so much slower than mixture-of-experts models?
A dense model activates every parameter for every token it generates. A Qwen 3.8 27B dense model, around 22GB in size, only managed 13.4 tokens per second on this hardware despite being far smaller than the 120B MoE model. That’s the tradeoff with dense architectures: every bit of the model has to be read from memory and computed for each token, which taxes bandwidth heavily regardless of total parameter count.
Mixture-of-experts (MoE) models like GPT-OSS-120B only activate a subset of their total parameters per token. That’s why a 120B parameter MoE model can run at 65 tokens per second while a 27B dense model crawls at 13.4. For anyone choosing which model architecture to run locally, this is the single biggest lever: MoE models make huge parameter counts practical on consumer and prosumer hardware, while dense models punish you for every billion parameters you add.
A newer architecture, Qwen 3.8 Flash Next at 125B parameters, was also tested. It loaded in 36 seconds, used 81GB of the 96GB GPU allocation (pushing close to the ceiling), and ran at 33.7 tokens per second, a workable speed for a model of that size on a laptop.
Is vLLM worth running over Llama.cpp on this hardware?
It depends entirely on whether you’re the only user. vLLM doesn’t build natively on Windows, but it compiles and runs fine inside WSL, and the laptop’s memory capacity makes it viable in a way that wasn’t practical on smaller local machines before.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
For prompt processing, vLLM is dramatically faster: 4,681 tokens per second versus about 1,800 for Llama.cpp on both Windows and WSL, a 2.5x improvement. But for single-user token generation, vLLM actually loses to Llama.cpp: 35 tokens per second versus 65 on GPT-OSS-120B. vLLM is architected to serve many concurrent requests efficiently, not to minimize latency for one user, so a single chat session is close to its worst-case scenario.
That changes completely under concurrent load. With eight simultaneous users (or agents) hitting the model at once, vLLM’s throughput jumped to 127.5 tokens per second combined, actually improving as load increased. For anyone building multi-agent systems or serving more than one person off a local model, vLLM is the better choice despite its single-user weakness. For a single developer chatting with a coding assistant, Llama.cpp (the engine behind tools like Ollama and LM Studio) wins.
Does running on battery or Windows vs. Linux change performance?
Unplugging the laptop had almost no effect on GPU-based LLM inference. GPT-OSS-120B generated at 65.9 tokens per second on battery versus 65.4 plugged in, essentially a wash. CPU-bound work told a different story: a multi-core fractal rendering benchmark (Mandelbrot set calculations across all 20 cores) took 22 seconds plugged in and 23.2 seconds on battery, a modest but real slowdown.
Windows versus WSL (Windows Subsystem for Linux) also produced mixed results depending on the task. Running Python-heavy workloads was substantially faster inside WSL than native Windows, even though both environments used all 20 CPU cores and single-core JavaScript benchmarks came out identical (1.24 seconds either way). The slowdown on native Windows traced specifically to how Python executed, not to the underlying chip. The practical takeaway: for GPU-based LLM inference, Windows and WSL perform almost identically, but for CPU-heavy Python work, WSL has a real edge.
Because this machine runs Windows on ARM, compatibility was also a question. The vast majority of developer tools, including VS Code, Zed, Node, Python, and even Adobe’s Creative Cloud apps, now run natively on ARM64, with only a handful of legacy x64 processes still running under emulation.
Frequently Asked Questions
What GPU memory allocation does the RTX Spark laptop support?
Up to 96GB of the 128GB of total system memory can be dedicated to the GPU, adjustable through Nvidia’s control panel. The remaining memory stays available for the operating system and other applications.
Can you really run a 120 billion parameter model on a laptop?
Yes. GPT-OSS-120B loaded in about 28 seconds, used around 61GB of GPU memory, and generated at roughly 65 tokens per second in testing, both plugged in and on battery.
Does the RTX Spark laptop throttle under sustained load?
In a 20-minute stress test running the GPU and all 20 CPU cores simultaneously, token generation speed stayed roughly flat, moving from about 50 tokens per second in the first five minutes to 52 in the last five, suggesting no significant thermal throttling.
Is Llama.cpp or vLLM better for local inference on this hardware?
Llama.cpp is faster for a single user (65 vs 35 tokens per second on GPT-OSS-120B). vLLM pulls ahead dramatically with multiple concurrent users or agents, reaching 127.5 tokens per second with eight simultaneous sessions, and it also has much faster prompt processing overall.
Does this laptop actually outperform the DGX Spark desktop?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
For the specific tests measured, including GPT-OSS-120B token generation and GPU memory bandwidth, the laptop outperformed a DGX Spark running the same version of Llama.cpp. Raw GPU compute came out roughly even between the two.
