Running Local Agent Swarms with vLLM and Hermes: A Config Guide
How to configure vLLM and Hermes agent to run parallel sub-agent swarms on a local dense model, with real settings for GPU memory, KV cache, and sequences.
Why run agent swarms on a local dense model instead of a fast quant?
Because throughput across many parallel agents matters more than single-response speed. A dense model at full precision responds slower per message than a quantized version of itself, but when you’re running five, ten, or twenty sub-agents at once through vLLM, the bottleneck stops being “how fast does one agent answer” and becomes “how much total work can the batch process together.” One creator running a local setup on quad RTX 3090s found that a heavily quantized model started failing on complex agentic coding tasks, while the same model at FP16 precision completed them, even though it was nominally the “slower” option in a single-chat context.
This is the core insight behind running agent swarms locally: raw chat latency and agentic throughput are different problems, and optimizing for one can hurt the other.
TL;DR
- Batch throughput beats single-response speed when running multiple sub-agents in parallel, since vLLM processes concurrent requests together rather than one at a time.
- A dense 27B parameter model at FP16 outperformed a heavily quantized (roughly int4-equivalent) variant of a related model on long, complex agentic coding sessions, even though the FP16 model runs slower per message.
- Key vLLM tuning parameters include GPU memory utilization, max model length, max number of sequences, max batch tokens, and KV cache quantization, all of which directly affect how many agents can run at once.
- Enabling prefix caching and chunked prefill is important for fast reloads and efficient reuse of shared context across agent calls.
- A max sequence count around 16 to 20 was found to be a workable ceiling on a quad 3090 setup, with higher values (up to 24) still functioning but adding significant wait time.
- Hermes agent connects to vLLM through a custom OpenAI-compatible endpoint, using auto-discovery to detect the model and context length automatically.
- Running GPU inference workloads in LXC containers instead of full VMs avoids virtualization overhead with no meaningful performance penalty.
What is an agent swarm, and why does it need special config?
An agent swarm is a setup where one orchestrating agent delegates a task to multiple sub-agents that work in parallel, each handling a piece of the overall job (research, code generation, review, summarization) before results get merged back together. Instead of a single model handling a task step by step in one conversation, the swarm spreads the work across concurrent requests to the same underlying model server.
This only works well if the inference server can actually handle many simultaneous requests without falling over or slowing to a crawl. That’s where vLLM configuration becomes critical. vLLM batches concurrent requests efficiently, but the defaults aren’t tuned for heavy multi-agent workloads. You need to explicitly configure how much GPU memory the server can claim, how many sequences it will run at once, how big the context window is, and how the KV cache gets stored.
Get these wrong and you’ll either run out of VRAM mid-swarm, or you’ll cap your parallelism so low that “spinning up five researchers” turns into a queue instead of a swarm.
How do you configure vLLM for parallel agents?
The core parameters that matter for swarm workloads, based on a real quad RTX 3090 setup running a 27B parameter dense model at FP16:
- GPU memory utilization: set to 0.95, meaning vLLM is allowed to claim nearly all available VRAM for model weights and cache.
- Max model length: set to 18244 tokens, defining the context window ceiling.
- Max number of sequences: set to 20, which caps how many concurrent requests (agents) can be processed at once. 16 was described as a safe number, with 20 giving a small buffer for overflow or compaction. Values up to 24 still worked but with a noticeable increase in wait time for outputs.
- Max number of batch tokens: set to 16K, controlling how much gets processed per batch step.
- KV cache quantization: set to FP8 (specifically the E4M3 format) to reduce memory pressure from cached context, which matters a lot when running many sequences simultaneously on Ampere-generation 3090s.
- Chunked prefill and prefix caching: both enabled. Prefix caching in particular was called out as critical for fast reload behavior, since it lets the server reuse already-processed context instead of recomputing it for every new agent call.
- Multimedia processor cache: set to 384MB, sized for workloads involving frequent screenshot-based evaluation by sub-agents. 256MB was noted as a safer, lighter alternative if you’re not doing heavy vision-based agent loops.
- Reasoning and tool-calling settings: a reasoning parser and auto tool choice were enabled for the model family in use, with a specific tool-call parser format needed depending on the model branch. Reasoning effort was set to “high,” and preserving thinking output was enabled, both of which increase token usage but were described as meaningfully improving output quality over lower reasoning settings.
- Mamba cache mode: set to “lean” and custom all-reduce disabled to suppress unnecessary warning messages.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
These settings were run on quad 3090s, with a separate 4090 handling desktop and recording duties on the same physical machine, illustrating that this isn’t a dedicated, isolated inference rig.
How does Hermes agent connect to a local vLLM server?
Hermes agent connects through a custom endpoint pointed at the vLLM server’s IP address and port, using the standard /v1 OpenAI-compatible API path and an API key (any string works if the server isn’t enforcing real auth). Once pointed at the endpoint, Hermes can auto-discover the running model and its context length rather than requiring manual entry. The model then shows up in a dropdown for selection, where naming it clearly (model name plus backend, like “Qwen 3.8 27B vLLM”) helps when juggling multiple model configs.
A practical infrastructure note: running the GPU server inside an LXC container rather than a full virtual machine avoids the performance hit that comes with VM-level GPU passthrough, while still keeping the workload isolated from the host.
What actually happens when a swarm runs?
In a real test, an orchestrating agent decided to delegate a research task to five sub-agents at once. As those sub-agents came online, running requests in the vLLM monitoring dashboard climbed from a handful up toward the configured maximum sequence count. Prefill token counts (the ingestion phase, where the model reads in context) rose first, followed by a surge in decode tokens (actual generation) once ingestion caught up.
Decode throughput scaled with the number of active agents: single-digit sub-agent counts produced decode rates in the low hundreds of tokens per second, and pushing toward the higher end of the configured sequence limit produced peaks in the 200 to 350 tokens-per-second range on this hardware. Prefix cache hit rates around 94 to 95 percent indicated that the caching setup was successfully reusing shared context across the swarm rather than reprocessing it from scratch for every agent.
Is it worth running a slower dense model for swarm work?
For agentic workloads where you’re delegating tasks and stepping away rather than chatting turn by turn, yes, based on this setup. The reasoning is straightforward: a single slow model still produces high quality output, and running many of them in parallel multiplies your effective throughput without sacrificing that quality. A “box of twenty geniuses” working slower but more carefully outperforms faster models that cut corners on complex tasks, particularly on long-running coding or research projects where correctness compounds over many steps.
The tradeoff shows up in wall-clock time per individual response and in how much you need to babysit maximum sequence limits versus how much output quality you’re willing to wait for. For pure chat use, where you want fast back-and-forth, a lighter or more heavily quantized model might still be the better fit. For agent swarms doing autonomous multi-step work, the calculus favors quality over per-message speed.
Frequently Asked Questions
What does “max number of sequences” control in vLLM?
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
It sets the ceiling on how many requests (in this case, agents or sub-agents) vLLM will process concurrently. Setting it too low limits your swarm size; setting it too high risks running out of VRAM or drastically increasing wait times as the server tries to juggle more parallel work than the hardware comfortably supports.
Why enable prefix caching for agent swarms?
Prefix caching lets vLLM reuse already-processed context across requests that share a common prefix, which speeds up reload times significantly when multiple agents are working from similar system prompts or shared context. It was specifically called out as critical for fast reload behavior in swarm setups.
Does a dense model at FP16 really outperform a quantized version for agents?
In the setup described, a heavily quantized variant (comparable to 4-bit) of a related model started failing on complex, long-running agentic coding tasks, while the same-class dense model at FP16 completed them successfully. This doesn’t mean quantization is always worse, but it suggests precision matters more as task complexity and session length increase.
Should GPU inference run in a VM or a container?
A container (specifically LXC) avoids the performance overhead associated with GPU passthrough in a full virtual machine, with no meaningful performance penalty observed compared to bare metal.
What’s a safe starting point for max sequences on a multi-GPU consumer setup?
Around 16 was described as a safe number, with 20 providing headroom for overflow or compaction without major risk. Pushing toward 24 still functioned but came with a noticeable increase in time spent waiting for outputs.
