How to Run Hy-Memory-30B-A3B Locally with vLLM
Step-by-step guide to deploying Tencent's Hy-Memory-30B-A3B memory writer locally with vLLM and its OpenAI-compatible API.
What is Hy-Memory-30B-A3B and why run it locally?
Hy-Memory-30B-A3B is a memory-writer model from Tencent Hunyuan built to handle the “write path” inside external memory frameworks like Mem0, MemOS, and Hy-Memory itself. Instead of calling a large general-purpose LLM to decide what a system should remember about a user or conversation, you swap in this model, which is purpose-trained for that one job. It has roughly 3B active parameters, which keeps inference cheap even though it’s distilled from much larger teacher models. Running it locally with vLLM gives you an OpenAI-compatible endpoint you can drop into any memory pipeline without sending conversation data to a third-party API.
TL;DR
- Hy-Memory-30B-A3B is a compact (~3B active parameter) model designed specifically to extract and update long-term memory inside frameworks like Mem0, MemOS, and Hy-Memory.
- It was trained in two stages, Memory On-Policy Distillation followed by Attributed Memory Reinforcement Learning, which lets it generalize to memory frameworks it never saw during training, including EverMemOS.
- On a fixed evaluation matrix spanning four harnesses, it posted an average score of 55.49, edging out larger models including Claude Opus 5 (54.62) and DeepSeek V4.1 Flash (53.64).
- Deployment runs through vLLM version 0.31.0, exposing a standard OpenAI-compatible chat completions endpoint.
- A no-think mode (set via
reasoning_effort: "no_think"in the chat template kwargs) is available for fast, low-latency memory extraction without chain-of-thought overhead. - The model and benchmark code are published on Hugging Face and GitHub under Tencent Hunyuan, along with a technical report detailing the training methodology.
- Because it only replaces the memory-writing component, you keep your existing retrieval, answer model, and judge configuration unchanged when swapping it in.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
How do you set up Hy-Memory-30B-A3B with vLLM?
The quickest path is to serve the model through vLLM and talk to it using the standard OpenAI Python client. Tencent’s documentation specifies vLLM version 0.31.0 as the tested baseline, and the model should be served with a custom name flag:
--served-model-name hy-memoryOnce the server is running (default local endpoint at http://127.0.0.1:8000/v1), you connect using the openai Python package, pointing the base_url at your local vLLM instance and passing a placeholder API key since no authentication is required for a local deployment:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="EMPTY",
)From there, calls look exactly like any OpenAI chat completion request. You specify model="hy-memory" to match the served name, and pass a messages array with system and user roles.
What does the no-think mode example actually do?
The published quickstart example uses Hy-Memory-30B-A3B for a common memory-writing task: pulling explicit facts and preferences out of a user message and returning them as structured JSON. The system prompt instructs the model to extract only explicit facts, avoid inference or invention, and return a memories list of concise, self-contained statements. The user message in the example includes details like a name, location, job, programming language preference, and a dietary restriction (vegetarian), and the model’s job is to isolate these into discrete memory entries rather than inventing extra context.
The key setting that makes this efficient is reasoning_effort: "no_think", passed through extra_body.chat_template_kwargs. This disables the model’s internal reasoning/chain-of-thought step, which matters for a memory writer because this task runs constantly in the background of a live application. You don’t want multi-second reasoning latency every time a chatbot needs to log a new fact about a user. Combined with temperature=0.0 for deterministic output and a max_tokens cap of 512, the example is tuned for fast, repeatable extraction rather than open-ended generation.
How does it perform compared to larger models?
Tencent evaluated Hy-Memory-30B-A3B against several general-purpose models used as memory writers, including Claude Opus 5, DeepSeek V4.1 Flash, and various Qwen and Hunyuan base models. The evaluation held everything else fixed, same harness version, same prompts, same retrieval setup, same answer model and judge, so the comparison isolates the writer model’s contribution.
Across four harnesses (Mem0, Hy-Memory, MemOS, and the held-out EverMemOS), each scored on six metrics (LoCoMo Judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, and Code Memory Bench), Hy-Memory-30B-A3B posted the top overall average at 55.49, ahead of Claude Opus 5 at 54.62 and DeepSeek V4.1 Flash at 53.64. Notably, it held up on EverMemOS, a framework excluded from training, scoring 56.89 there, which supports the claim that the two-stage training approach generalizes rather than just memorizing the specific harnesses it was trained against.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
It didn’t win every individual sub-metric. Claude Opus 5 led on raw LoCoMo and PersonaMem-v2 scores in several harnesses, and DeepSeek V4.1 Flash edged it out on some Code Memory Bench numbers. But as an aggregate memory writer across diverse scenarios and a much smaller active parameter count, the results suggest a specialized smaller model can match or beat general-purpose frontier models on this specific task.
Is Hy-Memory-30B-A3B worth deploying over a general-purpose LLM?
If you’re already running Mem0, MemOS, or a similar memory framework and currently routing memory-write calls through a large general LLM, swapping in Hy-Memory-30B-A3B is a reasonable trade to evaluate. The benefits are mainly cost and latency: a 3B active parameter model is far cheaper to run than a frontier model, and local deployment through vLLM means no per-call API cost and no data leaving your infrastructure.
The trade-off is scope. This model is trained specifically for memory writing tasks, deciding what to retain, how to represent it, and how to merge new evidence into existing memory banks. It is not a general chatbot replacement, and the benchmark results reflect performance within that narrow task, not general reasoning or conversational ability. If your pipeline already separates the memory-writer role from the answer-generation role (which most mature memory frameworks do), this is a drop-in replacement worth testing against your current writer model. If your architecture doesn’t separate these concerns, you’d need to restructure before this model is useful.
Frequently Asked Questions
What is the minimum vLLM version required for Hy-Memory-30B-A3B?
Tencent’s documentation lists version 0.31.0 as the tested vLLM version for serving the model.
Does Hy-Memory-30B-A3B replace the entire memory framework?
No. It replaces only the memory-writing component inside frameworks like Mem0, MemOS, and Hy-Memory. Retrieval, the answer-generation model, and the judge components in your pipeline stay unchanged.
What is “no-think mode” and when should I use it?
No-think mode disables the model’s internal reasoning or chain-of-thought process, set via reasoning_effort: "no_think" in the chat template kwargs. Use it for latency-sensitive memory extraction tasks where you need fast, deterministic output rather than deliberative reasoning.
How many active parameters does the model have?
Hy-Memory-30B-A3B has approximately 3B active parameters, despite the “30B” in its name referring to total parameters in its architecture, which keeps inference costs low relative to its performance on memory-writing benchmarks.
Can it work with memory frameworks it wasn’t trained on?
Yes. The model was tested on EverMemOS, a framework held out of training entirely, and still scored competitively (56.89 average), which Tencent attributes to the two-stage training process of on-policy distillation followed by reinforcement learning.
