How to Run SenseNova-Loop-9B-Preview Locally: Setup Guide
Step-by-step guide to running SenseNova-Loop-9B-Preview locally with Transformers, LightLLM, or SGLang, including setup and config details.
What is SenseNova-Loop-9B-Preview?
SenseNova-Loop-9B-Preview is an open-weight multimodal model from SenseNova built for long-context and agentic workflows, released under Apache 2.0. It takes text and images as input and produces text output, and it’s designed around a looped decoder architecture rather than the standard single-pass transformer stack most local AI builders are used to deploying. Running it requires a custom Transformers implementation, specific library versions, and awareness of how the looped execution path affects memory and compute.
TL;DR
- Looped architecture reuses 24 decoder layers twice in sequence, giving 56 decoder-layer computations per forward pass while only storing 32 layers’ worth of weights.
- Parameter footprint sits at roughly 9B stored parameters but behaves like a ~14.6B model at inference time because of the repeated computation.
- Transformers deployment needs version 5.18.0, PyTorch 2.11.0 with CUDA 13.0, and
trust_remote_code=Trueto load the custom model code. - Production serving options include LightLLM (Docker-based, tested with a 4-GPU tensor-parallel setup) and a SGLang branch built from an open pull request.
- Long-context training supports sequences up to 262,144 tokens, and the model mixes linear and full attention in a repeating four-layer pattern.
- Inference acceleration via MTP (multi-token prediction) is already available, while Dspark and Dflash2 speculative decoding methods are still in training.
- Precision is BF16 for the main backbone, with the mHC routing parameters kept in FP32 for numerical stability.
How does the looped decoder architecture work?
The model’s defining feature is that it doesn’t execute each decoder layer exactly once. Instead, the first four layers run normally, then a block spanning layers 4 through 27 (24 layers total) runs twice in sequence with the exact same weights, and finally layers 28 through 31 plus the output head finish the pass. If you call that repeated block B and its input h, the computation is literally B(B(h)).
The practical effect: a model with 32 distinct decoder layers performs 56 layer executions per forward pass. You get a deeper, more expressive computation path without the memory cost of storing twice as many unique weights. This is why the model card distinguishes between “backbone scale” (~9B total parameters) and “inference parameters” (~14.6B), since the effective compute during generation resembles a larger model even though disk space and weight loading reflect the smaller count.
Layered on top of this is a technique called two-stream manifold-constrained hyper-connections (mHC). Inside the looped section, two residual streams carry information around the decoder layers instead of one. Learned mappings control how each layer reads from the streams, blends information between them, and writes its output back. A Sinkhorn normalization step (20 iterations in the documented config) constrains how these streams mix. This routing logic adds a layer of configuration that any inference engine needs to replicate exactly, which is why the model ships with custom code rather than relying on a generic decoder class.
The attention pattern is also non-standard: every group of four layers contains three linear-attention layers followed by one full-attention layer, and the loop boundaries are aligned so this pattern stays consistent across both passes through the repeated block.
What hardware and software do you need?
The documented and verified environment uses Transformers 5.18.0 and PyTorch 2.11.0 with CUDA 13.0. Before installing Python packages, get a PyTorch build that matches your CUDA setup from the official PyTorch installation instructions, then install the rest:
python -m pip install "transformers==5.18.0" accelerate safetensorsThe model directory needs the configuration files, tokenizer, chat template, custom model code, and the weight shards themselves, whether you’re pulling from the Hugging Face repository or pointing at a local directory.
For precision, the exported weights use BF16 for the main backbone, with the mHC routing parameters kept in FP32. That mixed-precision setup matters if you’re planning to quantize further or convert the checkpoint, since routing parameters are more sensitive to precision loss than the rest of the network. The model card doesn’t publish explicit VRAM figures, so plan GPU memory around a ~14.6B-parameter-class model in BF16, plus overhead for the two residual streams held during the looped section and whatever context length you’re targeting.
How do you run it with Transformers?
The basic single-GPU text generation path uses the standard AutoModelForCausalLM and AutoTokenizer interfaces, with trust_remote_code=True set so Transformers loads the custom SenseNova Loop model class instead of trying to force it into an existing architecture:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sensenova/SenseNova-Loop-9B-Preview"
device = "cuda:0"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
device_map={"": device},
).eval()
messages = [{"role": "user", "content": "What is 13 times 7? Reply with the number only."}]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=256,
do_sample=False,
use_cache=True,
)
response = tokenizer.decode(
outputs[0, inputs.input_ids.shape[1]:],
skip_special_tokens=True,
)
print(response)- Name
- Maria Alvarez
- Date of birth
- SSN
- Diagnosis
- Balance due
- $412.80
This example runs greedy decoding and caps output at 256 new tokens. Setting enable_thinking=True turns on a thinking mode for more deliberate, multi-step responses, useful for agentic or tool-use tasks where the model needs to reason before acting. Swap the device string to target a different GPU if you’re running a multi-GPU box but want to pin inference to one card.
Which serving frameworks work for production deployment?
Two inference servers currently support the SenseNova Loop architecture: LightLLM and SGLang.
LightLLM has the more straightforward path since it’s available as a pre-built Docker image. The documented example uses four GPUs with tensor parallelism:
docker pull ghcr.io/modeltc/lightllm:main
docker run --rm --gpus all --ipc=host -p 8000:8000 \
-v /path/to/SenseNova-Loop-9B-Preview:/model \
-e LOADWORKER=4 -e LIGHTLLM_TRITON_AUTOTUNE_LEVEL=1 \
ghcr.io/modeltc/lightllm:main \
python -m lightllm.server.api_server \
--model_dir /model --model_name SenseNova-Loop-9B-Preview \
--tp 4 --host 0.0.0.0 --port 8000Once running, it exposes an OpenAI-compatible chat completions endpoint:
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "SenseNova-Loop-9B-Preview", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 128}'SGLang support currently lives in an open pull request rather than a mainline release, so you need to build from source:
git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/47281/head
git checkout -b sensenova-loop FETCH_HEAD
python -m pip install -e ./python
sglang serve --model-path /path/to/SenseNova-Loop-9B-PreviewBecause this is a PR branch rather than a tagged release, expect it to change or get superseded once the implementation merges upstream.
Is it worth running locally right now?
This is a preview release, and that label matters. The technical report is listed as TBD, the Transformers inference implementation’s “publication details” are also still pending, and acceleration methods like Dspark and Dflash2 are explicitly marked as still training. If you want a stable, well-documented production dependency today, that’s a reason for caution.
On the other hand, the architecture itself is interesting enough to justify hands-on experimentation. Looped transformers with shared weights are a real technique for getting more effective depth out of a smaller parameter budget, and pairing that with two-stream mHC routing is a distinctive design choice you won’t find in most open-weight releases. The benchmark comparisons on the model card cover a dense set of agentic and document-understanding tasks (coding, PPT generation, data analysis, workflow automation), suggesting the target use case is genuinely agent-oriented rather than general chat.
If your workload involves long documents, multimodal inputs, or tool-calling agents, and you’re comfortable working with a preview-stage model where some performance features (MTP is available now, but Dspark and Dflash2 aren’t) are only partially rolled out, it’s reasonable to test. If you need a drop-in production model with a finished technical report and mature serving support across more inference engines, wait for a stable release.
Frequently Asked Questions
Does SenseNova-Loop-9B-Preview support images as input?
Yes. The model accepts both text and images as input and generates text output, positioning it for document understanding and multimodal agentic tasks alongside pure text generation.
What’s the maximum context length it supports?
The documented training recipes go up to 262,144 tokens (256K), combining long sequences with multimodal inputs and tool-use trajectories.
Why does it need trust_remote_code=True?
The looped decoder, hybrid attention pattern, and two-stream mHC routing are custom architectural components not present in standard Transformers model classes. Setting trust_remote_code=True tells Transformers to load the model’s own Python implementation instead of trying to map it onto an existing architecture.
Can I fine-tune this model myself?
Yes. SenseNova provides a training recipe using torchtitan for supervised fine-tuning on a single 8-GPU node, along with separate training scripts for DSpark and DFlash2 speculative decoding draft models via the SenseSpec repository.
How many GPUs do I need to serve this model?
The documented LightLLM example uses four GPUs with tensor parallelism, though the model card doesn’t specify this as a hard minimum. Single-GPU inference works for basic text generation through Transformers, but production serving at scale is demonstrated with multi-GPU setups.
