Parallel Constrained Decoding: 7x Faster JSON on Apple Silicon
Parallel constrained decoding cuts structured JSON extraction latency up to 7x on Apple Silicon by scoring schema fields at once, not token by token.
What is parallel constrained decoding?
Parallel constrained decoding is a technique for generating structured JSON output from a language model by evaluating every schema field at once instead of producing tokens one after another. Built on Apple’s MLX framework, an implementation using mlx-community/Qwen2.5-1.5B-Instruct-4bit on an M4 Max shows latency reductions of 5.6x to 7.0x versus standard autoregressive decoding, while guaranteeing valid JSON syntax on every run. It works by prefilling the context once, broadcasting that cached state across all fields, and scoring only the valid candidate tokens for each field in parallel.
TL;DR
- Field-parallel scoring replaces sequential token generation: the model prefills context once and evaluates all schema fields simultaneously off a shared KV-cache instead of stepping through tokens one at a time.
- Benchmarks on an Apple Silicon M4 Max show speedups ranging from 5.6x on small 4-field schemas to 7.0x on a 28-field enterprise triage schema.
- The approach restricts each field to its valid candidate token set (booleans or enum choices), masking the rest of the vocabulary so invalid output becomes structurally impossible.
- Because values come from a constrained candidate slice, the method reports calibrated softmax probabilities for each field, giving a confidence score alongside every extracted value.
- Output is assembled programmatically from verified values rather than parsed from generated text, which is what produces the claimed 100% schema validity across all tested scenarios.
- The technique targets classification-style extraction (fraud routing, ticket triage, tariff codes) where fields map to bounded enums or booleans, not open-ended free-text generation.
- A published benchmark suite and web visualizer let developers compare parallel and autoregressive decoding side by side, including token-level hallucination and omission detection in the naive baseline.
How does parallel constrained decoding actually work?
Standard structured generation, even with JSON mode or grammar constraints, still runs autoregressively: the model predicts one token, appends it to the sequence, and repeats. A four-field JSON object might need 150 to 500 forward passes depending on value length and formatting overhead. Each pass requires its own round trip through GPU or NPU memory, so latency scales roughly linearly with output length.
Parallel constrained decoding restructures the problem around a specific property of extraction and classification tasks: the answer for each field belongs to a small, known set of options. A “risk_level” field might only ever be HIGH, MEDIUM, or LOW. A “requires_review” field is just a boolean. Instead of generating these values as free text, the engine:
- Prefills the context once. The source document and the schema’s field descriptions are run through the model a single time, producing one KV-cache state.
- Broadcasts that cache across fields. Rather than re-running the prefill for each field, the same cached state is reused for every field’s evaluation.
- Slices the vocabulary per field. For each field, only the token IDs corresponding to valid choices are scored. Everything else in the vocabulary is masked out before the softmax.
- Computes calibrated probabilities. A standard softmax (with temperature) is applied over just the candidate slice, producing normalized, comparable probabilities for each option.
- Handles multi-token choices via a token tree. When candidate values share a prefix (for example two enum choices that start with the same token), the engine runs a short continuation step against sliced cache states without reallocating memory.
- Assembles JSON programmatically. Because every field’s value is chosen from a verified candidate list, the final JSON is built directly from those values rather than generated and then parsed, which is what eliminates malformed output.
The net effect: a schema that would take one sequential forward pass per output token instead takes roughly one broadcast prefill plus a small number of parallel field evaluations, regardless of how many fields the schema has.
Why does this matter for structured extraction?
Most production use cases for LLM-based extraction (fraud triage, support ticket routing, tariff or product classification, security audit tagging) don’t need creative, open-ended text. They need a value chosen from a known list, attached to a confidence score, delivered fast and reliably. Autoregressive decoding is overkill for this: it pays the full cost of sequential generation to eventually spit out a token that was always going to come from a small enum.
Two problems specifically hurt autoregressive structured generation at scale:
- Latency scales with schema size. A 28-field ticket triage schema requires far more sequential steps than a 4-field fraud check, so latency grows with complexity even though the actual information content per field is small.
- Syntax and field reliability degrade. Free-form token generation is prone to malformed JSON, omitted keys, or hallucinated fields, especially as output length grows. Fixing this normally requires grammar constraints, retries, or post-hoc validation.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
By collapsing generation into a small number of parallel forward passes and constraining the vocabulary per field, the technique sidesteps both problems at once: fewer sequential steps and no invalid syntax to catch.
How much faster is it, and does accuracy hold up?
On an M4 Max, benchmarks using the 4-bit quantized Qwen2.5-1.5B-Instruct model show the following:
| Scenario | Fields | Autoregressive | Parallel | Speedup |
|---|---|---|---|---|
| Fintech fraud routing | 4 | 420 ms | 75 ms | 5.6x |
| Code security audit | 4 | 380 ms | 68 ms | 5.6x |
| High-cardinality tariff classification | 1 field, 255 choices | 500 ms | 89 ms | 5.6x |
| Enterprise support triage | 28 | 1,900 ms | 270 ms | 7.0x |
The pattern is consistent: schemas with more fields see larger speedups, since autoregressive latency grows with total output tokens while the parallel approach stays close to a single prefill plus lightweight field scoring. The high-cardinality case (one field with 255 possible tariff codes) is notable because it shows the technique isn’t just about field count. Even a single field with a large candidate set benefits, since the constraint is on vocabulary size per decision, not schema complexity per se.
On validity, every tested scenario reports 100% schema match and guaranteed valid JSON syntax for the parallel method, versus a naive autoregressive baseline that is susceptible to hallucinated or missing keys. Because values are pulled from calibrated probability distributions over the actual candidate set, each field also comes with an inspectable confidence score and a ranked list of alternative choices, which is useful for flagging low-confidence extractions for human review.
Where does this approach fall short?
The technique is scoped narrowly by design. It applies to fields defined as enums or booleans with bounded candidate sets, not to open-ended generation like summarization or free-text answers. If a field genuinely needs generated prose rather than a categorical choice, this method doesn’t apply and standard autoregressive decoding is still the right tool.
The published implementation is also tied to a specific stack: MLX on Apple Silicon (M1 through M4 series, macOS 14 or later), tested with mlx-community/Qwen2.5-1.5B-Instruct-4bit. The engine can load other decoder models supported by mlx-lm by changing the model ID, but benchmarks are only published for the one model and hardware combination. Results on other Apple Silicon chips, other quantizations, or non-Apple hardware aren’t documented.
Enum cardinality also isn’t unlimited. The schema format supports up to 255 choices per field, which covers many real-world classification tasks (tariff codes, department routing, priority tiers) but wouldn’t extend to open vocabularies or free-form categorical spaces without modification.
Is parallel constrained decoding worth using?
For teams running local, on-device extraction and classification pipelines on Apple Silicon, this is a clear latency and reliability win over naive autoregressive JSON generation, provided the task fits the enum/boolean field model. Sub-100-millisecond structured extraction on consumer hardware, with guaranteed valid syntax and per-field confidence scores, is meaningfully different from grammar-constrained sampling that still pays a per-token cost.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
The tradeoff is scope. It’s a specialized tool for categorical extraction, not a general replacement for LLM generation. Anyone building fraud triage, support routing, security tagging, or classification systems that run locally on Apple hardware gets a direct benefit. Anyone needing free-text generation, cloud-based serving on non-Apple hardware, or extraction fields with unbounded value spaces will need a different approach or a hybrid setup that reserves this method for the categorical fields within a larger schema.
Frequently Asked Questions
What hardware does parallel constrained decoding require?
It’s built on MLX and requires an Apple Silicon Mac (M1, M2, M3, or M4 series) running macOS 14.0 or later, with Python 3.10+.
Which models work with this technique?
The published benchmarks use mlx-community/Qwen2.5-1.5B-Instruct-4bit. Any decoder model supported by the mlx-lm library can be loaded by changing the model ID in the engine configuration, though only the Qwen2.5 1.5B model has published benchmark numbers.
How is 100% schema validity guaranteed?
Because each field’s value is selected from a masked, constrained set of valid candidate tokens and the JSON is assembled programmatically from those verified values, there’s no free-text generation step that could produce malformed syntax or invalid keys.
Does this work for free-text fields, not just categories?
No. The method is designed for fields with bounded candidate sets, enums (up to 255 choices) and booleans. Open-ended text generation still requires standard autoregressive decoding.
How much faster is it than normal JSON generation?
Benchmarks on an M4 Max show 5.6x speedup on smaller schemas (4 fields, or a single high-cardinality field) and up to 7.0x on a larger 28-field schema, with latency dropping from around 1.9 seconds to roughly 270 milliseconds in the largest tested case.
