Structured data extraction and categorical classification by Large Language Models (LLMs) traditionally suffer from an inherent inefficiency: token-by-token autoregressive decoding. For each field of a JSON schema, the model performs dozens or even hundreds of passes before finalizing the output.
On Apple Silicon's unified architecture, the Parallel Constrained Decoding project implemented via the MLX framework revolutionizes this approach. By simultaneously evaluating multi-field JSON schemas through KV-Cache Broadcasting and Logit Slicing, it reduces latency by 5.6x to 7.0x while ensuring absolute syntactic validity.
The Bottleneck of Classical Autoregressive Decoding
Standard structured generation methods—such as native API JSON mode or grammar-guided decoding (GBNF)—rely on sequential generation.
Each token requires a distinct pass through the neural network layers, consuming memory bandwidth at every step. Consequently, latency increases linearly with the length of the response.
Beyond latency, this approach exposes applications to risks of syntactic degradation, forgotten mandatory fields, or hallucinations on JSON keys.
The Architecture of Parallel Constrained Decoding
In extraction and classification scenarios, possible values typically belong to bounded sets (booleans, enumerations, choice lists). Parallel constrained decoding exploits this fundamental characteristic to restructure probability calculation.
The 6-Step Execution Pipeline
- Single Broadcast Prefill: The text context and semantic schema descriptions are passed just once into the model's Key-Value cache (KV Cache) via MLX.
- KV Cache Broadcasting: The KV cache is duplicated and simultaneously broadcast across all $M$ fields of the target schema.
- Sub-Vocabulary Logit Slicing: For each field, the engine masks the entire model vocabulary except for the token identifiers corresponding to valid choices authorized by the schema.
- Calibrated Softmax Probability Calculation: Exact normalized probabilities are computed over the restricted candidate slice.
- Token Tree Disambiguation: If candidate choices share multi-token prefixes, the engine evaluates continuation branches using the sliced cache states without redundant memory reallocation.
- Programmatic Assembly: The final JSON document is assembled directly from the validated values, guaranteeing 100% perfect syntactic compliance.
Performance Benchmarks on Apple Silicon M4 Max
Evaluation tests conducted with the mlx-community/Qwen2.5-1.5B-Instruct-4bit model under macOS Sequoia demonstrate spectacular performance gains:
- Fintech Anti-Fraud Routing (4 fields): 75 ms (5.6x speedup - 100% validity)
- Code Security Audit (4 fields): 68 ms (5.6x speedup - 100% validity)
- Pricing Classification (1 field, 255 choices): 89 ms (5.6x speedup - 100% validity)
- Enterprise Support Triage (28 fields): 270 ms (7.0x speedup - 100% validity)
Implementation with the Python SDK
The engine easily integrates into Python processing pipelines thanks to an intuitive API based on the MLX SDK.
1. Defining Data Schemas
Schemas are defined using typed structures (StructuredSchema), specifying for each field its type (enum or boolean), a textual description guiding the model's reasoning, and the allowed choices.
2. Executing Parallel Inference
Extraction is launched by passing the text context and the compiled schema to the execution function to obtain validated results and detailed telemetry per field.
Conclusion
Parallel constrained decoding on Apple Silicon demonstrates that it is possible to overcome the latency limits inherent in sequential autoregressive LLM generation. By combining MLX's unified hardware acceleration with logit slicing and KV cache broadcasting, structured data processing architectures achieve execution speeds compatible with the demands of critical production environments.