GGUF quantization Articles
Browse 7 articles about GGUF quantization.
Ternary Bonsai 2 9B Tested: How Much Does 2-Bit Quantization Cost You?
Hands-on tests of the 2-bit ternary Bonsai 2 9B model across coding, multilingual, reasoning, and vision tasks reveal where quality actually breaks down.
Underdog Saluki 27B: Run a 27B Tool-Calling Model on 8GB VRAM
Underdog Saluki 27B compresses Qwen3.8-27B into a 7.89GB GGUF that keeps tool-calling intact, runnable on stock llama.cpp.
Run GLM 5.3 Flash Locally: GSQ and RCO Quantization Explained
How GSQ and RCO quantization shrink the 320B GLM 5.3 Flash model to under 140GB, and how to build llama.cpp and serve it on local GPUs.
MiniCPM 5 2B: A Small Tool-Calling Model Built for Sub-Agents
MiniCPM 5 2B targets tool calling and sub-agent workloads. Here's how it benchmarks against 4B models and runs locally via llama.cpp and SGLang.
MiniCPM5-2B: Running OpenBMB's 2B On-Device Model Locally
MiniCPM5-2B packs 2.5B params, 131K context, and 4B-beating benchmarks. Here's how it works and how to run it via GGUF, MLX, or GPTQ.
Qwen3.8-4B Distilled: Q4 vs Q6 vs Q8 Quantization Compared
Hands-on benchmark of Q4, Q6, and Q8 GGUF quants for the Qwen3.8-4B distilled model, testing speed, VRAM use, and reasoning depth on llama.cpp.
How to Run fuse-1 Lite Locally: VRAM, Setup, and Formats
How to run fuse-1 Lite's 5.72B coding MoE model locally via GGUF, MLX, vLLM, or bitsandbytes, with VRAM needs for each backend.