How to Run Local AI Models with Claude Code to Cut Costs by 10x

The Hidden Cost Problem Inside Claude Code Workflows

If you’ve been running Claude Code on anything beyond small personal projects, you’ve probably noticed the bills. A few hundred dollars a month isn’t unusual for teams doing serious AI development work. Claude Code is excellent at what it does — but it routes nearly everything through frontier model APIs, and that adds up fast.

The thing is, a significant chunk of what Claude Code does in a typical workflow doesn’t actually require a frontier model. Embeddings, transcription, text classification, intent detection, simple summarization — these tasks are well within the capability of local open-source models that cost nothing per inference.

This guide covers exactly how to offload those tasks to local models running on your own machine, leaving Claude Code to do what it’s genuinely best at: complex reasoning, code generation, and high-stakes decisions. The result is a hybrid architecture that can cut your total AI inference costs by 10x or more.

---

Why Claude Code Sends Everything to the API (and Why That’s Expensive)

Claude Code is a terminal-based AI coding agent built by Anthropic. It reads your codebase, runs shell commands, edits files, and completes multi-step development tasks using Claude models as its reasoning engine.

The default setup routes all inference through Anthropic’s API. Every token costs money — and as your agent handles more complex pipelines, token costs compound quickly. A single agentic session that reads files, generates code, and handles tool calls can consume tens of thousands of tokens without you noticing.

Understanding how token-based pricing works helps clarify why the cost accumulates so fast. It’s not just the output tokens — it’s the full context window being sent on every call, including all prior conversation history.

The practical consequence: developers building real AI agent workflows on Claude Code often spend $200–$500/month on API costs, even with careful session management. Heavy users can go well beyond that.

---

The Core Idea: Task Routing by Complexity

Not all tasks need the same model. That’s the insight behind keeping costs manageable at scale.

A rough tier breakdown:

Task	Model Tier Needed	Example Local Alternative
Complex code generation	Frontier (Claude)	Not replaceable
Multi-step reasoning	Frontier (Claude)	Not replaceable
Text embeddings	Small local model	nomic-embed-text, mxbai-embed
Audio transcription	Specialized local model	Whisper
Text classification	Small local model	Llama 3.2 3B, Gemma
Intent detection	Small local model	Phi-3 Mini, Qwen 2.5
Simple summarization	Mid-tier local model	Mistral 7B, Gemma 9B
Entity extraction	Small local model	Llama 3.2 3B

The idea is straightforward: route each task to the cheapest model that can handle it reliably. This is the foundation of multi-model routing for AI agent cost optimization.

For Claude Code specifically, this means intercepting the tasks that don’t need Claude and handling them locally before they ever hit the API.

---

Setting Up Ollama as Your Local Inference Server

Ollama is the standard tool for running open-source models locally. It handles model downloads, memory management, and serves a local REST API that’s compatible with the OpenAI API format — which means you can point tools that expect an OpenAI-compatible endpoint at Ollama with minimal configuration.

Install and Configure Ollama

# macOS or Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download the installer from ollama.com

Once installed, start the server:

ollama serve

By default, Ollama listens on http://localhost:11434. You can verify it’s running with:

curl http://localhost:11434/api/tags

Pull the Models You Need

For a typical Claude Code offloading setup, you’ll want at minimum:

# A capable general model for classification and summarization
ollama pull llama3.2:3b

# A fast embedding model
ollama pull nomic-embed-text

# Whisper for audio transcription (via whisper.cpp or faster-whisper separately)

For transcription specifically, Ollama doesn’t bundle Whisper directly — you’ll handle that separately. More on that below.

If you want a stronger general model for mid-complexity tasks, consider pulling Gemma or Mistral. There’s a detailed step-by-step guide to running Gemma 4 locally with Ollama if you want to use Google’s latest open-weight model. For fine-tuning and self-hosting options, Mistral Small 4 is also worth considering.

---

Offloading Embeddings to Local Models

Embeddings are one of the clearest wins for local offloading. They’re called constantly in RAG pipelines, semantic search, and document indexing — and they don’t require frontier model intelligence.

Sending embeddings through Claude or GPT-4 APIs is almost always unnecessary. A well-trained local embedding model like nomic-embed-text or mxbai-embed-large produces embeddings that are competitive with commercial offerings for most use cases.

Setting Up Local Embeddings in Your Pipeline

With Ollama running, you can call embeddings via its REST API:

import requests

def get_embedding(text: str) -> list[float]:
    response = requests.post(
        "http://localhost:11434/api/embeddings",
        json={
            "model": "nomic-embed-text",
            "prompt": text
        }
    )
    return response.json()["embedding"]

If your codebase uses LangChain or LlamaIndex, both support Ollama embeddings as a drop-in replacement:

from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="nomic-embed-text")

This is particularly relevant for RAG pipelines where embedding costs can dominate. If you’re building knowledge retrieval into your Claude Code workflows, local embeddings are a near-zero-cost alternative.

What You Lose

Local embedding models are smaller and may have slightly lower quality on highly domain-specific or multilingual content. For most English-language code and documentation tasks, the difference is negligible. Run a quick comparison on your actual data before committing.

---

Offloading Transcription to Local Whisper

Audio transcription is another obvious offloading candidate. If your AI agent workflow ingests voice notes, meeting recordings, or any audio input, you’re probably paying for a cloud transcription API.

OpenAI’s Whisper model is open-source and runs locally. The faster-whisper library provides a highly optimized implementation:

pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("base.en", device="cpu", compute_type="int8")

def transcribe_audio(file_path: str) -> str:
    segments, info = model.transcribe(file_path)
    return " ".join([segment.text for segment in segments])

For better accuracy, use the medium or large-v3 model variants — though they require more RAM and CPU. On a modern MacBook Pro with Apple Silicon, large-v3 runs fast enough for most production workloads.

If you’re evaluating transcription quality before committing to local, the comparison of MAI Transcribe 1, Whisper, and Gemini Flash gives a clear breakdown of where each model excels.

Transcription Cost Comparison

Method	Cost per hour of audio
OpenAI Whisper API	~$0.36
Google Speech-to-Text	~$0.96–$1.44
Local Whisper (base.en)	$0.00
Local Whisper (large-v3)	$0.00

At scale, this is significant. A workflow that processes 100 hours of audio monthly saves $35–$140/month from transcription alone — with zero quality loss for most use cases.

---

Offloading Classification and Intent Detection

Classification tasks are well-suited for smaller local models. Sentiment analysis, intent detection, topic categorization, content moderation filtering — none of these require Claude Sonnet or Opus.

A 3B parameter model like Llama 3.2 3B handles classification reliably with a well-structured prompt. The key is giving it a constrained output format.

Example: Intent Classification with a Local Model

import requests
import json

def classify_intent(user_input: str) -> str:
    prompt = f"""Classify the following user input into exactly one of these categories:
[question, request, complaint, feedback, other]

User input: "{user_input}"

Respond with only the category label, nothing else."""

    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": "llama3.2:3b",
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.0}
        }
    )
    return response.json()["response"].strip().lower()

Setting temperature to 0 and constraining the output format dramatically improves reliability for classification tasks. You can also use structured output with models that support JSON mode.

---

Configuring Claude Code to Use Local Models for Specific Tasks

Here’s where it gets practical. Claude Code itself doesn’t natively support routing specific tasks to local models — you need to build a thin middleware layer that intercepts certain requests and handles them locally.

The cleanest approach is to build a local proxy server that sits between your code and the Anthropic API, routing requests based on task type.

Building a Simple Task Router

from fastapi import FastAPI, Request
import httpx
import json

app = FastAPI()

LOCAL_TASKS = ["embed", "classify", "transcribe"]

@app.post("/v1/messages")
async def route_request(request: Request):
    body = await request.json()
    
    # Check if this is a local-eligible task
    task_type = detect_task_type(body)
    
    if task_type in LOCAL_TASKS:
        return await handle_locally(task_type, body)
    else:
        # Forward to Anthropic API
        return await forward_to_anthropic(body)

This pattern — routing cheap tasks locally while forwarding complex tasks upstream — is the foundation of hybrid AI agent architecture. The key is instrumenting your task detection logic well enough to route accurately.

Task detection can be as simple as checking metadata flags you set in your prompts, or as sophisticated as a trained classifier that reads the request content.

Using Claude Code’s --api-url Flag

Claude Code supports a custom API URL via the ANTHROPIC_BASE_URL environment variable. You can point this at your local proxy:

export ANTHROPIC_BASE_URL="http://localhost:8080"
claude

Now every request from Claude Code passes through your proxy first. Local-eligible tasks get handled immediately; everything else gets forwarded to Anthropic.

This is conceptually similar to running Claude Code through Open Router, which uses the same ANTHROPIC_BASE_URL override to route requests through a different backend.

---

Practical Cost Breakdown: Before and After

Let’s put numbers to this. Here’s a representative workflow for a developer building an AI-powered document processing agent:

Monthly tasks:

• 50,000 embedding calls (for document indexing)
• 200 hours of audio transcription
• 500,000 classification calls (content routing)
• 1,000,000 tokens of complex reasoning (code generation, multi-step analysis)

Before local offloading (all through Claude API):

Task	Volume	Cost
Embeddings (claude-3-haiku)	50,000 calls	~$25
Transcription (Whisper API)	200 hours	~$72
Classification (claude-3-haiku)	500K calls	~$150
Complex reasoning (claude-3-5-sonnet)	1M tokens	~$30
Total		~$277/month

After local offloading:

Task	Volume	Cost
Embeddings (nomic-embed-text, local)	50,000 calls	$0
Transcription (local Whisper)	200 hours	$0
Classification (Llama 3.2 3B, local)	500K calls	$0
Complex reasoning (claude-3-5-sonnet)	1M tokens	~$30
Total		~$30/month

That’s roughly a 9x cost reduction — with no change to output quality on the tasks that matter. The reasoning and code generation that actually requires Claude still gets Claude.

---

Managing What’s Left: Token Efficiency for Claude Calls

Even after offloading cheap tasks, you still want to minimize waste on the remaining Claude API calls. This is where token budget management in Claude Code becomes relevant.

A few high-impact practices:

Compress context before sending. Instead of feeding raw files, pre-process them locally to extract only the relevant portions. A small local model can summarize or filter context that gets passed to Claude.

Use Plan Mode for complex tasks. Claude Code’s Opus Plan Mode lets you use Opus for the planning step and a cheaper model for execution — saving tokens on the expensive parts of the workflow.

Watch for MCP token overhead. If you’re using Claude Code with MCP servers, be aware that MCP server context can bloat token consumption significantly. Keep MCP schemas lean.

The 18 Claude Code token management techniques covers the full set of options for reducing API spend on sessions that do need to run through Anthropic.

---

Choosing the Right Local Models for Each Job

Not all local models are equal for offloaded tasks. Here’s what works well in practice:

For Embeddings

• nomic-embed-text — Fast, good quality, 137M parameters. Best default choice.
• mxbai-embed-large — Higher quality, slower. Worth it for production RAG systems.
• all-minilm — Tiny and fast, acceptable for simple similarity search.

For Classification and Simple Reasoning

• Llama 3.2 3B — Good baseline for classification tasks.
• Phi-3 Mini (3.8B) — Particularly strong at instruction following.
• Gemma 2B / 4B — Competitive performance at small sizes.

For a detailed comparison of current open-weight models for these kinds of workflows, the open-source vs. closed-source model comparison for agentic workflows is worth reading.

For Mid-Complexity Tasks

• Mistral 7B / Mistral Small — Solid all-rounder for summarization and structured extraction.
• Gemma 9B / 27B — Strong performance, especially on code-adjacent tasks.
• Qwen 2.5 7B — Very competitive on instruction following and multilingual tasks.

For comparative analysis of which open-weight models suit different agentic workloads, the Gemma 4 vs Qwen breakdown covers the tradeoffs clearly.

---

Where Remy Fits Into This Picture

If you’re building AI agent workflows — the kind where you’re wiring together embeddings, transcription, classification, and complex reasoning — you’re also managing a lot of infrastructure complexity.

Remy handles that infrastructure layer automatically. It’s a spec-driven development environment where you describe your application in annotated markdown, and the full-stack app — backend, database, auth, deployment — is compiled from that spec. The underlying platform (built on MindStudio’s infrastructure) supports 200+ AI models, including the ability to connect local models alongside frontier APIs.

This means you can build an application that routes embedding calls to a local model, transcription to Whisper, and complex reasoning to Claude — and have all of that reflected in a single spec that stays in sync as the project evolves. You’re not gluing five different APIs together manually in your editor.

For teams that want the hybrid architecture described in this article without the overhead of building and maintaining the routing layer themselves, that’s exactly what Remy addresses. You can try Remy at mindstudio.ai/remy.

---

Common Mistakes to Avoid

Routing complex tasks to local models to save money. This is the main failure mode. If you try to use a 7B local model for nuanced code generation or multi-step reasoning, you’ll get worse results and spend more time debugging. The savings aren’t worth it. Keep complex tasks on Claude.

Ignoring latency. Local model inference adds latency, especially on CPU. For real-time user-facing applications, benchmark your local inference times before committing. Most embedding and classification tasks are fast enough even on CPU. Large model inference on CPU is not.

Not validating output quality. Just because a local model returns an output doesn’t mean it’s correct. Add validation layers — especially for classification tasks. Check that the output is one of your expected categories. If it’s not, fall back to the API.

Forgetting to handle model loading time. Cold-starting Ollama takes a few seconds. In production, keep Ollama running as a persistent service and pre-load your models on startup.

Skipping caching. For embeddings especially, cache results aggressively. If you’re embedding the same documents repeatedly, you’re wasting compute even if it’s free.

---

Frequently Asked Questions

Can I use local models with Claude Code without building a custom proxy?

Not directly out of the box. Claude Code is designed to call Anthropic’s API. But you can override the base URL with ANTHROPIC_BASE_URL to point at a compatible proxy or router. Tools like LiteLLM can serve as a middleware layer that forwards requests to Ollama for supported task types while proxying everything else to Anthropic. Setting up that routing layer is the main engineering work involved.

Which tasks should never be offloaded to local models?

Complex, multi-step code generation should stay on Claude. The same goes for tasks requiring deep reasoning, long-context understanding across large codebases, and anything where errors have significant downstream consequences. The frontier models are genuinely better at these, and the cost difference is worth it.

How much GPU do I need to run local models effectively?

For embedding models and small classifiers (under 4B parameters), you can run entirely on CPU with acceptable latency. For 7B+ models, a modern Apple Silicon Mac handles inference well via Metal GPU acceleration. On Linux, an NVIDIA GPU with 8GB+ VRAM handles 7B models comfortably; 16GB for 13B models. CPU inference on 7B models is possible but slow — around 5–15 tokens/second on a modern machine.

Does this work with Claude Code’s Max subscription, or only the API?

This approach applies specifically to using Claude Code with API keys (pay-as-you-go), not the Max subscription, which is a flat-rate plan. If you’re on the Max subscription, you don’t pay per token — so local offloading won’t save you money on that subscription directly. However, the pattern is still relevant for any code you write that calls AI models from within Claude Code sessions. The comparison between Claude Code Ultra and local plan modes covers how these cost structures differ.

Can I use this same architecture for AI agents beyond Claude Code?

Yes. The hybrid local + frontier model approach works for any AI agent stack. The routing logic, Ollama setup, and local model choices described here apply equally to custom agent frameworks, LangChain agents, or agents built on platforms like MindStudio. For a broader look at connecting local LLMs to AI agent environments, that covers the integration patterns in more detail.

Is there a way to automate which tasks get routed where?

Yes — this is called model routing or intelligent request routing. There are dedicated tools for this, including AI model routers that optimize across multiple LLM providers. Some routers can classify incoming requests by complexity and cost, then automatically select the cheapest model that meets the quality threshold. This is more sophisticated than the manual routing described here, but the underlying principle is the same.

---

Key Takeaways

• Claude Code routes all inference through Anthropic’s API by default, making costs add up quickly for embedding-heavy, transcription-heavy, or classification-heavy workflows.
• Offloading those tasks to local models via Ollama and Whisper can reduce total AI costs by 8–10x without affecting the quality of your core reasoning and code generation.
• The practical setup involves running Ollama locally, pulling small open-source models, and optionally building a lightweight proxy that sits between Claude Code and the Anthropic API.
• Embeddings, transcription, and text classification are the highest-value offloading targets — they’re called frequently, well within local model capability, and cost nothing to run locally.
• Keep complex reasoning, code generation, and multi-step planning on Claude. The local/frontier split is about matching task complexity to model capability, not replacing Claude where it genuinely adds value.

If you’re building the kind of full-stack AI application where these cost considerations matter at scale, Remy handles the infrastructure so you can focus on what the application actually does.