SDKs · MCP

mcp-local-models

MCP server that exposes the ArcGrid local model fleet as Claude Code tools. Use ask_coder, ask_analyst, and ask_designer to route drafting work to on-prem inference instead of billing API tokens. Local models draft; Sonnet finalizes.

MCP Local Inference ArcRouter Python

Overview

The mcp-local-models server lives at mcp-local-models/server.py in the Grid repo. It runs as a Python process on ArcHelm, exposes five MCP tools over HTTP at port 8020, and proxies each tool call through ArcRouter's model aliases to the correct on-prem inference node.

The server uses streamable-http transport so it works as a standard MCP remote server.

Transport: streamable-http Port: 8020 Auth: Bearer token (MCP_AUTH_TOKEN env var — empty means no auth) Upstream: ArcRouter at http://172.17.0.1:4001 (LITELLM_BASE env var)

Routing rule

The routing decision tree for any drafting task in this codebase:

Task typeUse this toolModel aliasServed by
Mechanical code / JSON / configask_codercodingqwen2.5-coder:32b (ArcRig3)
Analysis, gap analysis, synthesisask_analystanalysisqwen3-fast (ArcRig3)
UI / design / long markdown draftsask_designerdesignqwen3-fast (ArcRig3)
Quick classification / short answersask_chatcommunicationsllama3.1:8b (ArcRig1/2)

Skip local routing when: the task needs knowledge of the current branch, session state, or tool results from the active conversation; the output needs project-specific entity names, voice, or context that can't fit in the prompt; or expected output is heavy prose (150+ lines). Route those directly in the main Sonnet session.

For Agent tool calls, default to model: "haiku" (API-billed traffic). Reserve Sonnet for the main Max plan session only.

Tools

ask_coder coding → qwen2.5-coder:32b
Generate TypeScript, Python, JSON, SQL, config files, and other mechanical code tasks. Best for code that follows clear patterns with no need for project-specific judgment.
max_tokens: 1024 (default) timeout: 60s ArcRig3
ask_analyst analysis → qwen3-fast
Structured analysis, gap analysis, comparison tables, and synthesis tasks. Use when you need the model to reason over a set of inputs and return a structured conclusion.
max_tokens: 2048 (default) timeout: 180s ArcRig3
ask_designer design → qwen3-fast
UI design descriptions, component layouts, long markdown drafts, and anything requiring extended prose output. Highest token budget of the five tools.
max_tokens: 4096 (default) timeout: 600s ArcRig3
ask_chat communications → llama3.1:8b
Short classification, formatting, quick answers, and yes/no decisions. Fastest and cheapest — served from ArcRig1 and ArcRig2. Not suitable for code or reasoning.
max_tokens: 512 (default) timeout: 30s ArcRig1 / ArcRig2
list_models
Returns the current model catalog from ArcRouter. Use this to check what aliases are active and which inference nodes are backing them. No arguments needed.
No arguments Live from ArcRouter

All four ask_* tools accept a prompt string and an optional max_tokens integer override. The response is the raw model completion as a string.

// Tool input schema (all ask_* tools) { "prompt": "...", "max_tokens": 1024 // optional, defaults vary per tool }

Configuration

The server reads two environment variables:

VariableDefaultPurpose
LITELLM_BASEhttp://172.17.0.1:4001ArcRouter base URL (Docker host gateway default)
MCP_AUTH_TOKEN(empty)Bearer token for inbound requests — leave empty to skip auth

The 172.17.0.1 default resolves to the Docker host gateway, which works when the MCP server runs inside a container that needs to reach ArcRouter on the host. For bare-metal installs, point this at ArcHelm's IP and port 4001.

Claude Code setup

Add to .mcp.json in the repo root or to the global Claude Code MCP config:

{ "mcpServers": { "local-models": { "type": "streamable-http", "url": "http://10.20.10.29:8020/mcp", "headers": { "Authorization": "Bearer <MCP_AUTH_TOKEN>" } } } }

After adding the server, the five tools become available in Claude Code as mcp__local-models__ask_coder, mcp__local-models__ask_analyst, etc.

The server must be running on ArcHelm. To start it:

cd ~/projects/the-grid python3 mcp-local-models/server.py

Timeouts and fallback

When a local model call times out or returns an error, note the failure and fall back to the main Sonnet session rather than retrying. Do not silently swallow the error. The recommended pattern:

// Pattern for using local model tools // 1. Call ask_coder / ask_analyst / ask_designer // 2. On success: use the draft output, then refine in the main session // 3. On timeout or error: note "ask_coder timed out, falling back to Sonnet" // and complete the task directly // Do not: // - Retry the same local call in a loop // - Route to Sonnet silently without noting the fallback // - Skip the local call when the task clearly fits a local model

Timeouts are expected during ArcRig maintenance windows. A cold model load on ArcRig3 adds 15-30s to the first request after idle.