Cost Deflection Analysis: Local GPU Inference vs. Cloud API Spend#

This document is the financial model behind ollama-mesh's savings tracking. It is designed for infrastructure directors and finance teams evaluating the ROI of shifting LLM inference from third-party cloud APIs to owned GPU hardware routed through ollama-mesh.

Every figure in the ollama-mesh dashboard is derived from the formulas below. When token data is unavailable, the dashboard displays "-" rather than an estimate. No number is ever fabricated.

Executive Summary#

An engineering organization running multi-agent LLM workflows at scale - coding copilots, RAG pipelines, automated code review, internal search - consumes millions of tokens per day. At cloud API rates, this translates to $5,000–$50,000/month in direct API spend, depending on model tier and volume.

ollama-mesh enables organizations to route this traffic to owned GPU hardware first, falling back to cloud APIs only when local capacity is exhausted. The savings dashboard tracks every token served locally and values it against the cloud rate the organization would otherwise pay.

Typical result: Platform teams running 2–4 GPU nodes with ollama-mesh report 60–85% reduction in cloud API spend within the first billing cycle, with the reduction visible in the dashboard from day one.

The Formulas#

Local Request Savings#

When a request is served by a local Ollama node, ollama-mesh parses the real token count from the upstream response:

• Ollama NDJSON responses: eval_count (completion tokens) + prompt_eval_count (input tokens) from the final streamed object
• OpenAI-compatible SSE responses: usage.total_tokens from the terminal [DONE]-adjacent chunk

Savings for each locally-served request:

saved_usd = (prompt_tokens + completion_tokens) / 1000 × reference_cost_per_1k

The reference_cost_per_1k is operator-configured and represents the blended cloud rate the organization would otherwise pay. Default: $0.002/1K tokens (deliberately conservative - adjust to match your actual cloud provider rate).

Cloud Overflow Spend#

When a request overflows to a configured cloud provider (OpenAI, Anthropic), spend is computed from the real token count parsed from the provider's response:

spent_usd = parsed_tokens / 1000 × cost_per_1k_tokens

cost_per_1k_tokens is configured per cloud provider in config.yaml. Local requests are always $0 spend.

Net Savings#

net_savings_usd = Σ saved_usd (all local requests) - Σ spent_usd (all cloud requests)

This is the number displayed on the dashboard savings widget. It represents the actual dollar amount the organization avoided paying to cloud providers by serving traffic locally.

Enterprise Workload Projections#

The following projections use conservative assumptions grounded in real-world multi-agent workflow patterns. All figures assume the default reference rate of $0.002/1K tokens. Adjust proportionally for your actual cloud rate.

Scenario A: Engineering Copilot (50-Person Team)#

Scenario B: Multi-Agent RAG Pipeline (Production)#

Scenario C: Internal Platform (200-Person Organization)#

Note: These projections calculate the difference between cloud API costs and amortized local hardware costs. The hardware amortization includes purchase price, power, cooling, and rack space over a 3-year depreciation schedule. Actual savings depend on GPU utilization rates, model sizes, and local-vs-cloud traffic split. ollama-mesh's dashboard shows the real split based on actual parsed token counts, not these projections.

Hardware ROI Calculator#

Break-Even Analysis#

The break-even point is when cumulative cloud savings equal the upfront GPU hardware investment:

break_even_months = hardware_cost / (monthly_cloud_savings - monthly_hardware_opex)

Monthly hardware opex = amortized purchase price (3yr) + power + rack/cooling. At $0.015/1K (Claude Sonnet 4 output-heavy workloads):

At GPT-4o blended rates ($0.005/1K), the monthly cloud equivalent is lower but break-even remains attractive: 2× RTX 4090 breaks even in ~4 months at GPT-4o rates.

After break-even, every locally-served token is pure cost deflection. The hardware continues serving traffic for 3-5 years.

Hardware pricing notes: RTX 4090 ~$1,800/card at current retail. A100 80GB ~$10,000-15,000/card (data center pricing, new); secondary market lower. Monthly opex above uses 3-year straight-line amortization plus ~$0.12/kWh power at typical GPU utilization. Rack/cooling adds 10-15% in enterprise colocations.

How the Dashboard Computes These Numbers#

Data Sources#

1. Local token counts - parsed from the real upstream Ollama response, not estimated. ollama-mesh reads eval_count and prompt_eval_count from the final NDJSON object (Ollama native) or usage.total_tokens from the terminal SSE chunk (OpenAI-compatible).

1. Cloud token counts - parsed from the real cloud provider response. OpenAI and Anthropic both include usage objects in their streaming responses.

1. Reference rate - operator-configured in config.yaml under savings.reference_cost_per_1k. Single flat rate applied to all locally-served tokens.

1. Cloud rates - per-provider cost_per_1k_tokens in the cloud_providers config block.

What "-" Means#

If requests were served but no token counts could be parsed from any response - for example, the upstream never sent a final usage object, or the stream was aborted before the terminal chunk - the API returns null and the dashboard renders "-". ollama-mesh never substitutes an estimated or random number for missing token data.

Counter Lifecycle#

Savings and spend counters are held in memory. Per-key token totals and quota counters are persisted to usage-state.json (configurable via auth.state_path) and survive restarts. Aggregate savings counters reset on process restart. The audit log (JSON-lines, if enabled) is the durable per-request record.

Configuration#

Setting the Reference Rate#

savings:
  reference_cost_per_1k: 0.015   # match your actual cloud model rate

Set this to the blended rate of the cloud model your traffic would otherwise consume:

When the field is missing or zero, the default $0.002/1K applies. Changing the rate only affects requests recorded after the change; it does not retroactively revalue earlier requests.

Per-Provider Cloud Rates#

cloud_providers:
  - name: openai-overflow
    provider: openai
    cost_per_1k_tokens: 0.0025    # GPT-4o blended rate
    enabled: true
  - name: anthropic-overflow
    provider: anthropic
    cost_per_1k_tokens: 0.004     # Claude Sonnet 4 blended rate
    enabled: true

Known Limitations#

Audit Trail for Financial Verification#

Every routed request is recorded in the JSON-lines audit log (when enabled) with:

• Request ID (crypto/rand generated)
• Timestamp
• API key name (never the key value)
• Model requested
• Target node (or cloud provider)
• Token counts (prompt + completion)
• Calculated cost (saved or spent)
• HTTP status and latency

This log is the source of truth for financial reconciliation. It can be ingested into any log aggregator (Splunk, Elastic, Datadog) for custom reporting, cost allocation, and audit compliance.