A single-binary control plane that turns your own GPUs into a high-availability alternative to cloud LLM APIs. ollama-mesh routes each request to the node already holding the model warm in VRAM - eliminating cold starts, balancing queue depth across nodes, and metering cost-aware cloud overflow with full token accounting. One OpenAI-compatible endpoint for Ollama, vLLM, TGI, and llama.cpp.
Measured, not simulated: on a deployed v0.14.0 instance routing to a single consumer-GPU node (8B Q4_K_M model, ~9.6 GB), cold TTFT was 11.5–18.1 s (median 17.3 s) vs a warm median of 8.1 s and a fastest warm response of 0.4 s - up to 43× faster than the median cold start. Reproduce it on your hardware with the open benchmark harness.
The Scheduling Problem
Serving LLMs across your own GPU fleet creates silent bottlenecks that generic load balancers can't see - wasting VRAM, wearing hardware, and breaking reliability.
Without scheduling coordination, developers load models ad-hoc. When GPU memory fills, the runtime terminates resident models to make room, leading to constant loading cycles that wear hardware and halt requests.
Standard load balancers balance network connections, not request complexity. A deep reasoning request is treated the same as a short completion, causing requests to pile up on a busy node while other GPUs sit completely idle.
When local hardware capacity is genuinely exceeded, users get connection errors. Alternatively, routing all requests to cloud APIs by default removes the cost benefits of self-hosting local models.
By actively polling node states and VRAM layouts every 2 seconds, ollama-mesh acts as a centralized control plane. It schedules requests based on real-time model residency, coordinates queues across nodes, and overflows to cloud providers only when local capacity is fully saturated - preserving local compute first.
Built for platform engineers
Point every app at one OpenAI-compatible URL. ollama-mesh handles warm-state routing, high-availability failover, authentication, rate limiting, and cost analytics - across Ollama, vLLM, TGI, and llama.cpp.
Schedules requests exclusively to GPU nodes where the target model is already resident in VRAM. Prevents the multi-second delay of thrashing models in and out of GPU memory, maximizing hardware lifetime and throughput.
Spills traffic to backup cloud APIs (OpenAI or Anthropic) only when local GPUs are fully saturated. The scheduler automatically pulls traffic back to the local cluster the instant queue depth or VRAM capacity is freed, ensuring you only pay for cloud compute when your local hardware is genuinely full.
Tracks, prices, and logs every cloud-overflow request per token. Provides a clear accounting of the dollar amount saved by serving requests locally versus what was spent on cloud fallback, directly in your terminal or dashboard.
Exposes real-time GPU utilization, active queue depths, VRAM occupancy, and model residency across all nodes. Served directly from the single binary with zero external telemetry agents, databases, or Grafana dashboards to configure.
A static, self-contained Go binary with zero runtime dependencies. Cold-starts in under 10ms, has a negligible memory footprint, and requires no Python environment, node modules, or Docker containers to run.
Enforces API key authentication and token quotas. Supports the standard /v1/chat/completions endpoint for tools like Continue, LibreChat, Open WebUI, and custom applications by changing a single base URL.
Provides a unified OpenAI-compatible endpoint that transparently orchestrates Ollama, vLLM, llama.cpp, and TGI nodes. Handles protocol translation automatically, allowing you to mix and match runtimes based on performance needs.
From zero to routed in three steps
No complex cluster agents or sidecars on your GPU nodes. ollama-mesh acts as a centralized control plane, communicating directly with your runtimes over standard HTTP APIs.
Grab the release for your platform - or build from source with Go. It's one file with no runtime dependencies.
Option 1: Download & Install only (Recommended)
curl -fsSL https://raw.githubusercontent.com/Anirudhx7/ollama-mesh/main/install.sh | sh
Option 2: Install, scan local network & run daemon in background
curl -fsSL https://raw.githubusercontent.com/Anirudhx7/ollama-mesh/main/install.sh | PROBE=1 START=1 sh
List your nodes and, optionally, a cloud key for fallback. Ollama, vLLM, TGI, and llama.cpp nodes all work - set runtime: per node.
# mesh.yaml nodes: - name: ollama-gpu url: http://gpu-01:11434 runtime: ollama - name: vllm-gpu url: http://10.0.1.20:8000 runtime: vllm fallback: openai # optional cloud overflow
Repoint your apps to the mesh. Same OpenAI-compatible endpoint - the routing happens behind it.
# your app, unchanged otherwise base_url = "http://mesh:8080/v1" # warm node picked automatically
Where it fits
Nginx, HAProxy, and Traefik balance TCP connections - they can't see which GPU holds your model warm. ollama-mesh schedules on live GPU and VRAM state instead. Here's the honest breakdown.
| Capability | ollama-mesh | Plain load balancer (nginx / HAProxy) |
|---|---|---|
| Warm-state routing (knows which nodes have which models loaded) | Yes | No (nginx has no visibility into VRAM contents) |
| Model residency awareness (avoids unnecessary reloads) | Yes | No (routes blindly, causing constant model thrashing) |
| GPU locality scheduling (routes by where the model already lives) | Yes | No (cannot query GPU layouts or topology) |
| Queue-depth awareness (avoids piling requests on a busy node) | Yes | Partial (nginx balances TCP connections, but does not know Ollama's internal queues) |
| VRAM visibility (knows what is consuming memory across the cluster) | Yes | No (cannot access /api/ps or nvidia-smi metrics) |
| Cold start handling (detects and routes around cold nodes) | Yes | No (routes to cold nodes, exposing 20-30s load times) |
| Multi-runtime support (Ollama, vLLM, TGI, llama.cpp) | Yes | No (requires complex custom routing and API translation layers) |
| Cluster observability (unified view of all nodes) | Yes | No (requires setting up external Prometheus exporters and Grafana) |
LiteLLM is excellent for provider abstraction, enterprise auth, and user-level rate limiting. ollama-mesh slots in directly below LiteLLM as your physical scheduling layer, handling raw GPU queue depths, model residency, and warm-state routing. Together, they form the standard enterprise stack for high-throughput, self-hosted LLM clusters.
Every request ollama-mesh keeps on your own GPUs is a request you don't pay OpenAI or Anthropic for. It meters each cloud-overflow token against your configured reference rate, so you see the exact dollar difference between local and cloud on every response - see the savings math.
Frequently Asked Questions
ollama-mesh is a single-binary control plane for self-hosted AI inference. It routes each request to the GPU node that already holds the model warm in VRAM across Ollama, vLLM, TGI, and llama.cpp, exposing one OpenAI-compatible endpoint so your own hardware becomes a high-availability alternative to cloud LLM APIs.
Nginx, HAProxy, and Traefik balance TCP connections with no visibility into which GPU holds a model warm, so they route blindly and cause repeated cold starts and VRAM thrashing. ollama-mesh reads live model residency and VRAM state on every node and schedules each request to a warm GPU.
All four. Despite the name, ollama-mesh is runtime-agnostic: it presents one unified OpenAI-compatible endpoint and translates protocols across Ollama, vLLM, TGI, and llama.cpp, so you can mix runtimes behind a single URL.
That is the core use case. It serves requests from your own GPUs first and overflows to OpenAI or Anthropic only when local capacity is fully saturated, metering every cloud token so you can see exactly what self-hosting saves versus cloud spend.
If a node goes offline, ollama-mesh marks it unhealthy and reroutes to other warm local nodes; if the whole local cluster is saturated or down, it can optionally fail over to a configured cloud provider. The routing path is stateless, so you can run multiple instances behind an L4 load balancer for HA.
Yes. ollama-mesh acts as an API gateway: Bearer-token authentication, per-key model allow-lists, and per-key rate limits with daily and monthly quotas - protecting self-hosted GPUs from abuse without changing your application code.
Install it, point it at your nodes, and open the dashboard. The numbers are yours - measured, not mocked up.