A self-hosted control plane with a marbor agent for remote telemetry, model operations, and node-side maintenance. marbor routes each request to the node already holding the model warm in VRAM - eliminating cold starts, balancing queue depth across nodes, and metering cost-aware cloud overflow with full token accounting. The core stays a single Go binary; the marbor agent extends it on remote GPU hosts with richer live data, lifecycle control, and fleet maintenance.
Measured, not simulated: on a deployed v0.13.1 instance routing to a single consumer-GPU node (8B Q4_K_M model, ~9.6 GB), cold TTFT was 11.5–18.1 s (median 17.3 s) vs a warm median of 8.1 s and a fastest warm response of 0.4 s - up to 43× faster than the median cold start. Honest context: only ~3.3 GB fit in VRAM so even warm was partly CPU-bound - full-VRAM hardware widens the gap. Reproduce it on your hardware with the open benchmark harness.
The Scheduling Problem
Serving LLMs across your own GPU fleet creates silent bottlenecks that generic load balancers can't see - wasting VRAM, wearing hardware, and breaking reliability.
Without scheduling coordination, developers load models ad-hoc. When GPU memory fills, the runtime terminates resident models to make room, leading to constant loading cycles that wear hardware and halt requests.
Standard load balancers balance network connections, not request complexity. A deep reasoning request is treated the same as a short completion, causing requests to pile up on a busy node while other GPUs sit completely idle.
When local hardware capacity is genuinely exceeded, users get connection errors. Alternatively, routing all requests to cloud APIs by default removes the cost benefits of self-hosting local models.
By actively polling node states and VRAM layouts every 2 seconds, marbor acts as a centralized control plane. It schedules requests based on real-time model residency, coordinates queues across nodes, and overflows to cloud providers only when local capacity is fully saturated - preserving local compute first.
Built for platform engineers
Point every app at one OpenAI-compatible URL. marbor handles warm-state routing, high-availability failover, authentication, rate limiting, and cost analytics - across Ollama, vLLM, TGI, llama.cpp, and MLX.
Schedules requests exclusively to GPU nodes where the target model is already resident in VRAM. Prevents the multi-second delay of thrashing models in and out of GPU memory, maximizing hardware lifetime and throughput.
Spills traffic to backup cloud APIs (OpenAI or Anthropic) only when local GPUs are fully saturated. The scheduler automatically pulls traffic back to the local cluster the instant queue depth or VRAM capacity is freed, ensuring you only pay for cloud compute when your local hardware is genuinely full.
Tracks, prices, and logs every cloud-overflow request per token. Provides a clear accounting of the dollar amount saved by serving requests locally versus what was spent on cloud fallback, directly in your terminal or dashboard.
Turns each remote GPU host into a live, operator-visible node: real-time GPU utilization, active queue depth, VRAM occupancy, model residency, and node-side operations are all available through the same product surface. No separate exporters, databases, or Grafana stack to assemble just to observe, control, and maintain your cluster.
A static, self-contained Go binary with zero runtime dependencies. Cold-starts in under 10ms, has a negligible memory footprint, and requires no Python environment, node modules, or Docker containers to run.
Enforces API key authentication and token quotas. Supports the standard /v1/chat/completions endpoint for tools like Continue, LibreChat, Open WebUI, and custom applications by changing a single base URL.
Provides a unified OpenAI-compatible endpoint that transparently orchestrates Ollama, vLLM, llama.cpp, TGI, and MLX nodes. Handles protocol translation automatically, allowing you to mix and match runtimes based on performance needs.
From zero to routed in three steps
The core control plane is a single Go binary. When you want richer remote telemetry, model lifecycle operations, or node-side maintenance, install the marbor agent on the GPU hosts that need it. marbor still communicates with every runtime over standard HTTP APIs.
Grab the release for your platform - or build from source with Go. It's one file with no runtime dependencies.
Option 1: Download & Install only (Recommended)
curl -fsSL raw.githubusercontent.com/Anirudhx7/marbor/main/install.sh | sh
Option 2: Install, scan local network & run daemon in background
curl -fsSL raw.githubusercontent.com/Anirudhx7/marbor/main/install.sh | START=1 sh
Add nodes from the dashboard, or seed them at install time. No config file to write. Ollama, vLLM, TGI, llama.cpp, and MLX nodes all work - set runtime per node.
# seed at install time (repeatable flag) marbor --seed-node "name=ollama-gpu,url=http://gpu-01:11434,runtime=ollama" --seed-node "name=vllm-gpu,url=http://10.0.1.20:8000,runtime=vllm"
Repoint your apps to the marbor. Same OpenAI-compatible endpoint - the routing happens behind it.
# your app, unchanged otherwise base_url = "http://marbor:11434/v1" # warm node picked automatically
Where it fits
Nginx, HAProxy, and Traefik balance TCP connections - they can't see which GPU holds your model warm. marbor schedules on live GPU and VRAM state instead. Here's the honest breakdown.
| Capability | marbor | Plain load balancer (nginx / HAProxy) |
|---|---|---|
| Warm-state routing (knows which nodes have which models loaded) | ✓ Yes | ✗ No - nginx has no VRAM visibility |
| Model residency awareness (avoids unnecessary reloads) | ✓ Yes | ✗ No - routes blindly, causes thrashing |
| GPU locality scheduling (routes by where the model already lives) | ✓ Yes | ✗ No - cannot query GPU layouts |
| Queue-depth awareness (avoids piling requests on a busy node) | ✓ Yes | ~ Partial - balances TCP, not LLM queues |
| VRAM visibility (knows what is consuming memory across the cluster) | ✓ Yes | ✗ No - cannot read /api/ps or nvidia-smi |
| Cold start handling (detects and routes around cold nodes) | ✓ Yes | ✗ No - exposes 20–30s cold load to user |
| Multi-runtime support (Ollama, vLLM, TGI, llama.cpp, MLX) | ✓ Yes | ✗ No - needs custom per-runtime config |
| Cluster observability (unified view of all nodes) | ✓ Yes | ✗ No - requires Prometheus + Grafana setup |
LiteLLM is excellent for provider abstraction, enterprise auth, and user-level rate limiting. marbor slots in directly below LiteLLM as your physical scheduling layer, handling raw GPU queue depths, model residency, and warm-state routing. Together, they form the standard enterprise stack for high-throughput, self-hosted LLM clusters.
Every request marbor keeps on your own GPUs is a request you don't pay OpenAI or Anthropic for. It meters each cloud-overflow token against your configured reference rate, so you see the exact dollar difference between local and cloud on every response - see the savings math.
Frequently Asked Questions
marbor is a single-binary control plane for self-hosted AI inference. It routes each request to the GPU node that already holds the model warm in VRAM across Ollama, vLLM, TGI, llama.cpp, and MLX, exposing one OpenAI-compatible endpoint so your own hardware becomes a high-availability alternative to cloud LLM APIs.
Nginx, HAProxy, and Traefik balance TCP connections with no visibility into which GPU holds a model warm, so they route blindly and cause repeated cold starts and VRAM thrashing. marbor reads live model residency and VRAM state on every node and schedules each request to a warm GPU.
All five. Despite the name, marbor is runtime-agnostic: it presents one unified OpenAI-compatible endpoint and translates protocols across Ollama, vLLM, TGI, llama.cpp, and MLX, so you can mix runtimes behind a single URL.
That is the core use case. It serves requests from your own GPUs first and overflows to OpenAI or Anthropic only when local capacity is fully saturated, metering every cloud token so you can see exactly what self-hosting saves versus cloud spend.
If a node goes offline, marbor marks it unhealthy and reroutes to other warm local nodes; if the whole local cluster is saturated or down, it can optionally fail over to a configured cloud provider. The routing path is stateless, so you can run multiple instances behind an L4 load balancer for HA.
Yes. marbor acts as an API gateway: Bearer-token authentication, per-key model allow-lists, and per-key rate limits with daily and monthly quotas - protecting self-hosted GPUs from abuse without changing your application code.
Install it, point it at your nodes, and open the dashboard. The numbers are yours - measured, not mocked up.