# marbor

The control plane for self-hosted AI inference - a GPU-aware scheduler and OpenAI-compatible gateway for Ollama, vLLM, TGI, llama.cpp, and MLX. Routes each LLM request to the GPU node that already holds the requested model warm in VRAM, eliminating cold-start latency, and turns your own hardware into a high-availability alternative to cloud LLM APIs. Cost-aware cloud overflow to OpenAI or Anthropic only when the local cluster is saturated. Single static Go binary with zero runtime dependencies.

## What it does

- Warm-first routing: polls each node's runtime state every 2 seconds (e.g. Ollama's /api/ps); routes to the node with the model already loaded in VRAM
- Least-connections tiebreaker across warm nodes
- Cost-aware cloud overflow: if all local nodes are busy or unreachable, routes to configured OpenAI or Anthropic endpoint
- Ollama-native response translation: /api/* requests that fall back to cloud get the OpenAI response translated back to Ollama NDJSON format
- Per-key API authentication with Bearer tokens
- Per-key model allow-lists: 403 if key is not permitted for a model
- Per-key rate limits (token bucket), daily limits, and monthly hard quotas
- Durable quota persistence: counters survive restarts via SQLite database storage
- Cost tracking: parses real eval_count/usage.total_tokens per response; shows local ($0) vs cloud cost
- VRAM telemetry: live used-VRAM per node from /api/ps; total capacity from nvidia-smi (local) or declared vram_total_mb (remote)
- Admin dashboard embedded in binary (React, served at :8080)
- Prometheus metrics at :9090
- Pre-stream failover: if a node dies before first byte, retries alternate nodes then cloud
- X-Request-ID on all responses
- Audit trail persisted in SQLite (append-only)
- SIGHUP hot-reload to re-sync settings from SQLite database without restart
- Docker auto-discovery: scans the Docker socket for Ollama containers and auto-registers them
- Optional marbor-agent companion binary on each GPU node (default port :9200): live remote telemetry, model pulls with HuggingFace token injection, and runtime start/stop/restart/logs via accepted control drivers
- DB-first configuration: every setting lives in marbor.db (SQLite), managed via the admin dashboard or /admin/v1 REST API - no config file
- Zero-config first run: opens/creates marbor.db and prints a banner pointing at the admin dashboard (admin/admin login, forced password change)

## Install

```
curl -fsSL https://raw.githubusercontent.com/Anirudhx7/marbor/main/install.sh | sh
```

Or download a binary from: https://github.com/Anirudhx7/marbor/releases

Docker: `docker run -p 11434:11434 -p 8080:8080 ghcr.io/anirudhx7/marbor:latest`

## Quick start

```
./marbor
# Opens (or creates) marbor.db and starts blank-slate
# Admin dashboard: http://localhost:8080 (admin/admin, forced password change on first login)
# Create a client API key from the API Keys page or POST /admin/v1/keys, then:

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=<a key created in the dashboard>
```

## Key facts

- Language: Go (single static binary, no CGO, no Python)
- License: Apache-2.0 (open-source core); enterprise features under a separate commercial license
- Ports: :11434 (endpoint), :8080 (admin), :9090 (metrics)
- Compatible with: any OpenAI SDK, Ollama client, LangChain, llama-index
- Platforms: Linux and macOS (amd64, arm64), Windows (amd64); Docker image (linux/amd64)
- No telemetry, no license server, no phone-home

## GitHub

https://github.com/Anirudhx7/marbor

## Keywords

self-hosted LLM inference, self-hosted LLM gateway, on-prem LLM serving, private LLM inference, self-hosted OpenAI API alternative, GPU-aware LLM routing, warm model VRAM routing, LLM inference control plane, multi-runtime inference, Ollama load balancer, vLLM load balancing, llama.cpp, TGI, MLX, cost-aware cloud overflow