Groat — Every token counts

A self-hosted LLM proxy that saves you money.

Vectors in, savings out, semantically cached, intelligently routed, never a token wasted.

$ cargo install groat

Read the Docs Github

groat — zsh

$ cargo install groat

Compiling groat v0.3.2

Finished release [optimized] target(s) in 4.12s

$ groat serve --port 8080

listening on 0.0.0.0:8080

routing → openai · anthropic · local

cache backend: redis (connected)

$ curl localhost:8080/v1/chat/completions

200 OK · 128 tokens · cache hit

cache_hit_rate: 68.4% | requests_today: 4,213

$ _

How it works

Point your app at Groat

Swap the base URL in your OpenAI SDK. Keep your prompts, your code, your framework exactly as they are.

Groat decides

Every request is checked against the semantic cache and scored for routing — before it ever leaves your machine.

Watch what you saved

The dashboard shows every cache hit, every downgraded request, and exactly how much smaller your bill got.

Built to actually reduce your bill

Not another dashboard that tells you what you already spent.

Drop-in compatibility

One line change. Point your existing OpenAI SDK at Groat instead of the provider — everything else in your codebase stays untouched.

Semantic cache

Not exact-match. Groat understands when two requests mean the same thing, even if the wording differs, and serves the cached response.

Local embeddings

Cache matching runs on-device with candle. No request data leaves your machine to compute a similarity score.

Prompt cache injection

Groat automatically structures requests to qualify for provider-side prompt caching discounts — no manual prompt engineering required.

Cost dashboard

See exactly what you're spending, what Groat saved, and which requests could have used a cheaper model.

Single binary

cargo install groat && groat up. No Docker, no dependencies, no config file required to get started.

Under the hood

semantic cache

Vector similarity, not string matching

// requests embedded locally via candle — nothing leaves the machine
// matched against prior requests in LanceDB by cosine similarity
// eviction policy: LRU — coldest entries drop first, hot paths stay cached
// similarity threshold is tunable per route

intent router

Three signals, one routing decision

// heuristic scoring on prompt length and structural complexity
// a lightweight local classifier estimates task difficulty
// your explicit routing rules always take priority when they match
// downgrades only fire once confidence clears your threshold

cache injection

Built for provider-side caching

// static content (system prompts, few-shot examples) reordered to a stable prefix
// that's what OpenAI and Anthropic prompt caching actually looks for
// done automatically — no manual prompt restructuring on your end

built withRustAxumTokioSQLitesqlxLanceDBcandle

Get started in two commands

No Docker, no config file, no signup. Install the binary, start it, and point your existing SDK at it.

install & run

$ cargo install groat

$ groat up

change one line

$ client = OpenAI(base_url="http://localhost:8787")

That's it, everything else in your code stays exactly the same.