Thrifty Router

Thrifty Router is an open-source, cost-optimizing LLM proxy for Google Cloud Vertex AI (Gemini 2.5 and 1.5). Running on Google Cloud Run with FastAPI, it intercepts prompt requests and dynamically determines the most cost-effective model tier — lite (Gemini 2.5 Flash at $0.075/$0.30 per 1M), standard (Gemini 1.5 Flash), or pro (Gemini 2.5 Pro at $1.25/$5.00 per 1M) — capable of fulfilling the request with high fidelity. The gateway implements four distinct routing strategies: static fixed mapping, semantic anchor classification using 768-dim embeddings via gemini-embedding-001, zero-shot LLM classification with automated fallback, and opportunistic cascade routing that attempts generation on cheaper tiers before validating and selectively escalating. Integrated with a two-tier semantic caching engine (SHA-256 exact match + Firestore vector search with cosine similarity >= 0.92) and a thread-safe micro-dollar cost ledger with a daily budget circuit breaker, Thrifty Router achieves up to 73% cost reductions on benchmark workloads while maintaining a 0.954 quality score evaluated by Gemini 2.5 Pro LLM-as-a-judge.

Defaulting all LLM workloads to flagship reasoning models (like Gemini 2.5 Pro or GPT-4o) inflates inference bills by 10x to 20x for simple tasks like factual extraction, sentiment analysis, text formatting, and conversational greetings. Conversely, locking an application into a smaller, cheaper model causes hallucinations or silent failures when users present complex reasoning, code synthesis, or nuanced logic. Furthermore, unmetered public API endpoints risk unbounded cloud charges without real-time spend caps, and repetitive queries incur full model inference latency and expense without semantic deduplication.