Thrifty Router

Thrifty Router is an open-source, cost-optimizing LLM proxy for Google Cloud Vertex AI (Gemini 3.1 and 3). Running on Google Cloud Run with FastAPI, it intercepts prompt requests and dynamically determines the most cost-effective model tier — lite (Gemini 3.1 Flash-Lite at $0.25/$1.50 per 1M tokens), standard (Gemini 3 Flash at $0.50/$3.00), or pro (Gemini 3.1 Pro at $2.00/$12.00) — capable of fulfilling the request with high fidelity. The gateway implements four distinct routing strategies: static fixed mapping, semantic anchor classification using 768-dim embeddings via gemini-embedding-001, zero-shot LLM classification with automated fallback, and opportunistic cascade routing that attempts generation on cheaper tiers before validating and selectively escalating. Integrated with a two-tier semantic caching engine (SHA-256 exact match + Firestore vector search with cosine similarity >= 0.95) and a thread-safe micro-dollar cost ledger with a daily budget circuit breaker, On the 300-prompt golden set, semantic routing matched the all-Pro baseline's 4.90 judge score at 39% of its cost, and Flash-Lite alone kept 98% of it at 3%, as scored by a Gemini 3.1 Pro LLM-as-a-judge.

Defaulting all LLM workloads to flagship reasoning models (like Gemini 3.1 Pro) inflates inference bills by 10x to 20x for simple tasks like factual extraction, sentiment analysis, text formatting, and conversational greetings. Conversely, locking an application into a smaller, cheaper model causes hallucinations or silent failures when users present complex reasoning, code synthesis, or nuanced logic. Furthermore, unmetered public API endpoints risk unbounded cloud charges without real-time spend caps, and repetitive queries incur full model inference latency and expense without semantic deduplication.