01 — Overview & Problem
Talking Tutor turns a generated study guide into a spoken quiz. You hold a button, answer out loud, and a tutor voice replies with feedback and the next question. Under the hood, the browser captures microphone audio through an AudioWorklet, downsamples it to 16 kHz PCM, and streams it over a WebSocket to a FastAPI relay on Cloud Run. The relay holds a Gemini Live API session, forwards audio in both directions, and turns the model's tool calls into scored answers and a final summary. Study guides come from Vision-First Study Buddy, which shares the same codebase and deployment; this page covers only the voice layer.
Reading a study guide is passive. The thing that makes knowledge stick is retrieval practice: closing the book and saying the answer out loud. Few students have a tutor on call to ask the questions, listen, and correct them. The engineering problem is just as real: bidirectional voice on a public, unauthenticated demo means open-ended audio streaming against a metered model, so every session needs a clock, a quota, and a way to end gracefully without cutting the tutor off mid-sentence.
02 — The Approach
The relay is the whole design. The browser never talks to Gemini directly; it opens one WebSocket to the backend, which opens one Live API session with a system prompt rendered from the study guide. Automatic activity detection is turned off so a press of the talk button maps exactly to activity_start and activity_end, which keeps the model from answering background noise. The model is given two tools: record_answer, which carries the question, the student's answer, a correctness flag, and one sentence of feedback, and end_quiz, which carries a summary. The relay converts those into answer_recorded and quiz_summary frames the client renders as a live scoreboard, and writes the final score into quiz history. Pacing is enforced server-side: the prompt tells the tutor to end its turn after feedback, and the relay waits 2.5 seconds before prompting it to ask the next question, cancelling that prompt if the student starts talking. Cost control is layered: an origin check, a per-device and global daily session cap, a concurrency cap, a 3-minute clock, an inbound audio byte quota, and a 45-second idle timeout. When the clock runs out the relay locks input and gives the tutor up to 30 seconds to finish its turn before ending the session, and the client drains queued audio rather than stopping playback. Everything runs on a single Cloud Run instance so the caps can live in memory.
System Architecture Diagrams
The quiz loop, and the two ways a session ends
What travels between the browser, the relay, and the Live API
03 — Tech Stack Spec
Gemini Live API
gemini-3.1-flash-live-preview, audio in and out
WebSocket Relay
FastAPI on Cloud Run, one socket per session
AudioWorklet Capture
48 kHz mic, downsampled to 16 kHz PCM16
Playback Queue
24 kHz PCM16 scheduled on a Web Audio clock
Tool Calling
record_answer and end_quiz drive scoring
Session Guard
Daily, concurrency, byte, and idle caps in memory
Turn Pacing
Server nudge after feedback, cancelled on speech
Graceful Time-Up
Input locks, tutor finishes, audio drains