← ALL EXPERIMENTS / talking-tutor
Talking Tutor
LANGUAGE / STACK: Python + React LIVE

Talking Tutor

Say your answers out loud. A voice tutor quizzes you on your own study guide and talks back in real time.

OVERVIEW

Talking Tutor turns a generated study guide into a spoken quiz. You hold a button, answer out loud, and a tutor voice replies with feedback and the next question. Under the hood, the browser captures microphone audio through an AudioWorklet, downsamples it to 16 kHz PCM, and streams it over a WebSocket to a FastAPI relay on Cloud Run. The relay holds a Gemini Live API session, forwards audio in both directions, and turns the model's tool calls into scored answers and a final summary. Study guides come from Vision-First Study Buddy, which shares the same codebase and deployment; this page covers only the voice layer.

THE PROBLEM

Reading a study guide is passive. The thing that makes knowledge stick is retrieval practice: closing the book and saying the answer out loud. Few students have a tutor on call to ask the questions, listen, and correct them. The engineering problem is just as real: bidirectional voice on a public, unauthenticated demo means open-ended audio streaming against a metered model, so every session needs a clock, a quota, and a way to end gracefully without cutting the tutor off mid-sentence.

The relay is the whole design. The browser never talks to Gemini directly; it opens one WebSocket to the backend, which opens one Live API session with a system prompt rendered from the study guide. Automatic activity detection is turned off so a press of the talk button maps exactly to activity_start and activity_end, which keeps the model from answering background noise. The model is given two tools: record_answer, which carries the question, the student's answer, a correctness flag, and one sentence of feedback, and end_quiz, which carries a summary. The relay converts those into answer_recorded and quiz_summary frames the client renders as a live scoreboard, and writes the final score into quiz history. Pacing is enforced server-side: the prompt tells the tutor to end its turn after feedback, and the relay waits 2.5 seconds before prompting it to ask the next question, cancelling that prompt if the student starts talking. Cost control is layered: an origin check, a per-device and global daily session cap, a concurrency cap, a 3-minute clock, an inbound audio byte quota, and a 45-second idle timeout. When the clock runs out the relay locks input and gives the tutor up to 30 seconds to finish its turn before ending the session, and the client drains queued audio rather than stopping playback. Everything runs on a single Cloud Run instance so the caps can live in memory.

Session Lifecycle

The quiz loop, and the two ways a session ends

Session Lifecycle CLICK TO ENLARGE
Audio Relay Path

What travels between the browser, the relay, and the Live API

Audio Relay Path CLICK TO ENLARGE
Medium EFFORT
graphic_eq

Gemini Live API

gemini-3.1-flash-live-preview, audio in and out

swap_vert

WebSocket Relay

FastAPI on Cloud Run, one socket per session

mic

AudioWorklet Capture

48 kHz mic, downsampled to 16 kHz PCM16

volume_up

Playback Queue

24 kHz PCM16 scheduled on a Web Audio clock

handyman

Tool Calling

record_answer and end_quiz drive scoring

timer

Session Guard

Daily, concurrency, byte, and idle caps in memory

pace

Turn Pacing

Server nudge after feedback, cancelled on speech

hourglass_bottom

Graceful Time-Up

Input locks, tutor finishes, audio drains

[PUSH-TO-TALK VOICE][BIDIRECTIONAL AUDIO STREAMING][MANUAL ACTIVITY DETECTION][TOOL-CALL SCORING][SERVER-SIDE PACING][GRACEFUL TIME LIMIT][SPEND CAPS IN MEMORY][LIVE TRANSCRIPT]