Show Dev: Your Face Is Now a Music Generator

Description and use

A real‑time music generator that reacts to user emotions, using client‑side face detection and server‑side audio synthesis.

Problem

Providing a continuous, real‑time music experience that reflects a user’s current emotional state without sending webcam data to a server.

Outcome

A real‑time, privacy‑preserving music generator that reacts to facial emotions, avoiding serverless cold starts and maintaining stateful audio transitions.

Workflow

• 1. Capture webcam frames in the browser.
• 2. Run face‑landmark detection client‑side to extract emotion probabilities.
• 3. Debounce updates (250 ms) and POST emotion vector to /api/mood.
• 4. Server receives vector, maps it to musical parameters (tempo, key, layers).
• 5. Server streams parameter updates to the client.
• 6. Client updates Web Audio API mixer with new parameters.
• 7. Music plays continuously, adapting to real‑time emotional changes.

Evidence

Face detection runs entirely client‑side. Webcam frames → face landmarks → a small vector of emotion probabilities. None of that leaves your machine.

Source

The music generation, though, lives in its own Railway service. That was deliberate.

Source