Show Dev: Your Face Is Now a Music Generator

Описание и применение

Создано приложение, которое в реальном времени генерирует музыку, реагируя на эмоции пользователя, используя клиентскую обработку лица и серверную генерацию звука.

Задача

Providing a continuous, real‑time music experience that reflects a user’s current emotional state without sending webcam data to a server.

Результат

A real‑time, privacy‑preserving music generator that reacts to facial emotions, avoiding serverless cold starts and maintaining stateful audio transitions.

Рабочий процесс

• 1. Capture webcam frames in the browser.
• 2. Run face‑landmark detection client‑side to extract emotion probabilities.
• 3. Debounce updates (250 ms) and POST emotion vector to /api/mood.
• 4. Server receives vector, maps it to musical parameters (tempo, key, layers).
• 5. Server streams parameter updates to the client.
• 6. Client updates Web Audio API mixer with new parameters.
• 7. Music plays continuously, adapting to real‑time emotional changes.

Доказательства

Face detection runs entirely client‑side. Webcam frames → face landmarks → a small vector of emotion probabilities. None of that leaves your machine.

Источник

The music generation, though, lives in its own Railway service. That was deliberate.

Источник