tonesearch: find the angry minute in your call recordings, on your own machine

Description and use

tonesearch enables small call teams to locally search for content and tone in recordings without uploading audio, leveraging open‑weight models EmbeddingGemma 2 and emotion2vec+ for privacy‑preserving analysis.

Problem

Small call teams and agencies need to locate angry moments in call recordings without sending audio to external cloud services.

Outcome

tonesearch enables local, privacy‑preserving search for content and tone in call recordings, achieving high correlation with human irritation ratings (Spearman +0.66 when combining Gemma audio embeddings with emotion2vec). It runs at 18× real time on an M3 Mac with GPU and 3.4× on CPU.

Workflow

• Index call recordings by splitting on silence into 2‑10s segments and running EmbeddingGemma 2 and emotion2vec+ to produce embeddings and tone scores.
• Store the embeddings and scores in a .npz file for fast retrieval.
• Run a query by embedding the text with Gemma, optionally specifying a tone weight, and compute a combined score via cosine similarity and tone log‑odds.
• Return the top N segments; clicking a row plays the audio segment; a separate tab shows the tone over the entire call with top spikes.
• All processing occurs locally; no audio leaves the machine.

Evidence

It needs no transcript and runs on a laptop, so no audio leaves the machine.

Source

The demo data is HaessigDB, a public set of acted call-center speech (CC-BY-4.0, by Niklas Weller, Marc Christopher Grau and Ivo Blohm). These are actors, not real customers.

Source

On an M3 Mac, with Gemma on the GPU, indexing runs at 18x real time. On CPU only, it runs at 3.4x real time.

Source

The audio embedding scored the same as the transcript embedding. It hears the words about as well as a transcript does, and it adds almost nothing about tone.

Source