I Went Looking for a Better Embedding Model. The Problem Was the Text.

Description and use

The author uncovered that Apple’s on‑device Turkish model collapses all Turkish sentences into a single vector, making retrieval ineffective. Encryption of summaries broke the keyword‑boost layer, forcing a redesign. Multilingual models tested did not improve the situation because the retrieval task was ill‑suited for embeddings alone. The lesson is to align the retrieval objective with the capabilities of the embedding model and to ensure that any preprocessing (encryption, keyword matching) does not invalidate the retrieval pipeline.

Problem

The author needed an on‑device retrieval system that could answer questions about personal messaging history, but Apple’s on‑device Turkish embedding model collapsed all Turkish sentences into a single vector, causing retrieval failures.

Outcome

The author uncovered that Apple’s on‑device Turkish model collapses all Turkish sentences into a single vector, making retrieval ineffective. Encryption of summaries broke the keyword‑boost layer, forcing a redesign. Multilingual models tested did not improve the situation because the retrieval task was ill‑suited for embeddings alone. The lesson is to align the retrieval objective with the capabilities of the embedding model and to ensure that any preprocessing (encryption, keyword matching) does not invalidate the retrieval pipeline.

Workflow

• 1. Read WhatsApp SQLite file (ChatStorage.sqlite) directly, noting it is unencrypted and not protected by macOS.
• 2. Store only encrypted summaries (AES‑256‑GCM) in a local vault, never persisting raw text.
• 3. Initially use Apple’s on‑device model (English) to embed Turkish text and apply keyword‑boost for retrieval.
• 4. Discover that keyword‑boost fails after encryption, so modify pipeline to decrypt candidates before scoring.
• 5. Test multilingual models (multilingual‑e5‑small, multilingual‑e5‑base, bge‑m3) on real WhatsApp data, measuring similarity gaps and top‑1 accuracy.
• 6. Realize that the test queries target speaker identity, which embeddings cannot infer because names are metadata, not text.
• 7. Conclude that embeddings alone are insufficient for the intended use case and that the system design must be revisited.

Evidence

WhatsApp keeps its local data in a SQLite file called ChatStorage.sqlite, inside a Group Container. I opened it from a normal terminal command. No prompt, no dialog, no "Terminal would like to access...", nothing.

Source

The encryption, the storage and the scoring code were all innocent. It gave 0.94 when querying the same summary.

Source

The three unrelated Turkish sentences scored between 0.79 and 0.85 against each other. The two unrelated English sentences scored 0.34.

Source

The average gaps came back as minus 0.0002 for e5-small, 0.0049 for e5-base and 0.0143 for bge-m3. Top‑1 accuracy was 0 percent for the small one and 33 percent for the other two.

Source