VelaVec: A 9.8M-Param Retrieval Encoder That Runs 53x Faster Than Its Teacher — on the CPU Alone

Описание и применение

VelaVec – 9,8‑млн‑параметровый энкодер, работающий на CPU в 53‑кратном ускорении по сравнению с учителем, с холодным стартом 5,4 мс и 39 МБ весами, без Python‑runtime.

Задача

Embedding models shrink in accuracy and grow in deployment cost; teams often rely on GPUs or heavy Python runtimes that incur long cold starts.

Результат

VelaVec delivers 53× faster query latency and 2600× faster cold start than its 33M teacher while maintaining 91–102% of teacher accuracy on symmetric tasks and 73–88% on retrieval tasks, all within a 39 MB footprint and pure‑Rust inference.

Рабочий процесс

• 1. Distill 33M teacher to 9.8M student using hybrid distillation: frozen token‑embedding table, 3‑layer bidirectional attention, dual heads.
• 2. Train with multi‑task objective (KD, InfoNCE, embedding regression) across five domains.
• 3. Build a pure‑Rust inference engine leveraging BNNS/CoreML pre‑packed GEMM graphs and NEON kernels.
• 4. Optimize element‑wise ops (RMSNorm, SwiGLU, RoPE) with hand‑written NEON kernels and polynomial approximations.
• 5. Deploy the model via Rust binary (weights.bin) or Hugging Face safetensors for edge or serverless use.
• 6. Optionally extend with a lightweight vision bridge (VelaVec‑T2I) for multimodal retrieval.
• 7. Monitor latency and accuracy; adjust projection heads for text→image or image→text retrieval as needed.

Доказательства

Query latency (M4 CPU) 103 µs vs 5.5 ms (teacher) – 53.9× faster

Источник

Cold start 5.4 ms (Rust) vs ~29 s (PyTorch) – ~2600× faster

Источник