VelaVec: A 9.8M-Param Retrieval Encoder That Runs 53x Faster Than Its Teacher — on the CPU Alone

Description and use

VelaVec is a 9.8 M‑parameter retrieval encoder that runs 53× faster than its 33 M teacher on Apple M4 CPU, with a 5.4 ms cold start and 39 MB weight size, all in pure Rust.

Problem

Embedding models shrink in accuracy and grow in deployment cost; teams often rely on GPUs or heavy Python runtimes that incur long cold starts.

Outcome

VelaVec delivers 53× faster query latency and 2600× faster cold start than its 33M teacher while maintaining 91–102% of teacher accuracy on symmetric tasks and 73–88% on retrieval tasks, all within a 39 MB footprint and pure‑Rust inference.

Workflow

• 1. Distill 33M teacher to 9.8M student using hybrid distillation: frozen token‑embedding table, 3‑layer bidirectional attention, dual heads.
• 2. Train with multi‑task objective (KD, InfoNCE, embedding regression) across five domains.
• 3. Build a pure‑Rust inference engine leveraging BNNS/CoreML pre‑packed GEMM graphs and NEON kernels.
• 4. Optimize element‑wise ops (RMSNorm, SwiGLU, RoPE) with hand‑written NEON kernels and polynomial approximations.
• 5. Deploy the model via Rust binary (weights.bin) or Hugging Face safetensors for edge or serverless use.
• 6. Optionally extend with a lightweight vision bridge (VelaVec‑T2I) for multimodal retrieval.
• 7. Monitor latency and accuracy; adjust projection heads for text→image or image→text retrieval as needed.

Evidence

Query latency (M4 CPU) 103 µs vs 5.5 ms (teacher) – 53.9× faster

Source

Cold start 5.4 ms (Rust) vs ~29 s (PyTorch) – ~2600× faster

Source