VelaVec: A 9.8M-Param Retrieval Encoder That Runs 53x Faster Than Its Teacher — on the CPU Alone

说明与用途

VelaVec 是一个 9.8M 参数的检索编码器，在 Apple M4 CPU 上比其 33M 教师模型快 53 倍，冷启动仅 5.4 ms，权重大小 39 MB，全部使用纯 Rust，无 Python 运行时。

问题

Embedding models shrink in accuracy and grow in deployment cost; teams often rely on GPUs or heavy Python runtimes that incur long cold starts.

结果

VelaVec delivers 53× faster query latency and 2600× faster cold start than its 33M teacher while maintaining 91–102% of teacher accuracy on symmetric tasks and 73–88% on retrieval tasks, all within a 39 MB footprint and pure‑Rust inference.

工作流

• 1. Distill 33M teacher to 9.8M student using hybrid distillation: frozen token‑embedding table, 3‑layer bidirectional attention, dual heads.
• 2. Train with multi‑task objective (KD, InfoNCE, embedding regression) across five domains.
• 3. Build a pure‑Rust inference engine leveraging BNNS/CoreML pre‑packed GEMM graphs and NEON kernels.
• 4. Optimize element‑wise ops (RMSNorm, SwiGLU, RoPE) with hand‑written NEON kernels and polynomial approximations.
• 5. Deploy the model via Rust binary (weights.bin) or Hugging Face safetensors for edge or serverless use.
• 6. Optionally extend with a lightweight vision bridge (VelaVec‑T2I) for multimodal retrieval.
• 7. Monitor latency and accuracy; adjust projection heads for text→image or image→text retrieval as needed.

证据

Query latency (M4 CPU) 103 µs vs 5.5 ms (teacher) – 53.9× faster

来源

Cold start 5.4 ms (Rust) vs ~29 s (PyTorch) – ~2600× faster

来源