Write a Receipt for Every Remote Inference

说明与用途

文章描述了测量本地和远程推理性能、根据预算做出路由决策，并记录包含提示哈希的收据以保护隐私并分析延迟和成本的工作流程。

问题

Teams often choose local or remote inference once and never re‑measure, leading to wasted time and privacy risk.

结果

The workflow enables objective routing decisions, protects private data via hashed prompts, and provides actionable logs for latency and privacy analysis.

工作流

• Measure local TPS with bench_local.py
• Measure remote TPS with bench_remote.py
• Measure round‑trip time with rtt.py
• Export metrics to .envrc
• Implement choose_route to compare local vs remote against budget
• Generate a receipt with prompt hash, bytes sent, tokens out, latency, and reason
• Append receipts as JSONL for later analysis

证据

Measure once, encode a small decision function, and keep a receipt for every request that went remote.

来源

A log line records that something happened. A receipt records why a decision was made.

来源

Notice what the function does not do. It does not ask whether you like cloud services. It does not care about brand.

来源