Text Summarization API Quality and Cost Comparison (Per‑Tenant Review Rules)

说明与用途

构建可重复测试的指南，用于比较 OpenAI、Claude 和 Gemini API 的质量、模式有效性和租户级成本，包括适配器、验证、令牌计数和使用记录表。

问题

A marketplace that turns code changes into structured review findings needs a repeatable test to compare summarization APIs on quality, schema validity, and cost per tenant, rather than a leaderboard.

结果

A repeatable evaluation harness that records quality signals and token usage per tenant, enabling fair comparison of OpenAI, Claude, and Gemini and transparent cost billing.

工作流

• 1. Define a bounded output schema (Finding, ReviewResult) in TypeScript.
• 2. Implement adapters that translate each provider’s response into the schema and validate it with validateResult().
• 3. Attach tenantId, changeId, and evaluationVersion to every request and record provider‑reported usage (inputTokens, outputTokens).
• 4. Store a UsageRow per request, including retryCount and evaluationVersion.
• 5. Run the harness on a representative set of 50 changes covering small fixes, permission changes, checkout work, and noisy diffs.
• 6. Score each result on finding precision, omission rate, contract validity, and summary usefulness.
• 7. Compute cost metrics: cost per accepted finding, cost per reviewed change, and retry share.
• 8. Compare providers using the same prompt, change set, and evaluation rules, producing a compatibility table.
• 9. Pin evaluation rules to a version number (e.g., v7) to isolate prompt changes from provider changes.
• 10. Use the ledger to bill tenants based on recorded usage, keeping unit rates separate from review results.

证据

The useful comparison is not a leaderboard. It is a repeatable test that records summary quality, schema validity, and spend per tenant.

来源

OpenAI, Claude, and Gemini can all sit behind the same application contract, but the contract must own the evaluation.

来源

The recommendation is boring: keep the application contract stable and test each API against the same labeled changes.

来源

Cost visibility starts before an API call. Attach a tenant id, change id, and evaluation version to every request.

来源