Text Summarization API Quality and Cost Comparison (Per‑Tenant Review Rules)

Description and use

Guide to building a repeatable test for comparing OpenAI, Claude, and Gemini APIs on quality, schema validity, and tenant‑level cost, with adapters, validation, token accounting, and a usage ledger.

Problem

A marketplace that turns code changes into structured review findings needs a repeatable test to compare summarization APIs on quality, schema validity, and cost per tenant, rather than a leaderboard.

Outcome

A repeatable evaluation harness that records quality signals and token usage per tenant, enabling fair comparison of OpenAI, Claude, and Gemini and transparent cost billing.

Workflow

• 1. Define a bounded output schema (Finding, ReviewResult) in TypeScript.
• 2. Implement adapters that translate each provider’s response into the schema and validate it with validateResult().
• 3. Attach tenantId, changeId, and evaluationVersion to every request and record provider‑reported usage (inputTokens, outputTokens).
• 4. Store a UsageRow per request, including retryCount and evaluationVersion.
• 5. Run the harness on a representative set of 50 changes covering small fixes, permission changes, checkout work, and noisy diffs.
• 6. Score each result on finding precision, omission rate, contract validity, and summary usefulness.
• 7. Compute cost metrics: cost per accepted finding, cost per reviewed change, and retry share.
• 8. Compare providers using the same prompt, change set, and evaluation rules, producing a compatibility table.
• 9. Pin evaluation rules to a version number (e.g., v7) to isolate prompt changes from provider changes.
• 10. Use the ledger to bill tenants based on recorded usage, keeping unit rates separate from review results.

Evidence

The useful comparison is not a leaderboard. It is a repeatable test that records summary quality, schema validity, and spend per tenant.

Source

OpenAI, Claude, and Gemini can all sit behind the same application contract, but the contract must own the evaluation.

Source

The recommendation is boring: keep the application contract stable and test each API against the same labeled changes.

Source

Cost visibility starts before an API call. Attach a tenant id, change id, and evaluation version to every request.

Source