I raced six models against each other on DigitalOcean Inference. The cheapest one won.

Описание и применение

Сравнение шести моделей на DigitalOcean Inference показало, что Mistral‑3‑14B обеспечивает лучшую скорость, стоимость и надёжность, в то время как самые большие модели часто возвращают пустые ответы.

Задача

Choosing the right LLM model for a production endpoint is hard because latency, cost, and availability vary widely and are hard to benchmark before deployment.

Результат

Mistral‑3‑14B outperformed all other models in latency, cost, and reliability for the tested prompts, while the largest model (Qwen3.5‑397b‑a17b) was slowest, most expensive, and frequently returned empty responses.

Рабочий процесс

• 1. Write a 390‑line Python script that sends the same prompt to six models via the DigitalOcean Inference API.
• 2. Use a single OpenAI client with the DigitalOcean base URL and a model‑access key.
• 3. Open six EventSource connections (one per model) to stream responses concurrently.
• 4. Run the script locally, capturing TTFT, total time, tokens, and cost per model.
• 5. Analyze results: compare TTFT, total time, cost, and empty‑response rate.
• 6. Adjust Gunicorn workers to threaded mode to reduce queuing latency.
• 7. Update model picker to filter out unavailable models based on 403 responses.
• 8. Document findings and recommend Mistral‑3‑14B for similar workloads.

Доказательства

Mistral 14B won on every axis I measured. Fastest to first token, fastest overall, cheapest per run, and it answered every single time.

Источник

The catalog will lie to you. The model picker is built from GET /v1/models, because hardcoding a model list is how you end up shipping a dead one.

Источник