5,000 Log Lines, One Free Model Server, 97.4% Success

Описание и применение

Пайплайн обрабатывает 5 000 логов, используя бесплатный LLM, с 97,4 % успеха, благодаря очереди SQLite, повторным попыткам и контролю токенов.

Задача

Processing 5,000 mixed-format log lines to extract error type, severity, and service using a free model server while handling rate limits, timeouts, and token budget.

Результат

The pipeline processed 5,000 logs with 97.4% success, demonstrating that a free model tier can handle batch workloads when combined with retry logic, checkpointing, and token budgeting.

Рабочий процесс

• Set MC_BASE_URL and MC_API_KEY as environment variables
• Generate 5,000 log lines in three formats
• Create SQLite queue file batch_queue.db
• Read logs, enqueue each line
• For each queued line: build prompt, call OpenAI client with retries (max 4) and exponential backoff (2s base), retry on 429/5xx
• Store successful responses in a results file
• Checkpoint progress in SQLite to resume after crashes

Доказательства

I built one. It survived. Here’s the design.

Источник

Survive 429s and timeouts. Resume after a crash. Never exceed the token budget.

Источник