Which answer formats do forecasting bots silently drop? A cross-tool benchmark

Описание и применение

Создано публичное сравнение извлечения ответов прогнозирующими ботами: два экстрактора показывают схожую точность, но различаются в обработке форматов текста.

Задача

Forecasting bots’ accuracy depends on their extractors correctly reading model outputs; many extractors silently drop or misinterpret valid answer formats.

Результат

Both extractors perform similarly; mv2-forecaster avoids false positives but drops some prose formats, while forecasting-tools captures prose but misreads base rates.

Рабочий процесс

• 1. Collected 20 labeled answer format cases from deployed forecasting bots.
• 2. Implemented test_forecast_answer_formats.py to evaluate each extractor offline.
• 3. Implemented compare_extractors.py to run live packages unmodified.
• 4. Ran mv2-forecaster 0.1.7 and forecasting-tools against the corpus.
• 5. Analyzed results: mv2-forecaster had 14 correct, 2 miss, 0 false positive, 4 correct_reject; forecasting-tools had 13 correct, 3 miss, 2 false positive, 2 correct_reject.
• 6. Derived takeaways for bot authors regarding extractor design.

Доказательства

Every deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them.

Источник

I built one: forecast-answer-formats, the first public cross-tool benchmark of which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop – mv2-forecaster and Metaculus's own forecasting-tools.

Источник

mv2-forecaster 0.1.7: 14 correct, 2 miss, 0 false positive, 4 correct_reject

Источник

forecasting-tools: 13 correct, 3 miss, 2 false positive, 2 correct_reject

Источник