Which answer formats do forecasting bots silently drop? A cross-tool benchmark

说明与用途

创建了一个公开基准，比较了两个预测机器人提取器的性能，显示它们在处理答案格式时的相似准确性和差异。

问题

Forecasting bots’ accuracy depends on their extractors correctly reading model outputs; many extractors silently drop or misinterpret valid answer formats.

结果

Both extractors perform similarly; mv2-forecaster avoids false positives but drops some prose formats, while forecasting-tools captures prose but misreads base rates.

工作流

• 1. Collected 20 labeled answer format cases from deployed forecasting bots.
• 2. Implemented test_forecast_answer_formats.py to evaluate each extractor offline.
• 3. Implemented compare_extractors.py to run live packages unmodified.
• 4. Ran mv2-forecaster 0.1.7 and forecasting-tools against the corpus.
• 5. Analyzed results: mv2-forecaster had 14 correct, 2 miss, 0 false positive, 4 correct_reject; forecasting-tools had 13 correct, 3 miss, 2 false positive, 2 correct_reject.
• 6. Derived takeaways for bot authors regarding extractor design.

证据

Every deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them.

来源

I built one: forecast-answer-formats, the first public cross-tool benchmark of which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop – mv2-forecaster and Metaculus's own forecasting-tools.

来源

mv2-forecaster 0.1.7: 14 correct, 2 miss, 0 false positive, 4 correct_reject

来源

forecasting-tools: 13 correct, 3 miss, 2 false positive, 2 correct_reject

来源