Same Evidence, Different Judgment? A Small Experiment on Implied Conclusions and Evidence Ordering

说明与用途

使用 gpt‑4.1‑mini 的实验表明，证据句子顺序可能影响模型判断，但样本量有限，无法得出最终结论。

问题

Determine whether ordering of evidence sentences affects a language model’s judgment of a relational conclusion.

结果

19 of 36 responses were accepted; 17 rejected due to Markdown code fences. Ordering of evidence sentences influenced model judgments in some conditions, but single observations per condition preclude definitive conclusions.

工作流

• Define four synthetic evidence cases with candidate conclusions and provisional labels.
• Create three presentation conditions: baseline, evidence_reverse, paragraph_reverse.
• For each case‑condition, send three stateless API requests: extract, judge, audit.
• Collect responses from gpt-4.1-mini via OpenAI Responses API.
• Analyze outcomes for evidence‑list and paragraph‑order effects.
• Document failures and acceptance rates.
• Publish results and protocol details.

证据

Four synthetic evidence cases × three presentation conditions × three independent tasks: 36 completed responses from gpt-4.1-mini

来源

Under the frozen parser, 19/36 responses were accepted and 17/36 were rejected solely for Markdown code fences surrounding JSON

来源

The requests used OpenAI's Responses API with requested model gpt-4.1-mini, max_output_tokens=900, store=false, and no temperature explicitly specified in the request.

来源