Same Evidence, Different Judgment? A Small Experiment on Implied Conclusions and Evidence Ordering

Описание и применение

Эксперимент с gpt‑4.1‑mini показал, что порядок предложений‑доказательств может влиять на выводы модели, однако ограниченный размер выборки не позволяет делать окончательные выводы.

Задача

Determine whether ordering of evidence sentences affects a language model’s judgment of a relational conclusion.

Результат

19 of 36 responses were accepted; 17 rejected due to Markdown code fences. Ordering of evidence sentences influenced model judgments in some conditions, but single observations per condition preclude definitive conclusions.

Рабочий процесс

• Define four synthetic evidence cases with candidate conclusions and provisional labels.
• Create three presentation conditions: baseline, evidence_reverse, paragraph_reverse.
• For each case‑condition, send three stateless API requests: extract, judge, audit.
• Collect responses from gpt-4.1-mini via OpenAI Responses API.
• Analyze outcomes for evidence‑list and paragraph‑order effects.
• Document failures and acceptance rates.
• Publish results and protocol details.

Доказательства

Four synthetic evidence cases × three presentation conditions × three independent tasks: 36 completed responses from gpt-4.1-mini

Источник

Under the frozen parser, 19/36 responses were accepted and 17/36 were rejected solely for Markdown code fences surrounding JSON

Источник

The requests used OpenAI's Responses API with requested model gpt-4.1-mini, max_output_tokens=900, store=false, and no temperature explicitly specified in the request.

Источник