Same Evidence, Different Judgment? A Small Experiment on Implied Conclusions and Evidence Ordering

Description and use

An experiment with gpt‑4.1‑mini showed that the order of evidence sentences can affect model judgments, but the limited sample size prevents definitive conclusions.

Problem

Determine whether ordering of evidence sentences affects a language model’s judgment of a relational conclusion.

Outcome

19 of 36 responses were accepted; 17 rejected due to Markdown code fences. Ordering of evidence sentences influenced model judgments in some conditions, but single observations per condition preclude definitive conclusions.

Workflow

• Define four synthetic evidence cases with candidate conclusions and provisional labels.
• Create three presentation conditions: baseline, evidence_reverse, paragraph_reverse.
• For each case‑condition, send three stateless API requests: extract, judge, audit.
• Collect responses from gpt-4.1-mini via OpenAI Responses API.
• Analyze outcomes for evidence‑list and paragraph‑order effects.
• Document failures and acceptance rates.
• Publish results and protocol details.

Evidence

Four synthetic evidence cases × three presentation conditions × three independent tasks: 36 completed responses from gpt-4.1-mini

Source

Under the frozen parser, 19/36 responses were accepted and 17/36 were rejected solely for Markdown code fences surrounding JSON

Source

The requests used OpenAI's Responses API with requested model gpt-4.1-mini, max_output_tokens=900, store=false, and no temperature explicitly specified in the request.

Source