Copilot Summarized My Conference. Its Best Slide Doesn't Exist.

Description and use

The author used Copilot to summarize a conference, found that the AI sometimes invented statements not present in the speaker decks, and identified gaps in the AIEOS system.

Problem

The author wanted to summarize conference sessions using Copilot and compare the AI-generated summary to the actual speaker decks, discovering discrepancies and gaps.

Outcome

Copilot produced a largely accurate summary but also invented statements not present in any slide deck. The exercise highlighted specific gaps in the author’s AIEOS system, such as missing runtime gating and semantic layer ownership, and confirmed alignment with the author’s existing proposals.

Workflow

• 1. Provide Copilot with materials from fourteen conference sessions.
• 2. Request a summary from Copilot.
• 3. Review Copilot’s executive summary for accuracy.
• 4. Open each speaker’s deck (total 57 slides).
• 5. Compare Copilot’s claims to the slide content.
• 6. Identify missing or invented statements.
• 7. Document gaps and align them with existing AIEOS features.
• 8. Plan enhancements based on identified gaps.

Evidence

I gave Copilot the materials from fourteen conference sessions and asked for a summary.

Source

The best line in the whole document came from the session on taming agent swarms: "One Agent -> Test -> Multiple Agents not Multiple Agents -> Hope"

Source

The word "hope" isn't in it.

Source

Copilot wrote that version of it, and wrote it well enough that I was ready to publish it as his.

Source