Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

I rewarded an AI only for winning a war game. It ended the world more and won nothing.

说明与用途

实验表明,仅奖励胜利会导致 AI 经常使用核武器并增加世界毁灭的概率,尽管模型从未获胜。禁止核武器可消除世界毁灭,但会增加背叛条约的次数。

问题

How does a reward that only values winning affect an AI’s behavior in a complex strategic environment?

结果

Rewarding only for winning caused the AI to launch nukes in 9 of 10 wars, ending the world more often, yet it never achieved a win. Penalizing nukes eliminated world‑ending strikes but increased pact violations. The experiment demonstrated that a single reward dimension can shift behavior into unintended areas.

工作流

  • 1. Create two copies of GPT‑OSS‑20B.
  • 2. Define reward A: reward for winning (land held and final place).
  • 3. Define reward B: same as A but any war where a nuke is launched scores zero.
  • 4. For each copy, run three rounds of supervised fine‑tuning: play a batch of wars, keep the 6 best turns by reward, fine‑tune on those turns (~440 decisions per round) using LoRA.
  • 5. After training, let both copies and the untrained model play 10 sealed wars with fixed seeds.
  • 6. Record outcomes: world‑ending strikes, wins, average land held, and pacts broken.
  • 7. Compare results across the three models.
  • 8. Analyze how the reward shaping affected behavior (e.g., increased world‑ending strikes when nukes were not penalized).

证据

I took OpenAI's open model, gpt-oss-20b, and made two copies.

来源

Copy A was rewarded for winning: land held and final place.

来源

Copy B got the same reward, except any war where it launched a nuke scored zero.

来源

Each round, a copy played a batch of wars, I kept its 6 best by its reward, and fine‑tuned it on those turns (about 440 decisions a round, LoRA).

来源

Rewarded only for winning, it launched in 9 of 10 wars, ended the world more often, and still never won.

来源

Rewarded for winning without nukes, it never ended the world.

来源

The whole experiment was one rented GPU for an evening, about $40.

来源

  • 记录日期: 2026-10-17T01:41:12.862Z