Statewave Guide — Day 4: We passed the safety test. Then failed the product test

Description and use

Safety testing succeeded, but product usefulness was low due to overly technical outputs. A Guidance Compiler and Renderer were added to translate verified technical knowledge into user‑friendly guidance.

Problem

Verified product knowledge from a model does not automatically translate into human‑friendly guidance, leading to technical, unhelpful outputs.

Outcome

Safety metrics remained strong while product usefulness improved by filtering technical details and presenting verified capabilities in user‑friendly language.

Workflow

• 1. Run safety test on model outputs. 2. Measure product usefulness with GPT‑5.6 Sol rubric. 3. Identify technical gaps and missing capabilities. 4. Add Guidance Compiler layer to filter what to tell the user. 5. Add Renderer layer to phrase guidance appropriately. 6. Re‑evaluate outputs for technicality and usefulness.

Evidence

Wrong-feature claims accepted 0 Unsupported claims accepted 0 Model restraint 30/30 Verified claim yield 95%

Source

1 / 3 median usefulness 24% rated genuinely useful

Source

14 / 21 outputs were too technical.

Source

Technically correct data structures do not automatically know how to talk to humans.

Source