Not Just a Working Demo: Why We Ran 162 Migration Tests for Two DSH Plugins

Description and use

The migration test of two DSH plugins for Codex and Claude Code ran 162 scenarios, identifying supported, partially supported, and unsupported features. After fixes, versions 0.1.3 and 0.1.4 were released, and the process is repeatable and documented.

Problem

Ensuring that DeepSeek Harness (DSH) plugins for Codex and Claude Code fully support real‑world development workflows after migration, rather than just passing a demo.

Outcome

The migration test suite confirmed which capabilities are fully supported, partially supported, or unsupported, enabling targeted fixes and transparent communication to users about migration boundaries.

Workflow

• 1. Stop running DSH Web process.
• 2. Install plugins: dsh plugin --profile web add [email protected] dsh plugin --profile web add [email protected] dsh web.
• 3. Run the 162 migration test suite covering conversation, images, files, code, shell tools, Git, Skills, MCP, permissions, env vars, session import, host restarts, long‑context continuation.
• 4. Review results: supported, partial, unsupported per capability.
• 5. Fix identified failures (image transport, session import, process termination, secret persistence).
• 6. Release updated plugin versions and publish new support matrix.
• 7. Repeat cycle for continuous validation.

Evidence

We split real development work into 162 atomic requirements: 76 for Codex and 86 for Claude.

Source

The matrix covers conversation and multi-turn context, images and files, code and shell tools, tests and Git, Skills, MCP, project configuration, permissions, environment variables, session import, host restarts, and long-context continuation.

Source

By August 29, 2026, all 162 cases had results. We retained 167 runs in total; five were reviewed Codex reruns and do not inflate the requirement count.

Source