AI-Generated UI Tests in Business Central, or How to Check That "Passed" Actually Means Something

Description and use

Automated UI tests in Business Central can falsely report success. Adding independent data checks and correcting plan syntax reveals real failures.

Problem

Automated UI tests in Business Central can report "passed" even when the actions did not actually succeed, leading to false confidence in test results.

Outcome

Tests that previously reported success were found to be false positives. Adding independent data checks and correcting plan syntax ensured accurate test outcomes.

Workflow

• 1. Record UI interactions with Page Scripting to generate a YAML plan.
• 2. Run the plan with npx replay or the Page Scripting pane.
• 3. Observe that the runner reports "passed" even when BC shows error banners or data is not saved.
• 4. Identify three failure scenarios: server validation banner, Boolean filter with value true, and steps with incorrect key (step instead of type).
• 5. Add an independent data check: query the database (on‑prem) or call the standard API (SaaS) to verify the record was created.
• 6. Correct the plan: use option numbers for Boolean filters (e.g., value: 2 for Yes) and ensure all steps use the correct "type:" key.
• 7. Re‑run the plan and confirm that failures are now reported and the data is present.

Evidence

The test passed. The official bc-replay runner reports passed, and the Page Scripting pane says "Script completed without errors".

Source

The runner reports passed every time, with no warning at all.

Source

The player silently fails to apply a filter on a Boolean field with the value true.

Source

If you write step: instead of type:, the player doesn't run the step, but it doesn't fail either.

Source