When We Audited Someone Else's Robot Dataset on GitHub, the Hardware Maker Responded

Description and use

Automated robot dataset audits can mislead without collection context. Feedback from the hardware maker revealed that high spike counts and idle ratios were artifacts of the collection method, not data defects. RDA now reports control mode metadata and transparent thresholds.

Problem

Automated quality audits of robot datasets can produce misleading metrics if collection context is ignored, leading to incorrect conclusions about data quality.

Outcome

The audit revealed that high spike counts were due to non-interpolated data collection, not data corruption. High idle ratios are normal for manipulation tasks. RDA now includes control mode metadata and transparent thresholds, improving metric interpretability.

Workflow

• 1. Run RDA v0.9.12 on target datasets.
• 2. Review audit report for spike counts, idle ratios, and other diagnostics.
• 3. Open a GitHub Issue with the report on the dataset repository.
• 4. Await feedback from hardware manufacturer or dataset authors.
• 5. Interpret metrics in context of collection mode and task type.
• 6. Update RDA to include control mode metadata and gap_multiplier_used.
• 7. Re-run audit and compare results.
• 8. Adjust data collection strategy (e.g., use mode 6/7) if needed.

Evidence

If operated with non-interpolated mode, then the spikes would appear easily.

Source

We normally recommend collecting data with online trajectory planning modes (6 or 7) to utilize the dynamic interpolation and velocity control in our control box.

Source