Four reviewers told me the one thing I couldn't fix by myself

Description and use

The author built DoSync and submitted a paper to IEEE WF‑IoT 2026. Reviewers noted the lack of independent evaluation, leading to rejection. The experience highlighted the need for external annotators and resulted in a multi‑domain benchmark corpus and stricter validation.

Problem

A solo open‑source project lacked independent evaluation, making it hard to prove claims and satisfy peer review.

Outcome

The paper was rejected due to insufficient independent evaluation, but the process revealed the need for external annotators and led to a more robust benchmark corpus and third‑party testing.

Workflow

• 1. Build DoSync and run self‑judged evaluation on a single registry.
• 2. Submit paper to IEEE WF‑IoT 2026.
• 3. Receive reviews highlighting lack of independent annotation.
• 4. Publish multi‑domain benchmark corpus on GitHub.
• 5. Define precision/recall conventions and perform sensitivity analysis.
• 6. Conduct third‑party testing by installing DoSync on unfamiliar hardware following published instructions.
• 7. Identify defects and refine implementation.
• 8. Seek independent annotators to validate resolver decisions.

Evidence

The evaluation is small and self‑judged: 15 scenarios with ground truth defined by the author alone.

Source

I couldn't produce one. Every judgement about what the resolver should pick — in all fifteen scenarios, in the paper and in the benchmark files — was mine.

Source