Four reviewers told me the one thing I couldn't fix by myself

Описание и применение

Автор создал протокол DoSync и отправил статью на IEEE WF‑IoT 2026. Рецензенты отметили отсутствие независимой оценки, что привело к отказу. Процесс выявил необходимость внешних аннотаторов и привёл к созданию много‑доменного корпуса тестов и более строгой валидации.

Задача

A solo open‑source project lacked independent evaluation, making it hard to prove claims and satisfy peer review.

Результат

The paper was rejected due to insufficient independent evaluation, but the process revealed the need for external annotators and led to a more robust benchmark corpus and third‑party testing.

Рабочий процесс

• 1. Build DoSync and run self‑judged evaluation on a single registry.
• 2. Submit paper to IEEE WF‑IoT 2026.
• 3. Receive reviews highlighting lack of independent annotation.
• 4. Publish multi‑domain benchmark corpus on GitHub.
• 5. Define precision/recall conventions and perform sensitivity analysis.
• 6. Conduct third‑party testing by installing DoSync on unfamiliar hardware following published instructions.
• 7. Identify defects and refine implementation.
• 8. Seek independent annotators to validate resolver decisions.

Доказательства

The evaluation is small and self‑judged: 15 scenarios with ground truth defined by the author alone.

Источник

I couldn't produce one. Every judgement about what the resolver should pick — in all fifteen scenarios, in the paper and in the benchmark files — was mine.

Источник