Printing the answer isn't solving the exercise: how I grade beginners' code

Описание и применение

Платформа CyberStars использует скрытые тесты и проверку структуры AST, чтобы убедиться, что учащиеся реализуют линейный поиск, а не просто печатают ожидаемый результат.

Задача

Beginners often cheat by hard‑coding expected outputs, so automated grading must detect such behavior and enforce correct algorithmic implementation.

Результат

The grading system now reliably detects hard‑coded solutions, enforces linear search implementation, and uses hidden tests plus structural AST checks to ensure students learn the intended algorithm.

Рабочий процесс

• Define exercise with a function signature and a list variable name.
• Create 50 hidden test cases with varied lists and inputs, including edge cases.
• Store only the input (inject and stdin) for each case; no expected output is kept.
• For each submission, parse the student’s code with Python’s ast module.
• Replace the assignment to the list variable with the test case’s list in both student and reference code.
• Run both programs in isolated Docker containers with no network.
• Compare printed outputs; if they match, the test passes.
• Apply structure rules (required variable, function, loop) by inspecting the AST; if missing, the test fails.
• Include accept‑* and reject‑* reference programs to validate rule coverage.
• If any hidden test fails, show the input and student output but not the expected answer.
• Cache reference outputs for generated cases to reduce runtime.
• Iteratively add or adjust tests and rules to maintain robustness.

Доказательства

When a hidden test fails, the student sees the input and what their program printed, but not the expected answer.

Источник

For every case, the grader runs both programs, the student's and the reference, with the same input, and compares what they print.

Источник

The grader swaps the value of the list variable before running, by parsing the student's code with Python's ast module.

Источник

Each exercise also carries structure rules, checked on the student's syntax tree.

Источник