Printing the answer isn't solving the exercise: how I grade beginners' code

说明与用途

CyberStars 通过隐藏测试和 AST 结构检查，确保学生实现线性搜索，而不是硬编码输出。

问题

Beginners often cheat by hard‑coding expected outputs, so automated grading must detect such behavior and enforce correct algorithmic implementation.

结果

The grading system now reliably detects hard‑coded solutions, enforces linear search implementation, and uses hidden tests plus structural AST checks to ensure students learn the intended algorithm.

工作流

• Define exercise with a function signature and a list variable name.
• Create 50 hidden test cases with varied lists and inputs, including edge cases.
• Store only the input (inject and stdin) for each case; no expected output is kept.
• For each submission, parse the student’s code with Python’s ast module.
• Replace the assignment to the list variable with the test case’s list in both student and reference code.
• Run both programs in isolated Docker containers with no network.
• Compare printed outputs; if they match, the test passes.
• Apply structure rules (required variable, function, loop) by inspecting the AST; if missing, the test fails.
• Include accept‑* and reject‑* reference programs to validate rule coverage.
• If any hidden test fails, show the input and student output but not the expected answer.
• Cache reference outputs for generated cases to reduce runtime.
• Iteratively add or adjust tests and rules to maintain robustness.

证据

When a hidden test fails, the student sees the input and what their program printed, but not the expected answer.

来源

For every case, the grader runs both programs, the student's and the reference, with the same input, and compares what they print.

来源

The grader swaps the value of the list variable before running, by parsing the student's code with Python's ast module.

来源

Each exercise also carries structure rules, checked on the student's syntax tree.

来源