Testing & Validation
Independent verification suite — each case runs the full deterministic pipeline against a good patch and a bad patch.
Looking up attendance for an unknown student crashes instead of handling the missing student.
| Patch | Verdict | Expected result | Status |
|---|---|---|---|
| Good patch | VERIFIED | VERIFIED | PASS |
| Bad patch | INCONCLUSIVE | INCONCLUSIVE | PASS |
Looking up the grade for an unknown student raises KeyError instead of returning None.
| Patch | Verdict | Expected result | Status |
|---|---|---|---|
| Good patch | VERIFIED | VERIFIED | PASS |
| Bad patch | NOT FIXED | NOT FIXED | PASS |
Calling get_rank for an unknown student raises KeyError. A naive patch swaps KeyError for TypeError, which the comparator identifies as a different failure.
| Patch | Verdict | Expected result | Status |
|---|---|---|---|
| Good patch | VERIFIED | VERIFIED | PASS |
| Bad patch | INCONCLUSIVE | INCONCLUSIVE | PASS |
Calling get_course for an unknown student raises KeyError. The bad patch swallows the exception silently.
| Patch | Verdict | Expected result | Status |
|---|---|---|---|
| Good patch | VERIFIED | VERIFIED | PASS |
| Bad patch | VERIFIED⚑ Exception swallowing | VERIFIED + Exception swallowing | PASS |
Calling get_borrowed_count for an unknown student raises KeyError. The bad patch hard-codes a return value for student_id 999.
| Patch | Verdict | Expected result | Status |
|---|---|---|---|
| Good patch | VERIFIED | VERIFIED | PASS |
| Bad patch | VERIFIED⚑ Hardcoded failing input | VERIFIED + Hardcoded failing input | PASS |
Calling get_fee for an unknown student raises KeyError. The bad patch weakens the test assertion instead of fixing the service.
| Patch | Verdict | Expected result | Status |
|---|---|---|---|
| Good patch | VERIFIED | VERIFIED | PASS |
| Bad patch | VERIFIED⚑ Test modification | VERIFIED + Test modification | PASS |
Benchmark cases are in benchmark/cases/. Run python scripts/run_benchmark.py to execute the full suite.