AI CODE REVIEW BENCHMARK & TRAINING ARENA
Coding agents write plausible pull requests that hide race conditions, injection sinks, and tool-execution escapes. Train your review reflex on real multi-language diffs and measure your precision, recall, and F1 score against ground-truth defect rules.
Refactored payment capture handler to check idempotency cache before charging customer card.
Syntax linters catch missing semicolons. Diffly trains engineers on the structural failure modes that pass CI and break production.
Agents frequently bypass AST validators with raw eval(), shell interpolation, or unverified tool-call arguments. Diffly tests whether you catch execution sinks disguised as helper refactors.
LLM-generated async handlers routinely split atomic operations into separate check-then-act reads and writes. Spot stale closure captures, missing cancellation tokens, and swallowed gather exceptions.
Even when parameterized queries protect WHERE clauses, agents interpolate sort columns, tenant headers, or redirect URLs directly. Audit subtle trust-boundary violations line by line.
Every submission computes exact line-level precision and recall so guessing every line tanks your score.
Read the threat model, environment specs, and unified or split diff exactly as it lands in code review.
Attach line-scoped review annotations explaining the root cause and failure mode without noisy false alarms.
Diffly evaluates your annotations against deterministic line-range and keyword rules or live Gemini BYOK grading.