`DeepSWE` Flags Leakage and False Negatives in AI Coding Benchmarks

Coding leaderboards can reward memorized answers and reject valid patches. Use SWE-bench-style rankings as a filter, not procurement truth.

• —DeepSWE ranks GPT-5.5 first, but the sharper signal is benchmark reliability, not the winner.
• —Answer leakage can inflate agent scores. A high leaderboard rank does not guarantee better repo-level work.
• —Valid solutions may be rejected, so benchmarks can undervalue agents that solve tasks in non-canonical ways.
• —Agent behavior suppression matters: tests built for narrow outputs can punish useful exploration and tool use.