EvaLooop: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming
ACM Transactions on Software Engineering and Methodology (TOSEM). arXiv · Leaderboard · Code
A loop-based robustness evaluation that alternates code generation and summarization on the same task. Applied to 96 LLMs (incl. Claude Opus 4, DeepSeek-V3), we observe a 2.65%–47.62% pass@1 drop over 10 loops, and find that robustness is not strictly tied to initial accuracy.