Many benchmarking papers

Benefits:

• reduce data contamination

• decentralize benchmark construction

• keep pace with evolving agents; benchmarks should also self-evolve

(2024 Mar) LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

(2024 June) Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models

(2024 June) LiveBench: A Challenging, Contamination-Limited LLM Benchmark

(2024 Dec) AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

(2025 Aug) DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis

https://futurex-ai.github.io/

(2024 July) AutoBencher: Towards Declarative Benchmark Construction

(2024, Jul) Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM