RepoBench

RepoBench is a comprehensive benchmark specifically designed for evaluating repository-level code auto-completion systems, testing models' ability to understand cross-file dependencies and demonstrate repository-level understanding. This evaluation goes beyond single-file code completion, challenging AI models to comprehend complex codebases, maintain context across multiple files, and generate contextually appropriate code completions.