BigCodeBench-Hard
text
+
+
+
+
About
BigCodeBench-Hard is a curated subset of approximately 150 challenging tasks from BigCodeBench, focusing on user-facing and particularly difficult programming problems. This variant emphasizes complex coding scenarios that require advanced reasoning, sophisticated function calling, and deep understanding of programming concepts. It serves as a rigorous benchmark for evaluating the limits of AI coding capabilities in realistic, demanding scenarios.
+
+
+
+
Evaluation Stats
Total Models1
Organizations1
Verified Results0
Self-Reported1
+
+
+
+
Benchmark Details
Max Score1
Language
en
+
+
+
+
Performance Overview
Score distribution and top performers
Score Distribution
1 models
Top Score
27.0%
Average Score
27.0%
High Performers (80%+)
0Top Organizations
#1Alibaba Cloud / Qwen Team
1 model
27.0%
+
+
+
+
Leaderboard
1 models ranked by performance on BigCodeBench-Hard
| License | Links | ||||
|---|---|---|---|---|---|
| Sep 19, 2024 | Apache 2.0 | 27.0% |
