CRUXEval-Output-CoT

text
+
+
+
+
About

CRUXEval-Output-CoT enhances the standard CRUXEval output prediction task by incorporating chain-of-thought reasoning methodology. This benchmark evaluates Large Language Models' ability to predict program outputs while showing their reasoning process step-by-step. CRUXEval-Output-CoT tests both code execution understanding and the capacity to articulate logical reasoning in code analysis scenarios.

+
+
+
+
Evaluation Stats
Total Models1
Organizations1
Verified Results0
Self-Reported1
+
+
+
+
Benchmark Details
Max Score1
Language
en
+
+
+
+
Performance Overview
Score distribution and top performers

Score Distribution

1 models
Top Score
56.0%
Average Score
56.0%
High Performers (80%+)
0

Top Organizations

#1Alibaba Cloud / Qwen Team
1 model
56.0%
+
+
+
+
Leaderboard
1 models ranked by performance on CRUXEval-Output-CoT
LicenseLinks
Sep 19, 2024
Apache 2.0
56.0%
+
+
+
+
Resources