Vibe-Eval

VIBE-eval is a challenging multimodal evaluation benchmark featuring 269 visual understanding prompts, including 100 hard-difficulty questions with expert-authored gold-standard responses. This rigorous assessment tests frontier multimodal chat models' visual comprehension capabilities, with over 50% of hard questions typically answered incorrectly, highlighting significant gaps in current AI visual reasoning abilities.