MMStar

MMStar is an elite vision-indispensable multimodal benchmark comprising 1,500 challenge samples meticulously selected by humans. It focuses on evaluating large vision-language models on tasks that absolutely require visual understanding, eliminating questions that can be answered through text alone. This curated benchmark provides more accurate assessment of models' true multimodal capabilities by ensuring vision-dependency in all evaluation tasks.