COLLIE

COLLIE is a systematic framework for evaluating constrained text generation capabilities in Large Language Models. The benchmark tests models' ability to generate text under various compositional constraints with diverse generation levels and modeling challenges. COLLIE provides comprehensive assessment of controlled text generation through constraint structure specification, example extraction, instruction rendering, and rigorous evaluation against specified constraints.