RULER

RULER is a synthetic benchmark designed to comprehensively evaluate long-context language models through flexible configurations for sequence length and task complexity. Expanding beyond needle-in-a-haystack tests, RULER includes information retrieval, multi-hop tracing, and aggregation tasks, revealing how model performance degrades as context length increases and testing behaviors beyond simple retrieval.