API-Bank

API-Bank is a comprehensive benchmark for tool-augmented Large Language Models that evaluates their ability to find, plan, and execute API calls to accomplish user-defined goals. Featuring 264 annotated dialogues with 568 API calls, it tests LLMs across three key capabilities: deciding when to call APIs, finding relevant tools via keyword search, and employing multiple APIs for complex requests. The benchmark uses automatic and manual evaluation to assess tool-augmented reasoning.