Claude 3.5 Haiku
Zero-eval
by Anthropic
+
+
+
+
About
Claude 3.5 Haiku is a language model developed by Anthropic. It achieves strong performance with an average score of 60.8% across 9 benchmarks. It excels particularly in HumanEval (88.1%), MGSM (85.6%), DROP (83.1%). It supports a 400K token context window for handling large documents. The model is available through 3 API providers. Released in 2024, it represents Anthropic's latest advancement in AI technology.
+
+
+
+
Pricing Range
Input (per 1M)$0.80 -$1.00
Output (per 1M)$4.00 -$5.00
Providers3
+
+
+
+
Timeline
AnnouncedOct 22, 2024
ReleasedOct 22, 2024
+
+
+
+
License & Family
License
Proprietary
Performance Overview
Performance metrics and category breakdown
Overall Performance
9 benchmarks
Average Score
60.8%
Best Score
88.1%
High Performers (80%+)
3Performance Metrics
Max Context Window
400.0KAvg Throughput
82.0 tok/sAvg Latency
0ms+
+
+
+
All Benchmark Results for Claude 3.5 Haiku
Complete list of benchmark scores with detailed information
| HumanEval | text | 0.88 | 88.1% | Self-reported | |
| MGSM | text | 0.86 | 85.6% | Self-reported | |
| DROP | text | 0.83 | 83.1% | Self-reported | |
| MATH | text | 0.69 | 69.4% | Self-reported | |
| MMLU-Pro | text | 0.65 | 65.0% | Self-reported | |
| TAU-bench Retail | text | 0.51 | 51.0% | Self-reported | |
| GPQA | text | 0.42 | 41.6% | Self-reported | |
| SWE-Bench Verified | text | 0.41 | 40.6% | Self-reported | |
| TAU-bench Airline | text | 0.23 | 22.8% | Self-reported |