RouteLLM
Like
A framework for serving and evaluating LLM routers — save LLM costs without compromising quality.
Cost / License
- Free
- Open Source
Platforms
- Python
- Self-Hosted
RouteLLM
Like
Features
No features, maybe you want to suggest one?
Tags
- mistral-ai
- mistral-7b
- gpt4
- Software Framework
RouteLLM News & Activities
Highlights All activities
Recent activities
No activities found.
RouteLLM information
Developed by
LMSYS Licensing
Open Source (Apache-2.0) and Free product.Written in
Alternatives
0 alternatives listedSupported Languages
- English
AlternativeTo Categories
Development, AI Tools & ServicesGitHub repository
- 4,487 Stars
- 355 Forks
- 36 Open Issues
- Updated
No comments or reviews, maybe you want to be first?
Post comment/reviewWhat is RouteLLM?
RouteLLM is a framework for serving and evaluating LLM routers.
Our core features include:
- Drop-in replacement for OpenAI's client (or launch an OpenAI-compatible server) to route simpler queries to cheaper models.
- Trained routers are provided out of the box, which we have shown to reduce costs by up to 85% while maintaining 95% GPT-4 performance on widely-used benchmarks like MT Bench.
- Benchmarks also demonstrate that these routers achieve the same performance as commercial offerings while being >40% cheaper.
- Easily extend the framework to include new routers and compare the performance of routers across multiple benchmarks.