llama.cpp
4 likes
The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
Cost / License
- Free
- Open Source (MIT)
Application types
EUPlatforms
- Windows
- Mac
- Linux
- Docker
- Homebrew
- Nix Package Manager
- MacPorts
- Self-Hosted
llama.cpp News & Activities
Highlights All activities
Recent activities
llama.cpp information
Developed by
ggml-org Licensing
Open Source (MIT) and Free product.Written in
Alternatives
27 alternatives listedSupported Languages
- English
AlternativeTo Categories
AI Tools & Services, System & HardwareGitHub repository
- 106,529 Stars
- 17,356 Forks
- 1540 Open Issues
- Updated
Featured in Lists
Wake up the NPU on your device
List by bugmenot with 46 apps, updated
What is llama.cpp?
The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is the main playground for developing new features for the ggml library.
Comments and Reviews
I like llama.cpp because it is the first and best open source application for LLM. It is not only the core engine of GPT4ALL, LM Studio, Jan, Ollama and so on, but also a friendly application with WebUI and router mode.