# LLM Inference

> Run and serve open-weight language models efficiently on your own hardware.

Navigation: [Index](../../llms.txt) · [HTML](https://awesomeopensource.dev/topic/llm-inference)

Local inference engines let you run open-weight models without sending data to a hosted API. This topic covers runtimes, quantization techniques, and gateways that make large models usable on consumer and server hardware.

Use this comparison when you need to choose between maximum portability, low memory usage, or high-throughput serving for your own applications.

## Feature comparison

| Project | OpenAI-compatible API | GPU acceleration | Quantization support | Batched inference | Docker deployment | Built-in web UI | Speculative decoding |
|---|---|---|---|---|---|---|---|
| [vllm-project/vllm](../vllm-project-vllm.md) | Yes | Yes | Yes | Yes | Yes | No | Yes |
| [JustVugg/colibri](../justvugg-colibri.md) | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| [neuron-core/neuron-ai](../neuron-core-neuron-ai.md) | No | No | No | No | No | No | No |
| [RunanywhereAI/wally](../runanywhereai-wally.md) | Yes | No | No | No | Yes | No | No |
| [ollama/ollama](../ollama-ollama.md) | Yes | Yes | Yes | – | Yes | No | No |
| [ggml-org/llama.cpp](../ggml-org-llama-cpp.md) | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| [Niko1221/Strata](../niko1221-strata.md) | Yes | Yes | Yes | – | – | No | No |
| [stew675/llama-cpp-rdna-boosts](../stew675-llama-cpp-rdna-boosts.md) | No | Yes | Yes | – | No | No | Yes |
| [nasone32/llama.cpp-RDNA3-7900xtx-opt](../nasone32-llama-cpp-rdna3-7900xtx-opt.md) | – | Yes | Yes | Yes | No | – | Yes |

## Features

- **OpenAI-compatible API**: Exposes an OpenAI-style chat completions endpoint.
- **GPU acceleration**: Uses CUDA, ROCm or other GPU backends.
- **Quantization support**: Loads quantized model formats.
- **Batched inference**: Improves throughput with batching or paged attention.
- **Docker deployment**: Ships a Docker image or compose setup.
- **Built-in web UI**: Ships a browser interface for chatting with loaded models.
- **Speculative decoding**: Accelerates generation by drafting tokens with a smaller model or n-grams.

## 9 projects

- [vllm-project/vllm](../vllm-project-vllm.md) · Activity score 5.0/5 (Thriving)
- [JustVugg/colibri](../justvugg-colibri.md) · Activity score 5.0/5 (Thriving)
- [neuron-core/neuron-ai](../neuron-core-neuron-ai.md) · Activity score 5.0/5 (Thriving)
- [RunanywhereAI/wally](../runanywhereai-wally.md) · Activity score 5.0/5 (Thriving)
- [ollama/ollama](../ollama-ollama.md) · Activity score 4.9/5 (Thriving)
- [ggml-org/llama.cpp](../ggml-org-llama-cpp.md) · Activity score 4.8/5 (Thriving)
- [Niko1221/Strata](../niko1221-strata.md) · Activity score 4.8/5 (Thriving)
- [stew675/llama-cpp-rdna-boosts](../stew675-llama-cpp-rdna-boosts.md) · Activity score 4.6/5 (Thriving)
- [nasone32/llama.cpp-RDNA3-7900xtx-opt](../nasone32-llama-cpp-rdna3-7900xtx-opt.md) · Activity score 3.6/5 (Active)
