LLM Inference
Run and serve open-weight language models efficiently on your own hardware.
LLM Inference
Local inference engines let you run open-weight models without sending data to a hosted API. This topic covers runtimes, quantization techniques, and gateways that make large models usable on consumer and server hardware.
Use this comparison when you need to choose between maximum portability, low memory usage, or high-throughput serving for your own applications.
9 projects
- vllm-project/vllm — High-throughput inference and serving engine for LLMs with an OpenAI-compatible API — OpenAI-compatible API: Yes; GPU acceleration: Yes; Quantization support: Yes; Batched inference: Yes; Docker deployment: Yes; Built-in web UI: No; Speculative decoding: Yes
- JustVugg/colibri — Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. — OpenAI-compatible API: Yes; GPU acceleration: Yes; Quantization support: Yes; Batched inference: Yes; Docker deployment: Yes; Built-in web UI: Yes; Speculative decoding: Yes
- neuron-core/neuron-ai — PHP framework for building and orchestrating AI agents — OpenAI-compatible API: No; GPU acceleration: No; Quantization support: No; Batched inference: No; Docker deployment: No; Built-in web UI: No; Speculative decoding: No
- RunanywhereAI/wally — Get up and running with GLM-5.3-flash, DeepSeek, Gemma and other open source frontier models — OpenAI-compatible API: Yes; GPU acceleration: No; Quantization support: No; Batched inference: No; Docker deployment: Yes; Built-in web UI: No; Speculative decoding: No
- ollama/ollama — Local runtime that runs open-weight LLMs on macOS, Windows and Linux with one command — OpenAI-compatible API: Yes; GPU acceleration: Yes; Quantization support: Yes; Docker deployment: Yes; Built-in web UI: No; Speculative decoding: No
- ggml-org/llama.cpp — C/C++ inference engine that runs quantized LLMs and vision models on CPU and GPU — OpenAI-compatible API: Yes; GPU acceleration: Yes; Quantization support: Yes; Batched inference: Yes; Docker deployment: Yes; Built-in web UI: Yes; Speculative decoding: Yes
- Niko1221/Strata — Run very large MoE models locally with SSD offloading and a local API — OpenAI-compatible API: Yes; GPU acceleration: Yes; Quantization support: Yes; Built-in web UI: No; Speculative decoding: No
- stew675/llama-cpp-rdna-boosts — Patch series that tunes llama.cpp inference for AMD RDNA GPUs — OpenAI-compatible API: No; GPU acceleration: Yes; Quantization support: Yes; Docker deployment: No; Built-in web UI: No; Speculative decoding: Yes
- nasone32/llama.cpp-RDNA3-7900xtx-opt — llama.cpp fork tuned for AMD RDNA3 GPUs, with multi-GPU ROCm patches — GPU acceleration: Yes; Quantization support: Yes; Batched inference: Yes; Docker deployment: No; Speculative decoding: Yes