vllm-project/vllm
A Python inference and serving engine for large language models on NVIDIA, AMD, Intel and CPU hardware. PagedAttention, continuous batching and prefix caching keep GPU memory tight while raising throughput, and an OpenAI-compatible server adds tensor parallelism, multi-LoRA and structured output. It loads 200+ Hugging Face architectures, supports FP8, INT4 and GGUF quantization, and runs from a wheel or container.
Maintenance activity