Neura Market
GPUStack logo

GPUStack

Free

An open-source GPU cluster manager for running LLMs

FreeFree tier
Inputs: text, audio, image, videoOutputs: text, audio, image, video
Type
Open Source
Visit Website GitHub (38 stars)

About GPUStack

GPUStack is an open-source GPU cluster manager designed for high-performance AI model serving and on-demand GPU instance provisioning. It orchestrates inference engines like vLLM, SGLang, and TensorRT-LLM, supports custom engines, and provides SSH-accessible GPU instances for development, fine-tuning, and interactive workloads. Key capabilities include multi-cluster management across on-premises, Kubernetes, and cloud environments, day-0 model support with pluggable engine architecture, performance-optimized configurations (low latency, high throughput, LMCache, HiCache, speculative decoding), and enterprise-grade operations with automated failure recovery, load balancing, monitoring, authentication, and access control. GPUStack supports a wide range of accelerators including NVIDIA, AMD, Ascend NPU, Hygon DCU, and MThreads GPUs, and offers industry-standard APIs for LLM, voice, image, and video models.

Key Features

Multi-Cluster GPU Management across on-premises, Kubernetes, and cloud environments
Pluggable Inference Engines with automated configuration for vLLM, SGLang, TensorRT-LLM, and custom engines
Day 0 Model Support for deploying newly released models immediately
Performance-Optimized Configurations with low-latency and high-throughput modes, LMCache, HiCache, and speculative decoding (EAGLE3, MTP, N-grams)
On-demand SSH-accessible GPU Instances for development, fine-tuning, and interactive workloads
Enterprise-Grade Operations including automated failure recovery, load balancing, monitoring, authentication, and access control
Wide Accelerator Support: NVIDIA GPU, AMD GPU, Ascend NPU, Hygon DCU, MThreads GPU

Pros & Cons

Pros
  • Open-source with no licensing fees
  • Supports a broad range of accelerators beyond NVIDIA GPUs
  • Performance-optimized configurations deliver strong inference performance out of the box
  • Enterprise-grade features like monitoring, authentication, and failure recovery
  • Multi-cluster management enables unified GPU orchestration across different environments
Cons
  • Relatively new project with a smaller community compared to established cluster managers
  • May require significant infrastructure and expertise to set up and maintain
  • Documentation and tutorials are still evolving

Best For

Deploying and scaling LLM, voice, image, and video models as a serviceProvisioning temporary GPU instances for development, fine-tuning, and experimentationManaging GPU resources across hybrid environments (on-premises, cloud, Kubernetes)

FAQ

What inference engines does GPUStack support?
GPUStack automatically configures vLLM, SGLang, and TensorRT-LLM, and allows adding custom inference engines.
What hardware accelerators are supported?
GPUStack supports NVIDIA GPU, AMD GPU, Ascend NPU, Hygon DCU, and MThreads GPU.
Is GPUStack free to use?
Yes, GPUStack is open-source and free to use.
Can I deploy models on the day they are released?
Yes, GPUStack's pluggable engine architecture enables day 0 model support, allowing deployment of newly released models immediately.