Tiny-vLLM Inference Engine and Course Released on GitHub

A developer known as jmaczan has released tiny-vllm, a high-performance LLM inference engine built in C++ and CUDA. The repository contains both the full source code of the inference server and a course that guides readers through the implementation process. The project is described as a younger, smaller sibling of vLLM, the popular inference engine.

The engine is designed to load a real LLM model from Safetensors format, specifically Llama 3.2 1B Instruct. It performs a full LLM forward pass including prefill and decode, with all computation handled through CUDA kernels. It implements KV cache, static batching, continuous batching, online softmax, a FlashAttention-like approach, and PagedAttention.

The author positions the project as a learning tool. It is meant for individual learners who want to understand the internals of LLM inference, as well as for lecturers who might use it as a teaching resource at universities.

Understanding LLM Inference

The course begins by explaining what an LLM is and why an inference engine is needed. Physically, an LLM is a file containing many floating-point numbers representing weights. These weights are learned during training. The architecture defines the operations and their order. To run the model, a program must turn that architecture into executable code and load the weights at runtime. That program is the inference server.

The author explains why C++ and CUDA are used: to maximize hardware efficiency and achieve high performance. CUDA allows writing code that runs on GPUs, which are better suited for the matrix multiplications that dominate LLM computation.

The course also provides a broader context for AI and computation. The author suggests that intelligence emerges from a large number of model parameters and extensive computation, not from any single component. The default attention mechanism is noted for its high computational complexity (O(n^2*d)), and the author mentions alternative mechanisms like linear attention as a potential future topic.

Technical Foundations

The course details the technical prerequisites. It can be built and run on any platform with an NVIDIA GPU, with minor adjustments to paths. The author develops and tests on Linux with CUDA Toolkit 13.1, C++17, GCC 15.2.1, an AMD Ryzen 7 9800X3D CPU, and an RTX 5090 GPU. The only external dependency is the nlohmann/json single-header JSON parser.

A significant portion of the course is dedicated to understanding the Safetensors format. A safetensor file has three sections: header size (8 bytes), a JSON header, and tensor data. The header contains metadata for each tensor including dtype, shape, and offsets. The author advises that it is easier to start by hardcoding for one specific model architecture (Llama 3.2 1B) rather than trying to make the engine model-independent from the beginning.

Floating-Point Number Representations

The course includes a deep dive into floating-point numbers, specifically float16 and bfloat16. The author explains the structure: sign, exponent, and fraction bits. For float16, the bias is 15, and the implicit leading 1 in the mantissa increases precision without using extra memory. An example encoding of the number 12.34 into float16 is worked through step by step, showing how precision is lost (the stored value becomes 12.3359375).

The author then discusses bfloat16, which is widely used in models like Llama 3.2 1B. Bfloat16 also takes 16 bits but allocates 8 bits to the exponent (same as float32) and only 7 bits to the fraction. This tradeoff reduces the risk of overflow or underflow at the cost of less precision, which is acceptable for LLM inference.

The course also touches on GPU and CPU memory management, explaining that data can live on the host (CPU DRAM) or the device (GPU HBM/VRAM). The author promises hands-on primer on CUDA memory operations.

The repository is available on GitHub under jmaczan/tiny-vllm. The author encourages forking, contributing platform-specific adjustments via pull requests, and opening issues for build problems.

Related on Neura Market

• AI Models, Explore other open-source LLM inference engines and model frameworks.
• AI Tools Directory, Find tools for building and deploying AI applications.
• Developer Resources, Learn about CUDA programming and GPU optimization techniques.