femtoGPT

Free

Pure Rust implementation of a minimal Generative Pretrained Transformer.

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (38 stars)

About femtoGPT

femtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer (GPT), built entirely from scratch including tensor processing logic, training, and inference code. It is designed for both training and inference of GPT-style language models on CPUs and GPUs (via OpenCL, supporting both NVIDIA and AMD cards). The implementation uses only basic Rust libraries (rand, serde, rayon) and verifies gradient correctness via gradient checking. While it is relatively slow on CPU and produces low-quality outputs on small models, it serves as an excellent educational tool for understanding how large language models work at a deep level. The project includes a simple CLI for training on custom datasets and running inference, with the ability to save and resume training.

Key Features

Pure Rust implementation from scratch, including tensor processing logic

Supports both training and inference of GPT-style language models

GPU acceleration via OpenCL (works on NVIDIA and AMD without CUDA)

Gradient checking to verify correctness of backpropagation

CPU training/inference with parallel computing (rayon)

Model saving and loading using serde and bincode

Simple CLI: train with `cargo run --release -- train`, infer with `cargo run --release -- infer`

Designed for small datasets (e.g., Shakespeare text with 65 unique characters)

Educational focus: helps understand LLMs at a deep level

Open-source with MIT license

Pros & Cons

Pros

Completely self-contained implementation with minimal dependencies
GPU support via OpenCL without needing CUDA toolkits
Gradient checking ensures training correctness
Great educational resource for understanding GPT internals
Supports saving and resuming training
Free and open-source

Cons

Extremely slow on CPU (as stated by the author)
Output quality is poor for small models (e.g., 300k parameters on Shakespeare)
Some layers may still have implementation bugs despite gradient checking
Limited to small datasets with few unique characters
Not suitable for production use

Best For

Learning how transformers and GPT models work at a low levelTraining small language models on custom text datasetsExperimenting with GPT architecture without heavy dependenciesTeaching or demonstrating neural network implementation in Rust

FAQ

What is femtoGPT?

femtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer (GPT) that can be used for both inference and training of GPT-style language models using CPUs and GPUs.

How do I train a model with femtoGPT?

Place your training text in dataset.txt (ensure it has a small number of unique characters), then run `cargo run --release -- train`.

Does femtoGPT support GPU acceleration?

Yes, femtoGPT supports GPU acceleration via OpenCL. You need to install OpenCL runtimes on your system and use the `--features gpu` flag when running. It works on both NVIDIA and AMD cards.

Is femtoGPT fast?

No, femtoGPT is extremely slow on CPU, as most primitive operations are implemented in the simplest way possible. GPU acceleration can help but performance is still limited.

What is the output quality like?

With small models (e.g., 300k parameters on Shakespeare), outputs are embarrassingly bad but show some ability to generate pronounceable words. The author notes it may improve with larger models (10M parameters).

What dependencies does femtoGPT use?

femtoGPT uses only random generation libraries (rand), data-serialization libraries (serde/bincode for model saving/loading), and a parallel computing library (rayon). It does not use any deep learning frameworks.