Benchmarks
Presents top-1 classification error rates for six learning methods on MNIST, Fashion-MNIST, CIFAR-10, and ImageNet using specific architectures and hyperparameters.
What this file does
Presents top-1 classification error rates for six learning methods on MNIST, Fashion-MNIST, CIFAR-10, and ImageNet using specific architectures and hyperparameters.
When to use it
- Comparing feedback alignment variants against backpropagation on standard benchmarks
- Reproducing the exact experimental setup for LeNet, ResNet-20, or ResNet-56
- Reporting baseline results in a paper or technical report on alternative gradient methods
- Validating your own implementation against published error rates
Assumes this stack
Benchmarks
MNIST & Fashion-MNIST
The training set is split into a 60k training and 10k validation partitions. The model with best validation accuracy is then benchmarked with the same validation set of 10k samples. The model used to compare is LeNet MNIST. Networks were trained with the Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9, and weight decay of 10^-3. Images were resized to 32x32 prior to being input to the network. We trained with a batch size of 64 for 100 epochs in one GPU. We decreased the initial learning rate by a factor of 10 at the 50th and the 75th epoch. The Top-1 Classification Error Rate is shown in the table.
| Method | MNIST | Fashion MNIST |
|---|---|---|
| BP | 0.91 | 9.2 |
| FA | 1.7 | 13.06 |
| uSF | 0.94 | 9.69 |
| brSF | 0.91 | 10.02 |
| frSF | 0.97 | 9.61 |
| DFA | 1.61 | 12.81 |
CIFAR 10
The training set is split into a 45k training and 5k validation partitions. The model with best validation accuracy is then benchmarked with the testing set of 10k samples as in He, Kaiming, et al.. The models used to compare are LeNet CIFAR10, ResNet-20 and ResNet-56. The configuration files attached contain the exact hyperparameters used per method. The Top-1 Classification Error Rate is shown in the table.
| Method | LeNet | LeNet (Adam) | ResNet-20 | ResNet-20 (Adam) | ResNet-56 (SGD) | ResNet-56 (Adam) |
|---|---|---|---|---|---|---|
| BP | 14.23 | 15.92 | 8.63 | 10.01 | 8.3 | 7.83 |
| FA | 46.69 | 40.67 | 32.16 | 29.59 | 34.88 | 29.23 |
| DFA | 54.21 | 37.59 | 45.94 | 32.16 | 38.01 | 32.02 |
| uSF | 16.22 | 16.34 | 10.05 | 10.59 | 8.2 | 9.19 |
| brSF | 16.02 | 17.08 | 11.02 | 11.08 | 8.69 | 10.13 |
| frSF | 16.86 | 16.83 | 11.2 | 11.22 | 9.49 | 10.02 |
ImageNet
A ResNet-18 network is trained with a batch size of 256 and 2 GPUs for 75 epochs using SGD with a initial learning rate of 0.1. A scheduler decreased the learning rate by a factor of 10 at the 20th, the 40th and the 60th epoch. We used a weight decay of 10^-4 and a momentum of 0.9. For DFA we used Adam with an initial learning rate of 0.001. At training time, a random resized crop of dimensions 224x224 of the original image or its horizontal flip with the per-pixel mean subtracted is used. When testing, the image is resized to 256x256 and then a center crop of 224x224 is used as input to the network.
| Method | ResNet-18 |
|---|---|
| BP | 30.39 |
| FA | 85.25 |
| DFA | 82.45 |
| uSF | 34.97 |
| brSF | 37.21 |
| frSF | 36.5 |
What's inside
Three benchmark sections (MNIST/Fashion-MNIST, CIFAR-10, ImageNet) each with training details and a results table.
Change this for your project
- Replace
https://github.com/jsalbert/biotorch/blob/main/biotorch/models/le_net.py#L6with your own model source - Replace
https://github.com/jsalbert/biotorch/blob/main/biotorch/models/small_resnet.py#L115with your own model source - Replace
https://github.com/jsalbert/biotorch/blob/main/biotorch/models/small_resnet.py#L127with your own model source
Where it goes
Keep it in your repository where the agent or team that needs it will read it.
Worth borrowing
- Splitting training data into train/validation partitions and selecting the best validation model for final testing
- Using a learning rate schedule that drops by factor 10 at fixed epoch milestones
Related Documents
Ruby 2.7
Documents every language, core class, and standard library change in Ruby 2.7 with code examples and rationale.
OABench: Benchmarking Large Language Models on the Brazilian Bar Examination
Evaluates 11 LLMs on the Brazilian Bar Exam's first phase, reporting accuracy, cost, and latency across three exam editions.
Prometheus Automation AI Marketplace - Project Documentation
Documents an enterprise AI marketplace built with Next.js 15, covering architecture, AI algorithms, security, and deployment.
Benchmarks
Compares Okra's read/write performance against raw LMDB across three dataset sizes using small key-value pairs.