Parameter-efficient fine-tuning with discrete fourier transform

Analysis

Why This Paper Matters

Parameter-efficient fine-tuning (PEFT) has become essential for adapting large pre-trained models without the prohibitive cost of full fine-tuning. Existing methods like LoRA and Adapters operate in the spatial domain, learning low-rank matrices or small adapter layers. FourierFT breaks new ground by shifting the learning process into the frequency domain via the discrete Fourier transform (DFT). This is significant because it opens up a new design space for PEFT, where the inductive bias of smoothness in weight updates can be exploited to achieve high efficiency.

The paper is particularly relevant as model sizes continue to grow, making even LoRA's rank-8 updates memory-intensive for billion-parameter models. By learning only a handful of Fourier coefficients, FourierFT drastically reduces the number of trainable parameters while maintaining or improving performance. This could democratize fine-tuning for practitioners with limited computational resources.

Technical Contributions

• Frequency-domain parameterization: Instead of learning weight updates directly, FourierFT learns a sparse set of coefficients in the frequency domain. The actual update is obtained via inverse DFT, which naturally enforces smoothness and reduces the number of free parameters.
• Theoretical justification: The authors show that learning in the frequency domain is equivalent to imposing a low-frequency bias, which aligns with the observation that fine-tuning often produces smooth weight changes.
• Efficient implementation: The method uses fast Fourier transform (FFT) algorithms, making the forward and backward passes computationally efficient even for large matrices.
• Compatibility: FourierFT can be applied to any linear layer in a transformer, similar to LoRA, and can be combined with other PEFT techniques.

Results

On the GLUE benchmark, FourierFT with only 0.1% of total parameters (e.g., 0.1M for BERT-base) achieves an average score of 83.5, compared to 83.2 for full fine-tuning and 82.9 for LoRA (rank=8). On SuperGLUE, FourierFT reaches 84.1 with 0.15% parameters, outperforming LoRA's 83.5 with 0.5% parameters. The method also shows strong performance on text classification and question answering tasks. Ablation studies confirm that learning only the lowest-frequency components (e.g., top 1% of Fourier coefficients) is sufficient for most tasks.

Significance

FourierFT challenges the prevailing low-rank assumption in PEFT by demonstrating that frequency-domain sparsity is an equally powerful inductive bias. This could lead to a new family of spectral PEFT methods that are both parameter-efficient and computationally lightweight. For AI practitioners, FourierFT offers a practical tool for adapting large models on modest hardware, potentially enabling fine-tuning of models with hundreds of billions of parameters on a single GPU. The work also bridges signal processing and deep learning, encouraging cross-pollination between these fields.