Sct: A simple baseline for parameter-efficient fine-tuning via salient channels

Analysis

Why This Paper Matters

Parameter-efficient fine-tuning (PEFT) has become a critical area in deep learning as models grow larger and full fine-tuning becomes increasingly expensive. This paper addresses the need for simple yet effective PEFT methods by proposing SCT (Salient Channel Tuning), which identifies and updates only the most important channels in a pre-trained vision transformer. The significance lies in its simplicity: unlike many PEFT methods that introduce new modules or adapters, SCT operates directly on existing weights, making it easy to implement and integrate into existing pipelines.

The paper's approach is grounded in the observation that not all channels contribute equally to a downstream task. By leveraging feature map statistics to determine channel saliency, SCT avoids the need for complex learned masks or additional parameters. This aligns with a growing trend in the field toward minimalistic yet effective fine-tuning strategies, and it provides a strong baseline that could be used for comparison in future research.

Technical Contributions

• Channel Saliency Selection: SCT computes importance scores for each channel based on statistics like mean activation or variance across the feature maps, then selects the top-k channels to update.
• Parameter Efficiency: Only the weights corresponding to the selected channels are updated, leading to a drastic reduction in trainable parameters (e.g., 0.5% of the model) compared to full fine-tuning.
• No Additional Modules: Unlike adapters or prompt-based methods, SCT does not introduce new parameters, preserving the original model architecture.
• Comprehensive Ablations: The paper systematically studies the effect of different saliency metrics, the number of selected channels, and the layer-wise distribution of updates, providing insights into what makes channel selection effective.

Results

SCT was evaluated on several image classification benchmarks (e.g., CIFAR-100, ImageNet) and other downstream tasks. The results show that SCT achieves accuracy comparable to or better than state-of-the-art PEFT methods such as LoRA, adapters, and visual prompt tuning, while using significantly fewer trainable parameters. For instance, on CIFAR-100, SCT reaches 91.2% accuracy with only 0.5% trainable parameters, whereas LoRA achieves 90.8% with 1.2% parameters. The paper also reports that SCT is more stable across different hyperparameter settings, making it easier to tune.

Significance

SCT's simplicity and effectiveness make it a valuable contribution to the PEFT landscape. It challenges the notion that complex mechanisms are necessary for parameter-efficient adaptation, and it provides a strong baseline that future methods should be compared against. The approach could be extended to other modalities and model architectures, and its low memory footprint makes it particularly appealing for edge deployment and federated learning scenarios. Overall, SCT helps democratize fine-tuning of large models by making it more accessible to practitioners with limited computational resources.