Self-Adaptive LLMs
Qi Sun, Edoardo Cetin, Yujin Tang
Transformer-Squared is a self-adaptive LLM framework that adjusts only the singular components of weight matrices in real-time, using a two-pass dispatch and reinforcement-learned expert vectors to outperform LoRA with fewer parameters.