A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
Essential AI Darsh J Shah, Peter Rushton, Somanshu Singla, et al.
This paper shows that language models' self-reflection ability emerges during pre-training, not just reinforcement learning, by introducing deliberate errors into chains-of-thought.
Qingxiu Dong, Li Dong, Yao Tang, et al.
Introduces Reinforcement Pre-Training (RPT), reframing next-token prediction as a reasoning task trained with RL and verifiable rewards, improving language modeling accuracy and providing a strong foundation for RL fine-tuning.
Unknown
This paper demonstrates that tabular foundation models can achieve generalization through in-context learning from a single table, without requiring pre-training on multiple tables.
Unknown
This paper explores knowledge pre-training for tabular foundation models, aiming to improve performance on small to moderate-sized tabular data.
Unknown
This paper investigates how pre-training can be leveraged for scientific machine learning, focusing on transfer learning behavior and scaling characteristics.
Bonan Min, Hayley Ross, Elior Sulem, et al.
This survey reviews the shift to PLM-driven NLP, covering pre-training, fine-tuning, prompting, and text generation, and discusses limitations and future directions.
Charlie Zhang, Graham Neubig, Xiang Yue
A controlled experimental framework isolates causal contributions of pre-training, mid-training, and RL to reasoning LM generalization.
Ioannis Maniadis Metaxas, Adrian Bulat, Alberto Baldrati, et al.
UltraViT is a latency-optimized vision encoder for LVLMs using a pyramidal architecture with heterogeneous spatial mixers and a two-stage generative pre-training strategy, achieving 1.7x speedup on-device.
Shuo Xing, Junyuan Hong, Yifan Wang, et al.
Shows continual pre-training on junk web text causes lasting cognitive decline in LLMs, with dose-response effects and partial healing.
Zorik Gekhman, G. Yona, Roee Aharoni, et al.
This paper shows that supervised fine-tuning introducing new factual knowledge to LLMs is learned slowly and linearly increases hallucination, supporting that factual knowledge is acquired during pre-training.
Unknown
CLIMB automates optimal data mixture discovery for language model pre-training via clustering and iterative bootstrapping.
Unknown
Instruction Pre-Training augments raw corpora with instruction-response pairs for supervised multitask pretraining of language models.