Preprint2025
SigLIP 2
Unknown
SigLIP 2 improves multilingual vision-language encoders with captioning pretraining, self-supervised losses, and online data curation, offering native aspect ratio preservation.
0Feb 1, 2025Computer Vision
AI Research
AI & ML Research Papers
A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
All Papers
8 papersPreprint2020
SimCLRv2
Unknown
SimCLRv2 presents a semi-supervised learning framework combining unsupervised pretraining, supervised fine-tuning, and distillation with unlabeled data.
0Jun 1, 2020Fine Tuning
Preprint2025
UltraLong
Unknown
Introduces a training method for developing ultra-long context LLMs with context windows extending up to 4 million tokens via efficient continued pretraining with YaRN-based scaling and instruction tuning.
0Apr 1, 2025Scaling LawsInstruction Tuning
PreprintInternational Conference on Learning Representations2023
Llemma
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, et al.
Llemma is an open-source LLM for mathematics, built by continued pretraining of Code Llama on a curated dataset of scientific papers, web math, and code, enabling tool use and formal theorem proving.
479Oct 1, 2023
arXiv
Preprint2023
LLaMA 2 Long
Unknown
LLaMA 2 Long extends LLaMA 2 to handle up to 32,768 tokens via continual pretraining with modified RoPE and synthetic instruction data.
0Sep 1, 2023
arXiv
PreprintNeural Information Processing Systems2019Featured
XLNet
Zhilin Yang, Zihang Dai, Yiming Yang, et al.
XLNet combines autoregressive and autoencoding pretraining via permutation language modeling, outperforming BERT on NLP tasks.
9.5kJun 1, 2019Transformers
arXiv
Preprint2023
The learnability of in-context learning
Unknown
Proves that in-context learning is guaranteed to emerge from multi-task pretraining under specific theoretical conditions.
0Jan 1, 2023In Context Learning
Preprintnpj Digital Medicine2021
Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction
Laila Rasmy, Yang Xiang, Ziqian Xie, et al.
Med-BERT adapts BERT to structured EHR data via pretraining on 28.5M patients, boosting disease prediction AUC by 1.21-6.14% and enabling small-dataset models to match those trained on tenfold larger data.
1.0kMay 20, 2021TransformersFine Tuning
arXiv