Preprint2026
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Haris Riaz, Hyungji Kim, Mihai Surdeanu
SiPE learns a lightweight syntactic prior from dependency parses during pretraining and injects it into positional embeddings, improving syntax and language understanding benchmarks.
0Aug 6, 2026TransformersAttention Mechanisms
arXiv