Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

Analysis

Why This Paper Matters

Positional embeddings are fundamental to Transformers, encoding token order and distance, but they are typically agnostic to syntactic structure. This paper addresses a critical gap: while syntax has been shown to improve language models, most existing methods either degrade perplexity or require marginalizing over many parses at inference, making them computationally expensive. SiPE offers a lightweight solution that injects syntactic priors directly into positional embeddings, achieving gains on both syntactic and general language understanding benchmarks without sacrificing inference efficiency.

The paper's significance is twofold. First, it systematically explores where and how syntactic information should enter the model, revealing architecture-dependent optimal injection points. This is a valuable insight for the community, as it moves beyond one-size-fits-all approaches. Second, it establishes a new Pareto frontier between syntactic supervision and inference cost, showing that a single parse suffices to achieve strong improvements, which is crucial for practical deployment.

Technical Contributions

• Syntax-informed Positional Embeddings (SiPE): Learns a lightweight syntactic prior from dependency parses during pretraining and injects it into all three dominant PE families (absolute, relative, rotary).
• Architecture-aware injection: For autoregressive decoders with relative PE, the prior is best coupled multiplicatively with the relative-position term in attention scores. For encoders, it is best added directly to input embeddings.
• Minimal architectural change: Leaves self-attention and the rest of the architecture untouched, making it easy to integrate into existing models.
• Single-parse conditioning: Unlike prior work that marginalizes over many parses at inference, SiPE conditions on a single parse, reducing computational overhead.

Results

The paper reports substantial improvements across multiple benchmarks. On SyntaxGym, SiPE improves performance by up to 10.3% over a base model with no syntactic supervision. Simultaneously, it reduces perplexity by 9.0%, a metric that most existing syntax-injection methods degrade. Crucially, these gains extend beyond syntactic generalization: SiPE also improves GLUE scores by up to 8.2%, demonstrating real-world language understanding improvements. These results are consistent across different PE families and architectures, highlighting the robustness of the approach.

Significance

SiPE challenges the assumption that syntactic supervision must come at the cost of general language modeling performance. By showing that a lightweight syntactic prior can be injected into positional embeddings to improve both syntax and general understanding, the paper opens new avenues for incorporating linguistic structure into Transformers. The architecture-dependent findings also provide practical guidance for future research on where to inject auxiliary information. Moreover, the single-parse inference requirement makes SiPE attractive for real-world applications where computational resources are limited. This work could inspire further exploration of syntax-informed positional encodings and their interaction with other architectural components.