LongCite

Analysis

Why This Paper Matters

Long-context language models are increasingly used for tasks like document analysis, summarization, and question answering over large corpora. However, their outputs often lack verifiability—users cannot easily trace claims back to source material. LongCite directly addresses this by teaching models to generate fine-grained, sentence-level citations alongside their answers. This is a significant step toward trustworthy AI, as it allows practitioners and end-users to validate model outputs, reducing the risk of hallucination and increasing transparency. The paper also provides a comprehensive benchmark (LongBench-Cite) and a large-scale dataset (LongCite-45k), enabling the community to systematically improve and evaluate citation capabilities.

How It Works

The core innovation is the CoF (Coarse to Fine) pipeline, which automates the creation of high-quality training data for long-context QA with citations. The pipeline starts with a long document and uses an LLM to generate a query and answer via Self-Instruct, guided by diverse task descriptions (e.g., summarization, multi-hop reasoning). Next, it retrieves relevant chunks from the document using a retriever (Zhipu Embedding-v2) and asks the LLM to add coarse chunk-level citations. Finally, the LLM refines these to sentence-level citations by extracting supporting sentences from the cited chunks. Low-quality instances (where fewer than 20% of statements have citations) are discarded.

This pipeline was applied to 50k documents from GLM-4's pre-training corpus, covering nine domains (books, encyclopedias, academic papers, code, etc.) in English and Chinese, with lengths from 256 to 128k tokens. After filtering, 44,600 high-quality instances formed the LongCite-45k dataset. The authors then fine-tuned two base models—GLM-4-9B and Llama-3.1-8B—on this dataset combined with 76k general SFT instances from ShareGPT to preserve general capabilities. The resulting models are LongCite-9B and LongCite-8B.

To isolate the effect of citation training, the authors also trained control models (LongSFT-9B and LongSFT-8B) on the same QA pairs but without citation instructions. This allowed them to compute a correctness ratio (CR) comparing citation-augmented vs. vanilla long-context QA performance.

Results

On the LongBench-Cite benchmark, LongCite-8B and LongCite-9B achieved the highest citation F1 scores among all evaluated models, including proprietary ones like GPT-4. Their citations were also significantly shorter (lower citation length), indicating finer granularity. Crucially, the correctness ratio (CR) exceeded 100% for both LongCite models, meaning that adding citations actually improved answer accuracy compared to vanilla long-context QA. This suggests that the citation generation process forces the model to ground its responses more carefully in the source text. The models also matched the correctness of officially post-trained versions (GLM-4-9B-chat and Llama-3.1-8B-Instruct), showing no degradation in general QA ability.

Significance

LongCite provides a practical, scalable solution for making long-context LLMs more trustworthy. By releasing the benchmark, dataset, and trained models, the authors enable the community to build on this work. The CoF pipeline is particularly valuable because it automates the creation of high-quality training data, reducing the need for expensive human annotation. The results demonstrate that citation generation can be integrated into a single forward pass without sacrificing accuracy, making it feasible for real-world deployment. Future work could extend this approach to more languages, longer contexts, and more complex citation formats (e.g., multiple citations per statement). For AI practitioners, LongCite offers a clear path to improving model transparency and user trust in applications like legal document review, medical literature analysis, and research paper summarization.