arXiv Knowledge Base — Specification

Version: 0.1.0 Last updated: 2026-01-31 Python: >=3.10

---

Current Status

Metric	Value
Total Papers	9,500
Legacy Chunks (v1)	12,443
Enriched Chunks (v2)	1,035
Extraction: completed	4,612
Extraction: pending	4,886
Extraction: failed	2
Embedding: completed	185
Embedding: pending	9,315
V2 Chunks by level	summary: 407, section: 628
V2 Contribution types	core: 664, comparison: 134, citation: 237
V2 Key results	97
V2 Embedding status	completed: 966, pending: 69

Top categories: cs.LG (2,791), cs.CL (1,655), cs.AI (1,242), cs.CV (1,107), cs.CR (305), stat.ML (224)

---

Architecture

arxiv_data/
├── config/
│   └── settings.py              # Pydantic Settings (env: ARXIV_KB_*)
├── src/
│   ├── cli.py                   # Typer CLI
│   ├── api/main.py              # FastAPI REST endpoints
│   ├── mcp/server.py            # FastMCP server for Claude
│   ├── collectors/
│   │   └── arxiv_fetcher.py     # arXiv API + PDF downloads
│   ├── extractors/
│   │   ├── pdf_extractor.py     # Docling PDF→Markdown + chunking
│   │   └── entity_extractor.py  # Hybrid entity extraction
│   ├── embedders/
│   │   └── embedder.py          # Sentence-transformers embeddings
│   └── storage/
│       ├── database.py          # SQLite (metadata + FTS5 + chunks)
│       └── vector_store.py      # LanceDB (vector search)
├── scripts/
│   └── migrate_to_v2.py         # v1→v2 migration tool
└── data/
    ├── arxiv.db                 # SQLite (82.8 MB)
    ├── lancedb/                 # Vector database
    ├── pdfs/{YYYY-MM}/          # Downloaded PDFs
    └── markdown/{YYYY-MM}/      # Extracted markdown + JSON metadata

---

Stack

Component	Technology
Collection	arxiv (pip) — official wrapper, rate-limiting
PDF Extraction	Docling (IBM) — GPU-accelerated, Markdown output
Metadata DB	SQLite + FTS5 — full-text search on titles/abstracts
Vector DB	LanceDB — serverless, cosine metric, IVF index
Embeddings	Qwen3-Embedding-0.6B (configurable, 1024-dim)
REST API	FastAPI
MCP Server	FastMCP — Claude integration
CLI	Typer + Rich
Config	Pydantic Settings (env prefix: ARXIV_KB_)

---

Configuration

Path Settings

Setting	Default
data_dir	{base_dir}/data
pdf_dir	{data_dir}/pdfs
markdown_dir	{data_dir}/markdown
sqlite_path	{data_dir}/arxiv.db
lancedb_path	{data_dir}/lancedb

arXiv Settings

Setting	Default
arxiv_categories	["cs.LG", "cs.CL", "cs.AI", "stat.ML"]
arxiv_delay_seconds	3.0
arxiv_page_size	500
arxiv_num_retries	5

Embedding Settings

Setting	Default
embedding_model	Qwen/Qwen3-Embedding-0.6B
embedding_dimension	1024
embedding_batch_size	32

Chunking Settings (v1 — Legacy)

Setting	Default
chunk_size	1024 tokens
chunk_overlap	100 tokens
min_chunk_size	100 tokens

Hierarchical Chunking Settings (v2)

Setting	Default
enable_hierarchical_chunking	True
summary_chunk_max_tokens	512
section_chunk_max_tokens	1024
atomic_chunk_max_tokens	256
atomic_chunk_min_tokens	50

Entity Extraction Settings

Setting	Default
enable_entity_extraction	True
use_llm_for_classification	False
entity_extraction_llm	Qwen/Qwen2.5-3B-Instruct
extract_atomic_facts	True

API & Scheduler

Setting	Default
api_host	0.0.0.0
api_port	8000
sync_hour	6
sync_minute	0

---

Database Schema

papers

Column	Type	Description
id	TEXT PK	arXiv ID (e.g. 2401.12345)
title	TEXT NOT NULL
abstract	TEXT
authors	TEXT	JSON array
categories	TEXT	JSON array
primary_category	TEXT
published_date	TEXT	ISO format
updated_date	TEXT	ISO format
doi	TEXT
journal_ref	TEXT
pdf_url	TEXT
pdf_path	TEXT	Local path
markdown_path	TEXT	Local path
extraction_status	TEXT	pending / completed / failed
embedding_status	TEXT	pending / completed / failed
version	INTEGER	Paper version number
created_at	TEXT
updated_at	TEXT

Indexes: published_date, primary_category, extraction_status, embedding_status FTS5: Virtual table on id, title, abstract, authors with auto-sync triggers

chunks (v1 — Legacy)

Column	Type	Description
id	INTEGER PK AUTO
paper_id	TEXT FK	→ papers.id
chunk_index	INTEGER	Position in sequence
chunk_type	TEXT	abstract, introduction, methodology, results, etc.
content	TEXT NOT NULL
token_count	INTEGER
embedding_id	TEXT	Vector store reference

chunks_v2 (Enriched — Hierarchical)

Column	Type	Description
id	TEXT PK	{paper_id}_{chunk_level}_{index}
paper_id	TEXT FK	→ papers.id
chunk_level	TEXT NOT NULL	summary / section / atomic
section_type	TEXT	abstract, introduction, methodology, results, etc.
content	TEXT NOT NULL
token_count	INTEGER
techniques	TEXT	JSON array: ["MoE", "attention"]
models_mentioned	TEXT	JSON array: ["BERT", "GPT-4"]
benchmarks	TEXT	JSON array: ["GLUE", "MMLU"]
metrics	TEXT	JSON object: {"accuracy": 85.3}
contribution_type	TEXT	core / comparison / baseline / citation
is_key_result	BOOLEAN
embedding_status	TEXT	pending / completed / failed
embedding_id	TEXT
created_at	TEXT
updated_at	TEXT

Indexes: paper_id, chunk_level, contribution_type, is_key_result, embedding_status

sync_state

Single-row table tracking last sync date, last paper ID, papers fetched count, and status (idle/in_progress/completed/failed).

---

Vector Store (LanceDB)

paper_chunks (v1 — Legacy)

Field	Type
id	str ({paper_id}_{chunk_index})
paper_id	str
chunk_index	int
chunk_type	str
content	str
title	str
authors	str
primary_category	str
published_date	str
vector	Vector(1024)

enriched_paper_chunks (v2)

Field	Type
id	str ({paper_id}_{chunk_level}_{index})
paper_id	str
chunk_level	str
section_type	str
content	str
title	str
authors	str
primary_category	str
published_date	str
techniques	str (JSON)
models_mentioned	str (JSON)
benchmarks	str (JSON)
metrics	str (JSON)
contribution_type	str
is_key_result	bool
vector	Vector(1024)

Index config: Cosine metric, 256 partitions, 96 sub-vectors (IVF)

---

Data Pipeline

1. SYNC (arxiv_fetcher.py)
   arXiv API → papers table + PDFs to data/pdfs/{YYYY-MM}/
   Status: extraction_status = "pending"

2. EXTRACT (pdf_extractor.py)
   PDF → Docling → Markdown → Sections → Chunks
   Saves: data/markdown/{YYYY-MM}/{id}.md + {id}.json
   v1: chunks table (flat)
   v2: chunks_v2 table (hierarchical + entities)
   Status: extraction_status = "completed"

3. EMBED (embedder.py)
   Chunks → Qwen3-Embedding-0.6B → LanceDB
   v1: paper_chunks table
   v2: enriched_paper_chunks table
   Status: embedding_status = "completed"

4. SEARCH
   Query → Embed → Vector similarity + metadata filters → Results

---

Hierarchical Chunking Strategy (v2)

Level 1: Summary (1 per paper)

• Abstract + first paragraph of introduction
• Entities extracted from abstract
• Always classified as core contribution
• Used for overview/discovery queries

Level 2: Section (variable per paper)

• Split by semantic boundaries (tables, figures, subsections, bold headers)
• Each section chunk gets independent entity extraction
• Contribution type classified per chunk
• Results sections split by experiment/benchmark
• Methodology sections split by component/step

Level 3: Atomic (variable per paper)

• Fine-grained facts: "DeltaNet achieves 85.3% on GLUE"
• Only generated from results/methodology sections
• Only when entities (models, benchmarks, metrics) are present
• Each fact links model + benchmark + metric

---

Entity Extraction

Approach: Hybrid (Rules + Patterns + Optional LLM)

1. Known entities — curated lists matched via compiled regex:

   • 100+ benchmarks (GLUE, ImageNet, MMLU, ...)
   • 150+ models (BERT, GPT-4, LLaMA, ...)
   • 100+ techniques (attention, MoE, LoRA, ...)

2. Metrics — regex patterns for 20+ metric types:

   • accuracy, F1, BLEU, ROUGE, perplexity, mAP, WER, FLOPs, etc.

3. Contribution classification — rule-based (optional LLM):

   • core: "we propose", "our method", "novel", methodology sections
   • comparison: "compared to", "outperforms", "baseline"
   • baseline: baseline references
   • citation: related work, default

4. Key result detection:

   • "state-of-the-art", "SOTA", "new record"
   • Chunks with metrics + core/comparison type

---

CLI Commands

Command	Description
arxiv-kb sync [--days N] [--incremental]	Fetch papers from arXiv
arxiv-kb extract [PAPER_ID] [--all] [--limit N]	Extract text from PDFs
arxiv-kb embed [PAPER_ID] [--all] [--limit N]	Generate embeddings
arxiv-kb search QUERY [--fulltext] [--cat CS.LG]	Search papers
arxiv-kb stats	Show database statistics
arxiv-kb paper PAPER_ID	Show paper details
arxiv-kb serve [--port 8000]	Start FastAPI server
arxiv-kb mcp	Start MCP server
arxiv-kb pipeline [--days 7] [--limit 100]	Full sync→extract→embed

---

REST API Endpoints

Search

Method	Path	Description
POST	/search	Semantic search (query, categories, date range, limit)
POST	/search/hybrid	Hybrid semantic + FTS (vector_weight: 0-1)
GET	/search/fulltext?query=...	Full-text search on titles/abstracts

Papers

Method	Path	Description
GET	/papers/{id}	Paper metadata
GET	/papers/{id}/chunks	All text chunks
GET	/papers/{id}/markdown	Extracted markdown text
GET	/papers/{id}/pdf	Download PDF file
GET	/papers/{id}/similar?limit=10	Find similar papers
POST	/papers/filter	Filter by categories, authors, dates

Browse

Method	Path	Description
GET	/recent?days=7&categories=cs.LG	Recent papers
GET	/authors/{name}	Papers by author

Metadata

Method	Path	Description
GET	/	API info
GET	/health	Health check
GET	/stats	Database statistics

---

MCP Tools (Claude Integration)

Legacy Tools (v1)

Tool	Args	Description
search_papers	query, categories?, limit	Semantic search
get_paper_details	paper_id, include_full_text?	Paper metadata + optional text
find_similar_papers	paper_id, limit	Similar papers
search_by_author	author_name, limit	Papers by author
get_recent_papers	days, categories?, limit	Recent papers
fulltext_search	query, limit	Keyword-based search
get_paper_chunks	paper_id	All chunks for a paper
get_knowledge_base_stats	—	DB statistics

Enriched Tools (v2)

Tool	Args	Description
search_model_results	model_name, benchmark?, include_baselines	Find benchmark results for a model
find_innovations	topic, techniques?, days, limit	Core contributions on a topic
compare_models	models (comma-sep), benchmark?	Compare model performance
search_by_technique	technique, contribution_type, limit	Papers by technique
get_key_results	query, limit	SOTA / significant results
get_paper_entities	paper_id	Aggregated entities for a paper
advanced_search	query, models?, benchmarks?, techniques?, contribution_type?, chunk_level?	Multi-filter search

---

Migration (v1 → v2)

python scripts/migrate_to_v2.py status           # Check progress
python scripts/migrate_to_v2.py init              # Create v2 tables
python scripts/migrate_to_v2.py migrate           # Migrate all papers
python scripts/migrate_to_v2.py migrate --limit 100
python scripts/migrate_to_v2.py migrate --paper-id 2401.12345
python scripts/migrate_to_v2.py migrate --no-skip # Re-migrate existing

The migration re-reads existing markdown files (no re-extraction needed), applies hierarchical chunking + entity extraction, generates new embeddings, and stores in both SQLite chunks_v2 and LanceDB enriched_paper_chunks.

---

Dependencies

[project]
dependencies = [
    "arxiv>=2.1.0",
    "httpx>=0.27.0",
    "aiofiles>=24.1.0",
    "marker-pdf>=1.0.0",
    "lancedb>=0.10.0",
    "sentence-transformers>=3.0.0",
    "fastapi>=0.115.0",
    "uvicorn>=0.30.0",
    "fastmcp>=0.3.0",
    "apscheduler>=3.10.0",
    "typer>=0.12.0",
    "rich>=13.0.0",
    "pydantic-settings>=2.0.0",
    "python-dotenv>=1.0.0",
]

---

File Storage Layout

data/
├── arxiv.db                          # SQLite (82.8 MB)
│   ├── papers (9,500 rows)
│   ├── chunks (12,443 rows)
│   ├── chunks_v2 (1,035 rows)
│   ├── papers_fts (FTS5 index)
│   └── sync_state (1 row)
├── lancedb/
│   ├── paper_chunks/                 # v1 embeddings (521 vectors)
│   └── enriched_paper_chunks/        # v2 embeddings (966 vectors)
├── pdfs/{YYYY-MM}/{paper_id}.pdf     # Original PDFs
└── markdown/{YYYY-MM}/
    ├── {paper_id}.md                 # Extracted text
    └── {paper_id}.json               # Section metadata