Arxiv Research Skill: Search, Fetch, and Analyze Academic Papers

The Arxiv skill bundled with Hermes Agent lets you search and retrieve academic papers from the arXiv preprint server using its free REST API. No API key, no extra dependencies, just curl and Python's standard library. You would reach for this when you need to quickly find papers by keyword, author, category, or ID, read abstracts or full PDFs, and generate BibTeX citations, all from the command line or within an agent workflow.

What it does

This skill wraps arXiv's public API into a set of curl commands and a helper Python script. You can:

• Search papers across all fields, titles, authors, abstracts, or categories.
• Fetch a specific paper by its arXiv ID, or multiple papers at once.
• Parse the Atom XML response into a clean, readable format.
• Generate BibTeX entries from paper metadata.
• Read abstracts via web_extract on the abstract page URL.
• Read full papers via web_extract on the PDF URL.
• Optionally augment results with citation counts, related papers, and author profiles from the Semantic Scholar API.

The skill is designed for researchers, students, and anyone who needs to survey literature programmatically without signing up for a third-party service.

Before you start

• Platforms: Linux, macOS, Windows.
• Dependencies: None beyond what Hermes Agent ships. The helper script uses only Python stdlib (xml.etree.ElementTree).
• API keys: None required for either arXiv or Semantic Scholar basic usage. Semantic Scholar allows 1 request per second without a key, 100 per second with a free API key.
• Rate limits: arXiv enforces roughly 1 request per 3 seconds. Semantic Scholar limits to 1 request per second without auth.
• Skill path: skills/research/arxiv.
• Related skill: ocr-and-documents for local PDF processing.

Searching Papers

The API returns Atom XML. You can parse it with grep/sed or pipe through python3 for clean output.

Basic search

curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5"

Clean output (parse XML to readable format)

curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5&sortBy=submittedDate&sortOrder=descending" | python3 -c "
import sys, xml.etree.ElementTree as ET
ns = {'a': 'http://www.w3.org/2005/Atom'}
root = ET.parse(sys.stdin).getroot()
for i, entry in enumerate(root.findall('a:entry', ns)):
    title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
    arxiv_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
    published = entry.find('a:published', ns).text[:10]
    authors = ', '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
    summary = entry.find('a:summary', ns).text.strip()[:200]
    cats = ', '.join(c.get('term') for c in entry.findall('a:category', ns))
    print(f'{i+1}. [{arxiv_id}] {title}')
    print(f'   Authors: {authors}')
    print(f'   Published: {published} | Categories: {cats}')
    print(f'   Abstract: {summary}...')
    print(f'   PDF: https://arxiv.org/pdf/{arxiv_id}')
    print()
"

Search Query Syntax

Prefix	Searches	Example
all:	All fields	all:transformer+attention
ti:	Title	ti:large+language+models
au:	Author	au:vaswani
abs:	Abstract	abs:reinforcement+learning
cat:	Category	cat:cs.AI
co:	Comment	co:accepted+NeurIPS

Boolean operators

# AND (default when using +)
search_query=all:transformer+attention

# OR
search_query=all:GPT+OR+all:BERT

# AND NOT
search_query=all:language+model+ANDNOT+all:vision

# Exact phrase
search_query=ti:"chain+of+thought"

# Combined
search_query=au:hinton+AND+cat:cs.LG

Sort and Pagination

Parameter	Options
sortBy	relevance, lastUpdatedDate, submittedDate
sortOrder	ascending, descending
start	Result offset (0-based)
max_results	Number of results (default 10, max 30000)

# Latest 10 papers in cs.AI
curl -s "https://export.arxiv.org/api/query?search_query=cat:cs.AI&sortBy=submittedDate&sortOrder=descending&max_results=10"

Fetching Specific Papers

# By arXiv ID
curl -s "https://export.arxiv.org/api/query?id_list=2402.03300"

# Multiple papers
curl -s "https://export.arxiv.org/api/query?id_list=2402.03300,2401.12345,2403.00001"

BibTeX Generation

After fetching metadata for a paper, generate a BibTeX entry:

{% raw %}

curl -s "https://export.arxiv.org/api/query?id_list=1706.03762" | python3 -c "
import sys, xml.etree.ElementTree as ET
ns = {'a': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
root = ET.parse(sys.stdin).getroot()
entry = root.find('a:entry', ns)
if entry is None: sys.exit('Paper not found')
title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
authors = ' and '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
year = entry.find('a:published', ns).text[:4]
raw_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
cat = entry.find('arxiv:primary_category', ns)
primary = cat.get('term') if cat is not None else 'cs.LG'
last_name = entry.find('a:author', ns).find('a:name', ns).text.split()[-1]
print(f'@article{{{last_name}{year}_{raw_id.replace(\".\", \"\")},')
print(f'  title     = {{{title}}},')
print(f'  author    = {{{authors}}},')
print(f'  year      = {{{year}}},')
print(f'  eprint    = {{{raw_id}}},')
print(f'  archivePrefix = {{arXiv}},')
print(f'  primaryClass  = {{{primary}}},')
print(f'  url       = {{https://arxiv.org/abs/{raw_id}}}')
print('}')
"

{% endraw %}

Reading Paper Content

After finding a paper, read it:

# Abstract page (fast, metadata + abstract)
web_extract(urls=["https://arxiv.org/abs/2402.03300"])

# Full paper (PDF → markdown via Firecrawl)
web_extract(urls=["https://arxiv.org/pdf/2402.03300"])

For local PDF processing, see the ocr-and-documents skill.

Common Categories

Category	Field
cs.AI	Artificial Intelligence
cs.CL	Computation and Language (NLP)
cs.CV	Computer Vision
cs.LG	Machine Learning
cs.CR	Cryptography and Security
stat.ML	Machine Learning (Statistics)
math.OC	Optimization and Control
physics.comp-ph	Computational Physics

Full list: https://arxiv.org/category_taxonomy

Helper Script

The scripts/search_arxiv.py script handles XML parsing and provides clean output:

python scripts/search_arxiv.py "GRPO reinforcement learning"
python scripts/search_arxiv.py "transformer attention" --max 10 --sort date
python scripts/search_arxiv.py --author "Yann LeCun" --max 5
python scripts/search_arxiv.py --category cs.AI --sort date
python scripts/search_arxiv.py --id 2402.03300
python scripts/search_arxiv.py --id 2402.03300,2401.12345

No dependencies, uses only Python stdlib.

Semantic Scholar (Citations, Related Papers, Author Profiles)

arXiv doesn't provide citation data or recommendations. Use the Semantic Scholar API for that, free, no key needed for basic use (1 req/sec), returns JSON.

Get paper details + citations

# By arXiv ID
curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300?fields=title,authors,citationCount,referenceCount,influentialCitationCount,year,abstract" | python3 -m json.tool

# By Semantic Scholar paper ID or DOI
curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1234/example?fields=title,citationCount"

Get citations OF a paper (who cited it)

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/citations?fields=title,authors,year,citationCount&limit=10" | python3 -m json.tool

Get references FROM a paper (what it cites)

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/references?fields=title,authors,year,citationCount&limit=10" | python3 -m json.tool

Search papers (alternative to arXiv search, returns JSON)

curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=GRPO+reinforcement+learning&limit=5&fields=title,authors,year,citationCount,externalIds" | python3 -m json.tool

Get paper recommendations

curl -s -X POST "https://api.semanticscholar.org/recommendations/v1/papers/" \
  -H "Content-Type: application/json" \
  -d '{"positivePaperIds": ["arXiv:2402.03300"], "negativePaperIds": []}' | python3 -m json.tool

Author profile

curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=Yann+LeCun&fields=name,hIndex,citationCount,paperCount" | python3 -m json.tool

Useful Semantic Scholar fields

title, authors, year, abstract, citationCount, referenceCount, influentialCitationCount, isOpenAccess, openAccessPdf, fieldsOfStudy, publicationVenue, externalIds (contains arXiv ID, DOI, etc.)

Complete Research Workflow

1. Discover: python scripts/search_arxiv.py "your topic" --sort date --max 10
2. Assess impact: curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID?fields=citationCount,influentialCitationCount"
3. Read abstract: web_extract(urls=["https://arxiv.org/abs/ID"])
4. Read full paper: web_extract(urls=["https://arxiv.org/pdf/ID"])
5. Find related work: curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID/references?fields=title,citationCount&limit=20"
6. Get recommendations: POST to Semantic Scholar recommendations endpoint
7. Track authors: curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=NAME"

Rate Limits

API	Rate	Auth
arXiv	~1 req / 3 seconds	None needed
Semantic Scholar	1 req / second	None (100/sec with API key)

Notes

• arXiv returns Atom XML, use the helper script or parsing snippet for clean output
• Semantic Scholar returns JSON, pipe through python3 -m json.tool for readability
• arXiv IDs: old format (hep-th/0601001) vs new (2402.03300)
• PDF: https://arxiv.org/pdf/{id}, Abstract: https://arxiv.org/abs/{id}
• HTML (when available): https://arxiv.org/html/{id}
• For local PDF processing, see the ocr-and-documents skill

ID Versioning

• arxiv.org/abs/1706.03762 always resolves to the latest version
• arxiv.org/abs/1706.03762v1 points to a specific immutable version
• When generating citations, preserve the version suffix you actually read to prevent citation drift (a later version may substantially change content)
• The API `` field returns the versioned URL (e.g., http://arxiv.org/abs/1706.03762v7)

Withdrawn Papers

Papers can be withdrawn after submission. When this happens:

• The `` field contains a withdrawal notice (look for "withdrawn" or "retracted")
• Metadata fields may be incomplete
• Always check the summary before treating a result as a valid paper

When not to use it

If you need citation data, recommendations, or author impact metrics, arXiv alone won't help. That is where the Semantic Scholar API section comes in. For local PDF processing, use the ocr-and-documents skill instead.

Limits and gotchas

• arXiv rate limits are generous but real: about 1 request every 3 seconds. Batch your queries.
• The API returns Atom XML, not JSON. The helper script and the parsing snippet handle this, but raw output is verbose.
• arXiv IDs come in two formats: old (hep-th/0601001) and new (2402.03300). Both work in queries.
• Papers can be withdrawn. Always check the summary field for withdrawal notices before relying on a result.
• Versioning matters: the URL without a version suffix always points to the latest version, which may differ from the version you originally read. Use versioned URLs in citations.

What pairs with this

The ocr-and-documents skill complements this one for local PDF processing. The Semantic Scholar API section is built into this skill page, so you do not need a separate skill for citation data.