Search and Read arXiv Papers with Hermes Agent

The arXiv Research skill turns Hermes Agent into a literature search tool. It queries the arXiv REST API, which needs no API key and no extra dependencies, so you can find papers by keyword, author, category, or ID straight from the command line. Researchers, students, and engineers who need to track a field, check a citation count, or pull a PDF for reading will reach for this skill first.

What it does

This skill gives you a set of curl commands and a helper script to search arXiv, fetch specific papers, and generate BibTeX entries. It also shows you how to read the abstract or full text through web_extract, and how to layer the Semantic Scholar API on top for citation data, recommendations, and author profiles. The workflow is built around the idea that discovery, assessment, and reading should happen without leaving your terminal.

Before you start

The skill is bundled with Hermes Agent, so it is installed by default. It works on Linux, macOS, and Windows. You need curl and, for the parsing snippets, Python with its standard library. No API keys are required for either arXiv or the basic Semantic Scholar endpoints. The helper script scripts/search_arxiv.py uses only Python stdlib, so there is nothing extra to install.

Searching Papers

The arXiv API returns Atom XML. You can parse it with grep or sed, but for clean output you will want to pipe it through Python. The basic search below returns five results for the query GRPO reinforcement learning.

Basic search

curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5"

Clean output (parse XML to readable format)

curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5&sortBy=submittedDate&sortOrder=descending" | python -c "
import sys, xml.etree.ElementTree as ET
ns = {'a': 'http://www.w3.org/2005/Atom'}
root = ET.parse(sys.stdin).getroot()
for i, entry in enumerate(root.findall('a:entry', ns)):
    title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
    arxiv_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
    published = entry.find('a:published', ns).text[:10]
    authors = ', '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
    summary = entry.find('a:summary', ns).text.strip()[:200]
    cats = ', '.join(c.get('term') for c in entry.findall('a:category', ns))
    print(f'{i+1}. [{arxiv_id}] {title}')
    print(f'   Authors: {authors}')
    print(f'   Published: {published} | Categories: {cats}')
    print(f'   Abstract: {summary}...')
    print(f'   PDF: https://arxiv.org/pdf/{arxiv_id}')
    print()
"

The parsing snippet extracts the title, arXiv ID, publication date, authors, first 200 characters of the abstract, and categories, then prints a readable list with a direct PDF link. This is the pattern you will use most often: search, then read the output to decide which papers deserve a closer look.

Search Query Syntax

arXiv's query language uses prefixes to target specific fields. The table below shows the common ones.

Prefix	Searches	Example
all:	All fields	all:transformer+attention
ti:	Title	ti:large+language+models
au:	Author	au:vaswani
abs:	Abstract	abs:reinforcement+learning
cat:	Category	cat:cs.AI
co:	Comment	co:accepted+NeurIPS

Boolean operators

# AND (default when using +)
search_query=all:transformer+attention

# OR
search_query=all:GPT+OR+all:BERT

# AND NOT
search_query=all:language+model+ANDNOT+all:vision

# Exact phrase
search_query=ti:"chain+of+thought"

# Combined
search_query=au:hinton+AND+cat:cs.LG

Note that + acts as AND, so all:transformer+attention finds papers that mention both terms. For OR, you write all:GPT+OR+all:BERT. The AND NOT operator lets you exclude terms, and exact phrases use quotes around the plus-joined words. You can combine prefixes with AND, as in the last example.

Sort and Pagination

Parameter	Options
sortBy	relevance, lastUpdatedDate, submittedDate
sortOrder	ascending, descending
start	Result offset (0-based)
max_results	Number of results (default 10, max 30000)

# Latest 10 papers in cs.AI
curl -s "https://export.arxiv.org/api/query?search_query=cat:cs.AI&sortBy=submittedDate&sortOrder=descending&max_results=10"

Use start to page through results, and max_results to control how many come back. The default is 10, but you can request up to 30000. Sorting by submittedDate descending is the standard way to see the newest work in a category.

Fetching Specific Papers

# By arXiv ID
curl -s "https://export.arxiv.org/api/query?id_list=2402.03300"

# Multiple papers
curl -s "https://export.arxiv.org/api/query?id_list=2402.03300,2401.12345,2403.00001"

If you already know the arXiv ID, you can fetch its metadata directly. The id_list parameter accepts a comma-separated list, so you can pull several papers at once.

BibTeX Generation

After fetching metadata for a paper, generate a BibTeX entry:

curl -s "https://export.arxiv.org/api/query?id_list=1706.03762" | python -c "
import sys, xml.etree.ElementTree as ET
ns = {'a': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}
root = ET.parse(sys.stdin).getroot()
entry = root.find('a:entry', ns)
if entry is None: sys.exit('Paper not found')
title = entry.find('a:title', ns).text.strip().replace('\n', ' ')
authors = ' and '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))
year = entry.find('a:published', ns).text[:4]
raw_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]
cat = entry.find('arxiv:primary_category', ns)
primary = cat.get('term') if cat is not None else 'cs.LG'
last_name = entry.find('a:author', ns).find('a:name', ns).text.split()[-1]
print(f'@article{{{last_name}{year}_{raw_id.replace(\".\", \"\")},')
print(f'  title     = {{{title}}},')
print(f'  author    = {{{authors}}},')
print(f'  year      = {{{year}}},')
print(f'  eprint    = {{{raw_id}}},')
print(f'  archivePrefix = {{arXiv}},')
print(f'  primaryClass  = {{{primary}}},')
print(f'  url       = {{https://arxiv.org/abs/{raw_id}}}')
print('}')
"

This script takes the XML for a single paper and prints a BibTeX entry. It uses the first author's last name and the year for the citation key, and it fills in the title, author list, year, eprint, archive prefix, primary class, and URL. If the paper is not found, it exits with a message.

Reading Paper Content

After finding a paper, read it:

# Abstract page (fast, metadata + abstract)
web_extract(urls=["https://arxiv.org/abs/2402.03300"])

# Full paper (PDF → markdown via Firecrawl)
web_extract(urls=["https://arxiv.org/pdf/2402.03300"])

For local PDF processing, see the ocr-and-documents skill.

The abstract page loads quickly and gives you the metadata and abstract. The PDF version goes through Firecrawl to convert the PDF to markdown, which is useful when you want to read the full text. If you need to process PDFs locally, the ocr-and-documents skill is the companion.

Common Categories

Category	Field
cs.AI	Artificial Intelligence
cs.CL	Computation and Language (NLP)
cs.CV	Computer Vision
cs.LG	Machine Learning
cs.CR	Cryptography and Security
stat.ML	Machine Learning (Statistics)
math.OC	Optimization and Control
physics.comp-ph	Computational Physics

Full list: https://arxiv.org/category_taxonomy

These are the categories you will use most often. The full taxonomy is available at the link if you need a more specific one.

Helper Script

The scripts/search_arxiv.py script handles XML parsing and provides clean output:

python scripts/search_arxiv.py "GRPO reinforcement learning"
python scripts/search_arxiv.py "transformer attention" --max 10 --sort date
python scripts/search_arxiv.py --author "Yann LeCun" --max 5
python scripts/search_arxiv.py --category cs.AI --sort date
python scripts/search_arxiv.py --id 2402.03300
python scripts/search_arxiv.py --id 2402.03300,2401.12345

No dependencies, uses only Python stdlib.

The helper script is the easiest way to search without writing your own XML parser. It accepts a query string, or flags for author, category, or ID, and it can sort by date and limit the number of results.

Semantic Scholar (Citations, Related Papers, Author Profiles)

arXiv doesn't provide citation data or recommendations. Use the Semantic Scholar API for that, free, no key needed for basic use (1 req/sec), returns JSON.

Get paper details + citations

# By arXiv ID
curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300?fields=title,authors,citationCount,referenceCount,influentialCitationCount,year,abstract" | python -m json.tool

# By Semantic Scholar paper ID or DOI
curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1234/example?fields=title,citationCount"

Get citations OF a paper (who cited it)

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/citations?fields=title,authors,year,citationCount&limit=10" | python -m json.tool

Get references FROM a paper (what it cites)

curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/references?fields=title,authors,year,citationCount&limit=10" | python -m json.tool

Search papers (alternative to arXiv search, returns JSON)

curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=GRPO+reinforcement+learning&limit=5&fields=title,authors,year,citationCount,externalIds" | python -m json.tool

Get paper recommendations

curl -s -X POST "https://api.semanticscholar.org/recommendations/v1/papers/" \
  -H "Content-Type: application/json" \
  -d '{"positivePaperIds": ["arXiv:2402.03300"], "negativePaperIds": []}' | python -m json.tool

Author profile

curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=Yann+LeCun&fields=name,hIndex,citationCount,paperCount" | python -m json.tool

Useful Semantic Scholar fields

title, authors, year, abstract, citationCount, referenceCount, influentialCitationCount, isOpenAccess, openAccessPdf, fieldsOfStudy, publicationVenue, externalIds (contains arXiv ID, DOI, etc.)

Semantic Scholar fills the gaps that arXiv leaves open. You can get citation counts, see who cited a paper, pull its reference list, search for papers, get recommendations, and look up author profiles with h-index and total citation counts. The responses are JSON, so piping through python -m json.tool makes them readable.

Complete Research Workflow

1. Discover: python scripts/search_arxiv.py "your topic" --sort date --max 10
2. Assess impact: curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID?fields=citationCount,influentialCitationCount"
3. Read abstract: web_extract(urls=["https://arxiv.org/abs/ID"])
4. Read full paper: web_extract(urls=["https://arxiv.org/pdf/ID"])
5. Find related work: curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID/references?fields=title,citationCount&limit=20"
6. Get recommendations: POST to Semantic Scholar recommendations endpoint
7. Track authors: curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=NAME"

This is the full loop: find papers, check their impact, read them, then branch out through references and recommendations. It is a practical sequence for a literature review or for staying current in a field.

Rate Limits

API	Rate	Auth
arXiv	~1 req / 3 seconds	None needed
Semantic Scholar	1 req / second	None (100/sec with API key)

Both APIs are free for basic use, but they throttle you. arXiv allows roughly one request every three seconds, and Semantic Scholar one per second. If you need higher throughput, Semantic Scholar offers an API key that raises the limit to 100 requests per second.

Notes

• arXiv returns Atom XML, use the helper script or parsing snippet for clean output
• Semantic Scholar returns JSON, pipe through python -m json.tool for readability
• arXiv IDs: old format (hep-th/0601001) vs new (2402.03300)
• PDF: https://arxiv.org/pdf/{id}, Abstract: https://arxiv.org/abs/{id}
• HTML (when available): https://arxiv.org/html/{id}
• For local PDF processing, see the ocr-and-documents skill

ID Versioning

• arxiv.org/abs/1706.03762 always resolves to the latest version
• arxiv.org/abs/1706.03762v1 points to a specific immutable version
• When generating citations, preserve the version suffix you actually read to prevent citation drift (a later version may substantially change content)
• The API `` field returns the versioned URL (e.g., http://arxiv.org/abs/1706.03762v7)

Withdrawn Papers

Papers can be withdrawn after submission. When this happens:

• The `` field contains a withdrawal notice (look for "withdrawn" or "retracted")
• Metadata fields may be incomplete
• Always check the summary before treating a result as a valid paper

When not to use it

If you need citation data, recommendations, or author metrics, arXiv alone will not help you. That is where the Semantic Scholar API comes in. Also, if you are looking for papers outside arXiv's scope, such as preprints from other servers or published versions with DOI metadata, you may need a different source. The skill is focused on arXiv and Semantic Scholar, so for other databases you would look elsewhere.

Limits and gotchas

The main limits are the rate limits: arXiv at about one request per three seconds, and Semantic Scholar at one per second without a key. The XML output from arXiv is not human-friendly, so you will want the helper script or the parsing snippet. Be aware of ID versioning: a URL without a version always points to the latest, which can change. For citations, preserve the version suffix to avoid drift. Also, papers can be withdrawn, so check the summary for a withdrawal notice before relying on a result.

What pairs with this

The ocr-and-documents skill is the natural companion when you need to process PDFs locally. After you download a paper, that skill can extract text from the PDF for further analysis. Together, they cover the full pipeline from discovery to reading to processing.