Research Paper Writing Pipeline: From Experiment to Submission
Write ML papers for NeurIPS/ICML/ICLR: design→submit.
Written by Neura Market from the official Hermes Agent documentation for Research Paper Writing. Commands, paths, and version numbers are reproduced from the source unchanged.
Read the official documentationThis skill turns a research idea or existing codebase into a publication-ready paper for top ML venues like NeurIPS, ICML, and ICLR. It is an end-to-end pipeline that covers experiment design, execution, analysis, drafting, self-review, and submission, with an emphasis on iteration and feedback loops. You would reach for this when you are starting a new paper, revising a draft, responding to reviews, or preparing camera-ready deliverables.
What it does
The skill is a structured workflow that guides an agent through the entire research lifecycle. It is not a linear checklist; it is an iterative loop where results trigger new experiments, reviews trigger revisions, and analysis feeds back into design. The pipeline is organized into phases, from project setup and literature review through experiment execution, analysis, drafting, self-review, and submission. It also covers post-acceptance tasks like posters, talks, and code release.
The core philosophy is proactive drafting: deliver complete drafts rather than asking questions, never hallucinate citations, treat the paper as a story with one clear contribution, and ensure every experiment supports a specific claim. Git discipline is emphasized throughout, with commits serving as the experiment history.
The overall flow is captured in the pipeline diagram below.
┌─────────────────────────────────────────────────────────────┐
│ RESEARCH PAPER PIPELINE │
│ │
│ Phase 0: Project Setup ──► Phase 1: Literature Review │
│ │ │ │
│ ▼ ▼ │
│ Phase 2: Experiment Phase 5: Paper Drafting ◄──┐ │
│ Design │ │ │
│ │ ▼ │ │
│ ▼ Phase 6: Self-Review │ │
│ Phase 3: Execution & & Revision ──────────┘ │
│ Monitoring │ │
│ │ ▼ │
│ ▼ Phase 7: Submission │
│ Phase 4: Analysis ─────► (feeds back to Phase 2 or 5) │
│ │
└─────────────────────────────────────────────────────────────┘
Before you start
This skill is bundled with Hermes Agent and installed by default. It is available on Linux and macOS. You will need a working LaTeX installation for compilation, git for version control, and access to the tools the skill relies on, such as terminal, execute_code, web_search, and delegate_task. The skill references several companion skills and reference documents that you should load when you reach the relevant phase.
Phase 0: Project Setup
The goal here is to establish the workspace, understand existing work, and identify the contribution. Start by exploring the repository to see what already exists.
# Understand project structure
ls -la
find . -name "*.py" | head -30
find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding"
Look for a README, results directories, configs, and existing .bib files. Then organize the workspace into a consistent structure.
workspace/
paper/ # LaTeX source, figures, compiled PDFs
experiments/ # Experiment runner scripts
code/ # Core method implementation
results/ # Raw experiment results (auto-generated)
tasks/ # Task/benchmark definitions
human_eval/ # Human evaluation materials (if needed)
Set up version control if it is not already in place.
git init # if not already
git remote add origin <repo-url>
git checkout -b paper-draft # or main
Commit every completed experiment batch with a descriptive message. The skill gives examples like "Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task)" and "Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier".
Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task)
Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier
Before writing anything, articulate the contribution: the What, the Why, and the So What. Propose a one-sentence framing to the scientist and ask for confirmation. Then create a TODO list using the todo tool, covering everything from defining the contribution to submission prep. Update this list throughout the project; it is the persistent state across sessions.
Research Paper TODO:
- [ ] Define one-sentence contribution
- [ ] Literature review (related work + baselines)
- [ ] Design core experiments
- [ ] Run experiments
- [ ] Analyze results
- [ ] Write first draft
- [ ] Self-review (simulate reviewers)
- [ ] Revise based on review
- [ ] Submission prep
Estimate the compute budget before running experiments. Consider API costs, GPU hours, and human evaluation costs, and add a 30-50% contingency. Track actual spend with a simple cost tracker.
Compute Budget Checklist:
- [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs)
- [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds)
- [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate)
- [ ] Total budget ceiling and contingency (add 30-50% for reruns)
# Simple cost tracker pattern
import json, os
from datetime import datetime
COST_LOG = "results/cost_log.jsonl"
def log_cost(experiment: str, model: str, input_tokens: int, output_tokens: int, cost_usd: float):
entry = {
"timestamp": datetime.now().isoformat(),
"experiment": experiment,
"model": model,
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"cost_usd": cost_usd,
}
with open(COST_LOG, "a") as f:
f.write(json.dumps(entry) + "\n")
When budget is tight, run pilot experiments with 1-2 seeds on a subset of tasks before committing to full sweeps. Use cheaper models for debugging, then switch to target models for final runs.
Most papers have 3-10 authors, so establish coordination workflows early. Choose between Overleaf, Git + LaTeX, or Overleaf + Git sync. Assign each section to one primary author to avoid merge conflicts. Agree on notation conventions, figure style, and LaTeX macros before anyone writes.
Author Coordination Checklist:
- [ ] Agree on section ownership (who writes what)
- [ ] Set up shared workspace (Overleaf or git repo)
- [ ] Establish notation conventions (before anyone writes)
- [ ] Schedule internal review rounds (not just at the end)
- [ ] Designate one person for final formatting pass
- [ ] Agree on figure style (colors, fonts, sizes) before creating figures
Phase 1: Literature Review
The goal is to find related work, identify baselines, and gather citations. Start from papers already referenced in the codebase.
# Via terminal:
grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py"
find . -name "*.bib"
Load the arxiv skill for structured paper discovery. Use web_search for broad discovery and web_extract for fetching specific papers.
# Via web_search:
web_search("[main technique] + [application domain] site:arxiv.org")
web_search("[baseline method] comparison ICML NeurIPS 2024")
# Via web_extract (for specific papers):
web_extract("https://arxiv.org/abs/2303.17651")
Additional search queries to try:
Search queries:
- "[main technique] + [application domain]"
- "[baseline method] comparison"
- "[problem name] state-of-the-art"
- Author names from existing citations
You can install Exa MCP for real-time academic search.
claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp"
A flat search misses important related work. Use an iterative breadth-then-depth pattern: start with 4-6 parallel queries covering different angles, then generate follow-up queries from what you learn, and finally fill specific gaps. Stop when a round returns more than 80% papers you already have. Typically 2-3 rounds suffice; surveys may need 4-5. For agent-based workflows, delegate each round's queries in parallel via delegate_task.
Iterative Literature Search:
Round 1 (Breadth): 4-6 parallel queries covering different angles
- "[method] + [domain]"
- "[problem name] state-of-the-art 2024 2025"
- "[baseline method] comparison"
- "[alternative approach] vs [your approach]"
→ Collect papers, extract key concepts and terminology
Round 2 (Depth): Generate follow-up queries from Round 1 learnings
- New terminology discovered in Round 1 papers
- Papers cited by the most relevant Round 1 results
- Contradictory findings that need investigation
→ Collect papers, identify remaining gaps
Round 3 (Targeted): Fill specific gaps
- Missing baselines identified in Rounds 1-2
- Concurrent work (last 6 months, same problem)
- Key negative results or failed approaches
→ Stop when new queries return mostly papers you've already seen
Never generate BibTeX from memory. The error rate for AI-generated citations is around 40%. Follow a mandatory 5-step process for every citation: search, verify in 2+ sources, retrieve BibTeX via DOI content negotiation, validate the claim, and add to the bibliography. If any step fails, mark it as [CITATION NEEDED] and inform the scientist.
Citation Verification (MANDATORY per citation):
1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords
2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef)
3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory)
4. VALIDATE → Confirm the claim you're citing actually appears in the paper
5. ADD → Add verified BibTeX to bibliography
If ANY step fails → mark as [CITATION NEEDED], inform scientist
# Fetch BibTeX via DOI
import requests
def doi_to_bibtex(doi: str) -> str:
response = requests.get(
f"https://doi.org/{doi}",
headers={"Accept": "application/x-bibtex"}
)
response.raise_for_status()
return response.text
If you cannot verify a citation, use a placeholder.
\cite{PLACEHOLDER_author2024_verify_this} % TODO: Verify this citation exists
Always tell the scientist how many citations are placeholders. Organize related work by methodology, not paper-by-paper. Group papers by approach and contrast assumptions, rather than listing "Smith et al. did X, Jones et al. did Y."
Phase 2: Experiment Design
Every experiment must answer a specific question that supports a paper claim. Create an explicit mapping from claims to experiments and expected evidence. If an experiment does not map to a claim, do not run it.
Design strong baselines: naive, strong, ablation, and compute-matched. Reviewers will ask whether you compared against the right methods. Define the evaluation protocol before running anything: metrics, aggregation, statistical tests, and sample sizes.
Write experiment scripts that save results incrementally for crash recovery.
# Save after each problem/task
result_path = f"results/{task}/{strategy}/result.json"
if os.path.exists(result_path):
continue # Skip already-completed work
# ... run experiment ...
with open(result_path, 'w') as f:
json.dump(result, f, indent=2)
Preserve all intermediate outputs.
results/<experiment>/
<task>/
<strategy>/
final_output.md # Final result
history.json # Full trajectory
pass_01/ # Per-iteration artifacts
version_a.md
version_b.md
critic.md
Keep generation, evaluation, and visualization separate.
run_experiment.py # Core experiment runner
run_baselines.py # Baseline comparison
run_comparison_judge.py # Blind evaluation
analyze_results.py # Statistical analysis
make_charts.py # Visualization
If human evaluation is needed, design it before automated experiments because it has longer lead times. Decide on annotator type, scale, sample size, agreement metric, and platform. Include attention checks, worked examples, and fair compensation. Report annotator qualifications, inter-annotator agreement, compensation details, and interface description.
- [ ] Clear task description with examples (good AND bad)
- [ ] Decision criteria for ambiguous cases
- [ ] At least 2 worked examples per category
- [ ] Attention checks / gold standard items (10-15% of total)
- [ ] Qualification task or screening round
- [ ] Estimated time per item and fair compensation (>= local minimum wage)
- [ ] IRB/ethics review if required by your institution
Phase 3: Experiment Execution & Monitoring
Launch long-running experiments with nohup.
nohup python run_experiment.py --config config.yaml > logs/experiment_01.log 2>&1 &
echo $! # Record the PID
Run independent experiments in parallel, but be aware of API rate limits. Set up periodic monitoring with a cron prompt that checks the process, reads the log tail, looks for completed results, and commits when done. If nothing has changed, respond with [SILENT] to suppress notifications.
Monitor Prompt Template:
1. Check if process is still running: ps aux | grep <pattern>
2. Read last 30 lines of log: tail -30 <logfile>
3. Check for completed results: ls <result_dir>
4. If results exist, read and report: cat <result_file>
5. If all done, commit: git add -A && git commit -m "<descriptive message>" && git push
6. Report in structured format (tables with key metrics)
7. Answer the key analytical question for this experiment
Handle failures by detecting them early and recovering. Common failures include API rate limits, process crashes, timeouts, and wrong model IDs. Scripts should always check for existing results and skip completed work, making re-runs safe.
After each batch, commit the results.
git add -A
git commit -m "Add <experiment name>: <key finding in 1 line>"
git push
Maintain an experiment journal that captures the exploration tree: why you tried something, what you learned, and what to try next. This is more valuable than git history alone for writing the Methods section and reporting failures honestly.
// experiment_journal.jsonl — append one entry per experiment attempt
{
"id": "exp_003",
"parent": "exp_001",
"timestamp": "2025-05-10T14:30:00Z",
"hypothesis": "Adding scope constraints will fix convergence failure from exp_001",
"plan": "Re-run autoreason with max_tokens=2000 and fixed structure template",
"config": {"model": "haiku", "strategy": "autoreason", "max_tokens": 2000},
"status": "completed",
"result_path": "results/exp_003/",
"key_metrics": {"win_rate": 0.85, "convergence_rounds": 3},
"analysis": "Scope constraints fixed convergence. Win rate jumped from 0.42 to 0.85.",
"next_steps": ["Try same constraints on Sonnet", "Test without structure template"],
"figures": ["figures/exp003_convergence.pdf"]
}
Snapshot the experiment script after each run for exact reproduction.
cp experiment.py results/exp_003/experiment_snapshot.py
Phase 4: Result Analysis
Aggregate results from all runs, compute per-task and aggregate metrics, and generate summary tables.
# Standard analysis pattern
import json, os
from pathlib import Path
results = {}
for result_file in Path("results/").rglob("result.json"):
data = json.loads(result_file.read_text())
strategy = result_file.parent.name
task = result_file.parent.parent.name
results.setdefault(strategy, {})[task] = data
# Compute aggregate metrics
for strategy, tasks in results.items():
scores = [t["score"] for t in tasks.values()]
print(f"{strategy}: mean={np.mean(scores):.1f}, std={np.std(scores):.1f}")
Always compute error bars, confidence intervals, pairwise tests like McNemar's, and effect sizes like Cohen's d. Identify the story: the main finding in one sentence, what surprised you, what failed, and what follow-up experiments are needed.
Negative results can be valuable. If your hypothesis was wrong but the why is informative, frame the paper around the analysis. If the method reveals something new, reframe as understanding. Clean negative results on popular claims are worth writing up. Venues like NeurIPS Datasets & Benchmarks, TMLR, and workshops welcome them.
Create figures with vector graphics, colorblind-safe palettes, and self-contained captions. Use booktabs for tables, bold the best value, and include direction symbols.
\usepackage{booktabs}
\begin{tabular}{lcc}
\toprule
Method & Accuracy $\uparrow$ & Latency $\downarrow$ \\
\midrule
Baseline & 85.2 & 45ms \\
\textbf{Ours} & \textbf{92.1} & 38ms \\
\bottomrule
\end{tabular}
Decide whether to run more experiments or start writing. If core claims are supported, move to drafting. If results are inconclusive, go back to design. If one ablation is missing, run it first.
Before writing, create an experiment_log.md that bridges results to prose. This is the single most important connective tissue. It should contain the contribution, each experiment with its claim, setup, key result, result files, figures, and surprising findings, plus a figures table, failed experiments, and open questions. This log lets a writing agent produce a grounded first draft without parsing raw JSON files, which is a common source of hallucinated numbers.
# Experiment Log
## Contribution (one sentence)
[The paper's main claim]
## Experiments Run
### Experiment 1: [Name]
- **Claim tested**: [Which paper claim this supports]
- **Setup**: [Model, dataset, config, number of runs]
- **Key result**: [One sentence with the number]
- **Result files**: results/exp1/final_info.json
- **Figures generated**: figures/exp1_comparison.pdf
- **Surprising findings**: [Anything unexpected]
### Experiment 2: [Name]
...
## Figures
| Filename | Description | Which section it belongs in |
|----------|-------------|---------------------------|
| figures/main_comparison.pdf | Bar chart comparing all methods on benchmark X | Results, Figure 2 |
| figures/ablation.pdf | Ablation removing components A, B, C | Results, Figure 3 |
...
## Failed Experiments (document for honesty)
- [What was tried, why it failed, what it tells us]
## Open Questions
- [Anything the results raised that the paper should address]
Iterative Refinement: Strategy Selection
Any output in the pipeline can be iteratively refined. The choice of strategy depends on the model tier and task type. The quick decision table summarizes when to use autoreason, critique-and-revise, or single pass.
| Your Situation | Strategy | Why |
|---|---|---|
| Mid-tier model + constrained task | Autoreason | Sweet spot. Generation-evaluation gap is widest. Baselines actively destroy weak model outputs. |
| Mid-tier model + open task | Autoreason with scope constraints added | Add fixed facts, structure, or deliverable to bound the improvement space. |
| Frontier model + constrained task | Autoreason | Wins 2/3 constrained tasks even at frontier. |
| Frontier model + unconstrained task | Critique-and-revise or single pass | Autoreason comes last. Model self-evaluates well enough. |
| Concrete technical task (system design) | Critique-and-revise | Direct find-and-fix loop is more efficient. |
| Template-filling task (one correct structure) | Single pass or conservative | Minimal decision space. Iteration adds no value. |
| Code with test cases | Autoreason (code variant) | Structured analysis of why it failed before fixing. Recovery rate 62% vs 43%. |
| Very weak model (Llama 8B class) | Single pass | Model too weak for diverse candidates. Invest in generation quality. |
The value of autoreason depends on the gap between generation and self-evaluation capability. This gap is structural and moves as costs drop, but never disappears.
Model Tier │ Generation │ Self-Eval │ Gap │ Autoreason Value
──────────────────┼────────────┼───────────┼────────┼─────────────────
Weak (Llama 8B) │ Poor │ Poor │ Small │ None — can't generate diverse candidates
Mid (Haiku 3.5) │ Decent │ Poor │ LARGE │ MAXIMUM — 42/42 perfect Borda
Mid (Gemini Flash)│ Decent │ Moderate │ Large │ High — wins 2/3
Strong (Sonnet 4) │ Good │ Decent │ Medium │ Moderate — wins 3/5
Frontier (S4.6) │ Excellent │ Good │ Small │ Only with constraints
The autoreason loop produces three candidates from fresh, isolated agents: a critic, an author, and a synthesizer, then a judge panel ranks them via Borda count. Convergence is reached when the incumbent wins two consecutive passes. Key parameters: k=2 convergence, CoT judges always, temperature 0.8 for authors and 0.3 for judges, conservative tiebreak, and fresh agents for every role.
When applying autoreason to paper drafts, provide ground truth to the critic, use at least 3 working judges, and scope-constrain the revision. Watch for failure modes like no convergence, synthesis drift, degradation below single pass, overfitting, and broken judges.
Phase 5: Paper Drafting
The complete drafting procedure lives in references/phase5-paper-drafting.md. Load it with read_file when you reach this phase, and pair it with references/writing-guide.md for prose-level style rules.
Phase 6: Self-Review & Revision
Simulate the review process before submission. Generate 3-5 independent reviews with different models or temperatures, defaulting to negative bias. Each reviewer sees only the paper.
You are an expert reviewer for [VENUE]. You are critical and thorough.
If a paper has weaknesses or you are unsure about a claim, flag it clearly
and reflect that in your scores. Do not give the benefit of the doubt.
Review this paper according to the official reviewer guidelines. Evaluate:
1. Soundness (are claims well-supported? are baselines fair and strong?)
2. Clarity (is the paper well-written? could an expert reproduce it?)
3. Significance (does this matter to the community?)
4. Originality (new insights, not just incremental combination?)
Provide your review as structured JSON:
{
"summary": "2-3 sentence summary",
"strengths": ["strength 1", "strength 2", ...],
"weaknesses": ["weakness 1 (most critical)", "weakness 2", ...],
"questions": ["question for authors 1", ...],
"missing_references": ["paper that should be cited", ...],
"soundness": 1-4,
"presentation": 1-4,
"contribution": 1-4,
"overall": 1-10,
"confidence": 1-5
}
Then feed all reviews to a meta-reviewer acting as an area chair.
You are an Area Chair at [VENUE]. You have received [N] independent reviews
of a paper. Your job is to:
1. Identify consensus strengths and weaknesses across reviewers
2. Resolve disagreements by examining the paper directly
3. Produce a meta-review that represents the aggregate judgment
4. Use AVERAGED numerical scores across all reviews
Be conservative: if reviewers disagree on whether a weakness is serious,
treat it as serious until the authors address it.
Reviews:
[review_1]
[review_2]
...
Optionally run a reflection loop where reviewers refine their reviews after seeing the meta-review, stopping when a reviewer says "I am done". Use the strongest available model for reviewing, independent of the writing model. Include 1-2 real published reviews as few-shot examples if available.
Run a separate visual review pass with a vision-capable model on the compiled PDF.
You are reviewing the visual presentation of this research paper PDF.
Check for:
1. Figure quality: Are plots readable? Labels legible? Colors distinguishable?
2. Figure-caption alignment: Does each caption accurately describe its figure?
3. Layout issues: Orphaned section headers, awkward page breaks, figures far from their references
4. Table formatting: Aligned columns, consistent decimal precision, bold for best results
5. Visual consistency: Same color scheme across all figures, consistent font sizes
6. Grayscale readability: Would the figures be understandable if printed in B&W?
For each issue, specify the page number and exact location.
Then run a claim verification pass to catch factual errors.
Claim Verification Protocol:
1. Extract every factual claim from the paper (numbers, comparisons, trends)
2. For each claim, trace it to the specific experiment/result that supports it
3. Verify the number in the paper matches the actual result file
4. Flag any claim without a traceable source as [VERIFY]
Delegate verification to a fresh sub-agent that receives only the paper text and raw result files, preventing confirmation bias.
Prioritize feedback into critical, high, medium, and low. Fix critical and high issues, possibly requiring new experiments. For each issue, identify affected sections, draft the fix, verify it does not break other claims, and update the paper.
When responding to actual reviews, write point-by-point rebuttals. Address every concern, lead with the strongest responses, be concise, include new results if available, and never be defensive. Use latexdiff to generate a marked-up PDF. Thank reviewers for specific feedback.
> R1-W1: "The paper lacks comparison with Method X."
We thank the reviewer for this suggestion. We have added a comparison with
Method X in Table 3 (revised). Our method outperforms X by 3.2pp on [metric]
(p<0.05). We note that X requires 2x our compute budget.
Save snapshots at key milestones.
paper/
paper.tex # Current working version
paper_v1_first_draft.tex # First complete draft
paper_v2_post_review.tex # After simulated review
paper_v3_pre_submission.tex # Final before submission
paper_v4_camera_ready.tex # Post-acceptance final
Phase 7: Submission Preparation
Complete every venue's mandatory checklist to avoid desk rejection. See references/checklists.md for NeurIPS, ICML, ICLR, and ACL requirements.
Anonymize thoroughly for double-blind review.
Anonymization Checklist:
- [ ] No author names or affiliations anywhere in the PDF
- [ ] No acknowledgments section (add after acceptance)
- [ ] Self-citations written in third person: "Smith et al. [1] showed..." not "We previously showed [1]..."
- [ ] No GitHub/GitLab URLs pointing to your personal repos
- [ ] Use Anonymous GitHub (https://anonymous.4open.science/) for code links
- [ ] No institutional logos or identifiers in figures
- [ ] No file metadata containing author names (check PDF properties)
- [ ] No "our previous work" or "in our earlier paper" phrasing
- [ ] Dataset names don't reveal institution (rename if needed)
- [ ] Supplementary materials don't contain identifying information
Verify formatting.
Pre-Submission Format Check:
- [ ] Page limit respected (excluding references and appendix)
- [ ] All figures are vector (PDF) or high-res raster (600 DPI PNG)
- [ ] All figures readable in grayscale
- [ ] All tables use booktabs
- [ ] References compile correctly (no "?" in citations)
- [ ] No overfull hboxes in critical areas
- [ ] Appendix clearly labeled and separated
- [ ] Required sections present (limitations, broader impact, etc.)
Run automated checks before compiling.
# 1. Lint with chktex (catches common LaTeX mistakes)
# Suppress noisy warnings: -n2 (sentence end), -n24 (parens), -n13 (intersentence), -n1 (command terminated)
chktex main.tex -q -n2 -n24 -n13 -n1
# 2. Verify all citations exist in .bib
# Extract \cite{...} from .tex, check each against .bib
python3 -c "
import re
tex = open('main.tex').read()
bib = open('references.bib').read()
cites = set(re.findall(r'\\\\cite[tp]?{([^}]+)}', tex))
for cite_group in cites:
for cite in cite_group.split(','):
cite = cite.strip()
if cite and cite not in bib:
print(f'WARNING: \\\\cite{{{cite}}} not found in references.bib')
"
# 3. Verify all referenced figures exist on disk
python3 -c "
import re, os
tex = open('main.tex').read()
figs = re.findall(r'\\\\includegraphics(?:\[.*?\])?{([^}]+)}', tex)
for fig in figs:
if not os.path.exists(fig):
print(f'WARNING: Figure file not found: {fig}')
"
# 4. Check for duplicate \label definitions
python3 -c "
import re
from collections import Counter
tex = open('main.tex').read()
labels = re.findall(r'\\\\label{([^}]+)}', tex)
dupes = {k: v for k, v in Counter(labels).items() if v > 1}
for label, count in dupes.items():
print(f'WARNING: Duplicate label: {label} (appears {count} times)')
"
Fix any warnings before proceeding. For agent-based workflows, feed chktex output back to the agent with instructions to make minimal fixes.
Compile cleanly.
# Clean build
rm -f *.aux *.bbl *.blg *.log *.out *.pdf
latexmk -pdf main.tex
# Or manual (triple pdflatex + bibtex for cross-references)
pdflatex -interaction=nonstopmode main.tex
bibtex main
pdflatex -interaction=nonstopmode main.tex
pdflatex -interaction=nonstopmode main.tex
# Verify output exists and has content
ls -la main.pdf
If compilation fails, parse the .log file for the first error. Common fixes include missing packages, math symbols outside math mode, wrong figure paths, and missing bib entries.
Each venue has specific requirements: NeurIPS wants a paper checklist in the appendix, ICML a broader impact statement, ICLR an LLM disclosure, ACL a limitations section, AAAI strict style adherence, and COLM a contribution framed for the language model community.
When converting between venues, never copy LaTeX preambles. Start fresh with the target template and copy only content sections.
# 1. Start fresh with target template
cp -r templates/icml2026/ new_submission/
# 2. Copy ONLY content sections (not preamble)
# - Abstract text, section content, figures, tables, bib entries
# 3. Adjust for page limits
# 4. Add venue-specific required sections
# 5. Update references
After rejection, address reviewer concerns without referencing the previous submission.
Prepare the camera-ready version after acceptance.
Camera-Ready Checklist:
- [ ] De-anonymize: add author names, affiliations, email addresses
- [ ] Add Acknowledgments section (funding, compute grants, helpful reviewers)
- [ ] Add public code/data URL (real GitHub, not anonymous)
- [ ] Address any mandatory revisions from meta-reviewer
- [ ] Switch template to camera-ready mode (if applicable — e.g., AAAI \anon → \camera)
- [ ] Add copyright notice if required by venue
- [ ] Update any "anonymous" placeholders in text
- [ ] Verify final PDF compiles cleanly
- [ ] Check page limit for camera-ready (sometimes differs from submission)
- [ ] Upload supplementary materials (code, data, appendix) to venue portal
Decide on arXiv timing carefully. For double-blind venues, post after the submission deadline. ICLR explicitly allows posting before. Do not update the arXiv version during review with changes that reference reviews. Choose primary and cross-listed categories.
# Check if your paper's title is already taken on arXiv
# (before choosing a title)
pip install arxiv
python -c "
import arxiv
results = list(arxiv.Search(query='ti:\"Your Exact Title\"', max_results=5).results())
print(f'Found {len(results)} matches')
for r in results: print(f' {r.title} ({r.published.year})')
"
Package clean, runnable code alongside the camera-ready submission.
your-method/
README.md # Setup, usage, reproduction instructions
requirements.txt # Or environment.yml for conda
setup.py # For pip-installable packages
LICENSE # MIT or Apache 2.0 recommended for research
configs/ # Experiment configurations
src/ # Core method implementation
scripts/ # Training, evaluation, analysis scripts
train.py
evaluate.py
reproduce_table1.sh # One script per main result
data/ # Small data or download scripts
download_data.sh
results/ # Expected outputs for verification
Use a README template for research code.
# [Paper Title]
Official implementation of "[Paper Title]" (Venue Year).
## Setup
[Exact commands to set up environment]
## Reproduction
To reproduce Table 1: `bash scripts/reproduce_table1.sh`
To reproduce Figure 2: `python scripts/make_figure2.py`
## Citation
[BibTeX entry]
Before release, verify the code runs from a clean clone, pin dependencies, remove hardcoded paths and credentials, and include a LICENSE. Use Anonymous GitHub for double-blind review.
- [ ] Code runs from a clean clone (test on fresh machine or Docker)
- [ ] All dependencies pinned to specific versions
- [ ] No hardcoded absolute paths
- [ ] No API keys, credentials, or personal data in repo
- [ ] README covers setup, reproduction, and citation
- [ ] LICENSE file present (MIT or Apache 2.0 for max reuse)
- [ ] Results are reproducible within expected variance
- [ ] .gitignore excludes data files, checkpoints, logs
# Use Anonymous GitHub for double-blind review
# https://anonymous.4open.science/
# Upload your repo → get an anonymous URL → put in paper
Phase 8: Post-Acceptance Deliverables
Prepare a conference poster with the title, authors, one-sentence contribution, method figure, 2-3 key results, and conclusion. Use a Z-pattern flow, readable text at distance, and high-resolution figures. Order posters 2+ weeks ahead.
For talks, adapt content to the duration: 5 minutes for a spotlight, 15-20 for an oral, 10-15 for a workshop. One idea per slide, minimal text, animated key figures, and a takeaway slide.
Write an accessible summary: a Twitter/X thread of 5-8 tweets, a blog post of 800-1500 words, or a project page. Post within 1-2 days of the paper appearing on proceedings or arXiv.
Workshop & Short Papers
Workshop papers have lower page limits and review standards, and value interesting ideas, preliminary results, and position pieces. Target a workshop for early-stage ideas, negative results, or position pieces. ACL short papers are 4 pages and should focus on one claim. Findings papers are 8 pages and represent solid work that narrowly missed the main conference.
Paper Types Beyond Empirical ML
Theory papers structure around theorems and proofs, with preliminaries, main results, proof sketches, and full proofs in the appendix. State theorems formally, provide intuition before proofs, and number assumptions.
Survey papers contribute organization and synthesis, requiring a clear taxonomy and comprehensive coverage. Benchmark papers must fill a genuine evaluation gap, document the dataset, and demonstrate the benchmark is challenging. Position papers contribute an argument, engaging seriously with counterarguments.
Hermes Agent Integration
This skill is designed for the Hermes agent and uses its tools, delegation, scheduling, and memory. Compose it with related skills for specific phases.
| Skill | When to Use | How to Load |
|---|---|---|
| arxiv | Phase 1 (Literature Review): searching arXiv, generating BibTeX, finding related papers via Semantic Scholar | skill_view("arxiv") |
| subagent-driven-development | Phase 5 (Drafting): parallel section writing with 2-stage review (spec compliance then quality) | skill_view("subagent-driven-development") |
| plan | Phase 0 (Setup): creating structured plans before execution. Writes to .hermes/plans/ | skill_view("plan") |
| qmd | Phase 1 (Literature): searching local knowledge bases (notes, transcripts, docs) via hybrid BM25+vector search | Install: skill_manage("install", "qmd") |
| diagramming | Phase 4-5: creating Excalidraw-based figures and architecture diagrams | skill_view("diagramming") |
| data-science | Phase 4 (Analysis): Jupyter live kernel for interactive analysis and visualization | skill_view("data-science") |
This skill supersedes ml-paper-writing, containing all of its content plus the full experiment and analysis pipeline.
Use the terminal tool for LaTeX compilation, git operations, and launching experiments. Use process for background experiment management. Use execute_code for Python analysis. Use read_file, write_file, and patch for editing. Use web_search and web_extract for literature. Use delegate_task for parallel section drafting and citation verification. Use todo as the primary state tracker, memory to persist key decisions, cronjob for monitoring, and clarify for targeted questions when blocked.
For experiment monitoring, the typical pattern is to check the process, read the log, list results, analyze, and commit. For parallel section drafting, delegate each section to a fresh sub-agent with all necessary context.
terminal("ps aux | grep <pattern>")
→ terminal("tail -30 <logfile>")
→ terminal("ls results/")
→ execute_code("analyze results JSON, compute metrics")
→ terminal("git add -A && git commit -m '<descriptive message>' && git push")
→ (final response auto-delivers "Experiment complete: <summary>"; for unattended runs, schedule via cron with a deliver: target)
delegate_task("Draft the Methods section based on these experiment scripts and configs.
Include: pseudocode, all hyperparameters, architectural details sufficient for
reproduction. Write in LaTeX using the neurips2025 template conventions.")
delegate_task("Draft the Related Work section. Use web_search and web_extract to
find papers. Verify every citation via Semantic Scholar. Group by methodology.")
delegate_task("Draft the Experiments section. Read all result files in results/.
State which claim each experiment supports. Include error bars and significance.")
For citation verification, use execute_code with the Semantic Scholar API.
# In execute_code:
from semanticscholar import SemanticScholar
import requests
sch = SemanticScholar()
results = sch.search_paper("attention mechanism transformers", limit=5)
for paper in results:
doi = paper.externalIds.get('DOI', 'N/A')
if doi != 'N/A':
bibtex = requests.get(f"https://doi.org/{doi}",
headers={"Accept": "application/x-bibtex"}).text
print(bibtex)
Use memory to persist key decisions across sessions.
memory("add", "Paper: autoreason. Venue: NeurIPS 2025 (9 pages).
Contribution: structured refinement works when generation-evaluation gap is wide.
Key results: Haiku 42/42, Sonnet 3/5, S4.6 constrained 2/3.
Status: Phase 5 — drafting Methods section.")
Use todo to track granular progress.
todo("add", "Design constrained task experiments for Sonnet 4.6")
todo("add", "Run Haiku baseline comparison")
todo("add", "Draft Methods section")
todo("update", id=3, status="in_progress")
todo("update", id=1, status="completed")
At the start of each session, check the todo list, read memory, review recent commits, check running experiments, and look for new results.
1. todo("list") # Check current task list
2. memory("read") # Recall key decisions
3. terminal("git log --oneline -10") # Check recent commits
4. terminal("ps aux | grep python") # Check running experiments
5. terminal("ls results/ | tail -20") # Check for new results
6. Report status to user, ask for direction
Schedule periodic experiment checks with cronjob.
cronjob("create", {
"schedule": "*/30 * * * *", # Every 30 minutes
"prompt": "Check experiment status:
1. ps aux | grep run_experiment
2. tail -30 logs/experiment_haiku.log
3. ls results/haiku_baselines/
4. If complete: read results, compute Borda scores,
git add -A && git commit -m 'Add Haiku results' && git push
5. Report: table of results, key finding, next step
6. If nothing changed: respond with [SILENT]"
})
Track deadlines with daily cron jobs.
cronjob("create", {
"schedule": "0 9 * * *", # Daily at 9am
"prompt": "NeurIPS 2025 deadline: May 22. Today is {date}.
Days remaining: {compute}.
Check todo list — are we on track?
If <7 days: warn user about remaining tasks."
})
Notify the user when experiments complete, unexpected findings occur, drafts are ready, or deadlines approach. Use [SILENT] when nothing has changed. Always report with structured data: status, results table, key finding, and next step.
## Experiment: <name>
Status: Complete / Running / Failed
| Task | Method A | Method B | Method C |
|------|---------|---------|---------|
| Task 1 | 85.2 | 82.1 | **89.4** |
Key finding: <one sentence>
Next step: <what happens next>
Use clarify only when genuinely blocked on venue choice, contribution framing, experiment priority, or submission readiness. Do not ask about word choice, section ordering, or which results to highlight; make a choice and flag it.
Reviewer Evaluation Criteria
Reviewers check quality, clarity, significance, and originality. The NeurIPS 6-point scale ranges from 6 (Strong Accept) to 1 (Strong Reject). Understanding this helps focus effort on technical soundness, clear writing, community impact, and new insights.
Common Issues and Solutions
Common problems include generic abstracts, introductions that are too long, experiments lacking explicit claims, missing statistical significance, scope creep, and weak human evaluations. Solutions include starting with the specific contribution, splitting background into related work, adding claim statements before each experiment, reporting error bars and tests, mapping every experiment to a claim, and following the human evaluation guidance. For negative results, consider workshops or reframing as analysis.
Reference Documents
The skill points to several reference documents for deeper guidance: writing-guide.md for prose style, citation-workflow.md for citation APIs, checklists.md for venue requirements, reviewer-guidelines.md for evaluation criteria, experiment-patterns.md for experiment design, autoreason-methodology.md for refinement strategies, human-evaluation.md for human studies, and paper-types.md for non-empirical papers.
LaTeX templates are available for NeurIPS 2025, ICML 2026, ICLR 2026, ACL, AAAI 2026, and COLM 2025.
Key external sources include writing guides by Neel Nanda, Sebastian Farquhar, Gopen & Swan, Lipton, and Perez, plus APIs from Semantic Scholar, CrossRef, and arXiv.
When not to use it
This skill is designed for ML/AI research papers targeting the listed venues. It is not suited for non-research writing tasks, papers outside these fields, or situations where you do not have access to the required tools or compute. If you are writing a paper that does not follow the empirical ML structure, see the guidance for theory, survey, benchmark, or position papers. For very short or informal writeups, a full pipeline may be overkill.
Limits and gotchas
The skill emphasizes that this is not a linear pipeline; it is an iterative loop. Results trigger new experiments, and reviews trigger revisions. The agent must handle these feedback loops. Citation verification is mandatory because AI-generated citations have a high error rate. The skill warns against hallucinating citations and requires marking unverifiable ones. It also cautions that autoreason is not always the best strategy; for frontier models on unconstrained tasks, critique-and-revise or single pass may be better. The skill notes that human evaluation has longer lead times and requires careful design and reporting. Finally, it stresses that every experiment must map to a claim, and that negative results can be valuable if handled honestly.
What pairs with this
This skill composes with the arxiv skill for literature review, subagent-driven-development for parallel drafting, plan for project setup, qmd for local knowledge search, diagramming for figures, and data-science for analysis. It supersedes ml-paper-writing.