Pokemon Player Skill: Run a Headless Pokemon Agent in Hermes

The Pokemon Player skill gives Hermes Agent the ability to play Generation 1 Pokemon games through a headless emulator, reading game state directly from RAM and confirming what it sees with screenshots. It is built for people who want to watch an autonomous agent work through Pokemon Red, Blue, Yellow, or FireRed, or who want a reproducible testbed for long-horizon agent behaviour. You would reach for it when the task is literally "play Pokemon" and you want the agent to handle navigation, battles, saving, and progress tracking on its own.

What it does

The skill wraps the pokemon-agent package, which runs a game ROM inside a headless emulator and exposes a small HTTP server. Hermes talks to that server to read position, HP, battle state, and dialog flags, and to send button presses. Because RAM alone does not tell you what is around the player, the skill pairs every state read with a screenshot that a vision model analyses. The result is a loop of observe, orient, decide, act, and verify, with saves and memory notes so a session can be resumed later.

In practice the agent starts the server, opens a public dashboard so you can watch, then plays in short bursts: two to four actions, a screenshot, a correction if needed. It saves before gyms and rival fights, records objectives and map knowledge under PKM: prefixes, and stops cleanly when you ask.

Before you start

The skill is optional and installed on demand. Its path is optional-skills/gaming/pokemon-player, version 1.0.0, authored by Teknium (teknium1) under the MIT licence. It supports linux, macos, and windows.

You need three things before the first run:

1. A checkout of the NousResearch/pokemon-agent repository on GitHub.
2. A Python 3.10+ virtual environment with the package installed in editable mode with the pyboy extra.
3. A ROM file. The skill is explicit: never download or provide ROM files. Ask the user for theirs. A previous setup may already have one at roms/pokemon_red.gb inside the checkout, but you should confirm rather than assume.

If a checkout already exists, for example ~/pokemon-agent with a working venv, skip the clone and just activate the environment.

Startup procedure

1. First-time setup

Clone the repository, then create the virtual environment. uv is preferred for speed; if it is not available, fall back to python -m venv plus pip. Install the package in editable mode with the pyboy extra so the emulator dependency is present.

If the checkout is already there, change into it and activate the existing environment instead of starting over.

2. Start the game server

From inside the pokemon-agent directory with the venv active, launch the server with pokemon-agent serve, pointing --rom at the ROM and --port at 9876. Run it in the background with &. To resume a saved game, add --load-state with the save name. Give it about four seconds to start, then check GET /health to confirm it is up.

3. Set up the live dashboard

The dashboard is how the user watches the run. The skill uses an SSH reverse tunnel through localhost.run: connect with ssh, forwarding local port 9876 to remote port 80 on the keyless localhost.run endpoint. Redirect the output to a log file, wait about ten seconds, then grep the log for the .lhr.life URL. Hand the user that URL with /dashboard/ appended.

The tunnel URL changes every time you restart it. If the tunnel drops, give the user the new address rather than the old one.

Save and load

Saving is not optional housekeeping; it is how you survive a bad gym attempt. The skill's schedule:

• Every 15 to 20 turns of play.
• Always before gym battles, rival encounters, or any risky fight.
• Before entering a new town or dungeon.
• Before any action you are unsure about.

Use POST /save with a descriptive name. Names like before_brock, route1_start, mt_moon_entrance, and got_cut tell you exactly where you were. Load with POST /load and the same name. GET /saves lists everything available.

When starting the server, --load-state auto-loads a save. The skill notes this is faster than loading through the API after startup, so prefer it when you already know which save you want.

The gameplay loop

The loop has seven steps. Skipping the verification step is the most common way a session goes wrong.

Step 1: Observe

Call GET /state for position, HP, battle status, and dialog. Then call GET /screenshot, save it to ~/.hermes/cache/scratch/pokemon.png, and run vision_analyze on it. Do both every time. RAM gives you numbers; vision gives you spatial awareness.

Step 2: Orient

Read the situation before acting. Dialog on screen means advance it. In battle means fight or run. Party hurt means head to a Pokemon Center. Near the objective means navigate carefully.

Step 3: Decide

The skill sets a priority order: dialog first, then battle, then healing, then the story objective, then training, then exploration. Following that order prevents the agent from wandering off mid-conversation or fighting when it should be healing.

Step 4: Act

Send POST /action with a short action list: two to four actions, not ten or fifteen. Long action lists overshoot targets and get stuck on obstacles.

Step 5: Verify

Take a screenshot and run vision_analyze to confirm you moved where you intended. The skill calls this the most important step and states plainly that without vision you will get lost.

Step 6: Record progress

Write what happened to memory using the PKM: prefix conventions described below.

Step 7: Save periodically

Follow the save schedule. Do not wait until something goes wrong.

Action reference

The server accepts these actions:

• press_a confirms, talks, and selects.
• press_b cancels and closes menus.
• press_start opens the game menu.
• walk_up, walk_down, walk_left, walk_right move one tile.
• hold_b_N holds B for N frames, useful for speeding through text.
• wait_60 waits about one second (60 frames).
• a_until_dialog_end presses A repeatedly until dialog clears.

Critical tips from experience

Use vision constantly

Screenshot every two to four movement steps. RAM tells you position and HP but not what surrounds you. Ledges, fences, signs, building doors, and NPCs are only visible in the image. Ask the vision model specific questions, such as "what is one tile north of me?" When stuck, screenshot before trying random directions.

Warp transitions need extra wait time

Walking through a door or stairs fades the screen to black during the map transition. You must wait for it to finish. Add two or three wait_60 actions after any door or stair warp. Without the wait, the position reads as stale and you will think you are still on the old map.

The building exit trap

When you exit a building you appear directly in front of the door. Walking north puts you right back inside. Always sidestep first: walk left or right two tiles, then continue in your intended direction.

Dialog handling

Generation 1 text scrolls slowly, letter by letter. To speed through it, hold B for 120 frames, then press A. Repeat as needed. Holding B makes text display at maximum speed; pressing A advances to the next line. The a_until_dialog_end action checks the RAM dialog flag, but that flag does not catch every text state. If dialog seems stuck, use the manual hold-B-plus-press-A pattern and verify with a screenshot.

Ledges are one-way

Ledges can only be jumped down (south), never climbed up (north). If a ledge blocks you going north, go left or right to find the gap around it. Use vision to identify which direction the gap is, and ask the vision model explicitly.

Navigation strategy

Move two to four steps at a time, then screenshot to check position. When entering a new area, screenshot immediately to orient. Ask the vision model "which direction to [destination]?" If you are stuck for three or more attempts, screenshot and re-evaluate completely. Do not spam ten to fifteen movements; you will overshoot or get stuck.

Running from wild battles

On the battle menu, RUN is bottom-right. From the default cursor position (FIGHT, top-left), press down then right to move the cursor to RUN, then press A. Wrap the sequence with hold_b to speed through text and animations.

Battling

On the battle menu, FIGHT is top-left and is the default cursor position. Press A to enter move selection, press A again to use the first move. Then hold B to speed through attack animations and text.

Battle strategy

Decision tree

1. Want to catch it? Weaken it, then throw a Poke Ball.
2. Wild Pokemon you do not need? Run.
3. Type advantage? Use the super-effective move.
4. No advantage? Use your strongest STAB move.
5. Low HP? Switch or use a Potion.

Gen 1 type chart (key matchups)

• Water beats Fire, Ground, Rock.
• Fire beats Grass, Bug, Ice.
• Grass beats Water, Ground, Rock.
• Electric beats Water, Flying.
• Ground beats Fire, Electric, Rock, Poison.
• Psychic beats Fighting, Poison (dominant in Gen 1).

Gen 1 quirks

• The Special stat serves as both offense and defense for special moves.
• Psychic is overpowered because Ghost moves are bugged.
• Critical hits are based on the Speed stat.
• Wrap and Bind prevent the opponent from acting.
• The Focus Energy bug reduces crit rate instead of raising it.

Memory conventions

Progress lives in Hermes memory under fixed prefixes so a later session can pick up where the last one stopped.

Prefix	Purpose	Example
PKM:OBJECTIVE	Current goal	Get Parcel from Viridian Mart
PKM:MAP	Navigation knowledge	Viridian: mart is northeast
PKM:STRATEGY	Battle/team plans	Need Grass type before Misty
PKM:PROGRESS	Milestone tracker	Beat rival, heading to Viridian
PKM:STUCK	Stuck situations	Ledge at y=28 go right to bypass
PKM:TEAM	Team notes	Squirtle Lv6, Tackle + Tail Whip

Progression milestones

The skill tracks the main story beats so the agent knows what comes next:

• Choose a starter.
• Deliver the Parcel from Viridian Mart and receive the Pokedex.
• Boulder Badge, Brock (Rock): use Water or Grass.
• Cascade Badge, Misty (Water): use Grass or Electric.
• Thunder Badge, Lt. Surge (Electric): use Ground.
• Rainbow Badge, Erika (Grass): use Fire, Ice, or Flying.
• Soul Badge, Koga (Poison): use Ground or Psychic.
• Marsh Badge, Sabrina (Psychic): the hardest gym.
• Volcano Badge, Blaine (Fire): use Water or Ground.
• Earth Badge, Giovanni (Ground): use Water, Grass, or Ice.
• Elite Four, then Champion.

Stopping play

End a session cleanly in four steps:

1. Save with a descriptive name via POST /save.
2. Update memory with PKM:PROGRESS.
3. Tell the user: "Game saved as [name]! Say 'play pokemon' to resume."
4. Kill the server and tunnel background processes.

When not to use it

This skill is for actually playing a Pokemon game through the emulator. If the user only wants to discuss Pokemon, look up type matchups, or plan a team without running a ROM, you do not need the server, the tunnel, or the emulator at all. The skill also assumes the user can supply a ROM; if they cannot or will not, the run cannot start, and the skill forbids downloading one on their behalf.

Limits and gotchas

The skill's own pitfalls list is short and worth memorising:

• Never download or provide ROM files.
• Do not send more than four or five actions without checking vision.
• Always sidestep after exiting a building before going north.
• Always add two or three wait_60 actions after door or stair warps.
• Dialog detection via RAM is unreliable; verify with screenshots.
• Save before risky encounters.
• The tunnel URL changes each time you restart it.

Two more constraints follow from the setup. The server must be running and reachable on the chosen port before any action call will work, and the dashboard depends on the localhost.run tunnel staying alive. If either process dies, the user loses the view and the agent loses its control channel.

What pairs with it

This skill sits in the Gaming category alongside other optional skills. It depends on the pokemon-agent package and its pyboy extra, and it uses Hermes's vision_analyze capability plus the scratch cache at ~/.hermes/cache/scratch/ for screenshots. The memory conventions tie into Hermes's general memory system, so anything that reads or writes agent memory will see the PKM: entries. If you are building a longer autonomous workflow, the save and resume pattern here is a useful model for any skill that needs to survive a restart.