Python Debugging with pdb and debugpy: A Practical Guide

When a test fails and the traceback doesn't explain why, or a long-running process misbehaves and can't be restarted, you need a debugger. This guide covers the three Python debugging tools bundled with Hermes Agent: breakpoint() with pdb for quick local inspection, python -m pdb for launching scripts under the debugger, and debugpy for remote and attach-to-process scenarios. You'll learn when to reach for each, how to use them effectively, and how to debug Hermes-specific processes like the gateway and subprocesses.

What it does

This skill gives you a decision tree for debugging Python code, from the simplest interactive breakpoint to attaching a debugger to an already-running process. The core idea is to start with the cheapest tool that works: breakpoint() drops you into a pdb REPL at the exact line you're interested in, with full access to local variables. When you need to debug a script without editing its source, python -m pdb launches it under the debugger. For long-lived processes like a gateway or daemon, or when you need IDE integration, debugpy speaks the Debug Adapter Protocol (DAP) and can attach to a running process by PID.

The pdb REPL itself is a full-featured command-line debugger. You can step through code line by line, set breakpoints, inspect and mutate variables, and even drop into a full Python REPL with the interact command. The skill also covers post-mortem debugging, where you land in the debugger at the exact frame where an exception was raised, and remote debugging with remote-pdb, which gives you a plain pdb prompt over a network socket.

Before you start

• Platform: This skill is available on Linux and macOS. Windows is not listed as supported.
• Installation: The skill is bundled with Hermes Agent, so it's installed by default. No separate installation is needed for the pdb parts.
• debugpy: For remote debugging, you need to install debugpy into the environment of the process you want to debug. The skill shows pip install debugpy after activating the Hermes virtual environment.
• remote-pdb: For the terminal-friendly remote debugging option, install remote-pdb with pip install remote-pdb.
• Permissions: Attaching to an already-running process via debugpy --pid may require adjusting the kernel's ptrace_scope setting, which needs root access.

pdb Quick Reference

Inside any pdb prompt (shown as (Pdb)), these commands are available:

Command	Action
h / h cmd	help
n	next line (step over)
s	step into
r	return from current function
c	continue
unt N	continue until line N
j N	jump to line N (same function only)
l / ll	list source around current line / full function
w	where (stack trace)
u / d	move up / down in the stack
a	print args of the current function
p expr / pp expr	print / pretty-print expression
display expr	auto-print expr on every stop
b file:line	set breakpoint
b func	break on function entry
b file:line, cond	conditional breakpoint
cl N	clear breakpoint N
tbreak file:line	one-shot breakpoint
!stmt	execute arbitrary Python (assignments included)
interact	drop into full Python REPL in current scope (Ctrl+D to exit)
q	quit

The interact command is the most powerful. It gives you a full Python REPL in the current scope, so you can import anything, inspect complex objects, and even call methods that mutate state. Note that locals are read-only by default; use !x = 42 from the (Pdb) prompt to mutate a local variable.

Recipe 1: Local breakpoint

The simplest debugging workflow. Edit your source file to add a breakpoint() call at the point where you want to pause:

def compute(x, y):
    result = some_helper(x)
    breakpoint()           # <-- drops into pdb here
    return result + y

Run the code normally. Execution stops at the breakpoint() line, and you get a (Pdb) prompt with full access to the local variables x, y, and result.

Don't forget to remove breakpoint() before committing. Use git diff or a pre-commit grep to catch stray calls:

rg -n 'breakpoint\(\)' --type py

Recipe 2: Launch a script under pdb (no source edits)

If you don't want to modify the source, launch the script directly under pdb:

python -m pdb path/to/script.py arg1 arg2
# Lands at first line of script
(Pdb) b path/to/script.py:42
(Pdb) c

The script starts paused at its first line. You can set a breakpoint at a specific line, then continue to it.

Recipe 3: Debug a pytest test

The Hermes test runner and pytest both support pdb integration:

# Drop to pdb on failure (or on any raised exception):
scripts/run_tests.sh tests/path/to/test_file.py::test_name --pdb

# Drop to pdb at the START of the test:
scripts/run_tests.sh tests/path/to/test_file.py::test_name --trace

# Show locals in tracebacks without pdb:
scripts/run_tests.sh tests/path/to/test_file.py --showlocals --tb=long

Note: scripts/run_tests.sh runs each test file in a captured subprocess via run_tests_parallel.py (no xdist), so interactive pdb does NOT work under the wrapper. Run pytest directly for --pdb:

source .venv/bin/activate
python -m pytest tests/foo_test.py::test_bar --pdb

This bypasses the hermetic-env guarantees, which is fine for debugging, but re-run under the wrapper to confirm before pushing.

Recipe 4: Post-mortem on any exception

When an exception is raised and you want to inspect the state at the crash site, use post-mortem debugging. The simplest way is to wrap the code in a try/except and call pdb.post_mortem:

import pdb, sys
try:
    run_the_thing()
except Exception:
    pdb.post_mortem(sys.exc_info()[2])

Or wrap a whole script so pdb catches any unhandled exception:

python -m pdb -c continue script.py
# When it crashes, pdb catches it and you're in the frame of the exception

You can also set a global hook in a REPL or Jupyter notebook:

import sys
def excepthook(etype, value, tb):
    import pdb; pdb.post_mortem(tb)
sys.excepthook = excepthook

After this, any uncaught exception in the session drops you into pdb at the crash frame.

Recipe 5: Remote debug with debugpy (attach to running process)

For long-lived processes like the Hermes gateway, tui_gateway, a daemon, or a process that's already misbehaving and can't be restarted cleanly, use debugpy. It speaks DAP and can attach to a running process.

Setup

source <hermes-agent-repo>/.venv/bin/activate
pip install debugpy

Pattern A: Source-edit, process waits for debugger at launch

Add this near the top of the entry point, or inside the function you want to debug:

import debugpy
debugpy.listen(("127.0.0.1", 5678))
print("debugpy listening on 5678, waiting for client...", flush=True)
debugpy.wait_for_client()
debugpy.breakpoint()       # optional: pause immediately once attached

Start the process; it blocks on wait_for_client() until a debugger client attaches.

Pattern B: No source edit, launch with -m debugpy

python -m debugpy --listen 127.0.0.1:5678 --wait-for-client your_script.py arg1

Equivalent for a module entry point:

python -m debugpy --listen 127.0.0.1:5678 --wait-for-client -m your.module

Pattern C: Attach to an already-running process

This requires the PID and debugpy preinstalled in the target's environment:

python -m debugpy --listen 127.0.0.1:5678 --pid <pid>
# debugpy injects itself into the process. Then attach a client as below.

Some kernels or security configs block the ptrace-based injection (/proc/sys/kernel/yama/ptrace_scope). Fix with:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Connecting a client from the terminal

The easiest terminal-side DAP client is VS Code CLI or a small script. From inside Hermes you have two practical options:

Option 1: debugpy's own CLI REPL, not an official feature, but a tiny DAP client script:

# /tmp/dap_client.py
import socket, json, itertools, time, sys

HOST, PORT = "127.0.0.1", 5678
s = socket.create_connection((HOST, PORT))
seq = itertools.count(1)

def send(msg):
    msg["seq"] = next(seq)
    body = json.dumps(msg).encode()
    s.sendall(f"Content-Length: {len(body)}\r\n\r\n".encode() + body)

def recv():
    header = b""
    while b"\r\n\r\n" not in header:
        header += s.recv(1)
    length = int(header.decode().split("Content-Length:")[1].split("\r\n")[0].strip())
    body = b""
    while len(body) < length:
        body += s.recv(length - len(body))
    return json.loads(body)

send({"type": "request", "command": "initialize", "arguments": {"adapterID": "python"}})
print(recv())
send({"type": "request", "command": "attach", "arguments": {}})
print(recv())
send({"type": "request", "command": "setBreakpoints",
      "arguments": {"source": {"path": sys.argv[1]},
                    "breakpoints": [{"line": int(sys.argv[2])}]}})
print(recv())
send({"type": "request", "command": "configurationDone"})
# ... loop reading events and sending continue/stepIn/etc.

This is fine for one-off automation but painful as an interactive UX.

Option 2: Attach from VS Code / Cursor / Zed, if the user has one open, they can add a launch.json:

{
  "name": "Attach to Hermes",
  "type": "debugpy",
  "request": "attach",
  "connect": { "host": "127.0.0.1", "port": 5678 },
  "justMyCode": false,
  "pathMappings": [
    { "localRoot": "${workspaceFolder}", "remoteRoot": "<hermes-agent-repo>" }
  ]
}

Option 3: Ditch DAP, use remote-pdb, usually what you actually want from a terminal agent:

pip install remote-pdb

In your code:

from remote_pdb import set_trace
set_trace(host="127.0.0.1", port=4444)   # blocks until connection

Then from the terminal:

nc 127.0.0.1 4444
# You get a (Pdb) prompt exactly as if debugging locally.

remote-pdb is the cleanest agent-friendly choice when debugpy's DAP protocol is overkill. Use debugpy only when you actually need IDE integration.

Debugging Hermes-specific Processes

Tests

See Recipe 3. The wrapper captures subprocess output, so run pytest directly for interactive pdb.

run_agent.py / CLI, one-shot

Easiest: add breakpoint() near the suspect line, then run hermes normally. Control returns to your terminal at the pause point.

tui_gateway subprocess (spawned by hermes --tui)

The gateway runs as a child of the Node TUI. Options:

A. Source-edit the gateway:

# tui_gateway/server.py near the top of serve()
import debugpy
debugpy.listen(("127.0.0.1", 5678))
debugpy.wait_for_client()

Start hermes --tui. The TUI will appear frozen (its backend is waiting). Attach a client; execution resumes when you continue.

B. Use remote-pdb at a specific handler:

from remote_pdb import set_trace
set_trace(host="127.0.0.1", port=4444)   # in the RPC handler you want to trap

Trigger the matching slash command from the TUI, then nc 127.0.0.1 4444 in another terminal.

_SlashWorker subprocess

Same pattern, remote-pdb with set_trace() inside the worker's exec path. The worker is persistent across slash commands, so the first trigger blocks until you connect; subsequent slash commands pass through normally unless you re-arm.

Gateway (gateway/run.py)

Long-lived. Use remote-pdb at a handler, or debugpy with --wait-for-client if you're restarting the gateway anyway.

Common Pitfalls

1. pdb under a parallel/output-capturing runner silently does nothing. You won't see the prompt, the test just hangs (true of pytest-xdist and of scripts/run_tests.sh's captured per-file subprocesses). Run pytest directly on a single file for interactive debugging.
2. breakpoint() in CI / non-TTY contexts hangs the process. Safe locally; never commit it. Add a pre-commit grep as a safety net.
3. PYTHONBREAKPOINT=0 disables all breakpoint() calls. Check the env if your breakpoint isn't hitting:

echo $PYTHONBREAKPOINT

4. debugpy.listen blocks only if you also call wait_for_client(). Without it, execution continues and your first breakpoint may fire before the client is attached.
5. Attach to PID fails on hardened kernels. ptrace_scope=1 (Ubuntu default) allows only same-user ptrace of child processes. Workaround: echo 0 > /proc/sys/kernel/yama/ptrace_scope (needs root) or launch under debugpy from the start.
6. Threads. pdb only debugs the current thread. For multithreaded code, use debugpy (thread-aware DAP) or set threading.settrace() per thread.
7. asyncio. pdb works in coroutines but await inside pdb requires Python 3.13+ or await from interact mode on older versions. For 3.11/3.12, use asyncio.run_coroutine_threadsafe tricks or !stmt-based awaits via asyncio.ensure_future.
8. scripts/run_tests.sh strips credentials and sets HOME=. If your bug depends on user config or real API keys, it won't reproduce under the wrapper. Debug with raw pytest first to repro, then re-confirm under the wrapper.
9. Forking / multiprocessing. pdb does not follow forks. Each child needs its own breakpoint() or set_trace(). For Hermes subagents, debug one process at a time.

Verification Checklist

• After pip install debugpy, confirm: python -c "import debugpy; print(debugpy.__version__)"
• For remote debug, confirm the port is actually listening: ss -tlnp | grep 5678
• First breakpoint actually hits (if it doesn't, you likely have PYTHONBREAKPOINT=0, you're under a parallel/capturing runner, or execution finished before attach)
• where / w shows the expected call stack
• Post-debug cleanup: no stray breakpoint() / set_trace() in committed code

rg -n 'breakpoint\(\)|set_trace\(|debugpy\.listen' --type py

One-Shot Recipes

"Why is this dict missing a key?"

# add above the KeyError site
breakpoint()
# then in pdb:
(Pdb) pp d
(Pdb) pp list(d.keys())
(Pdb) w                # how did we get here

"This test passes in isolation but fails in the suite."

scripts/run_tests.sh tests/the_test.py   # confirm it fails under the isolated runner first
# For interactive debugging, or if it only fails WITH other tests:
source .venv/bin/activate
python -m pytest tests/ -x --pdb
# Now it pdb-traps at the exact failing test after state accumulated.

"My async handler deadlocks."

# Add at handler entry
import remote_pdb; remote_pdb.set_trace(host="127.0.0.1", port=4444)

Trigger the handler. nc 127.0.0.1 4444, then w to see the suspended frame, !import asyncio; asyncio.all_tasks() to see what else is pending.

"Post-mortem on a crash in an Ink child process / subprocess."

PYTHONFAULTHANDLER=1 python -m pdb -c continue path/to/entrypoint.py
# On crash, pdb lands at the frame of the exception with full locals

When not to use it

Don't reach for a debugger when print() or logging.debug solves the problem in under a minute, or when pytest -vv --tb=long --showlocals already reveals the issue. The skill explicitly lists these as cases where a debugger is overkill.

Limits and gotchas

The skill lists several limitations you should keep in mind:

• pdb does not work under parallel or output-capturing test runners; it will silently hang. Run pytest directly for interactive debugging.
• breakpoint() in CI or non-TTY contexts hangs the process. Never commit it.
• PYTHONBREAKPOINT=0 disables all breakpoint() calls; check the environment if your breakpoint isn't hitting.
• debugpy.listen alone doesn't block; you must also call wait_for_client() to pause execution.
• Attaching to a PID can fail on hardened kernels due to ptrace_scope; you may need root to change it.
• pdb only debugs the current thread; use debugpy for multithreaded code.
• await inside pdb requires Python 3.13+ or the interact command on older versions.
• The test wrapper strips credentials and sets HOME=, so bugs depending on user config won't reproduce under it.
• pdb does not follow forks; each child process needs its own breakpoint.

Related skills

This skill pairs well with systematic-debugging for a structured approach to diagnosing issues, and node-inspect-debugger if you're also debugging JavaScript or TypeScript components in the same project.