Agent-operated HPC workflow

Use a local coding agent with shell access, Git, uv and a supported authenticated HPC connection. Remote helpers require Slurm, Bash, Python, tmux, tar and zstd; the full runtime requirements are in the resolved checkout's docs/usage.md.

The deliverable is a working connection and operating contract for the user's actual cluster. Resolve the repository checkout containing common/profile.py; paths below are relative to that checkout. This skill ships in that repository. The canonical source is https://github.com/playeriv65/hpc-agent-kit. Market installers copy this skill directory, not the remote helpers. If using an independently installed copy, locate the matching repository first; if absent, clone the canonical source into a new local workspace. Compare this installed SKILL.md with skills/hpc-workflow/SKILL.md in the checkout. If they differ, locate the matching source revision or update both from one reviewed revision; do not mix instructions from one version with helpers from another. Use the same version for instructions and scripts; record the resolved path and version in private site notes. Do not assume the skill directory contains helpers or invent a download URL. Read the checkout's AGENTS.md or identical CLAUDE.md. Run uv run python common/check_repo.py to verify its inventory and contracts.

Establish or adapt a site

Inspect the existing connection, jobs, tmux hierarchy and operator instructions first. Reuse a supported installation where possible. Read docs/msi-persistent-ssh.md in the resolved checkout when adapting an MSI/Duo route or an always-on authenticated holder, then read site probing to establish the transport, Slurm/QoS permissions, disk-backed local paths and persistent archive location. Unknown facts stay unknown; idle hardware does not establish project access.

Create a private profile using the observed facts and the examples under examples/. Select direct-ssh when login-to-compute SSH is proven, or psc-bridge for PSC's allocation-terminal bridge. The profile renderer rejects unfilled values, unverified local/persistent storage, non-isolated local paths and QoS outside the project/partition intersection. These checks validate the provided facts; they do not discover or independently prove those facts.

Run uv run --no-project --python 3.12 python common/profile.py PROFILE.json --output NEW_STAGING_DIRECTORY. Inspect the generated profile and manifest. common/deploy_bundle.py BUNDLE checks frozen bytes and shows its targets; --apply installs through the profile's already-authenticated exec route after quiescent-state checks, preserving old helpers and configuration. Use deployment within the user's authorized setup scope. Do not publish private profiles. For upgrades or failures, read recovery; use the shared transactional installer and its saved backup rather than ad hoc file replacement.

The adapters are templates, not a promise that every HPC has the same policy. For unsupported schedulers, storage layouts, GPU keys or connection mechanisms, adapt an isolated template and test it before replacing active helpers. Keep toolchain versions and framework setup in explicit site/repository configuration; do not carry MSI/PSC compiler choices to an unrelated cluster.

Read acceptance and perform the applicable checks. Write the site's operating notes with its supported entry command, paths, resource defaults, recovery commands and exact verification limits. Preserve the user's existing transport and research data. No human initialization wizard is required: the agent probes, writes configuration, deploys and verifies.

Operate and recover

The login tmux owns one gpu-request allocation pane. Each granted job owns gpu-JOB_ID compute tmux with setup, autosave and named task windows. A transport disconnect does not establish that Slurm released the allocation.

Persist an allocation request ID before submitting: SITE-gpu --request ID GPU_TYPE COUNT WALLTIME. Reuse exactly that ID and configuration after disconnects; SITE-gpu allocation ID reconciles the durable record with Slurm. Reservation or submission uncertainty never authorizes another ID. Inspect its log, named pane and scheduler evidence; preserve ambiguous records. READY is historical bootstrap evidence, not a current health guarantee.

Use SITE-gpu run --job JOB_ID --request TASK_ID --wait WINDOW -- COMMAND ... for experiments after bootstrap. A repeated task ID must retain the same job, window and literal command. Do not create extra srun steps for ordinary work. Use the repository's launcher to recreate environments and apply its toolchain. Read operating lessons when adapting desktop wrappers, preparing an application's environment/caches or observing a job through its end. It covers allocation namespaces, argument transport and workload readiness.

SITE-agent-state status reports queue, routes and allocation records. SITE-agent-state doctor also checks current compute access, autosave process identity, save freshness, current-writer identity and failure counts. doctor --verify-backup additionally reads the full archive chain; use it for acceptance or suspected corruption rather than routine high-frequency polling. Doctor uses an isolated hpc-health task window and records, without altering experiments. Unreachable, stale and unchecked are distinct from healthy.

Bootstrap restores and verifies before starting work, starts autosave and waits for a successful current-writer save. Defaults are five-minute increments, six-hour full compaction, and a save before Slurm's end time. Immutable generations commit through an atomic pointer; corrupt latest generations fall back to verified predecessors. Abrupt node loss can recover only already committed data. Store checkpoints/results explicitly on persistent storage; snapshots exclude .venv and large model caches. Never classify an incomplete checkpoint as resumable.

Normal monitoring reads status and logs. Allocation retries, authentication and destructive recovery are separate decisions: do not send repeated MFA challenges, cancel research jobs, discard uncertain records, reset live tmux, delete evidence or overwrite checkpoints as an automatic response to a failed observation.