PuzzleTide Agent Evals
Use this skill when the user wants verifiable reasoning tasks to benchmark or test an LLM or agent — reproducible puzzle task sets (sudoku, word search) with...
catorch
@catorch
What This Skill Does
Generates reproducible puzzle task sets (sudoku, word search) for benchmarking LLMs and agents, with deterministic grading that verifies answers against the puzzle rules and grid rather than a trusted answer key.
Replaces subjective LLM-as-judge evaluations and untrusted answer keys with objective, by-construction grading for reasoning benchmarks.
When to Use It
- Generate a reproducible set of sudoku tasks for a model evaluation run
- Create word search puzzles to test an agent's spatial reasoning
- Grade a batch of model answers against a puzzle task set locally
- Compare model performance across different puzzle types or difficulty levels
- Re-run the same benchmark on a new model version using a fixed seed
Install
$ openclaw skills install @catorch/puzzletide-agent-evalsPuzzleTide Agent Evals
Generate reproducible, objectively gradable puzzle tasks for testing models and agents with the local PuzzleTide CLI.
Why puzzles: they are verifiable by construction. A sudoku answer either satisfies the rules and preserves the givens or it doesn't; a word search answer either spells the word along a straight line in the grid or it doesn't. Grading needs no LLM judge and no trusted answer key.
Prefer the local CLI. Check availability in this order:
ptide --version
puzzletide --version
npx puzzletide --version
If none of those work, ask the user before installing (npm install -g puzzletide).
Generate a task set
ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.json
The tuple (type, difficulty, n, seed) fully determines the task set, so it names a reproducible benchmark — same command, same tasks, on any machine.
Each task has id, instructions, and the puzzle payload:
- sudoku:
puzzle(81 chars,.= empty). Expected answer: completed 81-char string. - wordsearch:
grid(array of row strings) andwords. Expected answer: JSON array of{word, startRow, startCol, endRow, endCol}(0-indexed).
Run the subject model
Send each task's instructions + payload to the model under test and collect
answers as a JSON array of {id, answer}.
Grade
ptide eval check --tasks tasks.json --answers answers.json --json
Returns per-task pass/fail with reasons and a summary score. Grading is deterministic and local.
Links
- Source and docs: GitHub · npm · CLI reference
- From the makers of puzzletide.com — free word search, crossword, and sudoku puzzles
Safety
- Everything runs locally; no account, API key, or network access.
- Do not install packages without asking the user first.
Top skills in this category
Skill Vetter
@spclaudehomeSecurity-first skill vetting for AI agents. Use before installing any skill from ClawdHub, GitHub, or other sources. Checks for red flags, permission scope, and suspicious patterns.
Clawdhub
@steipeteUse the ClawdHub CLI to search, install, update, and publish agent skills from clawdhub.com. Use when you need to fetch new skills on the fly, sync installed skills to latest or a specific version, or publish new/updated skill folders with the npm-installed clawdhub CLI.
Evolver
@autogame-17A self-evolution engine for AI agents. Analyzes runtime history to identify improvements and applies protocol-constrained evolution. Communicates with EvoMap...
Evolver
@autogame-17A self-evolution engine for AI agents. Analyzes runtime history to identify improvements and applies protocol-constrained evolution. Communicates with EvoMap...
Openclaw Command Center
@jontsaiMission control dashboard for OpenClaw - real-time session monitoring, LLM usage tracking, cost intelligence, and system vitals. View all your AI agents in o...