Drive tldraw Offline Canvases with an Agent: Scripting Guide

This skill turns the tldraw offline desktop app into a canvas your agent can drive from a terminal. Instead of clicking and dragging, you write code that reads the open document, generates shapes, and embeds durable JavaScript behavior into the file itself. Reach for this when you want diagrams, wireframes, or interactive whiteboards built by an agent, not by hand.

What it does

The tldraw offline app (offline.tldraw.com) exposes a local HTTP API on localhost:7236 by default. Your agent talks to that API with plain curl calls, the same way the app's own homepage demo shows Codex editing a canvas live. There is no computer-use or GUI clicking involved, and the agent never hand-edits the .tldraw file directly. Instead, it runs code against the live editor and, for durable behavior, edits a JavaScript file that the app watches and applies automatically.

Two workflows cover most needs. One-off edits, like laying out shapes or cleaning up a page, go through the /exec endpoint and take effect immediately. Durable behavior, like reactive shapes or clickable buttons that survive a reload, lives in a script/main.js file that reruns every time the document loads.

Before you start

You need the tldraw offline app installed and running with a document open. Releases are at https://github.com/tldraw/tldraw-offline/releases/latest, with macOS DMG, Windows x64/Arm64, and Linux x86_64/arm64 AppImage or amd64/arm64 .deb packages.

Install the agent skills inside the app via Develop → Install Agent Skills. The app writes its own tldraw skill into ~/.codex/skills/, ~/.claude/skills/, ~/.cursor/skills/, and ~/.gemini/skills/, teaching that agent the curl recipes below. This Hermes skill mirrors that guidance for Hermes.

On launch, the app writes server.json to its config dir: Linux ~/.config/tldraw/, macOS ~/Library/Application Support/tldraw/, Windows %APPDATA%\tldraw\. That file holds the port (default 7236), a bearer token, pid, and startedAt. Every request except GET / needs Authorization: Bearer <token>. A clean quit removes server.json; if the file is present but the port does not answer, the app quit uncleanly, so treat it as not running.

Re-read the port and token on every shell call. Each terminal call is a fresh shell, so an exported token does not persist. "Export once and reuse" sends an empty token and gets a 401. Read both inline at the top of each call with PORT=$(jq -r .port <server.json) and TOKEN=$(jq -r .token <server.json).

No account or network is needed for local editing.

How to run

Two distinct workflows. Pick by whether the change must survive a reload.

A. One-off canvas edits (/exec)

Layout, generating shapes, cleanup. This is a live edit, not a saved script:

BASE=http://localhost:7236
TOKEN=$(python -c "import json;print(json.load(open('$HOME/.config/tldraw/server.json'))['token'])")
# find the focused document id
DOC=$(curl -s "$BASE/api/search" -X POST -H 'content-type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"code":"return (await api.getFocusedDoc()).id"}' | python -c "import sys,json;print(json.load(sys.stdin)['result'])")
# run code with the live `editor` + `helpers` in scope
curl -s "$BASE/api/doc/$DOC/exec" -X POST -H 'content-type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"code":"const {createShapeId,toRichText}=await import(\"tldraw\"); editor.createShape({id:createShapeId(),type:\"geo\",x:0,y:0,props:{geo:\"rectangle\",w:200,h:100,color:\"blue\",fill:\"solid\",richText:toRichText(\"hello\")}}); return editor.getCurrentPageShapes().length"}'

The first call finds the focused document id. The second runs code with the live editor and helpers in scope, creating a blue rectangle with the text "hello" and returning the new shape count. You can read that count back to confirm the edit landed.

B. Durable behavior (script/main.js)

Reactive or interactive logic that must survive reload. Edit the file on disk; the app's watcher applies it:

# get the live script file path for the doc
curl -s "$BASE/api/doc/$DOC/script-workspace" -X POST \
  -H "Authorization: Bearer $TOKEN"          # -> result.mainJsPath, result.isDefaultScript
# edit result.mainJsPath with read_file / patch / write_file (see scripts/main.js)
# then confirm the watcher applied it:
curl -s "$BASE/api/doc/$DOC/script-status" -H "Authorization: Bearer $TOKEN"

The ready-to-adapt document script is scripts/main.js.

Quick reference

The document-script contract, verified against the app's bundled script-context.d.ts:

import { createShapeId, toRichText } from 'tldraw'   // primitives: import, not globals

export default function ({ editor, helpers, signal }) {
  editor.run(() => {                                 // batch = one undo step
    helpers.createShapeIfMissing({                   // idempotent furniture
      id: createShapeId('node-1'), type: 'geo', x: 0, y: 0,
      props: { geo: 'rectangle', w: 200, h: 100, richText: toRichText('hi') },
    })
  })

  const stop = editor.store.listen(() => { /* react */ })  // fires the tick AFTER a commit
  signal.addEventListener('abort', () => stop())           // REQUIRED cleanup on rerun/close
}

• ctx.editor, the live Editor (createShape, updateShape, deleteShapes, getCurrentPageShapes, getShape, getBindingsFromShape, zoomToFit, on('tick'|'event', fn), run(fn, { history: 'ignore' })).
• ctx.helpers, createShapeIfMissing, createShapesIfMissing, createArrowBetweenShapes(from, to, { arrowheadEnd }), translateShapes, onShapeTranslate(id, fn, { signal }), richTextToPlainText, boxShapes, getLints.
• ctx.signal, AbortSignal; attach every listener/interval teardown to it.
• config.js (separate file) registers custom shape/tool/component utils and runs before mount; main.js runs against the mounted editor and reruns on save.

Interactive UI (clickable buttons that drive state)

Drawn shapes can behave like a real app, which is the thing a static whiteboard cannot do. A full example lives in scripts/counter.js: a number display plus MINUS/RESET/PLUS buttons.

There is a verification boundary worth reading before you claim interaction works or does not. The app's own agent playbook says to verify a clickable-UI script with "one simulated click and one state read" via /exec: dispatch a pointer event with editor.dispatch, await a tick, then read the shape's state. Do not drive a real mouse. By that standard the counter is verified: dispatched clicks stepped it 0 → 1 → 2 → 1 → 0.

Two caveats are worth writing down:

• The script only runs once the app's file-watcher applies it. On Linux that watcher uses inotify. A host with an exhausted fs.inotify.max_user_instances logs inotify_add_watch ... No space left on device, script-status shows state: "not-watching" / hasEntry: false, and the script never executes. This is a host limit, not a script bug; a normal desktop is unaffected.
• Driving the canvas with computer-use needs FOREGROUND delivery, not background. cua-driver's default background delivery returns background_unavailable against Electron's occluded renderer, but that is the first rung, not a wall. cua-driver returns an escalation: "foreground" hint; set delivery_mode: "foreground" (pair with bring_to_front) and its X11 XTest path (x11_xtest_fg) clicks land. This dismisses the "Run Script" consent dialog and clicks the canvas. Verified by the Cua team on this exact v1.11.0 AppImage (Linux/X11). Do NOT conclude "Electron rejects synthetic clicks" and give up in background mode; climb to foreground. The real product path is still /exec, not clicks; this note is for computer-use-driven testing.

The pattern:

export default function ({ editor, helpers, signal }) {
  // 1. Build buttons idempotently; tag each with meta so the handler finds them.
  //    Give buttons a visible label AND a meta.action.
  // 2. Hit-test pointer_down in PAGE coordinates against the button bounds:
  const inside = (b, p) => p.x >= b.x && p.x <= b.x + b.w && p.y >= b.y && p.y <= b.y + b.h
  function onEvent(info) {
    if (!info || info.name !== 'pointer_down') return
    let p = null
    try { if (info.point && editor.screenToPage) p = editor.screenToPage(info.point) } catch {}
    p = p ?? editor.inputs?.currentPagePoint
    if (!p) return
    const hit = editor.getCurrentPageShapes().find(
      (s) => s.meta?.ui === 'button' &&
        inside({ x: s.x, y: s.y, w: s.props.w, h: s.props.h }, p)
    )
    if (hit) runAction(hit.meta.action)   // mutate state; store it in a shape's meta
  }
  editor.on('event', onEvent)
  signal.addEventListener('abort', () => editor.off('event', onEvent))  // REQUIRED
}

Find buttons by meta (or visible label via helpers.richTextToPlainText), not by hard-coded coordinates.

One script owns both build and read. If the shapes are created by one code path (with meta.action: 'inc') and the handler reads another convention (meta.action === 'PLUS'), clicks silently do nothing. Ship the buttons built by the same script that handles them, or ship an empty canvas so the script builds them fresh. Never pre-bake mismatched shapes into the file's db.

Keep app state in a shape's meta (for example meta.count) and render it as that shape's richText label, so it survives save and is readable for verification.

Detach the listener on signal abort. Skipping this is not cosmetic: on the next save the old onEvent stays attached alongside the new one, so every click fires twice and a counter jumps by 2 instead of 1.

For continuous motion use editor.on('tick', fn). For a moving anchor with attached pieces use helpers.onShapeTranslate(id, fn, { signal }).

Shipping a self-running scripted .tldraw

A .tldraw is a zip of metadata.json + session.json + db.sqlite + assets/. Only script/ entries are packable. For the script to auto-run without the "This document contains a script → Run Script" consent dialog:

• metadata.json must carry a script manifest: { "sha256": "<digest>" }, where the digest is sha256 over each sorted script/ path as `${path}\0${sha256hex(bytes)}\n`. A mismatch is rejected as tampered.
• Pre-trust the digest by adding it to ~/.tldraw/script-trust.json ({ "trusted": ["<digest>"] }, or $TLDRAW_SCRIPT_TRUST). The app skips consent when isScriptTrusted(digest) is true.

Procedure

1. Read the current token/port from server.json. Find the target doc with api.getFocusedDoc() (or api.getDocs()); name it explicitly if several are open.
2. For layout/generation, use /exec. For durable behavior, edit script/main.js via /script-workspace.
3. Make scripts idempotent: create durable shapes with helpers.createShapeIfMissing and stable createShapeId('name') ids. Scripts rerun on every load.
4. Keep script-owned writes out of the user's undo stack: editor.run(fn, { history: 'ignore' }) (or helpers.translateShapes, which already does).
5. For reactivity, editor.store.listen(cb) and tear it down on signal abort. For interaction, editor.on('event', h) (hit-test pointer_down in page coords); for animation, editor.on('tick', h).
6. For a single moving anchor + attached internals, prefer helpers.onShapeTranslate(anchorId, fn, { signal }) over a broad store listener. A broad listener can turn your own writes into feedback loops.

Shape props (validated against tldraw SDK v5 schema)

editor.createShape / createShapeIfMissing accept partial props; shape utils fill defaults. When building raw records for a file snapshot, every prop below is required. Run scripts/validate_shapes.mjs to check.

Shape	Required props
note	richText, color, labelColor, size, font, align, verticalAlign, growY, fontSizeAdjustment, url, scale, textLastEditedBy
text	richText, color, size, font, textAlign, w, scale, autoSize
frame	w, h, name, color
geo	geo, w, h, color, fill, richText (+ dash/size/etc. defaulted)

richText must be toRichText('...'); a bare string is rejected. color enum: black grey light-violet violet blue light-blue yellow orange green light-green light-red red white. font enum: draw sans serif mono.

Pitfalls

• store.listen fires on the tick AFTER a commit, not synchronously. If you write a shape and immediately read state expecting the listener to have run, it has not. Verified live: an in-turn read shows 0 fires; after one setTimeout tick it shows 1. Same reason the app notes editor.dispatch is async: await a tick before verifying.
• ctx, not globals. The entry is export default function ({ editor, helpers, signal }). There is no bare editor global in a document script. createShapeId / toRichText / Vec come from import ... from 'tldraw'.
• richText, not text. Text/note/geo labels use richText: toRichText(s).
• Raw records need every prop; createShape does not. In-app pass only the props you care about; a hand-built .tldraw snapshot needs the full set (table).
• Scripts rerun on every load: be idempotent. Use createShapeIfMissing with stable ids or you duplicate content and clobber user edits.
• Clean up on signal. signal.addEventListener('abort', () => stop()) for every store.listen / editor.on / setInterval; the signal fires before rerun and on close.
• Keep script writes out of undo: editor.run(fn, { history: 'ignore' }).
• editor.on('tick') pauses when the window is hidden (it is a RAF loop); setInterval keeps firing but Electron throttles it to ~1/s in the background.
• The API needs the bearer token from server.json; the port can be non-default (server.listen(0) picks one). Always read the file, do not hardcode 7236.
• Only tldraw / react / react-dom import, not a Node project.

Verification

• Shape schema (offline, no app): node scripts/validate_shapes.mjs, builds the real tldraw schema and validates note/text/frame. Passing prints 3/3.
• Live canvas edits: after /exec, read back with /api/search → api.getShapes(docId) (returns { page, viewport, shapes }) and api.getBindings(docId) (array). Confirm expected shapes/bindings exist. Grab api.getScreenshot(docId) (returns { filePath, ... }) and inspect the PNG/JPEG with vision_analyze.
• Durable script applied: GET /api/doc/:id/script-status. Success is state: "applied" (currentDiskDigest === lastAppliedDigest === manifestSha256, pendingApply === false, lastApplyError === null). If it stays "pending" after a short retry, report that instead of claiming success; "error" means the apply failed, so read errorLogPath.

When not to use it

Do not hand-place shapes to imitate a drawing. Write the code that generates them. Agents are far better at scripting the canvas than at drawing on it. If you need pixel-perfect visual art or freehand illustration, a different tool is a better fit.

Limits and gotchas

The script only runs once the app's file-watcher applies it. On Linux that watcher uses inotify; an exhausted fs.inotify.max_user_instances stops it silently. Computer-use clicks need foreground delivery, not background. The API needs the bearer token from server.json on every call. Scripts rerun on every load, so idempotency is not optional. Listeners must be detached on signal abort or clicks double-fire.

What pairs with this

This skill fits alongside other creative skills in the Hermes Agent optional-skills collection. For visual verification of canvas screenshots, pair it with vision_analyze. For broader document scripting, the same /exec and /script-workspace patterns extend to other tldraw documents.