Hermes Agent FindMy Skill: Track Apple Devices and AirTags

The FindMy skill lets a Hermes Agent answer questions like "where are my keys?" by driving the macOS Find My app and reading the screen. Apple ships no command-line interface for Find My, so this skill combines AppleScript, screen capture, and vision analysis to turn a GUI-only feature into something an agent can use. Reach for it when you want an agent to check on a device, an AirTag, or a pet's movement without you opening the app yourself.

What it does

Find My on macOS shows device and item locations in a window. The skill opens that window, switches between the Devices and Items tabs, and captures what is on screen. A vision model then reads the capture and reports locations in text the agent can reason about. Because the data only exists in the UI, every step is a form of UI automation: activate the app, click the right control, take a picture, interpret it.

Before you start

The skill is bundled with Hermes Agent, so there is nothing to install to get the basic method. You do need the following in place:

macOS with the Find My app installed and iCloud signed in.
The devices or AirTags already registered in Find My. The skill reads what the app already knows; it does not add new items.
Screen Recording permission granted to your terminal in System Settings → Privacy & Security → Screen Recording. Without it, screenshots will not capture the Find My window.
Optional but recommended: peekaboo, installed with brew install steipete/tap/peekaboo. The source describes it as giving better UI automation, which matters because AppleScript clicks are the fragile part of the basic method.

Method 1: AppleScript and screenshots

This is the path that works with nothing beyond the base system. The first block opens Find My, waits for it to load, and captures the window.

# Open Find My app
osascript -e 'tell application "FindMy" to activate'

# Wait for it to load
sleep 3

# Take a screenshot of the Find My window
screencapture -w -o ~/.hermes/cache/scratch/findmy.png

The -w flag tells screencapture to capture a window rather than the whole screen, and -o drops the window shadow. The file is written to the Hermes scratch cache. Once you have the image, hand it to the vision tool:

vision_analyze(image_url="~/.hermes/cache/scratch/findmy.png", question="What devices/items are shown and what are their locations?")

That call is how the agent turns pixels into an answer. The source is explicit that you should not try to parse the screenshot yourself; let the vision model read it.

Switching tabs

Find My separates devices from AirTags. These two AppleScript snippets click the toolbar buttons for each tab.

# Switch to Devices tab
osascript -e '
tell application "System Events"
    tell process "FindMy"
        click button "Devices" of toolbar 1 of window 1
    end tell
end tell'

# Switch to Items tab (AirTags)
osascript -e '
tell application "System Events"
    tell process "FindMy"
        click button "Items" of toolbar 1 of window 1
    end tell
end tell'

Use the Devices tab for iPhones, iPads, Macs, and AirPods, and the Items tab for AirTags. Because these scripts address the UI by button name and window position, they are the part most likely to need adjustment after a macOS update.

Method 2: Peekaboo UI automation

If peekaboo is installed, the source recommends it over raw AppleScript. It can annotate the UI with element identifiers and click by ID, which is less brittle than clicking a named button in a toolbar.

# Open Find My
osascript -e 'tell application "FindMy" to activate'
sleep 3

# Capture and annotate the UI
peekaboo see --app "FindMy" --annotate --path ~/.hermes/cache/scratch/findmy-ui.png

# Click on a specific device/item by element ID
peekaboo click --on B3 --app "FindMy"

# Capture the detail view
peekaboo image --app "FindMy" --path ~/.hermes/cache/scratch/findmy-detail.png

The annotated capture gives you the element IDs to click. After clicking into a device or item, capture the detail view and read it with vision:

vision_analyze(image_url="~/.hermes/cache/scratch/findmy-detail.png", question="What is the location shown for this device/item? Include address and coordinates if visible.")

This two-step pattern, annotate then click, is what makes Peekaboo more reliable for repeated runs. You are not guessing at UI structure; you are reading it from the capture.

Workflow: tracking an AirTag over time

A single location is a snapshot. If you want a route, for example a cat's patrol path, the skill supports a loop that captures on an interval.

# 1. Open FindMy to Items tab
osascript -e 'tell application "FindMy" to activate'
sleep 3

# 2. Click on the AirTag item (stay on page — AirTag only updates when page is open)

# 3. Periodically capture location
while true; do
    screencapture -w -o ~/.hermes/cache/scratch/findmy-$(date +%H%M%S).png
    sleep 300  # Every 5 minutes
done

Each capture is timestamped by filename. Analyze the screenshots with vision to pull out coordinates, then assemble them into a route. The source suggests a cronjob for ongoing tracking rather than leaving a loop running in a terminal.

When not to use it

There is no alternative CLI or API to fall back on. Apple does not provide one, which is why this skill exists at all. If you need programmatic location data without a GUI, this skill cannot give it to you. It is also the wrong tool for anything outside Find My: it reads what the app displays, nothing more.

Limits and gotchas

The source lists several constraints worth planning around:

Find My has no CLI or API, so everything here is UI automation and therefore subject to breakage across macOS versions.
AirTags only refresh their location while the Find My page for that item is actively displayed. Minimize the app or navigate away and updates stop.
Accuracy depends on nearby Apple devices in the Find My network. A location is only as good as the crowd around the tag.
Screen Recording permission is mandatory for screenshots.
Keep Find My in the foreground while tracking AirTags.
Use vision_analyze for reading screenshots rather than attempting pixel parsing.
Only track devices and items the user owns. The source states this as a rule, and it is the line between a useful agent and a privacy problem.

This skill sits in the Apple category alongside the other bundled macOS skills. If you are building an agent that works on a Mac, pair it with the rest of the Apple set so the agent can handle device and app tasks without you driving the UI.

Hermes Agent FindMy Skill: Track Apple Devices and AirTags on macOS