Tracking Apple Devices and AirTags with Hermes Agent on macOS

The Find My skill for Hermes Agent lets you locate Apple devices and AirTags through the macOS Find My app. Because Apple provides no command-line interface for Find My, this skill automates the app using AppleScript and screen captures, then reads the results with a vision model. You would reach for this when you need to answer location questions programmatically, build a tracking log, or integrate device location into an autonomous workflow.

What it does

This skill opens the Find My app on macOS, navigates between the Devices and Items tabs, takes screenshots of the interface, and optionally uses the peekaboo tool for more reliable UI automation. The captured images are then analyzed by a vision model (via vision_analyze) to extract location data, device names, and coordinates. It supports both one-shot queries and periodic monitoring, such as tracking an AirTag's position over time.

Before you start

• macOS with the Find My app installed and signed into iCloud.
• Devices or AirTags must already be registered in your Find My account.
• Screen Recording permission must be granted to your terminal app. Go to System Settings → Privacy & Security → Screen Recording and enable your terminal.
• Optional but recommended: Install peekaboo for better UI automation: brew install steipete/tap/peekaboo

Method 1: AppleScript + Screenshot (Basic)

This is the simplest approach. It opens Find My, waits for it to load, and takes a screenshot of the window.

Open FindMy and Navigate

# Open Find My app
osascript -e 'tell application "FindMy" to activate'

# Wait for it to load
sleep 3

# Take a screenshot of the Find My window
screencapture -w -o /tmp/findmy.png

Then use vision_analyze to read the screenshot:

vision_analyze(image_url="/tmp/findmy.png", question="What devices/items are shown and what are their locations?")

Switch Between Tabs

# Switch to Devices tab
osascript -e '
tell application "System Events"
    tell process "FindMy"
        click button "Devices" of toolbar 1 of window 1
    end tell
end tell'

# Switch to Items tab (AirTags)
osascript -e '
tell application "System Events"
    tell process "FindMy"
        click button "Items" of toolbar 1 of window 1
    end tell
end tell'

Method 2: Peekaboo UI Automation (Recommended)

If peekaboo is installed, use it for more reliable UI interaction:

# Open Find My
osascript -e 'tell application "FindMy" to activate'
sleep 3

# Capture and annotate the UI
peekaboo see --app "FindMy" --annotate --path /tmp/findmy-ui.png

# Click on a specific device/item by element ID
peekaboo click --on B3 --app "FindMy"

# Capture the detail view
peekaboo image --app "FindMy" --path /tmp/findmy-detail.png

Then analyze with vision:

vision_analyze(image_url="/tmp/findmy-detail.png", question="What is the location shown for this device/item? Include address and coordinates if visible.")

Workflow: Track AirTag Location Over Time

For monitoring an AirTag (e.g., tracking a cat's patrol route):

# 1. Open FindMy to Items tab
osascript -e 'tell application "FindMy" to activate'
sleep 3

# 2. Click on the AirTag item (stay on page — AirTag only updates when page is open)

# 3. Periodically capture location
while true; do
    screencapture -w -o /tmp/findmy-$(date +%H%M%S).png
    sleep 300  # Every 5 minutes
done

Analyze each screenshot with vision to extract coordinates, then compile a route.

When not to use it

The source does not list alternatives or exclusions. However, this skill is macOS-only and requires the Find My app to be open and visible. If you need to track devices on iOS or from a remote server, this skill will not work.

Limits and gotchas

• Find My has no CLI or API, must use UI automation.
• AirTags only update location while the Find My page is actively displayed.
• Location accuracy depends on nearby Apple devices in the Find My network.
• Screen Recording permission is required for screenshots.
• AppleScript UI automation may break across macOS versions.
• Keep Find My app in the foreground when tracking AirTags (updates stop when minimized).
• Use vision_analyze to read screenshot content, don't try to parse pixels.
• For ongoing tracking, use a cronjob to periodically capture and log locations.
• Respect privacy, only track devices/items the user owns.

Related skills

This skill pairs with other Apple automation skills in Hermes Agent, such as Apple Mail, Apple Messages, and Apple Notes, for a broader macOS automation workflow.