macOS 开源工具 bigarrow：为 AI 智能体提供屏幕视觉指引能力，实现无焦点抢占的精准 UI 交互引导

原标题：Let your AI agents paint big arrows, boxes and text on your screen

68AI 研判分

核心综述

bigarrow 是一款专为 AI 智能体设计的 macOS CLI 工具，通过 MIT 协议开源。它允许智能体在屏幕上绘制点击穿透、不抢占焦点的大箭头和文本框，以直观方式引导用户完成如“点击允许”等关键 UI 操作。该工具无需权限即可运行，解决了智能体仅靠终端文本提示难以有效引导用户进行图形界面交互的痛点。

报道全文原始报道全文

本文目录4 个章节

big-arrow-on-the-screen (bigarrow): one small macOS CLI and an agent skill. Click-through, never steals the focus, gone by itself. MIT.

Hacker News with five bigarrow arrows: the actual article is in here, finally an arrow bigger than this one, same design since 2007, today's thread already argued in 2014, not a lurker? click login

Your AI agent can refactor a monorepo, write a migration and explain monads, but when it needs you to click one button it prints "please click Allow in the dialog" into a terminal you are not looking at. bigarrow gives it a finger.

bigarrow pointing at a dialog's Allow button

BASH

bigarrow point --element "Allow" --app "System Settings" --text "Franz, click Allow"

A big, friendly arrow with a sign appears on top of everything, points at the thing, and goes away again. It is click-through, it never steals your focus, it works on every display and every Space, and it needs no permission at all to draw. It is one small Swift binary. There is no daemon, no menu-bar icon, no account, no telemetry, and, we checked twice, no AI inside. It is an arrow.

What is this actually for?

Fair question. Arrows have existed since roughly the Paleolithic. Here is what changed: software agents now do real work on your Mac, and they keep hitting the same wall, the part that only a human may do.

"Click Allow." macOS permission prompts, OAuth consent screens, "Open with...?" dialogs. The agent can find the button but must not, or cannot, press it for you. It can now point at it.
"Your turn." 2FA codes, CAPTCHAs, passkeys, a payment confirmation, a signature, a legal checkbox. The things an agent should never click on its own behalf. It points, you decide, it continues.
"It's this window, not that one." You have 14 Chrome windows. The agent knows which one it means: --window "Google Chrome:Pull request". It even picks the right tab: --app "Google Chrome:Pull request".
"I need you, and you're making coffee." --say reads the sign aloud. Your Mac will literally call you back to your desk.
Guided setups and onboarding. Walk a human through a settings pane step by step: start, wait until they acted, stop, next step. Like a product tour, minus the product.
Remote help. "No, the other gear icon." Point at it instead of describing it.
Demos, screencasts, docs. Highlight what matters while recording, or render the arrow straight into a PNG with --png for documentation.
Debugging coordinates. Not sure your Accessibility, screenshot or Peekaboo coordinates are right? Point at them and look. --dry-run --json tells you where it would point without drawing.

Situations we have all been in

Staged with a neutral demo dialog and recorded with the real bigarrow on a clean CI runner (BACKDROP_ARGS=--cover scripts/funny-scenes.sh). The dialogs are fake. The feelings are real.



`--color green`	`--shape zigzag --color orange`

`--color purple`	`--close-button`, because the human gets the last word

three `start`s, one button, zero ambiguity	`--style box --corners sharp`, plus a lesson about macOS permissions

`--shape spiral`: once around the sign, then to the button

What it is not: a screen annotator for humans, a click bot, or a screenshot tool. It never clicks, types or captures anything. It only points. Deliberately.

Install

BASH

brew install franzenzenhofer/tap/bigarrow
bigarrow install-skill          # teaches Claude Code (~/.claude/skills) and Codex (~/.agents/skills)

From source: swift build -c release (Xcode 16 or newer, macOS 14 or newer), binary at .build/release/bigarrow.

The three commands an agent needs

BASH

bigarrow point --element "Allow" --app "System Settings" --text "Franz, click Allow"   # by label
bigarrow point --at 760,500 --text "Franz, click HERE"                                 # by coordinate
bigarrow start --window "Safari:Inbox" --text "This window" && bigarrow stop            # until stopped

Every arrow ends by itself. Nobody has to clean up after an agent that forgot:


Time limit	`bigarrow point ... --duration 10` (default 8 s; `start` 300 s; `--duration 0` = no limit)
Start and stop	`bigarrow start ...` returns at once; `bigarrow stop` (or `stop --all`) removes it
The agent goes away	an arrow ends when the agent process that drew it exits (`CLAUDE_PID`, or `BIGARROW_OWNER_PID`)
The human answers	`bigarrow stop --hook` as a Claude Code `UserPromptSubmit` hook clears that session's arrows
The human closes it	`--close-button` puts a clickable X on the sign (opt-in)

Targets: --at X,Y, --rect X,Y,W,H, --mouse, --window App[:title], --element Label --app App, --peekaboo ID --snapshot see.json (from Peekaboo's see --json). Coordinates are global top-left logical points, the space Accessibility, CGWindowList and Peekaboo report. --display N makes --at and --rect relative to one display.

An arrow is tied to the app it points into. --app App[:window or tab title] (or --window) brings that app, window or Chrome/Safari tab to the front first, because pointing at a window hidden behind your terminal is a special kind of unhelpful; and while another app covers the target, the arrow hides and comes back with it. --no-raise leaves your windows alone. bigarrow elements --app X lists what --element can match. bigarrow doctor shows permissions, who owns them, and your displays.

Every command takes --json. Exit codes: 0 ok, 2 bad input, 3 target not found, 4 permission missing. Agents love exit codes. Humans tolerate them.

（注：更多技术实现细节与完整 API 文档请查阅原项目 README）