Mac developers comfortable with Xcode and API keys who want a working example of a screen-aware voice assistant; you need macOS 14.2+, Xcode 15+, Node 18+ and accounts for Cloudflare, Anthropic, AssemblyAI and ElevenLabs.

What can you build with clicky?

Fork it to build a voice tutor that watches your screen while you learn a new app. Study how a Swift menu bar app combines screen capture, streaming speech-to-text, an LLM and text-to-speech. Adapt the pointing tags to build guided walkthroughs that highlight buttons on screen. Swap in your own models or voices by editing the Cloudflare Worker proxy. Use it as a starting point for a company or product, since it is MIT licensed.

farzaa /

clicky

A Mac menu bar AI teacher that sees your screen, talks, and points at things

7.8k stars
64 stars since 3 Oct
Swift
Updated 6 months ago
Some experience
Productivity & personal

Open on GitHub (opens in a new tab)

Clicky is an AI buddy that sits next to your cursor on a Mac. You hold a keyboard shortcut and speak. It captures your screen, sends your words and the screenshot to Claude, and reads the answer back to you.

Under the hood, audio streams to AssemblyAI for transcription, Claude replies over streaming, and ElevenLabs turns the reply into speech. Claude can add a tag with screen coordinates to its answer, and the app uses it to fly a blue cursor overlay to the right control, even across several monitors. A small Cloudflare Worker holds the API keys so they never ship inside the app.

This repo is the open-source snapshot under the MIT license. The author says new work is now private and the latest version is a separate download, so treat this as a base to fork and tinker with, not the current product.

What it does

Push-to-talk voice input
Hold Control + Option to speak. Audio streams over a websocket to AssemblyAI for real-time transcription.
Screen-aware answers
Your transcript and a screenshot go to Claude, so it can answer questions about what you are looking at.
Cursor pointing
Claude can embed point tags in its reply, and a transparent overlay moves the cursor to that UI element, across multiple monitors.
Spoken replies
Responses play back through ElevenLabs text-to-speech.
Key-safe proxy
A Cloudflare Worker with three routes (/chat, /tts, /transcribe-token) keeps your Anthropic, AssemblyAI and ElevenLabs keys out of the app binary.

What you could build

Fork it to build a voice tutor that watches your screen while you learn a new app.
Study how a Swift menu bar app combines screen capture, streaming speech-to-text, an LLM and text-to-speech.
Adapt the pointing tags to build guided walkthroughs that highlight buttons on screen.
Swap in your own models or voices by editing the Cloudflare Worker proxy.
Use it as a starting point for a company or product, since it is MIT licensed.

Get started

Deploy the Cloudflare Worker with your three API keys, then replace the hardcoded proxy URLs in the Swift code with your Worker URL. Open the project in Xcode, set your signing team, run it, and grant the microphone, accessibility and screen recording permissions. The README also offers a Claude Code prompt that walks you through setup.