Pencil Agent
Pencil Agent
A Really Good Work Inc. project

Mark up your site on an iPad. A coding agent makes the change.

Draw on the live page, say what should change, and Claude Code edits the code on your Mac. You keep reviewing.

01 The problem

Small fixes still need a laptop.

You spot most problems on the real thing, often on a phone or iPad. The fix is usually small: a colour, a word, some spacing. But making it means opening the laptop, finding the file and switching context. So the fix waits, or gets lost.

02 How it works

Draw it, say it, it changes.

  1. 01Circle what you want changed. Apple Pencil on an iPad, or a finger on a phone.
  2. 02Say or type the change. “Make this our brand orange.”
  3. 03Your Mac gets the note, with the element, its text and its source file.
  4. 04Claude Code edits the file. The page reloads and the note turns green.
Fig. 01Demo · iPad
The demo site. Circle the headline, ask for it shorter, it updates.
03 The experience

Built to stay out of the way.

You’re reviewing, not operating a tool. Nothing asks you to stop.

Pencil draws
Fingers keep scrolling and tapping. On a phone, turn the pen on and use a finger.
Pause to finish
Draw as many strokes as you need. Stop for under a second and the note opens.
One pill
Type, talk or just send the sketch. A line above it says how it read your marks.
Status
Amber waitingBlue workingPurple questionGreen done
Keep moving
Mark the next thing while Claude works on the last one.
Tidy up
A tray lists every note. Close, reopen, or delete with five seconds to undo.

Voice, without paying for it.

Most voice features send your audio to a paid cloud service. This uses what’s already on your devices.

Keyboard mic
The dictation key on the iPad keyboard. Built in, free, no setup.
Whisper
whisper.cpp, an open-source speech model, runs on your Mac. Free, offline, the audio stays home.
Web Audio
The browser’s own audio tools drive the visual that moves with your voice.
04 Finding the element

How it knows what you circled.

A page is a tree of boxes and the browser knows where each one sits. So the overlay doesn’t look at pixels. It asks the browser.

  1. 01Ask what’s under the middle of the circle. Usually a word inside a button.
  2. 02Step out to the parent while the circle still covers 70% of it. The word becomes the button.
  3. 03Stop when it doesn’t fit. The section is too big, so the button is the answer.
Fig. 02Finding the element
section Shop notebooks 1the words: start here 2the button: circle covers it 3the section: too big, stop
Circle a button, get the button. Circle a card, get the card.

Then it collects what a developer would want: tag, classes, text, neighbours, a few styles and, in React, the component and source file.

05 Reading the marks

How it tells an arrow from a circle.

Everyone draws differently, so it doesn’t recognise pictures. It measures each stroke with loose rules that hold up in anyone’s hand.

Fig. 03Three rules
Ends meet → a loop: circle Hugs its edges → a box: add here Turns back at the tip → an arrow: move it
Measured, not recognised. Works in anyone’s hand.
Circle
The ends meet and the path is long. A loop.
Box
A loop whose points hug its edges.
Arrow
The point farthest from the start is the tip. The pen turns back from it to draw the head.
X
Two lines of similar size crossing near their middles.
Strike / underline
A flat line through something strikes it. Just below, it underlines it.
Handwriting
A scribble inside a box is its label.

Then the shape becomes an instruction. An arrow moves whatever sits at its tail to where the tip points, a whole card rather than its heading. A box in empty space adds something between the two elements it sits between.

It shows its reading before you send, like “Add after Link ‘See the paper’”. For anything beyond a circle, the agent also gets a picture: the page as outlines with your ink on top.

Fig. 04Sketch in, picture out
The demo page with a box sketched next to the two buttons, a scribble inside it, and an arrow from the third card to the first. The dark note pill below reads: Add after Link “See the paper”, with the note Add a Gift cards button, and lead with the fountain pen card.A · What you draw
The same area redrawn as grey outlines with labels, with the box, scribble and arrow on top.B · What the agent receives
06 The local server

The middle step, on your own Mac.

One Node file, nothing to install beyond Node. It sits between the iPad and Claude Code.

Serves
The drawing script your page loads. Nothing to install in your app.
Checks
Right token, right app, sensible size. Everything else is refused.
Holds
Notes in memory, numbered per page. Restart and they’re gone.
Streams
Each status change back to the page. No reloads.
Runs
Claude Code in your project, up to five notes at a time, two seconds after the last one.
Reports
Claude’s one line per note becomes the reply.
Code 01One note’s brief · plain text
[ann_c6j7hqC0Cmk] fix: Make this our brand orange
page: /
element: a.btn.btn-primary "Shop notebooks"
source: src/components/PrimaryButton.tsx:12
classes: btn btn-primary

That’s the whole brief for one note: about 60 tokens. Notes, sketches and voice stay on your Mac. Claude Code works in your project the same way it does at your desk, with your own login. Nothing runs until you send.

07 Security

Who gets to tell your computer to change code?

This lets a web page ask an agent to edit files on your Mac. Anyone who can reach the server could do the same. So only you can reach it, and the agent is limited even then.

Mac only
The server listens on 127.0.0.1. Other machines can’t connect. Tailscale hands requests in from inside the Mac.
Private network
Your iPad reaches the Mac over Tailscale. Only your signed-in devices can join.
Token
A long random key in a file only you can read. Every request carries it.
Your app only
The server answers your app’s address and refuses every other site.
Limited agent
Read, find, search, edit and write files. No terminal. No internet.

Why Tailscale.

Your iPad needs to reach your Mac. Opening a port to the internet invites the world. Home Wi-Fi only works at home, and anyone on it can reach the port. Tailscale is a private network between your own devices: install it on both, sign in once, and they connect, encrypted, from any network. It also gives the Mac a real HTTPS certificate, which Safari needs before a page can use the mic. The free plan covers it.

Without it: on a trusted Wi-Fi, run with HOST=0.0.0.0 and your Mac’s local IP. Plain HTTP, no mic button, home only. Not on café or office networks.

08 Try it

Run the demo.

Code 02Terminal
git clone https://github.com/reallygoodwork/pencil
cd pencil
npm run demo   # open localhost:5173, tap the pen

A Mac with Node 20 and Claude Code. The demo edits a copy of a sample site. The README covers your own project. Get the code

09 Studio

Made by Really Good Work Inc.

License
MIT. Use it for anything.Just keep the credit.Read it
Really Good Work Inc.