@pmndre/pine 0.0.0-stage → 0.22.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +226 -2
- package/bin/pine.js +27 -0
- package/docs/SKILL.md +234 -0
- package/docs/references/agents.md +454 -0
- package/docs/references/coding.md +311 -0
- package/docs/references/computer.md +212 -0
- package/docs/references/configuration.md +310 -0
- package/docs/references/events.md +267 -0
- package/docs/references/mcp.md +420 -0
- package/docs/references/operations.md +394 -0
- package/docs/references/overview.md +195 -0
- package/docs/references/security.md +896 -0
- package/docs/references/setup.md +385 -0
- package/docs/references/sources.md +772 -0
- package/docs/references/stack.md +121 -0
- package/docs/references/tui.md +575 -0
- package/examples/home/.env.example +7 -0
- package/examples/home/USER.md +4 -0
- package/examples/home/agents/main/agent.json +4 -0
- package/examples/home/agents/main/workspace/AGENTS.md +14 -0
- package/examples/home/agents/main/workspace/MEMORY.md +4 -0
- package/examples/home/agents/main/workspace/SOUL.md +4 -0
- package/examples/home/models.json +62 -0
- package/examples/home/pine.json +34 -0
- package/examples/home/skills/README.txt +7 -0
- package/examples/launchd/com.pine.daemon.plist +32 -0
- package/examples/systemd/pine.service +22 -0
- package/mcp/browser.d.ts +99 -0
- package/mcp/browser.js +496 -0
- package/mcp/commands.d.ts +20 -0
- package/mcp/commands.js +88 -0
- package/mcp/computer.d.ts +139 -0
- package/mcp/computer.js +698 -0
- package/mcp/computer.swift +989 -0
- package/mcp/lib.d.ts +23 -0
- package/mcp/lib.js +52 -0
- package/mcp/web.d.ts +6 -0
- package/mcp/web.js +140 -0
- package/package.json +32 -4
- package/src/access.d.ts +112 -0
- package/src/access.js +438 -0
- package/src/actions.d.ts +75 -0
- package/src/actions.js +167 -0
- package/src/admin.d.ts +167 -0
- package/src/admin.js +625 -0
- package/src/agent-admin.d.ts +24 -0
- package/src/agent-admin.js +55 -0
- package/src/agents.d.ts +540 -0
- package/src/agents.js +2776 -0
- package/src/background.d.ts +29 -0
- package/src/background.js +100 -0
- package/src/cli.d.ts +2 -0
- package/src/cli.js +777 -0
- package/src/computer.d.ts +30 -0
- package/src/computer.js +32 -0
- package/src/config.d.ts +396 -0
- package/src/config.js +549 -0
- package/src/dev/claude-model.d.ts +37 -0
- package/src/dev/claude-model.js +204 -0
- package/src/env-file.d.ts +4 -0
- package/src/env-file.js +40 -0
- package/src/goals.d.ts +133 -0
- package/src/goals.js +316 -0
- package/src/http.d.ts +46 -0
- package/src/http.js +509 -0
- package/src/limiter.d.ts +32 -0
- package/src/limiter.js +84 -0
- package/src/login.d.ts +23 -0
- package/src/login.js +130 -0
- package/src/manual.d.ts +61 -0
- package/src/manual.js +443 -0
- package/src/mcp-config.d.ts +49 -0
- package/src/mcp-config.js +182 -0
- package/src/models.d.ts +90 -0
- package/src/models.js +274 -0
- package/src/net.d.ts +50 -0
- package/src/net.js +150 -0
- package/src/pine.d.ts +32 -0
- package/src/pine.js +134 -0
- package/src/programs.d.ts +123 -0
- package/src/programs.js +412 -0
- package/src/router.d.ts +56 -0
- package/src/router.js +164 -0
- package/src/sandbox-fs.d.ts +34 -0
- package/src/sandbox-fs.js +186 -0
- package/src/sandbox-linux.d.ts +96 -0
- package/src/sandbox-linux.js +297 -0
- package/src/sandbox-setup.d.ts +70 -0
- package/src/sandbox-setup.js +216 -0
- package/src/security.d.ts +238 -0
- package/src/security.js +660 -0
- package/src/skills.d.ts +67 -0
- package/src/skills.js +163 -0
- package/src/snapshots.d.ts +68 -0
- package/src/snapshots.js +306 -0
- package/src/source-types.d.ts +61 -0
- package/src/source-types.js +1 -0
- package/src/sources/command.d.ts +4 -0
- package/src/sources/command.js +77 -0
- package/src/sources/folder.d.ts +6 -0
- package/src/sources/folder.js +116 -0
- package/src/sources/ftp.d.ts +4 -0
- package/src/sources/ftp.js +307 -0
- package/src/sources/http.d.ts +4 -0
- package/src/sources/http.js +201 -0
- package/src/sources/mqtt.d.ts +18 -0
- package/src/sources/mqtt.js +97 -0
- package/src/sources/telegram.d.ts +22 -0
- package/src/sources/telegram.js +283 -0
- package/src/sources/webhook.d.ts +4 -0
- package/src/sources/webhook.js +52 -0
- package/src/sources.d.ts +344 -0
- package/src/sources.js +966 -0
- package/src/store.d.ts +99 -0
- package/src/store.js +277 -0
- package/src/timers.d.ts +83 -0
- package/src/timers.js +318 -0
- package/src/tools.d.ts +11 -0
- package/src/tools.js +1297 -0
- package/src/tui/app.d.ts +305 -0
- package/src/tui/app.js +3195 -0
- package/src/tui/client.d.ts +236 -0
- package/src/tui/client.js +131 -0
- package/src/tui/code-pane.d.ts +68 -0
- package/src/tui/code-pane.js +535 -0
- package/src/tui/highlight.d.ts +8 -0
- package/src/tui/highlight.js +86 -0
- package/src/tui/picture.d.ts +206 -0
- package/src/tui/picture.js +801 -0
- package/src/tui/run.d.ts +8 -0
- package/src/tui/run.js +97 -0
- package/src/tui/theme.d.ts +32 -0
- package/src/tui/theme.js +128 -0
- package/src/tui/undo.d.ts +5 -0
- package/src/tui/undo.js +50 -0
- package/src/tui/voice.d.ts +49 -0
- package/src/tui/voice.js +155 -0
- package/src/tui/voice.swift +223 -0
- package/src/types.d.ts +139 -0
- package/src/types.js +1 -0
- package/src/undo.d.ts +224 -0
- package/src/undo.js +465 -0
- package/src/update.d.ts +35 -0
- package/src/update.js +130 -0
- package/src/util.d.ts +65 -0
- package/src/util.js +171 -0
- package/src/watching.d.ts +20 -0
- package/src/watching.js +68 -0
- package/src/workspace-view.d.ts +72 -0
- package/src/workspace-view.js +221 -0
|
@@ -0,0 +1,311 @@
|
|
|
1
|
+
# Working on code
|
|
2
|
+
|
|
3
|
+
Two things turn a conversation into a place to work on a project: a **folder** that is opened
|
|
4
|
+
for it, and the **code pane** beside it that shows what git says changed there, the files, and
|
|
5
|
+
what the agent's tool calls printed. A message whose result you do not want is taken back
|
|
6
|
+
with `/undo`, files included.
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
cd ~/dev/shop
|
|
10
|
+
pine .
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+

|
|
14
|
+
|
|
15
|
+
## A folder for a conversation
|
|
16
|
+
|
|
17
|
+
An agent works in its own workspace. A conversation can be given a folder of yours on top of
|
|
18
|
+
that: a repository, a project. From its next message on
|
|
19
|
+
|
|
20
|
+
- the agent's shell and file tools start in that folder (a relative path means a path there),
|
|
21
|
+
- it may read and change what is in it, like its own workspace,
|
|
22
|
+
- the folder's instruction files (`AGENTS.md`, `CLAUDE.md`) apply, next to the agent's own,
|
|
23
|
+
- the agent is told which folder it works in and that its own workspace is still there.
|
|
24
|
+
|
|
25
|
+
Everything else stays as it is: for an agent in the sandbox (`safe` or `manual` mode) the rest
|
|
26
|
+
of your home is closed and the network is closed unless you opened it, and the agent's mode
|
|
27
|
+
decides what it asks before doing.
|
|
28
|
+
|
|
29
|
+
The main agent works in `auto` mode and reaches your whole account anyway (see "Modes" in
|
|
30
|
+
[security.md](security.md)), so it can work on a project without a folder being opened.
|
|
31
|
+
Opening one still does two things for it: its tools start there, and the folder's instruction
|
|
32
|
+
files apply.
|
|
33
|
+
|
|
34
|
+
The folder belongs to that one conversation. Another conversation of the same agent does not
|
|
35
|
+
have it. Sub-agents it starts do, and so does another agent for the length of one task that
|
|
36
|
+
is handed to it from this conversation (`delegate` without a thread): a specialist has to see
|
|
37
|
+
the project it is asked about. A conversation that other agent keeps does not get it. It stays until
|
|
38
|
+
you take it away or remove the conversation, also across restarts. A renamed conversation
|
|
39
|
+
keeps it.
|
|
40
|
+
|
|
41
|
+
There are four ways to open one:
|
|
42
|
+
|
|
43
|
+
| | |
|
|
44
|
+
|---|---|
|
|
45
|
+
| `pine .` or `pine <folder>` | Starts the terminal UI with the tabs of last time and one more on that folder: a conversation of the main agent, named after the folder. If a conversation already works there, that one is opened again |
|
|
46
|
+
| The folder in the bottom bar | Click it (`no folder`, or the folder's path) for a list: no folder, this conversation's folder, the folder pine was started in (unless that is your home folder), folders other conversations of this agent use, and "Pick a folder …", which opens the system's folder chooser on macOS |
|
|
47
|
+
| `/folder <path>`, `/folder none` | The same by command (`off` and `-` mean `none` too). `/folder` alone opens the list |
|
|
48
|
+
| Drag a folder onto the terminal | It becomes the folder of the conversation in front. (A file dropped there is attached to your next message, as always) |
|
|
49
|
+
|
|
50
|
+

|
|
51
|
+
|
|
52
|
+
Only you can do this. There is no tool for it: an agent cannot open a folder for itself, and
|
|
53
|
+
nothing an agent reads can.
|
|
54
|
+
|
|
55
|
+
A folder is refused when it is, contains or lies in pine's home (your home folder contains it,
|
|
56
|
+
if pine's home is `~/.pine`), pine's own program, or another agent's workspace. It cannot be
|
|
57
|
+
changed while the conversation is working.
|
|
58
|
+
|
|
59
|
+
For a project an agent should always work in, without opening anything, set `workspace` in its
|
|
60
|
+
`agent.json` instead ([agents.md](agents.md)).
|
|
61
|
+
|
|
62
|
+
## The code pane
|
|
63
|
+
|
|
64
|
+
`▤ code` at the right end of the tab bar, `ctrl+g` or `/code` opens a pane beside the
|
|
65
|
+
conversation; the same closes it. It shows the conversation's folder, or the agent's workspace
|
|
66
|
+
when no folder is open. In a terminal of 100 columns or more it takes half the width (at most
|
|
67
|
+
96 columns); in a narrower one it takes the place of the conversation until it is closed.
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
pine 1 main × 2 main·shop × + ▤ code
|
|
71
|
+
───────────────────────────────────────────────────────────────────────────────────────────
|
|
72
|
+
make the total include shipping │ Changes Files Output Running ↻ ×
|
|
73
|
+
│ main · 2 changed · ~/dev/shop
|
|
74
|
+
✓ edit src/cart.ts 12 ms │─────────────────────────────────────────────────
|
|
75
|
+
✓ bash npm test 2.1 s │ M src/cart.ts +4 −1
|
|
76
|
+
│ + src/shipping.ts +12
|
|
77
|
+
The total now includes shipping ... │
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
| View | Shows | A click |
|
|
81
|
+
|---|---|---|
|
|
82
|
+
| **Changes** | What differs from the last commit: changed (`M`), added (`A`), new (`+`), deleted (`D`) and renamed (`R`) files with the lines added and removed, and the branch. Its first line switches to **by your last message**: the same list and diffs, counted from before your last message instead of from the last commit | on a file shows its diff, added lines green and removed ones red; `‹ back` returns to the list; `▸ by your last message` and `▸ since the last commit` switch between the two |
|
|
83
|
+
| **Files** | The folder as a tree | on a folder opens or closes it; on a file shows its text with line numbers and colours, or a picture as a picture |
|
|
84
|
+
| **Output** | Every tool call of the conversation's last 8 messages, newest first: the tool, what it was called with, how long it took | on a call shows everything it printed, not only the first lines the conversation shows |
|
|
85
|
+
| **Running** | What the conversation's commands started and left running: a development server, a watcher. Each with its command, the ports it listens on and how long it has run. A command the agent put into the background is listed too, with the file its output goes to | on `■ stop` ends it |
|
|
86
|
+
|
|
87
|
+
The pane follows the agent: when a tool call finishes, what is shown is read again, so a diff
|
|
88
|
+
grows while the agent edits. `↻` reads it again by hand, `×` closes the pane. The wheel scrolls
|
|
89
|
+
whichever side the pointer is over. Whether the pane was open is remembered for the next start.
|
|
90
|
+
|
|
91
|
+
`/code changes`, `/code files`, `/code output` and `/code running` open it on that view;
|
|
92
|
+
`/code turn` opens the changes of your last message; `/code off` closes it.
|
|
93
|
+
|
|
94
|
+
### What an agent left running
|
|
95
|
+
|
|
96
|
+
An agent that works on a web page starts its development server and leaves it running
|
|
97
|
+
(`npm run dev &`). pine notes every command's process group and so knows what is still there
|
|
98
|
+
when a command has ended, whatever the agent says about it. For each conversation:
|
|
99
|
+
|
|
100
|
+
- **Running** in the code pane lists it, and `■ stop` ends it (asked first, ended two seconds
|
|
101
|
+
later if it is still there).
|
|
102
|
+
- The agent is told in its next message what it left running and on which ports, and has
|
|
103
|
+
`stop_program` to end one. Ask it: "stop the servers you started".
|
|
104
|
+
- Resetting or removing the conversation stops what it left running. Quitting pine does not:
|
|
105
|
+
the programs are listed again at the next start. `/undo` does not either: it stops the
|
|
106
|
+
agent's work on a message, not a server that work started.
|
|
107
|
+
|
|
108
|
+
A program that detaches itself completely (a daemon that starts a session of its own) is not
|
|
109
|
+
found this way. On Linux, in the sandbox (`safe` or `manual` mode), nothing outlives its
|
|
110
|
+
command in the first place; a server there is started in the background (next section), or
|
|
111
|
+
by an agent in `auto` mode.
|
|
112
|
+
|
|
113
|
+
### Commands in the background
|
|
114
|
+
|
|
115
|
+
A shell command is waited for until it ends, and it is stopped when it reaches the time limit
|
|
116
|
+
(`bashTimeoutSec`). A long-running build, a test run, an install or a server does not fit
|
|
117
|
+
into that. Such a command the agent starts in the background: it does not wait for it to
|
|
118
|
+
finish, goes on working meanwhile, and is told when the command is done. For this the agent's
|
|
119
|
+
shell tool takes `background`:
|
|
120
|
+
|
|
121
|
+
| Name | What it does |
|
|
122
|
+
|---|---|
|
|
123
|
+
| `background` | An argument of the `bash` tool. With `background: true` the command is started and the call returns at once, with the command's id and the file its output goes to. The agent does not wait for it: pine tells the conversation when the command has ended. The time limit does not apply to it. |
|
|
124
|
+
| `timeout` | An argument of the `bash` tool: how many seconds a command that is waited for may take. Never more than `bashTimeoutSec`. It does not apply to a command in the background. |
|
|
125
|
+
|
|
126
|
+
The agent's `stop_program` tool ends a command in the background before it ends by itself, by
|
|
127
|
+
its id; without an id it lists what is running, with each output file. A command that was
|
|
128
|
+
waited for and reached the time limit is stopped as before; the agent is then told that
|
|
129
|
+
`background: true` would have given it the time.
|
|
130
|
+
|
|
131
|
+
What happens, step by step:
|
|
132
|
+
|
|
133
|
+
- The command runs in the same sandbox as every other command of that conversation, and starts
|
|
134
|
+
in the same folder. It cannot do anything in the background that it could not do otherwise.
|
|
135
|
+
- Everything it prints goes to a file in the agent's own temp folder
|
|
136
|
+
(`sandbox/<agent>/tmp/background/<name>.log` in pine's home). The agent reads it with its
|
|
137
|
+
ordinary file tools, also while the command runs. pine does not delete these files.
|
|
138
|
+
- When the command ends, the conversation that started it gets a message from pine, not from
|
|
139
|
+
you: the id, the command, how it ended (the exit code, a signal, or that you stopped it), how
|
|
140
|
+
long it ran, and its last lines. The agent then goes on with the task, and you get its answer
|
|
141
|
+
where you got the others (the terminal UI, Telegram).
|
|
142
|
+
- Commands that end while the agent is at work, or within a moment of each other, are told in
|
|
143
|
+
one message, when the agent has nothing else to do.
|
|
144
|
+
- A command that ends and leaves a program running (`npm run dev &`) is told as ended, and the
|
|
145
|
+
message says that something is still running. It is then listed like anything else an agent
|
|
146
|
+
left running.
|
|
147
|
+
- While such a command runs and the agent's turn is over, the status line under the
|
|
148
|
+
conversation says so (`● a command runs in the background: brew install ... · 3 min`), so
|
|
149
|
+
that a long install does not look like a hang. It says the same for a server or another
|
|
150
|
+
program that was left running.
|
|
151
|
+
- **Running** in the code pane lists a command in the background with its output file, and
|
|
152
|
+
`■ stop` ends it. The agent is then told that you stopped it. What the agent stops itself
|
|
153
|
+
with `stop_program` it is not told again.
|
|
154
|
+
- Resetting or removing the conversation ends its background commands, and nothing is told.
|
|
155
|
+
- Quitting pine does not end them. At the next start a command that is still running is listed
|
|
156
|
+
again and told when it ends; one that ended while pine was not running is told at once, with
|
|
157
|
+
its exit code. (Only when it was ended from outside while pine was away, with a signal it
|
|
158
|
+
could not outlive, the message says that how it ended is not known.)
|
|
159
|
+
|
|
160
|
+
What the commands print is shown to the agent as output, every line marked, and never as
|
|
161
|
+
something you said. If the conversation had read outside content when it started the command,
|
|
162
|
+
the message about its end counts as outside content too.
|
|
163
|
+
|
|
164
|
+
Limits: at most 8 commands of one conversation run in the background at a time. A run that an
|
|
165
|
+
outside event started cannot use `background`, and neither can a run of its own (the session
|
|
166
|
+
`new`, or a task that another agent handed over without a thread): there is no conversation
|
|
167
|
+
of yours that could be told. A sub-agent can; the conversation it works for is told. On Linux
|
|
168
|
+
a conversation that has read outside content cannot either ([security.md](security.md), "On
|
|
169
|
+
Linux"). A command that was waited for and reached the time limit is not moved into the
|
|
170
|
+
background: it is stopped, and has to be started again with `background: true`.
|
|
171
|
+
|
|
172
|
+
Apart from that stop, the pane only shows. It changes no file, runs no command of the project and sends nothing to a
|
|
173
|
+
model. Long lines wrap. A file is shown up to 256 KB, a diff up to 200 000 characters, a tool's
|
|
174
|
+
output up to 200 000 characters and 4000 lines; binary files are named, not shown. Changes
|
|
175
|
+
lists the first 300 files, and Files shows up to 500 entries of a folder.
|
|
176
|
+
|
|
177
|
+
While the pane is open, pictures and videos in the conversation are drawn with coloured half
|
|
178
|
+
blocks also on terminals that can show real pictures. A terminal draws a real picture only at
|
|
179
|
+
the left edge of a full line, where it would lie across the pane.
|
|
180
|
+
|
|
181
|
+
## Taking a message back
|
|
182
|
+
|
|
183
|
+
`/undo` takes back your last message in a conversation, with everything it led to:
|
|
184
|
+
|
|
185
|
+
1. What is still running is stopped: the agent's own work, its sub-agents, and what it handed
|
|
186
|
+
to other agents for that message.
|
|
187
|
+
2. The files of the conversation's folder go back to what they were before the message was
|
|
188
|
+
worked on (the agent's workspace when no folder is open).
|
|
189
|
+
3. The conversation forgets the message and what followed it: the answer, the tool calls, and
|
|
190
|
+
whatever else arrived in the conversation since (an answer that came back from another
|
|
191
|
+
agent, a schedule that ran there).
|
|
192
|
+
4. The message's text is put back into the input, to be changed and sent again.
|
|
193
|
+
|
|
194
|
+
`/undo` again takes back the message before that one. `/redo` brings back what the last
|
|
195
|
+
`/undo` took, files and conversation; after several undos, one message at a time. It works
|
|
196
|
+
until something new is said in the conversation: a new message makes the undo final.
|
|
197
|
+
|
|
198
|
+
| Command | What it does |
|
|
199
|
+
|---|---|
|
|
200
|
+
| `/undo` | Take back your last message in this conversation: stop its work, restore the files it changed, make the conversation forget it and what followed, and put its text back into the input. A mistake of the agent's is undone this way |
|
|
201
|
+
| `/redo` | Bring back what the last `/undo` took, files and conversation. Works until a new message is sent |
|
|
202
|
+
| `/code turn` | Show in the code pane what your last message changed in the folder, as a list and as diffs: what `/undo` would restore |
|
|
203
|
+
|
|
204
|
+
The conversation says each time what happened: which message was taken back, how many files
|
|
205
|
+
were restored and which, what was left alone, and how to bring it back.
|
|
206
|
+
|
|
207
|
+
```
|
|
208
|
+
↶ taken back: “make the total include shipping”
|
|
209
|
+
2 files restored in ~/dev/shop (1 changed back, 1 removed): src/cart.ts, src/shipping.ts
|
|
210
|
+
not taken back: what commands did outside this folder, files git ignores or that are over 2 MB, and anything that was sent out
|
|
211
|
+
the message is in the input again, to change and send
|
|
212
|
+
/redo brings it back, until you send a new message
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
### What is restored, and what is not
|
|
216
|
+
|
|
217
|
+
pine records the folder before the first tool call of a message and again when its work ends.
|
|
218
|
+
A message that is only answered records nothing. The first record of a large folder takes a
|
|
219
|
+
moment, because every file is read once; after that only what changed is looked at. An undo sets the files that differ between
|
|
220
|
+
the two back to how they were before. Files the message's work did not change are left as
|
|
221
|
+
they are, so what you changed elsewhere in the folder since then stays. What changed in the
|
|
222
|
+
folder while the message was being worked on counts as its work, whoever changed it: you in
|
|
223
|
+
an editor, or another conversation in the same folder.
|
|
224
|
+
|
|
225
|
+
Not recorded, and so never restored:
|
|
226
|
+
|
|
227
|
+
- Files the folder's ignore rules leave out: its `.gitignore` files, those above it in its
|
|
228
|
+
repository, and the repository's `info/exclude`. That is `node_modules`, build output and
|
|
229
|
+
the like. Your global git ignore file does not count.
|
|
230
|
+
- Files larger than 2 MB. An undo names the ones it left alone.
|
|
231
|
+
- A git repository inside the folder (a clone, a submodule).
|
|
232
|
+
- A folder in which more than 50 000 files or 512 MB are new or changed at once. Then no
|
|
233
|
+
file is restored, and the undo says so; the conversation is taken back all the same.
|
|
234
|
+
|
|
235
|
+
An undo cannot take back at all:
|
|
236
|
+
|
|
237
|
+
- what commands did outside the folder: a package installed for your account, a file written
|
|
238
|
+
somewhere else, a row in a database;
|
|
239
|
+
- commits, branches and everything else inside `.git`: the files you see are restored, the
|
|
240
|
+
repository's history is not touched;
|
|
241
|
+
- anything that was sent out: a message, a request to a web service, an event;
|
|
242
|
+
- what the agent set up in pine with its tools (a source, a schedule, an MCP server, a
|
|
243
|
+
skill): ask it to remove that, or remove it yourself.
|
|
244
|
+
|
|
245
|
+
What the conversation was allowed while the message was worked on goes with the message: an
|
|
246
|
+
"always" in an approval dialog, access given with `request_access`, and the outside-content
|
|
247
|
+
mark, if this was the message that brought outside content in. The conversation is then what
|
|
248
|
+
it was before the message, as far as pine can make it so. A redo brings these back too.
|
|
249
|
+
|
|
250
|
+
Only you can undo, in the terminal UI or through the control API. No agent has a tool for
|
|
251
|
+
it. What an outside event started in a conversation is not a message of yours and is not
|
|
252
|
+
what `/undo` takes back. A message that still waits behind another one is withdrawn, and
|
|
253
|
+
what runs before it goes on.
|
|
254
|
+
|
|
255
|
+
The states are kept in pine's home under `data/snapshots`: git trees in a git directory of
|
|
256
|
+
pine's own for each folder. Your own repository, its index and its history are never
|
|
257
|
+
touched, and a folder that is no repository is recorded the same way. Recording needs git
|
|
258
|
+
and pine's sandbox (macOS, or Linux with bubblewrap); without them an undo takes back the
|
|
259
|
+
conversation and says that no file was restored. pine keeps the last 40 messages of a
|
|
260
|
+
conversation undoable, and removes states that no message refers to after a few days. What
|
|
261
|
+
was said in a message that was taken back stays in the conversation's session file on disk,
|
|
262
|
+
unused and not shown to the model, until the conversation is reset or removed.
|
|
263
|
+
|
|
264
|
+
## What it is safe against
|
|
265
|
+
|
|
266
|
+
- **Looking is not running.** A repository can name programs for git to start: a diff driver, a
|
|
267
|
+
text converter, hooks, a file system monitor. The pane runs git in a sandbox that can write
|
|
268
|
+
nothing and has no network, and with those features switched off, so a folder you only opened
|
|
269
|
+
to look at cannot make anything run.
|
|
270
|
+
- **Nothing outside the folder.** Paths are resolved with their links and must stay inside the
|
|
271
|
+
folder; a link that leads out of it is neither listed nor opened.
|
|
272
|
+
- **An undo writes the folder and nothing else.** The files are put back by git, which runs
|
|
273
|
+
in a sandbox that can write the folder and pine's own record of it, has no network, and
|
|
274
|
+
reads neither the folder's git settings nor yours: no hook, filter or line-ending rule
|
|
275
|
+
applies, and a file comes back byte for byte. A folder that an agent swapped for a link to
|
|
276
|
+
somewhere else is made a folder again; nothing is written through the link.
|
|
277
|
+
- **A folder is for your conversation, not for what arrives in it.** A run that an outside
|
|
278
|
+
event started, not you, does not get the folder: it works in the agent's own workspace. A
|
|
279
|
+
conversation of yours that has read outside content (a web page) keeps the folder, with the
|
|
280
|
+
two limits below. See [security.md](security.md).
|
|
281
|
+
- **Instruction files.** `AGENTS.md` and `CLAUDE.md` of an opened folder are instructions to
|
|
282
|
+
the agent, like those of its workspace. Open folders you trust as much as you trust their
|
|
283
|
+
authors with your agent. Once a conversation has read outside content, its file tools no
|
|
284
|
+
longer change these files, and in `safe` and `manual` mode the sandbox refuses a shell
|
|
285
|
+
command that writes them as well. In `auto` mode nothing checks a shell command for them.
|
|
286
|
+
- **Git hooks and configuration.** What lies in `.git/hooks` and `.git/config` runs with your
|
|
287
|
+
rights the next time you use git in that folder. A conversation that has read outside content
|
|
288
|
+
can still commit and branch there, but its file tools do not change those, and in `safe` and
|
|
289
|
+
`manual` mode the sandbox refuses a shell command that writes them as well. One that has read none can, like
|
|
290
|
+
the rest of the folder: it acts on what you told it.
|
|
291
|
+
|
|
292
|
+
## From a script
|
|
293
|
+
|
|
294
|
+
The control API ([operations.md](operations.md)) has the same things:
|
|
295
|
+
|
|
296
|
+
| Request | Purpose |
|
|
297
|
+
|---|---|
|
|
298
|
+
| `GET /agents/<id>/folder?session=` | The folder of a conversation, and the agent's workspace |
|
|
299
|
+
| `GET /agents/<id>/folders` | Every conversation of the agent that has a folder |
|
|
300
|
+
| `POST /agents/<id>/folder` `{ session, folder }` | Open a folder for a conversation; `folder: null` closes it |
|
|
301
|
+
| `GET /agents/<id>/view/changes?session=` | What git says changed in the conversation's folder |
|
|
302
|
+
| `GET /agents/<id>/view/diff?session=&path=` | The diff of one file (path from the top of the repository) |
|
|
303
|
+
| `GET /agents/<id>/view/tree?session=&path=` | What is in one folder (path from the conversation's folder) |
|
|
304
|
+
| `GET /agents/<id>/view/file?session=&path=` | A file's text |
|
|
305
|
+
| `GET /agents/<id>/view/find?session=&q=` | The files of the conversation's folder whose name fits a few letters: what `@` offers |
|
|
306
|
+
| `GET /tasks/<id>/transcript?full=1` | A task's steps with the whole output of each tool call |
|
|
307
|
+
| `GET /agents/<id>/programs?session=`, `POST /agents/<id>/programs/stop` `{ session, id }` | What the conversation's commands left running, background commands among them, and stopping one |
|
|
308
|
+
| `POST /agents/<id>/undo` `{ session }` | Take back your last message in a conversation. The answer has the message's text (`prompt`), what was restored (`restored`: `root`, the counts `changed`, `removed` and `back`, the first `files`, and `left` for files too large to record), a `note` when no file was restored, `stopped`, how many messages a redo can bring back (`redo`), and whether there is another message to take back (`more`) |
|
|
309
|
+
| `POST /agents/<id>/redo` `{ session }` | Bring back what the last undo took. The same answer |
|
|
310
|
+
| `GET /agents/<id>/view/turn?session=` | What your last message changed in the conversation's folder: `files` with their line counts, or a `note` saying why there is nothing |
|
|
311
|
+
| `GET /agents/<id>/view/turndiff?session=&path=` | The difference of one file over your last message (path from the conversation's folder) |
|
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
# App control: operating the apps on a Mac
|
|
2
|
+
|
|
3
|
+
The bundled MCP server `computer` lets an agent use the apps on the computer pine runs on, the
|
|
4
|
+
way a person at the screen does: open an app, read a window, press a button, fill in a field,
|
|
5
|
+
pick from a menu, click, type. It is for things that have no command line and no web page: a
|
|
6
|
+
note in Notes, a setting in an app, a document in TextEdit, an app the owner built and wants
|
|
7
|
+
tried out.
|
|
8
|
+
|
|
9
|
+
It works on macOS only. On Linux and Windows the server is not started and the tools are not
|
|
10
|
+
there; `pine check` says so on the agent's `apps` line.
|
|
11
|
+
|
|
12
|
+
```json
|
|
13
|
+
"computer": { "pine": "computer" }
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
`pine init` writes this entry into `pine.json`. To take app control away, remove the entry or
|
|
17
|
+
set `"enabled": false` on it.
|
|
18
|
+
|
|
19
|
+
## Who has it
|
|
20
|
+
|
|
21
|
+
If an agent does not have the tools to operate apps in its list, the reason is in this table:
|
|
22
|
+
who may operate apps, and who may not.
|
|
23
|
+
|
|
24
|
+
| Who | App control |
|
|
25
|
+
|---|---|
|
|
26
|
+
| The main agent, in a conversation with its owner | Yes. The owner is asked once per app (below) |
|
|
27
|
+
| Another agent whose mode is `auto` or `bypass`, in a conversation with its owner | Yes, the same way |
|
|
28
|
+
| An agent that works in its sandbox: `safe` or `manual` mode (every agent but the main one, unless configured otherwise) | No. The tools are not in its list. Operating apps is acting with the owner's whole account, which such an agent does not have |
|
|
29
|
+
| A run that an event from outside started (a mail, a webhook, a message from somebody else) | Never, whatever the agent's `untrusted` profile lists |
|
|
30
|
+
| Sub-agents of a conversation that has it | Yes, when the agent's `subagents.mcp` is on; they share the conversation's answers |
|
|
31
|
+
|
|
32
|
+
An agent without the tools cannot get them by asking: `request_access` does not give app
|
|
33
|
+
control. The owner gives it by putting the agent in `auto` mode (`/mode auto`, or `mode` in
|
|
34
|
+
its `agent.json`; [security.md](security.md), "The whole account"). To be asked about every
|
|
35
|
+
action in an app, not once per app, set `"approve": "always"` on the server.
|
|
36
|
+
|
|
37
|
+
## Two ways to work
|
|
38
|
+
|
|
39
|
+
The tools serve a model that only reads text and a model that can see pictures.
|
|
40
|
+
|
|
41
|
+
**By outline** (any model). `app_snapshot` returns a window as indented text taken from the
|
|
42
|
+
accessibility tree, the description every Mac app keeps of its own windows for screen readers:
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
TextEdit, window 1 "Untitled".
|
|
46
|
+
Controls with [ref=...] can be used: app_press presses one, app_set_text writes into a text field.
|
|
47
|
+
|
|
48
|
+
- close button [ref=e4]
|
|
49
|
+
- toolbar
|
|
50
|
+
- button "Bold" [ref=e7]
|
|
51
|
+
- pop-up button "Font" [ref=e8]: "Helvetica"
|
|
52
|
+
- text area [ref=e2] (focused): "Dear Ada"
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
A control that can be used has a reference. `app_press` presses one, `app_set_text` writes into
|
|
56
|
+
a text field, `app_menu` picks from the menu bar by name. A control keeps its reference from
|
|
57
|
+
one snapshot to the next for as long as it exists. After something has changed (a dialog
|
|
58
|
+
opened, another view), take a new snapshot.
|
|
59
|
+
|
|
60
|
+
**By picture** (a model that can see). `app_screenshot` returns a picture of a window, or of
|
|
61
|
+
the whole screen, and says how large the picture is. `app_click`, `app_drag` and `app_scroll`
|
|
62
|
+
take positions in that picture: `x` and `y` are pixels counted from its top left corner. pine
|
|
63
|
+
turns them into the right point on the screen, whatever the display's resolution and wherever
|
|
64
|
+
the window is, also on a second display. A window that was moved after its picture was taken
|
|
65
|
+
is clicked where it is now.
|
|
66
|
+
|
|
67
|
+
A picture of a window shows that window alone, also while another app covers it, and taking it
|
|
68
|
+
does not bring the app to the front. When the app has something open over the window (a menu, a
|
|
69
|
+
sheet, a bubble), the picture holds that too, also where it reaches beyond the window, and the
|
|
70
|
+
answer says "and what is open over it". For such a picture the app is brought to the front, and
|
|
71
|
+
what lies beside the app on the screen is left grey: a picture of one app does not show another.
|
|
72
|
+
|
|
73
|
+
**For both.** `app_type` types text where the cursor is, `app_key` presses a key or a
|
|
74
|
+
shortcut, `app_open` starts an app or brings it to the front, `apps` lists what is running.
|
|
75
|
+
|
|
76
|
+
Use the outline wherever it works: it is exact, small, and needs no eyes. Take a picture when
|
|
77
|
+
the outline does not show what matters (a drawing, a map, a custom view, an app that describes
|
|
78
|
+
its window poorly), or to check what something looks like.
|
|
79
|
+
|
|
80
|
+
## The tools
|
|
81
|
+
|
|
82
|
+
They appear as `mcp__computer__apps`, `mcp__computer__app_open` and so on. Every tool but
|
|
83
|
+
`apps` takes `app`: the app's name as `apps` lists it (`TextEdit`). Capital letters do not
|
|
84
|
+
matter; a bundle id (`com.apple.TextEdit`) works too.
|
|
85
|
+
|
|
86
|
+
| Tool | What it does |
|
|
87
|
+
|---|---|
|
|
88
|
+
| `apps` | Lists the running apps and their windows, the one in front first, and the screens when there are several. Not asked about |
|
|
89
|
+
| `app_open` | Opens an app by name, or brings it to the front when it runs already. A window that was put away into the Dock comes back. Takes a name, not a path |
|
|
90
|
+
| `app_snapshot` | Reads a window as an outline with references. `window`: its number or a part of its title when the app has several; otherwise the front one. Long windows are cut at 24,000 characters; tables show their visible rows |
|
|
91
|
+
| `app_press` | Presses the control with the reference `ref`: a button, a checkbox, a link, a tab. A row of a list is selected. A control that has nothing to press by name is clicked in its middle |
|
|
92
|
+
| `app_set_text` | Makes the text field with the reference `ref` hold `text`, replacing what was in it. A search or address field acts only on the return key: `app_key` with `return` afterwards |
|
|
93
|
+
| `app_menu` | Brings the app to the front and picks an entry from its menu bar: `path` is the menu names from the top down, separated by ` > ` (`File > Save`, `Format > Font > Bold`). Without `path` it lists the menus, with the name of a menu what is in it, with shortcuts |
|
|
94
|
+
| `app_screenshot` | Takes a picture of an app's window (`app`, and `window` like `app_snapshot`), or of the whole screen without `app` (`screen`: which one, when there are several). The answer gives the picture's size in pixels. An open menu or sheet of the app is in the picture; other apps are not |
|
|
95
|
+
| `app_click` | Clicks at `x`, `y` in the last picture. `count`: 2 for a double click, 3 for a triple one. `button`: `"right"` for the context menu |
|
|
96
|
+
| `app_drag` | Presses the mouse button at `x`, `y` in the last picture, moves to `to_x`, `to_y` and lets go |
|
|
97
|
+
| `app_scroll` | Turns the scroll wheel over `x`, `y` in the last picture. `direction`: `up`, `down`, `left` or `right`. `amount`: lines, 5 unless given |
|
|
98
|
+
| `app_type` | Types `text` on the keyboard into the app, where its cursor is. A line break presses the return key, which in a chat app sends the message. The app treats it as typing: it may capitalize or correct words as it does for a person. `app_set_text` writes exactly |
|
|
99
|
+
| `app_key` | Presses a key or shortcut in the app: `return`, `tab`, `escape`, `space`, `delete`, `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`, `f1` to `f12`, a single character, or these with `cmd`, `ctrl`, `alt`, `shift`, `fn` in front (`cmd+s`, `cmd+shift+z`). Several, separated by spaces, are pressed in order. The keyboard layout in use is respected |
|
|
100
|
+
|
|
101
|
+
Calls run one after another: there is one mouse and one keyboard. While an agent works in an
|
|
102
|
+
app, the app is in front on the owner's screen and the pointer moves; the owner sees all of it
|
|
103
|
+
and can take over at any time.
|
|
104
|
+
|
|
105
|
+
## What the owner is asked
|
|
106
|
+
|
|
107
|
+
App control acts as the owner, so the owner says where.
|
|
108
|
+
|
|
109
|
+
- **Once per app, per conversation.** The first call that names an app, whether it looks or
|
|
110
|
+
acts, is put to the owner: "operate the app TextEdit". A yes holds for that app in that
|
|
111
|
+
conversation and its sub-agents until the conversation is reset or the daemon restarts. After
|
|
112
|
+
it, nothing the agent does in that app is asked about again.
|
|
113
|
+
- **"Always here"** in the dialog stops the question for every app in that conversation.
|
|
114
|
+
- **The whole screen** (`app_screenshot` without `app`) is asked about once per conversation
|
|
115
|
+
in the same way: such a picture shows every app that is open.
|
|
116
|
+
- **`apps`** is not asked about.
|
|
117
|
+
- A **no** blocks the call. The agent says what it wanted to do and does not try another way.
|
|
118
|
+
- In **bypass** mode nothing is asked. In **manual** mode every action waits for a go-ahead as
|
|
119
|
+
all actions do there; looking (`apps`, `app_snapshot`, `app_screenshot`) does not, once the
|
|
120
|
+
app was allowed. In **plan** mode the agent only looks, after the same yes per app; every
|
|
121
|
+
other tool here (`app_open`, `app_press`, `app_set_text`, `app_menu` also when it would only
|
|
122
|
+
list, clicking, typing) is refused without a question.
|
|
123
|
+
- To be asked about every call to these tools, set `"approve": "always"` on the server
|
|
124
|
+
([mcp.md](mcp.md)).
|
|
125
|
+
|
|
126
|
+
What a window shows was written by whoever wrote the mail, the page or the document in it: it
|
|
127
|
+
is outside content. A conversation that has used app control is marked like one that has used
|
|
128
|
+
the browser ([security.md](security.md), "Taint and approvals").
|
|
129
|
+
|
|
130
|
+
The action lands in the app that was named, or not at all: a reference has to belong to that
|
|
131
|
+
app, a position has to lie in one of its windows (otherwise the answer says which app is
|
|
132
|
+
there), and keys are sent to that app's process and to no other, after it was brought to the
|
|
133
|
+
front: if the owner clicks into another window while the agent types, the rest of the text
|
|
134
|
+
does not end up there.
|
|
135
|
+
|
|
136
|
+
pine does not click, type or press in **the app pine itself runs in** (the terminal app the
|
|
137
|
+
daemon was started from): that is where the owner answers pine's questions. Looking at it is
|
|
138
|
+
allowed. To run a command, an agent uses its own shell.
|
|
139
|
+
|
|
140
|
+
There is no list of forbidden apps. An app that can run commands or show files (a terminal, the
|
|
141
|
+
Finder, an editor) reaches everything the owner reaches, including what the sandbox keeps from
|
|
142
|
+
the agent's shell. The yes for such an app is the owner's decision to make.
|
|
143
|
+
|
|
144
|
+
## What macOS has to be told, once
|
|
145
|
+
|
|
146
|
+
macOS lets a program operate other apps only when the owner has allowed it, per app that asks.
|
|
147
|
+
What asks here is the app pine runs in: the terminal app the daemon was started from (Terminal,
|
|
148
|
+
iTerm, Ghostty, an editor's terminal).
|
|
149
|
+
|
|
150
|
+
| Grant | Needed for | Where |
|
|
151
|
+
|---|---|---|
|
|
152
|
+
| Accessibility | Everything but `apps` and `app_open`: reading windows, pressing, clicking, typing | System Settings > Privacy & Security > Accessibility: switch on the app pine runs in |
|
|
153
|
+
| Screen Recording | `app_screenshot` | System Settings > Privacy & Security > Screen & System Audio Recording (older systems: Screen Recording): switch on the app pine runs in. macOS then wants that app quit and opened again, so pine has to be started again too |
|
|
154
|
+
|
|
155
|
+
When a grant is missing, the tool's answer says which one and where to click, and macOS puts
|
|
156
|
+
its own window on the screen that leads to the switch. The agent tells the owner and tries
|
|
157
|
+
again when they say it is done. Without Screen Recording, `app_snapshot` and everything that
|
|
158
|
+
works by reference still work.
|
|
159
|
+
|
|
160
|
+
## The helper, and what it needs
|
|
161
|
+
|
|
162
|
+
Node cannot reach the accessibility API or make mouse and key events. A small program does
|
|
163
|
+
that: `mcp/computer.swift`, which ships with pine as source. The first time app control is
|
|
164
|
+
used pine compiles it with Apple's `swiftc` and keeps the result in its home
|
|
165
|
+
(`data/computer/`); that takes up to half a minute, once, and again after an update of pine
|
|
166
|
+
that changes the helper. A call that arrives while it is being built is answered with "try
|
|
167
|
+
again in a moment".
|
|
168
|
+
|
|
169
|
+
The compiler comes with Apple's command line developer tools, which a Mac that has `git`
|
|
170
|
+
already has. Where they are missing, the tool's answer says so and what to type:
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
xcode-select --install
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
The helper is pine's only compiled part. It decides nothing: which app may be operated and
|
|
177
|
+
what an answer says is pine's.
|
|
178
|
+
|
|
179
|
+
## Options
|
|
180
|
+
|
|
181
|
+
Under `options` of the server's entry.
|
|
182
|
+
|
|
183
|
+
| Option | Default | Meaning |
|
|
184
|
+
|---|---|---|
|
|
185
|
+
| `images` | `true` | `false`: for a model that cannot see pictures. `app_screenshot`, `app_click`, `app_drag` and `app_scroll` are not offered, which leaves eight tools that work by outline |
|
|
186
|
+
| `imageSize` | `1280` | The longer side of a picture in pixels. Pictures are scaled down to it (and to about 1.15 million pixels) so that a model is sent what it will actually look at; positions are mapped back whatever the size |
|
|
187
|
+
| `block` | none | Tools to leave out, by name: `["app_type", "app_key"]` |
|
|
188
|
+
|
|
189
|
+
`description`, `enabled`, `agents`, `approve` and `timeout` work as for every MCP server
|
|
190
|
+
([mcp.md](mcp.md)).
|
|
191
|
+
|
|
192
|
+
## When something does not work
|
|
193
|
+
|
|
194
|
+
| What the agent is told | What it means |
|
|
195
|
+
|---|---|
|
|
196
|
+
| "macOS does not let pine operate apps yet" | The Accessibility grant is missing (above) |
|
|
197
|
+
| "macOS does not let pine take pictures of the screen yet" | The Screen Recording grant is missing (above) |
|
|
198
|
+
| "App control needs Apple's command line developer tools" | The compiler is missing: `xcode-select --install` |
|
|
199
|
+
| "pine is building its helper" | The first use; try again in a moment |
|
|
200
|
+
| "That reference does not exist any more" | The window changed since the snapshot: take a new `app_snapshot` |
|
|
201
|
+
| "At that position is ..., not ..." | Another app's window is where the click would land: take a new `app_screenshot`, or name that app |
|
|
202
|
+
| "... is the app pine itself runs in" | pine does not act in its own terminal app |
|
|
203
|
+
| "... could not be brought to the front" | Something holds the screen: a system dialog, an app in full screen |
|
|
204
|
+
| "not approved" | The owner said no to that app in this conversation |
|
|
205
|
+
| "Not done: you are in plan mode" | The agent's mode is `plan`: it may look at apps and acts in none. The owner switches the mode when the plan is to be carried out |
|
|
206
|
+
| "Operating apps uses your owner's whole account, which this conversation does not have" | The agent works in its sandbox in this conversation: its mode is `safe` or `manual`. The owner can switch it to `auto` mode |
|
|
207
|
+
| The tools are not in the agent's list | The agent is in `safe` or `manual` mode (it works in its sandbox), the run was started from outside, the system is not macOS, or the server is not configured |
|
|
208
|
+
|
|
209
|
+
An app that draws its window itself (some games, some cross-platform apps) describes little
|
|
210
|
+
of it: the outline is short or empty. Use pictures there. Apps built on a browser (Slack,
|
|
211
|
+
Visual Studio Code) describe their windows once they are asked, which pine does; the first
|
|
212
|
+
snapshot of such an app can be thinner than the second.
|