pi-superagent 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADAPTATION.md +73 -0
- package/LICENSE +21 -0
- package/NOTICE +19 -0
- package/README.md +151 -2
- package/config.example.env +81 -0
- package/docs/architecture.md +160 -0
- package/docs/design-decisions.md +230 -0
- package/docs/formal-verification.md +106 -0
- package/docs/operator-guide.md +137 -0
- package/formal/Intake.cfg +18 -0
- package/formal/Intake.tla +206 -0
- package/formal/README.md +31 -0
- package/formal/Ticket.cfg +23 -0
- package/formal/Ticket.tla +244 -0
- package/formal/Workers.cfg +20 -0
- package/formal/Workers.tla +152 -0
- package/launchd/com.example.superagent-health.plist +31 -0
- package/package.json +72 -4
- package/scripts/superagent +219 -0
- package/src/core/assignments.ts +140 -0
- package/src/core/engine.ts +270 -0
- package/src/core/errors.ts +61 -0
- package/src/core/evals.ts +340 -0
- package/src/core/ids.ts +5 -0
- package/src/core/index.ts +10 -0
- package/src/core/intake.ts +63 -0
- package/src/core/memory.ts +50 -0
- package/src/core/model.ts +353 -0
- package/src/core/part.ts +80 -0
- package/src/core/ports.ts +134 -0
- package/src/core/reasons.ts +38 -0
- package/src/core/replies.ts +95 -0
- package/src/core/stage.ts +26 -0
- package/src/core/tickets.ts +423 -0
- package/src/pi/assessor.ts +97 -0
- package/src/pi/authors.ts +10 -0
- package/src/pi/executor.ts +267 -0
- package/src/pi/extension.ts +703 -0
- package/src/pi/gh-api.ts +147 -0
- package/src/pi/herdr-workers.ts +299 -0
- package/src/pi/herdr.ts +91 -0
- package/src/pi/judge.ts +60 -0
- package/src/pi/log.ts +35 -0
- package/src/pi/outbox.ts +82 -0
- package/src/pi/pulls.ts +154 -0
- package/src/pi/receipts.ts +43 -0
- package/src/pi/report-tool.ts +35 -0
- package/src/pi/router.ts +125 -0
- package/src/pi/sdk-contract.ts +51 -0
- package/src/pi/server.ts +171 -0
- package/src/pi/sink-session-inbox.ts +28 -0
- package/src/pi/source-github.ts +209 -0
- package/src/pi/store.ts +132 -0
- package/src/pi/worker-role.ts +79 -0
- package/src/pi/worker-view.ts +80 -0
- package/src/pi/workspace.ts +155 -0
- package/vendor/dsh-superagent/LICENSE +21 -0
- package/vendor/dsh-superagent/UPSTREAM_SHA +1 -0
- package/vendor/dsh-superagent/docs/000-philosophy.md +33 -0
- package/vendor/dsh-superagent/docs/001-domain.md +112 -0
- package/vendor/dsh-superagent/docs/002-authorization.md +155 -0
- package/vendor/dsh-superagent/docs/003-execution.md +203 -0
- package/vendor/dsh-superagent/docs/004-prompts.md +96 -0
- package/vendor/dsh-superagent/docs/005-panel.md +58 -0
- package/vendor/dsh-superagent/docs/006-deployment.md +274 -0
- package/vendor/dsh-superagent/docs/007-architecture.md +617 -0
- package/vendor/dsh-superagent/docs/008-cookbook.md +425 -0
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
# Design decisions
|
|
2
|
+
|
|
3
|
+
Each section states a rule the system keeps and the failure that taught it. Where each rule lives is
|
|
4
|
+
mapped in [architecture.md](architecture.md); the proofs are in [formal-verification.md](formal-verification.md).
|
|
5
|
+
|
|
6
|
+
## GitHub is the only way in
|
|
7
|
+
|
|
8
|
+
Every request arrives as an issue, a comment or a review on one repository. There is no command-line
|
|
9
|
+
submission, no board and no label vocabulary.
|
|
10
|
+
|
|
11
|
+
Each extra door grows its own answers to "is this a person?" and "is this addressed to me?", and the
|
|
12
|
+
agent's behavior then depends on which door a request happened to use. With one door, the identity rules
|
|
13
|
+
live in one file (`source-github.ts`), and who is heard and who may start work in two functions
|
|
14
|
+
(`scopeOf` and `policyOf` in `router.ts`).
|
|
15
|
+
|
|
16
|
+
## Only people with write access are heard
|
|
17
|
+
|
|
18
|
+
The router hears an author only when GitHub says they can write to the repository: the OWNER, MEMBER and
|
|
19
|
+
COLLABORATOR author associations. Anyone else's issue, comment or review is stored as filtered and goes no
|
|
20
|
+
further: no judgement, no ticket, no input to a running worker. `PI_SUPERAGENT_ALLOWED` replaces the rule
|
|
21
|
+
with a list of logins.
|
|
22
|
+
|
|
23
|
+
On a public repository anyone can open an issue, and an accepted issue starts a worker with a shell on
|
|
24
|
+
the operator's machine. The judge cannot be that boundary, since a stranger's request reads like the
|
|
25
|
+
owner's. The admission gate cannot either: a comment on an issue that already has a ticket goes straight
|
|
26
|
+
to its worker and is never admitted again. So the check sits at the source scope, the engine's first step,
|
|
27
|
+
where it covers every way in. An author whose association is missing counts as having none.
|
|
28
|
+
|
|
29
|
+
## The first look is silent
|
|
30
|
+
|
|
31
|
+
The first time the source sees a repository, it records every open issue and fires none of them. Only
|
|
32
|
+
issues opened afterward are requests.
|
|
33
|
+
|
|
34
|
+
The first version, pointed at a repository with a backlog, fired every open issue at once. A backlog is
|
|
35
|
+
history, not a queue of fresh requests. Closing and reopening an issue, though, is a person saying "go
|
|
36
|
+
again": a closed issue leaves the seen set and fires when it returns, under a new trigger id, because the
|
|
37
|
+
id includes the issue's last update. The review cursor seeds the same way. (`Intake.tla` C1, C2.)
|
|
38
|
+
|
|
39
|
+
## Read positions commit late; seeds commit at once
|
|
40
|
+
|
|
41
|
+
A poll stages its read positions, and they commit at the start of the next poll, once the engine holds
|
|
42
|
+
every trigger they produced.
|
|
43
|
+
|
|
44
|
+
Committing at the moment of reading leaves a window: if the process dies between reading a comment and
|
|
45
|
+
handing it to the engine, the saved position already points past a request the engine never saw, and
|
|
46
|
+
that request is gone. A position that commits late can only cause a replay, and replays are free because
|
|
47
|
+
the engine accepts each trigger id once. A seed is the mirror image: losing one would replay the backlog,
|
|
48
|
+
and a seed can never swallow a request, so it commits immediately. The model checker found the early
|
|
49
|
+
commit in shipped code; the fix ships with the counterexample as a regression test. (`Intake.tla` C3, C6.)
|
|
50
|
+
|
|
51
|
+
## Code admits; the model judges
|
|
52
|
+
|
|
53
|
+
A policy gate in code admits a trigger only from a person who is not on the bot list. Only then does
|
|
54
|
+
the judge see it. When the judge accepts a thread that already has an open ticket, the router overrules
|
|
55
|
+
it and records an ignore.
|
|
56
|
+
|
|
57
|
+
A model's verdicts wobble; the deterministic layer keeps a wobble from turning bot traffic, the agent's
|
|
58
|
+
own comments or a duplicate into work. Both checks were learned. A trigger stored before bot names were
|
|
59
|
+
matched as patterns still said "human", and an agent asking the operator to approve a test became a ticket
|
|
60
|
+
to run it; the gate now checks the author again. And the judge once opened a second ticket on a thread
|
|
61
|
+
whose first ticket was right there in its prompt.
|
|
62
|
+
|
|
63
|
+
## The judge works alone, one question at a time
|
|
64
|
+
|
|
65
|
+
Judgements go to a headless pi session that loads no extensions, skills or context files, never to the
|
|
66
|
+
operator's own pane, and the assessor hands it one candidate per turn.
|
|
67
|
+
|
|
68
|
+
Judging in the operator's pane made the pane look frozen for every model turn. Sending candidates in a
|
|
69
|
+
burst was worse: 41 of them steered into one running turn came back out of order, and the tail expired
|
|
70
|
+
unanswered. The judge is also opened once and shared: opening a session takes 20 to 45 seconds, and
|
|
71
|
+
before that rule every eval arriving in the window opened its own.
|
|
72
|
+
|
|
73
|
+
## Workers report through a tool, never in prose
|
|
74
|
+
|
|
75
|
+
A worker ends its assignment by calling `ticket_report` with a state: ready, blocked, failed or
|
|
76
|
+
cancelled. Only *ready* completes a ticket.
|
|
77
|
+
|
|
78
|
+
A model that says "done" in prose while having done nothing must be unable to complete anything. In the
|
|
79
|
+
lifecycle model, completing without a *ready* report is not merely untested but unreachable
|
|
80
|
+
(`Ticket.tla` I2). A run that ends without a report completes nothing either: the worker files it as a
|
|
81
|
+
failure, and the person hears about it on the thread.
|
|
82
|
+
|
|
83
|
+
## Every ending reaches the thread
|
|
84
|
+
|
|
85
|
+
Whatever ends a worker's run is posted on the issue: its report, its question, a crash, a run that ended
|
|
86
|
+
without a report, a launch that never got going. Each message says what happened and what the person can
|
|
87
|
+
do next. The one exception is the 15-minute quiet flag, which stays on the panel: a long build is not news.
|
|
88
|
+
|
|
89
|
+
The person who asked is on GitHub, not on the panel. An ending they cannot see is a request that died in
|
|
90
|
+
silence, and a question they cannot see is a worker waiting forever. The same ending posts once per
|
|
91
|
+
assignment, so a launch that fails on every poll says so once, not every 30 seconds.
|
|
92
|
+
|
|
93
|
+
## A worker is fenced in
|
|
94
|
+
|
|
95
|
+
Inside a worker, a guard refuses delegation tools, refuses blocked commands, and refuses writes outside
|
|
96
|
+
the ticket's workspace. The workspace's setup script, which runs before pi does, meets the blocked
|
|
97
|
+
commands as refusing shims first on its PATH. The brief also tells the worker that the machine is shared.
|
|
98
|
+
|
|
99
|
+
Delegated work leaves the mandate, the workspace and the report behind. Blocked commands are the
|
|
100
|
+
operator's personal tools (`gh`, for instance), and the operator's identity is not the worker's to use.
|
|
101
|
+
The shared-machine brief exists because workers run side by side with each other and with people:
|
|
102
|
+
sequential builds, bounded parallel jobs, and no process left running after the report.
|
|
103
|
+
|
|
104
|
+
## A crash gets one fresh worker, not a loop
|
|
105
|
+
|
|
106
|
+
When a worker crashes, the router retries the ticket once on its own, through the engine's retry. The
|
|
107
|
+
crashed attempt ends on record, the comments it never read go to the new worker, and the new worker's brief
|
|
108
|
+
says what happened, because the old one's changes are still in the workspace. A second crash on the same
|
|
109
|
+
ticket waits for a person. Either way, the crashed worker's slot is freed at once.
|
|
110
|
+
|
|
111
|
+
A lost tab or a killed process is usually bad luck, and one retry rides it out without waking anyone. A
|
|
112
|
+
worker that crashes every time is a bug, and looping on it would open a new tab every few seconds while
|
|
113
|
+
hiding the problem. One retry per ticket per router process is the same bound that startup recovery uses,
|
|
114
|
+
and the lifecycle model already covers it: `Ticket.tla` lets a retry happen at any moment after a failed
|
|
115
|
+
report.
|
|
116
|
+
|
|
117
|
+
## A person's stop is final
|
|
118
|
+
|
|
119
|
+
Once a person stops a ticket, nothing revives it: no new message, no wake-up, no retry. A request made
|
|
120
|
+
again becomes a new ticket.
|
|
121
|
+
|
|
122
|
+
A stop is a person saying "not now". An agent that resumes what it was told to stop teaches everyone that
|
|
123
|
+
their words do not count. (`Ticket.tla` I1.)
|
|
124
|
+
|
|
125
|
+
## A closed conversation stops its ticket
|
|
126
|
+
|
|
127
|
+
GitHub ends a conversation in three ways: a merge closes the issue it names, a close says no, and a
|
|
128
|
+
deletion says never mind. All three look the same from a poll, because the thread leaves the set of open
|
|
129
|
+
issues and pull requests. The router then stops the ticket with a person's stop, posts one notice on the
|
|
130
|
+
thread, and stays quiet until the issue is reopened. A ticket that follows its own pull request is the
|
|
131
|
+
exception: that pull request's merge or close ends it, as merged or as closed, not as a stop.
|
|
132
|
+
|
|
133
|
+
Closing the issue is the one stop that needs no tool and no knowledge of the system. Tickets on threads
|
|
134
|
+
from other sources are left alone: the router can only tell whether a GitHub thread is open.
|
|
135
|
+
|
|
136
|
+
## Work that changes a repository ends with its pull request
|
|
137
|
+
|
|
138
|
+
A worker whose work changes a repository opens a pull request and names it in `ticket_report`. The router
|
|
139
|
+
binds the pull request to the ticket before the engine reads the report, so *ready* then means awaiting
|
|
140
|
+
review rather than done. Comments and reviews on the pull request go to the worker. The merge ends the
|
|
141
|
+
ticket as merged, a close ends it as closed, and the router says which on the issue. A report that names
|
|
142
|
+
no pull request ends the ticket as before.
|
|
143
|
+
|
|
144
|
+
The work decides, not the request. A deployment, an answer or an investigation changes no repository, and a
|
|
145
|
+
ticket made to wait for a pull request would never end; the worker knows by the end of its run whether it
|
|
146
|
+
opened one. The worker opens it, not the router, because the worker knows which repository the work
|
|
147
|
+
belongs in: the watched repository is often only where people ask. Only a pull request URL counts, since
|
|
148
|
+
`#5` may be an issue, and a ticket bound to an issue would wait for a merge that never comes. The binding
|
|
149
|
+
comes first because the engine completes a ticket on a *ready* report unless it already follows an open
|
|
150
|
+
pull request. The lifecycle itself is the vendored engine's (`bindThread`, `threadClosed`, and upstream's
|
|
151
|
+
`docs/003-execution.md`); the router feeds it facts.
|
|
152
|
+
|
|
153
|
+
The watched repository's pull requests cost nothing extra: their comments and reviews arrive with every
|
|
154
|
+
poll. One in another repository is read every two minutes, three calls each time, so a dozen open reviews
|
|
155
|
+
stay well inside GitHub's hourly limit.
|
|
156
|
+
|
|
157
|
+
## A review is a request; opening a pull request is not
|
|
158
|
+
|
|
159
|
+
A submitted review, whether it requests changes or approves, is a person speaking about delivered work,
|
|
160
|
+
on a conversation the ticket already owns. It fires a fresh judgement, and the reply lands in the pull
|
|
161
|
+
request's own conversation (on GitHub a pull request is also issue N, so no second address is needed).
|
|
162
|
+
|
|
163
|
+
Opening a pull request is a delivery, not a request. Treating every opening as a trigger would make the
|
|
164
|
+
agent a mandatory reviewer of its own output, a loop that feeds itself.
|
|
165
|
+
|
|
166
|
+
## Only work holds a slot
|
|
167
|
+
|
|
168
|
+
The cap counts workers on a run. A worker between runs, waiting on a person's answer or simply done, is
|
|
169
|
+
parked: it keeps its session and its context, and its slot goes to the next ticket in the queue. A reply
|
|
170
|
+
puts it back on a run, even past the cap for a while.
|
|
171
|
+
|
|
172
|
+
A slot is the machine's capacity, and an idle session uses almost none of it. When every live worker held
|
|
173
|
+
a slot, a queue of 22 tickets waited while one of the four slots belonged to a worker that had finished its
|
|
174
|
+
run without saying so.
|
|
175
|
+
|
|
176
|
+
## A tab is closed only when nobody is in it
|
|
177
|
+
|
|
178
|
+
A finished worker's tab stays open for ten minutes of follow-up and then closes only if its pi sits idle;
|
|
179
|
+
while the agent is busy, the router looks again every ten minutes instead of giving up.
|
|
180
|
+
The startup sweep closes a worker tab only when no record owns it or herdr no longer knows its agent.
|
|
181
|
+
|
|
182
|
+
Worker tabs do outlive their purpose: a restart adopts a worker, a stop starts a newer one, and the older
|
|
183
|
+
tab stays behind, so a sweep has to exist. But herdr has no guarded close, and a tab is someone's
|
|
184
|
+
terminal. The liveness check is the single guard that keeps the sweep away from an agent that is still
|
|
185
|
+
working. (`Workers.tla` W2, W4, W5.)
|
|
186
|
+
|
|
187
|
+
## One router, one store, one state directory
|
|
188
|
+
|
|
189
|
+
One router runs at a time, and one process holds the engine. The store is synchronous SQLite, and
|
|
190
|
+
everything the system knows lives in one directory.
|
|
191
|
+
|
|
192
|
+
The engine's commands are atomic only if the store is synchronous: an asynchronous driver would turn that
|
|
193
|
+
atomicity into a comment. One directory makes the crash model concrete. A restart is "start again with
|
|
194
|
+
this directory", which is exactly what the specifications model. One router per process matters too: the
|
|
195
|
+
judge and every worker load this same extension, and each of them used to boot a router of its own.
|
|
196
|
+
|
|
197
|
+
## Health is an exit code
|
|
198
|
+
|
|
199
|
+
`superagent status` answers with a number: 0 healthy, 1 down, 2 a part is failing, 3 waiting for setup.
|
|
200
|
+
A router that fails three polls in a row reports its GitHub source as failing.
|
|
201
|
+
|
|
202
|
+
A scheduled check should act without reading prose, and each code has one remedy: a restart may fix 1
|
|
203
|
+
and fixes neither 2 nor 3. A router that cannot reach GitHub otherwise looks healthy forever, answering
|
|
204
|
+
nothing.
|
|
205
|
+
|
|
206
|
+
## The panel is local and cannot be steered from a web page
|
|
207
|
+
|
|
208
|
+
The panel binds 127.0.0.1, refuses any Host header but 127.0.0.1 and localhost, and serves its API only
|
|
209
|
+
to callers that send `x-superagent-request: 1`.
|
|
210
|
+
|
|
211
|
+
A browser will not let another site forge either one. Without them, any page you visit could read the
|
|
212
|
+
queue, or pause the engine, by rebinding a hostname to localhost.
|
|
213
|
+
|
|
214
|
+
## The engine stays byte-identical to upstream
|
|
215
|
+
|
|
216
|
+
`src/core/` and its test suite are vendored from upstream and never edited. Every adaptation, from the
|
|
217
|
+
judge's prompt to the GitHub shapes, lives in `src/pi/`.
|
|
218
|
+
|
|
219
|
+
The engine is the part the lifecycle specification models and upstream maintains. Untouched, an upstream
|
|
220
|
+
fix can be re-vendored by copying, and the specification keeps describing the code that runs. The
|
|
221
|
+
[adaptation ledger](../ADAPTATION.md) says how to prove the copy is exact.
|
|
222
|
+
|
|
223
|
+
## The specifications live beside the code
|
|
224
|
+
|
|
225
|
+
`formal/` holds three TLA+ specifications, and `scripts/check-formal` checks all of them exhaustively in
|
|
226
|
+
about five minutes. Every action names the function it mirrors; every header states what the model
|
|
227
|
+
leaves out.
|
|
228
|
+
|
|
229
|
+
A proof that cannot be re-run in one command rots. These are cheap to keep honest, and they have paid:
|
|
230
|
+
the intake specification found a request-losing bug in shipped code.
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# Formal verification
|
|
2
|
+
|
|
3
|
+
Three parts of pi-superagent, the places where a bad ordering would quietly lose or duplicate work, are
|
|
4
|
+
specified in TLA+ and checked exhaustively by TLC, the TLA+ model checker:
|
|
5
|
+
|
|
6
|
+
| Specification | Models | States explored (distinct) | Claims |
|
|
7
|
+
| --- | --- | --- | --- |
|
|
8
|
+
| [`Ticket.tla`](../formal/Ticket.tla) | the engine's ticket lifecycle (`src/core`) | 12,325,891 (1,398,882) | 7 |
|
|
9
|
+
| [`Intake.tla`](../formal/Intake.tla) | the GitHub source (`source-github.ts`) | 565,551 (113,281) | 6 |
|
|
10
|
+
| [`Workers.tla`](../formal/Workers.tla) | worker records, tabs and agents (`herdr-workers.ts`) | 347 (94) | 5 |
|
|
11
|
+
|
|
12
|
+
Every claim holds in every reachable state. The specifications have already paid for themselves: the
|
|
13
|
+
intake model found a bug in shipped code, a read position saved before the engine held its triggers, so
|
|
14
|
+
that a crash at the wrong moment lost a request for good. TLC printed the exact interleaving, and the fix
|
|
15
|
+
shipped with that interleaving as a regression test. Along the way TLC also caught two mistakes in the
|
|
16
|
+
models themselves (a dropped evaluation batch, and a stopped ticket brought back to life), which is
|
|
17
|
+
exactly what a checker is for.
|
|
18
|
+
|
|
19
|
+
## Running the checks
|
|
20
|
+
|
|
21
|
+
```sh
|
|
22
|
+
scripts/check-formal # all three specifications, about five minutes
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The script needs a JDK (`brew install openjdk`, or point `JAVA` at one) and downloads `tla2tools.jar` to
|
|
26
|
+
`~/.cache/tla2tools/` on first use. It exits 0 only when every specification is checked to completion. A
|
|
27
|
+
missing JDK, a parse error, a violated claim or a truncated run each exits 1 with a short reason: a check
|
|
28
|
+
that stays silent is treated as a failed one.
|
|
29
|
+
|
|
30
|
+
## What each specification proves
|
|
31
|
+
|
|
32
|
+
### `Ticket.tla`: the ticket lifecycle
|
|
33
|
+
|
|
34
|
+
- **I1 StopAbsorbing.** A ticket a person stopped is never live again.
|
|
35
|
+
- **I2 DoneFromReady.** A completed ticket always has a *ready* report, so completing without one is
|
|
36
|
+
unreachable, not merely untested.
|
|
37
|
+
- **I3 OneRunning.** A ticket has at most one running assignment.
|
|
38
|
+
- **I4 NoSecondTicket.** No ticket is created for a thread that a live ticket already follows, in any
|
|
39
|
+
interleaving, including a follow-up that arrives in the middle of an evaluation.
|
|
40
|
+
- **I5 OutboxOnce and I6 SentMonotonic.** Each acknowledgement is queued at most once and sent at most
|
|
41
|
+
once, and a sent message stays sent.
|
|
42
|
+
- **LT_Delivered.** Under the stated fairness, every incoming message eventually reaches intake: intake
|
|
43
|
+
never stalls.
|
|
44
|
+
|
|
45
|
+
### `Intake.tla`: the GitHub source
|
|
46
|
+
|
|
47
|
+
- **C1 SeedSilence.** An issue that was open at the first look is never accepted unless it is closed and
|
|
48
|
+
reopened.
|
|
49
|
+
- **C2 ReopenOnce.** An issue fires at most one request per open interval.
|
|
50
|
+
- **C3 NoGhostCommit.** The committed read position never passes a comment the engine has not accepted.
|
|
51
|
+
This is the claim that found the bug: commit at read time, and TLC produces a crash that loses a request.
|
|
52
|
+
- **C4 NoDoubleAccept.** Acceptance is by id and permanent: a replayed trigger adds nothing, and nothing
|
|
53
|
+
accepted is ever lost.
|
|
54
|
+
- **C5 BotNeverHuman.** A bot's comment is never accepted as a person's.
|
|
55
|
+
- **C6 NoLoss.** Through a crash, every posted comment is eventually accepted, and every issue opened
|
|
56
|
+
after the first look eventually fires.
|
|
57
|
+
|
|
58
|
+
### `Workers.tla`: tabs, records and the sweep
|
|
59
|
+
|
|
60
|
+
- **W1 OneLive.** A ticket has at most one live worker.
|
|
61
|
+
- **W2 SweepKeepsLiveAgents and W4 NoTabLossUnderWork.** A worker whose record is live and whose agent is
|
|
62
|
+
alive always keeps its tab. The sweep's liveness check makes this an invariant rather than a race.
|
|
63
|
+
- **W3 NoResurrect.** A record only moves forward, from free to live to ended, and returns to free only
|
|
64
|
+
when it is retired. An ended record never becomes live again.
|
|
65
|
+
- **W5 ZombieTabsGone.** After any finite number of launches, every dead agent's tab is eventually closed:
|
|
66
|
+
the workspace cannot fill up with abandoned tabs.
|
|
67
|
+
|
|
68
|
+
## Why the results mean something
|
|
69
|
+
|
|
70
|
+
**Each action mirrors real code.** Every action in a specification names the function whose guards it
|
|
71
|
+
copies. When that function changes, the action must change with it, and the check must run again.
|
|
72
|
+
|
|
73
|
+
**Each header says what the model leaves out.** Leaving behavior out narrows what a claim covers. A claim
|
|
74
|
+
holds for the behaviors the model includes, and for a left-out feature only as far as that feature's code
|
|
75
|
+
passes through the same guards. For example, every path that could resume a ticket checks the same
|
|
76
|
+
`ended.by === 'human'` guard that I1 relies on. The headers list what is missing so that a reader can
|
|
77
|
+
check this for themselves.
|
|
78
|
+
|
|
79
|
+
**The bounds are small on purpose.** Two threads, two ticket slots and four messages; two issues, two
|
|
80
|
+
comments and one crash; two workers and three launches. Small bounds let TLC visit every state in
|
|
81
|
+
minutes, and ordering bugs almost always show up at small sizes. They are still bounds, not a proof for
|
|
82
|
+
every size, and each header states its own.
|
|
83
|
+
|
|
84
|
+
**Liveness rests on stated fairness.** The "eventually" claims (LT_Delivered, C6, W5) assume that an
|
|
85
|
+
action which stays enabled eventually happens. Each specification writes its fairness assumptions down,
|
|
86
|
+
next to the claims, so they can be argued with. Crashes and launches are bounded, so "eventually" always
|
|
87
|
+
has an end.
|
|
88
|
+
|
|
89
|
+
**Lessons from TLC, so you can skip them.** A property with primed variables (`[][...]_vars`) goes under
|
|
90
|
+
`PROPERTY`, never `INVARIANT`. `UNCHANGED` takes a tuple `<<...>>`, not a set. An action that assigns
|
|
91
|
+
`x'` must not also list `x` in its `UNCHANGED`: the contradiction silently disables the action, and only
|
|
92
|
+
`-coverage 1` will show it. A specification whose runs can finish needs an explicit stuttering step
|
|
93
|
+
(`Terminate == UNCHANGED vars`), or TLC reports every finished run as a deadlock; stuck work is then
|
|
94
|
+
caught by the liveness claims. And naming a state `open` broke the parse, which is why the intake model
|
|
95
|
+
says `opened`.
|
|
96
|
+
|
|
97
|
+
## What the proofs do not say
|
|
98
|
+
|
|
99
|
+
A model proves things about the model. These cover the control flow described above, not the judge's
|
|
100
|
+
decisions, GitHub's behavior (status codes, rate limits, pagination), SQLite, herdr, or the network.
|
|
101
|
+
Newer behavior is tested but not modeled: reviews, the stop on a closed conversation, the outbox's check
|
|
102
|
+
before a retry, and how each way a worker's run ends reaches the thread.
|
|
103
|
+
|
|
104
|
+
What the specifications do guarantee: no implementation that follows these transitions can reach a
|
|
105
|
+
forbidden state, and the liveness claims hold under the stated fairness. When the code a specification
|
|
106
|
+
mirrors changes, upstream's ticket transitions included, the specification must change with it.
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# Operator guide
|
|
2
|
+
|
|
3
|
+
Everything you need to ask for work, watch it happen, and set it right when it goes wrong.
|
|
4
|
+
|
|
5
|
+
## Asking for work
|
|
6
|
+
|
|
7
|
+
Open an issue on the watched repository (`PI_SUPERAGENT_GITHUB_REPO`) and say plainly what you want
|
|
8
|
+
done:
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
Title: Rename the Export button to "Ship"
|
|
12
|
+
Body: The primary button in the header should say Ship instead of Export. Keep the keyboard shortcut.
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Within about a minute the judge's acknowledgement appears on the issue, and a tab named for the issue
|
|
16
|
+
opens in the `superagent` herdr workspace with the worker inside. When the worker finishes, its report
|
|
17
|
+
arrives on the issue as a reply.
|
|
18
|
+
|
|
19
|
+
The judge accepts a concrete action someone asks the agent to take, even one that calls itself a test.
|
|
20
|
+
It ignores chatter, notes addressed to other people, and new requests on a thread that already has an
|
|
21
|
+
open ticket. An ignore posts nothing on the issue; the reason is in the engine log.
|
|
22
|
+
|
|
23
|
+
- **Who can ask.** People with write access to the repository: its owner, members of the organization
|
|
24
|
+
that owns it, and collaborators. Anyone else's issues, comments and reviews are dropped before the judge
|
|
25
|
+
sees them, and the panel counts them as filtered. To hear someone else, add them as a collaborator. To
|
|
26
|
+
hear fewer people, list the logins in `PI_SUPERAGENT_ALLOWED`; the list then replaces the rule.
|
|
27
|
+
- **Follow-ups.** Comment on the same issue. While its ticket is open, your comments go straight to its
|
|
28
|
+
worker.
|
|
29
|
+
- **Pull requests.** When the work changes a repository, the worker opens a pull request and names it in
|
|
30
|
+
its report, and the receipt on the issue links it. The ticket then waits for review, shown on the panel
|
|
31
|
+
as `review of #N`: your comments and reviews on the pull request go to the same worker. Merge it, and
|
|
32
|
+
the ticket ends and the issue says it is done. Close it unmerged, and the ticket ends too; a comment on
|
|
33
|
+
the issue starts the work again. A pull request in another repository works the same way, checked every
|
|
34
|
+
two minutes. A request that changes no repository, such as a question, an investigation or an
|
|
35
|
+
operation, ends with its report.
|
|
36
|
+
- **Who opens the pull request.** The worker, with the `gh` on this machine, so the pull request's author
|
|
37
|
+
is whoever `gh` is signed in as. Blocking `gh` for workers (`PI_SUPERAGENT_BLOCKED`) turns pull requests
|
|
38
|
+
off: workers then report without one.
|
|
39
|
+
- **The agent's own account.** By default the router reads and replies as the account `gh` is signed in
|
|
40
|
+
as, so its comments look like yours apart from a hidden marker. To give it an account of its own, put a
|
|
41
|
+
token for that account in a file only you can read, set `PI_SUPERAGENT_GITHUB_TOKEN_FILE` to the file
|
|
42
|
+
and `PI_SUPERAGENT_GITHUB_SELF` to the account's login, give the account access to the watched
|
|
43
|
+
repository, and restart. The engine log's `router.start` line then says `githubAs="token file"`.
|
|
44
|
+
Workers still act as `gh`'s account, so their pull requests are yours, and the router reads those pull
|
|
45
|
+
requests as that account too.
|
|
46
|
+
- **Pull requests no ticket opened.** A comment or a submitted review on one reaches the judge like any
|
|
47
|
+
other request, and the answer lands in the pull request's conversation.
|
|
48
|
+
- **Questions from a worker.** A worker that needs you reports *blocked* with one question. The question
|
|
49
|
+
is posted on the issue (and listed on the panel under *Needs a person*); answer it with a comment.
|
|
50
|
+
- **Stopping.** Close the issue. Within one poll the ticket stops and says so on the thread. A ticket
|
|
51
|
+
waiting on its own pull request ends with that pull request instead, as above. Reopen the issue to run the request again. Without GitHub, the panel
|
|
52
|
+
API stops a ticket too:
|
|
53
|
+
`curl -X POST -H 'x-superagent-request: 1' http://127.0.0.1:8788/api/tickets/<id>/stop`.
|
|
54
|
+
|
|
55
|
+
## Watching
|
|
56
|
+
|
|
57
|
+
- **The herdr workspace `superagent`.** The `engine log` tab streams every poll, judgement, launch and
|
|
58
|
+
receipt. Each worker has its own tab, labelled `#<issue> · <title>`; it is an ordinary pi session, so
|
|
59
|
+
you can read along, type into it, or press Esc to interrupt it.
|
|
60
|
+
- **The panel** at <http://127.0.0.1:8788/>. It shows what needs a person, what is running and queued,
|
|
61
|
+
and what the judge decided.
|
|
62
|
+
- **The log itself:** `tail -F ~/.pi/superagent/engine.log`. Each line is one event with its fields, so
|
|
63
|
+
`grep eval.decided` lists every judgement with its reason.
|
|
64
|
+
|
|
65
|
+
## Running it
|
|
66
|
+
|
|
67
|
+
```sh
|
|
68
|
+
superagent status # 0 healthy · 1 down · 2 a part is failing · 3 waiting for setup
|
|
69
|
+
superagent restart # safe at any time: workers keep running and the new router adopts them
|
|
70
|
+
superagent stop # leaves a marker, so the health check will not restart it
|
|
71
|
+
superagent start # clears the marker
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
In a checkout, the same script is `scripts/superagent`.
|
|
75
|
+
|
|
76
|
+
Settings live in `~/.pi/superagent/config.env`. [`config.example.env`](../config.example.env), shipped
|
|
77
|
+
with the package, explains each setting and its default. Restart to apply a change.
|
|
78
|
+
|
|
79
|
+
On macOS, the launchd job in [`launchd/`](../launchd/com.example.superagent-health.plist) runs `status`
|
|
80
|
+
every five minutes and starts a router that is down. It leaves a deliberately stopped router alone.
|
|
81
|
+
Install it once:
|
|
82
|
+
|
|
83
|
+
```sh
|
|
84
|
+
plist="$(npm root -g)/pi-superagent/launchd/com.example.superagent-health.plist" # launchd/ in a checkout
|
|
85
|
+
sed "s|__HOME__|$HOME|g" "$plist" > ~/Library/LaunchAgents/com.example.superagent-health.plist
|
|
86
|
+
launchctl load ~/Library/LaunchAgents/com.example.superagent-health.plist
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
The job finds `superagent`, `pi` and `herdr` on its own PATH: Homebrew's directories, `/usr/local/bin` and
|
|
90
|
+
`~/.npm-global/bin`. If `command -v superagent` or `command -v pi` prints a directory outside those, as nvm's
|
|
91
|
+
do, add it to the job's PATH. From a checkout, replace `superagent` in the job with the script's full path.
|
|
92
|
+
|
|
93
|
+
## Capacity
|
|
94
|
+
|
|
95
|
+
Up to four workers can be on a run at once (`PI_SUPERAGENT_MAX_WORKERS`); further tickets wait their
|
|
96
|
+
turn, queued on the panel. A worker between runs, waiting on your answer or simply done, is *parked*: it
|
|
97
|
+
keeps its tab and everything it knows, but gives up its slot, so the queue keeps moving. A reply puts it
|
|
98
|
+
back on a run. The judge answers one request at a time. Every worker's brief says the machine is shared:
|
|
99
|
+
sequential builds, bounded parallel jobs, nothing left running after the report.
|
|
100
|
+
|
|
101
|
+
A worker on a run that emits nothing for 15 minutes is flagged on the panel, and only there, under *Needs a
|
|
102
|
+
person*: steer it, stop it, or accept its work. The flag does not stop the worker. If it is only running a
|
|
103
|
+
long build, let it be; the flag clears when the worker reports.
|
|
104
|
+
|
|
105
|
+
## When something goes wrong
|
|
106
|
+
|
|
107
|
+
**No acknowledgement after a few minutes.** Run `status` first. Then
|
|
108
|
+
`grep eval.decided ~/.pi/superagent/engine.log | tail`: `kind=ignore` comes with the judge's reason, and a
|
|
109
|
+
clearer comment or a reopened issue asks again. A `judge.prompt` line with no `eval.decided` after it
|
|
110
|
+
means the judge is wedged; `restart` fixes it.
|
|
111
|
+
|
|
112
|
+
**Someone else's issue gets no acknowledgement.** Check that they can write to the repository, or that
|
|
113
|
+
their login is in `PI_SUPERAGENT_ALLOWED` when that is set. The panel's count of filtered triggers rises
|
|
114
|
+
by one for each request it dropped this way.
|
|
115
|
+
|
|
116
|
+
**A ticket stays open after its worker reported ready.** It is waiting for its pull request: the panel
|
|
117
|
+
shows `review of #N`. Merge or close the pull request; the ticket ends within one poll, or within two
|
|
118
|
+
minutes for a pull request in another repository.
|
|
119
|
+
|
|
120
|
+
**`status` exits 2.** Read `~/.pi/superagent/health.json`. A failing `source` means the router cannot
|
|
121
|
+
reach GitHub; check `gh auth status`. The error clears on the next good poll.
|
|
122
|
+
|
|
123
|
+
**`status` exits 3.** Read `~/.pi/superagent/setup.json`: it names what is missing and where to fix it.
|
|
124
|
+
|
|
125
|
+
**A worker looks stuck.** Open its tab and read it. Type into it to steer, or press Esc to interrupt.
|
|
126
|
+
|
|
127
|
+
**The issue says the worker stopped without reporting a result.** It ended a run without calling
|
|
128
|
+
`ticket_report`, so whatever it concluded is still in its tab. The worker is still open: a reply on the
|
|
129
|
+
issue steers it, and closing the issue drops the ticket.
|
|
130
|
+
|
|
131
|
+
**The issue says the worker crashed.** The router has already started a fresh worker on it, once, and the
|
|
132
|
+
new worker's brief says what happened. If the issue says the fresh worker crashed too, nothing runs the
|
|
133
|
+
ticket until `superagent restart`, and closing the issue drops it.
|
|
134
|
+
|
|
135
|
+
**The issue says a worker could not start.** The notice quotes the error, usually from the setup script
|
|
136
|
+
(`PI_SUPERAGENT_PREPARE`). The router retries on every poll, without posting again, until the cause is
|
|
137
|
+
fixed or the issue is closed.
|