pi-superagent 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADAPTATION.md +71 -0
- package/LICENSE +21 -0
- package/NOTICE +19 -0
- package/README.md +98 -2
- package/config.example.env +81 -0
- package/docs/architecture.md +139 -0
- package/docs/design-decisions.md +130 -0
- package/docs/formal-verification.md +102 -0
- package/docs/operator-guide.md +149 -0
- package/formal/Intake.cfg +18 -0
- package/formal/Intake.tla +206 -0
- package/formal/README.md +31 -0
- package/formal/Ticket.cfg +23 -0
- package/formal/Ticket.tla +244 -0
- package/formal/Workers.cfg +20 -0
- package/formal/Workers.tla +152 -0
- package/launchd/com.example.superagent-health.plist +31 -0
- package/package.json +72 -4
- package/scripts/superagent +219 -0
- package/src/core/assignments.ts +140 -0
- package/src/core/engine.ts +270 -0
- package/src/core/errors.ts +61 -0
- package/src/core/evals.ts +340 -0
- package/src/core/ids.ts +5 -0
- package/src/core/index.ts +10 -0
- package/src/core/intake.ts +63 -0
- package/src/core/memory.ts +50 -0
- package/src/core/model.ts +353 -0
- package/src/core/part.ts +80 -0
- package/src/core/ports.ts +134 -0
- package/src/core/reasons.ts +38 -0
- package/src/core/replies.ts +95 -0
- package/src/core/stage.ts +26 -0
- package/src/core/tickets.ts +423 -0
- package/src/pi/assessor.ts +97 -0
- package/src/pi/authors.ts +10 -0
- package/src/pi/executor.ts +267 -0
- package/src/pi/extension.ts +703 -0
- package/src/pi/gh-api.ts +147 -0
- package/src/pi/herdr-workers.ts +299 -0
- package/src/pi/herdr.ts +91 -0
- package/src/pi/judge.ts +60 -0
- package/src/pi/log.ts +35 -0
- package/src/pi/outbox.ts +82 -0
- package/src/pi/pulls.ts +154 -0
- package/src/pi/receipts.ts +43 -0
- package/src/pi/report-tool.ts +35 -0
- package/src/pi/router.ts +125 -0
- package/src/pi/sdk-contract.ts +51 -0
- package/src/pi/server.ts +171 -0
- package/src/pi/sink-session-inbox.ts +28 -0
- package/src/pi/source-github.ts +209 -0
- package/src/pi/store.ts +132 -0
- package/src/pi/worker-role.ts +79 -0
- package/src/pi/worker-view.ts +80 -0
- package/src/pi/workspace.ts +155 -0
- package/vendor/dsh-superagent/LICENSE +21 -0
- package/vendor/dsh-superagent/UPSTREAM_SHA +1 -0
- package/vendor/dsh-superagent/docs/000-philosophy.md +33 -0
- package/vendor/dsh-superagent/docs/001-domain.md +112 -0
- package/vendor/dsh-superagent/docs/002-authorization.md +155 -0
- package/vendor/dsh-superagent/docs/003-execution.md +203 -0
- package/vendor/dsh-superagent/docs/004-prompts.md +96 -0
- package/vendor/dsh-superagent/docs/005-panel.md +58 -0
- package/vendor/dsh-superagent/docs/006-deployment.md +274 -0
- package/vendor/dsh-superagent/docs/007-architecture.md +617 -0
- package/vendor/dsh-superagent/docs/008-cookbook.md +425 -0
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# Formal verification
|
|
2
|
+
|
|
3
|
+
When is it safe to mark a GitHub comment as read? An early router saved its place as soon as it
|
|
4
|
+
read the comment. If it crashed before passing the message to the ticket engine, it restarted one
|
|
5
|
+
comment too far ahead. From the requester’s point of view, the worker had simply ignored them.
|
|
6
|
+
|
|
7
|
+
TLC found the precise sequence that lost the comment. It works by trying every ordering of events
|
|
8
|
+
allowed by a small model written in TLA+. That sequence became a regression test, and the router
|
|
9
|
+
now waits until the engine has received the comment before saving its place.
|
|
10
|
+
|
|
11
|
+
The project checks three such models. The ticket model asks who owns each conversation and how
|
|
12
|
+
work can end. The intake model follows messages across polls and crashes. The worker model follows
|
|
13
|
+
terminal tabs as agents start, finish and leave tabs behind.
|
|
14
|
+
|
|
15
|
+
| Specification | Models | States explored (distinct) | Claims |
|
|
16
|
+
| --- | --- | --- | --- |
|
|
17
|
+
| [`Ticket.tla`](../formal/Ticket.tla) | the engine's ticket lifecycle (`src/core`) | 12,325,891 (1,398,882) | 7 |
|
|
18
|
+
| [`Intake.tla`](../formal/Intake.tla) | the GitHub source (`source-github.ts`) | 565,551 (113,281) | 6 |
|
|
19
|
+
| [`Workers.tla`](../formal/Workers.tla) | worker records, tabs and agents (`herdr-workers.ts`) | 347 (94) | 5 |
|
|
20
|
+
|
|
21
|
+
The counts include repeat visits to a state. The numbers in parentheses count each distinct
|
|
22
|
+
state once. All listed properties hold within the bounds below. TLC also caught two errors in the
|
|
23
|
+
models themselves, one that dropped a batch of requests and another that revived a stopped ticket.
|
|
24
|
+
|
|
25
|
+
## Running the checks
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
scripts/check-formal # all three specifications, about five minutes
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The script needs a JDK (`brew install openjdk`, or point `JAVA` at one) and downloads `tla2tools.jar` to
|
|
32
|
+
`~/.cache/tla2tools/` on first use. It exits 0 only when every specification is checked to completion.
|
|
33
|
+
It exits 1 and prints the reason if Java is missing, a specification fails to parse, TLC finds a
|
|
34
|
+
violation or a run ends early.
|
|
35
|
+
|
|
36
|
+
## What each specification proves
|
|
37
|
+
|
|
38
|
+
### `Ticket.tla`: the ticket lifecycle
|
|
39
|
+
|
|
40
|
+
- **I1 StopAbsorbing.** A ticket a person stopped is never live again.
|
|
41
|
+
- **I2 DoneFromReady.** Every completed ticket has a recorded *ready* report.
|
|
42
|
+
- **I3 OneRunning.** A ticket has at most one running assignment.
|
|
43
|
+
- **I4 NoSecondTicket.** No ticket is created for a thread that a live ticket already follows, in any
|
|
44
|
+
interleaving, including a follow-up that arrives in the middle of an evaluation.
|
|
45
|
+
- **I5 OutboxOnce and I6 SentMonotonic.** Each acknowledgement is queued at most once and sent at most
|
|
46
|
+
once, and a sent message stays sent.
|
|
47
|
+
- **LT_Delivered.** Under the stated fairness, every incoming message eventually reaches intake: intake
|
|
48
|
+
never stalls.
|
|
49
|
+
|
|
50
|
+
### `Intake.tla`: the GitHub source
|
|
51
|
+
|
|
52
|
+
- **C1 SeedSilence.** An issue that was open at the first look is never accepted unless it is closed and
|
|
53
|
+
reopened.
|
|
54
|
+
- **C2 ReopenOnce.** An issue fires at most one request per open interval.
|
|
55
|
+
- **C3 NoGhostCommit.** The committed read position never passes a comment the engine has not accepted.
|
|
56
|
+
This is the claim that found the bug: commit at read time, and TLC produces a crash that loses a request.
|
|
57
|
+
- **C4 NoDoubleAccept.** Acceptance is by id and permanent: a replayed trigger adds nothing, and nothing
|
|
58
|
+
accepted is ever lost.
|
|
59
|
+
- **C5 BotNeverHuman.** A bot's comment is never accepted as a person's.
|
|
60
|
+
- **C6 NoLoss.** Through a crash, every posted comment is eventually accepted, and every issue opened
|
|
61
|
+
after the first look eventually fires.
|
|
62
|
+
|
|
63
|
+
### `Workers.tla`: tabs, records and the sweep
|
|
64
|
+
|
|
65
|
+
- **W1 OneLive.** A ticket has at most one live worker.
|
|
66
|
+
- **W2 SweepKeepsLiveAgents and W4 NoTabLossUnderWork.** A worker whose record is live and whose agent is
|
|
67
|
+
alive always keeps its tab. The sweep's liveness check makes this an invariant rather than a race.
|
|
68
|
+
- **W3 NoResurrect.** A record only moves forward, from free to live to ended, and returns to free only
|
|
69
|
+
when it is retired. An ended record never becomes live again.
|
|
70
|
+
- **W5 ZombieTabsGone.** After any finite number of launches, every dead agent's tab is eventually closed:
|
|
71
|
+
the workspace cannot fill up with abandoned tabs.
|
|
72
|
+
|
|
73
|
+
## Interpreting the results
|
|
74
|
+
|
|
75
|
+
Each action names the function whose guards it mirrors. A change to that function may change which
|
|
76
|
+
states the implementation can reach. Update the model and rerun the check when a transition changes.
|
|
77
|
+
For example, I1 depends on every path that might resume a ticket checking `ended.by === 'human'`.
|
|
78
|
+
|
|
79
|
+
The ticket model has two threads, two ticket slots and four messages. The intake model has two
|
|
80
|
+
issues, two comments and one crash. The worker model has two workers and three launches. These
|
|
81
|
+
bounds let TLC explore every ordering in minutes. Each model header records its exclusions.
|
|
82
|
+
|
|
83
|
+
The eventual-delivery and cleanup properties (LT_Delivered, C6 and W5) also depend on fairness.
|
|
84
|
+
An action that stays enabled must eventually run. The bounds on crashes and launches give the
|
|
85
|
+
system a chance to finish delivery or cleanup.
|
|
86
|
+
|
|
87
|
+
## Maintaining the models
|
|
88
|
+
|
|
89
|
+
- Put temporal formulas such as `[][A]_vars` under `PROPERTY`. Use `INVARIANT` for state predicates.
|
|
90
|
+
- Pass a tuple such as `<<x, y>>` to `UNCHANGED`. If an action assigns `x'`, omit `x` from that
|
|
91
|
+
tuple unless the assignment really must preserve its value. Use `-coverage 1` to spot disabled actions.
|
|
92
|
+
- Give a model with a valid terminal state an explicit stuttering step (`Terminate == UNCHANGED vars`).
|
|
93
|
+
This lets TLC distinguish normal termination from a deadlock. Liveness checks still detect stuck work.
|
|
94
|
+
- The intake model uses the state name `opened` because `open` caused a parse error.
|
|
95
|
+
|
|
96
|
+
## Scope
|
|
97
|
+
|
|
98
|
+
The models describe control flow. The judge’s decisions, GitHub API behavior, SQLite, herdr and network
|
|
99
|
+
failures have their own behavior outside these bounds. Integration tests cover review routing,
|
|
100
|
+
closing a conversation, checking for a previous post before retrying, and notifying the thread
|
|
101
|
+
when a worker’s run ends. Consult the test suite for those cases and the model headers for the
|
|
102
|
+
assumptions behind each property.
|
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
# Operator guide
|
|
2
|
+
|
|
3
|
+
Most of the time, the issue is the only window you need. Ask for a change there, answer any questions
|
|
4
|
+
in the thread, and review the pull request when it arrives. If you want to see a command run or help
|
|
5
|
+
a worker past a snag, its terminal is open to you too.
|
|
6
|
+
|
|
7
|
+
## Ask for work
|
|
8
|
+
|
|
9
|
+
Open an issue in the repository you configured with `superagent init`. A sentence about the outcome
|
|
10
|
+
and one detail that matters to you usually give the worker a good place to start.
|
|
11
|
+
|
|
12
|
+
> **Show the export shortcut in the Help menu**
|
|
13
|
+
>
|
|
14
|
+
> I always forget how to export from the keyboard. Could you show the existing shortcut beside
|
|
15
|
+
> Export in the Help menu? Please use the same wording as the File menu.
|
|
16
|
+
|
|
17
|
+
The **judge** reads new requests and decides which ones call for action. When it accepts yours,
|
|
18
|
+
the router acknowledges it on the issue. Look for a tab with that issue number in the
|
|
19
|
+
`superagent` herdr workspace. Inside is the **worker**, the pi session doing the work.
|
|
20
|
+
|
|
21
|
+
An aside to a teammate can stay an aside. If the judge passes over a comment, you can find its
|
|
22
|
+
reason in the engine log. Once your issue has an open ticket, further comments go to the worker.
|
|
23
|
+
|
|
24
|
+
### Who can start work
|
|
25
|
+
|
|
26
|
+
By default the router accepts authors whose GitHub association is `OWNER`, `MEMBER` or
|
|
27
|
+
`COLLABORATOR`. The panel counts requests from other authors as filtered. To choose the permitted
|
|
28
|
+
accounts yourself, set `PI_SUPERAGENT_ALLOWED` to their logins. This list replaces the default rule.
|
|
29
|
+
|
|
30
|
+
### Follow up and review changes
|
|
31
|
+
|
|
32
|
+
Comment on the issue to steer an active worker. If it needs a decision, it posts a question on
|
|
33
|
+
the thread and the panel lists it under *Needs a person*. Reply there to continue.
|
|
34
|
+
|
|
35
|
+
For a code change, the worker opens a pull request and the router links it on the issue. The panel
|
|
36
|
+
shows `review of #N` while the PR is open. Comments and reviews on that PR reach the same worker.
|
|
37
|
+
Merging ends the ticket as merged. Closing the PR unmerged also ends the ticket, and you can
|
|
38
|
+
comment on the issue to ask for another attempt. The router checks PRs in other repositories every
|
|
39
|
+
two minutes. An answer, investigation or operation can finish with the worker’s ready report.
|
|
40
|
+
|
|
41
|
+
A comment or submitted review on a PR without a ticket goes to the judge. The router replies in
|
|
42
|
+
that PR’s conversation.
|
|
43
|
+
|
|
44
|
+
### Choose the GitHub accounts
|
|
45
|
+
|
|
46
|
+
Workers open PRs as the account signed in through the local `gh` CLI. Blocking `gh` with
|
|
47
|
+
`PI_SUPERAGENT_BLOCKED` disables that workflow. By default the router uses the same account and
|
|
48
|
+
adds a hidden marker to its comments so it can distinguish them from yours.
|
|
49
|
+
|
|
50
|
+
To have the router reply under its own name, give that account access to the watched repository.
|
|
51
|
+
Save its token in a file only you can read, set `PI_SUPERAGENT_GITHUB_TOKEN_FILE` to the path and
|
|
52
|
+
`PI_SUPERAGENT_GITHUB_SELF` to the login, then restart. Look for `githubAs="token file"` in the
|
|
53
|
+
`router.start` log entry. Workers still use the local CLI account, which the router also uses to
|
|
54
|
+
read their PRs.
|
|
55
|
+
|
|
56
|
+
### Stop a ticket
|
|
57
|
+
|
|
58
|
+
Close the issue to stop its ticket on the next poll. When a ticket is waiting for its own PR,
|
|
59
|
+
merge or close that PR to record the review outcome. Reopen the issue to request work again.
|
|
60
|
+
You can also stop a ticket through the local panel API.
|
|
61
|
+
|
|
62
|
+
```sh
|
|
63
|
+
curl -X POST -H 'x-superagent-request: 1' "http://127.0.0.1:8788/api/tickets/<id>/stop"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## Watch a worker
|
|
67
|
+
|
|
68
|
+
To see the work itself, open the herdr tab labeled `#<issue> · <title>`. It’s an ordinary pi
|
|
69
|
+
session. You can read the output, type a suggestion or press Esc to interrupt a command. For a
|
|
70
|
+
view across all your issues, open the local panel. It shows what’s running, what’s queued and
|
|
71
|
+
which workers need an answer. [Open the panel](http://127.0.0.1:8788/)
|
|
72
|
+
|
|
73
|
+
The **engine log** tab tells the other half of the story. It records when the router checked
|
|
74
|
+
GitHub, what the judge decided and when a worker started. To follow the same log from another
|
|
75
|
+
terminal, run `tail -F ~/.pi/superagent/engine.log`.
|
|
76
|
+
|
|
77
|
+
## Manage the router
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
superagent status # 0 healthy · 1 down · 2 a part is failing · 3 waiting for setup
|
|
81
|
+
superagent restart # safe at any time: workers keep running and the new router adopts them
|
|
82
|
+
superagent stop # leaves a marker, so the health check will not restart it
|
|
83
|
+
superagent start # clears the marker
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
In a checkout, the same script is `scripts/superagent`.
|
|
87
|
+
|
|
88
|
+
Settings live in `~/.pi/superagent/config.env`. [`config.example.env`](../config.example.env), shipped
|
|
89
|
+
with the package, explains each setting and its default. Restart to apply a change.
|
|
90
|
+
|
|
91
|
+
On macOS, the launchd job in [`launchd/`](../launchd/com.example.superagent-health.plist) runs `status`
|
|
92
|
+
every five minutes and starts a router that is down. It leaves a deliberately stopped router alone.
|
|
93
|
+
Install it once:
|
|
94
|
+
|
|
95
|
+
```sh
|
|
96
|
+
plist="$(npm root -g)/pi-superagent/launchd/com.example.superagent-health.plist" # launchd/ in a checkout
|
|
97
|
+
sed "s|__HOME__|$HOME|g" "$plist" > ~/Library/LaunchAgents/com.example.superagent-health.plist
|
|
98
|
+
launchctl load ~/Library/LaunchAgents/com.example.superagent-health.plist
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
The job finds `superagent`, `pi` and `herdr` on its own PATH: Homebrew's directories, `/usr/local/bin` and
|
|
102
|
+
`~/.npm-global/bin`. If `command -v superagent` or `command -v pi` prints a directory outside those, as nvm's
|
|
103
|
+
do, add it to the job's PATH. From a checkout, replace `superagent` in the job with the script's full path.
|
|
104
|
+
|
|
105
|
+
## When several issues arrive at once
|
|
106
|
+
|
|
107
|
+
By default, four workers can run at once. If you open a fifth issue while all four are busy, its
|
|
108
|
+
ticket waits in the queue on the panel. A worker waiting for your answer can **park**. It keeps
|
|
109
|
+
its tab and its conversation history, and another ticket can use the free slot. When you reply,
|
|
110
|
+
the parked worker picks up where it left off. Set `PI_SUPERAGENT_MAX_WORKERS` to change the limit.
|
|
111
|
+
|
|
112
|
+
The judge reads one request at a time. Workers can build and test side by side, so their
|
|
113
|
+
instructions ask them to limit parallel jobs and clean up any processes they start. If a worker
|
|
114
|
+
goes quiet for 15 minutes, it appears under *Needs a person* on the panel. A long build can be
|
|
115
|
+
quiet, too. Take a look in the tab before interrupting it.
|
|
116
|
+
|
|
117
|
+
## When something goes wrong
|
|
118
|
+
|
|
119
|
+
**No acknowledgement after a few minutes.** Run `superagent status`, then inspect the most recent
|
|
120
|
+
decisions with `grep eval.decided ~/.pi/superagent/engine.log | tail`. For `kind=ignore`, read the
|
|
121
|
+
reason before posting a clearer request. If the log stays at `judge.prompt` without a decision,
|
|
122
|
+
restart the router with `superagent restart`.
|
|
123
|
+
|
|
124
|
+
**Someone else's issue gets no acknowledgement.** Check their GitHub author association, or look for
|
|
125
|
+
their login in `PI_SUPERAGENT_ALLOWED` if you configured a list. The panel's count of filtered triggers rises
|
|
126
|
+
by one for each request it dropped this way.
|
|
127
|
+
|
|
128
|
+
**A ticket stays open after a ready report.** Look for `review of #N` on the panel. Merge or close
|
|
129
|
+
that PR to finish the ticket. The router checks the watched repository on each poll and other
|
|
130
|
+
repositories every two minutes.
|
|
131
|
+
|
|
132
|
+
**Status exits 2.** Read `~/.pi/superagent/health.json` to identify the failing component. For a
|
|
133
|
+
GitHub source error, check `gh auth status`. The next successful poll clears the error.
|
|
134
|
+
|
|
135
|
+
**Status exits 3.** Read `~/.pi/superagent/setup.json` for the missing prerequisite and its location.
|
|
136
|
+
|
|
137
|
+
**A worker looks stuck.** Open its herdr tab to inspect the current command. Type a message to steer
|
|
138
|
+
the worker or press Esc to interrupt it.
|
|
139
|
+
|
|
140
|
+
**A run ended without a report.** Inspect the worker’s tab for its last output. The session stays
|
|
141
|
+
open. Reply on the issue to continue or close the issue to drop the ticket.
|
|
142
|
+
|
|
143
|
+
**A worker crashed.** The router retries the first crash automatically and tells the replacement
|
|
144
|
+
what happened. After a second crash, inspect the failure and run `superagent restart` to retry,
|
|
145
|
+
or close the issue to stop.
|
|
146
|
+
|
|
147
|
+
**A worker could not start.** Read the error on the issue. Check repository access if the clone
|
|
148
|
+
failed, or inspect `PI_SUPERAGENT_PREPARE` if you use a custom setup command. The router retries
|
|
149
|
+
on each poll and posts the launch failure once.
|
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
--------------------- MODULE Intake ---------------------
|
|
2
|
+
(*
|
|
3
|
+
The GitHub intake source: its read position, its seed rule, and comment delivery.
|
|
4
|
+
Its first version fired every pre-existing issue as a request on its first start,
|
|
5
|
+
so its rules are worth checking in every interleaving.
|
|
6
|
+
|
|
7
|
+
Mirrors src/pi/source-github.ts: bootstrap seeds `since` one day back; the first
|
|
8
|
+
poll records the open issues and fires none of them; a later poll fires issues
|
|
9
|
+
opened since, drops closed ones from the seen set (so a reopened issue fires again),
|
|
10
|
+
and reads comments from the committed cursor forward.
|
|
11
|
+
|
|
12
|
+
The cursor follows the deferred-commit rule: a poll stages its new cursor, and the
|
|
13
|
+
stage commits at the next poll, once the engine has taken every trigger the poll
|
|
14
|
+
emitted (engine.receive dedups by content-derived trigger id). A crash before the
|
|
15
|
+
commit rewinds to the committed cursor: replay, dedup, no loss. Committing at read
|
|
16
|
+
time instead makes C3 false, which is the bug this model found in shipped code.
|
|
17
|
+
|
|
18
|
+
Leaves out: reviews (they follow the same deferred cursor), pull requests and the
|
|
19
|
+
stop on a closed thread, pagination, edited or deleted comments, comment bodies.
|
|
20
|
+
Bounds: two issues, two comments, one crash.
|
|
21
|
+
*)
|
|
22
|
+
EXTENDS Integers, FiniteSets
|
|
23
|
+
|
|
24
|
+
CONSTANTS i1, i2, c1, c2 (* model values: two issues, two comments *)
|
|
25
|
+
|
|
26
|
+
Issues == {i1, i2}
|
|
27
|
+
(* Comment facts are fixed: issue, time, and whether a person wrote it. c2 is a bot's. *)
|
|
28
|
+
CmOf(c) == [issue |-> i1,
|
|
29
|
+
at |-> IF c = c1 THEN 2 ELSE 3,
|
|
30
|
+
human |-> c # c2]
|
|
31
|
+
Comments == {c1, c2}
|
|
32
|
+
Times == 1..4
|
|
33
|
+
MaxNat(S) == IF S = {} THEN 0 ELSE CHOOSE m \in S : \A n \in S : n <= m
|
|
34
|
+
|
|
35
|
+
VARIABLES
|
|
36
|
+
crashes, (* bounded restarts, for liveness *)
|
|
37
|
+
seeded, firstSeeded, reopened, (* the import-replay ghost *)
|
|
38
|
+
opened, posted, (* the outside world *)
|
|
39
|
+
seen, cursor, seenStaged, cursorStaged, (* committed and staged positions *)
|
|
40
|
+
acceptedCm, acceptedHuman, acceptedIss, (* engine.receive, dedup by id *)
|
|
41
|
+
pendCm, pendIss, openCnt (* emissions in flight *)
|
|
42
|
+
|
|
43
|
+
vars == <<crashes, seeded, firstSeeded, reopened, opened, posted, seen, cursor,
|
|
44
|
+
seenStaged, cursorStaged, acceptedCm, acceptedHuman, acceptedIss,
|
|
45
|
+
pendCm, pendIss, openCnt>>
|
|
46
|
+
|
|
47
|
+
Init ==
|
|
48
|
+
/\ crashes = 0
|
|
49
|
+
/\ seeded = FALSE /\ firstSeeded = {} /\ reopened = [i \in Issues |-> FALSE]
|
|
50
|
+
/\ opened = {i1} (* i1 predates the source — it must seed silent *)
|
|
51
|
+
/\ posted = {}
|
|
52
|
+
/\ seen = {} /\ cursor = 1 /\ seenStaged = {} /\ cursorStaged = 1
|
|
53
|
+
/\ acceptedCm = {} /\ acceptedHuman = {} /\ acceptedIss = {}
|
|
54
|
+
/\ pendCm = {} /\ pendIss = {} /\ openCnt = [i \in Issues |-> 0]
|
|
55
|
+
|
|
56
|
+
(* ---- the outside world ---- *)
|
|
57
|
+
(* Posting causality: a comment materializes at wall time, never behind the newest
|
|
58
|
+
timestamp any poll has observed (the cursor only advances by reading, and GitHub
|
|
59
|
+
`at` is creation time). *)
|
|
60
|
+
PostComment(c) ==
|
|
61
|
+
/\ c \notin posted
|
|
62
|
+
/\ CmOf(c).at >= cursorStaged
|
|
63
|
+
/\ posted' = posted \cup {c}
|
|
64
|
+
/\ UNCHANGED <<crashes, seeded, firstSeeded, reopened, opened, seen, cursor,
|
|
65
|
+
seenStaged, cursorStaged, acceptedCm, acceptedHuman, acceptedIss,
|
|
66
|
+
pendCm, pendIss, openCnt>>
|
|
67
|
+
|
|
68
|
+
OpenIssue(i) ==
|
|
69
|
+
/\ i \notin opened
|
|
70
|
+
/\ opened' = opened \cup {i}
|
|
71
|
+
/\ openCnt' = [openCnt EXCEPT ![i] = 0] (* a fresh open interval may fire once *)
|
|
72
|
+
/\ UNCHANGED <<crashes, seeded, firstSeeded, reopened, posted, seen, cursor,
|
|
73
|
+
seenStaged, cursorStaged, acceptedCm, acceptedHuman, acceptedIss,
|
|
74
|
+
pendCm, pendIss>>
|
|
75
|
+
|
|
76
|
+
CloseIssue(i) ==
|
|
77
|
+
/\ i \in opened
|
|
78
|
+
/\ opened' = opened \ {i}
|
|
79
|
+
/\ openCnt' = [openCnt EXCEPT ![i] = 0]
|
|
80
|
+
/\ reopened' = [reopened EXCEPT ![i] = TRUE]
|
|
81
|
+
/\ UNCHANGED <<crashes, seeded, firstSeeded, posted, seen, cursor, seenStaged,
|
|
82
|
+
cursorStaged, acceptedCm, acceptedHuman, acceptedIss, pendCm, pendIss>>
|
|
83
|
+
|
|
84
|
+
(* ---- the source: first poll seeds; later polls commit, then stage ---- *)
|
|
85
|
+
|
|
86
|
+
(* The first look records the open issues and fires none of them. The seed commits at
|
|
87
|
+
once: losing it to a crash would fire the whole backlog, and the shipped source
|
|
88
|
+
saves it inside the poll. Comments are read in the same poll, as in the shipped
|
|
89
|
+
source. *)
|
|
90
|
+
FirstPoll ==
|
|
91
|
+
/\ seeded = FALSE
|
|
92
|
+
/\ seeded' = TRUE
|
|
93
|
+
/\ firstSeeded' = opened
|
|
94
|
+
/\ seen' = opened
|
|
95
|
+
/\ seenStaged' = opened
|
|
96
|
+
/\ pendCm' = {c \in posted : CmOf(c).at >= cursor}
|
|
97
|
+
/\ cursorStaged' = IF pendCm' = {} THEN cursor ELSE 1 + MaxNat({CmOf(c).at : c \in pendCm'})
|
|
98
|
+
/\ pendIss' = {}
|
|
99
|
+
/\ UNCHANGED <<crashes, reopened, opened, posted, cursor,
|
|
100
|
+
acceptedCm, acceptedHuman, acceptedIss, openCnt>>
|
|
101
|
+
|
|
102
|
+
(* A later poll: commit the previous stage — only once every trigger it emitted has
|
|
103
|
+
been accepted (the deferred-commit rule). Then stage the next look. *)
|
|
104
|
+
Poll ==
|
|
105
|
+
/\ seeded
|
|
106
|
+
/\ pendCm = {} /\ pendIss = {}
|
|
107
|
+
/\ cursor' = cursorStaged
|
|
108
|
+
/\ seen' = seenStaged
|
|
109
|
+
/\ LET newIss == opened \ seenStaged IN
|
|
110
|
+
/\ pendIss' = newIss
|
|
111
|
+
/\ pendCm' = {c \in posted : CmOf(c).at >= cursor'}
|
|
112
|
+
/\ seenStaged' = (seenStaged \cup newIss) \cap opened
|
|
113
|
+
/\ cursorStaged' = IF pendCm' = {} THEN cursor' ELSE 1 + MaxNat({CmOf(c).at : c \in pendCm'})
|
|
114
|
+
/\ UNCHANGED <<crashes, seeded, firstSeeded, reopened, opened, posted,
|
|
115
|
+
acceptedCm, acceptedHuman, acceptedIss, openCnt>>
|
|
116
|
+
|
|
117
|
+
(* The engine takes each emitted trigger once. Ids are content hashes, so a replayed
|
|
118
|
+
trigger is processed as a no-op (engine.receive reports a duplicate) rather than
|
|
119
|
+
jamming the queue: the commit needs only every emitted trigger processed. *)
|
|
120
|
+
AcceptComment(c) ==
|
|
121
|
+
/\ c \in pendCm
|
|
122
|
+
/\ acceptedCm' = acceptedCm \cup {c}
|
|
123
|
+
/\ acceptedHuman' = IF CmOf(c).human THEN acceptedHuman \cup {c} ELSE acceptedHuman
|
|
124
|
+
/\ pendCm' = pendCm \ {c}
|
|
125
|
+
/\ UNCHANGED <<crashes, seeded, firstSeeded, reopened, opened, posted, seen, cursor,
|
|
126
|
+
seenStaged, cursorStaged, acceptedIss, pendIss, openCnt>>
|
|
127
|
+
|
|
128
|
+
AcceptIssue(i) ==
|
|
129
|
+
/\ i \in pendIss
|
|
130
|
+
/\ acceptedIss' = acceptedIss \cup {i}
|
|
131
|
+
/\ pendIss' = pendIss \ {i}
|
|
132
|
+
/\ openCnt' = IF i \in acceptedIss THEN openCnt ELSE [openCnt EXCEPT ![i] = openCnt[i] + 1]
|
|
133
|
+
/\ UNCHANGED <<crashes, seeded, firstSeeded, reopened, opened, posted, seen, cursor,
|
|
134
|
+
seenStaged, cursorStaged, acceptedCm, acceptedHuman, pendCm>>
|
|
135
|
+
|
|
136
|
+
(* Crash: the staged position dies with the process; committed state and accepted
|
|
137
|
+
triggers survive; emissions in flight are lost and replay from the committed cursor.
|
|
138
|
+
The crash budget is finite and WF(Crash) spends it, so the liveness claim is checked
|
|
139
|
+
through a restart; the replay argument is the same for each further one. *)
|
|
140
|
+
Crash ==
|
|
141
|
+
/\ crashes < 1
|
|
142
|
+
/\ crashes' = crashes + 1
|
|
143
|
+
/\ cursorStaged' = cursor
|
|
144
|
+
/\ seenStaged' = seen
|
|
145
|
+
/\ pendCm' = {} /\ pendIss' = {}
|
|
146
|
+
/\ UNCHANGED <<seeded, firstSeeded, reopened, opened, posted, seen, cursor,
|
|
147
|
+
acceptedCm, acceptedHuman, acceptedIss, openCnt>>
|
|
148
|
+
|
|
149
|
+
Terminate == UNCHANGED vars
|
|
150
|
+
|
|
151
|
+
Next ==
|
|
152
|
+
\/ Terminate
|
|
153
|
+
\/ \E c \in Comments : PostComment(c)
|
|
154
|
+
\/ \E i \in Issues : OpenIssue(i)
|
|
155
|
+
\/ \E i \in Issues : CloseIssue(i)
|
|
156
|
+
\/ FirstPoll
|
|
157
|
+
\/ Poll
|
|
158
|
+
\/ \E c \in Comments : AcceptComment(c)
|
|
159
|
+
\/ \E i \in Issues : AcceptIssue(i)
|
|
160
|
+
\/ Crash
|
|
161
|
+
|
|
162
|
+
Fairness ==
|
|
163
|
+
/\ \A c \in Comments : WF_vars(PostComment(c)) /\ SF_vars(AcceptComment(c))
|
|
164
|
+
/\ \A i \in Issues : WF_vars(OpenIssue(i)) /\ SF_vars(AcceptIssue(i))
|
|
165
|
+
/\ SF_vars(FirstPoll) /\ SF_vars(Poll) /\ WF_vars(Crash)
|
|
166
|
+
|
|
167
|
+
Spec == Init /\ [][Next]_vars /\ Fairness
|
|
168
|
+
|
|
169
|
+
(* ------------------------------------------------------------ the claims ---- *)
|
|
170
|
+
|
|
171
|
+
TypeOK ==
|
|
172
|
+
/\ crashes \in 0..1
|
|
173
|
+
/\ seeded \in BOOLEAN /\ firstSeeded \in SUBSET Issues /\ opened \in SUBSET Issues
|
|
174
|
+
/\ posted \in SUBSET Comments
|
|
175
|
+
/\ seen \in SUBSET Issues /\ seenStaged \in SUBSET Issues
|
|
176
|
+
/\ cursor \in Times /\ cursorStaged \in Times
|
|
177
|
+
/\ acceptedCm \in SUBSET Comments /\ acceptedHuman \in SUBSET Comments
|
|
178
|
+
/\ acceptedIss \in SUBSET Issues
|
|
179
|
+
/\ pendCm \in SUBSET Comments /\ pendIss \in SUBSET Issues
|
|
180
|
+
/\ openCnt \in [Issues -> 0..2] /\ reopened \in [Issues -> BOOLEAN]
|
|
181
|
+
|
|
182
|
+
(* C1. The first look is silent: a seeded issue fires only after a real close and
|
|
183
|
+
reopen. *)
|
|
184
|
+
C1_SeedSilence == acceptedIss \cap firstSeeded \subseteq {i \in Issues : reopened[i]}
|
|
185
|
+
|
|
186
|
+
(* C2. At most one request per open interval of an issue. *)
|
|
187
|
+
C2_ReopenOnce == \A i \in Issues : openCnt[i] <= 1
|
|
188
|
+
|
|
189
|
+
(* C3. The committed cursor only stands over accepted comments: after any crash, every
|
|
190
|
+
comment the cursor has passed was accepted — nothing is lost between poll and
|
|
191
|
+
accept. (Commit-at-emit fails this; the deferred commit is what the source must do.) *)
|
|
192
|
+
C3_NoGhostCommit == [][\A c \in posted : CmOf(c).at < cursor => c \in acceptedCm]_vars
|
|
193
|
+
|
|
194
|
+
(* C4. Acceptance is permanent. Accepting by id is a set union, so a replay adds
|
|
195
|
+
nothing by construction; what TLC checks is that nothing accepted is ever lost. *)
|
|
196
|
+
C4_NoDoubleAccept == [][acceptedCm \subseteq acceptedCm' /\ acceptedIss \subseteq acceptedIss']_vars
|
|
197
|
+
|
|
198
|
+
(* C5. A bot's comment is never taken for a person's request. *)
|
|
199
|
+
C5_BotNeverHuman == acceptedHuman \subseteq {c \in acceptedCm : CmOf(c).human}
|
|
200
|
+
|
|
201
|
+
(* C6. Liveness through a crash: every posted comment is eventually accepted, and
|
|
202
|
+
every issue opened after the seed fires. The model allows one crash; the replay
|
|
203
|
+
argument is the same for each further one. *)
|
|
204
|
+
C6_NoLoss == []<>(posted \subseteq acceptedCm /\ \A i \in opened : i \in acceptedIss \/ i \in firstSeeded)
|
|
205
|
+
|
|
206
|
+
=====================
|
package/formal/README.md
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# TLA+ models
|
|
2
|
+
|
|
3
|
+
A worker finishes just as the router restarts. Should the router start another worker, adopt the
|
|
4
|
+
one it finds, or close the tab? Small differences in timing matter here. These three models let
|
|
5
|
+
TLC, the TLA+ model checker, explore the possible orderings within a set of bounds. Start with
|
|
6
|
+
the model closest to the code you’re changing.
|
|
7
|
+
|
|
8
|
+
| Specification | Code it models | Distinct states |
|
|
9
|
+
| --- | --- | ---: |
|
|
10
|
+
| [Ticket.tla](Ticket.tla) | Ticket lifecycle in `src/core/` | 1,398,882 |
|
|
11
|
+
| [Intake.tla](Intake.tla) | GitHub polling in `src/pi/source-github.ts` | 113,281 |
|
|
12
|
+
| [Workers.tla](Workers.tla) | Worker records and herdr tabs in `src/pi/herdr-workers.ts` | 94 |
|
|
13
|
+
|
|
14
|
+
Each accompanying `.cfg` file sets the bounds and selects the properties to check.
|
|
15
|
+
[Property definitions and assumptions](../docs/formal-verification.md)
|
|
16
|
+
|
|
17
|
+
## Run the checks
|
|
18
|
+
|
|
19
|
+
From the repository root, run `scripts/check-formal`. All three checks take about five minutes and
|
|
20
|
+
require a JDK. The script downloads the TLC jar on first use. To rerun just the intake model, run
|
|
21
|
+
this command from the `formal/` directory.
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
java -XX:+UseParallelGC -Xmx4g -cp ~/.cache/tla2tools/tla2tools.jar tlc2.TLC -config Intake.cfg Intake.tla
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Update a model
|
|
28
|
+
|
|
29
|
+
Each action names the implementation function it represents. When you change a transition or its
|
|
30
|
+
guards, update the corresponding action and rerun TLC. Keep the scope and exclusions in the model
|
|
31
|
+
header current. Choose bounds small enough to explore exhaustively during development.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
CONSTANTS
|
|
2
|
+
t1 = "t1"
|
|
3
|
+
t2 = "t2"
|
|
4
|
+
k1 = "k1"
|
|
5
|
+
k2 = "k2"
|
|
6
|
+
m11 = "m11"
|
|
7
|
+
m12 = "m12"
|
|
8
|
+
m21 = "m21"
|
|
9
|
+
m22 = "m22"
|
|
10
|
+
|
|
11
|
+
SPECIFICATION Spec
|
|
12
|
+
|
|
13
|
+
INVARIANT
|
|
14
|
+
TypeOK
|
|
15
|
+
I2_DoneFromReady
|
|
16
|
+
I3_OneRunning
|
|
17
|
+
I5_OutboxOnce
|
|
18
|
+
|
|
19
|
+
PROPERTY
|
|
20
|
+
I4_NoSecondTicket
|
|
21
|
+
I1_StopAbsorbing
|
|
22
|
+
I6_SentMonotonic
|
|
23
|
+
LT_Delivered
|