@botiverse/k-carrier 0.1.8 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/NOTICE +5 -2
- package/README.md +72 -26
- package/core/src/artifact/download.ts +23 -1
- package/core/src/artifact/gzip.ts +26 -0
- package/core/src/artifact/source.ts +2 -0
- package/core/src/{createUpgrader.ts → createRunner.ts} +13 -15
- package/core/src/index.ts +11 -3
- package/core/src/launcher/launch.ts +15 -0
- package/core/src/launcher/supervise.ts +170 -0
- package/core/src/lifecycle/commandHost.ts +111 -0
- package/core/src/lifecycle/hostAdapter.ts +28 -16
- package/core/src/operation.ts +47 -25
- package/core/src/operationLifecycle.ts +2 -7
- package/core/src/platform/ops.ts +7 -0
- package/core/src/platform/posix.ts +26 -6
- package/core/src/platform/windows.ts +8 -2
- package/core/src/protocol/runner.ts +81 -0
- package/core/src/provenance/journal.ts +1 -1
- package/core/src/runner/cli.ts +27 -0
- package/core/src/runner/execute.ts +68 -0
- package/core/src/txn/engine.ts +41 -85
- package/core/src/txn/fileEffects.ts +15 -1
- package/core/src/txn/hostCallBudget.ts +4 -1
- package/core/src/txn/hostCallUncertain.ts +2 -0
- package/core/src/txn/lock.ts +81 -37
- package/core/src/txn/state.ts +1 -1
- package/core/src/upgrade/drive.ts +31 -2
- package/core/src/upgrade/outcome.ts +1 -1
- package/core/src/upgrade/recover.ts +22 -1
- package/core/src/upgrade/retire.ts +1 -1
- package/core/src/upgrader.ts +4 -9
- package/docs/design.md +173 -0
- package/docs/guide.md +196 -0
- package/docs/harness-design.md +75 -170
- package/docs/integration.md +221 -364
- package/docs/prior-art/design-influences.md +26 -0
- package/docs/prior-art/external-runner-research.md +49 -0
- package/docs/reference.md +209 -0
- package/docs/test-plan.md +89 -92
- package/harness/src/adapter/releaseKnob.ts +1 -1
- package/harness/src/adapter/serviceChecks.ts +5 -5
- package/harness/src/artifact/m1.ts +8 -8
- package/harness/src/artifact/m1Resume.ts +2 -2
- package/harness/src/artifact/m3.ts +25 -104
- package/harness/src/artifact/m3Hosts.ts +9 -61
- package/harness/src/artifact/m4.ts +3 -3
- package/harness/src/artifact/m5.ts +5 -5
- package/harness/src/artifact/m6.ts +6 -6
- package/harness/src/artifact/m6Status.ts +1 -1
- package/harness/src/examples/checks.ts +10 -13
- package/harness/src/fixtures/cliToolSource.ts +166 -0
- package/harness/src/fixtures/externalCrashAdapter.ts +19 -0
- package/harness/src/fixtures/managedHost.ts +100 -0
- package/harness/src/fixtures/serviceSource.ts +181 -0
- package/harness/src/fixtures/supervisedAdapter.ts +57 -0
- package/harness/src/scenario/processScan.ts +3 -1
- package/harness/src/scenario/sandbox.ts +2 -2
- package/harness/src/teeth/artifact.ts +3 -3
- package/harness/src/teeth/examples.ts +1 -1
- package/package.json +5 -3
- package/docs/design-v1.md +0 -246
- package/docs/prior-art.md +0 -150
package/docs/design.md
ADDED
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
# K design
|
|
2
|
+
|
|
3
|
+
This document is the normative contract for K's execution model, transaction,
|
|
4
|
+
supervision and controller boundary. It states obligations; it does not
|
|
5
|
+
explain them. Read [how an upgrade works](guide.md) first for the narrative
|
|
6
|
+
and vocabulary, and the [reference](reference.md) for wire formats, exit
|
|
7
|
+
codes, budgets and file layout.
|
|
8
|
+
|
|
9
|
+
## Execution and trust
|
|
10
|
+
|
|
11
|
+
```mermaid
|
|
12
|
+
flowchart LR
|
|
13
|
+
Entry[Bootstrap / CLI / remote control] --> Supervisor[Verify and launch runner]
|
|
14
|
+
Supervisor --> Worker[K + product adapter]
|
|
15
|
+
Worker --> State[Lock / journal / slots / receipts]
|
|
16
|
+
Worker --> Controller[Product lifecycle controller]
|
|
17
|
+
Controller --> App[Application]
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
- The runner and its runtime live outside the application slots and service
|
|
21
|
+
unit. Stopping the application must leave the worker alive. A child
|
|
22
|
+
process can still belong to a systemd cgroup or Windows job; the publisher
|
|
23
|
+
must arrange isolation.
|
|
24
|
+
- The supervisor retains recovery code until settlement. Installation state
|
|
25
|
+
persists. An OS hook or operator must restart installation after reboot.
|
|
26
|
+
- The adapter is fixed at build time. Requests select an action and a target
|
|
27
|
+
version, never code, commands or download URLs.
|
|
28
|
+
- The publisher authenticates distribution metadata and caller authority. K
|
|
29
|
+
checks artifact SHA-256 and size; these checks do not establish publisher
|
|
30
|
+
identity. See [packaging](integration.md#distribute-a-built-installer).
|
|
31
|
+
|
|
32
|
+
## Components
|
|
33
|
+
|
|
34
|
+
Paths are relative to `core/src/`.
|
|
35
|
+
|
|
36
|
+
| Component | Responsibility |
|
|
37
|
+
|---|---|
|
|
38
|
+
| `launcher/` | Acquire, verify, execute and clean runner code; supervise recovery |
|
|
39
|
+
| `protocol/` | Validate the bounded request/response contract |
|
|
40
|
+
| `runner/` | Serve stdin/stdout, execute requests and map outcomes to exit codes |
|
|
41
|
+
| `createRunner.ts` | Assemble source, host, policy and transaction state |
|
|
42
|
+
| `lifecycle/` | Stop/start/probe the application, including command-based control |
|
|
43
|
+
| `artifact/`, `txn/`, `converge/` | Verified acquisition, durable transactions and readiness predicates |
|
|
44
|
+
|
|
45
|
+
`createRunner(options)` constructs the transaction interface. `serveRunner(factory)`
|
|
46
|
+
serves it in the runner process. `launchRunner({release, request, scratchDir,
|
|
47
|
+
interpreter?})` verifies and supervises a worker, writes one final response and
|
|
48
|
+
returns its exit code. `superviseRunner` returns the same result as data,
|
|
49
|
+
including any retained `recoveryFile`. `resumeRunner(recoveryFile)` verifies
|
|
50
|
+
that retained runner and performs operation-bound recovery without
|
|
51
|
+
distribution access.
|
|
52
|
+
|
|
53
|
+
## Transaction and recovery
|
|
54
|
+
|
|
55
|
+
Each installation has one lock, stable and experiment executable slots, and a
|
|
56
|
+
journal. K stages verified candidate bytes, quiesces work, stops the old
|
|
57
|
+
service, starts the candidate and evaluates live readiness. Passing candidates
|
|
58
|
+
are promoted; failed candidates restore stable. Application data belongs
|
|
59
|
+
outside the slots.
|
|
60
|
+
|
|
61
|
+
The phases are `idle`, `staged`, `handing-over`, `running-experiment`,
|
|
62
|
+
`readback`, `promoted` and `rolled-back`. Intent is journaled before effects.
|
|
63
|
+
Recovery takes the same lock and uses existing slot bytes without release
|
|
64
|
+
lookup:
|
|
65
|
+
|
|
66
|
+
- Before durable promote intent: restore stable.
|
|
67
|
+
- After durable promote intent: replay commit idempotently.
|
|
68
|
+
|
|
69
|
+
A running candidate alone does not authorize commit. Corrupt or unknown state,
|
|
70
|
+
and another live lock owner, prevent conflicting operations. Filesystem
|
|
71
|
+
durability and controller behavior determine the real platform guarantees.
|
|
72
|
+
|
|
73
|
+
## Transaction completion
|
|
74
|
+
|
|
75
|
+
**Required for the initial release.** Every started operation has an owner
|
|
76
|
+
that waits for a durable terminal result or explicitly reports unresolved
|
|
77
|
+
recovery. An installer invocation must first settle unfinished work before
|
|
78
|
+
admitting a new upgrade. Recovering an interrupted operation does not retry
|
|
79
|
+
its requested upgrade.
|
|
80
|
+
|
|
81
|
+
The supervisor:
|
|
82
|
+
|
|
83
|
+
- runs outside the application service unit;
|
|
84
|
+
- retains the verified runner while the operation is active;
|
|
85
|
+
- enforces execution and recovery budgets ([values](reference.md#supervisor-budgets));
|
|
86
|
+
- starts a recovery worker after an abnormal exit;
|
|
87
|
+
- stops supervising after completion or an explicit unresolved result.
|
|
88
|
+
|
|
89
|
+
Recovery attempts and total elapsed time are bounded. Exhaustion preserves
|
|
90
|
+
state and provides a recovery command rather than reporting success.
|
|
91
|
+
|
|
92
|
+
Before starting a successor, the supervisor must observe the prior worker
|
|
93
|
+
exit. The recovery worker then takes the lock and fences outstanding
|
|
94
|
+
controller effects before replaying lifecycle actions. An expired deadline
|
|
95
|
+
alone is insufficient; an unconfirmed exit forbids takeover.
|
|
96
|
+
|
|
97
|
+
Recovery must bind to the original operation id under K's transaction lock.
|
|
98
|
+
If another operation has since run, recovery inspects or replays the original
|
|
99
|
+
result without modifying the newer operation. Recovery reuses the existing
|
|
100
|
+
journal and receipts; supervision does not create another transaction log.
|
|
101
|
+
|
|
102
|
+
Cleanup follows settlement: persist the outcome, release owned resources,
|
|
103
|
+
then remove disposable code. Never delete slots, a live owner's lock, or
|
|
104
|
+
recovery logs to make an interrupted operation appear complete. If recovery
|
|
105
|
+
remains unresolved, preserve the evidence and a verified means to invoke it
|
|
106
|
+
again. Installer startup handles leftover work; machine reboot still requires
|
|
107
|
+
an OS hook or operator to start the installer.
|
|
108
|
+
|
|
109
|
+
Every engine host call has a positive budget. Uncertain effects retain the
|
|
110
|
+
worker's lock until it exits. Bundled workers exit after flushing their
|
|
111
|
+
response; custom in-process callers must also exit on `HostCallUncertain`
|
|
112
|
+
rather than reuse that worker.
|
|
113
|
+
|
|
114
|
+
The lock is a local filesystem protocol requiring atomic creation and
|
|
115
|
+
coherent directory reads ([details](reference.md#lock-protocol)). It is not a
|
|
116
|
+
distributed or NFS lock. Operation ids, receipt archival and replay rules are
|
|
117
|
+
in the [reference](reference.md#receipts-and-retries).
|
|
118
|
+
|
|
119
|
+
## Host control contract
|
|
120
|
+
|
|
121
|
+
`createRunner` requires a HostAdapter. Its operations and the controller's
|
|
122
|
+
obligations:
|
|
123
|
+
|
|
124
|
+
| Operation | Controller obligation |
|
|
125
|
+
|---|---|
|
|
126
|
+
| fence | Drain or fence previous controller effects; required when effects can outlive the worker |
|
|
127
|
+
| quiesce | Stop admission and durably park promised workloads; repeated calls safe |
|
|
128
|
+
| stop | Stop the specified slot's service and confirm termination |
|
|
129
|
+
| start | Start the selected artifact idempotently, without creating duplicate residents |
|
|
130
|
+
| healthProbe | Return version, pid and startId from one ready live instance |
|
|
131
|
+
| resume | Restore parked work on either candidate or rolled-back stable |
|
|
132
|
+
|
|
133
|
+
- The controller must work while the old application is down.
|
|
134
|
+
- `start` returning does not establish readiness; the probe does.
|
|
135
|
+
- K probes once before handover and records that incarnation's `startId`
|
|
136
|
+
with the handing-over intent. Readback evidence carrying the same
|
|
137
|
+
`startId` is refused and rolls back: the old service was not replaced.
|
|
138
|
+
- `start` must not execute the artifact in place inside the slot on
|
|
139
|
+
Windows; promotion renames slot directories and a running executable locks
|
|
140
|
+
its directory. Copy or hard-link to a runtime path outside the slots.
|
|
141
|
+
- A stateless service satisfies `quiesce` and `resume` by acknowledging. The
|
|
142
|
+
obligations apply to workloads the product promises to preserve.
|
|
143
|
+
- Adapters with no effects surviving their worker may omit `fence`. All
|
|
144
|
+
other adapters must supply it and test it against their real service
|
|
145
|
+
manager. `createCommandHost` always delivers `fence`; a command controller
|
|
146
|
+
that queues nothing acknowledges it.
|
|
147
|
+
- If OS lifecycle surfaces are declared, K also requires them to reference
|
|
148
|
+
the promoted artifact before retiring their previous manager. Undeclared
|
|
149
|
+
surfaces are not observed.
|
|
150
|
+
|
|
151
|
+
`createCommandHost` runs an external controller via argv without a shell,
|
|
152
|
+
records the controller pid durably before each call, and drains recorded
|
|
153
|
+
controllers before `fence` during recovery. Wire format, bounds and pid
|
|
154
|
+
handling are in the [reference](reference.md#command-controller-protocol).
|
|
155
|
+
The recorded pid is never used to kill an arbitrary process. `fence`
|
|
156
|
+
acknowledgement is a product contract, not something K infers from process
|
|
157
|
+
exit. Fire-and-forget stop cannot establish termination.
|
|
158
|
+
|
|
159
|
+
## Product responsibilities
|
|
160
|
+
|
|
161
|
+
The adapter defines release lookup, installation ownership, consent,
|
|
162
|
+
notification and compatibility policy. Package-manager-owned installations
|
|
163
|
+
defer to their owner. K restores executables; products provide data-migration
|
|
164
|
+
compatibility, backup and restore, and any promised workload continuity.
|
|
165
|
+
Service upgrades can interrupt availability. Remote authorization,
|
|
166
|
+
distribution channels and cloud reconnection belong to the product
|
|
167
|
+
integration.
|
|
168
|
+
|
|
169
|
+
Use the [integration guide](integration.md) and
|
|
170
|
+
[service example](../examples/external-service/README.md) to build an
|
|
171
|
+
installer. The [test plan](test-plan.md) describes framework and product
|
|
172
|
+
acceptance; [prior art](prior-art/design-influences.md) records design
|
|
173
|
+
influences.
|
package/docs/guide.md
ADDED
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
# How an upgrade works
|
|
2
|
+
|
|
3
|
+
This is the narrative walkthrough of K's mechanism: which processes exist,
|
|
4
|
+
what happens in what order, what breaks, and how to read the result. It is
|
|
5
|
+
written to be read once, top to bottom. It is **not** the contract; where it
|
|
6
|
+
and the [design](design.md) or [reference](reference.md) disagree, they win.
|
|
7
|
+
|
|
8
|
+
For what K is for and whether you need it, read the [README](../README.md).
|
|
9
|
+
For how to build and ship an installer, read the
|
|
10
|
+
[integration guide](integration.md). Terms are defined where they first
|
|
11
|
+
appear in bold; this document is the vocabulary the other documents use.
|
|
12
|
+
|
|
13
|
+
## The pieces
|
|
14
|
+
|
|
15
|
+
```mermaid
|
|
16
|
+
flowchart LR
|
|
17
|
+
B[Bootstrap<br/>install.sh / self upgrade] -->|request| S[Supervisor<br/>launchRunner]
|
|
18
|
+
S -->|verify, spawn| W[Worker<br/>K + adapter]
|
|
19
|
+
W --> D[(State directory<br/>lock · journal · slots · receipts)]
|
|
20
|
+
W -->|fence · quiesce · stop<br/>start · probe · resume| C[Controller]
|
|
21
|
+
C --> A[Application]
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
- The **bootstrap** is your `install.sh`, or the `self upgrade` command inside
|
|
25
|
+
your application. It downloads and verifies the installer, launches it,
|
|
26
|
+
and relays the exit code. It contains no upgrade logic.
|
|
27
|
+
- The **installer** is what you ship. It has two halves. The **supervisor**
|
|
28
|
+
is a temporary process that acquires and verifies the runner, executes it
|
|
29
|
+
and, if it dies, runs bounded recovery. The **runner** is K plus your
|
|
30
|
+
adapter bundled into one executable; it serves one request and exits. A
|
|
31
|
+
running runner is a **worker**.
|
|
32
|
+
- The **adapter** is your trusted code, fixed at build time: where releases
|
|
33
|
+
come from, whether an upgrade may proceed, and how to reach the controller.
|
|
34
|
+
- The **controller** is your program that actually stops, starts and probes
|
|
35
|
+
the application. The adapter can call it directly or through `createCommandHost`.
|
|
36
|
+
- The **state directory** holds the lock, the journal, two **slots** and the
|
|
37
|
+
receipts. Recovery uses this state together with a compatible runner, its
|
|
38
|
+
runtime and the product controller; it does not need release distribution.
|
|
39
|
+
|
|
40
|
+
The runner and the controller live outside the application's service unit.
|
|
41
|
+
That is the whole point: stopping the application must not stop the thing
|
|
42
|
+
that is upgrading it.
|
|
43
|
+
|
|
44
|
+
The slots are **stable** and **experiment**; these are positions
|
|
45
|
+
on disk, not release channels, so a product channel also called "stable" is
|
|
46
|
+
unrelated. *Hands*, mentioned in the integration guide, is the release
|
|
47
|
+
platform K's authors publish with; K does not depend on it.
|
|
48
|
+
|
|
49
|
+
## One upgrade, start to finish
|
|
50
|
+
|
|
51
|
+
```mermaid
|
|
52
|
+
sequenceDiagram
|
|
53
|
+
participant B as Bootstrap
|
|
54
|
+
participant S as Supervisor
|
|
55
|
+
participant W as Worker
|
|
56
|
+
participant C as Controller
|
|
57
|
+
participant A as Application
|
|
58
|
+
B->>S: upgrade id=job-1 target=2.0.0
|
|
59
|
+
S->>S: download runner, check sha256 + size
|
|
60
|
+
S->>W: spawn, request on stdin
|
|
61
|
+
W->>W: take upgrade.lock, inspect operation identity
|
|
62
|
+
W->>C: fence previous controller effects
|
|
63
|
+
W->>W: settle leftovers, begin requested operation
|
|
64
|
+
W->>W: fetch release, verify bytes
|
|
65
|
+
Note over W: journal: staged intent
|
|
66
|
+
W->>W: write experiment slot
|
|
67
|
+
Note over W: journal: handing-over intent
|
|
68
|
+
W->>C: quiesce
|
|
69
|
+
C->>A: park work
|
|
70
|
+
W->>C: stop stable
|
|
71
|
+
C->>A: terminate, confirm exit
|
|
72
|
+
W->>C: start experiment
|
|
73
|
+
C->>A: launch candidate
|
|
74
|
+
W->>C: healthProbe
|
|
75
|
+
C-->>W: version 2.0.0, pid, startId
|
|
76
|
+
Note over W: journal: readback ok → promote intent
|
|
77
|
+
W->>W: experiment becomes stable
|
|
78
|
+
W->>C: resume
|
|
79
|
+
Note over W: persist promoted operation receipt
|
|
80
|
+
W-->>S: {result: promoted}, exit 0
|
|
81
|
+
S-->>B: exit 0
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
In words:
|
|
85
|
+
|
|
86
|
+
1. The bootstrap asks the supervisor for one operation: an id and a target
|
|
87
|
+
version. The id is what everything else binds to.
|
|
88
|
+
2. The supervisor downloads the runner, checks its hash and size, writes it
|
|
89
|
+
to scratch space and runs it with the request on stdin.
|
|
90
|
+
3. The worker takes the installation lock. If an earlier operation was left
|
|
91
|
+
unfinished, it settles that first. The same id replays its result; a new
|
|
92
|
+
id may proceed only after recovery succeeds.
|
|
93
|
+
4. It asks your release source for exactly the target version, verifies the
|
|
94
|
+
bytes, and writes them into the *experiment* slot. Nothing running has
|
|
95
|
+
changed yet.
|
|
96
|
+
5. It asks the controller to quiesce (park in-flight work) and to stop the
|
|
97
|
+
current service, and waits for confirmation that the old process is gone.
|
|
98
|
+
6. It asks the controller to start the candidate from the experiment slot,
|
|
99
|
+
then probes it. The probe must come from one live process and report the
|
|
100
|
+
expected version with a pid and a **startId**. K probed once before
|
|
101
|
+
stopping the old service and journaled that startId; if the same one
|
|
102
|
+
comes back now, the old process was never replaced and the upgrade rolls
|
|
103
|
+
back.
|
|
104
|
+
7. Only now does the worker write **promote intent** to the journal. That
|
|
105
|
+
line is the point of no return: before it, the safe move is always to put
|
|
106
|
+
the old version back; after it, the safe move is always to finish the
|
|
107
|
+
promotion.
|
|
108
|
+
8. Experiment becomes stable. The controller resumes parked work. The
|
|
109
|
+
receipt is persisted in `operation.json`; it is archived before a later
|
|
110
|
+
operation begins. The worker prints its response and exits 0.
|
|
111
|
+
|
|
112
|
+
The transaction journal phases are, in order: `idle`,
|
|
113
|
+
`staged`, `handing-over`, `running-experiment`, `readback`, and then either
|
|
114
|
+
`promoted` or `rolled-back`. Operation receipts additionally track stages such as
|
|
115
|
+
`downloading` and policy outcomes such as `held`.
|
|
116
|
+
|
|
117
|
+
## When something goes wrong
|
|
118
|
+
|
|
119
|
+
### The candidate is bad
|
|
120
|
+
|
|
121
|
+
It does not start, it reports the wrong version, or it never passes the probe.
|
|
122
|
+
The worker stops it, starts stable again, resumes parked work, and records
|
|
123
|
+
`rolled-back`. Exit code 1. The stable bytes were never touched. This is the
|
|
124
|
+
routine failure and the one K is built around.
|
|
125
|
+
|
|
126
|
+
### The worker dies
|
|
127
|
+
|
|
128
|
+
A crash, a kill, or a budget timeout. The supervisor:
|
|
129
|
+
|
|
130
|
+
1. terminates a timed-out worker and waits for its observed exit; a deadline
|
|
131
|
+
alone does not permit takeover;
|
|
132
|
+
2. starts a recovery worker bound to the same operation id;
|
|
133
|
+
3. that worker takes the transaction lock, checks the operation identity,
|
|
134
|
+
and asks the controller to **fence** before replaying lifecycle effects.
|
|
135
|
+
Fencing ensures actions queued by the earlier worker cannot land later.
|
|
136
|
+
An already terminal original operation simply replays its receipt.
|
|
137
|
+
|
|
138
|
+
Recovery does not need the network and does not guess. It reads the journal:
|
|
139
|
+
|
|
140
|
+
```mermaid
|
|
141
|
+
flowchart LR
|
|
142
|
+
idle --> staged --> ho[handing-over] --> re[running-experiment] --> rb[readback]
|
|
143
|
+
rb -->|promote intent| promoted
|
|
144
|
+
rb -.-> rolledback[rolled-back]
|
|
145
|
+
subgraph before [before promote intent: restore stable]
|
|
146
|
+
staged
|
|
147
|
+
ho
|
|
148
|
+
re
|
|
149
|
+
rb
|
|
150
|
+
end
|
|
151
|
+
subgraph after [after promote intent: replay commit]
|
|
152
|
+
promoted
|
|
153
|
+
end
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Before promote intent, recovery restores stable. After it, recovery replays
|
|
157
|
+
the commit. Both are idempotent, so a crash *during* recovery is handled the
|
|
158
|
+
same way next time. Recovery never starts a new upgrade, even if the original
|
|
159
|
+
request was an upgrade.
|
|
160
|
+
|
|
161
|
+
Attempts and elapsed time are bounded. If recovery does not settle within
|
|
162
|
+
those bounds, the supervisor exits 3, leaves the verified runner and a
|
|
163
|
+
`recovery.json` in scratch space, and prints where. Later, `resumeRunner` on
|
|
164
|
+
that file retries the same recovery offline.
|
|
165
|
+
|
|
166
|
+
### Everything dies
|
|
167
|
+
|
|
168
|
+
Power loss, reboot, or the whole invocation killed. The state directory
|
|
169
|
+
survives. The next installer invocation, whether from `install.sh`, from
|
|
170
|
+
`self upgrade`, or from an operator, settles the unfinished operation before
|
|
171
|
+
it accepts new work. K does not install a permanent watchdog, so *something*
|
|
172
|
+
has to run the installer again: your product's OS startup hook, or a person.
|
|
173
|
+
|
|
174
|
+
## Reading the result
|
|
175
|
+
|
|
176
|
+
Every run prints one JSON response on stdout and returns an exit code.
|
|
177
|
+
Inspect both. The short version:
|
|
178
|
+
|
|
179
|
+
| Exit | Meaning | What to do |
|
|
180
|
+
|---|---|---|
|
|
181
|
+
| 0 | Successful upgrade outcome; readable status or recovery with no recorded outcome | Check the action and receipt |
|
|
182
|
+
| 1 | Rolled back, or failed before any change | Read `operation.operation.outcome` and the reason |
|
|
183
|
+
| 2 | Held by policy, ownership or compatibility | Nothing changed; a new attempt needs a new id |
|
|
184
|
+
| 3 | Unresolved | Keep the recovery file and run recovery |
|
|
185
|
+
|
|
186
|
+
Two things surprise people. First, a `recover` action that successfully
|
|
187
|
+
restored stable after a bad candidate exits **1**, because the recorded
|
|
188
|
+
outcome is a rolled-back upgrade. That is the honest answer. Second, `status`
|
|
189
|
+
reads the last receipt, not the live service; it can say `genesis` (no
|
|
190
|
+
operation ever recorded) on a machine that is running fine.
|
|
191
|
+
|
|
192
|
+
The exact response shape and every exit code are in the
|
|
193
|
+
[reference](reference.md#protocol-v1).
|
|
194
|
+
|
|
195
|
+
What each step asks of your controller, and how to package and ship the
|
|
196
|
+
result, is the [integration guide](integration.md).
|
package/docs/harness-design.md
CHANGED
|
@@ -1,170 +1,75 @@
|
|
|
1
|
-
#
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
```
|
|
77
|
-
- **分档执行**:`--profile swap|daemon|managed` 选齿集;cli 档误挂 L2 齿 ⇒ 注册期报错(档界齿)。
|
|
78
|
-
- **断言二分机械化**:`kind` 必填 invariant 或 baseline-带失效条件;CI 扫无标注断言(断言纪律的执行器)。
|
|
79
|
-
- **mutation-runner 对接**:registry 导出齿清单 + must-red 表,Lincan 的 runner 直接消费(未变异 baseline 0 失败 / 每齿变异必红 / 全红也不发结论)。
|
|
80
|
-
|
|
81
|
-
### 1.6 自验(M0 出口,harness 的上岗证)
|
|
82
|
-
harness 判定别人之前先判自己,三样本缺一不可:
|
|
83
|
-
- **known-green**:正确实现走完整升级 → 必须全绿;
|
|
84
|
-
- **known-red**:注入一个已知故障 → 对应齿必须红、且只红该红的;
|
|
85
|
-
- **对抗样本**:**结构上能过齿的检查、但违反真实 oracle** 的假实现(例:probe 换 pid 不换 startId 报新版本;quiesce 把账本备份再恢复伪装等价)→ 必须被抓。对抗样本清单随齿长(每次真实逃逸事后加一条)。
|
|
86
|
-
自验不过 ⇒ harness 拒绝运行任何评审(exit 非零 + typed 原因)。
|
|
87
|
-
|
|
88
|
-
### 1.7 接入方模式(`k-harness --profile X --adapter path`)
|
|
89
|
-
同一套齿对**外部真 adapter** 跑合规子集(不跑需要故障开关的齿,跑契约齿:quiesce↔resume 等价、probe 活性、ownership 响应)。绿 = 接入方契约达标;这也是 examples 三 demo 的验收方式——**demo 和接入方走同一道门**。
|
|
90
|
-
|
|
91
|
-
## 1.75 两个测试平面:黑盒优先(08-05:"就像启动一个 CLI、跑它的命令")
|
|
92
|
-
|
|
93
|
-
harness 有两个平面,**默认用外面那个**:
|
|
94
|
-
|
|
95
|
-
- **黑盒平面(主平面)**:spawn **真实打包好的二进制**,只通过它的命令行驱动(`mytool self upgrade` / `mytool status`),从外面断言:exit code、输出、盘上文件、进程状态、下次运行的版本。**就是用户的用法** —— 它顺带真正验证了"每个入口构造同一 Upgrader"这类 claim(library 平面验不了打包/入口接线)。examples 三 demo 都是真 CLI,端到端齿全在这层写。
|
|
96
|
-
- **library 平面(辅助)**:import core API 直驱 Upgrader——只留给黑盒够不着的内部齿(如 journal 重放细节)。
|
|
97
|
-
|
|
98
|
-
**规则:能在黑盒层表达的齿必须写在黑盒层**;library 层是例外、要说明为什么外面够不着。(同我们 symptom-layer 教义:用户层的红是最不可伪造的 oracle。)
|
|
99
|
-
|
|
100
|
-
接入方黑盒模式随之而来:`k-harness --profile swap --bin ./mytool` —— **零代码集成**:给你的真二进制,harness 起 fake-server、跑你的升级命令、断言下次运行版本/回滚/held。比 `--adapter` 还轻(cli 档接入方连 adapter 都不用给)。
|
|
101
|
-
|
|
102
|
-
## 1.76 黑盒 CLI 契约(08-05:"需要定义查版本子命令、状态 predicate 之类")
|
|
103
|
-
|
|
104
|
-
黑盒平面要从外面问二进制三类问题,因此有一个**小的 CLI 契约** —— 但它是"声明"不是"每家自己发明":
|
|
105
|
-
|
|
106
|
-
**① core 白送命令实现**(app 只做一行接线):
|
|
107
|
-
- `versionCommand()` → 打印二进制自身版本(cli 档够用);
|
|
108
|
-
- `statusCommand()` → **问活进程**(走宿主 socket,同 same-PID 纪律)输出机读 JSON:`{ProcessEvidence, TxnState, ConvergenceReport}` —— 就是 core 已有的三个类型,不另造 schema;
|
|
109
|
-
- `selfUpgradeCommand()` → 包装 `upgrader.upgrade()`,outcome 四态按结构化输出。
|
|
110
|
-
|
|
111
|
-
**② app 声明命令名**(K 不规定你的 CLI 长相,但声明是**必须的**——harness 不猜命令,缺声明立即 typed FAIL):二进制旁边放一个 `k.target.ts`(或 `--target <path>`),default export typed `BlackBoxTarget`,命令名全部显式声明:
|
|
112
|
-
```ts
|
|
113
|
-
import type { BlackBoxTarget } from "@k-carrier/harness";
|
|
114
|
-
|
|
115
|
-
export default {
|
|
116
|
-
version: ["--version"],
|
|
117
|
-
status: ["k-status", "--json"], // 可选
|
|
118
|
-
selfUpgrade: ["self", "upgrade"],
|
|
119
|
-
env: { K_RELEASE_BASE: "..." }, // 可选
|
|
120
|
-
} satisfies BlackBoxTarget;
|
|
121
|
-
```
|
|
122
|
-
`k-harness --bin ./mytool` 动态 import 它来驱动(node 原生跑 TS,不用 build)。**没有 `k.target.ts` = 立即 typed FAIL(`BLACKBOX_TARGET_REQUIRED`),不探测、不猜**——猜对了省一行配置,猜错了给出的是一个可信的错误结论(不确定时要求显式声明,不替用户发明规矩)。
|
|
123
|
-
|
|
124
|
-
**③ 与透明性原则自洽**:这些不是测试后门,是**产品本来就该有的面**(用户和 support 一样需要 `status --json`)——harness 只是恰好消费它们。cli 档最小契约 = `version` + `selfUpgrade` 两条;daemon/managed 档 + `status`(活进程 JSON)。
|
|
125
|
-
|
|
126
|
-
**齿**:契约自身可验 —— `k.target.ts` 声明的命令跑不通 / status 输出不合 schema ⇒ 黑盒验收直接 FAIL(typed,不进齿评审);缺 target 文件 ⇒ 必 FAIL 且信息可操作(齿 `blackbox.missing-target-fails`)。
|
|
127
|
-
|
|
128
|
-
## 1.77 版本工件工厂 + 清场(08-05:"准备相应版本的二进制?删除清空?")
|
|
129
|
-
|
|
130
|
-
**① artifact-factory(升级测试需要"同一个 app 的 vX 和 vY")**:
|
|
131
|
-
- `makeRelease({version, behavior})` → 产出**盖了版本戳的真二进制** + manifest,落到 fake-server。实现 = **一次构建、多次盖戳**(构建 demo 源码一次,post-build 往二进制里注入版本串——与真 SEA 嵌版本同型,快且真实;不用"版本写在旁边文件"那种假形态)。
|
|
132
|
-
- `behavior` 旋钮让某个"新版本"**故意坏**:`crash-on-start / wrong-probe / hang-on-quiesce ...` —— 回滚齿、known-red、对抗样本的 fixture 都从这来("升到坏版本→自动回滚→stable 完好"整条链可黑盒复现)。
|
|
133
|
-
- 内容寻址缓存(key = demo 源 hash × version × behavior),跨场景复用,不重复构建。
|
|
134
|
-
|
|
135
|
-
**② 清场(teardown)**:
|
|
136
|
-
- 沙箱边界即清场边界:install dir + stateDir + fake-server 存储全在场景沙箱内。teardown = **杀进程树并确认真死**(按沙箱标记 pgrep 复核零残留——僵尸 `__service` 是我们的真实产线教训,"发了 kill"≠"死了")→ 删沙箱目录。崩溃场景故意留下的中间态也被同一动作清干净(一切都在沙箱里,所以 rm 恒有效)。
|
|
137
|
-
- **越界写齿**:core 在任何场景中写沙箱外任何路径 ⇒ RED。这颗齿顺带保证了产品级卫生(升级器不污染全局 HOME/系统目录),也让"清空"永远可信——**能一键删干净,是因为先机械保证了它只写在自己地盘**。
|
|
138
|
-
- 顺带的产品映射:沙箱清单 = 将来"干净卸载"要删的东西的权威地图(卸载功能本身另立,不在本期)。
|
|
139
|
-
|
|
140
|
-
## 1.8 透明性原则(08-05:测试框架对升级框架透明)
|
|
141
|
-
|
|
142
|
-
**core 对 harness 零感知,机械强制**:
|
|
143
|
-
- **禁 test-conditional**:core 内不得存在 "if under test" 任何形态(环境变量开关/测试模式 flag/AllowUnsigned 之类后门)。CI ratchet 扫 core 源码(同我们 clock-ratchet 手法),出现即红。
|
|
144
|
-
- harness 需要的一切必须走**产品级注入面**——这些面是产品本来就需要的,不是为测试开的:
|
|
145
|
-
- `HostAdapter`:产品 API 本体,fake-host 只是又一个 adapter;
|
|
146
|
-
- `releaseBase`:指向 localhost 是配置,不是测试感知;
|
|
147
|
-
- `clock`:时钟 seam 是正当的生产抽象(默认真时钟),不是测试后门;
|
|
148
|
-
- `stateDir`:本就按 app 配置。
|
|
149
|
-
- 崩溃注入 = 对真进程 kill -9,零 core 配合;故障注入全在 fake-host(harness 侧代码);对抗样本 = 假 adapter——全部外部。
|
|
150
|
-
- **反向信号**:若某颗齿写不出来、除非给 core 开后门 ⇒ 判定为**公共 API 不足**(dogfood 信号),修 API 而不是开门。透明性由此与 forcing-function 同构:测试框架也只能是 core 的一个普通消费者。
|
|
151
|
-
|
|
152
|
-
## 2. 关键设计决定(为什么这样)
|
|
153
|
-
1. **真进程优先**:崩溃/双跑/probe 活性只在真 spawn 的假 daemon 上验——mock 崩溃 = 没测崩溃。inproc 只服务快速逻辑齿。
|
|
154
|
-
2. **枚举生成覆盖面**:kill 矩阵、齿-档对账、断言标注扫描全由代码生成/校验,**人列清单在这三处非法**(人会漏,且漏的方向总是"看起来覆盖够了")。
|
|
155
|
-
3. **虚拟时钟 + 沙箱**:决定论优先;flaky 即 bug。
|
|
156
|
-
4. **齿注册表是唯一入口**:declaration = 纪律载体(分档/must-red/二分标注都在注册时强制),绕开注册表的测试 CI 拒收。
|
|
157
|
-
5. **对抗样本制度化**:自验含"骗过检查但违反 oracle"的样本,且逃逸事后必须沉淀为新对抗样本——今天 uninstall 设计 那套 BLIND 对抗采样的教训直接机械化。
|
|
158
|
-
|
|
159
|
-
## 3. 实现顺序(M0 内部)
|
|
160
|
-
1. teeth 注册表 + 分档执行器 + 二分标注检查(纯逻辑,先立规矩);
|
|
161
|
-
2. scenario 沙箱 + 虚拟时钟;
|
|
162
|
-
3. fake-server(静态服务 + 篡改 API);
|
|
163
|
-
4. fake-host inproc → fake-host daemon(真进程);
|
|
164
|
-
5. crash 枚举器(吃 core 状态机表——此时 core 只需 `txn/state.ts` 的类型,已存在);
|
|
165
|
-
6. 自验三样本 → **M0 出口**。
|
|
166
|
-
此后每个功能层(M1–M6)的落地 = 先在 registry 写该层的齿(RED)→ 实现层 → 齿转 GREEN。
|
|
167
|
-
|
|
168
|
-
## 4. 边界
|
|
169
|
-
- harness 不测 UI/产品语义(壳仓库自己的事);只测 core 契约 + 接入方 adapter 合规。
|
|
170
|
-
- 真机/平台矩阵(mac launchd、Windows 服务)走 Testbed 轮,harness 出可移植齿、Testbed 供真床。
|
|
1
|
+
# Harness design
|
|
2
|
+
|
|
3
|
+
The harness tests K's transaction mechanisms. The external runner tests exercise
|
|
4
|
+
application integration. Neither replaces acceptance tests for a product's actual
|
|
5
|
+
service manager, installer packaging and data.
|
|
6
|
+
|
|
7
|
+
## Test layers
|
|
8
|
+
|
|
9
|
+
| Layer | Implementation | Evidence |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| Mechanisms | Core unit tests, harness checks, injected effects | Locking, journal ordering, slots, policy, rollback and convergence |
|
|
12
|
+
| Execution boundary | Protocol, launcher, runner and command-controller tests | Input rejection, verified runner execution, bounded calls, receipt and exit-code binding |
|
|
13
|
+
| Real processes | `core/src/runner/process.test.ts` | A separate runner upgrades an application with no K dependency; another worker recovers after a crash |
|
|
14
|
+
| Product acceptance | Product repository and target machines | Real service isolation, packaging, workload restoration and data compatibility |
|
|
15
|
+
|
|
16
|
+
`pnpm test` runs the first three layers; `pnpm test:runner` selects the runner
|
|
17
|
+
boundary and integration tests. Direct Node tests are not all registered harness
|
|
18
|
+
checks, so `--list` is not the complete test inventory.
|
|
19
|
+
|
|
20
|
+
## Components
|
|
21
|
+
|
|
22
|
+
All paths below are under `harness/src/`.
|
|
23
|
+
|
|
24
|
+
| Directory | Role |
|
|
25
|
+
|---|---|
|
|
26
|
+
| `fake-host/` | In-process hosts and spawned service processes with controllable faults |
|
|
27
|
+
| `fake-server/` | Local artifact delivery, range requests and corrupted responses |
|
|
28
|
+
| `artifact-factory/` | Runnable, version-stamped fixture artifacts with selectable failures |
|
|
29
|
+
| `fixtures/` | Internal byte-replacement, service and workload-ledger fixtures |
|
|
30
|
+
| `scenario/` | Isolated sandboxes and virtual time |
|
|
31
|
+
| `crash/` | Transition-derived fault matrix and recovery assertions |
|
|
32
|
+
| `sim/` | Seeded fault scheduling, invariant checks and replayable failure records |
|
|
33
|
+
| `teeth/` | Named checks, selection and known-green/known-red tests |
|
|
34
|
+
|
|
35
|
+
Core receives host, source, clock and effects through normal interfaces. It must
|
|
36
|
+
not detect tests or gain test-only bypasses. Logical tests use virtual time;
|
|
37
|
+
process/network tests use bounded waits and clean up their resources.
|
|
38
|
+
|
|
39
|
+
## Registered checks
|
|
40
|
+
|
|
41
|
+
A registered check (called a *tooth* in the code) declares its id, tested layers,
|
|
42
|
+
profile, invariant or baseline failure condition, and a mutation that must fail.
|
|
43
|
+
The registry rejects invalid declarations and duplicate ids. Known-green cases
|
|
44
|
+
establish that correct behavior passes; known-red and adversarial cases establish
|
|
45
|
+
that the check catches its intended failure. A declared mutation is not evidence
|
|
46
|
+
that a separate mutation campaign has run.
|
|
47
|
+
|
|
48
|
+
The harness profiles `swap` and `service` select mechanism checks. Internal
|
|
49
|
+
fixtures invoke the engine directly; product examples use the external runner.
|
|
50
|
+
|
|
51
|
+
`--adapter` runs the subset supported by a harness adapter's declared contract;
|
|
52
|
+
it expects the harness driver interface, not an arbitrary product HostAdapter.
|
|
53
|
+
`--bin` drives a fixture or binary through explicitly declared commands in
|
|
54
|
+
`k.target.ts` (or `--target`). Missing command declarations fail; the harness does
|
|
55
|
+
not infer CLI names. See `node harness/src/cli.ts --help` for current options.
|
|
56
|
+
|
|
57
|
+
## Fault coverage and limits
|
|
58
|
+
|
|
59
|
+
The crash enumerator crosses the transaction transition table with three points:
|
|
60
|
+
before journal write, after journal write and after the action. The matrix uses
|
|
61
|
+
injected effects and simulated crashes. Directed real-process tests separately
|
|
62
|
+
kill a worker between stop and start and verify recovery. The generated matrix
|
|
63
|
+
is not a claim that every point was tested with OS kills or physical power loss.
|
|
64
|
+
|
|
65
|
+
Seeded simulation explores additional effect interleavings. Its receipts preserve
|
|
66
|
+
the seed and transcript hash; failing seeds are saved for replay. Regressions
|
|
67
|
+
should become fixed cases. Simulation does not establish filesystem durability,
|
|
68
|
+
service-manager isolation or platform-specific executable replacement.
|
|
69
|
+
|
|
70
|
+
Recovery uses the same contract at every layer: before durable promote intent,
|
|
71
|
+
restore stable; after it, replay commit. Product controllers must also isolate
|
|
72
|
+
unfinished asynchronous effects before retrying a timed-out operation.
|
|
73
|
+
|
|
74
|
+
See the [test plan](test-plan.md) for commands and acceptance criteria, and the
|
|
75
|
+
[design](design.md) for the transaction and wire contracts.
|