@botiverse/k-carrier 0.1.8 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (62) hide show
  1. package/NOTICE +5 -2
  2. package/README.md +72 -26
  3. package/core/src/artifact/download.ts +23 -1
  4. package/core/src/artifact/gzip.ts +26 -0
  5. package/core/src/artifact/source.ts +2 -0
  6. package/core/src/{createUpgrader.ts → createRunner.ts} +13 -15
  7. package/core/src/index.ts +11 -3
  8. package/core/src/launcher/launch.ts +15 -0
  9. package/core/src/launcher/supervise.ts +170 -0
  10. package/core/src/lifecycle/commandHost.ts +111 -0
  11. package/core/src/lifecycle/hostAdapter.ts +28 -16
  12. package/core/src/operation.ts +47 -25
  13. package/core/src/operationLifecycle.ts +2 -7
  14. package/core/src/platform/ops.ts +7 -0
  15. package/core/src/platform/posix.ts +26 -6
  16. package/core/src/platform/windows.ts +8 -2
  17. package/core/src/protocol/runner.ts +81 -0
  18. package/core/src/provenance/journal.ts +1 -1
  19. package/core/src/runner/cli.ts +27 -0
  20. package/core/src/runner/execute.ts +68 -0
  21. package/core/src/txn/engine.ts +41 -85
  22. package/core/src/txn/fileEffects.ts +15 -1
  23. package/core/src/txn/hostCallBudget.ts +4 -1
  24. package/core/src/txn/hostCallUncertain.ts +2 -0
  25. package/core/src/txn/lock.ts +81 -37
  26. package/core/src/txn/state.ts +1 -1
  27. package/core/src/upgrade/drive.ts +31 -2
  28. package/core/src/upgrade/outcome.ts +1 -1
  29. package/core/src/upgrade/recover.ts +22 -1
  30. package/core/src/upgrade/retire.ts +1 -1
  31. package/core/src/upgrader.ts +4 -9
  32. package/docs/design.md +173 -0
  33. package/docs/guide.md +196 -0
  34. package/docs/harness-design.md +75 -170
  35. package/docs/integration.md +221 -364
  36. package/docs/prior-art/design-influences.md +26 -0
  37. package/docs/prior-art/external-runner-research.md +49 -0
  38. package/docs/reference.md +209 -0
  39. package/docs/test-plan.md +89 -92
  40. package/harness/src/adapter/releaseKnob.ts +1 -1
  41. package/harness/src/adapter/serviceChecks.ts +5 -5
  42. package/harness/src/artifact/m1.ts +8 -8
  43. package/harness/src/artifact/m1Resume.ts +2 -2
  44. package/harness/src/artifact/m3.ts +25 -104
  45. package/harness/src/artifact/m3Hosts.ts +9 -61
  46. package/harness/src/artifact/m4.ts +3 -3
  47. package/harness/src/artifact/m5.ts +5 -5
  48. package/harness/src/artifact/m6.ts +6 -6
  49. package/harness/src/artifact/m6Status.ts +1 -1
  50. package/harness/src/examples/checks.ts +10 -13
  51. package/harness/src/fixtures/cliToolSource.ts +166 -0
  52. package/harness/src/fixtures/externalCrashAdapter.ts +19 -0
  53. package/harness/src/fixtures/managedHost.ts +100 -0
  54. package/harness/src/fixtures/serviceSource.ts +181 -0
  55. package/harness/src/fixtures/supervisedAdapter.ts +57 -0
  56. package/harness/src/scenario/processScan.ts +3 -1
  57. package/harness/src/scenario/sandbox.ts +2 -2
  58. package/harness/src/teeth/artifact.ts +3 -3
  59. package/harness/src/teeth/examples.ts +1 -1
  60. package/package.json +5 -3
  61. package/docs/design-v1.md +0 -246
  62. package/docs/prior-art.md +0 -150
package/docs/design.md ADDED
@@ -0,0 +1,173 @@
1
+ # K design
2
+
3
+ This document is the normative contract for K's execution model, transaction,
4
+ supervision and controller boundary. It states obligations; it does not
5
+ explain them. Read [how an upgrade works](guide.md) first for the narrative
6
+ and vocabulary, and the [reference](reference.md) for wire formats, exit
7
+ codes, budgets and file layout.
8
+
9
+ ## Execution and trust
10
+
11
+ ```mermaid
12
+ flowchart LR
13
+ Entry[Bootstrap / CLI / remote control] --> Supervisor[Verify and launch runner]
14
+ Supervisor --> Worker[K + product adapter]
15
+ Worker --> State[Lock / journal / slots / receipts]
16
+ Worker --> Controller[Product lifecycle controller]
17
+ Controller --> App[Application]
18
+ ```
19
+
20
+ - The runner and its runtime live outside the application slots and service
21
+ unit. Stopping the application must leave the worker alive. A child
22
+ process can still belong to a systemd cgroup or Windows job; the publisher
23
+ must arrange isolation.
24
+ - The supervisor retains recovery code until settlement. Installation state
25
+ persists. An OS hook or operator must restart installation after reboot.
26
+ - The adapter is fixed at build time. Requests select an action and a target
27
+ version, never code, commands or download URLs.
28
+ - The publisher authenticates distribution metadata and caller authority. K
29
+ checks artifact SHA-256 and size; these checks do not establish publisher
30
+ identity. See [packaging](integration.md#distribute-a-built-installer).
31
+
32
+ ## Components
33
+
34
+ Paths are relative to `core/src/`.
35
+
36
+ | Component | Responsibility |
37
+ |---|---|
38
+ | `launcher/` | Acquire, verify, execute and clean runner code; supervise recovery |
39
+ | `protocol/` | Validate the bounded request/response contract |
40
+ | `runner/` | Serve stdin/stdout, execute requests and map outcomes to exit codes |
41
+ | `createRunner.ts` | Assemble source, host, policy and transaction state |
42
+ | `lifecycle/` | Stop/start/probe the application, including command-based control |
43
+ | `artifact/`, `txn/`, `converge/` | Verified acquisition, durable transactions and readiness predicates |
44
+
45
+ `createRunner(options)` constructs the transaction interface. `serveRunner(factory)`
46
+ serves it in the runner process. `launchRunner({release, request, scratchDir,
47
+ interpreter?})` verifies and supervises a worker, writes one final response and
48
+ returns its exit code. `superviseRunner` returns the same result as data,
49
+ including any retained `recoveryFile`. `resumeRunner(recoveryFile)` verifies
50
+ that retained runner and performs operation-bound recovery without
51
+ distribution access.
52
+
53
+ ## Transaction and recovery
54
+
55
+ Each installation has one lock, stable and experiment executable slots, and a
56
+ journal. K stages verified candidate bytes, quiesces work, stops the old
57
+ service, starts the candidate and evaluates live readiness. Passing candidates
58
+ are promoted; failed candidates restore stable. Application data belongs
59
+ outside the slots.
60
+
61
+ The phases are `idle`, `staged`, `handing-over`, `running-experiment`,
62
+ `readback`, `promoted` and `rolled-back`. Intent is journaled before effects.
63
+ Recovery takes the same lock and uses existing slot bytes without release
64
+ lookup:
65
+
66
+ - Before durable promote intent: restore stable.
67
+ - After durable promote intent: replay commit idempotently.
68
+
69
+ A running candidate alone does not authorize commit. Corrupt or unknown state,
70
+ and another live lock owner, prevent conflicting operations. Filesystem
71
+ durability and controller behavior determine the real platform guarantees.
72
+
73
+ ## Transaction completion
74
+
75
+ **Required for the initial release.** Every started operation has an owner
76
+ that waits for a durable terminal result or explicitly reports unresolved
77
+ recovery. An installer invocation must first settle unfinished work before
78
+ admitting a new upgrade. Recovering an interrupted operation does not retry
79
+ its requested upgrade.
80
+
81
+ The supervisor:
82
+
83
+ - runs outside the application service unit;
84
+ - retains the verified runner while the operation is active;
85
+ - enforces execution and recovery budgets ([values](reference.md#supervisor-budgets));
86
+ - starts a recovery worker after an abnormal exit;
87
+ - stops supervising after completion or an explicit unresolved result.
88
+
89
+ Recovery attempts and total elapsed time are bounded. Exhaustion preserves
90
+ state and provides a recovery command rather than reporting success.
91
+
92
+ Before starting a successor, the supervisor must observe the prior worker
93
+ exit. The recovery worker then takes the lock and fences outstanding
94
+ controller effects before replaying lifecycle actions. An expired deadline
95
+ alone is insufficient; an unconfirmed exit forbids takeover.
96
+
97
+ Recovery must bind to the original operation id under K's transaction lock.
98
+ If another operation has since run, recovery inspects or replays the original
99
+ result without modifying the newer operation. Recovery reuses the existing
100
+ journal and receipts; supervision does not create another transaction log.
101
+
102
+ Cleanup follows settlement: persist the outcome, release owned resources,
103
+ then remove disposable code. Never delete slots, a live owner's lock, or
104
+ recovery logs to make an interrupted operation appear complete. If recovery
105
+ remains unresolved, preserve the evidence and a verified means to invoke it
106
+ again. Installer startup handles leftover work; machine reboot still requires
107
+ an OS hook or operator to start the installer.
108
+
109
+ Every engine host call has a positive budget. Uncertain effects retain the
110
+ worker's lock until it exits. Bundled workers exit after flushing their
111
+ response; custom in-process callers must also exit on `HostCallUncertain`
112
+ rather than reuse that worker.
113
+
114
+ The lock is a local filesystem protocol requiring atomic creation and
115
+ coherent directory reads ([details](reference.md#lock-protocol)). It is not a
116
+ distributed or NFS lock. Operation ids, receipt archival and replay rules are
117
+ in the [reference](reference.md#receipts-and-retries).
118
+
119
+ ## Host control contract
120
+
121
+ `createRunner` requires a HostAdapter. Its operations and the controller's
122
+ obligations:
123
+
124
+ | Operation | Controller obligation |
125
+ |---|---|
126
+ | fence | Drain or fence previous controller effects; required when effects can outlive the worker |
127
+ | quiesce | Stop admission and durably park promised workloads; repeated calls safe |
128
+ | stop | Stop the specified slot's service and confirm termination |
129
+ | start | Start the selected artifact idempotently, without creating duplicate residents |
130
+ | healthProbe | Return version, pid and startId from one ready live instance |
131
+ | resume | Restore parked work on either candidate or rolled-back stable |
132
+
133
+ - The controller must work while the old application is down.
134
+ - `start` returning does not establish readiness; the probe does.
135
+ - K probes once before handover and records that incarnation's `startId`
136
+ with the handing-over intent. Readback evidence carrying the same
137
+ `startId` is refused and rolls back: the old service was not replaced.
138
+ - `start` must not execute the artifact in place inside the slot on
139
+ Windows; promotion renames slot directories and a running executable locks
140
+ its directory. Copy or hard-link to a runtime path outside the slots.
141
+ - A stateless service satisfies `quiesce` and `resume` by acknowledging. The
142
+ obligations apply to workloads the product promises to preserve.
143
+ - Adapters with no effects surviving their worker may omit `fence`. All
144
+ other adapters must supply it and test it against their real service
145
+ manager. `createCommandHost` always delivers `fence`; a command controller
146
+ that queues nothing acknowledges it.
147
+ - If OS lifecycle surfaces are declared, K also requires them to reference
148
+ the promoted artifact before retiring their previous manager. Undeclared
149
+ surfaces are not observed.
150
+
151
+ `createCommandHost` runs an external controller via argv without a shell,
152
+ records the controller pid durably before each call, and drains recorded
153
+ controllers before `fence` during recovery. Wire format, bounds and pid
154
+ handling are in the [reference](reference.md#command-controller-protocol).
155
+ The recorded pid is never used to kill an arbitrary process. `fence`
156
+ acknowledgement is a product contract, not something K infers from process
157
+ exit. Fire-and-forget stop cannot establish termination.
158
+
159
+ ## Product responsibilities
160
+
161
+ The adapter defines release lookup, installation ownership, consent,
162
+ notification and compatibility policy. Package-manager-owned installations
163
+ defer to their owner. K restores executables; products provide data-migration
164
+ compatibility, backup and restore, and any promised workload continuity.
165
+ Service upgrades can interrupt availability. Remote authorization,
166
+ distribution channels and cloud reconnection belong to the product
167
+ integration.
168
+
169
+ Use the [integration guide](integration.md) and
170
+ [service example](../examples/external-service/README.md) to build an
171
+ installer. The [test plan](test-plan.md) describes framework and product
172
+ acceptance; [prior art](prior-art/design-influences.md) records design
173
+ influences.
package/docs/guide.md ADDED
@@ -0,0 +1,196 @@
1
+ # How an upgrade works
2
+
3
+ This is the narrative walkthrough of K's mechanism: which processes exist,
4
+ what happens in what order, what breaks, and how to read the result. It is
5
+ written to be read once, top to bottom. It is **not** the contract; where it
6
+ and the [design](design.md) or [reference](reference.md) disagree, they win.
7
+
8
+ For what K is for and whether you need it, read the [README](../README.md).
9
+ For how to build and ship an installer, read the
10
+ [integration guide](integration.md). Terms are defined where they first
11
+ appear in bold; this document is the vocabulary the other documents use.
12
+
13
+ ## The pieces
14
+
15
+ ```mermaid
16
+ flowchart LR
17
+ B[Bootstrap<br/>install.sh / self upgrade] -->|request| S[Supervisor<br/>launchRunner]
18
+ S -->|verify, spawn| W[Worker<br/>K + adapter]
19
+ W --> D[(State directory<br/>lock · journal · slots · receipts)]
20
+ W -->|fence · quiesce · stop<br/>start · probe · resume| C[Controller]
21
+ C --> A[Application]
22
+ ```
23
+
24
+ - The **bootstrap** is your `install.sh`, or the `self upgrade` command inside
25
+ your application. It downloads and verifies the installer, launches it,
26
+ and relays the exit code. It contains no upgrade logic.
27
+ - The **installer** is what you ship. It has two halves. The **supervisor**
28
+ is a temporary process that acquires and verifies the runner, executes it
29
+ and, if it dies, runs bounded recovery. The **runner** is K plus your
30
+ adapter bundled into one executable; it serves one request and exits. A
31
+ running runner is a **worker**.
32
+ - The **adapter** is your trusted code, fixed at build time: where releases
33
+ come from, whether an upgrade may proceed, and how to reach the controller.
34
+ - The **controller** is your program that actually stops, starts and probes
35
+ the application. The adapter can call it directly or through `createCommandHost`.
36
+ - The **state directory** holds the lock, the journal, two **slots** and the
37
+ receipts. Recovery uses this state together with a compatible runner, its
38
+ runtime and the product controller; it does not need release distribution.
39
+
40
+ The runner and the controller live outside the application's service unit.
41
+ That is the whole point: stopping the application must not stop the thing
42
+ that is upgrading it.
43
+
44
+ The slots are **stable** and **experiment**; these are positions
45
+ on disk, not release channels, so a product channel also called "stable" is
46
+ unrelated. *Hands*, mentioned in the integration guide, is the release
47
+ platform K's authors publish with; K does not depend on it.
48
+
49
+ ## One upgrade, start to finish
50
+
51
+ ```mermaid
52
+ sequenceDiagram
53
+ participant B as Bootstrap
54
+ participant S as Supervisor
55
+ participant W as Worker
56
+ participant C as Controller
57
+ participant A as Application
58
+ B->>S: upgrade id=job-1 target=2.0.0
59
+ S->>S: download runner, check sha256 + size
60
+ S->>W: spawn, request on stdin
61
+ W->>W: take upgrade.lock, inspect operation identity
62
+ W->>C: fence previous controller effects
63
+ W->>W: settle leftovers, begin requested operation
64
+ W->>W: fetch release, verify bytes
65
+ Note over W: journal: staged intent
66
+ W->>W: write experiment slot
67
+ Note over W: journal: handing-over intent
68
+ W->>C: quiesce
69
+ C->>A: park work
70
+ W->>C: stop stable
71
+ C->>A: terminate, confirm exit
72
+ W->>C: start experiment
73
+ C->>A: launch candidate
74
+ W->>C: healthProbe
75
+ C-->>W: version 2.0.0, pid, startId
76
+ Note over W: journal: readback ok → promote intent
77
+ W->>W: experiment becomes stable
78
+ W->>C: resume
79
+ Note over W: persist promoted operation receipt
80
+ W-->>S: {result: promoted}, exit 0
81
+ S-->>B: exit 0
82
+ ```
83
+
84
+ In words:
85
+
86
+ 1. The bootstrap asks the supervisor for one operation: an id and a target
87
+ version. The id is what everything else binds to.
88
+ 2. The supervisor downloads the runner, checks its hash and size, writes it
89
+ to scratch space and runs it with the request on stdin.
90
+ 3. The worker takes the installation lock. If an earlier operation was left
91
+ unfinished, it settles that first. The same id replays its result; a new
92
+ id may proceed only after recovery succeeds.
93
+ 4. It asks your release source for exactly the target version, verifies the
94
+ bytes, and writes them into the *experiment* slot. Nothing running has
95
+ changed yet.
96
+ 5. It asks the controller to quiesce (park in-flight work) and to stop the
97
+ current service, and waits for confirmation that the old process is gone.
98
+ 6. It asks the controller to start the candidate from the experiment slot,
99
+ then probes it. The probe must come from one live process and report the
100
+ expected version with a pid and a **startId**. K probed once before
101
+ stopping the old service and journaled that startId; if the same one
102
+ comes back now, the old process was never replaced and the upgrade rolls
103
+ back.
104
+ 7. Only now does the worker write **promote intent** to the journal. That
105
+ line is the point of no return: before it, the safe move is always to put
106
+ the old version back; after it, the safe move is always to finish the
107
+ promotion.
108
+ 8. Experiment becomes stable. The controller resumes parked work. The
109
+ receipt is persisted in `operation.json`; it is archived before a later
110
+ operation begins. The worker prints its response and exits 0.
111
+
112
+ The transaction journal phases are, in order: `idle`,
113
+ `staged`, `handing-over`, `running-experiment`, `readback`, and then either
114
+ `promoted` or `rolled-back`. Operation receipts additionally track stages such as
115
+ `downloading` and policy outcomes such as `held`.
116
+
117
+ ## When something goes wrong
118
+
119
+ ### The candidate is bad
120
+
121
+ It does not start, it reports the wrong version, or it never passes the probe.
122
+ The worker stops it, starts stable again, resumes parked work, and records
123
+ `rolled-back`. Exit code 1. The stable bytes were never touched. This is the
124
+ routine failure and the one K is built around.
125
+
126
+ ### The worker dies
127
+
128
+ A crash, a kill, or a budget timeout. The supervisor:
129
+
130
+ 1. terminates a timed-out worker and waits for its observed exit; a deadline
131
+ alone does not permit takeover;
132
+ 2. starts a recovery worker bound to the same operation id;
133
+ 3. that worker takes the transaction lock, checks the operation identity,
134
+ and asks the controller to **fence** before replaying lifecycle effects.
135
+ Fencing ensures actions queued by the earlier worker cannot land later.
136
+ An already terminal original operation simply replays its receipt.
137
+
138
+ Recovery does not need the network and does not guess. It reads the journal:
139
+
140
+ ```mermaid
141
+ flowchart LR
142
+ idle --> staged --> ho[handing-over] --> re[running-experiment] --> rb[readback]
143
+ rb -->|promote intent| promoted
144
+ rb -.-> rolledback[rolled-back]
145
+ subgraph before [before promote intent: restore stable]
146
+ staged
147
+ ho
148
+ re
149
+ rb
150
+ end
151
+ subgraph after [after promote intent: replay commit]
152
+ promoted
153
+ end
154
+ ```
155
+
156
+ Before promote intent, recovery restores stable. After it, recovery replays
157
+ the commit. Both are idempotent, so a crash *during* recovery is handled the
158
+ same way next time. Recovery never starts a new upgrade, even if the original
159
+ request was an upgrade.
160
+
161
+ Attempts and elapsed time are bounded. If recovery does not settle within
162
+ those bounds, the supervisor exits 3, leaves the verified runner and a
163
+ `recovery.json` in scratch space, and prints where. Later, `resumeRunner` on
164
+ that file retries the same recovery offline.
165
+
166
+ ### Everything dies
167
+
168
+ Power loss, reboot, or the whole invocation killed. The state directory
169
+ survives. The next installer invocation, whether from `install.sh`, from
170
+ `self upgrade`, or from an operator, settles the unfinished operation before
171
+ it accepts new work. K does not install a permanent watchdog, so *something*
172
+ has to run the installer again: your product's OS startup hook, or a person.
173
+
174
+ ## Reading the result
175
+
176
+ Every run prints one JSON response on stdout and returns an exit code.
177
+ Inspect both. The short version:
178
+
179
+ | Exit | Meaning | What to do |
180
+ |---|---|---|
181
+ | 0 | Successful upgrade outcome; readable status or recovery with no recorded outcome | Check the action and receipt |
182
+ | 1 | Rolled back, or failed before any change | Read `operation.operation.outcome` and the reason |
183
+ | 2 | Held by policy, ownership or compatibility | Nothing changed; a new attempt needs a new id |
184
+ | 3 | Unresolved | Keep the recovery file and run recovery |
185
+
186
+ Two things surprise people. First, a `recover` action that successfully
187
+ restored stable after a bad candidate exits **1**, because the recorded
188
+ outcome is a rolled-back upgrade. That is the honest answer. Second, `status`
189
+ reads the last receipt, not the live service; it can say `genesis` (no
190
+ operation ever recorded) on a machine that is running fine.
191
+
192
+ The exact response shape and every exit code are in the
193
+ [reference](reference.md#protocol-v1).
194
+
195
+ What each step asks of your controller, and how to package and ship the
196
+ result, is the [integration guide](integration.md).
@@ -1,170 +1,75 @@
1
- # K 测试框架(harness)设计 v1
2
-
3
- 08-05:测试框架要**提前设计成完整的一块**,不是随层补测试。本文是 harness 的架构设计;`test-plan.md` 是跑在它上面的计划。
4
-
5
- ## 0. 定位:harness = 框架的可执行规格(executable spec)
6
-
7
- 顺序反转:**先有 harness,后有功能层**。每个功能层落地的定义 = "它让 harness 里预先写好的那组齿从 RED 变 GREEN"。测试不是功能的附件,是功能的规格。三个推论:
8
- 1. M0(harness 自举)先于一切层实现;
9
- 2. **不允许 harness 外的 ad-hoc 测试**——新齿必须进 registry(否则齿的 must-red/分档/自验纪律管不到它);
10
- 3. harness 只吃 core 的公共 API + HostAdapter ⇒ 它同时是 **API 的第一个消费者**(dogfood:API 不好用,harness 先痛)。
11
-
12
- ## 1. 组件架构
13
-
14
- ```
15
- harness/
16
- ├─ fake-host/ 假宿主(两种形态)
17
- │ ├─ inproc.ts 进程内 HostAdapter 实现(快速单元级)
18
- │ └─ daemon.ts 可 spawn 的真进程假 daemon(kill -9 是真的)
19
- ├─ fake-server/ 本地静态发布服务器 + 篡改 API
20
- ├─ artifact-factory/ 版本工件工厂(一次构建多次盖戳 + behavior 旋钮)
21
- ├─ scenario/ 场景运行器(隔离沙箱 + 虚拟时钟)
22
- ├─ crash/ 崩溃注入编排器(枚举生成,禁手列)
23
- ├─ teeth/ 齿注册表 + 分档 + 自验
24
- └─ cli.ts `k-harness` 入口(含 --adapter 接入方模式)
25
- ```
26
-
27
- ### 1.1 fake-host(假宿主)
28
- - **两形态**:`inproc`(进程内实现,跑快速逻辑齿)+ `process`(编译成真二进制、真 spawn、真 PID/startId —— kill -9、双跑检测、probe 活性都必须在真进程上验,mock 验不了崩溃)。
29
- - **故障注入开关**(per 方法):`fail-on-quiesce / hang-on-stop / wrong-version-probe / stale-startId-probe / crash-during-start ...`——每颗齿测"故障被抓",开关关掉齿必须转绿(证明齿测的是故障不是常态)。
30
- - **虚拟负载账本**:假宿主维护一个确定性"会话状态"文件(计数器+校验和);`quiesce↔resume` 等价断言 = 账本逐字节比对(**含 rolled-back resume**)。这是"会话保留"的可机械判定形态。
31
-
32
- ### 1.2 fake-server(假发布端)
33
- - 本地静态文件服务 + manifest 构造器(含 Range 续传——不认 Range 的桩会让"续传"悄悄退化成普通下载)。
34
- - **篡改 API**:`corruptByte(file, offset) / swapFiles / serveOlderVersion / dropFile` —— 完整性齿全部走"真篡改→真拒绝",不 mock 校验函数。判据是 sha256:K 验完整性不验来源(design-v1 §L0.5),所以篡改的判据也只能是"服务的字节还对不对得上 manifest 的摘要"。
35
-
36
- ### 1.3 scenario(场景运行器)
37
- - **一场景一沙箱**:独立 temp stateDir + 独立 fake-server 端口 全部并行安全、可重复。
38
- - **虚拟时钟注入**:core 的超时/重试全走注入 clock(框架级 clock seam —— 我们 web 侧 clock-ratchet 的同款纪律),场景可快进;无真实 sleep。
39
- - 场景 = 声明式脚本(步骤 + 期望 outcome + 期望 journal 尾部),跑完输出结构化 receipt(给 CI 和人两用)。
40
-
41
- ### 1.4 crash(崩溃注入编排器,承重件)
42
- - **覆盖面由代码生成**:从 core 导入状态机迁移表,自动枚举 `迁移边 × kill 点`(每个动作的 journal-写前/写后至少两点)→ 生成场景矩阵。**手列 kill 点非法**。
43
- - **完备性齿**:core 新增一个 phase/迁移而矩阵没覆盖 harness 自身 RED(枚举器数量对账)。防"加了状态忘了测崩溃"。
44
- - 每格断言同一组不变式:重启后 = 恢复 stable 或完成迁移;**永不双跑**(真进程存活探测);**永不砖**(stable 可再启动);journal 可重放。
45
-
46
- ### 1.45 DST:确定性模拟测试(与崩溃矩阵互补)
47
-
48
- **实现状态:已落地。** 入口为 `k-harness sim`;实现位于
49
- `harness/src/sim/`,三颗注册齿为 `sim.seed-replay-identical`、
50
- `sim.smoke-invariants`、`sim.fault-surface-covered`。
51
-
52
- **两条腿**:崩溃矩阵 = **枚举穷举**(在它的粒度上完备,给保证);DST = **种子随机深探**(FoundationDB/TigerBeetle 手法,找枚举想不到的交错,给发现)。
53
-
54
- - **前提(对 txn 引擎的架构约束,写引擎前就定)**:L1 引擎 = **纯状态机 + Effects 接口**——journal append/fsync、槽操作、宿主调用、时钟全部经注入的 effects 层,引擎本体零直接 IO/时间/随机。这**不是**测试后门(§1.8 自洽):effects 层就是平台适配器的天然挂点(各平台 fsync/swap 本来就不同实现),是产品级抽象。
55
- - **模拟器**:SimEffects = 内存盘(可模拟 partial write / fsync 丢失 / 重排)+ VirtualClock + **种子 PRNG 故障调度器**(在任意 effect 点注入 crash/fail/delay,按种子决定)。跑 N 千个种子 × 每种子一条完整升级/回滚剧本 → 断言同一组不变式(永不双跑/永不砖/journal 可重放/谓词诚实)。
56
- - **可复现**:任何失败 = 一个种子号,`k-harness sim --seed X` 逐字节重放。**失败种子沉淀为枚举矩阵的新固定格**(发现→保证的转化管道)。
57
- - **跑法**:PR 门跑固定 smoke 种子集(快、确定);夜跑扩大随机种子量;语料库(历史失败种子)永久保留。
58
- - PR/本地:`k-harness sim`(固定、评审可见的 smoke seeds)。
59
- - 单种子重放:`k-harness sim --seed X --json`(同 seed receipt 逐字节相同)。
60
- - 扩量:`k-harness sim --start-seed X --seeds N`;nightly 默认 50,000 seeds。
61
- - 失败自动原子合并进 `.k-harness/sim-failures.json`,workflow 保存为 artifact;记录含 seed、失败原因、transcript SHA-256 与一条可直接运行的 replay command。
62
- - 范围诚实:DST 覆盖 txn/converge 的逻辑交错;真进程/真 OS 面(信号、真 fsync 语义)仍归崩溃矩阵与真机轮——**模拟不替代真进程层,两者叠加**。
63
-
64
- ### 1.5 teeth(齿注册表)
65
- 每颗齿是一条注册记录,声明即纪律(缺任一字段注册失败):
66
- ```ts
67
- registerTooth({
68
- id: "txn.no-dual-run",
69
- profiles: ["service", "hosted"], // 分档
70
- kind: "invariant", // { kind: "baseline", failureCondition: "..." }
71
- mustRed: [ // mutation 契约:≥1 条,且答得出"不被我抓还会被谁抓"
72
- { mutate: "skip journal fsync before handover", caughtOnlyBy: "this" },
73
- ],
74
- run: async (ctx) => { ... },
75
- });
76
- ```
77
- - **分档执行**:`--profile swap|daemon|managed` 选齿集;cli 档误挂 L2 齿 ⇒ 注册期报错(档界齿)。
78
- - **断言二分机械化**:`kind` 必填 invariant 或 baseline-带失效条件;CI 扫无标注断言(断言纪律的执行器)。
79
- - **mutation-runner 对接**:registry 导出齿清单 + must-red 表,Lincan 的 runner 直接消费(未变异 baseline 0 失败 / 每齿变异必红 / 全红也不发结论)。
80
-
81
- ### 1.6 自验(M0 出口,harness 的上岗证)
82
- harness 判定别人之前先判自己,三样本缺一不可:
83
- - **known-green**:正确实现走完整升级 → 必须全绿;
84
- - **known-red**:注入一个已知故障 → 对应齿必须红、且只红该红的;
85
- - **对抗样本**:**结构上能过齿的检查、但违反真实 oracle** 的假实现(例:probe 换 pid 不换 startId 报新版本;quiesce 把账本备份再恢复伪装等价)→ 必须被抓。对抗样本清单随齿长(每次真实逃逸事后加一条)。
86
- 自验不过 ⇒ harness 拒绝运行任何评审(exit 非零 + typed 原因)。
87
-
88
- ### 1.7 接入方模式(`k-harness --profile X --adapter path`)
89
- 同一套齿对**外部真 adapter** 跑合规子集(不跑需要故障开关的齿,跑契约齿:quiesce↔resume 等价、probe 活性、ownership 响应)。绿 = 接入方契约达标;这也是 examples 三 demo 的验收方式——**demo 和接入方走同一道门**。
90
-
91
- ## 1.75 两个测试平面:黑盒优先(08-05:"就像启动一个 CLI、跑它的命令")
92
-
93
- harness 有两个平面,**默认用外面那个**:
94
-
95
- - **黑盒平面(主平面)**:spawn **真实打包好的二进制**,只通过它的命令行驱动(`mytool self upgrade` / `mytool status`),从外面断言:exit code、输出、盘上文件、进程状态、下次运行的版本。**就是用户的用法** —— 它顺带真正验证了"每个入口构造同一 Upgrader"这类 claim(library 平面验不了打包/入口接线)。examples 三 demo 都是真 CLI,端到端齿全在这层写。
96
- - **library 平面(辅助)**:import core API 直驱 Upgrader——只留给黑盒够不着的内部齿(如 journal 重放细节)。
97
-
98
- **规则:能在黑盒层表达的齿必须写在黑盒层**;library 层是例外、要说明为什么外面够不着。(同我们 symptom-layer 教义:用户层的红是最不可伪造的 oracle。)
99
-
100
- 接入方黑盒模式随之而来:`k-harness --profile swap --bin ./mytool` —— **零代码集成**:给你的真二进制,harness 起 fake-server、跑你的升级命令、断言下次运行版本/回滚/held。比 `--adapter` 还轻(cli 档接入方连 adapter 都不用给)。
101
-
102
- ## 1.76 黑盒 CLI 契约(08-05:"需要定义查版本子命令、状态 predicate 之类")
103
-
104
- 黑盒平面要从外面问二进制三类问题,因此有一个**小的 CLI 契约** —— 但它是"声明"不是"每家自己发明":
105
-
106
- **① core 白送命令实现**(app 只做一行接线):
107
- - `versionCommand()` → 打印二进制自身版本(cli 档够用);
108
- - `statusCommand()` → **问活进程**(走宿主 socket,同 same-PID 纪律)输出机读 JSON:`{ProcessEvidence, TxnState, ConvergenceReport}` —— 就是 core 已有的三个类型,不另造 schema;
109
- - `selfUpgradeCommand()` → 包装 `upgrader.upgrade()`,outcome 四态按结构化输出。
110
-
111
- **② app 声明命令名**(K 不规定你的 CLI 长相,但声明是**必须的**——harness 不猜命令,缺声明立即 typed FAIL):二进制旁边放一个 `k.target.ts`(或 `--target <path>`),default export typed `BlackBoxTarget`,命令名全部显式声明:
112
- ```ts
113
- import type { BlackBoxTarget } from "@k-carrier/harness";
114
-
115
- export default {
116
- version: ["--version"],
117
- status: ["k-status", "--json"], // 可选
118
- selfUpgrade: ["self", "upgrade"],
119
- env: { K_RELEASE_BASE: "..." }, // 可选
120
- } satisfies BlackBoxTarget;
121
- ```
122
- `k-harness --bin ./mytool` 动态 import 它来驱动(node 原生跑 TS,不用 build)。**没有 `k.target.ts` = 立即 typed FAIL(`BLACKBOX_TARGET_REQUIRED`),不探测、不猜**——猜对了省一行配置,猜错了给出的是一个可信的错误结论(不确定时要求显式声明,不替用户发明规矩)。
123
-
124
- **③ 与透明性原则自洽**:这些不是测试后门,是**产品本来就该有的面**(用户和 support 一样需要 `status --json`)——harness 只是恰好消费它们。cli 档最小契约 = `version` + `selfUpgrade` 两条;daemon/managed 档 + `status`(活进程 JSON)。
125
-
126
- **齿**:契约自身可验 —— `k.target.ts` 声明的命令跑不通 / status 输出不合 schema ⇒ 黑盒验收直接 FAIL(typed,不进齿评审);缺 target 文件 ⇒ 必 FAIL 且信息可操作(齿 `blackbox.missing-target-fails`)。
127
-
128
- ## 1.77 版本工件工厂 + 清场(08-05:"准备相应版本的二进制?删除清空?")
129
-
130
- **① artifact-factory(升级测试需要"同一个 app 的 vX 和 vY")**:
131
- - `makeRelease({version, behavior})` → 产出**盖了版本戳的真二进制** + manifest,落到 fake-server。实现 = **一次构建、多次盖戳**(构建 demo 源码一次,post-build 往二进制里注入版本串——与真 SEA 嵌版本同型,快且真实;不用"版本写在旁边文件"那种假形态)。
132
- - `behavior` 旋钮让某个"新版本"**故意坏**:`crash-on-start / wrong-probe / hang-on-quiesce ...` —— 回滚齿、known-red、对抗样本的 fixture 都从这来("升到坏版本→自动回滚→stable 完好"整条链可黑盒复现)。
133
- - 内容寻址缓存(key = demo 源 hash × version × behavior),跨场景复用,不重复构建。
134
-
135
- **② 清场(teardown)**:
136
- - 沙箱边界即清场边界:install dir + stateDir + fake-server 存储全在场景沙箱内。teardown = **杀进程树并确认真死**(按沙箱标记 pgrep 复核零残留——僵尸 `__service` 是我们的真实产线教训,"发了 kill"≠"死了")→ 删沙箱目录。崩溃场景故意留下的中间态也被同一动作清干净(一切都在沙箱里,所以 rm 恒有效)。
137
- - **越界写齿**:core 在任何场景中写沙箱外任何路径 ⇒ RED。这颗齿顺带保证了产品级卫生(升级器不污染全局 HOME/系统目录),也让"清空"永远可信——**能一键删干净,是因为先机械保证了它只写在自己地盘**。
138
- - 顺带的产品映射:沙箱清单 = 将来"干净卸载"要删的东西的权威地图(卸载功能本身另立,不在本期)。
139
-
140
- ## 1.8 透明性原则(08-05:测试框架对升级框架透明)
141
-
142
- **core 对 harness 零感知,机械强制**:
143
- - **禁 test-conditional**:core 内不得存在 "if under test" 任何形态(环境变量开关/测试模式 flag/AllowUnsigned 之类后门)。CI ratchet 扫 core 源码(同我们 clock-ratchet 手法),出现即红。
144
- - harness 需要的一切必须走**产品级注入面**——这些面是产品本来就需要的,不是为测试开的:
145
- - `HostAdapter`:产品 API 本体,fake-host 只是又一个 adapter;
146
- - `releaseBase`:指向 localhost 是配置,不是测试感知;
147
- - `clock`:时钟 seam 是正当的生产抽象(默认真时钟),不是测试后门;
148
- - `stateDir`:本就按 app 配置。
149
- - 崩溃注入 = 对真进程 kill -9,零 core 配合;故障注入全在 fake-host(harness 侧代码);对抗样本 = 假 adapter——全部外部。
150
- - **反向信号**:若某颗齿写不出来、除非给 core 开后门 ⇒ 判定为**公共 API 不足**(dogfood 信号),修 API 而不是开门。透明性由此与 forcing-function 同构:测试框架也只能是 core 的一个普通消费者。
151
-
152
- ## 2. 关键设计决定(为什么这样)
153
- 1. **真进程优先**:崩溃/双跑/probe 活性只在真 spawn 的假 daemon 上验——mock 崩溃 = 没测崩溃。inproc 只服务快速逻辑齿。
154
- 2. **枚举生成覆盖面**:kill 矩阵、齿-档对账、断言标注扫描全由代码生成/校验,**人列清单在这三处非法**(人会漏,且漏的方向总是"看起来覆盖够了")。
155
- 3. **虚拟时钟 + 沙箱**:决定论优先;flaky 即 bug。
156
- 4. **齿注册表是唯一入口**:declaration = 纪律载体(分档/must-red/二分标注都在注册时强制),绕开注册表的测试 CI 拒收。
157
- 5. **对抗样本制度化**:自验含"骗过检查但违反 oracle"的样本,且逃逸事后必须沉淀为新对抗样本——今天 uninstall 设计 那套 BLIND 对抗采样的教训直接机械化。
158
-
159
- ## 3. 实现顺序(M0 内部)
160
- 1. teeth 注册表 + 分档执行器 + 二分标注检查(纯逻辑,先立规矩);
161
- 2. scenario 沙箱 + 虚拟时钟;
162
- 3. fake-server(静态服务 + 篡改 API);
163
- 4. fake-host inproc → fake-host daemon(真进程);
164
- 5. crash 枚举器(吃 core 状态机表——此时 core 只需 `txn/state.ts` 的类型,已存在);
165
- 6. 自验三样本 → **M0 出口**。
166
- 此后每个功能层(M1–M6)的落地 = 先在 registry 写该层的齿(RED)→ 实现层 → 齿转 GREEN。
167
-
168
- ## 4. 边界
169
- - harness 不测 UI/产品语义(壳仓库自己的事);只测 core 契约 + 接入方 adapter 合规。
170
- - 真机/平台矩阵(mac launchd、Windows 服务)走 Testbed 轮,harness 出可移植齿、Testbed 供真床。
1
+ # Harness design
2
+
3
+ The harness tests K's transaction mechanisms. The external runner tests exercise
4
+ application integration. Neither replaces acceptance tests for a product's actual
5
+ service manager, installer packaging and data.
6
+
7
+ ## Test layers
8
+
9
+ | Layer | Implementation | Evidence |
10
+ |---|---|---|
11
+ | Mechanisms | Core unit tests, harness checks, injected effects | Locking, journal ordering, slots, policy, rollback and convergence |
12
+ | Execution boundary | Protocol, launcher, runner and command-controller tests | Input rejection, verified runner execution, bounded calls, receipt and exit-code binding |
13
+ | Real processes | `core/src/runner/process.test.ts` | A separate runner upgrades an application with no K dependency; another worker recovers after a crash |
14
+ | Product acceptance | Product repository and target machines | Real service isolation, packaging, workload restoration and data compatibility |
15
+
16
+ `pnpm test` runs the first three layers; `pnpm test:runner` selects the runner
17
+ boundary and integration tests. Direct Node tests are not all registered harness
18
+ checks, so `--list` is not the complete test inventory.
19
+
20
+ ## Components
21
+
22
+ All paths below are under `harness/src/`.
23
+
24
+ | Directory | Role |
25
+ |---|---|
26
+ | `fake-host/` | In-process hosts and spawned service processes with controllable faults |
27
+ | `fake-server/` | Local artifact delivery, range requests and corrupted responses |
28
+ | `artifact-factory/` | Runnable, version-stamped fixture artifacts with selectable failures |
29
+ | `fixtures/` | Internal byte-replacement, service and workload-ledger fixtures |
30
+ | `scenario/` | Isolated sandboxes and virtual time |
31
+ | `crash/` | Transition-derived fault matrix and recovery assertions |
32
+ | `sim/` | Seeded fault scheduling, invariant checks and replayable failure records |
33
+ | `teeth/` | Named checks, selection and known-green/known-red tests |
34
+
35
+ Core receives host, source, clock and effects through normal interfaces. It must
36
+ not detect tests or gain test-only bypasses. Logical tests use virtual time;
37
+ process/network tests use bounded waits and clean up their resources.
38
+
39
+ ## Registered checks
40
+
41
+ A registered check (called a *tooth* in the code) declares its id, tested layers,
42
+ profile, invariant or baseline failure condition, and a mutation that must fail.
43
+ The registry rejects invalid declarations and duplicate ids. Known-green cases
44
+ establish that correct behavior passes; known-red and adversarial cases establish
45
+ that the check catches its intended failure. A declared mutation is not evidence
46
+ that a separate mutation campaign has run.
47
+
48
+ The harness profiles `swap` and `service` select mechanism checks. Internal
49
+ fixtures invoke the engine directly; product examples use the external runner.
50
+
51
+ `--adapter` runs the subset supported by a harness adapter's declared contract;
52
+ it expects the harness driver interface, not an arbitrary product HostAdapter.
53
+ `--bin` drives a fixture or binary through explicitly declared commands in
54
+ `k.target.ts` (or `--target`). Missing command declarations fail; the harness does
55
+ not infer CLI names. See `node harness/src/cli.ts --help` for current options.
56
+
57
+ ## Fault coverage and limits
58
+
59
+ The crash enumerator crosses the transaction transition table with three points:
60
+ before journal write, after journal write and after the action. The matrix uses
61
+ injected effects and simulated crashes. Directed real-process tests separately
62
+ kill a worker between stop and start and verify recovery. The generated matrix
63
+ is not a claim that every point was tested with OS kills or physical power loss.
64
+
65
+ Seeded simulation explores additional effect interleavings. Its receipts preserve
66
+ the seed and transcript hash; failing seeds are saved for replay. Regressions
67
+ should become fixed cases. Simulation does not establish filesystem durability,
68
+ service-manager isolation or platform-specific executable replacement.
69
+
70
+ Recovery uses the same contract at every layer: before durable promote intent,
71
+ restore stable; after it, replay commit. Product controllers must also isolate
72
+ unfinished asynchronous effects before retrying a timed-out operation.
73
+
74
+ See the [test plan](test-plan.md) for commands and acceptance criteria, and the
75
+ [design](design.md) for the transaction and wire contracts.