owenloop 0.5.1 → 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (88) hide show
  1. package/CHANGELOG.md +30 -0
  2. package/README.md +9 -2
  3. package/dist/packages/work/src/agent/brief.d.ts +21 -0
  4. package/dist/packages/work/src/agent/brief.js +42 -1
  5. package/dist/packages/work/src/agent/capability-model.d.ts +123 -0
  6. package/dist/packages/work/src/agent/capability-model.js +148 -0
  7. package/dist/packages/work/src/agent/loop.d.ts +47 -4
  8. package/dist/packages/work/src/agent/loop.js +83 -53
  9. package/dist/packages/work/src/exec/loop.d.ts +17 -0
  10. package/dist/packages/work/src/exec/loop.js +195 -34
  11. package/dist/packages/work/src/exec/runner.js +15 -5
  12. package/dist/packages/work/src/harness/claude.js +69 -9
  13. package/dist/packages/work/src/harness/codex.js +56 -8
  14. package/dist/packages/work/src/harness/contract.d.ts +65 -1
  15. package/dist/packages/work/src/harness/contract.js +56 -0
  16. package/dist/packages/work/src/harness/session-store.d.ts +3 -1
  17. package/dist/packages/work/src/harness/session-store.js +46 -22
  18. package/dist/packages/work/src/hold/mcp.js +9 -2
  19. package/dist/packages/work/src/hub/client.d.ts +7 -1
  20. package/dist/packages/work/src/hub/client.js +1 -0
  21. package/dist/packages/work/src/hub/types.d.ts +71 -3
  22. package/dist/packages/work/src/roles/agent-run.js +8 -9
  23. package/dist/packages/work/src/roles/hold.d.ts +2 -0
  24. package/dist/packages/work/src/roles/hold.js +39 -8
  25. package/dist/packages/work/src/roles/settings.js +18 -6
  26. package/dist/packages/work/src/roles/shift.js +13 -1
  27. package/dist/packages/work/src/settings/settings.d.ts +60 -7
  28. package/dist/packages/work/src/settings/settings.js +69 -22
  29. package/dist/packages/work/src/shift/logretention.d.ts +253 -0
  30. package/dist/packages/work/src/shift/logretention.js +407 -0
  31. package/dist/packages/work/src/shift/logsink.d.ts +14 -0
  32. package/dist/packages/work/src/shift/logsink.js +98 -0
  33. package/dist/packages/work/src/shift/loop.d.ts +18 -1
  34. package/dist/packages/work/src/shift/loop.js +250 -12
  35. package/dist/packages/work/src/shift/protocol.d.ts +124 -1
  36. package/dist/packages/work/src/shift/protocol.js +14 -0
  37. package/dist/packages/work/src/shift/runtime.d.ts +14 -0
  38. package/dist/packages/work/src/shift/runtime.js +229 -6
  39. package/dist/packages/work/src/shift/server.d.ts +13 -0
  40. package/dist/packages/work/src/shift/server.js +61 -36
  41. package/dist/packages/work/src/shift/spawn.d.ts +78 -5
  42. package/dist/packages/work/src/shift/spawn.js +158 -12
  43. package/dist/packages/work/src/shift/state.d.ts +25 -0
  44. package/dist/packages/work/src/shift/state.js +29 -5
  45. package/dist/packages/work/src/shift/truncate.d.ts +35 -0
  46. package/dist/packages/work/src/shift/truncate.js +73 -0
  47. package/dist/packages/work/src/submit-proof.d.ts +31 -6
  48. package/dist/packages/work/src/submit-proof.js +33 -7
  49. package/dist/packages/work/src/submit-value.d.ts +48 -0
  50. package/dist/packages/work/src/submit-value.js +118 -0
  51. package/dist/packages/work/src/usage.d.ts +1 -1
  52. package/dist/packages/work/src/usage.js +11 -0
  53. package/dist/src/bundle/runtime.d.ts +1 -1
  54. package/dist/src/bundle/runtime.js +9 -0
  55. package/dist/src/capabilities.d.ts +93 -0
  56. package/dist/src/capabilities.js +101 -0
  57. package/dist/src/cli.js +37 -11
  58. package/dist/src/crypto/records.d.ts +3 -0
  59. package/dist/src/crypto/records.js +11 -0
  60. package/dist/src/defs.js +192 -4
  61. package/dist/src/engine.d.ts +112 -3
  62. package/dist/src/engine.js +224 -28
  63. package/dist/src/hub.d.ts +27 -11
  64. package/dist/src/hub.js +24 -18
  65. package/dist/src/index.d.ts +3 -1
  66. package/dist/src/index.js +4 -1
  67. package/dist/src/mcp/serve.js +11 -4
  68. package/dist/src/model.d.ts +36 -0
  69. package/dist/src/model.js +46 -0
  70. package/dist/src/schemas/index.d.ts +17 -0
  71. package/dist/src/schemas/order.v1.schema.json +17 -3
  72. package/dist/src/store.d.ts +33 -1
  73. package/dist/src/store.js +57 -4
  74. package/dist/src/types.d.ts +102 -2
  75. package/docs/agent-runner.md +26 -1
  76. package/docs/authoring.md +7 -0
  77. package/docs/bundles.md +103 -3
  78. package/docs/cli.md +127 -9
  79. package/docs/crypto.md +23 -7
  80. package/docs/shift-logs.md +642 -0
  81. package/docs/wire-contracts.md +34 -19
  82. package/package.json +1 -1
  83. package/plugins/claude-code/plugin/.claude-plugin/plugin.json +1 -1
  84. package/plugins/claude-code/plugin/.mcp.json +1 -1
  85. package/plugins/codex/plugins/owenloop/.codex-plugin/plugin.json +1 -1
  86. package/plugins/codex/plugins/owenloop/.mcp.json +1 -1
  87. package/dist/packages/work/src/agent/model-policy.d.ts +0 -49
  88. package/dist/packages/work/src/agent/model-policy.js +0 -138
@@ -0,0 +1,642 @@
1
+ # Shift logs
2
+
3
+ A shift writes two kinds of file into one directory:
4
+
5
+ | File | What it holds | Format |
6
+ | --- | --- | --- |
7
+ | `shift.log` | the shift process's own dispatch record | JSON Lines, one `ShiftEvent` per line |
8
+ | `<run>.log` | one worker process's raw stdout and stderr | opaque bytes, no format at all |
9
+
10
+ Both are **append-only** and both **outlive the shift process**. That is the
11
+ point of the feature: before it, a shift's dispatch record existed only in the
12
+ socket FIFO of a running daemon, and a dispatched worker's output was written to
13
+ `/dev/null`. If a shift died, why it died died with it.
14
+
15
+ This page is a **contract**, not a description. `owenloop-service` implements a
16
+ log uploader against it. Fields, filenames, and resolution order stated here are
17
+ stable; anything not stated here is not promised.
18
+
19
+ ## Three processes, three different things
20
+
21
+ Do not collapse these — the rest of this page depends on telling them apart.
22
+
23
+ 1. **The shift process** — `owenloop shift start` (or `owenloop work shift`).
24
+ Writes `shift.log`. One per machine, per state directory.
25
+ 2. **A worker process** — `owenloop work exec` or `owenloop work agent-run`,
26
+ spawned detached by the shift, one per dispatched order. Its stdout and
27
+ stderr are what `<run>.log` holds.
28
+ 3. **The command a worker runs** — the `command:` text of a command step,
29
+ executed by the worker under `/bin/sh -c`. **Its** streams are captured into
30
+ the receipt the worker submits to the hub. They are **not** in `<run>.log`.
31
+
32
+ So a `<run>.log` holds the worker's own diagnostics — "holding this order",
33
+ "running this step", the `workdir` fallback warning, a crash stack — and not the
34
+ command's output. One destination each, no duplication. `packages/work/test/
35
+ shift-logs-acceptance.test.ts` asserts both halves, the presence and the
36
+ absence.
37
+
38
+ ## Where the directory is
39
+
40
+ Resolution order, highest priority first:
41
+
42
+ 1. `--log-dir <path>` on `owenloop shift start` or `owenloop work shift`
43
+ 2. `OWENLOOP_SHIFT_LOG_DIR`
44
+ 3. `shiftLogDir` in settings
45
+ 4. **the resolved state directory** — the default
46
+
47
+ Defaulting to the state directory is what makes correlation free: `<run>.log`
48
+ lands beside the `<run>.json` in-flight record the shift already writes, with
49
+ the same basename.
50
+
51
+ **The correlation key is the BASENAME, not the adjacency.** `<run>.log` and
52
+ `<run>.json` are sanitized by the same function (`safeRun` in
53
+ `packages/work/src/shift/state.ts`), so pointing `--log-dir` at a different
54
+ filesystem moves the logs away from the records and correlation still holds.
55
+
56
+ The shift creates the directory at startup with `mkdirSync(…, {recursive: true,
57
+ mode: 0o700})`. If that fails it prints one line to its own stderr —
58
+
59
+ ```
60
+ owenloop shift: cannot create shift log directory <dir>: <reason> — continuing with logging disabled
61
+ ```
62
+
63
+ — and serves with no logging at all: no `shift.log`, and every worker launched
64
+ with `stdio: ['ignore','ignore','ignore']` exactly as before this feature
65
+ existed. **Losing observability never costs a dispatch.**
66
+
67
+ ## Permissions
68
+
69
+ Every file this feature creates is **owner-only**, and the mode is explicit at
70
+ every creation site rather than inherited from the umask:
71
+
72
+ | Path | Mode | Created by |
73
+ | --- | --- | --- |
74
+ | `<log-dir>/` | `0700` *when this feature creates it* — see below | `prepareShiftLogDir` (`logretention.ts`) |
75
+ | `<log-dir>/shift.log` | `0600` | `createShiftLogSink` (`logsink.ts`) |
76
+ | `<log-dir>/<run>.log` | `0600` | `createDefaultSpawner` (`spawn.ts`) |
77
+ | `<log-dir>/.owners/` | `0700` | `registerShiftLogOwner` (`logretention.ts`) |
78
+ | `<log-dir>/.owners/<hash>` | `0600` | `registerShiftLogOwner` (`logretention.ts`) |
79
+
80
+ **Why, specifically.** Before this feature a worker's stdout and stderr went to
81
+ `/dev/null`; this is the change that makes them persist. `<run>.log` is raw
82
+ output from authored workflow content, so a step that echoes a token puts that
83
+ token in the file, and `shift.log` quotes hub and workflow messages verbatim in
84
+ its `hub-error` and `order-dropped` records. That is the same data class as the
85
+ receipt artifact `packages/work/src/exec/loop.ts` already writes with
86
+ `mode: 0o600`, and the rule this repository states there — "the file is `0600`
87
+ because its content is agent-produced artifact data" — applies here unchanged.
88
+ Without an explicit mode these files would be `0666 & ~umask`, which is `0644`
89
+ under the usual `022`: readable by every local account on the machine.
90
+
91
+ **A mode applies only when the call CREATES the path.** Three consequences worth
92
+ knowing before you rely on it:
93
+
94
+ - **In the DEFAULT configuration the log directory is `0755`, not `0700`.** With
95
+ no `--log-dir`, the log directory *is* the state directory, and the state
96
+ directory is created earlier in the same startup by `ensureStateDir`
97
+ (`packages/work/src/shift/state.ts`) with `mkdirSync(stateDir, {recursive:
98
+ true})` and no `mode` — `0755` under the usual umask `022`.
99
+ `prepareShiftLogDir` then calls `mkdirSync(dir, {mode: 0o700})` on a directory
100
+ that already exists, which does not chmod it. So on a fresh machine running
101
+ the defaults, any local account can list the directory and read the run ids in
102
+ it. **The files inside are still `0600`**, so no log content is exposed — only
103
+ its filenames. `chmod 700` the state directory if the names matter to you.
104
+ - A `<run>.log`, a `shift.log`, or a `.owners/` directory left behind by a build
105
+ from before this was explicit **keeps its old, wider permissions**. Nothing
106
+ re-chmods an existing path. Fix those with one `chmod` if it matters to you;
107
+ we do not tighten a file an operator may already have handed to something
108
+ else.
109
+ - `mkdirSync` does not chmod a directory that already exists, and that is the
110
+ supported escape hatch for a **shared drop directory**: pre-create
111
+ `--log-dir` with the owner, group, and mode a separate uploader account
112
+ needs, and the shift leaves it exactly as it found it.
113
+
114
+ **The two log files stay `0600` either way.** A shared *directory* is an
115
+ operator's decision to make; it does not make the bytes inside less sensitive.
116
+ So an uploader running as a different account cannot read `shift.log` or
117
+ `<run>.log` out of the box. The intended deployment is an uploader running as
118
+ the **same user as the shift**. If you need a genuinely separate uploader
119
+ account, that is an explicit operator act today — a group-readable ACL on the
120
+ directory plus a deliberate `chmod` of the files, or a small privileged shipper
121
+ — and it is not something this feature arranges for you.
122
+
123
+ ## `shift.log` — the structured log
124
+
125
+ One JSON object per line, `\n`-terminated, UTF-8. Every line is independently
126
+ parseable; there is no header, no framing, and no trailer other than the
127
+ optional `ended` record described below.
128
+
129
+ ### The envelope, on every record
130
+
131
+ | Field | Type | Meaning |
132
+ | --- | --- | --- |
133
+ | `type` | string | the discriminator; see the table below |
134
+ | `ts` | string | ISO-8601 UTC with milliseconds, e.g. `2026-08-13T18:04:11.412Z` |
135
+ | `shift` | string | the shift's human name |
136
+ | `shiftId` | string | `shf_<uuid>` — the shift **process incarnation** |
137
+
138
+ On the wire and on disk those four fields sit flat alongside the record's own
139
+ fields, which is all a JSON consumer needs. A consumer that MIRRORS the source
140
+ types should know the split is drawn one field further in: `ShiftEventEnvelope`
141
+ in `packages/work/src/shift/protocol.ts` is `ts`/`shift`/`shiftId` only, and
142
+ `type` is the discriminator on each event body, not part of the envelope type.
143
+ Same bytes, different type decomposition.
144
+
145
+ Two properties an uploader can rely on:
146
+
147
+ - **`shiftId` is stable for the life of one process and unique across
148
+ processes.** Two shifts sharing a log directory interleave their records into
149
+ one `shift.log`; `shiftId` is how you separate them. It is `''` only for an
150
+ emitter that declared no shift id, which the shipped runtime never does.
151
+ (Sharing a log directory across shifts with separate state directories is
152
+ supported for retention too — see [The owner registry](#the-owner-registry--log-dirowners).)
153
+ - **`shift` is NOT stable.** `clock_in` renames a running shift, and each record
154
+ carries the name in force when it was written. Group by `shiftId`, display
155
+ `shift`.
156
+
157
+ ### Record types
158
+
159
+ | `type` | Fields beyond the envelope | Meaning |
160
+ | --- | --- | --- |
161
+ | `parked` | `origin`, `cap`, `serveCrews[]`, `hostname`, `cwd` | the shift started serving; see below |
162
+ | `dispatched` | `workflow`, `run`, `step`, `kind`, `pid` | a worker was spawned |
163
+ | `reaped` | `workflow`, `run`, `kind`, `pid` | a worker exited and was collected |
164
+ | `failed` | `workflow`, `run`, `step`, `kind`, `message`, and — from the exit path only — `executable`, `exitStatus`, `signal`, optional `harness` | a worker died or could not be spawned; **two producers, two shapes** — see below |
165
+ | `capacity` | `inFlight`, `cap` | no free slot, so the `whats_next` sweep was deferred — explains a shift running nothing new while work is outstanding. **Edge-triggered**, and emitted only on a *changed* wake — see below |
166
+ | `hub-error` | `op` (`wake`\|`whats_next`), `message`, optional `workflow` | a hub call failed. `workflow` present = the targeted `whats_next` for that one workflow; `workflow` absent with `op: 'whats_next'` = the untargeted inbox call, which aborts the whole sweep. **File-only** — see below |
167
+ | `bundle-miss` | `workflow`, `def` | a legacy order named a def with no cached bundle |
168
+ | `order-dropped` | `workflow`, `run`, `step`, `reason`, `message` | the shift refused one order and left it for hub pickup |
169
+ | `event-queue-overflow` | `dropped` | the socket FIFO evicted events; see below |
170
+ | `ended` | — | an operator ran `owenloop shift end` |
171
+ | `gate` | optional `workflow`, `run`, `name`, `question` | reserved; not emitted today |
172
+
173
+ `kind` is `'exec' | 'agent-run'`.
174
+
175
+ **Four of these are file-only.** `parked`, `capacity`, `hub-error`, and
176
+ `event-queue-overflow` are written to `shift.log` and are never delivered over
177
+ the daemon's Unix socket to `owenloop shift next`. Every other type goes to
178
+ both. The routing rule is `FILE_ONLY_EVENTS` in
179
+ `packages/work/src/shift/runtime.ts`, applied at the one point where an event
180
+ fans out to its two sinks.
181
+
182
+ The split is not importance. It is what each consumer is:
183
+
184
+ - A record about a **unit of work that moved** — `dispatched`, `reaped`,
185
+ `failed`, `order-dropped`, `bundle-miss`, `ended` — tells a socket client
186
+ something it cannot otherwise learn. Both sinks get it.
187
+ - A record about the **shift's own condition** is redundant or harmful on the
188
+ socket, and load-bearing in the file.
189
+
190
+ For `parked` and `capacity`, the socket-side objection is concrete: an idle
191
+ shift's `next` must **block** until there is work to report, and either record
192
+ sitting in the queue satisfies it instantly with news that nothing happened. A
193
+ `parked` record would make the first `next` after every start return
194
+ immediately; a `capacity` record would wake every attending terminal to say the
195
+ shift is full. Neither adds information — every `next` response already carries
196
+ live `cap`, `free`, and `running` in its own envelope, which is exactly what
197
+ `capacity` restates, and a socket client already knows which shift it reached
198
+ because it connected to that shift's socket (`op: 'status'` answers name,
199
+ crews, and cap on demand).
200
+
201
+ `hub-error` is file-only for the same reason plus a volume argument that makes
202
+ it a correctness matter rather than a tidiness one. **A failed hub call is not a
203
+ unit of work moving** — nothing was dispatched, reaped, or dropped; the shift
204
+ failed to ask — so it cannot claim the "a socket client cannot learn it
205
+ otherwise" justification, and it is subject to the blocking contract that
206
+ `parked` and `capacity` are subject to.
207
+
208
+ It is also the one record type that cannot self-limit. `hub-error` is
209
+ **level-triggered**: one record per failed call, for as long as the failure
210
+ lasts. The loop backs off only for HTTP 429 (`noteServerBackoff` requires a
211
+ `HubError` with `status === 429`), so an unreachable hub — `ECONNREFUSED`, DNS
212
+ failure, timeout, HTTP 500 — produces one record per poll tick with nothing
213
+ slowing it down: roughly 720 per hour per workflow at the 5s default. The socket
214
+ FIFO holds 1000 records and evicts the **oldest**, so about 83 minutes of outage
215
+ would evict every `dispatched`, `failed`, and `reaped` record a parked client
216
+ was waiting for, and every `owenloop shift next` during the outage would return
217
+ instantly carrying something that is not work.
218
+
219
+ The file keeps all of them. `shift.log` is append-only and unbounded, so an
220
+ operator can still count and time every attempt — which was the reason to record
221
+ each one, and it is a reason about the *file*, not about the wire.
222
+
223
+ `event-queue-overflow` has a second, independent reason: a record placed in the
224
+ queue that just overflowed would evict another event, overflow again, and
225
+ recurse under exactly the load that produced it. It is handed straight to the
226
+ log sink and never enters the fan-out at all.
227
+
228
+ Nothing is lost, because the file is the consumer with no envelope and no
229
+ context. `shift.log` has no per-response `cap`/`free`/`running`, so without
230
+ these four records a reader cannot tell an **idle** shift (no orders offered)
231
+ from a **saturated** one (orders offered, no slots) from a **stranded** one (hub
232
+ unreachable, so nothing was ever offered), nor tell a quiet log from a lossy
233
+ one.
234
+
235
+ `order-dropped.reason` is the stable machine discriminator and is one of
236
+ `malformed-digest`, `malformed-worker`, `unsupported-worker`,
237
+ `verification-failed`, `metadata-unavailable`. `message` is human text — match
238
+ on `reason`, display `message`.
239
+
240
+ ### The first record is self-describing
241
+
242
+ A shift's first record is `parked`, and it exists so a reader holding only the
243
+ file can resolve every later record. Later records identify the shift by `shift`
244
+ and `shiftId` alone; `parked` is what those names mean:
245
+
246
+ ```json
247
+ {"type":"parked","origin":"https://hub.example","cap":3,"serveCrews":["crew-a"],
248
+ "hostname":"box.local","cwd":"/srv/work","ts":"2026-08-13T18:04:11.412Z",
249
+ "shift":"box.local/work#a1b2c3","shiftId":"shf_…"}
250
+ ```
251
+
252
+ `--once` suppresses it, matching the console `parked as …` line: a one-shot
253
+ drain is not a parked shift.
254
+
255
+ **`serveCrews: []` means EVERY crew, not none.** `owenloop shift start --all`
256
+ maps to the empty array, so an empty list is the widest possible scope rather
257
+ than the narrowest (`packages/work/src/shift/loop.ts`, `setShift`/`getShift`). A
258
+ non-empty list is the crews named on the command line. Anything joining on this
259
+ field must special-case the empty array or it will report the busiest shifts on
260
+ a host as serving nothing.
261
+
262
+ The value is the one in force AT STARTUP. `clock_in` can change a running
263
+ shift's crew set, and `parked` is written once, so a long-lived shift's current
264
+ scope is not recoverable from this record alone.
265
+
266
+ ### The two `failed` shapes
267
+
268
+ `failed` has two producers and they do not emit the same fields. A consumer that
269
+ types the record off one of them rejects the other.
270
+
271
+ - **The exit path** — `reportWorkerFailure` in
272
+ `packages/work/src/shift/runtime.ts`, reached when a worker process was
273
+ spawned and then died. It ALWAYS includes `executable`, `exitStatus` and
274
+ `signal`. `exitStatus` and `signal` are commonly JSON `null` (a process that
275
+ exits on its own has no signal; one killed by a signal has no exit status), so
276
+ type them as nullable-and-present, not optional. `harness` is the one
277
+ genuinely optional field here — present only for an `agent-run` worker.
278
+ - **The spawn-threw path** — `emit()` in `packages/work/src/shift/loop.ts`,
279
+ reached when `spawn()` itself threw and no process ever existed. It carries
280
+ `workflow`, `run`, `step`, `kind` and `message` and OMITS `executable`,
281
+ `exitStatus`, `signal` and `harness` entirely — there was no process to
282
+ describe.
283
+
284
+ So `message` is the only failure detail present in both. Treat the four process
285
+ fields as "absent means the worker never started", not as "unknown".
286
+
287
+ One sentinel to know: on the exit path `step` is `"(unknown)"` — that literal
288
+ string, not `null` or an absent field — when the failure could not be attributed
289
+ to a named step. Anything indexing or joining on `step` must exclude it.
290
+
291
+ ### The last record, when there is one
292
+
293
+ `ended` is written **only** when an operator ran `owenloop shift end`. A signal
294
+ (`SIGINT`, `SIGTERM`), a loop failure, or a `kill -9` ends the process without
295
+ one.
296
+
297
+ So, for a consumer: a `shift.log` whose last record is `ended` describes a shift
298
+ that stopped on purpose. A `shift.log` with no trailing `ended` describes a
299
+ shift that is still running, or one that stopped some other way. Those are two
300
+ different questions and the file answers both; do not treat a missing `ended` as
301
+ corruption.
302
+
303
+ ### Line size
304
+
305
+ Each line is bounded at **512 KiB** (`MAX_RESPONSE_LINE_BYTES`) — the same
306
+ ceiling the Unix socket applies to a response, so the file never carries a
307
+ record the wire protocol would have refused, and an uploader can size its reads
308
+ from one number.
309
+
310
+ A record that would exceed it has its string fields shortened, largest first,
311
+ halving each pass, with the removed tail replaced by the literal marker
312
+ `[truncated]`. A field reduced to nothing becomes exactly `[truncated]`.
313
+
314
+ `type`, `ts`, and `shiftId` are **protected**: they are never shortened, because
315
+ a line must stay attributable and placeable in time even when its payload is
316
+ gone. `shift` is deliberately **not** protected — it is operator-supplied,
317
+ unbounded input, and the name is recoverable from any other record carrying the
318
+ same `shiftId`.
319
+
320
+ One consequence to code against: the 512 KiB bound applies to **string** fields
321
+ only. If some future record's non-string payload alone exceeded the ceiling, the
322
+ writer emits the oversized line rather than blank the identity that makes it
323
+ readable. No shipped record type can reach that case; an uploader should still
324
+ read lines with a bound rather than assume one.
325
+
326
+ ### Overflow records
327
+
328
+ The shift daemon's in-memory socket FIFO holds 1000 events for parked
329
+ `owenloop shift next` clients. When it overflows it evicts the oldest event and
330
+ counts the loss.
331
+
332
+ `event-queue-overflow` records that count **into `shift.log`**, never back into
333
+ the queue that just overflowed. `dropped` is **cumulative for the process**, so
334
+ the newest such record states the running total lost so far, not a delta.
335
+
336
+ Records are written at powers of ten — the 1st, 10th, 100th, 1000th eviction and
337
+ so on. That gives an immediate first record, an always-current total, and at
338
+ most a handful of records for any real run. It is a deliberate trade: reporting
339
+ every eviction would roughly double the volume of a file this feature does not
340
+ rotate, and reporting only the first would leave an unattended shift claiming
341
+ `dropped: 1` forever.
342
+
343
+ **Loss is bounded to the socket, not the file.** `shift.log` gets every record;
344
+ only a parked client's view can lose events.
345
+
346
+ ### Known limitation: `shift.log` is never rotated or reaped
347
+
348
+ It grows without bound for the life of a log directory, and the retention sweep
349
+ below explicitly refuses to touch it.
350
+
351
+ This is a decision, not an oversight. Rotation renames or truncates the file,
352
+ and byte-offset resumption (below) is defined against a file that only ever
353
+ grows. Making the file rotatable and making it resumable by offset are mutually
354
+ exclusive; resumability won because the uploader is the reason the file exists.
355
+ Bounding `shift.log` — by size cap, by rotation with an uploader-visible
356
+ sequence number, or by moving the whole directory per shift — is open.
357
+
358
+ Volume is low: a handful of records per dispatch. This is a slow leak, not a
359
+ fast one.
360
+
361
+ **One asymmetry to know about.** `capacity` is edge-triggered — one record when
362
+ the shift becomes full, not one per poll — precisely because repetition into a
363
+ never-rotated file actively harms it.
364
+
365
+ Read `capacity` as *at most* one record per at-capacity stretch, not exactly
366
+ one. Two things narrow it, both visible in `packages/work/src/shift/loop.ts`:
367
+
368
+ - **The hub IS still polled while a shift is full.** `opts.hub.wake(cursor)`
369
+ runs every tick regardless of local capacity. Only the follow-up `whats_next`
370
+ sweep is deferred. So a `capacity` record means "there was news and I could
371
+ not go get it", never "I stopped talking to the hub".
372
+ - **The emit sits inside the `changed && k <= 0` branch.** It fires only on a
373
+ wake that reports a *changed* cursor. A shift that fills up and stays full
374
+ while the hub reports nothing new produces **no `capacity` record at all**.
375
+ The absence of a record is therefore not evidence the shift had free slots.
376
+
377
+ An uploader should treat `capacity` as a hint about a moment, and derive
378
+ occupancy from `dispatched`/`reaped`/`failed` pairs, which are unconditional.
379
+
380
+ `hub-error` deliberately did **not** get
381
+ that treatment **in the file**: it is level-triggered, so a hub that is
382
+ unreachable for a day writes roughly one record per poll interval per workflow
383
+ (~17k/day at the 5s default). The reasoning is that a repeated `capacity` says
384
+ nothing new, whereas each `hub-error` is a distinct failed attempt an operator
385
+ may want to count and time. If that trade proves wrong in practice,
386
+ edge-triggering `hub-error` on `(op, message)` transitions is the obvious
387
+ change, and it belongs with the rotation decision above rather than ahead of it.
388
+
389
+ Keep the two questions apart, because they have different answers for the same
390
+ record type. **Frequency** is a property of the file: how many records one
391
+ outage writes into an append-only file that is never rotated — still open, as
392
+ above. **Routing** is a property of the wire: whether those records reach a
393
+ parked `owenloop shift next` at all — decided, and the answer is no.
394
+ `hub-error` is in `FILE_ONLY_EVENTS` precisely *because* it is level-triggered
395
+ into a socket queue that is bounded at 1000 and evicts the oldest. Neither
396
+ decision substitutes for the other.
397
+
398
+ ## `<run>.log` — one worker's raw output
399
+
400
+ `<log-dir>/<run>.log`, where `<run>` is the run id sanitized by `safeRun`.
401
+
402
+ **Opaque.** No structure is imposed and none should be assumed. It is whatever
403
+ bytes the worker wrote to fd 1 and fd 2, interleaved by the kernel in write
404
+ order.
405
+
406
+ **Untrusted.** A worker runs authored workflow content. Treat the contents as
407
+ attacker-influenceable data: never echo it into a shell, an HTML page, or a log
408
+ viewer that interprets escape sequences. Untrusted is a reason not to *repeat*
409
+ those bytes, never a reason to discard them.
410
+
411
+ **Owner-only on disk.** See [Permissions](#permissions) — the file is created
412
+ `0600`, so an uploader that runs as a different account cannot read it until an
413
+ operator decides otherwise.
414
+
415
+ **Unbounded in size, by design.** No cap and no truncation. A cap would remove
416
+ exactly the pathological run you needed to read. Volume is bounded by age
417
+ instead — see retention.
418
+
419
+ **Append, never truncate.** A retried or re-armed run reuses its run id, and its
420
+ `<run>.log` keeps the previous attempt's output above the new one. Reading a log
421
+ top to bottom gives you every attempt in order.
422
+
423
+ ### How the bytes get there
424
+
425
+ `createDefaultSpawner` (`packages/work/src/shift/spawn.ts`) opens the log once
426
+ with `openSync(path, 'a', 0o600)` and hands **that one descriptor** to stdio
427
+ slots 1 and 2 — the same redirection a shell writes as `2>&1`. The parent's copy
428
+ is closed in a `finally` immediately after `spawn` returns.
429
+
430
+ Three details that are load-bearing:
431
+
432
+ - **Open once, use twice.** Opening the path twice would create two independent
433
+ file offsets on one file and silently corrupt the output.
434
+ - **A file descriptor is not a pipe.** The invariant the spawner protects is
435
+ that no worker stdio slot is a *parent-owned pipe*: a detached worker that
436
+ outlives its shift would keep writing into a pipe whose reader has vanished. A
437
+ file descriptor survives the parent's exit, needs no reader, and cannot raise
438
+ `EPIPE`.
439
+ - **Close the parent's copy.** One leaked descriptor per dispatch reaches
440
+ `EMFILE` and stops dispatch entirely.
441
+
442
+ If the log cannot be opened, the shift prints one line —
443
+
444
+ ```
445
+ owenloop shift: could not open worker log <path>: <reason> — dispatching with its output discarded
446
+ ```
447
+
448
+ — and spawns the worker with its output discarded. The dispatch still happens.
449
+
450
+ That line is reported **once per shift**, not once per dispatch: every condition
451
+ that stops a log from opening — a full disk, a read-only directory, a log
452
+ directory deleted underneath a running shift — persists across the dispatches
453
+ that follow, so an unlatched report would write one stderr line per dispatch for
454
+ as long as the shift runs. The event sink latches its own failure report the
455
+ same way.
456
+
457
+ ## Retention
458
+
459
+ Runs **once, at shift startup**, before any dispatch. It removes **worker logs
460
+ only**.
461
+
462
+ A `<run>.log` is removed when **both** hold:
463
+
464
+ 1. no `<run>.json` in-flight record exists in **any state directory that writes
465
+ logs into this directory** (see *The owner registry* below), **and**
466
+ 2. the log's own mtime is at least `maxAge` old.
467
+
468
+ Condition 1 is not tidiness. A live worker holds an open descriptor on its log;
469
+ on POSIX, unlinking that file leaves the child writing into an orphaned inode —
470
+ bytes produced, charged against the filesystem, and unreadable. The in-flight
471
+ gate is what prevents that data loss.
472
+
473
+ ### The owner registry — `<log-dir>/.owners/`
474
+
475
+ `OWENLOOP_SHIFT_LOG_DIR` is a single global setting, so pointing several crews'
476
+ shifts at **one** log directory while each crew keeps its **own** state
477
+ directory is a supported and natural layout. It is also what makes condition 1
478
+ non-trivial: a sweeping shift can only see the `<run>.json` records in the state
479
+ directory it owns. Another shift's live `<run>.log` has no record the sweeper can
480
+ see, so a naive sweep would classify it as finished and unlink it — precisely
481
+ the orphaned-inode loss the gate exists to prevent. At the 14-day default that
482
+ needs a worker quiet for 14 days; at a reduced `--log-max-age` it would fire
483
+ against **every** live worker of every other shift.
484
+
485
+ So each shift, at startup and before it sweeps, writes one claim:
486
+
487
+ ```
488
+ <log-dir>/.owners/<sha256(stateDir)[0:16]>.json → {"stateDir":"/abs/path"}
489
+ ```
490
+
491
+ The sweep reads every claim and treats a run as in flight if **any** claimed
492
+ state directory still holds its `<run>.json`. Properties that matter:
493
+
494
+ - **One file per shift, named by a hash of the state directory it names.** Two
495
+ shifts starting simultaneously each write their own path, so no
496
+ read-modify-write can lose a claim.
497
+ - **A stale claim errs safe, but it is not free.** If the retired shift's state
498
+ directory no longer exists, or exists and holds no `<run>.json`, the claim
499
+ contributes no match and costs one `stat` per candidate log. That is the
500
+ common case and it is genuinely harmless.
501
+
502
+ The case to know about is a state directory that **survives with records in
503
+ it**. `<run>.json` is removed by `reconcileInFlight`, which only ever runs
504
+ against a state directory a live shift is attending
505
+ (`packages/work/src/shift/state.ts`). A retired crew's directory that no shift
506
+ attends again keeps its records indefinitely — and `isShiftLogReapable`
507
+ returns false whenever `hasRunRecord`, so every matching `<run>.log` becomes
508
+ permanently un-reapable. The cost is unbounded disk, not one `stat`.
509
+
510
+ The direction is deliberate: erring toward keeping a log costs disk, erring
511
+ the other way costs an operator the evidence they went looking for, and an
512
+ orphaned inode is unrecoverable while a wasted byte is not. **Operationally**:
513
+ deleting a retired shift's state directory outright also retires its claim's
514
+ effect, because a claim naming a directory that does not exist matches
515
+ nothing. Deleting the claim file alone is not enough if some *other* shift
516
+ also claims that directory.
517
+ - **A corrupt or unreadable claim costs only its claimant**, never the sweep,
518
+ and the sweeping shift's own state directory is always consulted regardless.
519
+ - If the claim cannot be written, the shift reports one line to its stderr and
520
+ keeps dispatching — its logs are then at risk from another shift sharing the
521
+ directory, which is the only reason that line exists.
522
+
523
+ `.owners/` is not a log. It is never swept, and an uploader should ignore it.
524
+
525
+ `maxAge` resolution, highest priority first:
526
+
527
+ 1. `--log-max-age <ms>`
528
+ 2. `OWENLOOP_SHIFT_LOG_MAX_AGE_MS`
529
+ 3. `shiftLogMaxAgeMs` in settings
530
+ 4. **14 days** — the default
531
+
532
+ `0` is a real choice, not "off": it means *reap every completed run's log at the
533
+ next shift startup*. There is no value that disables the sweep.
534
+
535
+ The environment variable is accepted only as a **non-negative integer number of
536
+ milliseconds**. An unparseable value, a negative one, and a non-integer one
537
+ (`86400000.5` is dropped, not rounded) are all ignored, and resolution falls
538
+ through to the next source — a typo in an env var must not stop a shift from
539
+ serving. The ignoring is silent, so a shift running on the default after a typo
540
+ looks exactly like a shift running on the default on purpose.
541
+
542
+ **Never reaped at any age:** `shift.log`, `.owners/`, anything not ending in
543
+ `.log`, and any `*.log` whose basename is not a `run_…` id. The log directory
544
+ defaults to the state directory, which also holds `shift.sock`,
545
+ `.dispatch.lock`, and atomic-write temporaries; a sweep that removed by age
546
+ alone would eat them.
547
+
548
+ The sweep is best-effort and never throws. A locked, unreadable, or
549
+ concurrently-removed file costs that one entry, not the sweep, and never the
550
+ shift.
551
+
552
+ ### Rolling back leaves the files behind
553
+
554
+ Downgrading to a build from before this feature is safe but not self-cleaning.
555
+ `shift.log`, every `<run>.log`, and `.owners/` stay in the log directory, and
556
+ under the default configuration that directory is the state directory the older
557
+ build still uses.
558
+
559
+ - **The old build ignores them.** `readStateRecords` reads only names ending in
560
+ `.json`, so reconciliation and `owenloop shift status` behave exactly as they
561
+ did — a `.log` file and the `.owners/` directory are invisible to them.
562
+ - **Nothing removes them.** The retention sweep is part of this feature, so the
563
+ older build has no reaper. The files persist at whatever size they had reached
564
+ until an operator deletes them by hand.
565
+
566
+ Delete `shift.log`, `run_*.log`, and `.owners/` from the log directory if the
567
+ disk matters; leave them if you intend to roll forward again, because a
568
+ re-upgraded shift resumes appending to the same `shift.log`.
569
+
570
+ ## Notes for an uploader
571
+
572
+ **Resume by byte offset.** Both file kinds are opened `O_APPEND` and only ever
573
+ grow while a shift is running. Record the offset you have consumed and resume
574
+ from it. This is why `shift.log` is not rotated.
575
+
576
+ **Interleaving is safe on a local filesystem.** Each `shift.log` record is one
577
+ complete `…\n` string written with a **single** `appendFileSync` call, and two
578
+ shifts sharing a log directory both append with `O_APPEND`. Lines stay
579
+ individually parseable and `shiftId` says which process wrote each one.
580
+
581
+ Be precise about where that comes from, because it bounds where it holds: the
582
+ guarantee is a property of an `O_APPEND` `write(2)` to a **regular file on a
583
+ local filesystem**, not of `appendFileSync` itself. `appendFileSync` loops on a
584
+ short write, and a second process can append between iterations of that loop. At
585
+ the sizes involved — one record, bounded at 512 KiB — a local filesystem does
586
+ not short-write, which is why it holds in practice. On NFS it is not promised.
587
+ Parse whole lines either way and carry a trailing fragment forward, as below.
588
+
589
+ **Durability over throughput.** `shift.log` holds no descriptor between writes:
590
+ every line is on disk before `write()` returns. A buffered stream would lose its
591
+ tail in exactly the crash the log exists to explain. There is consequently
592
+ nothing to flush and no `close()` to call.
593
+
594
+ **The writer never redacts; the uploader must.** The writer's job is to lose
595
+ nothing. Deciding what may leave the machine — and stripping it — belongs to
596
+ whatever ships the bytes off the machine. Assume `<run>.log` can contain
597
+ anything a workflow author's step printed.
598
+
599
+ **A record you cannot parse is a bug to report, not a line to drop silently.**
600
+ The writer bounds and escapes every line; a malformed one means something else
601
+ wrote to the file.
602
+
603
+ **A trailing line with no `\n` is not yet a record.** Split on `\n` and parse
604
+ only complete lines; carry the remainder into the next read. A live shift may be
605
+ mid-append when you read, and a shift that was killed can leave a final
606
+ fragment. This is the one parse failure that is expected rather than a bug —
607
+ which is why it is stated as "not yet a record" rather than as an exception to
608
+ the rule above.
609
+
610
+ **Run as the same user as the shift, or arrange access deliberately.**
611
+ `shift.log` and `<run>.log` are created `0600` — always, in every configuration
612
+ — see [Permissions](#permissions). An uploader under a different account gets
613
+ `EACCES` on the files until an operator widens something on purpose, and it gets
614
+ that even where the *directory* happens to be traversable, which under the
615
+ defaults it is.
616
+
617
+ ## Flags and settings
618
+
619
+ | Flag | Env | Setting | Default |
620
+ | --- | --- | --- | --- |
621
+ | `--log-dir <path>` | `OWENLOOP_SHIFT_LOG_DIR` | `shiftLogDir` | the state directory |
622
+ | `--log-max-age <ms>` | `OWENLOOP_SHIFT_LOG_MAX_AGE_MS` | `shiftLogMaxAgeMs` | `1209600000` (14 days) |
623
+
624
+ Both flags are accepted by `owenloop shift start` and `owenloop work shift`.
625
+
626
+ `owenloop work settings` prints `shiftLogDir = (unset)` and
627
+ `shiftLogMaxAgeMs = (unset)` when the settings file names neither. `(unset)`
628
+ there means *this file sets no value*, not *no default applies* — the defaults
629
+ in the table above are the ones in force. Only a few keys (`dispatchCap` and
630
+ friends) carry their built-in default into that listing.
631
+
632
+ ## Source map
633
+
634
+ | Concern | File |
635
+ | --- | --- |
636
+ | event shapes, the envelope, `stampShiftEvent` | `packages/work/src/shift/protocol.ts` |
637
+ | the `shift.log` sink | `packages/work/src/shift/logsink.ts` |
638
+ | paths, resolution, the reap gate, the sweep | `packages/work/src/shift/logretention.ts` |
639
+ | line-size bounding | `packages/work/src/shift/truncate.ts` |
640
+ | worker stdio wiring | `packages/work/src/shift/spawn.ts` |
641
+ | where the emitters are wired together | `packages/work/src/shift/runtime.ts` |
642
+ | the socket FIFO and its overflow | `packages/work/src/shift/server.ts` |