harnex 0.8.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +188 -0
- data/GUIDE.md +17 -10
- data/README.md +61 -54
- data/TECHNICAL.md +38 -15
- data/docs/codex-appserver.md +175 -0
- data/docs/configuration.md +118 -0
- data/docs/dispatch-telemetry.md +502 -0
- data/docs/events.md +132 -0
- data/guides/01_dispatch.md +31 -28
- data/guides/04_monitoring.md +10 -9
- data/guides/05_naming.md +18 -8
- data/lib/harnex/adapters/base.rb +10 -0
- data/lib/harnex/adapters/codex_appserver.rb +25 -0
- data/lib/harnex/artifact_report.rb +455 -6
- data/lib/harnex/cli.rb +3 -3
- data/lib/harnex/commands/artifact_report.rb +8 -7
- data/lib/harnex/commands/doctor.rb +38 -4
- data/lib/harnex/commands/history.rb +7 -3
- data/lib/harnex/commands/run.rb +55 -43
- data/lib/harnex/commands/status.rb +0 -2
- data/lib/harnex/commands/wait.rb +0 -1
- data/lib/harnex/config.rb +170 -0
- data/lib/harnex/core.rb +200 -21
- data/lib/harnex/dispatch_history.rb +16 -7
- data/lib/harnex/pricing.rb +135 -0
- data/lib/harnex/retention.rb +320 -0
- data/lib/harnex/runtime/session.rb +451 -166
- data/lib/harnex/terminal_status.rb +18 -21
- data/lib/harnex/version.rb +2 -2
- data/lib/harnex.rb +3 -0
- metadata +9 -2
|
@@ -0,0 +1,502 @@
|
|
|
1
|
+
# Dispatch Telemetry
|
|
2
|
+
|
|
3
|
+
`harnex run` captures predicted-vs-actual dispatch telemetry for a wrapped
|
|
4
|
+
agent session. Raw measurements remain on the v1 per-session events stream.
|
|
5
|
+
Durable summaries use one canonical v2 dispatch stream: every run appends one
|
|
6
|
+
`dispatch_start` row at registration and one rich `dispatch_end` row at
|
|
7
|
+
teardown.
|
|
8
|
+
|
|
9
|
+
Inside a git repo the stream is `<git-root>/.harnex/dispatch.jsonl`; outside a
|
|
10
|
+
git repo it is `~/.local/state/harnex/dispatch.jsonl`. `harnex history`,
|
|
11
|
+
`status --id`, and `wait` consume the same rows. Legacy v1 thin rows and
|
|
12
|
+
pre-v2 envelope-less summaries may coexist and remain readable/skippable.
|
|
13
|
+
|
|
14
|
+
## CLI flags
|
|
15
|
+
|
|
16
|
+
```text
|
|
17
|
+
harnex run codex --meta '{"model":"gpt-5.3-codex","effort":"high","predicted":{"input_tokens":[200000,800000]}}'
|
|
18
|
+
harnex run pi --context 'Implement the task; Harnex writes the final receipt'
|
|
19
|
+
harnex run pi --artifact-report .harnex/receipts/pi-r-64.json --context 'Optionally write review claims to $HARNEX_ARTIFACT_CLAIMS_PATH'
|
|
20
|
+
harnex run pi --project-id harnex --queue-id queue-005 --entry-id SP-4 --phase implement --intent queue-work --require-attribution
|
|
21
|
+
harnex run pi --orchestration-run-id queue-005 --orchestration-generation-id gen-1 --orchestration-role worker
|
|
22
|
+
harnex orchestration report --dispatch .harnex/dispatch.jsonl --run-id queue-005 --json
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
- `--meta JSON` must be a JSON object. The parsed object is echoed verbatim on
|
|
26
|
+
the `started.meta` event.
|
|
27
|
+
- Every run allocates a harness-owned `harnex.artifact_report.v1` receipt path.
|
|
28
|
+
The default is a repo-keyed file under `~/.local/state/harnex/receipts/`; live status
|
|
29
|
+
and the end row expose it as `artifact_report_path` / `artifact_report.path`.
|
|
30
|
+
- `--artifact-report PATH` overrides that output destination.
|
|
31
|
+
`--validation-report PATH` remains a compatibility alias. The worker receives
|
|
32
|
+
the final path as `HARNEX_ARTIFACT_REPORT_PATH` /
|
|
33
|
+
`HARNEX_VALIDATION_REPORT_PATH`, but Harnex owns and overwrites that file.
|
|
34
|
+
- `HARNEX_ARTIFACT_CLAIMS_PATH` is a separate optional input for bounded
|
|
35
|
+
`summary`, `verdict`, and `P1`/`P2`/`P3` counts. Claims are never acceptance
|
|
36
|
+
evidence. Stale, malformed, or oversized claims are ignored.
|
|
37
|
+
- `--require-artifact-report` is retained for script compatibility and exports
|
|
38
|
+
`HARNEX_ARTIFACT_REPORT_REQUIRED=1`, but it no longer needs an explicit path
|
|
39
|
+
or model-authored report. Receipt write failure is fail-closed for every run.
|
|
40
|
+
- `harnex artifact-report validate PATH --final` validates the generated
|
|
41
|
+
receipt without echoing its contents. `init` and the old final-validation
|
|
42
|
+
contract remain available for legacy/manual v1 documents.
|
|
43
|
+
- `--project-id`, `--queue-id`, `--entry-id`, `--entry-title`, `--phase`,
|
|
44
|
+
`--tier`, `--issue`, `--plan`, `--intent`, `--model`, `--effort`,
|
|
45
|
+
`--parent-dispatch-id`, `--parent-attempt-id`, and `--attempt-kind` are
|
|
46
|
+
first-class queue/agent telemetry flags. They are persisted as caller-provided
|
|
47
|
+
strings and override same-named `--meta` values. `--attempt-kind` is one of
|
|
48
|
+
`initial`, `retry`, `fix`, `review`, `fallback`, or `superseding`; linkage fields keep
|
|
49
|
+
independently-run follow-ups joinable without merging their raw usage.
|
|
50
|
+
- `--orchestration-run-id`, `--orchestration-generation-id`,
|
|
51
|
+
`--orchestration-role`, `--orchestration-session-id`, and
|
|
52
|
+
`--orchestration-rotation-reason` opt a dispatch row into logical
|
|
53
|
+
primary-orchestrator rollups. `--orchestration-role` is `primary` or
|
|
54
|
+
`worker`; Harnex-managed primaries should use `primary` instead of emitting a
|
|
55
|
+
duplicate external sample for the same usage row.
|
|
56
|
+
- `--require-attribution` fails before launch unless `project_id`, `phase`,
|
|
57
|
+
`intent`, and at least one of `queue_id` / `entry_id` / `issue` / `plan` are
|
|
58
|
+
present through first-class flags or `--meta`.
|
|
59
|
+
- The canonical stream is the only destination a dispatch writes telemetry to.
|
|
60
|
+
There is no second copy and no flag to request one; a flag asking for a
|
|
61
|
+
mirror file is rejected as unknown. See CHANGELOG for the removal note.
|
|
62
|
+
- The v2 `dispatch_end` combines the history envelope (`schema_version`,
|
|
63
|
+
`record_type`, id/status/timing fields) with all rich telemetry sections.
|
|
64
|
+
A default dispatch therefore adds exactly two rows, not separate thin and
|
|
65
|
+
rich end rows.
|
|
66
|
+
|
|
67
|
+
Use `harnex history --json | jq .` for pipelines over the repo-local log.
|
|
68
|
+
|
|
69
|
+
## Metadata and prediction contract
|
|
70
|
+
|
|
71
|
+
The v2 `dispatch_end` always has `meta`, `predicted`, `actual`, `agent`,
|
|
72
|
+
`usage`, `context`, `attribution`, `outcome`, `attempt`, `reliability`,
|
|
73
|
+
`artifact_report`, `receipt`, `observed`, and `validation` blocks. When queue
|
|
74
|
+
attribution fields are provided, harnex also adds a top-level `queue` block.
|
|
75
|
+
When orchestration fields are provided, harnex adds a top-level
|
|
76
|
+
`orchestration` block. A sanitized `claims` block is additive only when the
|
|
77
|
+
worker supplied one.
|
|
78
|
+
|
|
79
|
+
Harnex-owned `meta` fields are always populated when derivable: `id`,
|
|
80
|
+
`tmux_session`, `description`, `started_at`, `ended_at`, `harness`,
|
|
81
|
+
`harness_version`, `agent`, `agent_version`, `agent_provider`, `host`,
|
|
82
|
+
`platform`, `repo`, `branch`, `start_sha`, and `end_sha`.
|
|
83
|
+
|
|
84
|
+
These top-level `--meta` keys pass through into `meta` when provided:
|
|
85
|
+
`orchestrator`, `orchestrator_session`, `chain_id`, `parent_dispatch_id`,
|
|
86
|
+
`parent_attempt_id`, `attempt_kind`, `tier`, `phase`, `issue`, `plan`, and
|
|
87
|
+
`task_brief`. Queue-specific keys such as
|
|
88
|
+
`project_id`, `queue_id`, `entry_id`, `entry_title`, and `intent` are used for
|
|
89
|
+
the top-level `queue` block but are not duplicated into legacy `meta`. Unknown
|
|
90
|
+
top-level keys are kept on `started.meta` but are not copied into the v2 end
|
|
91
|
+
row.
|
|
92
|
+
|
|
93
|
+
`predicted` is copied verbatim from `--meta.predicted` when it is a JSON object;
|
|
94
|
+
otherwise it is `{}`. Harnex does no profile lookup or recommendation-table
|
|
95
|
+
resolution.
|
|
96
|
+
|
|
97
|
+
## Queue, agent, and reliability blocks
|
|
98
|
+
|
|
99
|
+
The top-level `queue` block is emitted only when at least one queue attribution
|
|
100
|
+
field is known. When present, it has a stable key set and preserves values as
|
|
101
|
+
strings:
|
|
102
|
+
|
|
103
|
+
```json
|
|
104
|
+
{
|
|
105
|
+
"queue": {
|
|
106
|
+
"project_id": "harnex",
|
|
107
|
+
"queue_id": "queue-005",
|
|
108
|
+
"entry_id": "SP-4",
|
|
109
|
+
"entry_title": "Implement sidecar ingestion",
|
|
110
|
+
"issue": "52",
|
|
111
|
+
"plan": "52",
|
|
112
|
+
"phase": "implement",
|
|
113
|
+
"tier": "B",
|
|
114
|
+
"intent": "queue-work"
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
The top-level `agent` block is always emitted and is the preferred home for
|
|
120
|
+
routing details; legacy `meta.agent*` and `actual.model` stay for compatibility:
|
|
121
|
+
|
|
122
|
+
```json
|
|
123
|
+
{
|
|
124
|
+
"agent": {
|
|
125
|
+
"cli": "codex",
|
|
126
|
+
"provider": "openai",
|
|
127
|
+
"model_requested": "gpt-5.3-codex",
|
|
128
|
+
"model_effective": "gpt-5.3-codex",
|
|
129
|
+
"reasoning_effort": "high",
|
|
130
|
+
"service_tier": "flex",
|
|
131
|
+
"adapter_transport": "stdio_jsonrpc"
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
The top-level `reliability` block is always emitted and should be preferred over
|
|
137
|
+
legacy `actual.disconnections` for reliability analytics:
|
|
138
|
+
|
|
139
|
+
```json
|
|
140
|
+
{
|
|
141
|
+
"reliability": {
|
|
142
|
+
"adapter_close": "normal",
|
|
143
|
+
"real_disconnections": 0,
|
|
144
|
+
"stream_interruptions": 0,
|
|
145
|
+
"stalls": 0,
|
|
146
|
+
"force_resumes": 0,
|
|
147
|
+
"compactions": 0,
|
|
148
|
+
"recovered": false
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
`adapter_close` is `normal` for ordinary process/adapter completion,
|
|
154
|
+
`interrupted` for timeout/signal termination, `lost` for boot failure or real
|
|
155
|
+
transport loss, and `unknown` when harnex cannot classify it. Successful
|
|
156
|
+
structured runs that close normally after task completion should report
|
|
157
|
+
`real_disconnections: 0` even if old consumers still read the legacy counter.
|
|
158
|
+
|
|
159
|
+
Example grouping for queue analysis:
|
|
160
|
+
|
|
161
|
+
```bash
|
|
162
|
+
jq -r 'select(.queue) | [.queue.project_id, .queue.queue_id, .queue.entry_id, .queue.phase, .agent.model_effective] | @tsv' .harnex/dispatch.jsonl
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
## Usage, context pressure, attribution, outcomes, and attempts
|
|
166
|
+
|
|
167
|
+
`usage` makes nullable legacy `actual` token and cost fields interpretable:
|
|
168
|
+
|
|
169
|
+
```json
|
|
170
|
+
{
|
|
171
|
+
"usage": {
|
|
172
|
+
"status": "observed",
|
|
173
|
+
"cost_usd": 1.42,
|
|
174
|
+
"cost_source": "price_table",
|
|
175
|
+
"cost_price_as_of": "2026-08-03",
|
|
176
|
+
"input_tokens": 120000,
|
|
177
|
+
"output_tokens": 8000,
|
|
178
|
+
"cached_input_tokens": 2000,
|
|
179
|
+
"reasoning_tokens": null,
|
|
180
|
+
"total_tokens": 130000
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
`usage.status` is `observed` for an adapter measurement, `zero` for an explicit
|
|
186
|
+
all-zero adapter measurement, `estimated` for caller-supplied
|
|
187
|
+
`--meta '{"usage":{"status":"estimated",...}}'` values, `unsupported` when
|
|
188
|
+
the adapter has no supported usage source, or `missing` when a supported source
|
|
189
|
+
provided no observation. `cost_source` is `provider_reported` for a reliable
|
|
190
|
+
adapter value, `price_table` when Harnex computes exact maintained
|
|
191
|
+
provider/model/service-tier/context-band list pricing, and `caller_estimate` for
|
|
192
|
+
a declared estimate. Price-table rows carry `cost_price_as_of`; unknown models,
|
|
193
|
+
service/context tiers, missing context evidence, or required token components
|
|
194
|
+
remain null. Provider-reported values are never
|
|
195
|
+
overwritten, and all cost telemetry is operational estimation rather than a
|
|
196
|
+
billing invoice.
|
|
197
|
+
|
|
198
|
+
`context` is separate from cumulative `usage`: it describes how full the active
|
|
199
|
+
model context became, not how many tokens all requests accumulated:
|
|
200
|
+
|
|
201
|
+
```json
|
|
202
|
+
{
|
|
203
|
+
"context": {
|
|
204
|
+
"status": "observed",
|
|
205
|
+
"source": "pi_get_session_stats",
|
|
206
|
+
"terminal_tokens": 64000,
|
|
207
|
+
"window_tokens": 200000,
|
|
208
|
+
"terminal_percent": 32.0,
|
|
209
|
+
"peak_tokens": 118000,
|
|
210
|
+
"peak_percent": 59.0,
|
|
211
|
+
"samples": 7,
|
|
212
|
+
"missing_samples": 1,
|
|
213
|
+
"latest_sample_status": "missing"
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
`terminal_*` is the final **valid** occupancy sample and `peak_*` is the
|
|
219
|
+
independent high-water mark across valid samples. `window_tokens` is the model
|
|
220
|
+
window paired with that terminal sample. `samples` counts bounded source
|
|
221
|
+
samples, including unavailable ones; `missing_samples` counts that unavailable
|
|
222
|
+
subset. Consequently, a null sample immediately after compaction leaves the
|
|
223
|
+
last valid terminal and peak values intact while setting
|
|
224
|
+
`latest_sample_status: "missing"`. Null never means zero.
|
|
225
|
+
|
|
226
|
+
`context.status` is `observed` for Pi's dedicated
|
|
227
|
+
`get_session_stats.contextUsage` signal, `estimated` for Codex app-server,
|
|
228
|
+
`missing` when a supported source yielded no valid occupancy, or `unsupported`
|
|
229
|
+
when the adapter has no active-context source. `source` is
|
|
230
|
+
`pi_get_session_stats` or `codex_thread_token_usage_last` for those structured
|
|
231
|
+
adapters and is null for unsupported adapters. Pi's percentage is adapter
|
|
232
|
+
reported. Codex's `tokenUsage.last.totalTokens` is the latest model-reported
|
|
233
|
+
active context size, but it excludes local items appended after that response;
|
|
234
|
+
Harnex therefore labels it estimated and derives
|
|
235
|
+
`terminal_percent = last.totalTokens / modelContextWindow * 100`. That is
|
|
236
|
+
full-window pressure, not Codex TUI's baseline-adjusted “context left” display.
|
|
237
|
+
No prompt, transcript, message, tool payload, or compaction summary is copied
|
|
238
|
+
into this block.
|
|
239
|
+
|
|
240
|
+
`attribution.status` is `complete` when `project_id`, `phase`, `intent`, and a
|
|
241
|
+
work id are present; `partial` when any attribution is known but that contract
|
|
242
|
+
is incomplete; otherwise `missing`. `outcome.status` is derived from the
|
|
243
|
+
harness receipt: `accepted`, `rejected`, `no_change`, or `unknown`. Optional
|
|
244
|
+
worker claims cannot set it. Its additive `class` records the work verdict
|
|
245
|
+
(`completed_with_proof`, `completed_no_activity`, `report_invalid`,
|
|
246
|
+
`task_failed`, plus legacy classes retained in old rows), and `report_status`
|
|
247
|
+
is normally `accepted` or `rejected`. `source=harnex_observed_state` identifies
|
|
248
|
+
new receipts. The block also contains final commit/path/LOC observations; those
|
|
249
|
+
facts prove the delta, not semantic quality or human authorship.
|
|
250
|
+
|
|
251
|
+
Every end row has one Harnex-session `attempt`. Its random `id` is distinct
|
|
252
|
+
from operator-visible `run_id`; `parent_attempt_id` and `parent_dispatch_id`
|
|
253
|
+
link retries/fixes/reviews/fallbacks while each row keeps separate raw usage.
|
|
254
|
+
At finalization Harnex walks the canonical stream's parent chain once to derive
|
|
255
|
+
`actual.attempts_total`, succeeded/failed counts, `fallback_triggered`, and
|
|
256
|
+
`reliability.recovered`; missing parents and malformed/cyclic history degrade
|
|
257
|
+
safely to the resolvable chain. `retry_count` remains the separate in-run event
|
|
258
|
+
counter. The events JSONL also carries `attempt_started`, `attempt_finished`,
|
|
259
|
+
and adapter-reported retry/fallback events.
|
|
260
|
+
|
|
261
|
+
## Orchestration tax rollups
|
|
262
|
+
|
|
263
|
+
`harnex orchestration` joins one logical primary-orchestrator run across
|
|
264
|
+
primary generations and child dispatches. It is opt-in and bounded: the sample
|
|
265
|
+
path stores counters and lifecycle labels only.
|
|
266
|
+
|
|
267
|
+
Harnex-managed primaries can be represented directly by their dispatch row:
|
|
268
|
+
|
|
269
|
+
```bash
|
|
270
|
+
harnex run pi --orchestration-run-id queue-005 \
|
|
271
|
+
--orchestration-generation-id primary-1 --orchestration-role primary ...
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
External interactive primaries can emit bounded samples through an integration
|
|
275
|
+
or shell command:
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
harnex orchestration sample --out .harnex/orchestrator.jsonl \
|
|
279
|
+
--run-id queue-005 --generation-id primary-1 --project-id harnex \
|
|
280
|
+
--queue-id queue-005 --session-id pi-primary-1 \
|
|
281
|
+
--context-status observed --context-tokens 64000 \
|
|
282
|
+
--context-window-tokens 200000 --context-percent 32 \
|
|
283
|
+
--usage-status observed --usage-input-tokens 120000 \
|
|
284
|
+
--usage-output-tokens 9000 --usage-total-tokens 129000 \
|
|
285
|
+
--tool-calls 31 --compactions 1
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
The sample schema is `harnex.orchestrator_sample.v1`. Valid sample events are
|
|
289
|
+
`sample`, `generation_started`, `generation_finished`, `rotation`, `recovery`,
|
|
290
|
+
and `compaction`. Samples must never include prompts, transcripts, hidden
|
|
291
|
+
reasoning, tool arguments/results, secrets, or private payloads.
|
|
292
|
+
|
|
293
|
+
Reports join dispatch rows whose `orchestration.run_id` matches the requested
|
|
294
|
+
run and optional external samples with the same `orchestration_run_id`:
|
|
295
|
+
|
|
296
|
+
```bash
|
|
297
|
+
harnex orchestration report --dispatch .harnex/dispatch.jsonl \
|
|
298
|
+
--samples .harnex/orchestrator.jsonl --run-id queue-005 --json
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
The report schema is `harnex.orchestration_tax.v1`. It includes primary usage
|
|
302
|
+
and context coverage, per-generation peaks and rotation reasons, worker usage,
|
|
303
|
+
accepted/rejected/blocked/unknown child outcomes deduplicated by work id,
|
|
304
|
+
primary usage/tool calls per accepted entry, and explicit `missing` /
|
|
305
|
+
`unsupported` statuses instead of treating absent telemetry as zero.
|
|
306
|
+
|
|
307
|
+
## Observed-state receipts and optional claims
|
|
308
|
+
|
|
309
|
+
Every dispatch receives a bounded receipt authored by Harnex, not by the
|
|
310
|
+
worker. The schema identifier remains `harnex.artifact_report.v1` so existing
|
|
311
|
+
`artifact-report validate --final` consumers keep one command and one result
|
|
312
|
+
contract. The additive `receipt.author=harnex` marker selects the observed-state
|
|
313
|
+
final contract.
|
|
314
|
+
|
|
315
|
+
A generated receipt contains:
|
|
316
|
+
|
|
317
|
+
```json
|
|
318
|
+
{
|
|
319
|
+
"schema": "harnex.artifact_report.v1",
|
|
320
|
+
"status": "pass",
|
|
321
|
+
"receipt": {
|
|
322
|
+
"version": 1,
|
|
323
|
+
"author": "harnex",
|
|
324
|
+
"generated_at": "2026-08-03T05:00:00Z",
|
|
325
|
+
"id": "cx-r-64",
|
|
326
|
+
"session_id": "8d8f3f07c8fc343d"
|
|
327
|
+
},
|
|
328
|
+
"outcome": {
|
|
329
|
+
"status": "no_change",
|
|
330
|
+
"summary": "Harnex observed successful completion with no Git delta."
|
|
331
|
+
},
|
|
332
|
+
"validation": {
|
|
333
|
+
"status": "pass",
|
|
334
|
+
"final_reported": true,
|
|
335
|
+
"commands": [
|
|
336
|
+
{ "cmd": "git diff --check", "exit_code": 0, "status": "completed" }
|
|
337
|
+
]
|
|
338
|
+
},
|
|
339
|
+
"observed": {
|
|
340
|
+
"git": {
|
|
341
|
+
"status": "observed",
|
|
342
|
+
"start_sha": "0123456789abcdef0123456789abcdef01234567",
|
|
343
|
+
"end_sha": "0123456789abcdef0123456789abcdef01234567",
|
|
344
|
+
"branch": "main",
|
|
345
|
+
"changed_paths": [],
|
|
346
|
+
"loc_added": 0,
|
|
347
|
+
"loc_removed": 0,
|
|
348
|
+
"files_changed": 0,
|
|
349
|
+
"commits": 0
|
|
350
|
+
},
|
|
351
|
+
"commands": [
|
|
352
|
+
{ "cmd": "git diff --check", "exit_code": 0, "status": "completed" }
|
|
353
|
+
],
|
|
354
|
+
"command_observation": "observed",
|
|
355
|
+
"turn": {
|
|
356
|
+
"status": "completed",
|
|
357
|
+
"outcome_class": "completed_with_proof",
|
|
358
|
+
"task_complete": true,
|
|
359
|
+
"task_failed": false,
|
|
360
|
+
"accepted": true,
|
|
361
|
+
"exit_code": 0
|
|
362
|
+
},
|
|
363
|
+
"usage": { "status": "observed", "input_tokens": 100, "output_tokens": 20, "total_tokens": 120 }
|
|
364
|
+
}
|
|
365
|
+
}
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
`observed.git` is sufficient for a queue to distinguish commit proof
|
|
369
|
+
(`start_sha != end_sha`, commit count/path/LOC evidence) from an observed
|
|
370
|
+
`no_change` result. `start_dirty`, `end_dirty`, and `worktree_changed` expose
|
|
371
|
+
worktree caveats while unchanged pre-session dirt is excluded from the delta.
|
|
372
|
+
Codex app-server `commandExecution` items contribute
|
|
373
|
+
bounded command text, integer exit code, status, and optional duration. Other
|
|
374
|
+
transports currently report `command_observation: "unsupported"` rather than
|
|
375
|
+
guessing from prose or generic tool events. If the 256-KiB receipt cap requires
|
|
376
|
+
trimming, `commands_truncated` / `changed_paths_truncated` make that explicit;
|
|
377
|
+
full aggregate counts remain in the dispatch row. Usage carries the same
|
|
378
|
+
measured/missing/unsupported semantics as the dispatch
|
|
379
|
+
row and is refreshed at teardown when final adapter usage becomes available.
|
|
380
|
+
|
|
381
|
+
`validation.commands` mirrors all bounded observed command exits for legacy
|
|
382
|
+
readers. Its aggregate status is `not_run`, `pass`, or `fail`; a failed
|
|
383
|
+
exploratory command may precede a successful turn. For harness receipts,
|
|
384
|
+
`validate --final` therefore validates receipt authorship/shape, accepted turn,
|
|
385
|
+
and accepted/no-change outcome instead of letting an intermediate command exit
|
|
386
|
+
rewrite the harness verdict. Queue policy may impose a stricter command gate by
|
|
387
|
+
inspecting `observed.commands`.
|
|
388
|
+
|
|
389
|
+
Reviewers can write one optional input file at
|
|
390
|
+
`HARNEX_ARTIFACT_CLAIMS_PATH`:
|
|
391
|
+
|
|
392
|
+
```json
|
|
393
|
+
{
|
|
394
|
+
"claims": {
|
|
395
|
+
"summary": "Review complete; one P2 remains.",
|
|
396
|
+
"verdict": "changes_requested",
|
|
397
|
+
"findings": { "P1": 0, "P2": 1, "P3": 0 }
|
|
398
|
+
}
|
|
399
|
+
}
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
Only those bounded fields are copied. Claims are informational and never affect
|
|
403
|
+
receipt validity, `outcome.status`, or the completion gate. The fingerprint of
|
|
404
|
+
a configured pre-existing file is used only to avoid ingesting stale claims;
|
|
405
|
+
Harnex always atomically replaces the final receipt, eliminating the old
|
|
406
|
+
missing/stale-proof acceptance race. Legacy workers that still write a full v1
|
|
407
|
+
file may contribute its outcome summary/status as advisory claims, but Harnex
|
|
408
|
+
replaces the document and ignores it for acceptance.
|
|
409
|
+
|
|
410
|
+
`artifact_report` in the dispatch row records final path, bytes, SHA-256,
|
|
411
|
+
schema, ingest status, report status, and author. `receipt`, `observed`, and
|
|
412
|
+
`validation` are compact top-level copies; `claims` appears only when present.
|
|
413
|
+
The default receipt path is outside the checkout so proof generation cannot
|
|
414
|
+
pollute the Git delta. An explicit path is supported when a queue requires one.
|
|
415
|
+
Receipt write/validation failure is typed `report_invalid` and fails closed.
|
|
416
|
+
Harnex never scrapes report-shaped final prose or copies full transcripts.
|
|
417
|
+
|
|
418
|
+
`harnex artifact-report init` still creates the older manual skeleton, and the
|
|
419
|
+
validator still accepts that legacy final contract. It is compatibility tooling,
|
|
420
|
+
not a required worker step for new dispatches.
|
|
421
|
+
|
|
422
|
+
## Autonomous completion gate
|
|
423
|
+
|
|
424
|
+
For Codex app-server runs launched with `--context`, provider turn completion
|
|
425
|
+
is not by itself accepted work completion. Harnex emits `task_complete` only
|
|
426
|
+
when it has at least one structured command/tool/file-change item or a Git
|
|
427
|
+
delta. If both are absent, it emits `task_failed` with
|
|
428
|
+
`outcome_class=completed_no_activity`, writes a rejected observed receipt, and
|
|
429
|
+
normalizes auto-stop to non-zero. This applies equally to
|
|
430
|
+
`service_tier=flex` and `service_tier=fast` and deliberately ignores final
|
|
431
|
+
answer text and claims. Intentional no-op work must still perform observable
|
|
432
|
+
inspection/validation; a model assertion of `no_change` cannot self-approve.
|
|
433
|
+
|
|
434
|
+
PTY transports do not expose equivalent completion-item metadata, so their
|
|
435
|
+
existing prompt-return auto-stop behavior remains. Non-Codex transports label
|
|
436
|
+
command observation `unsupported`; Git and terminal state are still recorded
|
|
437
|
+
without guessing.
|
|
438
|
+
|
|
439
|
+
## Actuals
|
|
440
|
+
|
|
441
|
+
At process exit, harnex collects usage through the active adapter. JSON-RPC
|
|
442
|
+
Codex sessions read cumulative `thread/tokenUsage/updated` data, Pi RPC sessions
|
|
443
|
+
read `get_session_stats`, and PTY adapters parse the last 16 KB of transcript
|
|
444
|
+
when they support a parser. Adapters without a parser emit nullable usage
|
|
445
|
+
fields. Separately, Pi aggregates bounded `contextUsage` samples and Codex
|
|
446
|
+
aggregates `tokenUsage.last` plus `modelContextWindow`; neither source is
|
|
447
|
+
substituted with cumulative usage when active occupancy is unavailable.
|
|
448
|
+
|
|
449
|
+
Git actuals capture the start/end SHA plus committed, staged, unstaged, and
|
|
450
|
+
untracked changes relative to the worktree state observed at session start.
|
|
451
|
+
Unchanged pre-existing dirt is not credited to the worker, and harness-owned
|
|
452
|
+
dispatch/receipt paths are excluded. Git failures leave the corresponding
|
|
453
|
+
consolidated fields `null` and omit `git` events.
|
|
454
|
+
|
|
455
|
+
The `actual` block includes model/effort hints from `--meta`, duration, token
|
|
456
|
+
counts, `agent_session_id`, compatibility `cost_usd`, adapter transport, git
|
|
457
|
+
deltas, exit reason, task completion state, signal/exit code, last error,
|
|
458
|
+
operational counters (`stalls`, `force_resumes`, `disconnections`,
|
|
459
|
+
`compactions`, `turn_count`, `tool_calls`, `commands_executed`), rate-limit
|
|
460
|
+
payloads, output/event volume measurements, and output/events log paths. New
|
|
461
|
+
additive attempt counters are `attempts_total`, `attempts_succeeded`,
|
|
462
|
+
`attempts_failed`, `retry_count`, `throttle_429_count`, `disconnect_count`, and
|
|
463
|
+
`fallback_triggered`. Throughput values are populated only for a
|
|
464
|
+
harness-accepted outcome: `throughput_tokens_per_s` and
|
|
465
|
+
`throughput_successes_per_h`; `retry_tax_pct` is `0.0` when no retry occurred
|
|
466
|
+
and `null` until a retry source can measure attributable wasted tokens.
|
|
467
|
+
|
|
468
|
+
Legacy `actual.cost_usd` reflects adapter/provider-reported cost only. Prefer
|
|
469
|
+
the top-level `usage` block: it also carries price-table-derived cost and its
|
|
470
|
+
`cost_source` / `cost_price_as_of` provenance. Claude PTY currently has no
|
|
471
|
+
bounded usage producer and reports `unsupported`; it is not silently priced.
|
|
472
|
+
|
|
473
|
+
Examples for downstream analysis (never treat missing usage as zero):
|
|
474
|
+
|
|
475
|
+
```bash
|
|
476
|
+
# Accepted successes per hour, grouped by project/phase/effective model.
|
|
477
|
+
jq -s 'map(select(.outcome.status == "accepted" and .attribution.status == "complete"))
|
|
478
|
+
| group_by([.attribution.project_id, .attribution.phase, .agent.model_effective])
|
|
479
|
+
| map({group: .[0].attribution.project_id + "/" + .[0].attribution.phase + "/" + .[0].agent.model_effective,
|
|
480
|
+
successes_per_hour: ((length * 3600) / (map(.actual.duration_s) | add))})' .harnex/dispatch.jsonl
|
|
481
|
+
|
|
482
|
+
# Retry and real-disconnect rates for completed rows.
|
|
483
|
+
jq -s 'map(select(.actual.attempts_total > 0))
|
|
484
|
+
| {retry_rate: ((map(.actual.retry_count) | add) / (map(.actual.attempts_total) | add)),
|
|
485
|
+
disconnect_rate: ((map(.actual.disconnect_count) | add) / (map(.actual.attempts_total) | add))}' .harnex/dispatch.jsonl
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
## Exit taxonomy
|
|
489
|
+
|
|
490
|
+
- `success`: wrapped process exited `0` with task completion, accepted observed
|
|
491
|
+
receipt proof, or an adapter session summary.
|
|
492
|
+
- `failure`: wrapped process exited non-zero, the observed-activity gate emitted
|
|
493
|
+
`task_failed`, or Harnex could not write/validate the receipt.
|
|
494
|
+
- `timeout`: wrapped process exited with code `124`.
|
|
495
|
+
- `boot_failure`: JSON-RPC app-server exited within the startup window before a
|
|
496
|
+
turn was observed.
|
|
497
|
+
- `disconnected`: wrapped process exited `0` but no session summary was parsed.
|
|
498
|
+
|
|
499
|
+
Canonical dispatch-stream writes remain best-effort. Receipt
|
|
500
|
+
writes are different: proof-generation failure is fail-closed and changes the
|
|
501
|
+
work verdict. Repo phase allowlists and runtime-log retention are documented in
|
|
502
|
+
[configuration.md](configuration.md).
|
data/docs/events.md
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# `harnex events` (v1)
|
|
2
|
+
|
|
3
|
+
## 1. Purpose and non-goals
|
|
4
|
+
|
|
5
|
+
`harnex events` provides a per-session JSONL stream for orchestration and
|
|
6
|
+
monitoring tooling.
|
|
7
|
+
|
|
8
|
+
This layer is transport + contract only. It does not implement watcher policy
|
|
9
|
+
or preset logic.
|
|
10
|
+
|
|
11
|
+
## 2. CLI usage
|
|
12
|
+
|
|
13
|
+
```text
|
|
14
|
+
harnex events --id ID [--repo PATH] [--cli CLI] [--from ISO8601]
|
|
15
|
+
harnex events --id ID --snapshot
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
- `--id ID` is required.
|
|
19
|
+
- `--follow` is enabled by default.
|
|
20
|
+
- `--snapshot` is non-blocking (`--no-follow` alias behavior).
|
|
21
|
+
- `--from` accepts ISO-8601 only (`ts >= from` replay filter).
|
|
22
|
+
|
|
23
|
+
Follow mode exits `0` when the target session emits `type: "exited"`.
|
|
24
|
+
|
|
25
|
+
## 3. Transport and file location
|
|
26
|
+
|
|
27
|
+
Events are append-only JSONL rows at:
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
~/.local/state/harnex/events/<repo_key>--<id_key>.jsonl
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Writers append one event per line and flush after each append.
|
|
34
|
+
Readers can snapshot + tail the same file.
|
|
35
|
+
|
|
36
|
+
## 4. v1 schema reference
|
|
37
|
+
|
|
38
|
+
Every row includes this envelope:
|
|
39
|
+
|
|
40
|
+
- `schema_version` (Integer): always `1`
|
|
41
|
+
- `seq` (Integer): monotonic per session, starts at `1`
|
|
42
|
+
- `ts` (String): UTC ISO-8601 timestamp
|
|
43
|
+
- `id` (String): session ID
|
|
44
|
+
- `type` (String): event type
|
|
45
|
+
|
|
46
|
+
Emitted now (Layer 4):
|
|
47
|
+
|
|
48
|
+
- `started`: adds `pid` (Integer)
|
|
49
|
+
- `send`: adds
|
|
50
|
+
- `msg` (String): first 200 characters of the original text; if longer,
|
|
51
|
+
a trailing `…` is appended
|
|
52
|
+
- `msg_truncated` (Boolean): whether truncation occurred
|
|
53
|
+
- `forced` (Boolean): send force mode
|
|
54
|
+
- `exited`: adds
|
|
55
|
+
- `code` (Integer): synthesized numeric exit code
|
|
56
|
+
- `signal` (Integer, optional): present for signaled exits
|
|
57
|
+
|
|
58
|
+
## 5. Stability promise
|
|
59
|
+
|
|
60
|
+
Schema v1 is additive-only:
|
|
61
|
+
|
|
62
|
+
- existing fields will not be removed, renamed, or type-changed
|
|
63
|
+
- new fields and new event types may be added
|
|
64
|
+
|
|
65
|
+
Breaking changes require a major schema bump.
|
|
66
|
+
|
|
67
|
+
## 6. Consumer patterns
|
|
68
|
+
|
|
69
|
+
Snapshot:
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
harnex events --id worker --snapshot
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Follow:
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
harnex events --id worker | jq -c '.'
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Replay from a timestamp:
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
harnex events --id worker --snapshot --from 2026-04-29T10:00:00Z
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## 7. Layer 2 integration note
|
|
88
|
+
|
|
89
|
+
Layer 4 defines the bus and schema. In a follow-up layer, watcher-owned
|
|
90
|
+
producers can publish `resume`, `log_active`, and `log_idle` events to this
|
|
91
|
+
same stream. `harnex events` remains a read-only consumer surface.
|
|
92
|
+
|
|
93
|
+
## 8. Layer 5: dispatch telemetry
|
|
94
|
+
|
|
95
|
+
Layer 5 adds dispatch telemetry events without changing schema version `1`.
|
|
96
|
+
The additions are optional for legacy consumers: existing event types keep
|
|
97
|
+
their fields, and new event types can be ignored by readers that do not need
|
|
98
|
+
telemetry.
|
|
99
|
+
|
|
100
|
+
New optional fields on existing event types:
|
|
101
|
+
|
|
102
|
+
- `started.meta` (Object, optional): parsed verbatim from `harnex run --meta`.
|
|
103
|
+
It is absent when `--meta` is not provided.
|
|
104
|
+
- `exited.reason` (String, optional): one of `success`, `failure`, `timeout`,
|
|
105
|
+
or `disconnected`.
|
|
106
|
+
|
|
107
|
+
New event types:
|
|
108
|
+
|
|
109
|
+
- `usage`: emitted once after the wrapped process exits and before `exited`.
|
|
110
|
+
It includes nullable `input_tokens`, `output_tokens`, `reasoning_tokens`,
|
|
111
|
+
`cached_tokens`, `total_tokens`, and `agent_session_id`.
|
|
112
|
+
- `git`: emitted when git metadata is available. `phase: "start"` includes
|
|
113
|
+
`sha` and `branch`; `phase: "end"` includes `sha`, `loc_added`,
|
|
114
|
+
`loc_removed`, `files_changed`, and `commits`.
|
|
115
|
+
- `summary`: emitted last before `exited`. `path` is the canonical tracked
|
|
116
|
+
dispatch stream — the only destination a dispatch record is written to.
|
|
117
|
+
`exit` is `success`, `failure`, `timeout`, or `disconnected`.
|
|
118
|
+
|
|
119
|
+
Example telemetry sequence:
|
|
120
|
+
|
|
121
|
+
```json
|
|
122
|
+
{"schema_version":1,"seq":1,"ts":"2026-05-01T11:30:00Z","id":"cx-i-372","type":"started","pid":12345,"meta":{"issue":"23","plan":"27","predicted":{"input_tokens":[200000,800000]}}}
|
|
123
|
+
{"schema_version":1,"seq":2,"ts":"2026-05-01T11:30:00Z","id":"cx-i-372","type":"git","phase":"start","sha":"a8114695c1f0","branch":"main"}
|
|
124
|
+
{"schema_version":1,"seq":3,"ts":"2026-05-01T11:42:13Z","id":"cx-i-372","type":"usage","input_tokens":104158,"output_tokens":2709,"reasoning_tokens":870,"cached_tokens":250880,"total_tokens":106867,"agent_session_id":"019ddf05-0f03-7d70-904f-23db7f00640f"}
|
|
125
|
+
{"schema_version":1,"seq":4,"ts":"2026-05-01T11:42:13Z","id":"cx-i-372","type":"git","phase":"end","sha":"abc1234567","loc_added":312,"loc_removed":65,"files_changed":7,"commits":1}
|
|
126
|
+
{"schema_version":1,"seq":5,"ts":"2026-05-01T11:42:13Z","id":"cx-i-372","type":"summary","path":"/home/u/proj/.harnex/dispatch.jsonl","exit":"success"}
|
|
127
|
+
{"schema_version":1,"seq":6,"ts":"2026-05-01T11:42:13Z","id":"cx-i-372","type":"exited","code":0,"reason":"success"}
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Events files are runtime logs, not the durable dispatch summary. Their age and
|
|
131
|
+
size retention is configurable; current/live-session files are protected. See
|
|
132
|
+
[configuration.md](configuration.md).
|