@deniscodp/skills 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +89 -2
- package/bin/deniscodp-skills.mjs +4 -0
- package/package.json +23 -4
- package/scripts/catalog.mjs +68 -0
- package/scripts/cli.mjs +264 -0
- package/scripts/install.mjs +184 -0
- package/scripts/recommended.mjs +102 -0
- package/skills/denisco-handoff-implementation/SKILL.md +254 -0
- package/skills/denisco-handoff-implementation/references/blocker-checklist.md +71 -0
- package/skills/denisco-handoff-implementation/references/delegated-implementation.md +168 -0
- package/skills/denisco-handoff-implementation/references/ownership.md +105 -0
- package/skills/denisco-handoff-implementation/references/session-plan.md +195 -0
- package/skills/denisco-handoff-implementation/scripts/ownership.mjs +153 -0
- package/skills/denisco-implementation-coordinator/SKILL.md +237 -0
- package/skills/denisco-implementation-coordinator/agents/openai.yaml +2 -0
- package/skills/denisco-implementation-coordinator/references/documents.md +175 -0
- package/skills/denisco-implementation-coordinator/references/runtimes.md +105 -0
- package/skills/denisco-interactive-spec-definition/SKILL.md +126 -0
- package/skills/denisco-interactive-spec-definition/references/blocker-checklist.md +71 -0
- package/skills/denisco-interactive-spec-definition/references/handoff-template.md +124 -0
- package/skills/denisco-project-context/SKILL.md +60 -0
- package/skills/denisco-project-context/references/project-documents.md +70 -0
- package/skills/denisco-visual-check/SKILL.md +136 -0
- package/skills/denisco-visual-check/references/judge-prompt.md +55 -0
- package/skills/denisco-visual-check/references/project-config.md +118 -0
- package/skills/denisco-workflow-benchmark/SKILL.md +113 -0
- package/skills/denisco-workflow-benchmark/agents/openai.yaml +2 -0
- package/skills/denisco-workflow-benchmark/references/measurement.md +166 -0
- package/skills/denisco-workflow-benchmark/references/workflow-contract.md +371 -0
- package/skills/denisco-workflow-benchmark/scripts/bench.mjs +156 -0
- package/skills/denisco-workflow-benchmark/scripts/cases.mjs +46 -0
- package/skills/denisco-workflow-benchmark/scripts/configuration.mjs +71 -0
- package/skills/denisco-workflow-benchmark/scripts/discover.mjs +88 -0
- package/skills/denisco-workflow-benchmark/scripts/eval-worker.mjs +58 -0
- package/skills/denisco-workflow-benchmark/scripts/io.mjs +82 -0
- package/skills/denisco-workflow-benchmark/scripts/prepare.mjs +69 -0
- package/skills/denisco-workflow-benchmark/scripts/runs.mjs +235 -0
- package/skills/denisco-workflow-benchmark/scripts/state-prompt.mjs +96 -0
- package/skills/denisco-workflow-benchmark/scripts/tracking.mjs +91 -0
- package/skills/denisco-workflow-benchmark/scripts/usage.mjs +175 -0
- package/skills/denisco-workflow-benchmark/scripts/workflow.mjs +334 -0
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# Visual configuration contract
|
|
2
|
+
|
|
3
|
+
Execution/authentication instructions belong to the project operating guide. This file defines
|
|
4
|
+
capture coordinates only. Reuse existing project configuration before creating another file.
|
|
5
|
+
|
|
6
|
+
## Location and compatibility
|
|
7
|
+
|
|
8
|
+
Use the path named by project instructions. Otherwise look for an existing
|
|
9
|
+
`.agent/visual-check.json`. If needed, use the established agent-local directory.
|
|
10
|
+
Legacy `$schema: "visual-check/1"` configurations remain readable; the value is a format marker,
|
|
11
|
+
not a URL to fetch. Preserve unknown fields. Do not automatically migrate or overwrite them.
|
|
12
|
+
|
|
13
|
+
Configuration is optional for user-supplied screenshots and standalone mockups. For real app
|
|
14
|
+
capture require a known URL and executable access procedure, not a mandatory new JSON file.
|
|
15
|
+
If startup is needed, resolve a command from the operating guide or legacy app section.
|
|
16
|
+
|
|
17
|
+
## Fields and defaults
|
|
18
|
+
|
|
19
|
+
| Field | Meaning/default |
|
|
20
|
+
|---|---|
|
|
21
|
+
| app | Optional URL and serveCommand; omit when there is no runnable app |
|
|
22
|
+
| app.reuseIfListening | true; verify it is the intended app before reusing |
|
|
23
|
+
| app.readyWhen | Optional path and selector indicating readiness |
|
|
24
|
+
| operatingGuide | Optional path to shared start/auth/test instructions |
|
|
25
|
+
| auth | Legacy none or interactive-once; absence means unspecified, not proof of no auth |
|
|
26
|
+
| auth.signedInWhen | Selector proving authenticated state |
|
|
27
|
+
| auth.sessionFile | Local ignored session state; treat as sensitive |
|
|
28
|
+
| environment | Explicit shared/readOnly flags; unspecified write policy means no write tests |
|
|
29
|
+
| artifactsDir | Default .agent/visual-check; local ignored capture outputs, grouped per task |
|
|
30
|
+
| toolingDir | Optional path outside the worktree holding tooling reused across worktrees |
|
|
31
|
+
| viewport | Default width 1440, height 900, deviceScaleFactor 1 |
|
|
32
|
+
| theme | Default system; record actual rendered theme for each capture |
|
|
33
|
+
| routes | Symbolic names to paths |
|
|
34
|
+
| fixtures | Symbolic case names to reproducible, non-sensitive data coordinates |
|
|
35
|
+
|
|
36
|
+
For interactive-once auth, use a headed human login, save state in an ignored location and verify
|
|
37
|
+
signedInWhen when reusing. An expired session needs renewed login or the project's authorized
|
|
38
|
+
authentication procedure, not password guessing.
|
|
39
|
+
|
|
40
|
+
The saved session, the login helper and helpers reused across worktrees belong in toolingDir:
|
|
41
|
+
machine-local, never credentials, and named in the operating guide so a new worktree finds them.
|
|
42
|
+
artifactsDir stays per worktree and holds outputs. Without toolingDir the tooling is per worktree
|
|
43
|
+
and does not survive a new checkout; say so rather than leaving it to be discovered.
|
|
44
|
+
|
|
45
|
+
Navigation, menus, flags, login and environment behavior belong to the operating guide, not to this
|
|
46
|
+
file. Keep coordinates here: routes, viewport, symbolic fixture names pointing at the guide.
|
|
47
|
+
Duplicated prose goes stale and then contradicts the guide. Migrate an existing note bag as you
|
|
48
|
+
touch it, not in one pass.
|
|
49
|
+
|
|
50
|
+
Use the same operating-guide reference for business behavior and visual checks. Do not duplicate
|
|
51
|
+
service/login facts in every handoff. Record discovered versus actually executed procedures.
|
|
52
|
+
Creating local artifact/config directories must respect project instructions and active write
|
|
53
|
+
permissions. Never automatically broaden shared-environment write permissions.
|
|
54
|
+
|
|
55
|
+
## Artifact layout and index
|
|
56
|
+
|
|
57
|
+
Artifacts live under `artifactsDir`, one folder per task named with the handoff task ID, and the
|
|
58
|
+
outputs of a single run in a subfolder of it. Only the index and helpers shared across tasks stay
|
|
59
|
+
at its root; what is reused across worktrees, the browser profile included, belongs to toolingDir.
|
|
60
|
+
A task's probes, captures and logs are then retrievable from its name alone, without grepping
|
|
61
|
+
unrelated files.
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
<toolingDir>/ shared across worktrees, outside the repository
|
|
65
|
+
profile/ saved session state
|
|
66
|
+
login.mjs headed login helper
|
|
67
|
+
lib/ helpers reused across worktrees
|
|
68
|
+
|
|
69
|
+
.agent/
|
|
70
|
+
visual-check.json configuration
|
|
71
|
+
visual-check/ artifactsDir, per worktree
|
|
72
|
+
INDEX.md index of every probe and diagnostic
|
|
73
|
+
dss-fase3-special-fields/ one folder per task
|
|
74
|
+
probe-v1-v3.mjs
|
|
75
|
+
probe-v1-v3/ outputs of one run
|
|
76
|
+
stato.png
|
|
77
|
+
diag.json
|
|
78
|
+
diag-tabella-vuota.mjs
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
`INDEX.md` carries one row per probe or diagnostic:
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
| Task | Artifact | Covers | Needs | Status | Updated |
|
|
85
|
+
|---|---|---|---|---|---|
|
|
86
|
+
| dss-fase3-special-fields | dss-fase3-special-fields/probe-v1-v3.mjs | V1, V2, V3 | selected group, saved filter cleared | current | 2026-09-14 |
|
|
87
|
+
| dss-fase3-appuntamento | dss-fase3-appuntamento/probe-punto2.mjs | P2-A1 | selected group | superseded by dss-fase3-special-fields/probe-v1-v3.mjs | 2026-09-11 |
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Read it before creating a probe. A principal (the implementation owner, including a bounded
|
|
91
|
+
programming worker with its own claim) adds or updates the row when finishing one; a delegated visual-only
|
|
92
|
+
child returns proposed rows for the principal to merge, without writing the shared index. Give each
|
|
93
|
+
child an exclusive `<taskId>/<attemptId>/` directory for both probe copies and outputs. Children
|
|
94
|
+
read shared helpers but report needed repairs to the principal rather than changing them concurrently.
|
|
95
|
+
An artifact missing from the index is harder for the next session to find; absence from the index
|
|
96
|
+
alone does not prove missing evidence. Keep superseded rows and name the successor: deleting
|
|
97
|
+
them loses why a probe was replaced. Name in Needs the environment preconditions a probe depends
|
|
98
|
+
on, such as a selected group or a saved filter; their values live in the operating guide, where
|
|
99
|
+
they serve everyone. A rediscovered prerequisite costs a full diagnostic run. Criterion outcomes
|
|
100
|
+
do not belong here.
|
|
101
|
+
|
|
102
|
+
Legacy flat layouts stay valid. Index them where they are and adopt the per-task layout for new
|
|
103
|
+
artifacts instead of rewriting an accumulated directory in one pass.
|
|
104
|
+
|
|
105
|
+
## Example
|
|
106
|
+
|
|
107
|
+
```json
|
|
108
|
+
{
|
|
109
|
+
"$schema": "visual-check/1",
|
|
110
|
+
"operatingGuide": "docs/agent-workflow/OPERATIONS.md",
|
|
111
|
+
"app": { "url": "http://localhost:5173", "serveCommand": "npm run dev" },
|
|
112
|
+
"environment": { "shared": false, "readOnly": true },
|
|
113
|
+
"viewport": { "width": 1440, "height": 900, "deviceScaleFactor": 1 },
|
|
114
|
+
"routes": { "orders": "/orders" }
|
|
115
|
+
}
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Missing fixtures remain unresolved prerequisites for those cases; do not invent values.
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: denisco-workflow-benchmark
|
|
3
|
+
description: "Prepare reproducible workflow comparisons, register task sessions, import local Codex/Claude usage and report evidence-backed outcomes. Use only when workflow measurement or benchmarking is explicitly requested; not for ordinary task execution, implicit measurement, prose-only completion judgments or launching paid runs."
|
|
4
|
+
metadata:
|
|
5
|
+
version: "5.1.0"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Workflow benchmark
|
|
9
|
+
|
|
10
|
+
Use [the CLI and data contract](references/measurement.md) before collecting or preparing a run.
|
|
11
|
+
The executable is [scripts/bench.mjs](scripts/bench.mjs); it works with Node 20+ and never launches
|
|
12
|
+
a model. Locate it relative to this installed skill, not a remembered developer workstation path.
|
|
13
|
+
For real tasks read [persistent handoff and closure](references/workflow-contract.md).
|
|
14
|
+
Use `bind` to persist the explicit run in `<handoff-path>.workflow.json`; a later session must
|
|
15
|
+
discover it from the handoff alone, not rely on external PROMPTS.md as the only connection.
|
|
16
|
+
|
|
17
|
+
Task workflow measurement is opt-in. Use this skill only after the current user request or applicable
|
|
18
|
+
project instructions explicitly activate measurement for this task. Installation, skill discovery,
|
|
19
|
+
acceptance criteria, an old register and `measurement.status: unknown` are not activation. Disabled
|
|
20
|
+
is authoritative: do not bind, register, collect or rewrite prompts for that task. When explicitly
|
|
21
|
+
enabled, the agent owns setup, retries, collection and reporting; users receive outcomes rather than
|
|
22
|
+
session IDs, nonces or commands. Follow the operational contract and preserve existing bindings.
|
|
23
|
+
After specification save/check `planning-prompt` in HANDOFF; it requires no plan or block.
|
|
24
|
+
Planning prompts assign PLAN/STATE, the block map and a saved/checked first execution prompt, then
|
|
25
|
+
the start question; only the user's yes to it, or a later submitted execution prompt, starts
|
|
26
|
+
implementation. Native Plan Mode approval never does. Submitting a saved execution prompt as the
|
|
27
|
+
user's request continues the task from the named block; reading or generating it does not.
|
|
28
|
+
A working session then continues block by block until a real stop defined by denisco-handoff-implementation.
|
|
29
|
+
Keep phase boundaries independent of tool mode.
|
|
30
|
+
For handoff planning/implementation, acquire the denisco-handoff-implementation worktree claim before
|
|
31
|
+
registration or any other write. Conversation registration deduplicates usage; it does not claim
|
|
32
|
+
execution ownership. A second process with the same conversation ID remains a separate writer.
|
|
33
|
+
An observer blocked by ownership may read records or run `checkpoint`, but not `open`, `track`,
|
|
34
|
+
`collect`, `report` or `close`, which write. Delegated visual-only children return evidence for the
|
|
35
|
+
implementation owner to register; they do not mutate the shared run themselves. A bounded programming
|
|
36
|
+
worker with its own claim records its own evidence while the coordinator stays read-only. Follow
|
|
37
|
+
the coordinated execution section of the persistent workflow contract.
|
|
38
|
+
Between workers, a coordinator may register/refresh permitted measurement records under its own
|
|
39
|
+
claim, but leaves technical documents unchanged: its prompts and history belong in COORDINATOR
|
|
40
|
+
and COORDINATOR-HISTORY. `close`, `bind` and technical prompt save/clear stay with the standard
|
|
41
|
+
specification/implementation workflow; they are not coordinator bookkeeping commands.
|
|
42
|
+
|
|
43
|
+
Keep the task and acceptance fixed before measurement. Separate exploration investment, task work
|
|
44
|
+
and correction. Register native session IDs explicitly; do not attribute the latest unrelated log.
|
|
45
|
+
Use one task per session in v1, or explicit time bounds for mixed/resumed histories.
|
|
46
|
+
|
|
47
|
+
After registration, use `open` when the planned block map exists: it refreshes registered logs
|
|
48
|
+
and prints the register and assignment. An unchanged log does not prove its conversation ended.
|
|
49
|
+
The agent runs collect/report at closure and refreshes them on the next entry; the final response's
|
|
50
|
+
trailing usage remains pending until an observer can refresh it. Close with `close`, which saves the prepared
|
|
51
|
+
continuation, checkpoints the block and refreshes usage. Both compose existing commands and attest
|
|
52
|
+
nothing: coverage, acceptance and the claimed status stay explicit decisions.
|
|
53
|
+
|
|
54
|
+
Register the current conversation with `track --handoff <file> --provider <actual-provider>
|
|
55
|
+
--phase <actual-phase>`, then execute the returned nextCall yourself in a separate tool call.
|
|
56
|
+
The helper generates the marker, resolves the unique log and registers it; existing registration
|
|
57
|
+
is reused without double counting. The helper snapshots the provider's installed skill bundle
|
|
58
|
+
when available and associates it with the new session. If pending, retry once later in the same
|
|
59
|
+
conversation, then report the gap. Keep the returned ticket internal; use it with
|
|
60
|
+
`close --ticket <ticket>` in this same conversation so close reuses the registered identity.
|
|
61
|
+
Never copy the ticket into the next-session prompt. Low-level whoami/session remain available for
|
|
62
|
+
diagnostics and historical recovery with positive attribution evidence.
|
|
63
|
+
|
|
64
|
+
Collect usage from local logs through the CLI. Never estimate tokens or copy conversations into
|
|
65
|
+
reports. Inspect coverage warnings, duplicates, parent/child attribution and version information.
|
|
66
|
+
Missing metrics remain null, not zero. Reimport after the session has finished for final accounting.
|
|
67
|
+
The working session records evidence and its identity when known; an operator/observer finalizes
|
|
68
|
+
usage after closure. If identity is ambiguous, leave an explicit pending action. Do not launch a
|
|
69
|
+
model solely to measure known IDs. Keep measurement-only agent overhead separately visible.
|
|
70
|
+
|
|
71
|
+
Evaluate acceptance using recorded evidence or the prepared external verifier. An agent's summary
|
|
72
|
+
does not mark a task verified. Record user acceptance/reopening separately and preserve history.
|
|
73
|
+
|
|
74
|
+
Compare matched cases within the same provider/model/settings first. Include failures and blocked
|
|
75
|
+
runs, not just successful ones. Report quality before resource use; do not turn different tokenizers
|
|
76
|
+
into a claim that one provider is universally more efficient. Small pilots are descriptive only.
|
|
77
|
+
|
|
78
|
+
Preparing a benchmark does not authorize a model run. Return ready-to-paste prompts and commands.
|
|
79
|
+
For controlled cases, keep evaluator and reference solutions outside the working copy. This is a
|
|
80
|
+
procedural boundary, not an OS sandbox: record evidence of inspection or instruction contamination.
|
|
81
|
+
|
|
82
|
+
After each block's validation and evidence registration, update STATE and run `checkpoint` for that
|
|
83
|
+
block. When the working session stops, use `next-prompt --save` for the proposed next assignment or
|
|
84
|
+
same-block resume, then `checkpoint --next-block` to check both the last block worked and the
|
|
85
|
+
persisted prompt. A same-block resume is allowed; no command chooses the assignment.
|
|
86
|
+
`next-prompt --check` is read-only; plain `next-prompt` only prints and does not satisfy delivery.
|
|
87
|
+
The scripts check integrity/completeness and their generated section, not artifact relevance or
|
|
88
|
+
arbitrary Markdown claims. Keep custom notes outside that section. Link the saved STATE in the
|
|
89
|
+
response; any prompt copied into chat must match the saved text. When no task work remains, or the user ends the task or asks for no prompt, use `--clear`
|
|
90
|
+
and checkpoint without `--next-block`, retaining unresolved acceptance/accounting in STATE.
|
|
91
|
+
These structural operations work with measurement disabled/unknown. In Plan Mode leave writes
|
|
92
|
+
pending. Later changes to code or evidence require refreshed results and another checkpoint.
|
|
93
|
+
|
|
94
|
+
A skill can be loaded by the Skill tool or by typing its slash command; the register records both
|
|
95
|
+
and being installed still proves neither. A report marks each session's usage final, stale or
|
|
96
|
+
unavailable by rehashing its log, so a skipped collect is visible instead of silent.
|
|
97
|
+
|
|
98
|
+
Capture skill configuration with `configuration` and associate it explicitly with sessions.
|
|
99
|
+
Read `metadata.version`, not a top-level version; retain hashes if the version is unknown.
|
|
100
|
+
|
|
101
|
+
For visual checks the principal is the default executor. Delegate the whole check only on explicit
|
|
102
|
+
user request; a blind perceptual judge remains independently delegated when available and permitted.
|
|
103
|
+
Use visual-start --executor principal|subagent before the check and visual-finish after it, with the
|
|
104
|
+
registered executor session, explicit skill loading, criterion results and artifacts. Refresh the
|
|
105
|
+
executor's log before finish so loading is observed. A principal check requires a top-level session;
|
|
106
|
+
a delegated check requires a child attributed to the named principal. Register judge usage separately.
|
|
107
|
+
Legacy records keep their executor; never reinterpret a historical delegated attempt as a principal
|
|
108
|
+
one. Inspect artifacts and native logs to repair registration gaps before scheduling a replacement.
|
|
109
|
+
For Claude children, use the native JSONL under the conversation's subagents directory with its
|
|
110
|
+
recorded root sessionId plus agentId, not a temporary task .output file or agentId as sessionId.
|
|
111
|
+
Use actual native metadata and invocation evidence on every provider; ambiguity stays pending.
|
|
112
|
+
Installed files are not proof of loading. Preserve configuration/environment history across
|
|
113
|
+
upgrades; never attribute today's installed versions to earlier sessions without evidence.
|
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
# Measurement contract and CLI
|
|
2
|
+
|
|
3
|
+
Run `node <installed-skill>/scripts/bench.mjs help`. Node 20+ is the only runtime dependency.
|
|
4
|
+
The CLI never launches a model. Its `evaluate` command runs local case code in a child Node process.
|
|
5
|
+
|
|
6
|
+
## Run identity and commands
|
|
7
|
+
|
|
8
|
+
For ordinary task work the agent uses `track` and `planning-prompt` following
|
|
9
|
+
[setup and registration](workflow-contract.md#agent-managed-setup-and-first-handoff).
|
|
10
|
+
`track --handoff FILE --provider codex|claude --phase PHASE` emits an internal ticket and nextCall;
|
|
11
|
+
execute that nextCall in a separate tool call to identify and register this conversation together.
|
|
12
|
+
When the provider skill root is present in the project, the helper snapshots and associates it
|
|
13
|
+
with the new session. Keep the ticket for the same conversation's `close --ticket` call; never
|
|
14
|
+
put it in the next-session prompt.
|
|
15
|
+
It works before planning and reports delayed/ambiguous identification as pending, also in reports.
|
|
16
|
+
Optional scope/parent/configuration flags and retry limits are in that contract. The low-level
|
|
17
|
+
commands below remain available; the user does not fill their IDs or nonces in the normal flow.
|
|
18
|
+
|
|
19
|
+
A task is the feature/bug, a block is a plan unit, and a session is a native conversation/log.
|
|
20
|
+
Planning and implementation in the same conversation are counted once, preserving the registered
|
|
21
|
+
phase. A resumed conversation is not automatically a new session. A nonce is only the helper's
|
|
22
|
+
temporary marker for locating a log; it is not a task or block identifier.
|
|
23
|
+
|
|
24
|
+
`init --out DIR --task ID --project DIR --variant NAME --criteria FILE` creates a new run manifest.
|
|
25
|
+
Criteria input is a JSON array of `{ "id": "A1", "expectation": "observable result" }`; required
|
|
26
|
+
defaults to true, and at least one required criterion is necessary. Existing directories are refused.
|
|
27
|
+
|
|
28
|
+
`open --handoff FILE --block ID [--phase PHASE] [--nonce TOKEN]` inspects an assigned block: it
|
|
29
|
+
validates the sidecar, runs `collect` on the bound run, reports each session's usage delta and
|
|
30
|
+
freshness, prints the register and criteria, checks the saved prompt for the block and returns a
|
|
31
|
+
marker plus the `whoami`/`session` commands. Its only write is that reimport. A failed prompt
|
|
32
|
+
check exits 2, as with `checkpoint`.
|
|
33
|
+
|
|
34
|
+
`close --handoff FILE --block ID --implementation STATE (--next-block ID|--final)` closes one: it
|
|
35
|
+
saves the assigned continuation (or clears it with `--final`), runs the checkpoint and refreshes
|
|
36
|
+
usage, then lists what remains manual. Exactly one continuation form is required. Steps run in that
|
|
37
|
+
order and stop at the first failure, naming what was already applied: each underlying command keeps
|
|
38
|
+
its own atomicity, the composition does not add a transaction. It never passes
|
|
39
|
+
`--coverage-confirmed`, never records acceptance and never infers `--claim`.
|
|
40
|
+
|
|
41
|
+
`whoami --provider codex|claude --nonce TOKEN [--project DIR --root DIR]` returns the session whose
|
|
42
|
+
log contains a marker that session emitted, and only when exactly one log matches. Zero matches
|
|
43
|
+
means the transcript has not been written yet: repeat it in a later call. `--project` narrows the
|
|
44
|
+
candidates to the logs recorded in that directory before they are counted, since it is a declared
|
|
45
|
+
constraint; if it leaves several, or none where the marker was found elsewhere, that is an error.
|
|
46
|
+
No fallback selects a log by recency.
|
|
47
|
+
|
|
48
|
+
`discover --provider codex|claude --project DIR` lists native IDs and log paths using metadata only.
|
|
49
|
+
Without `--root` the search starts at the CLI home named by `CODEX_HOME`
|
|
50
|
+
or `CLAUDE_CONFIG_DIR`, otherwise the default one. Optional `--root DIR` supports other machines
|
|
51
|
+
and CLI homes; `--limit N` defaults to 20. Discovery never assigns a log to a task. It only reads
|
|
52
|
+
a metadata prefix and may omit files without metadata there.
|
|
53
|
+
|
|
54
|
+
`session --run FILE --provider codex|claude --id NATIVE_ID --log FILE --phase PHASE` registers and
|
|
55
|
+
imports one explicitly selected log. Phases: exploration, specification, planning, implementation,
|
|
56
|
+
verification, correction. The registration rejects a native ID not present in the source.
|
|
57
|
+
|
|
58
|
+
Use `--from ISO --to ISO` to delimit mixed/resumed histories; from is inclusive and to exclusive.
|
|
59
|
+
Do not reuse the same native session for multiple phases in one run: assign its main phase or use
|
|
60
|
+
separate sessions. A session must not be attributed to several tasks without disjoint bounds.
|
|
61
|
+
The comparison warns when native identities are reused across runs.
|
|
62
|
+
|
|
63
|
+
For a child with its own native session ID, add `--parent provider:session-id --parent-evidence TEXT`.
|
|
64
|
+
For Claude logs sharing the root session ID, also use `--agent ID`; that agentId must occur in the
|
|
65
|
+
log. Parent evidence records the actual native invocation/metadata relationship, not an assumption.
|
|
66
|
+
Missing or unidentifiable child logs leave coverage incomplete. Native formats lacking these fields
|
|
67
|
+
are not silently inferred from filenames. Counts duplicated across registered logs are flagged.
|
|
68
|
+
|
|
69
|
+
Optional `session --configuration ID` associates a previously captured configuration explicitly.
|
|
70
|
+
Existing sessions can be linked with `session-config`; never infer attribution from the latest snapshot.
|
|
71
|
+
See [persistent task commands](workflow-contract.md) for bind, checkpoint, next-prompt and configuration.
|
|
72
|
+
`next-prompt --save` persists the assigned prompt in STATE and checks the write; `--check` checks
|
|
73
|
+
without writing, `--clear` removes the obsolete generated section. At handoff use
|
|
74
|
+
`checkpoint --next-block ID` to check persistence separately from technical verification.
|
|
75
|
+
|
|
76
|
+
`--coverage-confirmed` is an operator attestation that the source contains only this task/range,
|
|
77
|
+
all attributable subagents are included, and the source has been flushed. It is not a parser repair.
|
|
78
|
+
Set it only after checking scope. The reimport after closure happens in the next session's `open`,
|
|
79
|
+
or explicitly with `collect --run FILE`.
|
|
80
|
+
If initially left unconfirmed, `collect --run FILE --coverage-confirmed` attests coverage for all
|
|
81
|
+
registered sources after reviewing them; parser warnings are still retained.
|
|
82
|
+
The agent refreshes registered logs through `open` at the next block entry and runs collect/report
|
|
83
|
+
at closure. The last response's usage remains pending until a later observer can refresh the log.
|
|
84
|
+
A different conversation starting does not prove an earlier conversation has ended.
|
|
85
|
+
Repeated imports replace prior snapshots and are idempotent. A failed import does not partially
|
|
86
|
+
update a run. One writer per manifest: a leftover `.lock` after a crash needs operator inspection
|
|
87
|
+
before removal; never remove the lock of a live writer.
|
|
88
|
+
|
|
89
|
+
## Token semantics
|
|
90
|
+
|
|
91
|
+
| Source | Accounting |
|
|
92
|
+
|---|---|
|
|
93
|
+
| Codex event_msg/token_count | Differences between consecutive total_token_usage counters; duplicates contribute zero |
|
|
94
|
+
| Claude message.usage | One message ID counted once; stream fragments consolidate growing output counts |
|
|
95
|
+
| Skill loading | Skill tool calls and typed slash commands, recorded separately; a command name may be a CLI built-in |
|
|
96
|
+
|
|
97
|
+
Codex input already includes cached input; reasoning is a detail of output. Neither is added again.
|
|
98
|
+
Claude normalized input is input_tokens + cache_read_input_tokens + cache_creation_input_tokens.
|
|
99
|
+
Normalized total is input plus output. Preserve numeric native counts separately. Missing categories
|
|
100
|
+
are null, not guessed zero. Do not sum native snapshots and normalized totals together.
|
|
101
|
+
|
|
102
|
+
A Codex counter decrease indicates a gap/reset and makes coverage partial. A bounded import needs
|
|
103
|
+
a preceding counter; otherwise the first in-range total is omitted with a warning. Full-history
|
|
104
|
+
imports require review for fork prefixes and unrelated earlier work. A Claude request straddling
|
|
105
|
+
a time boundary is omitted as ambiguous. Unknown formats and partial lines produce warnings.
|
|
106
|
+
|
|
107
|
+
These are **observed local log adapters**, with read-only smoke checks on Codex 0.153.4 and Claude
|
|
108
|
+
Code 2.1.237 logs, plus synthetic tests (the installed Claude CLI reports 2.1.266).
|
|
109
|
+
They are not a guaranteed stable vendor API. Record actual CLI versions,
|
|
110
|
+
models and effort when present; unknown metadata remains explicit. Add format tests before expanding
|
|
111
|
+
support. Structured programmatic alternatives are documented by
|
|
112
|
+
[Codex](https://learn.chatgpt.com/docs/non-interactive-mode) and
|
|
113
|
+
[Claude](https://code.claude.com/docs/en/agent-sdk/cost-tracking).
|
|
114
|
+
[Claude monitoring](https://code.claude.com/docs/en/monitoring-usage) is an optional future source;
|
|
115
|
+
this version does not enable telemetry or alter CLI settings.
|
|
116
|
+
|
|
117
|
+
## Evidence, review and report
|
|
118
|
+
|
|
119
|
+
- `result --run FILE --criterion ID --status pass|fail|not-run --observation TEXT --evidence FILE`
|
|
120
|
+
records an actual observation. Pass/fail require an existing artifact, stored by path and hash.
|
|
121
|
+
`not-run` records a missing prerequisite without inventing evidence.
|
|
122
|
+
- `review --run FILE --status accepted|reopened|pending --note TEXT` records explicit user feedback.
|
|
123
|
+
Reopening retains past evidence and requires new results after the reopening event.
|
|
124
|
+
- `intervention --run FILE --kind clarification|correction|approval|environment --note TEXT`
|
|
125
|
+
records manual intervention; silence is not evidence of zero intervention effort.
|
|
126
|
+
- `environment --run FILE --note TEXT` records inspected loaded skill paths/versions, model/effort,
|
|
127
|
+
global instructions, memory and environment differences. Updates append environmentHistory,
|
|
128
|
+
preserving a legacy note on first update; environmentReview remains the latest view.
|
|
129
|
+
A note cannot prove isolation by itself.
|
|
130
|
+
- `report --run FILE` writes report.md and report.json beside the manifest. Each session's usage is
|
|
131
|
+
marked final, stale or unavailable by rehashing its log, so a missing collect is visible.
|
|
132
|
+
- `compare --runs FILE,FILE [--out JSON]` compares runs, retaining unsuccessful outcomes and warnings.
|
|
133
|
+
|
|
134
|
+
Manifest schemaVersion 1 stores task/variant/project, fixed criteria, sessions, metric snapshots,
|
|
135
|
+
append-only evidence/history/interventions and user review. Local manifests include native log paths;
|
|
136
|
+
do not commit or share them without reviewing metadata. No message bodies or credentials are exported.
|
|
137
|
+
The report's observed duration is the span of token events, not wall time since launch or active work.
|
|
138
|
+
|
|
139
|
+
Verified means all required criteria have passing evidence. It is distinct from user acceptance.
|
|
140
|
+
Artifact hash changes invalidate evidence. Controlled cases also invalidate verification if source
|
|
141
|
+
files change after evaluation. For ordinary project tasks the operator must reopen/reverify after
|
|
142
|
+
code changes; v1 does not fingerprint arbitrary large repositories automatically.
|
|
143
|
+
|
|
144
|
+
Quality precedes token comparisons. Keep exploration costs separate and visible in project totals;
|
|
145
|
+
do not hide them or count them once per downstream task. Compare variants within one provider/model
|
|
146
|
+
configuration first. Different tokenizers, hidden subagents, cache state and human interventions
|
|
147
|
+
can confound cross-provider comparisons. A single matched pair does not establish statistical benefit.
|
|
148
|
+
|
|
149
|
+
## Prepared cases
|
|
150
|
+
|
|
151
|
+
`prepare --out DIR --case order-filter|order-summary --variant baseline|candidate --provider codex|claude`
|
|
152
|
+
creates a workspace, identical decided handoff, fixed criteria, installed applicable skills,
|
|
153
|
+
PROMPT.md, OPERATOR.md and run.json. Baseline additionally requires `--baseline MANIFEST` pointing to
|
|
154
|
+
the immutable original snapshot. Candidate uses sibling skill folders in the installed bundle.
|
|
155
|
+
|
|
156
|
+
Only the three original skill responsibilities are installed in controlled workspaces; the benchmark
|
|
157
|
+
tool/evaluator and reference solutions stay outside. Run `evaluate --run FILE` externally after work.
|
|
158
|
+
Before launching any model, initialize Git inside the prepared workspace and verify that
|
|
159
|
+
`git rev-parse --show-toplevel` resolves to that workspace, not the enclosing toolkit repository.
|
|
160
|
+
The evaluator is procedural separation, not a hostile-code sandbox. Protected file changes block CLI
|
|
161
|
+
evaluation. Do not inspect the evaluator during a model run or treat a contaminated run as controlled.
|
|
162
|
+
|
|
163
|
+
Exit codes: 0 success (including reports/read-only blocked checkpoints), 1 invalid input/tool failure,
|
|
164
|
+
2 a contradictory checkpoint --claim, a failed continuation check (`next-prompt --check` or
|
|
165
|
+
`checkpoint --next-block`), or fixed acceptance
|
|
166
|
+
failure from CLI evaluate. A successful report command is not a successful task.
|