@tech-leads-club/harness-toolkit 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +95 -0
- package/LICENSE +84 -0
- package/NOTICE +9 -0
- package/README.md +480 -0
- package/bin/tlc +10 -0
- package/bin/tlc-build +80 -0
- package/bin/tlc-cli.ts +1509 -0
- package/bin/tlc-exec +10 -0
- package/bin/tlc-exec.cmd +4 -0
- package/bin/tlc-exec.d.mts +60 -0
- package/bin/tlc-exec.mjs +248 -0
- package/bin/tlc.cmd +4 -0
- package/bin/tlc.mjs +7 -0
- package/bin/write-user-hooks.d.mts +40 -0
- package/bin/write-user-hooks.mjs +123 -0
- package/capabilities/catalog.json +418 -0
- package/config.example.json +51 -0
- package/dist/compact-before.mjs +7839 -0
- package/dist/doctor.mjs +8263 -0
- package/dist/help-topic.mjs +82 -0
- package/dist/init-project.mjs +993 -0
- package/dist/install-runtime.mjs +969 -0
- package/dist/lessons-cli.mjs +7258 -0
- package/dist/obs-cli.mjs +7025 -0
- package/dist/price-lookup.mjs +221 -0
- package/dist/prompt-submit.mjs +7860 -0
- package/dist/refresh-model-prices.mjs +158 -0
- package/dist/response-after.mjs +7879 -0
- package/dist/run.mjs +7830 -0
- package/dist/session-end.mjs +7937 -0
- package/dist/session-start.mjs +8056 -0
- package/dist/shim.mjs +52 -0
- package/dist/stop.mjs +8494 -0
- package/dist/subagent-start.mjs +7861 -0
- package/dist/subagent-stop.mjs +7851 -0
- package/dist/support.mjs +7045 -0
- package/dist/tlc-cli.mjs +8063 -0
- package/dist/tool-after.mjs +8113 -0
- package/dist/tool-before.mjs +7985 -0
- package/dist/tool-failure.mjs +7850 -0
- package/dist/uninstall-runtime.mjs +1307 -0
- package/docs/architecture.md +270 -0
- package/docs/concepts.md +562 -0
- package/docs/coverage.md +55 -0
- package/docs/decisions/ad-001.md +32 -0
- package/docs/decisions/ad-002.md +51 -0
- package/docs/decisions/ad-003.md +30 -0
- package/docs/decisions/ad-004.md +37 -0
- package/docs/decisions/ad-005.md +36 -0
- package/docs/decisions/ad-006.md +49 -0
- package/docs/decisions/ad-007.md +36 -0
- package/docs/decisions/ad-008.md +54 -0
- package/docs/decisions/ad-009.md +61 -0
- package/docs/decisions/ad-010.md +45 -0
- package/docs/decisions/ad-011.md +59 -0
- package/docs/decisions/ad-012.md +71 -0
- package/docs/decisions/ad-013.md +87 -0
- package/docs/decisions/ad-014.md +56 -0
- package/docs/decisions/ad-015.md +33 -0
- package/docs/decisions/ad-016.md +98 -0
- package/docs/decisions/ad-017.md +65 -0
- package/docs/decisions/ad-018.md +77 -0
- package/docs/decisions/ad-019.md +75 -0
- package/docs/decisions/ad-020.md +88 -0
- package/docs/decisions/ad-021.md +57 -0
- package/docs/decisions/ad-022.md +120 -0
- package/docs/decisions/ad-023.md +87 -0
- package/docs/decisions/ad-024.md +85 -0
- package/docs/decisions/ad-025.md +92 -0
- package/docs/decisions/ad-026.md +131 -0
- package/docs/decisions/ad-027.md +90 -0
- package/docs/decisions/ad-028.md +90 -0
- package/docs/decisions/ad-029.md +73 -0
- package/docs/decisions/ad-030.md +97 -0
- package/docs/decisions/ad-031.md +94 -0
- package/docs/decisions/ad-032.md +83 -0
- package/docs/decisions/ad-033.md +89 -0
- package/docs/decisions/ad-034.md +86 -0
- package/docs/decisions/ad-035.md +86 -0
- package/docs/decisions/ad-036.md +68 -0
- package/docs/decisions/ad-037.md +47 -0
- package/docs/decisions/ad-038.md +52 -0
- package/docs/decisions/ad-039.md +69 -0
- package/docs/decisions/ad-040.md +89 -0
- package/docs/decisions/ad-041.md +98 -0
- package/docs/decisions/ad-042.md +82 -0
- package/docs/decisions/ad-043.md +79 -0
- package/docs/decisions/ad-044.md +61 -0
- package/docs/decisions/ad-045.md +94 -0
- package/docs/decisions/ad-046.md +111 -0
- package/docs/decisions/ad-047.md +96 -0
- package/docs/decisions/ad-048.md +85 -0
- package/docs/decisions/ad-049.md +66 -0
- package/docs/decisions/ad-050.md +94 -0
- package/docs/decisions/ad-051.md +69 -0
- package/docs/decisions/ad-052.md +69 -0
- package/docs/decisions/ad-053.md +78 -0
- package/docs/decisions/ad-054.md +98 -0
- package/docs/decisions/ad-055.md +74 -0
- package/docs/decisions/ad-056.md +85 -0
- package/docs/decisions/ad-057.md +68 -0
- package/docs/decisions/ad-058.md +97 -0
- package/docs/decisions/ad-059.md +82 -0
- package/docs/decisions/ad-060.md +75 -0
- package/docs/decisions/ad-061.md +68 -0
- package/docs/decisions/ad-062.md +72 -0
- package/docs/decisions/ad-063.md +84 -0
- package/docs/decisions/ad-064.md +79 -0
- package/docs/decisions/ad-065.md +81 -0
- package/docs/decisions/ad-066.md +111 -0
- package/docs/decisions/ad-067.md +64 -0
- package/docs/decisions/ad-068.md +79 -0
- package/docs/decisions/ad-069.md +74 -0
- package/docs/decisions/ad-070.md +86 -0
- package/docs/decisions/ad-071.md +93 -0
- package/docs/decisions/ad-072.md +82 -0
- package/docs/decisions/ad-073.md +102 -0
- package/docs/decisions/ad-074.md +91 -0
- package/docs/decisions/ad-075.md +79 -0
- package/docs/decisions/ad-076.md +102 -0
- package/docs/decisions/ad-077.md +94 -0
- package/docs/decisions/ad-078.md +84 -0
- package/docs/decisions/ad-079.md +73 -0
- package/docs/decisions/ad-080.md +86 -0
- package/docs/decisions/ad-081.md +70 -0
- package/docs/decisions/ad-082.md +79 -0
- package/docs/decisions/ad-083.md +88 -0
- package/docs/decisions/index.md +111 -0
- package/docs/diagnose.md +248 -0
- package/docs/index.md +52 -0
- package/docs/init.md +101 -0
- package/docs/lessons.md +294 -0
- package/docs/log.md +139 -0
- package/docs/measure.md +178 -0
- package/docs/providers/claude-code.md +118 -0
- package/docs/providers/cursor.md +110 -0
- package/docs/providers/index.md +71 -0
- package/docs/troubleshooting.md +138 -0
- package/model-aliases.json +12 -0
- package/model-prices.cursor.json +410 -0
- package/model-prices.json +1 -0
- package/package.json +57 -0
- package/skills/harness-init/SKILL.md +241 -0
- package/skills/harness-init/evals/trigger_evals.json +86 -0
- package/skills/harness-init/references/capabilities.md +113 -0
- package/src/contracts/capabilities.ts +30 -0
- package/src/contracts/decision.ts +21 -0
- package/src/contracts/effort.ts +15 -0
- package/src/contracts/harness-event.ts +87 -0
- package/src/contracts/index.ts +6 -0
- package/src/contracts/policy-defaults.ts +14 -0
- package/src/contracts/wiring.ts +20 -0
- package/src/core/attest/attest.service.ts +0 -0
- package/src/core/capability/capability.service.ts +62 -0
- package/src/core/capability/capability.store.ts +51 -0
- package/src/core/capability/capability.types.ts +56 -0
- package/src/core/comment-policy/comment-policy.service.ts +307 -0
- package/src/core/comment-policy/comment-policy.types.ts +6 -0
- package/src/core/comment-policy/comment-resolvability.ts +134 -0
- package/src/core/comment-policy/comment-syntax.catalog.ts +310 -0
- package/src/core/comment-policy/comment-syntax.store.ts +65 -0
- package/src/core/comment-policy/comment-syntax.types.ts +24 -0
- package/src/core/core.facade.ts +461 -0
- package/src/core/duplication/duplication.service.ts +219 -0
- package/src/core/floor/floor.catalog.ts +54 -0
- package/src/core/floor/floor.paths.ts +90 -0
- package/src/core/floor/floor.policy-surface.ts +311 -0
- package/src/core/floor/floor.service.ts +306 -0
- package/src/core/floor/floor.tokenize.ts +179 -0
- package/src/core/floor/floor.verb.ts +28 -0
- package/src/core/floor/index.ts +1 -0
- package/src/core/gate/gate.artifact.ts +157 -0
- package/src/core/gate/gate.command.ts +130 -0
- package/src/core/gate/gate.findings.ts +201 -0
- package/src/core/gate/gate.inputs.ts +0 -0
- package/src/core/gate/gate.lock.ts +283 -0
- package/src/core/gate/gate.service.ts +28 -0
- package/src/core/gate/gate.types.ts +67 -0
- package/src/core/handoff/handoff.service.ts +43 -0
- package/src/core/handoff/handoff.store.ts +59 -0
- package/src/core/handoff/handoff.types.ts +89 -0
- package/src/core/index.ts +52 -0
- package/src/core/integrity/state-seal.ts +81 -0
- package/src/core/lesson/lesson.authored.ts +96 -0
- package/src/core/lesson/lesson.credit.ts +51 -0
- package/src/core/lesson/lesson.garden.ts +283 -0
- package/src/core/lesson/lesson.link.ts +93 -0
- package/src/core/lesson/lesson.score.ts +61 -0
- package/src/core/lesson/lesson.select.ts +209 -0
- package/src/core/lesson/lesson.service.ts +110 -0
- package/src/core/lesson/lesson.store.ts +317 -0
- package/src/core/lesson/lesson.sync.ts +70 -0
- package/src/core/lesson/lesson.types.ts +92 -0
- package/src/core/lesson/lesson.validity.ts +44 -0
- package/src/core/observability/observability.report.ts +326 -0
- package/src/core/observability/observability.service.ts +328 -0
- package/src/core/observability/observability.store.ts +269 -0
- package/src/core/observability/observability.types.ts +193 -0
- package/src/core/observability/observability.why.ts +206 -0
- package/src/core/observe/observe.service.ts +76 -0
- package/src/core/plan/plan.detect.ts +47 -0
- package/src/core/plan/plan.service.ts +81 -0
- package/src/core/plan/plan.types.ts +25 -0
- package/src/core/policy/policy.defaults.ts +103 -0
- package/src/core/policy/policy.guard.ts +34 -0
- package/src/core/policy/policy.integrity.ts +238 -0
- package/src/core/policy/policy.loader.ts +118 -0
- package/src/core/policy/policy.operator.ts +82 -0
- package/src/core/policy/policy.posture.ts +79 -0
- package/src/core/policy/policy.rails.ts +45 -0
- package/src/core/policy/policy.types.ts +157 -0
- package/src/core/presence/presence.service.ts +111 -0
- package/src/core/presence/presence.store.ts +59 -0
- package/src/core/presence/presence.types.ts +9 -0
- package/src/core/release/release.decisions.ts +147 -0
- package/src/core/release/release.seen.ts +38 -0
- package/src/core/shell-policy/shell-policy.service.ts +215 -0
- package/src/core/shell-policy/shell-policy.stall.ts +57 -0
- package/src/core/shell-policy/shell-policy.types.ts +26 -0
- package/src/core/ship/ship.ledger.ts +76 -0
- package/src/core/ship/ship.service.ts +137 -0
- package/src/core/ship/ship.types.ts +19 -0
- package/src/core/stagnation/stagnation.resolution.ts +87 -0
- package/src/core/stagnation/stagnation.service.ts +20 -0
- package/src/core/stagnation/stagnation.store.ts +49 -0
- package/src/core/stagnation/stagnation.types.ts +12 -0
- package/src/core/subagent-policy/subagent-policy.parent-model.ts +202 -0
- package/src/core/subagent-policy/subagent-policy.service.ts +139 -0
- package/src/core/subagent-policy/subagent-policy.types.ts +22 -0
- package/src/core/supply-chain/supply-chain.catalog.ts +51 -0
- package/src/core/supply-chain/supply-chain.service.ts +217 -0
- package/src/core/turn/turn.activity.ts +94 -0
- package/src/core/turn/turn.autopilot.ts +131 -0
- package/src/core/turn/turn.failure-signals.ts +193 -0
- package/src/core/turn/turn.loop-counter.ts +71 -0
- package/src/core/turn/turn.types.ts +14 -0
- package/src/core/untrusted/untrusted.detect.ts +53 -0
- package/src/core/untrusted/untrusted.recall.ts +87 -0
- package/src/core/untrusted/untrusted.service.ts +129 -0
- package/src/core/untrusted/untrusted.store.ts +75 -0
- package/src/core/untrusted/untrusted.types.ts +42 -0
- package/src/entrypoints/compact-before.ts +14 -0
- package/src/entrypoints/prompt-submit.ts +25 -0
- package/src/entrypoints/response-after.ts +60 -0
- package/src/entrypoints/run.ts +179 -0
- package/src/entrypoints/session-end.ts +58 -0
- package/src/entrypoints/session-start.ts +172 -0
- package/src/entrypoints/shim.ts +46 -0
- package/src/entrypoints/stop.ts +876 -0
- package/src/entrypoints/subagent-start.ts +16 -0
- package/src/entrypoints/subagent-stop.ts +32 -0
- package/src/entrypoints/support.ts +182 -0
- package/src/entrypoints/tool-after.ts +111 -0
- package/src/entrypoints/tool-before.ts +186 -0
- package/src/entrypoints/tool-failure.ts +26 -0
- package/src/platform/backoff.ts +49 -0
- package/src/platform/cli-output.ts +30 -0
- package/src/platform/env-scope.ts +22 -0
- package/src/platform/fs-atomic.ts +142 -0
- package/src/platform/fs-jsonl.ts +24 -0
- package/src/platform/git.ts +187 -0
- package/src/platform/paths.ts +80 -0
- package/src/platform/pricing.ts +229 -0
- package/src/platform/process.ts +64 -0
- package/src/platform/sanitize.ts +23 -0
- package/src/platform/screen.ts +46 -0
- package/src/platform/style.ts +103 -0
- package/src/providers/claude/claude.capabilities.ts +25 -0
- package/src/providers/claude/claude.detect.ts +14 -0
- package/src/providers/claude/claude.inbound.ts +252 -0
- package/src/providers/claude/claude.lessons-view.ts +33 -0
- package/src/providers/claude/claude.outbound.ts +65 -0
- package/src/providers/claude/claude.policy-defaults.ts +9 -0
- package/src/providers/claude/claude.transcript.ts +62 -0
- package/src/providers/claude/claude.wiring.ts +245 -0
- package/src/providers/claude/index.ts +17 -0
- package/src/providers/cursor/cursor.capabilities.ts +28 -0
- package/src/providers/cursor/cursor.detect.ts +14 -0
- package/src/providers/cursor/cursor.inbound.ts +203 -0
- package/src/providers/cursor/cursor.lessons-view.ts +30 -0
- package/src/providers/cursor/cursor.outbound.ts +43 -0
- package/src/providers/cursor/cursor.policy-defaults.ts +9 -0
- package/src/providers/cursor/cursor.wiring.ts +214 -0
- package/src/providers/cursor/index.ts +17 -0
- package/src/providers/index.ts +9 -0
- package/src/providers/provider.degrade.ts +152 -0
- package/src/providers/provider.port.ts +20 -0
- package/src/providers/provider.registry.ts +28 -0
- package/tools/doctor.ts +631 -0
- package/tools/help-topic.ts +74 -0
- package/tools/init-project.ts +277 -0
- package/tools/install-runtime.ts +164 -0
- package/tools/lessons-cli.ts +474 -0
- package/tools/obs-cli.ts +175 -0
- package/tools/price-lookup.ts +57 -0
- package/tools/refresh-model-prices.ts +210 -0
- package/tools/test-env.mjs +28 -0
- package/tools/test-env.names.d.mts +1 -0
- package/tools/test-env.names.mjs +14 -0
- package/tools/uninstall-runtime.ts +463 -0
package/docs/lessons.md
ADDED
|
@@ -0,0 +1,294 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Concept
|
|
3
|
+
title: "Lessons"
|
|
4
|
+
description: "Durable, ranked lessons that keep the agent from repeating gate failures — three tiers, staleness against a named reference, a validity window, effectiveness measured after injection, lifecycle, config, ranking, and the per-provider rendered view."
|
|
5
|
+
tags: [concept, lessons, intelligence]
|
|
6
|
+
timestamp: "2026-08-04"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Lessons
|
|
10
|
+
|
|
11
|
+
Durable, compact lessons that keep the agent from repeating **gate failures**. Ranked inject under a char
|
|
12
|
+
budget — not conversational memory.
|
|
13
|
+
|
|
14
|
+
Four things decide whether a lesson reaches a turn: which **tier** it lives in, whether it is still **true**
|
|
15
|
+
(its refs resolve, its window is open), how it **ranks**, and whether the budget has room.
|
|
16
|
+
|
|
17
|
+
## The three tiers
|
|
18
|
+
|
|
19
|
+
| Tier | Where | Who reads it | Written by |
|
|
20
|
+
|------|-------|--------------|------------|
|
|
21
|
+
| `core` | inside the runtime | every install, identically | shipped — immutable |
|
|
22
|
+
| `global` | `<runtime home>/state/lessons.json` | every product on this machine | `lessons add --global`, `lessons promote` |
|
|
23
|
+
| `project` | `<repo>/.tlc/harness/state/lessons.json` | this repository only | gate stagnation, `lessons add` |
|
|
24
|
+
|
|
25
|
+
`<runtime home>` is `$TLC_HOME`, or `~/.tlc/harness` when it is unset.
|
|
26
|
+
|
|
27
|
+
**Choosing a tier.** Ask whether the lesson would still be true in a different product:
|
|
28
|
+
|
|
29
|
+
- "this repository's CI sets `TLC_HOME`" → **project**
|
|
30
|
+
- "run the gate itself, never an approximation of its steps" → **global**
|
|
31
|
+
|
|
32
|
+
Ranking reads all three; `projectBoost` keeps the project tier above the global tier, so local knowledge
|
|
33
|
+
outranks knowledge carried in. On a duplicate id the nearer tier wins, so a project that took a global lesson
|
|
34
|
+
and rewrote it reads its own version.
|
|
35
|
+
|
|
36
|
+
**Nothing crosses between products by itself.** There is no automatic promotion, by design: a lesson mined from
|
|
37
|
+
a gate failure in one product names that product's test runner and file layout, and no threshold can tell
|
|
38
|
+
whether it applies elsewhere. Only the operator can ([/decisions/ad-040.md](/decisions/ad-040.md)).
|
|
39
|
+
|
|
40
|
+
An injected lesson renders as `[gate/status/tier]`, so a turn reading `[test/active/global]` can tell the advice
|
|
41
|
+
was written about a different repository.
|
|
42
|
+
|
|
43
|
+
`source` is a different fact from `tier`: it records *how* the lesson was learned — `core`, `project` (mined
|
|
44
|
+
from a gate failure) or `manual` (authored) — not where it lives.
|
|
45
|
+
|
|
46
|
+
## Staleness — a lesson names what makes it true
|
|
47
|
+
|
|
48
|
+
A lesson may carry **refs**, each a repository-relative `path` or `path:symbol`. When a ref stops resolving the
|
|
49
|
+
lesson is withheld: a lesson naming a renamed file is worse than absent, because it sends the next turn looking
|
|
50
|
+
for something that no longer exists, with the authority of a lesson.
|
|
51
|
+
|
|
52
|
+
| Status | Meaning | Stale? |
|
|
53
|
+
|--------|---------|--------|
|
|
54
|
+
| `present` | the path exists, and the symbol appears in it | no |
|
|
55
|
+
| `path-missing` | the path is gone, or was absolute | **yes** |
|
|
56
|
+
| `symbol-missing` | the file survived, the name did not | **yes** |
|
|
57
|
+
| `unreadable` | the file could not be read | no — deferred |
|
|
58
|
+
|
|
59
|
+
`unreadable` is deliberately not stale. A file this process cannot open is not evidence the lesson stopped being
|
|
60
|
+
true. A lesson with **no refs is never stale** — most lessons are about conduct.
|
|
61
|
+
|
|
62
|
+
Refs are repository-relative; an absolute path never resolves, or a global lesson would report `present` in every
|
|
63
|
+
product on the machine that happens to contain the file.
|
|
64
|
+
|
|
65
|
+
`garden` sets and clears staleness for **project** lessons. A **global** lesson is judged per repository at
|
|
66
|
+
selection time instead — its refs may legitimately be missing here and present in the product it came from, so
|
|
67
|
+
one stored flag cannot be right for all of them ([/decisions/ad-036.md](/decisions/ad-036.md)).
|
|
68
|
+
|
|
69
|
+
## Validity window
|
|
70
|
+
|
|
71
|
+
`validFrom` / `validTo` (ISO) express knowledge with a known end — "pin the formatter until the toolchain moves".
|
|
72
|
+
Active when `(validFrom absent or ≤ now) and (validTo absent or > now)`.
|
|
73
|
+
|
|
74
|
+
An **unparseable bound withholds the lesson**. `--until "next tuesday"` is a typo, and treating a broken
|
|
75
|
+
declaration as no declaration would inject exactly what the author meant to limit. `garden` prunes an expired
|
|
76
|
+
lesson, because unlike a broken ref the end was declared by the author
|
|
77
|
+
([/decisions/ad-037.md](/decisions/ad-037.md)).
|
|
78
|
+
|
|
79
|
+
## Effectiveness — did the lesson help?
|
|
80
|
+
|
|
81
|
+
Ranking is built from proxies for usefulness. This is the measurement.
|
|
82
|
+
|
|
83
|
+
When lessons are injected on a retry for gate G, their ids and G go on the handoff. The **next run of gate G**
|
|
84
|
+
grades them: passed → `helped`, failed → `neutral`. The gate name is compared, so lessons injected for `lint` are
|
|
85
|
+
not graded by `test`.
|
|
86
|
+
|
|
87
|
+
| Reading | Meaning |
|
|
88
|
+
|---------|---------|
|
|
89
|
+
| `helped n/m` | present when that gate recovered, at least once |
|
|
90
|
+
| `neutral 0/m` | present at m failures and no recoveries |
|
|
91
|
+
| `unproven` | injected **for a gate** and never graded — no evidence, not "fine" |
|
|
92
|
+
| `session-only` | injected at session start and never for a gate, so this mechanism cannot measure it |
|
|
93
|
+
| `not-injected` | never shown yet |
|
|
94
|
+
|
|
95
|
+
Only `unproven` is a `doctor` warning. **A session-start injection is never graded** — only the retry path has a
|
|
96
|
+
gate whose next run can decide — so a lesson with gate `any` is unprovable by this mechanism rather than
|
|
97
|
+
unjustified, and saying otherwise would warn about a healthy store on every run
|
|
98
|
+
([/decisions/ad-044.md](/decisions/ad-044.md)).
|
|
99
|
+
|
|
100
|
+
The rate is `null` over zero graded injections, never `0` — zero would read as "measured and it never helped",
|
|
101
|
+
which is a claim the harness has not earned.
|
|
102
|
+
|
|
103
|
+
**This is not causal.** A gate passing after a lesson was injected does not prove the lesson caused it, and
|
|
104
|
+
`neutral` does not mean the lesson was wrong. A causal answer needs the same task run twice, and real work does
|
|
105
|
+
not repeat. The counters do not feed ranking, because boosting on a non-causal signal would make the ranking
|
|
106
|
+
self-confirming ([/decisions/ad-039.md](/decisions/ad-039.md)).
|
|
107
|
+
|
|
108
|
+
## Lifecycle
|
|
109
|
+
|
|
110
|
+
```text
|
|
111
|
+
gate stagnation (fingerprint ≥ 2)
|
|
112
|
+
→ upsert candidate lesson (project store), recording the session key
|
|
113
|
+
garden (sessionEnd / tlc harness lessons garden) — both writable tiers
|
|
114
|
+
→ promote candidates (distinct sessions ≥ promoteHitCount)
|
|
115
|
+
→ mark / clear staleness (project tier)
|
|
116
|
+
→ prune expired
|
|
117
|
+
→ decay / quarantine / prune
|
|
118
|
+
inject
|
|
119
|
+
→ sessionStart: active only, top N / maxChars
|
|
120
|
+
→ stop retry: active + matching candidates, gate-scoped; records a pending credit
|
|
121
|
+
grade
|
|
122
|
+
→ next run of the same gate: helped / neutral
|
|
123
|
+
optional
|
|
124
|
+
→ sync provider-native durable view (see Provider views below)
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## Config (`intelligence.lessons`)
|
|
128
|
+
|
|
129
|
+
| Key | Default | Meaning |
|
|
130
|
+
|-----|---------|---------|
|
|
131
|
+
| `enabled` | false | Master switch |
|
|
132
|
+
| `maxInjectSession` | 5 | Cap at sessionStart |
|
|
133
|
+
| `maxInjectRetry` | 8 | Cap on stop follow-up |
|
|
134
|
+
| `maxCharsSession` | 900 | Char budget session |
|
|
135
|
+
| `maxCharsRetry` | 1400 | Char budget retry |
|
|
136
|
+
| `promoteHitCount` | 2 | Candidate → active, counted in **distinct sessions** |
|
|
137
|
+
| `decayLambda` | 0.02 | Exponential decay per hour since the failure last **recurred** (`lastSeenAt`) |
|
|
138
|
+
| `projectBoost` | 1.5 | Score multiplier for the **project** tier |
|
|
139
|
+
| `syncRulesFile` | `auto` | Write the provider-native durable view: `auto` where the provider does not deliver hook context, `always`, or `never` |
|
|
140
|
+
| `gardenOnSessionEnd` | true | Garden on sessionEnd |
|
|
141
|
+
|
|
142
|
+
## Ranking
|
|
143
|
+
|
|
144
|
+
`score = relevance(gate, tokens) × confidence × exp(-λ · hours since lastSeenAt) × projectBoost?`
|
|
145
|
+
|
|
146
|
+
**Hours are counted from recurrence, never from exposure.** `lastSeenAt` moves only when the failure
|
|
147
|
+
signature happens again; `lastAccessedAt` moves when a lesson is *shown* and is telemetry only. Decay and
|
|
148
|
+
pruning both read `lastSeenAt` — reading the exposure field made relevance self-fulfilling, so a lesson that
|
|
149
|
+
merely matched a gate name kept resetting its own clock and never faded
|
|
150
|
+
([/decisions/ad-023.md](/decisions/ad-023.md)).
|
|
151
|
+
|
|
152
|
+
**Promotion counts distinct sessions, not `hitCount`.** `hitCount` counts recurrences of the same gate
|
|
153
|
+
fingerprint, and one stuck session produces those by definition — the stagnation rail exists because sessions
|
|
154
|
+
repeat themselves. A record written before session keys existed falls back to `hitCount` so it can still promote
|
|
155
|
+
([/decisions/ad-038.md](/decisions/ad-038.md)).
|
|
156
|
+
|
|
157
|
+
A lesson also retires when its cause is gone: the garden prunes a `verification` lesson whose stored signal is
|
|
158
|
+
an unresolved gate command, because AD-021 made that class classify as `config` and it can no longer recur.
|
|
159
|
+
|
|
160
|
+
Pack **whole lesson blocks** under the char budget. Never mid-string `slice` a lesson. When the budget is
|
|
161
|
+
full, omit lower-ranked lessons entirely and append `_(N more active lessons omitted under char budget)_`.
|
|
162
|
+
|
|
163
|
+
Session inject stops at the first lesson that does not fit (no filler with lower-ranked leftovers).
|
|
164
|
+
Quarantine never injects. Stale and out-of-window lessons are excluded **before** ranking, not scored low.
|
|
165
|
+
|
|
166
|
+
**The budget usually binds, and says so.** `maxCharsSession` defaults to 900 and a rendered block is four lines,
|
|
167
|
+
so about two fit while `maxInjectSession` says five. The injected block names what it dropped:
|
|
168
|
+
|
|
169
|
+
```text
|
|
170
|
+
(3 more eligible lessons omitted under the char budget — raise maxCharsSession to see them)
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
## Pinning a standing rule
|
|
174
|
+
|
|
175
|
+
Ranking weighs recurrence, decay and gate match — all estimates about a lesson the harness *inferred*. An
|
|
176
|
+
instruction the operator wrote deliberately has no recurrence to accumulate, so making it compete on score is a
|
|
177
|
+
category error: it loses to a shipped seed and never arrives.
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
tlc harness lessons add "Never declare done without an end-to-end run pasted into the reply." --global --pin
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
A pinned lesson is placed **before every scored lesson**, in store order. Everything else still binds — staleness,
|
|
184
|
+
the validity window, mode filtering and the char budget — so pinning changes order, not eligibility, and a pinned
|
|
185
|
+
rule naming a renamed file is withheld like any other. Nothing caps how many can be pinned; the budget bounds
|
|
186
|
+
delivery and `lessons list` marks each one `PINNED` ([/decisions/ad-043.md](/decisions/ad-043.md)).
|
|
187
|
+
|
|
188
|
+
## Provider views
|
|
189
|
+
|
|
190
|
+
`.tlc/harness/lessons.md` is the source of truth, and the store, the ranking, the budget and the rendered text are
|
|
191
|
+
shared by every provider. What differs is **transport** — how the text reaches the model — and that is a declared
|
|
192
|
+
capability rather than a preference ([/decisions/ad-050.md](/decisions/ad-050.md)):
|
|
193
|
+
|
|
194
|
+
| Provider | `sessionStartContextReliable` | Rendered view |
|
|
195
|
+
| --- | --- | --- |
|
|
196
|
+
| Cursor | `false` — `additional_context` returned from `sessionStart` is accepted, logged as merged, and dropped; acknowledged by Cursor as a race between the hook and the composer handle (forum 158452, 2026-04-20; reported again against 3.14.7 on 2026-08-02) | `.cursor/rules/harness-lessons.mdc` (`alwaysApply: true`) — the durable route, and on this host the only reliable one |
|
|
197
|
+
| Claude Code | `true` — `SessionStart` delivers `hookSpecificOutput.additionalContext` | a single `@.tlc/harness/lessons.md` import line appended to `CLAUDE.md`, under `always` |
|
|
198
|
+
|
|
199
|
+
Under the default `auto`, the view is written where the provider does not deliver hook context and withheld where it
|
|
200
|
+
does. `always` writes it everywhere; `never` writes it nowhere, which is how an operator declines a file in their
|
|
201
|
+
repo. A config carrying the field's old boolean still works — `true` reads as `always`, `false` as `never` — and
|
|
202
|
+
`tlc harness lessons list` names the coercion.
|
|
203
|
+
|
|
204
|
+
The view carries all three tiers and only what would actually be injected, so a withheld lesson never appears there
|
|
205
|
+
and a core or global lesson is not missing from it. It is written at session start as well as session end, because a
|
|
206
|
+
transport that is one session behind carries the previous session's guidance.
|
|
207
|
+
|
|
208
|
+
`tlc harness obs report` says which of the two is paid: the emission where the host delivers it, the rules file where
|
|
209
|
+
it does not.
|
|
210
|
+
|
|
211
|
+
## Design notes
|
|
212
|
+
|
|
213
|
+
| Insight | Applied here |
|
|
214
|
+
|---------|--------------|
|
|
215
|
+
| Lessons are atoms | Whole-block pack in `packLessonsUnderBudget` / the provider-view renderers |
|
|
216
|
+
| Rank before cut | Inject by `rankScore`; sync by priority → hitCount → confidence |
|
|
217
|
+
| Promote on repeat across sessions | distinct `sessionKeys` ≥ `promoteHitCount` |
|
|
218
|
+
| A claim outlives its subject | Refs + staleness, withheld not deleted |
|
|
219
|
+
| Some knowledge expires | Validity window, pruned on expiry |
|
|
220
|
+
| Usefulness is measured, not assumed | helped / neutral / unproven after the next gate run |
|
|
221
|
+
| Knowledge travels, context does not | Three tiers, operator-invoked promotion, tier in the rendered line |
|
|
222
|
+
| Grounded only | Gate stagnation / failures / an author — not chat memory |
|
|
223
|
+
| Noise control | Cap N + maxChars; omit note instead of half-sentences; garden decay/quarantine |
|
|
224
|
+
|
|
225
|
+
## CLI
|
|
226
|
+
|
|
227
|
+
```bash
|
|
228
|
+
tlc harness lessons add "<instruction>" [--gate <name>] [--avoid "..."] [--prefer "..."] \
|
|
229
|
+
[--tokens a,b] [--ref path[:symbol]] [--until <iso>] [--global] [--pin]
|
|
230
|
+
tlc harness lessons promote <id> # copy a project lesson into the global tier
|
|
231
|
+
tlc harness lessons list [--all] [--json]
|
|
232
|
+
tlc harness lessons show <id>
|
|
233
|
+
tlc harness lessons garden
|
|
234
|
+
tlc harness lessons sync-rules
|
|
235
|
+
tlc harness lessons path
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
`--ref` may repeat. `list` marks a withheld lesson `WITHHELD` and reports `effect=`, `validity=`, `stale=` and
|
|
239
|
+
`refs=` per lesson.
|
|
240
|
+
|
|
241
|
+
**Two different counts at the end, on purpose.** The tier line counts what would be *injected*, after the nearer
|
|
242
|
+
tier wins a duplicate id. The store lines count what is *on disk*. A promoted lesson lives in both stores and
|
|
243
|
+
resolves to the project copy, so without the second pair an operator reads `core=6 project=5` and concludes
|
|
244
|
+
`promote` did nothing:
|
|
245
|
+
|
|
246
|
+
```text
|
|
247
|
+
11 lessons — core=6 project=5
|
|
248
|
+
stale=0 out-of-window=0 unproven=0 not-injected=11
|
|
249
|
+
project store: <repo>/.tlc/harness/state/lessons.json (5 lessons)
|
|
250
|
+
global store: ~/.tlc/harness/state/lessons.json (5 lessons, 5 also in this project)
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
`doctor` reports stale, out-of-window and unproven lessons as separate warnings, and stays silent when the
|
|
254
|
+
capability is off or the writable tiers are empty. When everything is healthy it prints one row:
|
|
255
|
+
|
|
256
|
+
```text
|
|
257
|
+
OK lesson health — 10 lessons across the writable tiers, none stale, none out of window
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
## When the synced file looks empty
|
|
261
|
+
|
|
262
|
+
The synced markdown is a rendering of the store, so an empty one has a reason and now names it
|
|
263
|
+
([/decisions/ad-049.md](/decisions/ad-049.md)):
|
|
264
|
+
|
|
265
|
+
| What the file says | What is happening |
|
|
266
|
+
| --- | --- |
|
|
267
|
+
| `switched off … intelligence.lessons.enabled is false` | the capability is off, so no gate failure is ever recorded — and off is the default |
|
|
268
|
+
| `No lesson recorded yet` | on, but no failure has repeated inside a session yet |
|
|
269
|
+
| `N candidate lessons recorded, none promoted` | recorded, waiting for the same failure in `promoteHitCount` distinct sessions |
|
|
270
|
+
| `N active lesson is withheld` | promoted, but a ref stopped resolving or a window closed |
|
|
271
|
+
|
|
272
|
+
Three hurdles stand between a gate failing and that file showing anything: the capability has to be on, the *same*
|
|
273
|
+
failure has to repeat inside a session, and the candidate has to be promoted across distinct sessions. One sentence
|
|
274
|
+
covered all of them, which reads as broken.
|
|
275
|
+
|
|
276
|
+
**`.specs/LESSONS.md` is not this file.** It belongs to the `/sdd` skill, which keeps its own lessons layer. The
|
|
277
|
+
harness writes one file and nothing else.
|
|
278
|
+
|
|
279
|
+
## Reading the store from a test
|
|
280
|
+
|
|
281
|
+
The suite runs with `TLC_HOME` pointed at an empty temporary directory, so no test can read the lessons an
|
|
282
|
+
operator happened to promote on that machine. A test that needs the real runtime home sets it explicitly
|
|
283
|
+
([/decisions/ad-042.md](/decisions/ad-042.md)).
|
|
284
|
+
|
|
285
|
+
## Trade-offs
|
|
286
|
+
|
|
287
|
+
| Benefit | Cost |
|
|
288
|
+
|---------|------|
|
|
289
|
+
| Stops repeating the same gate mistake across sessions | Uses context tokens |
|
|
290
|
+
| Gate-scoped + decay stays relevant | Needs enable + occasional garden |
|
|
291
|
+
| A lesson retires when its subject is renamed | Refs are author-supplied; substring matching accepts a false `present` |
|
|
292
|
+
| Knowledge follows the operator across products | A global lesson can be irrelevant in some product; refs and the boost bound it |
|
|
293
|
+
| The store can be defended with numbers | The grading is correlational, not causal |
|
|
294
|
+
| Provider-view sync survives hook races | Can dirty the provider's own rules/memory file if enabled |
|
package/docs/log.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Aggregate
|
|
3
|
+
title: "Documentation log"
|
|
4
|
+
description: "Chronological, ISO 8601 record of every architectural decision, grouped by the date it was taken. Generated from docs/decisions/."
|
|
5
|
+
tags: [log, history, okf]
|
|
6
|
+
timestamp: "2026-08-12"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Log
|
|
10
|
+
|
|
11
|
+
Generated from `docs/decisions/` — do not edit by hand. Run `node tools/render-log.ts`.
|
|
12
|
+
|
|
13
|
+
A reserved file of the [OKF v0.1](/decisions/ad-013.md) bundle: entries grouped under ISO 8601 headings,
|
|
14
|
+
newest first. For what landed in which npm release, see `CHANGELOG.md` at the repository root.
|
|
15
|
+
|
|
16
|
+
## 2026-08-19
|
|
17
|
+
|
|
18
|
+
- **AD-081** — The manifest npm publishes is checked here, because the release runner was the only thing reading it ([/decisions/ad-081.md](/decisions/ad-081.md))
|
|
19
|
+
- **AD-082** — npm is the documented install, and the first version cannot come from CI ([/decisions/ad-082.md](/decisions/ad-082.md))
|
|
20
|
+
- **AD-083** — Going public switched the branch ruleset on, and an unattended release needs the App to bypass it ([/decisions/ad-083.md](/decisions/ad-083.md))
|
|
21
|
+
|
|
22
|
+
## 2026-08-17
|
|
23
|
+
|
|
24
|
+
- **AD-073** — A neighbour mid-gate is not a reason to block a turn ([/decisions/ad-073.md](/decisions/ad-073.md))
|
|
25
|
+
- **AD-074** — Code the gate cannot read is refused, and a credential is not always a file ([/decisions/ad-074.md](/decisions/ad-074.md))
|
|
26
|
+
- **AD-075** — A dependency a turn adds outlives the turn, so two mechanical failures are worth a stop ([/decisions/ad-075.md](/decisions/ad-075.md))
|
|
27
|
+
- **AD-076** — Goal-hijack enforcement needs the tool's output, and the host does not deliver it ([/decisions/ad-076.md](/decisions/ad-076.md))
|
|
28
|
+
- **AD-077** — A command that appears verbatim in fetched content is put to the operator ([/decisions/ad-077.md](/decisions/ad-077.md))
|
|
29
|
+
- **AD-078** — The two files the harness reads aloud are sealed on write and verified on injection ([/decisions/ad-078.md](/decisions/ad-078.md))
|
|
30
|
+
- **AD-079** — The coverage claim is a generated page with its gaps in it, not a badge in the README ([/decisions/ad-079.md](/decisions/ad-079.md))
|
|
31
|
+
- **AD-080** — Every hand-written list of our own rules is now checked, including the one that drifted while writing this ([/decisions/ad-080.md](/decisions/ad-080.md))
|
|
32
|
+
|
|
33
|
+
## 2026-08-13
|
|
34
|
+
|
|
35
|
+
- **AD-068** — A directory decides what ships, and dist is derived from disk in both directions ([/decisions/ad-068.md](/decisions/ad-068.md))
|
|
36
|
+
- **AD-069** — A decision record declares its shape, and cites by link so a move cannot break it ([/decisions/ad-069.md](/decisions/ad-069.md))
|
|
37
|
+
- **AD-070** — A comment has to read for somebody who was not in the session ([/decisions/ad-070.md](/decisions/ad-070.md))
|
|
38
|
+
- **AD-071** — The turn's added lines are checked against the code the project already has ([/decisions/ad-071.md](/decisions/ad-071.md))
|
|
39
|
+
- **AD-072** — A record can leave the corpus, and removing is a change worth recording ([/decisions/ad-072.md](/decisions/ad-072.md))
|
|
40
|
+
|
|
41
|
+
## 2026-08-12
|
|
42
|
+
|
|
43
|
+
- **AD-060** — The gate records the environment it ran under, and names it only once the cheap explanations are spent ([/decisions/ad-060.md](/decisions/ad-060.md))
|
|
44
|
+
- **AD-061** — A decision that refuses names its rule, and degrading preserves it ([/decisions/ad-061.md](/decisions/ad-061.md))
|
|
45
|
+
- **AD-062** — One command answers whether the harness did that, and says so plainly when it did not ([/decisions/ad-062.md](/decisions/ad-062.md))
|
|
46
|
+
- **AD-063** — One palette for human output, and a checker keeping it out of everything else ([/decisions/ad-063.md](/decisions/ad-063.md))
|
|
47
|
+
- **AD-064** — A snapshot is assigned, never accumulated, and a table does not list what it cannot count ([/decisions/ad-064.md](/decisions/ad-064.md))
|
|
48
|
+
- **AD-065** — The obs bus has a contract, and the gate checks both sides of it ([/decisions/ad-065.md](/decisions/ad-065.md))
|
|
49
|
+
- **AD-066** — Uninstall reads the artefact, and the plan is the confirmation ([/decisions/ad-066.md](/decisions/ad-066.md))
|
|
50
|
+
- **AD-067** — A reserved file that cannot be retired is rendered, and the gate holds it there ([/decisions/ad-067.md](/decisions/ad-067.md))
|
|
51
|
+
|
|
52
|
+
## 2026-08-10
|
|
53
|
+
|
|
54
|
+
- **AD-058** — Gates diff against the turn's base, and language knowledge is one table ([/decisions/ad-058.md](/decisions/ad-058.md))
|
|
55
|
+
- **AD-059** — A rail never writes a field it reads, and a counter reads the plane its events land on ([/decisions/ad-059.md](/decisions/ad-059.md))
|
|
56
|
+
|
|
57
|
+
## 2026-08-07
|
|
58
|
+
|
|
59
|
+
- **AD-054** — npm is the distribution, and a merged release PR is the only thing that publishes ([/decisions/ad-054.md](/decisions/ad-054.md))
|
|
60
|
+
- **AD-055** — The changelog is rendered from the decision records, and git says which release each landed in ([/decisions/ad-055.md](/decisions/ad-055.md))
|
|
61
|
+
- **AD-056** — The package delivers the runtime, and the runtime path stays where hooks already point ([/decisions/ad-056.md](/decisions/ad-056.md))
|
|
62
|
+
- **AD-057** — The release PR is a mechanism, not a gate, and the bot merges it ([/decisions/ad-057.md](/decisions/ad-057.md))
|
|
63
|
+
|
|
64
|
+
## 2026-08-06
|
|
65
|
+
|
|
66
|
+
- **AD-053** — The harness ships no model allowlist, and a list that names nothing enforces nothing ([/decisions/ad-053.md](/decisions/ad-053.md))
|
|
67
|
+
|
|
68
|
+
## 2026-08-05
|
|
69
|
+
|
|
70
|
+
- **AD-045** — A gate verdict is reused when the content hash of its inputs did not change ([/decisions/ad-045.md](/decisions/ad-045.md))
|
|
71
|
+
- **AD-046** — The runtime path is an artifact, and update never touches what it does not own ([/decisions/ad-046.md](/decisions/ad-046.md))
|
|
72
|
+
- **AD-047** — An instruction is not an affordance, and a refusal names the route that works ([/decisions/ad-047.md](/decisions/ad-047.md))
|
|
73
|
+
- **AD-048** — A broken updater cannot deliver its own fix, so the installer is the recovery route ([/decisions/ad-048.md](/decisions/ad-048.md))
|
|
74
|
+
- **AD-049** — An empty synced file says which of four reasons made it empty ([/decisions/ad-049.md](/decisions/ad-049.md))
|
|
75
|
+
- **AD-050** — Lesson transport is a provider capability, not an operator preference ([/decisions/ad-050.md](/decisions/ad-050.md))
|
|
76
|
+
- **AD-051** — A warning fails the gate, and a suppression states what breaks without it ([/decisions/ad-051.md](/decisions/ad-051.md))
|
|
77
|
+
- **AD-052** — The repository moves to the org and the runtime paths do not ([/decisions/ad-052.md](/decisions/ad-052.md))
|
|
78
|
+
|
|
79
|
+
## 2026-08-04
|
|
80
|
+
|
|
81
|
+
- **AD-024** — The gate tells the truth about its own environment, its own lock, and the cause of a failure ([/decisions/ad-024.md](/decisions/ad-024.md))
|
|
82
|
+
- **AD-025** — Posture governs surfacing only, and each posture has exactly one name ([/decisions/ad-025.md](/decisions/ad-025.md))
|
|
83
|
+
- **AD-026** — An interruption is worth what it costs: narrower asks, a deadline on questions, and a rate the operator can see ([/decisions/ad-026.md](/decisions/ad-026.md))
|
|
84
|
+
- **AD-027** — Evidence is ordered against the code, every rail's firing is recorded, and a checker can run with its rule off ([/decisions/ad-027.md](/decisions/ad-027.md))
|
|
85
|
+
- **AD-028** — A resolved failure is kept, a session attests to itself, and provider neutrality becomes a proof ([/decisions/ad-028.md](/decisions/ad-028.md))
|
|
86
|
+
- **AD-029** — A capability the init skill cannot correctly initialise is not shipped ([/decisions/ad-029.md](/decisions/ad-029.md))
|
|
87
|
+
- **AD-030** — Clearing a policy divergence is one operator command behind four independent locks, and a refusal never points an agent at a door the floor holds shut ([/decisions/ad-030.md](/decisions/ad-030.md))
|
|
88
|
+
- **AD-031** — The decisions are the changelog, a breaking change carries its own instruction, and looking never changes anything ([/decisions/ad-031.md](/decisions/ad-031.md))
|
|
89
|
+
- **AD-032** — A hook is healthy when it can run, not when a marker string is present ([/decisions/ad-032.md](/decisions/ad-032.md))
|
|
90
|
+
- **AD-033** — The gate says what it costs, appendFiles stops promising what it cannot deliver, and a dead capability leaves ([/decisions/ad-033.md](/decisions/ad-033.md))
|
|
91
|
+
- **AD-034** — A warning that fires on a healthy install is not a warning, and the author reads the operator's output before anyone else does ([/decisions/ad-034.md](/decisions/ad-034.md))
|
|
92
|
+
- **AD-035** — A lesson learned by reasoning can be written down, and the harness never learns where lessons come from ([/decisions/ad-035.md](/decisions/ad-035.md))
|
|
93
|
+
- **AD-036** — A lesson names what makes it true, and stops being injected when that is gone ([/decisions/ad-036.md](/decisions/ad-036.md))
|
|
94
|
+
- **AD-037** — A lesson can be true for a period, and an unparseable bound withholds it ([/decisions/ad-037.md](/decisions/ad-037.md))
|
|
95
|
+
- **AD-038** — Promotion counts distinct sessions, because one stuck session is one observation ([/decisions/ad-038.md](/decisions/ad-038.md))
|
|
96
|
+
- **AD-039** — A lesson is graded by the gate it was injected for, and unproven is not a passing reading ([/decisions/ad-039.md](/decisions/ad-039.md))
|
|
97
|
+
- **AD-040** — Three lesson tiers, and nothing crosses between products by itself ([/decisions/ad-040.md](/decisions/ad-040.md))
|
|
98
|
+
- **AD-041** — A member something reads and nothing writes fails the gate ([/decisions/ad-041.md](/decisions/ad-041.md))
|
|
99
|
+
- **AD-042** — The suite gets an empty runtime home, and one renderer renders a lesson ([/decisions/ad-042.md](/decisions/ad-042.md))
|
|
100
|
+
- **AD-043** — A standing rule is pinned, not ranked, and the budget says what it dropped ([/decisions/ad-043.md](/decisions/ad-043.md))
|
|
101
|
+
- **AD-044** — Only an injection a gate could grade can be unproven ([/decisions/ad-044.md](/decisions/ad-044.md))
|
|
102
|
+
|
|
103
|
+
## 2026-08-03
|
|
104
|
+
|
|
105
|
+
- **AD-023** — One finding per failure, and lesson relevance is recurrence rather than exposure ([/decisions/ad-023.md](/decisions/ad-023.md))
|
|
106
|
+
|
|
107
|
+
## 2026-07-31
|
|
108
|
+
|
|
109
|
+
- **AD-022** — The policy surface is a floor rule, detection sits behind interception, and the operator/agent line is structural ([/decisions/ad-022.md](/decisions/ad-022.md))
|
|
110
|
+
|
|
111
|
+
## 2026-07-30
|
|
112
|
+
|
|
113
|
+
- **AD-017** — The docs gate delegates to the project's tool, and the catalog is the only source of capability metadata ([/decisions/ad-017.md](/decisions/ad-017.md))
|
|
114
|
+
- **AD-018** — Three rails adopted from an external review, each off by default and declared rather than inferred ([/decisions/ad-018.md](/decisions/ad-018.md))
|
|
115
|
+
- **AD-019** — A resource is identified by what it resolves to, and a declared capability must be read where it matters ([/decisions/ad-019.md](/decisions/ad-019.md))
|
|
116
|
+
- **AD-020** — One resolution for the install path, one source for posture, and a config that only advertises what it reads ([/decisions/ad-020.md](/decisions/ad-020.md))
|
|
117
|
+
- **AD-021** — A gate command that never resolved is a config fault, and a recipe runner does not receive file arguments ([/decisions/ad-021.md](/decisions/ad-021.md))
|
|
118
|
+
|
|
119
|
+
## 2026-07-29
|
|
120
|
+
|
|
121
|
+
- **AD-002** — Provider-neutral naming and layout ([/decisions/ad-002.md](/decisions/ad-002.md))
|
|
122
|
+
- **AD-003** — No backward compatibility ([/decisions/ad-003.md](/decisions/ad-003.md))
|
|
123
|
+
- **AD-004** — Ports and adapters with an anti-corruption layer per provider ([/decisions/ad-004.md](/decisions/ad-004.md))
|
|
124
|
+
- **AD-005** — Local test runner is the gate; CI matrix runs on every push ([/decisions/ad-005.md](/decisions/ad-005.md))
|
|
125
|
+
- **AD-006** — Windows ships in scope, with CI covering the suite and the build ([/decisions/ad-006.md](/decisions/ad-006.md))
|
|
126
|
+
- **AD-007** — Vendor check applies to core tests; absence checks do not ([/decisions/ad-007.md](/decisions/ad-007.md))
|
|
127
|
+
- **AD-008** — Biome + TypeScript in the gate; @types/node pinned to the declared floor ([/decisions/ad-008.md](/decisions/ad-008.md))
|
|
128
|
+
- **AD-009** — Event kinds are provider-agnostic; capabilities are data, not flags ([/decisions/ad-009.md](/decisions/ad-009.md))
|
|
129
|
+
- **AD-010** — Shared vocabulary moves to src/contracts/ ([/decisions/ad-010.md](/decisions/ad-010.md))
|
|
130
|
+
- **AD-011** — Vendor-specific data belongs to the provider, not to core ([/decisions/ad-011.md](/decisions/ad-011.md))
|
|
131
|
+
- **AD-012** — Prefer Bun at runtime, keep dist/ for the Node fallback, ship no binary ([/decisions/ad-012.md](/decisions/ad-012.md))
|
|
132
|
+
- **AD-013** — Documentation follows the Open Knowledge Format (OKF v0.1) ([/decisions/ad-013.md](/decisions/ad-013.md))
|
|
133
|
+
- **AD-014** — Claude Code hook payload field paths, pinned ([/decisions/ad-014.md](/decisions/ad-014.md))
|
|
134
|
+
- **AD-015** — Wiring handler names are the entrypoint filenames ([/decisions/ad-015.md](/decisions/ad-015.md))
|
|
135
|
+
- **AD-016** — Field semantics, state writers, and the core export surface ([/decisions/ad-016.md](/decisions/ad-016.md))
|
|
136
|
+
|
|
137
|
+
## 2026-07-27
|
|
138
|
+
|
|
139
|
+
- **AD-001** — Optional parent-Fast sticky deny for Task spawns ([/decisions/ad-001.md](/decisions/ad-001.md))
|
package/docs/measure.md
ADDED
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Runbook
|
|
3
|
+
title: "Measure"
|
|
4
|
+
description: "How to read harness observability: status, live signal, raw signal, session reports, audit trail, price catalogs, and the on-disk project state files, all provider-tagged."
|
|
5
|
+
tags: [runbook, measure, observability, pricing]
|
|
6
|
+
timestamp: "2026-07-29"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Measure
|
|
10
|
+
|
|
11
|
+
Run from the project root, or set `TLC_PROJECT_DIR`.
|
|
12
|
+
|
|
13
|
+
## Status
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
tlc harness status
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Shows mode, grind on/off, and whether stop gates are paused.
|
|
20
|
+
|
|
21
|
+
## Live signal
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
tlc harness obs live
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Allowlisted tail of signal events (session, prompt, fails, denials, gates, compact, subagents, cost alerts).
|
|
28
|
+
|
|
29
|
+
## Raw signal
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
tlc harness obs events [n]
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Last N JSON lines from `obs.jsonl`.
|
|
36
|
+
|
|
37
|
+
## Session report
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
tlc harness obs report [conversation_id]
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Markdown rollup: tokens, estimated USD, tools, subagents, gates. Writes under
|
|
44
|
+
`.tlc/harness/state/reports/`.
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
tlc harness obs rollup <conversation_id>
|
|
48
|
+
tlc harness obs prune
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
`rollup` prints the raw JSON rollup for one session; `prune` deletes rollups older than
|
|
52
|
+
`retentionDays` (default 14).
|
|
53
|
+
|
|
54
|
+
## Machine-readable output
|
|
55
|
+
|
|
56
|
+
Every read command accepts `--json` and then writes exactly one JSON value to stdout and no prose, so a CI
|
|
57
|
+
step or an agent can parse it instead of scraping text:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
tlc harness status --json
|
|
61
|
+
tlc harness doctor --json # { ok, failed, warned, checks: [{ id, name, status, detail }] }
|
|
62
|
+
tlc harness obs live --json # { count, events: [...] }
|
|
63
|
+
tlc harness obs report --json # { session, path, rollup }
|
|
64
|
+
tlc harness lessons list --json
|
|
65
|
+
tlc harness prices lookup <model> --json
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Without the flag, output is byte-identical to what it has always been. Exit codes do not change either: a
|
|
69
|
+
failing `doctor --json` still exits 1 and still emits a parseable value, with `ok: false`, so a caller can
|
|
70
|
+
branch on the code or on the payload.
|
|
71
|
+
|
|
72
|
+
## Global spool
|
|
73
|
+
|
|
74
|
+
`obs.globalSpool` (off by default) mirrors every obs and audit record into a single file under the runtime
|
|
75
|
+
home — `~/.tlc/harness/state/obs-spool.jsonl` — wrapping each one with the repository path and project name:
|
|
76
|
+
|
|
77
|
+
```json
|
|
78
|
+
{"repo":"/work/my-repo","project":"my-repo","stream":"obs","record":{...}}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
That is what makes cost and gate history readable across every repository at once; per-repo files stay
|
|
82
|
+
authoritative and untouched. Writes are best-effort — an unwritable runtime home degrades to project-only
|
|
83
|
+
recording rather than failing a hook. `tlc harness obs prune` prunes the spool on the same retention window
|
|
84
|
+
as session rollups and reports how many records it dropped.
|
|
85
|
+
|
|
86
|
+
When the runtime home is itself a checkout reached through a symlink, the spool lands in that checkout's
|
|
87
|
+
`state/` directory, which `.gitignore` already covers for exactly this case.
|
|
88
|
+
|
|
89
|
+
## Retention
|
|
90
|
+
|
|
91
|
+
`tlc harness obs prune` reads `obs.retentionDays` from project policy (default 14) and applies it to session
|
|
92
|
+
rollups and to the spool, reporting how many spool records it dropped. The other tunables — `includePayloads`,
|
|
93
|
+
`maxAttrChars`, `sessionCostAlertUsd` — are described in [/concepts.md](/concepts.md).
|
|
94
|
+
|
|
95
|
+
## Observability planes
|
|
96
|
+
|
|
97
|
+
Every record carries a `provider` field (`"cursor" | "claude"`) so signal, debug, and audit records from a
|
|
98
|
+
multi-provider project stay attributable per-event, not just per-project (see
|
|
99
|
+
[/decisions/ad-011.md](/decisions/ad-011.md)).
|
|
100
|
+
|
|
101
|
+
| Plane | File | Default | Contents |
|
|
102
|
+
|-------|------|---------|----------|
|
|
103
|
+
| Signal | `.tlc/harness/state/obs.jsonl` | ON | lifecycle, fails, denials, gates, cost alerts, ship claims |
|
|
104
|
+
| Debug | `.tlc/harness/state/debug.jsonl` | OFF | happy-path tool/shell/mcp noise |
|
|
105
|
+
| Audit | `.tlc/harness/state/audit.jsonl` | ON | one record per hook invocation (`{ ts, event, payload }`), restored per [/decisions/ad-016.md](/decisions/ad-016.md) item 7 so a denied/asked shell command is never silently unaudited |
|
|
106
|
+
|
|
107
|
+
Set `"observability": { "debugEnabled": true }` in user or project config to also capture debug-level
|
|
108
|
+
events. `shell.end` is promoted from debug to signal automatically whenever the permission was not a plain
|
|
109
|
+
allow — an audited denial should never require opting into debug mode to see.
|
|
110
|
+
|
|
111
|
+
18 `HarnessEventKind` values map onto a smaller set of `ObsKind` values (`session.start`, `tool.start`,
|
|
112
|
+
`shell.end`, `gate.outcome`, `cost.turn`, …) — see `src/core/observability/observability.types.ts` for the
|
|
113
|
+
full mapping table.
|
|
114
|
+
|
|
115
|
+
## Prices
|
|
116
|
+
|
|
117
|
+
Cost estimates use on-disk catalogs under `~/.tlc/harness/`, resolved provider-first:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
tlc harness prices refresh
|
|
121
|
+
tlc harness prices refresh all
|
|
122
|
+
tlc harness prices refresh cursor
|
|
123
|
+
tlc harness prices refresh litellm
|
|
124
|
+
tlc harness prices lookup <model-id> [provider]
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
| Command | Effect |
|
|
128
|
+
|---------|--------|
|
|
129
|
+
| `refresh` / `refresh all` | Update the Cursor catalog and the LiteLLM fallback |
|
|
130
|
+
| `refresh cursor` | Write `model-prices.cursor.json` (commit when rates change) |
|
|
131
|
+
| `refresh litellm` | Write `model-prices.litellm.json` (gitignored; regenerate locally) |
|
|
132
|
+
| `lookup <model-id> [provider]` | Resolve catalog key, pool, and USD for 1M input + 1M output |
|
|
133
|
+
|
|
134
|
+
### Catalogs
|
|
135
|
+
|
|
136
|
+
| File | Role | In git |
|
|
137
|
+
|------|------|--------|
|
|
138
|
+
| `model-prices.<provider>.json` (e.g. `model-prices.cursor.json`) | Primary, per provider | Yes |
|
|
139
|
+
| `model-prices.litellm.json` | Fallback (LiteLLM public JSON) | No |
|
|
140
|
+
| `model-prices.json` | Local overrides | Empty `{}` template only |
|
|
141
|
+
| `model-aliases.json` | Model id → catalog key | Yes |
|
|
142
|
+
|
|
143
|
+
### Resolution order
|
|
144
|
+
|
|
145
|
+
1. `model-prices.json` (local overrides)
|
|
146
|
+
2. `model-prices.<provider>.json` (this provider's own catalog)
|
|
147
|
+
3. `model-prices.litellm.json`
|
|
148
|
+
4. otherwise `cost_usd: null`
|
|
149
|
+
|
|
150
|
+
Pools (neutral names in observability records; see [/decisions/ad-011.md](/decisions/ad-011.md) item 2):
|
|
151
|
+
`provider_native` | `other` | `auto` | `unknown`. The on-disk catalog files still use vendor-named pool
|
|
152
|
+
keys internally (`cursor_models`, `anthropic_models`, …) since pricing must name real vendors — those are
|
|
153
|
+
mapped to the neutral names before they reach `core/`.
|
|
154
|
+
|
|
155
|
+
### When to refresh
|
|
156
|
+
|
|
157
|
+
| Situation | Command |
|
|
158
|
+
|-----------|---------|
|
|
159
|
+
| A provider published new rates or models | `tlc harness prices refresh cursor` (then commit) |
|
|
160
|
+
| Missing LiteLLM file or obscure model | `tlc harness prices refresh litellm` |
|
|
161
|
+
| Update both catalogs | `tlc harness prices refresh` |
|
|
162
|
+
| Inspect one model | `tlc harness prices lookup <model-id> [provider]` |
|
|
163
|
+
|
|
164
|
+
`tlc harness doctor` requires at least one provider catalog to be present. LiteLLM is optional until needed
|
|
165
|
+
as fallback.
|
|
166
|
+
|
|
167
|
+
## Project state files
|
|
168
|
+
|
|
169
|
+
| Path | Contents |
|
|
170
|
+
|------|----------|
|
|
171
|
+
| `.tlc/harness/state/obs.jsonl` | Signal |
|
|
172
|
+
| `.tlc/harness/state/debug.jsonl` | Debug (if enabled) |
|
|
173
|
+
| `.tlc/harness/state/audit.jsonl` | Verbose per-hook audit trail |
|
|
174
|
+
| `.tlc/harness/state/sessions/*.json` | Per-conversation rollups |
|
|
175
|
+
| `.tlc/harness/state/handoff.json` | Cross-turn handoff |
|
|
176
|
+
| `.tlc/harness/state/ship-ledger.jsonl` | Ship claim / challenge / pass rows |
|
|
177
|
+
| `.tlc/harness/state/lessons.json` | Project lessons store |
|
|
178
|
+
| `.tlc/harness/state/parent-model.json` | Sticky parent-model snapshot (see [/decisions/ad-001.md](/decisions/ad-001.md)) |
|