@haaaiawd/loom 2.0.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -10
- package/CONTRIBUTING.md +37 -37
- package/EVIL_EVAL.md +112 -112
- package/README.md +209 -204
- package/README.zh-CN.md +171 -169
- package/SECURITY.md +11 -11
- package/cli/src/protocol.js +404 -360
- package/cli/src/store.js +624 -622
- package/design.md +198 -191
- package/docs/PROMPT_CATALOG.md +99 -92
- package/docs/RELEASE_CHECKLIST.md +53 -53
- package/docs/UX_FLOW.md +171 -171
- package/docs/brand/loom-mark.svg +18 -18
- package/docs/brand/loom-readme-header.svg +34 -34
- package/docs/brand/loom-readme-header.zh-CN.svg +29 -29
- package/docs/loom-eval-loop.drawio +21 -21
- package/docs/loom-eval-loop.svg +56 -56
- package/docs/loom-production-loop.drawio +41 -41
- package/docs/loom-production-loop.svg +92 -92
- package/package.json +52 -52
package/CHANGELOG.md
CHANGED
|
@@ -1,13 +1,18 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 2.0.
|
|
3
|
+
## 2.0.1
|
|
4
|
+
|
|
5
|
+
- Added an active-Task execution protocol that restores workspace-aware coding guidance, risk-based testing, restartable progress, criterion-level proof, and conditional commit/PR boundaries after context resets.
|
|
6
|
+
- Strengthened the generated `AGENTS.md` anchor so Agents reload LOOM at project entry, after context loss, and before resumed editing without invoking it before every tool call.
|
|
4
7
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
-
|
|
8
|
-
-
|
|
9
|
-
- Added
|
|
10
|
-
-
|
|
11
|
-
- Added
|
|
12
|
-
-
|
|
13
|
-
-
|
|
8
|
+
## 2.0.0
|
|
9
|
+
|
|
10
|
+
- Rebuilt LOOM as invisible Agent infrastructure around project understanding, a scalable design-document graph, separate professional-field dossiers, a broad Work Map, and one active Task.
|
|
11
|
+
- Replaced fixed roles and stages with an adaptive clarification protocol and explicit convergence conditions.
|
|
12
|
+
- Added human-editable project truth, structured continuity state, and superseding decision history.
|
|
13
|
+
- Added context selection that keeps very large Task maps on disk and injects only the active horizon.
|
|
14
|
+
- Added seven adaptable design-document templates and `loom prompts` as a complete live prompt inventory.
|
|
15
|
+
- Reduced Keeper to one independent build-readiness handoff with revision, changed-digest, fresh-run, and stale-state enforcement.
|
|
16
|
+
- Added controlled Evil Eval scaffolding for equal-condition, repeated, reset-heavy blind comparison.
|
|
17
|
+
- Added bilingual English/Chinese release documentation and a flat, minimal visual system with editable loop diagrams.
|
|
18
|
+
- Deliberately stopped automatic mutation of legacy `.loom/v1` projects.
|
package/CONTRIBUTING.md
CHANGED
|
@@ -1,37 +1,37 @@
|
|
|
1
|
-
# Contributing to LOOM
|
|
2
|
-
|
|
3
|
-
LOOM should become simpler when it changes. A contribution is strongest when it preserves the natural
|
|
4
|
-
human–Agent conversation, keeps project truth inspectable on disk, and proves its effect without adding
|
|
5
|
-
another ritual.
|
|
6
|
-
|
|
7
|
-
## Before changing code
|
|
8
|
-
|
|
9
|
-
1. Read [the system design](design.md) and [the UX loop](docs/UX_FLOW.md).
|
|
10
|
-
2. Run the current suite and CLI locally:
|
|
11
|
-
|
|
12
|
-
```bash
|
|
13
|
-
npm test
|
|
14
|
-
node cli/bin/loom.js --help
|
|
15
|
-
node cli/bin/loom.js prompts
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
3. Keep a change inside one observable problem. If a contract or stored shape changes, update its design,
|
|
19
|
-
prompt, help text, and tests together.
|
|
20
|
-
|
|
21
|
-
## Pull requests
|
|
22
|
-
|
|
23
|
-
A useful pull request explains:
|
|
24
|
-
|
|
25
|
-
- the user or Agent failure it fixes;
|
|
26
|
-
- the smallest complete change that fixes it;
|
|
27
|
-
- the exact command or artifact that proves the result;
|
|
28
|
-
- any compatibility, migration, or unverified boundary.
|
|
29
|
-
|
|
30
|
-
Prompt changes must remain visible through `loom prompts` and be reflected in
|
|
31
|
-
[`docs/PROMPT_CATALOG.md`](docs/PROMPT_CATALOG.md). Changes that claim better quality need a baseline-relative
|
|
32
|
-
proof; a nicer-looking output alone is not evidence that LOOM caused the improvement.
|
|
33
|
-
|
|
34
|
-
## Release checks
|
|
35
|
-
|
|
36
|
-
Run `npm test` and `npm pack --dry-run` before requesting release. See the
|
|
37
|
-
[release checklist](docs/RELEASE_CHECKLIST.md) for the maintainer flow.
|
|
1
|
+
# Contributing to LOOM
|
|
2
|
+
|
|
3
|
+
LOOM should become simpler when it changes. A contribution is strongest when it preserves the natural
|
|
4
|
+
human–Agent conversation, keeps project truth inspectable on disk, and proves its effect without adding
|
|
5
|
+
another ritual.
|
|
6
|
+
|
|
7
|
+
## Before changing code
|
|
8
|
+
|
|
9
|
+
1. Read [the system design](design.md) and [the UX loop](docs/UX_FLOW.md).
|
|
10
|
+
2. Run the current suite and CLI locally:
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
npm test
|
|
14
|
+
node cli/bin/loom.js --help
|
|
15
|
+
node cli/bin/loom.js prompts
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
3. Keep a change inside one observable problem. If a contract or stored shape changes, update its design,
|
|
19
|
+
prompt, help text, and tests together.
|
|
20
|
+
|
|
21
|
+
## Pull requests
|
|
22
|
+
|
|
23
|
+
A useful pull request explains:
|
|
24
|
+
|
|
25
|
+
- the user or Agent failure it fixes;
|
|
26
|
+
- the smallest complete change that fixes it;
|
|
27
|
+
- the exact command or artifact that proves the result;
|
|
28
|
+
- any compatibility, migration, or unverified boundary.
|
|
29
|
+
|
|
30
|
+
Prompt changes must remain visible through `loom prompts` and be reflected in
|
|
31
|
+
[`docs/PROMPT_CATALOG.md`](docs/PROMPT_CATALOG.md). Changes that claim better quality need a baseline-relative
|
|
32
|
+
proof; a nicer-looking output alone is not evidence that LOOM caused the improvement.
|
|
33
|
+
|
|
34
|
+
## Release checks
|
|
35
|
+
|
|
36
|
+
Run `npm test` and `npm pack --dry-run` before requesting release. See the
|
|
37
|
+
[release checklist](docs/RELEASE_CHECKLIST.md) for the maintainer flow.
|
package/EVIL_EVAL.md
CHANGED
|
@@ -1,112 +1,112 @@
|
|
|
1
|
-
# Evil Eval
|
|
2
|
-
|
|
3
|
-
Evil Eval asks a hostile question: does LOOM improve real Agent work, or does it merely generate more process
|
|
4
|
-
that resembles rigor?
|
|
5
|
-
|
|
6
|
-
## Core comparison
|
|
7
|
-
|
|
8
|
-
Each scenario runs two conditions:
|
|
9
|
-
|
|
10
|
-
- Baseline: a normal capable Agent receives the raw brief and all ordinary tools, but no LOOM.
|
|
11
|
-
- LOOM: the same model receives the same raw brief, workspace, tools, user answers, and budget, with LOOM available.
|
|
12
|
-
|
|
13
|
-
The baseline is not weakened. It may inspect files, browse, ask questions, plan, use memory supplied by the host,
|
|
14
|
-
and create any artifacts it normally would. The intended experimental variable is LOOM's protocol and persistent
|
|
15
|
-
state, not permission, information, or model quality.
|
|
16
|
-
|
|
17
|
-
This is the primary causal comparison: **no framework versus LOOM 2**. LOOM 1 may be added as a third regression
|
|
18
|
-
condition when we want to know whether the redesign improved on its ancestor, but it cannot replace the no-framework
|
|
19
|
-
baseline and does not answer whether LOOM itself adds value.
|
|
20
|
-
|
|
21
|
-
## Scenario shape
|
|
22
|
-
|
|
23
|
-
A scenario records:
|
|
24
|
-
|
|
25
|
-
- the identical raw brief;
|
|
26
|
-
- workspace setup and immutable snapshot;
|
|
27
|
-
- hidden user facts and a deterministic answer script;
|
|
28
|
-
- success criteria that judges can observe;
|
|
29
|
-
- context-reset points;
|
|
30
|
-
- equal token, time, and tool budgets;
|
|
31
|
-
- at least three repetitions per condition.
|
|
32
|
-
|
|
33
|
-
High-signal scenarios include ambiguous greenfield work, changes to an existing project, multidisciplinary
|
|
34
|
-
personalized products, operational command work, and a mid-project user correction that supersedes an earlier
|
|
35
|
-
decision.
|
|
36
|
-
|
|
37
|
-
## The evil parts
|
|
38
|
-
|
|
39
|
-
1. Force a context reset after clarification and again mid-Task.
|
|
40
|
-
2. Include one attractive but wrong implementation assumption.
|
|
41
|
-
3. Include a professional domain where generic advice is plausible but harmful.
|
|
42
|
-
4. Change one consequential user answer after the first plan is formed.
|
|
43
|
-
5. Give both conditions equal access to discoverable repository facts.
|
|
44
|
-
6. Penalize unnecessary questions, ceremony, document volume, time, and token cost.
|
|
45
|
-
7. Require an implementation attempt or operational transcript; documents alone cannot win.
|
|
46
|
-
|
|
47
|
-
## Measures
|
|
48
|
-
|
|
49
|
-
Every score cites observable evidence:
|
|
50
|
-
|
|
51
|
-
| Measure | What it asks |
|
|
52
|
-
|---|---|
|
|
53
|
-
| Intent fidelity | Did the result solve the user's actual problem without unauthorized expansion? |
|
|
54
|
-
| Question value | Did questions change decisions, and did the Agent avoid asking discoverable facts? |
|
|
55
|
-
| Whole-project coverage | Were important behaviors, boundaries, dependencies, and failure paths understood? |
|
|
56
|
-
| Capability depth | Did specialist knowledge produce a distinctive project decision rather than a summary? |
|
|
57
|
-
| Buildability | Could a fresh Agent select and begin the right work without guessing? |
|
|
58
|
-
| Reset continuity | What important truth survived forced context loss? |
|
|
59
|
-
| User burden | How much unnecessary explanation, questioning, and framework operation reached the human? |
|
|
60
|
-
| Cost and time | Was any quality gain worth its added tokens, latency, and artifacts? |
|
|
61
|
-
|
|
62
|
-
## Blinding and repetition
|
|
63
|
-
|
|
64
|
-
Runs are anonymized. A fresh judge receives A and B in randomized order without framework names. The judge scores
|
|
65
|
-
both, then receives the order swapped and judges again. Order-sensitive conclusions are marked unstable. Each
|
|
66
|
-
condition runs at least three times because a single model trajectory cannot distinguish a system effect from luck.
|
|
67
|
-
|
|
68
|
-
Anonymization may replace condition labels and framework metadata, but it must preserve each runnable artifact's
|
|
69
|
-
internal directory names and relative paths. Before the judge sees a packet, the harness records a source digest/file
|
|
70
|
-
manifest and reruns the condition's declared tests and smoke commands from the anonymized location. A missing file or
|
|
71
|
-
path failure introduced by copying is a harness failure, not a condition failure; repair the packet and rerun the judge
|
|
72
|
-
while preserving the erroneous first judgment as audit evidence.
|
|
73
|
-
|
|
74
|
-
Human judgment should be used for product taste or domain harm when an automated judge cannot observe it. An LLM
|
|
75
|
-
judge may organize evidence but does not become ground truth by being verbose.
|
|
76
|
-
|
|
77
|
-
## Ablations
|
|
78
|
-
|
|
79
|
-
If full LOOM wins, run smaller comparisons to find the causal mechanism:
|
|
80
|
-
|
|
81
|
-
- Task persistence only;
|
|
82
|
-
- project whole plus Task persistence;
|
|
83
|
-
- capability dossiers without the clarification protocol;
|
|
84
|
-
- full LOOM without forced resets.
|
|
85
|
-
|
|
86
|
-
LOOM should keep only mechanisms whose benefit survives ablation. If the same result comes from a smaller subset,
|
|
87
|
-
delete the rest.
|
|
88
|
-
|
|
89
|
-
## CLI scaffold
|
|
90
|
-
|
|
91
|
-
Create a scenario description:
|
|
92
|
-
|
|
93
|
-
```json
|
|
94
|
-
{
|
|
95
|
-
"id": "EVAL-001",
|
|
96
|
-
"title": "Ambiguous multidisciplinary build",
|
|
97
|
-
"brief": "Build a calming recovery tool from this repository.",
|
|
98
|
-
"hidden_user_facts": ["The user must retain manual control."],
|
|
99
|
-
"success_criteria": ["A fresh Agent resumes correctly after a forced reset."],
|
|
100
|
-
"context_reset_points": ["after-shaping", "mid-task"],
|
|
101
|
-
"repetitions": 3
|
|
102
|
-
}
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
Then run:
|
|
106
|
-
|
|
107
|
-
```bash
|
|
108
|
-
loom eval scaffold --json-file scenario.json
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
LOOM writes a manifest, equalized condition prompts, and a blind judge prompt under `.loom/eval/<scenario>/`.
|
|
112
|
-
Launching isolated Agents remains host-specific; do not claim the evaluation ran merely because the scaffold exists.
|
|
1
|
+
# Evil Eval
|
|
2
|
+
|
|
3
|
+
Evil Eval asks a hostile question: does LOOM improve real Agent work, or does it merely generate more process
|
|
4
|
+
that resembles rigor?
|
|
5
|
+
|
|
6
|
+
## Core comparison
|
|
7
|
+
|
|
8
|
+
Each scenario runs two conditions:
|
|
9
|
+
|
|
10
|
+
- Baseline: a normal capable Agent receives the raw brief and all ordinary tools, but no LOOM.
|
|
11
|
+
- LOOM: the same model receives the same raw brief, workspace, tools, user answers, and budget, with LOOM available.
|
|
12
|
+
|
|
13
|
+
The baseline is not weakened. It may inspect files, browse, ask questions, plan, use memory supplied by the host,
|
|
14
|
+
and create any artifacts it normally would. The intended experimental variable is LOOM's protocol and persistent
|
|
15
|
+
state, not permission, information, or model quality.
|
|
16
|
+
|
|
17
|
+
This is the primary causal comparison: **no framework versus LOOM 2**. LOOM 1 may be added as a third regression
|
|
18
|
+
condition when we want to know whether the redesign improved on its ancestor, but it cannot replace the no-framework
|
|
19
|
+
baseline and does not answer whether LOOM itself adds value.
|
|
20
|
+
|
|
21
|
+
## Scenario shape
|
|
22
|
+
|
|
23
|
+
A scenario records:
|
|
24
|
+
|
|
25
|
+
- the identical raw brief;
|
|
26
|
+
- workspace setup and immutable snapshot;
|
|
27
|
+
- hidden user facts and a deterministic answer script;
|
|
28
|
+
- success criteria that judges can observe;
|
|
29
|
+
- context-reset points;
|
|
30
|
+
- equal token, time, and tool budgets;
|
|
31
|
+
- at least three repetitions per condition.
|
|
32
|
+
|
|
33
|
+
High-signal scenarios include ambiguous greenfield work, changes to an existing project, multidisciplinary
|
|
34
|
+
personalized products, operational command work, and a mid-project user correction that supersedes an earlier
|
|
35
|
+
decision.
|
|
36
|
+
|
|
37
|
+
## The evil parts
|
|
38
|
+
|
|
39
|
+
1. Force a context reset after clarification and again mid-Task.
|
|
40
|
+
2. Include one attractive but wrong implementation assumption.
|
|
41
|
+
3. Include a professional domain where generic advice is plausible but harmful.
|
|
42
|
+
4. Change one consequential user answer after the first plan is formed.
|
|
43
|
+
5. Give both conditions equal access to discoverable repository facts.
|
|
44
|
+
6. Penalize unnecessary questions, ceremony, document volume, time, and token cost.
|
|
45
|
+
7. Require an implementation attempt or operational transcript; documents alone cannot win.
|
|
46
|
+
|
|
47
|
+
## Measures
|
|
48
|
+
|
|
49
|
+
Every score cites observable evidence:
|
|
50
|
+
|
|
51
|
+
| Measure | What it asks |
|
|
52
|
+
|---|---|
|
|
53
|
+
| Intent fidelity | Did the result solve the user's actual problem without unauthorized expansion? |
|
|
54
|
+
| Question value | Did questions change decisions, and did the Agent avoid asking discoverable facts? |
|
|
55
|
+
| Whole-project coverage | Were important behaviors, boundaries, dependencies, and failure paths understood? |
|
|
56
|
+
| Capability depth | Did specialist knowledge produce a distinctive project decision rather than a summary? |
|
|
57
|
+
| Buildability | Could a fresh Agent select and begin the right work without guessing? |
|
|
58
|
+
| Reset continuity | What important truth survived forced context loss? |
|
|
59
|
+
| User burden | How much unnecessary explanation, questioning, and framework operation reached the human? |
|
|
60
|
+
| Cost and time | Was any quality gain worth its added tokens, latency, and artifacts? |
|
|
61
|
+
|
|
62
|
+
## Blinding and repetition
|
|
63
|
+
|
|
64
|
+
Runs are anonymized. A fresh judge receives A and B in randomized order without framework names. The judge scores
|
|
65
|
+
both, then receives the order swapped and judges again. Order-sensitive conclusions are marked unstable. Each
|
|
66
|
+
condition runs at least three times because a single model trajectory cannot distinguish a system effect from luck.
|
|
67
|
+
|
|
68
|
+
Anonymization may replace condition labels and framework metadata, but it must preserve each runnable artifact's
|
|
69
|
+
internal directory names and relative paths. Before the judge sees a packet, the harness records a source digest/file
|
|
70
|
+
manifest and reruns the condition's declared tests and smoke commands from the anonymized location. A missing file or
|
|
71
|
+
path failure introduced by copying is a harness failure, not a condition failure; repair the packet and rerun the judge
|
|
72
|
+
while preserving the erroneous first judgment as audit evidence.
|
|
73
|
+
|
|
74
|
+
Human judgment should be used for product taste or domain harm when an automated judge cannot observe it. An LLM
|
|
75
|
+
judge may organize evidence but does not become ground truth by being verbose.
|
|
76
|
+
|
|
77
|
+
## Ablations
|
|
78
|
+
|
|
79
|
+
If full LOOM wins, run smaller comparisons to find the causal mechanism:
|
|
80
|
+
|
|
81
|
+
- Task persistence only;
|
|
82
|
+
- project whole plus Task persistence;
|
|
83
|
+
- capability dossiers without the clarification protocol;
|
|
84
|
+
- full LOOM without forced resets.
|
|
85
|
+
|
|
86
|
+
LOOM should keep only mechanisms whose benefit survives ablation. If the same result comes from a smaller subset,
|
|
87
|
+
delete the rest.
|
|
88
|
+
|
|
89
|
+
## CLI scaffold
|
|
90
|
+
|
|
91
|
+
Create a scenario description:
|
|
92
|
+
|
|
93
|
+
```json
|
|
94
|
+
{
|
|
95
|
+
"id": "EVAL-001",
|
|
96
|
+
"title": "Ambiguous multidisciplinary build",
|
|
97
|
+
"brief": "Build a calming recovery tool from this repository.",
|
|
98
|
+
"hidden_user_facts": ["The user must retain manual control."],
|
|
99
|
+
"success_criteria": ["A fresh Agent resumes correctly after a forced reset."],
|
|
100
|
+
"context_reset_points": ["after-shaping", "mid-task"],
|
|
101
|
+
"repetitions": 3
|
|
102
|
+
}
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Then run:
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
loom eval scaffold --json-file scenario.json
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
LOOM writes a manifest, equalized condition prompts, and a blind judge prompt under `.loom/eval/<scenario>/`.
|
|
112
|
+
Launching isolated Agents remains host-specific; do not claim the evaluation ran merely because the scaffold exists.
|