@dzhechkov/keysarium-core 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +126 -0
- package/governance/checkpoint-protocol.md +129 -0
- package/governance/constitution.md +144 -0
- package/governance/shard-protocol.md +118 -0
- package/index.md +89 -0
- package/memory/dream-engine.md +148 -0
- package/memory/memory-protocol.md +197 -0
- package/memory/reward-tracker.md +162 -0
- package/orchestration/background-workers.md +141 -0
- package/orchestration/model-routing.md +92 -0
- package/orchestration/queen-protocol.md +154 -0
- package/orchestration/topology-selection.md +175 -0
- package/package.json +44 -0
- package/platform/adapter-registry.md +93 -0
- package/platform/templates/copilot.md +81 -0
- package/platform/templates/cursor.md +65 -0
- package/platform/templates/opencode.md +69 -0
- package/trust-tiers/promotion-protocol.md +144 -0
- package/trust-tiers/tier-system.md +111 -0
- package/verification/audit-trail.md +154 -0
- package/verification/judge-attestation.md +130 -0
- package/verification/witness-chain.md +138 -0
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Cursor Platform Template
|
|
2
|
+
|
|
3
|
+
> Generates `.cursorrules` and optionally `.cursor/skills/` from pipeline source configuration.
|
|
4
|
+
|
|
5
|
+
## Target Format
|
|
6
|
+
|
|
7
|
+
Cursor uses a `.cursorrules` file in the project root. This is a plain-text/markdown file that Cursor's AI reads as project-level instructions. Optionally, `.cursor/skills/` can hold longer skill documents.
|
|
8
|
+
|
|
9
|
+
## Generation Protocol
|
|
10
|
+
|
|
11
|
+
### Step 1: Generate `.cursorrules`
|
|
12
|
+
|
|
13
|
+
Create a `.cursorrules` file with the following structure:
|
|
14
|
+
|
|
15
|
+
```
|
|
16
|
+
# Project: {project name}
|
|
17
|
+
|
|
18
|
+
## Project Overview
|
|
19
|
+
{2-3 paragraphs describing the project purpose}
|
|
20
|
+
|
|
21
|
+
## Rules
|
|
22
|
+
{For each rule file, create a subsection with key bullet points.
|
|
23
|
+
Target: 3-8 bullet points per rule. Keep each to 1-2 lines.}
|
|
24
|
+
|
|
25
|
+
## Skills
|
|
26
|
+
{For each skill, extract:
|
|
27
|
+
- One-line description
|
|
28
|
+
- Core protocol steps (3-5 key steps)
|
|
29
|
+
- Key constraints (2-3 bullet points)
|
|
30
|
+
Keep each skill to 15-30 lines.}
|
|
31
|
+
|
|
32
|
+
## Pipeline
|
|
33
|
+
{Simplified pipeline overview: stage name | description | key output}
|
|
34
|
+
|
|
35
|
+
## Anti-Patterns
|
|
36
|
+
{Copy the anti-patterns table}
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
### Step 2: Generate `.cursor/skills/` (for large skills)
|
|
40
|
+
|
|
41
|
+
If any skill exceeds 200 lines, create `.cursor/skills/{skill-name}.md` with the full content adapted for Cursor.
|
|
42
|
+
|
|
43
|
+
### Step 3: Size Check
|
|
44
|
+
|
|
45
|
+
Verify `.cursorrules` is under 10,000 tokens (~7,500 words). If over, further summarize skill sections.
|
|
46
|
+
|
|
47
|
+
## Content Adaptation Rules
|
|
48
|
+
|
|
49
|
+
| Source Element | Cursor Adaptation |
|
|
50
|
+
|----------------|-------------------|
|
|
51
|
+
| `Read: path/SKILL.md` | "Follow the {X} skill protocol" |
|
|
52
|
+
| Agent tool references | "For parallel work, break into sub-tasks" |
|
|
53
|
+
| Model routing | Omit (Cursor uses its own model) |
|
|
54
|
+
| `$ARGUMENTS` | "User-provided input" |
|
|
55
|
+
| Promise tags | Keep as documentation markers |
|
|
56
|
+
| Checkpoint protocol | "Pause and confirm with user before proceeding" |
|
|
57
|
+
|
|
58
|
+
## Example Output
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
.cursorrules (main instructions, <10K tokens)
|
|
62
|
+
.cursor/
|
|
63
|
+
└── skills/
|
|
64
|
+
└── {large-skill}.md (full skill, if >200 lines)
|
|
65
|
+
```
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# OpenCode Platform Template
|
|
2
|
+
|
|
3
|
+
> Generates `.opencode/` directory structure from pipeline source configuration.
|
|
4
|
+
|
|
5
|
+
## Target Format
|
|
6
|
+
|
|
7
|
+
OpenCode uses a `.opencode/` directory with `config.yaml` and per-file skills and rules. This is the most structurally similar platform to Claude Code.
|
|
8
|
+
|
|
9
|
+
## Generation Protocol
|
|
10
|
+
|
|
11
|
+
### Step 1: Generate `.opencode/config.yaml`
|
|
12
|
+
|
|
13
|
+
```yaml
|
|
14
|
+
# OpenCode configuration for {project name}
|
|
15
|
+
# Generated from source pipeline configuration
|
|
16
|
+
|
|
17
|
+
project:
|
|
18
|
+
name: "{project name}"
|
|
19
|
+
description: "{project description}"
|
|
20
|
+
|
|
21
|
+
instructions: |
|
|
22
|
+
{Critical rules: file conventions, quality requirements, checkpoints.
|
|
23
|
+
Keep to 30-50 lines.}
|
|
24
|
+
|
|
25
|
+
skills_dir: ".opencode/skills"
|
|
26
|
+
rules_dir: ".opencode/rules"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
### Step 2: Generate `.opencode/skills/`
|
|
30
|
+
|
|
31
|
+
For each skill, create `.opencode/skills/{skill-name}.md` with the full SKILL.md content adapted:
|
|
32
|
+
- Replace source-specific paths with `.opencode/` paths
|
|
33
|
+
- Replace Agent tool references with "Break into sub-tasks"
|
|
34
|
+
- Omit model routing (platform-managed)
|
|
35
|
+
|
|
36
|
+
### Step 3: Generate `.opencode/rules/`
|
|
37
|
+
|
|
38
|
+
For each rule file, create `.opencode/rules/{rule-name}.md` with minimal adaptation:
|
|
39
|
+
- Replace source paths
|
|
40
|
+
- Keep tables and structured content intact
|
|
41
|
+
- Remove platform-specific tool references
|
|
42
|
+
|
|
43
|
+
### Step 4: Generate `.opencode/README.md`
|
|
44
|
+
|
|
45
|
+
Brief README explaining the configuration structure and how to regenerate.
|
|
46
|
+
|
|
47
|
+
## Content Adaptation Rules
|
|
48
|
+
|
|
49
|
+
| Source Element | OpenCode Adaptation |
|
|
50
|
+
|----------------|---------------------|
|
|
51
|
+
| `source/skills/X/SKILL.md` | `.opencode/skills/X.md` |
|
|
52
|
+
| `source/rules/X.md` | `.opencode/rules/X.md` |
|
|
53
|
+
| `Read: path` | `See: adapted-path` |
|
|
54
|
+
| Agent tool references | "Break into sub-tasks for parallel work" |
|
|
55
|
+
| Model routing | Omit (platform-managed) |
|
|
56
|
+
|
|
57
|
+
## Example Output
|
|
58
|
+
|
|
59
|
+
```
|
|
60
|
+
.opencode/
|
|
61
|
+
├── config.yaml
|
|
62
|
+
├── README.md
|
|
63
|
+
├── skills/
|
|
64
|
+
│ ├── skill-1.md
|
|
65
|
+
│ └── skill-2.md
|
|
66
|
+
└── rules/
|
|
67
|
+
├── rule-1.md
|
|
68
|
+
└── rule-2.md
|
|
69
|
+
```
|
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
# Promotion Protocol — Advancing Skills Between Tiers
|
|
2
|
+
|
|
3
|
+
> How to promote a skill from one trust tier to the next.
|
|
4
|
+
|
|
5
|
+
## Overview
|
|
6
|
+
|
|
7
|
+
Skills start at the lowest tier that matches their current evidence level. They can be promoted to higher tiers by meeting the requirements defined in `tier-system.md`. This protocol defines the promotion process.
|
|
8
|
+
|
|
9
|
+
## Promotion Paths
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
Tier 0 (Advisory)
|
|
13
|
+
↓ Add references/ or modules/ + complete documentation
|
|
14
|
+
Tier 1 (Structured)
|
|
15
|
+
↓ Pass multi-evaluator panel with score >= 7.0
|
|
16
|
+
Tier 2 (Validated)
|
|
17
|
+
↓ Add deterministic eval tests + score >= 8.5
|
|
18
|
+
Tier 3 (Verified)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Promotion: Tier 0 to Tier 1
|
|
22
|
+
|
|
23
|
+
### Requirements
|
|
24
|
+
1. SKILL.md has complete protocol documentation (not just a stub)
|
|
25
|
+
2. At least ONE of:
|
|
26
|
+
- `references/` directory with 2+ example files
|
|
27
|
+
- `modules/` directory with sub-components
|
|
28
|
+
- Structured output format documented (JSON schema or template)
|
|
29
|
+
|
|
30
|
+
### Process
|
|
31
|
+
1. Review SKILL.md for completeness
|
|
32
|
+
2. Verify supporting materials exist
|
|
33
|
+
3. Update skill metadata:
|
|
34
|
+
```
|
|
35
|
+
trust_tier: 1
|
|
36
|
+
trust_tier_label: "Structured"
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
### No formal evaluation needed — this is a structural check.
|
|
40
|
+
|
|
41
|
+
## Promotion: Tier 1 to Tier 2
|
|
42
|
+
|
|
43
|
+
### Requirements
|
|
44
|
+
1. All Tier 1 requirements met
|
|
45
|
+
2. Pass a multi-evaluator panel evaluation
|
|
46
|
+
3. Average score >= 7.0 out of 10.0
|
|
47
|
+
4. No single judge score below 5.0
|
|
48
|
+
|
|
49
|
+
### Process
|
|
50
|
+
1. Submit the skill for multi-evaluator evaluation
|
|
51
|
+
2. Panel of 3 judges independently evaluates the skill:
|
|
52
|
+
- **Domain Expert** (weight: 0.4) — evaluates domain accuracy and depth
|
|
53
|
+
- **Critic** (weight: 0.3) — looks for weaknesses and edge cases
|
|
54
|
+
- **Completeness Auditor** (weight: 0.3) — checks structural coverage
|
|
55
|
+
3. Compute weighted average score
|
|
56
|
+
4. If score >= 7.0:
|
|
57
|
+
- Update skill metadata:
|
|
58
|
+
```
|
|
59
|
+
trust_tier: 2
|
|
60
|
+
trust_tier_label: "Validated"
|
|
61
|
+
bto_score: {score}
|
|
62
|
+
bto_date: "{YYYY-MM-DD}"
|
|
63
|
+
```
|
|
64
|
+
- Record the full evaluation results
|
|
65
|
+
5. If score < 7.0:
|
|
66
|
+
- Provide judge feedback for improvement
|
|
67
|
+
- Skill remains at Tier 1
|
|
68
|
+
- May re-evaluate after improvements
|
|
69
|
+
|
|
70
|
+
### Judge Panel Rules
|
|
71
|
+
- Judges operate in strict isolation (see judge-attestation.md)
|
|
72
|
+
- Judges MUST be a different model tier than the skill's generation model
|
|
73
|
+
- Judge scores are final unless disagreement > 3 points (escalate to meta-judge)
|
|
74
|
+
|
|
75
|
+
## Promotion: Tier 2 to Tier 3
|
|
76
|
+
|
|
77
|
+
### Requirements
|
|
78
|
+
1. All Tier 2 requirements met
|
|
79
|
+
2. Multi-evaluator panel score >= 8.5
|
|
80
|
+
3. Deterministic eval test suite exists
|
|
81
|
+
4. All eval tests pass consistently
|
|
82
|
+
|
|
83
|
+
### Process
|
|
84
|
+
1. Create eval test suite:
|
|
85
|
+
- Define test cases with known-good inputs and expected outputs
|
|
86
|
+
- Tests must be deterministic (same input always produces the same pass/fail)
|
|
87
|
+
- Minimum 5 test cases covering core functionality
|
|
88
|
+
2. Run eval tests and verify all pass
|
|
89
|
+
3. Re-evaluate with multi-evaluator panel (or use existing score if >= 8.5)
|
|
90
|
+
4. If all conditions met:
|
|
91
|
+
- Update skill metadata:
|
|
92
|
+
```
|
|
93
|
+
trust_tier: 3
|
|
94
|
+
trust_tier_label: "Verified"
|
|
95
|
+
bto_score: {score}
|
|
96
|
+
bto_date: "{YYYY-MM-DD}"
|
|
97
|
+
eval_tests: {count}
|
|
98
|
+
eval_tests_passing: {count}
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Demotion
|
|
102
|
+
|
|
103
|
+
Skills can be demoted if:
|
|
104
|
+
- Eval tests start failing (Tier 3 -> Tier 2)
|
|
105
|
+
- Re-evaluation score drops below threshold (Tier 2 -> Tier 1)
|
|
106
|
+
- Supporting materials are removed (Tier 1 -> Tier 0)
|
|
107
|
+
|
|
108
|
+
Demotion is logged with reason and date.
|
|
109
|
+
|
|
110
|
+
## Promotion History
|
|
111
|
+
|
|
112
|
+
Each skill should maintain a promotion history:
|
|
113
|
+
|
|
114
|
+
```json
|
|
115
|
+
{
|
|
116
|
+
"skill_id": "{name}",
|
|
117
|
+
"current_tier": 2,
|
|
118
|
+
"history": [
|
|
119
|
+
{
|
|
120
|
+
"date": "2026-02-15",
|
|
121
|
+
"from_tier": 0,
|
|
122
|
+
"to_tier": 1,
|
|
123
|
+
"reason": "Added references/ directory with 3 examples"
|
|
124
|
+
},
|
|
125
|
+
{
|
|
126
|
+
"date": "2026-03-01",
|
|
127
|
+
"from_tier": 1,
|
|
128
|
+
"to_tier": 2,
|
|
129
|
+
"reason": "Passed BTO evaluation with score 7.8",
|
|
130
|
+
"score": 7.8,
|
|
131
|
+
"panel": ["domain-expert: 8.2", "critic: 7.5", "auditor: 8.0"]
|
|
132
|
+
}
|
|
133
|
+
]
|
|
134
|
+
}
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## Cross-Project Tier Transfer
|
|
138
|
+
|
|
139
|
+
When importing a skill from another project (via brain import):
|
|
140
|
+
- The imported tier is treated as a **recommendation**, not a guarantee
|
|
141
|
+
- The importing project may choose to:
|
|
142
|
+
1. Accept the tier as-is (trust the source)
|
|
143
|
+
2. Require re-evaluation at the current project (verify locally)
|
|
144
|
+
3. Demote by one tier (conservative approach)
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# Trust Tier System — 4-Tier Classification
|
|
2
|
+
|
|
3
|
+
> Classify skills and artifacts by their validation evidence level.
|
|
4
|
+
|
|
5
|
+
## Overview
|
|
6
|
+
|
|
7
|
+
Not all skills and artifacts have the same level of validation. The Trust Tier system provides a 4-level classification that communicates how much confidence to place in a skill's output.
|
|
8
|
+
|
|
9
|
+
## Tier Definitions
|
|
10
|
+
|
|
11
|
+
| Tier | Label | Requirements | Confidence Level |
|
|
12
|
+
|------|-------|-------------|-----------------|
|
|
13
|
+
| **Tier 3** | Verified | Eval test suites with deterministic validation | Highest — production-ready |
|
|
14
|
+
| **Tier 2** | Validated | Passed multi-evaluator panel with score >= 7.0 | High — tested and scored |
|
|
15
|
+
| **Tier 1** | Structured | Documented protocol + references or modules | Medium — well-organized |
|
|
16
|
+
| **Tier 0** | Advisory | Basic documentation only | Low — use with caution |
|
|
17
|
+
|
|
18
|
+
## Tier Criteria (Detailed)
|
|
19
|
+
|
|
20
|
+
### Tier 0 — Advisory
|
|
21
|
+
|
|
22
|
+
**Minimum requirements:**
|
|
23
|
+
- A SKILL.md (or equivalent) file exists
|
|
24
|
+
- Basic instructions are documented
|
|
25
|
+
|
|
26
|
+
**What it means:** The skill exists and has instructions, but has not been tested or validated. Output quality is unpredictable.
|
|
27
|
+
|
|
28
|
+
**Flag in pipeline:** Display a warning when a Tier 0 skill is loaded:
|
|
29
|
+
```
|
|
30
|
+
WARNING: Skill '{name}' is Tier 0 (Advisory). Output may need extra review.
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### Tier 1 — Structured
|
|
34
|
+
|
|
35
|
+
**Minimum requirements:**
|
|
36
|
+
- SKILL.md with complete protocol documentation
|
|
37
|
+
- At least ONE of:
|
|
38
|
+
- `references/` directory with example inputs/outputs
|
|
39
|
+
- `modules/` directory with sub-components
|
|
40
|
+
- Structured output format (JSON schema, template)
|
|
41
|
+
|
|
42
|
+
**What it means:** The skill is well-organized and has supporting materials, but has not been formally evaluated.
|
|
43
|
+
|
|
44
|
+
### Tier 2 — Validated
|
|
45
|
+
|
|
46
|
+
**Minimum requirements:**
|
|
47
|
+
- All Tier 1 requirements
|
|
48
|
+
- Passed a multi-evaluator panel (3+ judges) with average score >= 7.0 out of 10.0
|
|
49
|
+
- Evaluation results recorded (date, scores, panel composition)
|
|
50
|
+
|
|
51
|
+
**What it means:** The skill has been formally tested by multiple evaluators and scored above the quality threshold.
|
|
52
|
+
|
|
53
|
+
**Recording:** Add to the skill's metadata:
|
|
54
|
+
```
|
|
55
|
+
trust_tier: 2
|
|
56
|
+
trust_tier_label: "Validated"
|
|
57
|
+
bto_score: 7.8
|
|
58
|
+
bto_date: "2026-03-01"
|
|
59
|
+
bto_panel: "domain-expert (8.2), critic (7.5), completeness-auditor (8.0)"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
### Tier 3 — Verified
|
|
63
|
+
|
|
64
|
+
**Minimum requirements:**
|
|
65
|
+
- All Tier 2 requirements
|
|
66
|
+
- Deterministic eval test suite exists
|
|
67
|
+
- Tests pass consistently (reproducible results)
|
|
68
|
+
- Score >= 8.5 on the multi-evaluator panel
|
|
69
|
+
|
|
70
|
+
**What it means:** The skill has both human evaluation AND automated testing. This is the highest confidence level.
|
|
71
|
+
|
|
72
|
+
## Classification Checklist
|
|
73
|
+
|
|
74
|
+
Use this checklist to determine a skill's current tier:
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
[ ] SKILL.md exists → Tier 0 minimum
|
|
78
|
+
[ ] SKILL.md has complete protocol documentation → Tier 0
|
|
79
|
+
[ ] references/ OR modules/ OR structured output exists → Tier 1
|
|
80
|
+
[ ] Multi-evaluator panel score >= 7.0 recorded → Tier 2
|
|
81
|
+
[ ] Deterministic eval test suite exists and passes → Tier 3
|
|
82
|
+
[ ] Multi-evaluator panel score >= 8.5 → Tier 3
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Tier Display Format
|
|
86
|
+
|
|
87
|
+
When displaying skill tiers (e.g., in a health check):
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
Skills Health Check:
|
|
91
|
+
skill-name-1 Tier 2 — Validated (score: 7.8)
|
|
92
|
+
skill-name-2 Tier 1 — Structured
|
|
93
|
+
skill-name-3 Tier 0 — Advisory (missing: references/)
|
|
94
|
+
skill-name-4 Tier 3 — Verified (score: 8.9, tests: 12/12)
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Enforcement Rules
|
|
98
|
+
|
|
99
|
+
| Context | Rule |
|
|
100
|
+
|---------|------|
|
|
101
|
+
| Production pipeline | Warn on Tier 0 skills, recommend alternatives |
|
|
102
|
+
| Critical decisions | Require Tier 2+ skills for decision-making stages |
|
|
103
|
+
| Evaluation panels | Judges should be Tier 1+ (Tier 0 judges produce unreliable scores) |
|
|
104
|
+
| Knowledge export | Include tier metadata in brain export |
|
|
105
|
+
|
|
106
|
+
## Integration
|
|
107
|
+
|
|
108
|
+
The tier system integrates with:
|
|
109
|
+
- **Memory Protocol:** Tier affects reward weight (higher tier = more trusted outcomes)
|
|
110
|
+
- **Model Routing:** Tier can influence model selection (Tier 0 skills may need Tier 3 model to compensate)
|
|
111
|
+
- **Brain Export:** Tier metadata is included in portable brain containers
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
# Audit Trail Protocol — Complete Evaluation History
|
|
2
|
+
|
|
3
|
+
> Protocol for maintaining a complete, verifiable history of all evaluations and decisions in a pipeline.
|
|
4
|
+
|
|
5
|
+
## Overview
|
|
6
|
+
|
|
7
|
+
The Audit Trail combines the Witness Chain (artifact integrity) and Judge Attestation (evaluator isolation) into a unified evaluation history. It provides a single point of reference for auditing any decision made during pipeline execution.
|
|
8
|
+
|
|
9
|
+
## Audit Trail Structure
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
{audit-root}/
|
|
13
|
+
├── audit-log.json ← Master log of all events
|
|
14
|
+
├── witness-chains/
|
|
15
|
+
│ └── {project-slug}.json ← Per-project witness chain
|
|
16
|
+
├── attestations/
|
|
17
|
+
│ └── {evaluation-id}.json ← Per-evaluation attestation set
|
|
18
|
+
└── decisions/
|
|
19
|
+
└── {decision-id}.json ← Per-decision record
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Event Types
|
|
23
|
+
|
|
24
|
+
| Event Type | Description | Source Protocol |
|
|
25
|
+
|------------|-------------|----------------|
|
|
26
|
+
| `artifact_created` | A stage artifact was produced | witness-chain.md |
|
|
27
|
+
| `artifact_verified` | An artifact's hash was verified | witness-chain.md |
|
|
28
|
+
| `artifact_modified` | An artifact was legitimately changed | witness-chain.md (repair) |
|
|
29
|
+
| `evaluation_started` | A judge panel began evaluation | judge-attestation.md |
|
|
30
|
+
| `evaluation_completed` | A judge panel finished scoring | judge-attestation.md |
|
|
31
|
+
| `checkpoint_reached` | A human checkpoint was displayed | checkpoint-protocol.md |
|
|
32
|
+
| `checkpoint_approved` | Human approved at checkpoint | checkpoint-protocol.md |
|
|
33
|
+
| `checkpoint_revised` | Human requested changes at checkpoint | checkpoint-protocol.md |
|
|
34
|
+
| `reward_stored` | A reward record was persisted | memory-protocol.md |
|
|
35
|
+
| `decision_made` | A pipeline decision was recorded | (this protocol) |
|
|
36
|
+
|
|
37
|
+
## Audit Log Schema
|
|
38
|
+
|
|
39
|
+
```json
|
|
40
|
+
{
|
|
41
|
+
"version": "1.0",
|
|
42
|
+
"pipeline_id": "string — unique pipeline execution ID",
|
|
43
|
+
"started_at": "ISO-8601",
|
|
44
|
+
"events": [
|
|
45
|
+
{
|
|
46
|
+
"event_id": "evt-{sequence:06d}",
|
|
47
|
+
"type": "string — event type from table above",
|
|
48
|
+
"timestamp": "ISO-8601",
|
|
49
|
+
"stage": "string — stage where event occurred",
|
|
50
|
+
"actor": "string — agent ID or 'human'",
|
|
51
|
+
"details": {
|
|
52
|
+
"description": "string — human-readable event description",
|
|
53
|
+
"artifact": "string — file path (if applicable)",
|
|
54
|
+
"hash": "sha256:... (if applicable)",
|
|
55
|
+
"score": 8.2,
|
|
56
|
+
"reward": 0.7
|
|
57
|
+
},
|
|
58
|
+
"references": {
|
|
59
|
+
"witness_chain_sequence": 3,
|
|
60
|
+
"attestation_id": "bto-eval-...",
|
|
61
|
+
"checkpoint_number": 2
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
]
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## Decision Record Schema
|
|
69
|
+
|
|
70
|
+
For significant decisions (e.g., which solution variant to choose, which candidate to promote):
|
|
71
|
+
|
|
72
|
+
```json
|
|
73
|
+
{
|
|
74
|
+
"decision_id": "dec-{YYYYMMDD}-{HHmmss}-{seq}",
|
|
75
|
+
"timestamp": "ISO-8601",
|
|
76
|
+
"stage": "string — stage where decision was made",
|
|
77
|
+
"decision_type": "variant_selection | tier_promotion | escalation | rollback",
|
|
78
|
+
"description": "string — what was decided",
|
|
79
|
+
"options_considered": [
|
|
80
|
+
{
|
|
81
|
+
"option_id": "A",
|
|
82
|
+
"description": "string",
|
|
83
|
+
"score": 8.2,
|
|
84
|
+
"selected": true
|
|
85
|
+
},
|
|
86
|
+
{
|
|
87
|
+
"option_id": "B",
|
|
88
|
+
"description": "string",
|
|
89
|
+
"score": 7.1,
|
|
90
|
+
"selected": false
|
|
91
|
+
}
|
|
92
|
+
],
|
|
93
|
+
"rationale": "string — why this option was chosen",
|
|
94
|
+
"decided_by": "string — agent ID or 'human'",
|
|
95
|
+
"evidence": {
|
|
96
|
+
"attestation_id": "string (if decision was based on judge scores)",
|
|
97
|
+
"witness_chain_hash": "sha256:... (if decision was about an artifact)"
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
## Audit Trail Operations
|
|
103
|
+
|
|
104
|
+
### Initialize
|
|
105
|
+
|
|
106
|
+
At pipeline start:
|
|
107
|
+
1. Create `{audit-root}/` directory if it does not exist
|
|
108
|
+
2. Create a new `audit-log.json` with the pipeline execution ID
|
|
109
|
+
3. Log the `pipeline_started` event
|
|
110
|
+
|
|
111
|
+
### Record Event
|
|
112
|
+
|
|
113
|
+
At each significant point:
|
|
114
|
+
1. Read the current audit-log.json
|
|
115
|
+
2. Append the new event with an incremental event_id
|
|
116
|
+
3. Write the updated audit-log.json
|
|
117
|
+
|
|
118
|
+
### Query
|
|
119
|
+
|
|
120
|
+
To find events related to a specific artifact or stage:
|
|
121
|
+
1. Load audit-log.json
|
|
122
|
+
2. Filter events by `stage`, `type`, or `details.artifact`
|
|
123
|
+
3. Return matching events in chronological order
|
|
124
|
+
|
|
125
|
+
### Verify Integrity
|
|
126
|
+
|
|
127
|
+
To verify the audit trail has not been tampered with:
|
|
128
|
+
1. For each `artifact_created` event, verify the hash in the witness chain
|
|
129
|
+
2. For each `evaluation_completed` event, verify attestations
|
|
130
|
+
3. Check that event timestamps are monotonically increasing
|
|
131
|
+
4. Report any inconsistencies
|
|
132
|
+
|
|
133
|
+
## Integration Points
|
|
134
|
+
|
|
135
|
+
| Source | Audit Event | When |
|
|
136
|
+
|--------|------------|------|
|
|
137
|
+
| Witness Chain | `artifact_created` | After each stage artifact is hashed |
|
|
138
|
+
| Judge Attestation | `evaluation_started`, `evaluation_completed` | At judge panel start/end |
|
|
139
|
+
| Checkpoint Protocol | `checkpoint_reached`, `checkpoint_approved` | At each human checkpoint |
|
|
140
|
+
| Memory Protocol | `reward_stored` | After each memory_store() call |
|
|
141
|
+
|
|
142
|
+
## Retention Policy
|
|
143
|
+
|
|
144
|
+
- Audit logs are retained for the lifetime of the project
|
|
145
|
+
- They are NOT subject to the memory protocol's expiration rules
|
|
146
|
+
- On brain export, the audit trail is included as metadata (event counts, not full events)
|
|
147
|
+
|
|
148
|
+
## Regulatory Compliance Note
|
|
149
|
+
|
|
150
|
+
For regulated domains (banking, healthcare), the audit trail provides:
|
|
151
|
+
- Complete decision traceability
|
|
152
|
+
- Proof of human oversight (checkpoint events)
|
|
153
|
+
- Proof of evaluator independence (attestation events)
|
|
154
|
+
- Artifact integrity verification (witness chain events)
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# Judge Attestation Protocol — Evaluator Isolation Proofs
|
|
2
|
+
|
|
3
|
+
> Cryptographic proof that evaluators in a multi-evaluator panel operated independently.
|
|
4
|
+
|
|
5
|
+
## Overview
|
|
6
|
+
|
|
7
|
+
When multiple evaluators (judges) assess the same artifact, it is critical to prove they operated independently. This protocol defines how each judge creates a cryptographic attestation of its evaluation before seeing other judges' scores.
|
|
8
|
+
|
|
9
|
+
## Constants
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
NULL_HASH = "0000000000000000000000000000000000000000000000000000000000000000"
|
|
13
|
+
ATTESTATION_FILE = ".judge-attestations.json"
|
|
14
|
+
HASH_PREFIX = "sha256:"
|
|
15
|
+
STANDARD_PANEL_SIZE = 3
|
|
16
|
+
HIGH_STAKES_PANEL_SIZE = 5
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Attestation Creation
|
|
20
|
+
|
|
21
|
+
### Hash Input Construction
|
|
22
|
+
|
|
23
|
+
Each judge's evaluation hash is computed from:
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
evaluation_hash = SHA-256(judge_id + "|" + artifact_hash + "|" + score + "|" + rationale_summary)
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
- `judge_id`: Unique identifier for the judge role
|
|
30
|
+
- `artifact_hash`: SHA-256 hash of the artifact being evaluated
|
|
31
|
+
- `score`: Numeric score as a string with one decimal place (e.g., "8.2")
|
|
32
|
+
- `rationale_summary`: First 500 characters of the judge's rationale text
|
|
33
|
+
|
|
34
|
+
### Computing the Artifact Hash
|
|
35
|
+
|
|
36
|
+
Before any judge starts, compute the artifact hash:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
ARTIFACT_HASH=$(${SHA_CMD} "path/to/artifact.md" | awk '{print $1}')
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
This proves all judges evaluated the same artifact.
|
|
43
|
+
|
|
44
|
+
### Computing the Evaluation Hash
|
|
45
|
+
|
|
46
|
+
After a judge completes its evaluation:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
EVAL_HASH=$(printf '%s' "${JUDGE_ID}|${ARTIFACT_HASH}|${SCORE}|${RATIONALE_SUMMARY}" | ${SHA_CMD} | awk '{print $1}')
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## Chain Linking
|
|
53
|
+
|
|
54
|
+
Judges are recorded in a fixed order:
|
|
55
|
+
|
|
56
|
+
1. First judge (previous = NULL_HASH)
|
|
57
|
+
2. Second judge (previous = first judge's evaluation_hash)
|
|
58
|
+
3. Third judge (previous = second judge's evaluation_hash)
|
|
59
|
+
4. Additional judges continue the chain
|
|
60
|
+
|
|
61
|
+
The chain records the ORDER of attestation finalization, proving each was recorded after the previous.
|
|
62
|
+
|
|
63
|
+
## Isolation Proof Logic
|
|
64
|
+
|
|
65
|
+
1. **Hash Independence:** Each evaluation_hash is computed from the judge's own data only
|
|
66
|
+
2. **Chain Integrity:** The previous_attestation_hash links attestations in order but does NOT include other judges' scores
|
|
67
|
+
3. **Timestamp Monotonicity:** Timestamps must be strictly increasing
|
|
68
|
+
|
|
69
|
+
## Attestation File Schema
|
|
70
|
+
|
|
71
|
+
```json
|
|
72
|
+
{
|
|
73
|
+
"evaluations": [
|
|
74
|
+
{
|
|
75
|
+
"evaluation_id": "string — unique ID for this round",
|
|
76
|
+
"artifact_path": "string — path to evaluated artifact",
|
|
77
|
+
"artifact_hash": "sha256:<hex>",
|
|
78
|
+
"panel_size": 3,
|
|
79
|
+
"started_at": "ISO-8601",
|
|
80
|
+
"completed_at": "ISO-8601",
|
|
81
|
+
"attestations": [
|
|
82
|
+
{
|
|
83
|
+
"judge_id": "string — judge role identifier",
|
|
84
|
+
"score": 8.2,
|
|
85
|
+
"rationale_summary": "string — first 500 chars",
|
|
86
|
+
"evaluation_hash": "sha256:<hex>",
|
|
87
|
+
"timestamp": "ISO-8601",
|
|
88
|
+
"previous_attestation_hash": "sha256:<hex or null hash>"
|
|
89
|
+
}
|
|
90
|
+
],
|
|
91
|
+
"final_score": 7.94,
|
|
92
|
+
"weights": {
|
|
93
|
+
"judge-1": 0.4,
|
|
94
|
+
"judge-2": 0.3,
|
|
95
|
+
"judge-3": 0.3
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
]
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
## Verification
|
|
103
|
+
|
|
104
|
+
```
|
|
105
|
+
1. Load attestation file
|
|
106
|
+
2. For each evaluation round:
|
|
107
|
+
a. Verify artifact_hash matches across all attestations
|
|
108
|
+
b. For each attestation (index i):
|
|
109
|
+
i. Reconstruct evaluation_hash from (judge_id, artifact_hash, score, rationale_summary)
|
|
110
|
+
ii. Compare with stored evaluation_hash
|
|
111
|
+
iii. Verify chain link (previous_attestation_hash)
|
|
112
|
+
iv. Verify timestamp monotonicity
|
|
113
|
+
3. Produce verification report
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## Meta-Judge Attestation
|
|
117
|
+
|
|
118
|
+
When a meta-judge is invoked (e.g., disagreement exceeds threshold), it creates its own attestation appended to the chain. The meta-judge MAY reference other judges' scores (it is explicitly a synthesizer, not an independent evaluator).
|
|
119
|
+
|
|
120
|
+
## Disagreement Detection Enhancement
|
|
121
|
+
|
|
122
|
+
With attestations, disagreement detection gains cryptographic backing:
|
|
123
|
+
|
|
124
|
+
```
|
|
125
|
+
If max_score - min_score > threshold:
|
|
126
|
+
1. Verify all attestations are valid
|
|
127
|
+
2. Verify isolation (no influence detected)
|
|
128
|
+
3. If isolation verified: escalate to meta-judge (genuine disagreement)
|
|
129
|
+
4. If isolation violated: flag conformity collapse warning
|
|
130
|
+
```
|