mishkan-harness 0.2.6 → 0.2.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (133) hide show
  1. package/README.md +49 -11
  2. package/bin/mishkan.js +562 -79
  3. package/docs/design/MISHKAN_decisions.md +398 -8
  4. package/docs/design/MISHKAN_observability.md +8 -8
  5. package/docs/engineer/profile-readable.md +201 -0
  6. package/docs/engineer/profile.md +754 -0
  7. package/docs/usage/01-installation.md +22 -8
  8. package/docs/usage/02-project-init.md +30 -13
  9. package/docs/usage/03-orchestration.md +46 -7
  10. package/docs/usage/04-memory-layer.md +135 -69
  11. package/docs/usage/06-llm-providers.md +2 -1
  12. package/docs/usage/07-troubleshooting.md +23 -0
  13. package/docs/usage/08-glossary.md +12 -3
  14. package/docs/usage/09-workflows.md +15 -3
  15. package/docs/usage/10-observability.md +3 -3
  16. package/docs/usage/11-graphify.md +38 -18
  17. package/docs/usage/12-cli.md +87 -0
  18. package/docs/usage/12-skill-discovery.md +11 -0
  19. package/docs/usage/README.md +35 -19
  20. package/package.json +1 -1
  21. package/payload/mishkan/AGENT_SPEC.md +3 -1
  22. package/payload/mishkan/agents/ahikam.md +3 -1
  23. package/payload/mishkan/agents/aholiab.md +3 -1
  24. package/payload/mishkan/agents/asaph.md +3 -1
  25. package/payload/mishkan/agents/baruch.md +8 -1
  26. package/payload/mishkan/agents/benaiah.md +3 -1
  27. package/payload/mishkan/agents/bezalel.md +3 -1
  28. package/payload/mishkan/agents/caleb.md +3 -1
  29. package/payload/mishkan/agents/deborah.md +3 -1
  30. package/payload/mishkan/agents/elasah.md +3 -1
  31. package/payload/mishkan/agents/eliashib.md +3 -1
  32. package/payload/mishkan/agents/ezra.md +3 -1
  33. package/payload/mishkan/agents/hanun.md +3 -1
  34. package/payload/mishkan/agents/hiram.md +3 -1
  35. package/payload/mishkan/agents/hizkiah.md +3 -1
  36. package/payload/mishkan/agents/huldah.md +3 -1
  37. package/payload/mishkan/agents/huram.md +3 -1
  38. package/payload/mishkan/agents/hushai.md +3 -1
  39. package/payload/mishkan/agents/igal.md +3 -1
  40. package/payload/mishkan/agents/ira.md +3 -1
  41. package/payload/mishkan/agents/jahaziel.md +3 -1
  42. package/payload/mishkan/agents/jakin.md +3 -1
  43. package/payload/mishkan/agents/jehonathan.md +3 -1
  44. package/payload/mishkan/agents/jehoshaphat.md +3 -1
  45. package/payload/mishkan/agents/joab.md +3 -1
  46. package/payload/mishkan/agents/joah.md +3 -1
  47. package/payload/mishkan/agents/maaseiah.md +3 -1
  48. package/payload/mishkan/agents/meremoth.md +3 -1
  49. package/payload/mishkan/agents/meshullam.md +3 -1
  50. package/payload/mishkan/agents/nathan.md +3 -1
  51. package/payload/mishkan/agents/nehemiah.md +3 -1
  52. package/payload/mishkan/agents/obed.md +3 -1
  53. package/payload/mishkan/agents/oholiab.md +3 -1
  54. package/payload/mishkan/agents/palal.md +3 -1
  55. package/payload/mishkan/agents/phinehas.md +3 -1
  56. package/payload/mishkan/agents/rehum.md +3 -1
  57. package/payload/mishkan/agents/salma.md +3 -1
  58. package/payload/mishkan/agents/seraiah.md +3 -1
  59. package/payload/mishkan/agents/shallum.md +3 -1
  60. package/payload/mishkan/agents/shaphan.md +3 -1
  61. package/payload/mishkan/agents/shemaiah.md +6 -1
  62. package/payload/mishkan/agents/shevna.md +3 -1
  63. package/payload/mishkan/agents/uriah.md +3 -1
  64. package/payload/mishkan/agents/zaccur.md +3 -1
  65. package/payload/mishkan/agents/zadok.md +3 -1
  66. package/payload/mishkan/agents/zerubbabel.md +3 -1
  67. package/payload/mishkan/cognee/.env.example +11 -0
  68. package/payload/mishkan/cognee/Dockerfile +20 -0
  69. package/payload/mishkan/cognee/README.md +34 -14
  70. package/payload/mishkan/cognee/docker-compose.curated.yml +33 -6
  71. package/payload/mishkan/cognee/docker-compose.selfhosted.yml +35 -5
  72. package/payload/mishkan/cognee/docker-compose.work.yml +191 -0
  73. package/payload/mishkan/cognee/docker-compose.yml +18 -3
  74. package/payload/mishkan/cognee/patches/cognee-mcp-core-align.py +72 -0
  75. package/payload/mishkan/cognee/patches/cognee-mcp-recall-user.py +82 -0
  76. package/payload/mishkan/cognee/promote-curated.py +77 -0
  77. package/payload/mishkan/cognee/prune-store.py +23 -0
  78. package/payload/mishkan/commands/mishkan-init.md +4 -1
  79. package/payload/mishkan/commands/mishkan-org-reference.md +2 -2
  80. package/payload/mishkan/config/curated-library.yaml +1 -0
  81. package/payload/mishkan/config/model-routing.yaml +23 -14
  82. package/payload/mishkan/evals/baruch/fixtures/invalid/bad-curated-candidate.json +22 -0
  83. package/payload/mishkan/evals/baruch/fixtures/valid/resolved-curated-candidate.json +23 -0
  84. package/payload/mishkan/hooks/model-route.py +14 -1
  85. package/payload/mishkan/observability/usage_parser.py +1 -0
  86. package/payload/mishkan/observability/watch/pyproject.toml +1 -1
  87. package/payload/mishkan/observability/watch/src/mishkan_watch/__main__.py +78 -21
  88. package/payload/mishkan/observability/watch/src/mishkan_watch/app.py +56 -3
  89. package/payload/mishkan/observability/watch/src/mishkan_watch/tabs/knowledge.py +76 -5
  90. package/payload/mishkan/observability/watch/src/mishkan_watch/tabs/workflows.py +22 -1
  91. package/payload/mishkan/observability/watchd/pyproject.toml +1 -1
  92. package/payload/mishkan/observability/watchd/src/mishkan_watchd/__main__.py +35 -7
  93. package/payload/mishkan/observability/watchd/src/mishkan_watchd/server.py +102 -8
  94. package/payload/mishkan/observability/watchd/src/mishkan_watchd/sources/cognee_poll.py +136 -49
  95. package/payload/mishkan/ontology.md +8 -1
  96. package/payload/mishkan/ontology.ttl +188 -0
  97. package/payload/mishkan/scripts/ensure-work-store.sh +230 -0
  98. package/payload/mishkan/scripts/mishkan-ingest.sh +49 -3
  99. package/payload/mishkan/scripts/promote-curated.sh +77 -0
  100. package/payload/mishkan/scripts/reset-knowledge-data.sh +70 -0
  101. package/payload/mishkan/scripts/validate-research-log.sh +27 -0
  102. package/payload/mishkan/skills/baruch-research-reporting-craft/SKILL.md +42 -8
  103. package/payload/mishkan/skills/cognee-promote/SKILL.md +13 -0
  104. package/payload/mishkan/skills/cognee-quickstart/SKILL.md +7 -1
  105. package/payload/mishkan/skills/ezra-research-formulation-craft/SKILL.md +9 -0
  106. package/payload/mishkan/skills/mishkan-ingest/SKILL.md +22 -0
  107. package/payload/mishkan/skills/mishkan-init/SKILL.md +31 -19
  108. package/payload/mishkan/skills/shemaiah-evaluation-craft/SKILL.md +33 -5
  109. package/payload/mishkan/skills/team-lead-craft/SKILL.md +66 -0
  110. package/payload/mishkan/templates/mcp.json +7 -2
  111. package/payload/mishkan/templates/observability-log.schema.json +1 -1
  112. package/payload/mishkan/templates/research-log.schema.json +15 -1
  113. package/payload/mishkan/workflows/README.md +19 -13
  114. package/payload/mishkan/workflows/chosheb-feature-ship.js +63 -18
  115. package/payload/mishkan/workflows/migdal-dr-drill.js +4 -0
  116. package/payload/mishkan/workflows/migdal-infra-change.js +48 -7
  117. package/payload/mishkan/workflows/mishkan-architecture-panel.js +4 -0
  118. package/payload/mishkan/workflows/mishkan-blast-radius.js +4 -0
  119. package/payload/mishkan/workflows/mishkan-codebase-audit.js +4 -0
  120. package/payload/mishkan/workflows/mishkan-deep-research.js +4 -0
  121. package/payload/mishkan/workflows/mishkan-init.js +4 -0
  122. package/payload/mishkan/workflows/mishkan-knowledge-gap-discovery.js +4 -0
  123. package/payload/mishkan/workflows/mishkan-migration-wave.js +4 -0
  124. package/payload/mishkan/workflows/mishkan-release-readiness.js +4 -0
  125. package/payload/mishkan/workflows/mishkan-sprint-close.js +4 -0
  126. package/payload/mishkan/workflows/mishkan-standards-rollout.js +4 -0
  127. package/payload/mishkan/workflows/mishmar-security-gate.js +45 -6
  128. package/payload/mishkan/workflows/panim-ds-rollout.js +4 -0
  129. package/payload/mishkan/workflows/panim-feature-ship.js +132 -0
  130. package/payload/mishkan/workflows/sefer-release-notes.js +4 -0
  131. package/payload/mishkan/workflows/yasad-data-migration-wave.js +4 -0
  132. package/payload/mishkan/workflows/yasad-feature-ship.js +130 -0
  133. package/payload/mishkan/workflows/yasad-schema-evolution.js +4 -0
@@ -88,7 +88,7 @@ Phases:
88
88
  6. **Stamp** — record version + timestamp for `status` / `uninstall` to read.
89
89
  7. **Observability (opt-in)** — prompts to install `mishkan-watchd` and
90
90
  `mishkan-watch` via `uv tool`. Skipped cleanly if `uv` isn't on PATH; you
91
- can re-run it later with `npx mishkan-harness observability`.
91
+ can re-run it later with `npx mishkan-harness observability install`.
92
92
 
93
93
  ## Verify the install
94
94
 
@@ -114,10 +114,22 @@ cat ~/.claude/mishkan/.install-stamp.json 2>/dev/null
114
114
 
115
115
  ## Bring the cognee memory stack up
116
116
 
117
- The harness ships the memory layer as a Docker Compose stack. Two boxes:
118
- **work** (per-project knowledge, port `:7777`) and **curated** (shared reference
119
- library, port `:7730`). See [memory layer](./04-memory-layer.md) for the
120
- full design.
117
+ The harness ships the memory layer as a Docker Compose stack. Three pillars:
118
+ **per-project work stores** (one isolated Ladybug container per project,
119
+ provisioned at `/mishkan-init`), **cognee-memory** (`:7777`, shared per-client
120
+ session memory), and **cognee-curated** (`:7730`, shared reference library).
121
+ See [memory layer](./04-memory-layer.md) for the full design (D-007 + D-012).
122
+
123
+ The guided path is one command — it preflights the config, names any gap, then
124
+ brings up the shared stack (base + hardening + selfhosted overlays) and seeds the
125
+ curated box:
126
+
127
+ ```bash
128
+ mishkan knowledge configure # wizard: LLM_API_KEY + provider profile → .env (see chapter 06)
129
+ mishkan knowledge-stack up # memory :7777 + curated :7730, idempotent (~5 min cold start)
130
+ ```
131
+
132
+ <details><summary>What <code>knowledge-stack up</code> wraps, if you prefer the manual steps</summary>
121
133
 
122
134
  ```bash
123
135
  cd ~/.claude/mishkan/cognee
@@ -126,8 +138,8 @@ cd ~/.claude/mishkan/cognee
126
138
  cp .env.example .env
127
139
  # … fill LLM_API_KEY and pick a provider profile (see chapter 06)
128
140
 
129
- # 2. bring up the work stack (always with the hardening overlay)
130
- docker compose -f docker-compose.yml -f docker-compose.hardening.yml up -d --build
141
+ # 2. bring up the stack (base + hardening + selfhosted overlays)
142
+ docker compose -f docker-compose.yml -f docker-compose.hardening.yml -f docker-compose.selfhosted.yml up -d --build
131
143
 
132
144
  # 3. bring up the curated box (one-time per host, idempotent helper)
133
145
  bash ~/.claude/mishkan/scripts/ensure-curated-box.sh
@@ -136,6 +148,8 @@ bash ~/.claude/mishkan/scripts/ensure-curated-box.sh
136
148
  bash ~/.claude/mishkan/scripts/seed-curated-library.sh
137
149
  ```
138
150
 
151
+ </details>
152
+
139
153
  Health checks:
140
154
 
141
155
  ```bash
@@ -187,7 +201,7 @@ The installer's phase 7 asks whether to install the cross-session observability
187
201
  stack. You can also run it standalone any time:
188
202
 
189
203
  ```bash
190
- npx mishkan-harness observability
204
+ npx mishkan-harness observability install
191
205
  ```
192
206
 
193
207
  Requirements: `uv` (https://astral.sh/uv) and Python 3.11+. If `uv` is not on
@@ -52,7 +52,20 @@ The full sequence (when chosen):
52
52
  8. **Automated** cognee setup (see below).
53
53
  9. **Automated** project `CLAUDE.md` write + sprint S0.
54
54
 
55
- Each step that touches a contract requires `/plan` to run first.
55
+ ```mermaid
56
+ flowchart TD
57
+ PRD["1 · Nehemiah<br/>docs/PRD.md"] --> SRS["2 · Nathan<br/>docs/SRS.md"]
58
+ SRS --> CON["3 · Zadok<br/>docs/CONTRACT.md"]
59
+ CON --> ARC["4 · Bezalel + Nathan<br/>docs/ARCHITECTURE.md"]
60
+ ARC --> TM["5 · Benaiah<br/>docs/THREAT_MODEL.md · STRIDE"]
61
+ TM --> C4["6 · Meshullam<br/>docs/diagrams/C4/"]
62
+ C4 --> DOCS["7 · Jehoshaphat<br/>README · adr · runbooks"]
63
+ DOCS --> KN["8 · automated<br/>cognee setup"]
64
+ KN --> CL["9 · automated<br/>CLAUDE.md + Sprint S0"]
65
+ ```
66
+
67
+ Each step that touches a contract requires `/plan` to run first — nothing is generated
68
+ without its upstream artifact.
56
69
 
57
70
  ## Step 8 in detail: cognee setup at init
58
71
 
@@ -81,8 +94,10 @@ A lean, dynamic file that loads **after** the user-level identity. It carries:
81
94
  main session needs every turn.
82
95
  - Sprint slot — *current sprint*, *what's in flight*, *blockers*. Updated by
83
96
  `/mishkan-resume`, `/sprint-close`, and you.
84
- - Note that there are two cognee stores (`cognee` = work, `cognee-curated` =
85
- reference) and that `cognee-curated` is read-only.
97
+ - Note that init provisions two project-level MCP aliases (`cognee` = this
98
+ project's work store, `cognee-curated` = shared reference library; the latter
99
+ is read-only). The third cognee store — `cognee-memory` (`:7777`, shared
100
+ session memory) — is a pre-existing singleton, not provisioned here.
86
101
  - A pointer to the existing `docs/` if there is one (does not duplicate).
87
102
 
88
103
  ## Brownfield handling — what does *not* happen
@@ -110,7 +125,8 @@ find .claude/rules -type f | sort
110
125
  # settings.local.json gitignored
111
126
  grep -E '\.claude/settings\.local\.json' .gitignore
112
127
 
113
- # the two cognee servers declared
128
+ # the two project-level cognee aliases written to .mcp.json
129
+ # (cognee-memory is a shared singleton; it does not appear in per-project .mcp.json)
114
130
  python3 -c "import json; print(list(json.load(open('.mcp.json'))['mcpServers'].keys()))"
115
131
  # expected: ['cognee', 'cognee-curated']
116
132
  ```
@@ -130,13 +146,14 @@ claude # fresh session
130
146
  ## Common edge cases
131
147
 
132
148
  - **No remote / private repo:** `.mcp.json` is tracked; do not put secrets in
133
- it. The cognee MCP URLs point at `http://localhost:7777` and `:7730`your
134
- own host, no third-party endpoints.
135
- - **Multiple projects on one host:** safe. The curated box is shared
136
- (singleton); the work store holds each project as its own dataset, keyed by
137
- the project directory basename by default (see
138
- [Selective ingest](./05-selective-ingest.md) for the dataset naming
139
- rules).
149
+ it. The cognee MCP URLs point at local ports on your own host no
150
+ third-party endpoints. The per-project work store runs on a dynamically
151
+ assigned port; `cognee-memory` is `:7777`; `cognee-curated` is `:7730`.
152
+ - **Multiple projects on one host:** safe. Each project has its own physically
153
+ isolated work store container (`mishkan-work-<slug>`) provisioned by
154
+ `ensure-work-store.sh`. The curated box and the `cognee-memory` (`:7777`)
155
+ session-memory box are shared singletons. See [Memory layer](./04-memory-layer.md)
156
+ for the three-pillar layout (D-012).
140
157
  - **Running init twice:** safe. The four artifacts are not overwritten; the
141
158
  curated ensure step is idempotent; rules are re-copied verbatim.
142
159
 
@@ -147,5 +164,5 @@ claude # fresh session
147
164
  selective).
148
165
  - [Orchestration](./03-orchestration.md) — how the main session routes work
149
166
  once init has run.
150
- - [Memory layer](./04-memory-layer.md) — the two cognee stores and what they
151
- hold.
167
+ - [Memory layer](./04-memory-layer.md) — the three cognee stores (work,
168
+ memory, curated) and what they hold.
@@ -9,14 +9,19 @@
9
9
  subagent — confirmed in the official subagent docs. The `Task` tool in a
10
10
  subagent's frontmatter is **inert** when the subagent runs.
11
11
 
12
+ ```mermaid
13
+ flowchart TD
14
+ You(["You — talk"]) --> MS[["MAIN SESSION — leadership / orchestrator<br/>acts as Nehemiah (PM) + Bezalel (CTO)"]]
15
+ MS -->|Task| A["Team Lead / Specialist"]
16
+ MS -->|Task| B["project-specific agent (e.g. aiobi-ops)"]
17
+ MS -->|Task| C["research pipeline"]
18
+ A --> SYN[["main session SYNTHESISES the outputs"]]
19
+ B --> SYN
20
+ C --> SYN
12
21
  ```
13
- You ──talk──▶ MAIN SESSION = leadership ← orchestrator
14
-
15
- ├─Task Team Lead / Specialist ┐
16
- ├─Task→ project-specific agent │ one level deep
17
- └─Task→ research pipeline ┘ siblings, no nesting
18
- then main session SYNTHESISES their outputs
19
- ```
22
+
23
+ *One level deep, siblings, no nesting* — the main session delegates via `Task`, then
24
+ synthesises. A subagent cannot spawn another: its own `Task` tool is inert.
20
25
 
21
26
  Three things follow:
22
27
 
@@ -57,10 +62,44 @@ its own work.
57
62
  | **Migdal** | Infrastructure | `eliashib` (lead), `meshullam` (design), `palal` (systems), `meremoth` (devops), `hanun` (devsecops), `rehum` (SRE) |
58
63
  | **Sefer** | Documentation (pull-based) | `jehoshaphat` (lead), `seraiah` / `joah` / `shevna` (org/project/team), `jehonathan` (publication) |
59
64
 
65
+ ```mermaid
66
+ graph TD
67
+ N[["Nehemiah — PM"]] --- Bz[["Bezalel — CTO"]]
68
+ Bz --> CH["Chosheb<br/>Design / UX"]
69
+ Bz --> PA["Panim<br/>Frontend"]
70
+ Bz --> YA["Yasad<br/>Backend"]
71
+ Bz --> MI["Migdal<br/>Infrastructure"]
72
+ Bz --> SE["Sefer<br/>Docs · pull-based"]
73
+ Bz --> RP["Research pipeline<br/>6 stages · shared"]
74
+ MM["Mishmar — Security<br/>cross-cutting"]
75
+ MM -. gates .-> CH
76
+ MM -. gates .-> PA
77
+ MM -. gates .-> YA
78
+ MM -. gates .-> MI
79
+ MM -. gates .-> SE
80
+ ```
81
+
82
+ *Two orchestrators (the main session embodies both) route to six teams; Mishmar gates
83
+ every team's output; the research pipeline is shared. Within each team: Lead → Specialists
84
+ → QA → Reporter, with QA and Reporter structurally separate from the producing agents.*
85
+
60
86
  **Research pipeline** (6 stages): `jakin` → `ezra` → `caleb` → `shaphan` →
61
87
  `shemaiah` → `baruch`. Each stage is a single-purpose agent. The pipeline is
62
88
  also a skill (`research-pipeline`) the main session can invoke whole.
63
89
 
90
+ ```mermaid
91
+ flowchart LR
92
+ J["jakin<br/>clarify"] --> E["ezra<br/>formulate"]
93
+ E --> Ca["caleb<br/>web research"]
94
+ Ca --> Sh["shaphan<br/>summarise"]
95
+ Sh --> Se["shemaiah<br/>evaluate"]
96
+ Se --> B["baruch<br/>report + Cognee node"]
97
+ E -. curated library already answers .-> B
98
+ ```
99
+
100
+ *Six single-purpose stages. Ezra short-circuits straight to Baruch when the curated
101
+ library already holds the answer — no web research spent.*
102
+
64
103
  Full glossary including each agent's role: [Glossary](./08-glossary.md).
65
104
  Naming rationale: [`docs/design/MISHKAN_agent_aliases.md`](../design/MISHKAN_agent_aliases.md).
66
105
 
@@ -4,56 +4,96 @@
4
4
  > agents query it. This is the layer that makes MISHKAN *accumulate* rather
5
5
  > than just function.
6
6
 
7
- ## Two physically-isolated stores (decision D-007)
8
-
9
- Cognee runs locally on the host as a Docker stack. The core architectural
10
- choice (introduced in commit `418d10a`, documented in
11
- [`docs/design/MISHKAN_decisions.md`](../design/MISHKAN_decisions.md) §D-007):
12
-
13
- ```
14
- shared Ollama (embeddings, local) ─┐ stateless, serves both
15
- shared Postgres server ────────────┤ separate databases
16
-
17
- WORK box :7777 (cognee) CURATED box :7730 (cognee-curated)
18
- own Neo4j + cognee_db own Neo4j + curated_db
19
- ── per-project knowledge ── ── cross-project reference ──
20
- • aiobi-mail (project dataset) • curated_library (96 nodes seed)
21
- claude_code_memory (per-client memory) • read-mostly
22
- UI :7724 · Neo4j :7716/:7709 UI :7734 · Neo4j :7731/:7732
7
+ ## Three physically-isolated pillars (decisions D-007 + D-012)
8
+
9
+ Cognee runs locally on the host as a Docker stack. The current architecture
10
+ has three pillars (D-007 separated work from curated; D-012 further split
11
+ per-project work stores from the shared session-memory box; both documented in
12
+ [`docs/design/MISHKAN_decisions.md`](../design/MISHKAN_decisions.md)):
13
+
14
+ ```mermaid
15
+ flowchart LR
16
+ OLL[("Ollama<br/>embeddings · local · serves all")]
17
+ PG[("Postgres server<br/>cognee_db + curated_db")]
18
+ WORK[("cognee WORK — per-project<br/>mishkan-work-{slug}<br/>embedded Ladybug + LanceDB<br/>own port + volume · isolated")]
19
+ MEM[("cognee-memory :7777<br/>Neo4j + cognee_db<br/>claude_code_memory · shared · never prune")]
20
+ CUR[("cognee-curated :7730<br/>own Neo4j + curated_db<br/>curated_library · shared · read-mostly")]
21
+ OLL -. embeddings .-> WORK
22
+ OLL -. embeddings .-> MEM
23
+ OLL -. embeddings .-> CUR
24
+ PG --- MEM
25
+ PG --- CUR
23
26
  ```
24
27
 
25
- Why two stores and not one?
28
+ *Three physically-isolated pillars (D-007 + D-012). Shared Ollama embeds for all; the
29
+ shared Postgres backs memory + curated; each per-project work store is self-contained
30
+ (embedded Ladybug + LanceDB, own volume) so cross-project reads are impossible by construction.*
31
+
32
+ Why physical separation and not logical datasets?
26
33
 
27
34
  - **PII isolation.** Project ingestion pulls in code and docs that can contain
28
35
  PII (real email addresses in incident reports were the trigger during the
29
- build). The cross-project curated library must stay clean.
30
- - **Logical tags aren't enough on Neo4j Community.** Community Edition allows
31
- only one database per Neo4j instance, and with cognee's access control
32
- disabled (the only mode that works against Neo4j today) all datasets share
33
- one graph. So logical dataset tags commingle in one store. Physical
34
- separation = a separate Neo4j container.
35
-
36
- The `claude_code_memory` dataset is the **per-client session memory**, created
37
- on demand by cognee-mcp when Claude Code connects. **It belongs in the work
38
- store and must never be pruned** (D-007 calls this out explicitly).
36
+ build). The cross-project curated library and session-memory box must stay clean.
37
+ - **`datasets=` filter is advisory-only on Neo4j.** With `ENABLE_BACKEND_ACCESS_CONTROL=false`
38
+ (the only mode that works against Neo4j Community), cognee ignores the
39
+ `datasets=` filter and searches the whole graph verified against cognee
40
+ v1.1.0 / issue #1023. Logical tags alone do not isolate.
41
+ - **Physical separation = topology.** Each project work store is a separate
42
+ container + volume + on-disk graph file. Cross-project reads are impossible
43
+ by construction.
44
+
45
+ The `claude_code_memory` dataset is the **per-client session memory**, held in
46
+ the `cognee-memory` (`:7777`) box. It is shared across all your work — one
47
+ continuous thing, not re-derivable — and **must never be pruned** (D-012).
48
+ Keep it scrubbed of project-specific secrets and PII because it is
49
+ cross-project by nature.
50
+
51
+ ### Per-project work stores (decision D-012)
52
+
53
+ D-007 separated *work* from *curated*. **D-012** extends the same physical-
54
+ separation principle one axis further: each project gets its **own** work store,
55
+ not a shared dataset in one box. The trigger was a confidentiality bug — with
56
+ `ENABLE_BACKEND_ACCESS_CONTROL=false` (required on Neo4j) cognee's `datasets=`
57
+ filter is **advisory only**: it is silently ignored and search runs the whole
58
+ graph, so a query scoped to one project returned another project's data (with a
59
+ live API key among it). Verified against cognee v1.1.0 / issue #1023.
60
+
61
+ The fix: each project runs its own lightweight cognee-mcp container with an
62
+ **embedded Ladybug** graph (no Neo4j) and its own volume — `mishkan-work-<slug>`
63
+ on its own port, provisioned by `ensure-work-store.sh` at `/mishkan-init`.
64
+ Isolation is by topology (container + volume + on-disk graph file), never the
65
+ `datasets=` filter.
66
+
67
+ Each project's `.mcp.json` then carries **three doorways** (the alias is the
68
+ doorway; the backend behind each differs):
69
+
70
+ - **`cognee`** → that per-project Ladybug store — isolated project knowledge.
71
+ - **`cognee-memory`** → the **kept** Neo4j box on `:7777`, now holding only
72
+ `claude_code_memory` — shared per-client session memory, one continuous thing
73
+ across all your work, not re-derivable, so never fragmented per project or
74
+ pruned. (Cross-project by nature → keep it scrubbed of project secrets/PII.)
75
+ - **`cognee-curated`** → the shared reference library (`:7730`).
76
+
77
+ The shared box's old *project* graphs are discarded (re-derivable from tagged
78
+ docs); the box itself is **repurposed** as the session-memory pillar, not retired.
39
79
 
40
80
  ## The data flow
41
81
 
42
- ```
43
- docs / project content curated resources (small, static)
44
- │ │
45
- ▼ ▼
46
- cognee.add() raw files staged add_data_points() ← structured
47
- │ │
48
- cognify() ← LLM extracts entities (no LLM — embeddings only)
49
- │ + relationships │
50
- ▼ ▼
51
- memify() ← embeds the triplet layer memify() optional
52
- │ into the vector store
53
-
54
- search() ← retrieval (vector + graph)
82
+ ```mermaid
83
+ flowchart LR
84
+ subgraph D["docs / project content"]
85
+ ADD["cognee.add()<br/>stage raw files"] --> CFY["cognify()<br/>LLM extracts entities + relationships"]
86
+ CFY --> MFY["memify()<br/>embed triplet layer into vector store"]
87
+ MFY --> SR["search()<br/>retrieval · vector + graph"]
88
+ end
89
+ subgraph C["curated resources (small, static)"]
90
+ ADP["add_data_points()<br/>structured · no LLM"] --> MFY2["memify() (optional)<br/>embeddings only"]
91
+ end
55
92
  ```
56
93
 
94
+ *Two write paths into a store: project docs go `add → cognify` (LLM extraction) `→ memify
95
+ → search`; curated resources are written structurally (`add_data_points`, no LLM — embeddings only).*
96
+
57
97
  Each phase, in words:
58
98
 
59
99
  - **`add`** stages files for processing. The raw file content is stored under
@@ -62,7 +102,13 @@ Each phase, in words:
62
102
  - **`cognify`** is the LLM-heavy step. It chunks each document, calls the LLM
63
103
  to extract entities and relationships as structured output (instructor mode),
64
104
  embeds chunks + entities, and writes to Neo4j + pgvector. This is the step
65
- that costs LLM tokens and runs into rate caps.
105
+ that costs LLM tokens and runs into rate caps. Each work store attaches the
106
+ **MISHKAN ontology** (`ontology.ttl`, the machine form of `ontology.md` — ADR
107
+ D-013) to cognify via the container's env config (`ONTOLOGY_FILE_PATH` +
108
+ `rdflib`/`fuzzy`, staged by `ensure-work-store`): entities whose extracted type
109
+ matches a schema class are validated (`ontology_valid: true`) and enriched with
110
+ parent-class / object-property edges. Fails open — a missing ontology ingests
111
+ unconstrained.
66
112
  - **`memify`** is the enrichment step that runs **after** cognify. The default
67
113
  enrichment embeds the **edge / triplet** layer into the vector store
68
114
  (`EdgeType_relationship_name` and `graph_relationship_ledger` tables in
@@ -77,63 +123,76 @@ by enrichment, never manually.
77
123
 
78
124
  ## The MCP — how agents reach memory
79
125
 
80
- Every MISHKAN-initialised project declares **both** servers in `.mcp.json`:
126
+ Every MISHKAN-initialised project declares **three** servers in `.mcp.json`
127
+ (written by `ensure-work-store.sh` at `/mishkan-init`):
81
128
 
82
129
  ```json
83
130
  {
84
131
  "mcpServers": {
85
- "cognee": { "type": "http", "url": "http://localhost:7777/mcp" },
86
- "cognee-curated": { "type": "http", "url": "http://localhost:7730/mcp" }
132
+ "cognee": { "type": "http", "url": "http://localhost:<per-project-port>/mcp" },
133
+ "cognee-memory": { "type": "http", "url": "http://localhost:7777/mcp" },
134
+ "cognee-curated": { "type": "http", "url": "http://localhost:7730/mcp" }
87
135
  }
88
136
  }
89
137
  ```
90
138
 
91
- So when an agent searches, it can target either store explicitly:
139
+ The `<per-project-port>` is assigned by `ensure-work-store.sh` at init time
140
+ and recorded in the project's `.mcp.json`. It is NOT `:7777`.
141
+
142
+ So when an agent searches, it targets the right pillar explicitly:
92
143
 
93
- - `cognee` — read+write the project's own graph (typical).
144
+ - `cognee` — read+write the project's own isolated Ladybug graph (typical for
145
+ project knowledge retrieval and ingest).
146
+ - `cognee-memory` — read+write per-client session memory (`claude_code_memory`)
147
+ shared across all work. Use when the agent needs to recall or record
148
+ cross-session context.
94
149
  - `cognee-curated` — read the cross-project reference library (typical for
95
150
  Shemaiah cross-referencing curated resources).
96
151
 
97
152
  MCP servers connect at **session start**. A fresh session is needed for
98
153
  `/mishkan-init`-written `.mcp.json` to take effect.
99
154
 
100
- ## Datasets — the logical layer inside each store
155
+ ## Datasets — a label inside a store, NOT an isolation boundary
101
156
 
102
- Datasets are cognee's logical partitioning. In the work store, every project
103
- gets its own dataset (named after the project directory by convention). A
104
- typical work store after a few projects:
157
+ Within a single store, `datasets` is cognee's logical *label* by convention the
158
+ project dir name (plus `claude_code_memory` for per-client session memory). It is
159
+ **not a security boundary**: with access control off (the only mode that works on
160
+ Neo4j), `datasets=[...]` on `cognee.search` is **advisory only** — cognee ignores
161
+ it and searches the whole graph (verified, cognee v1.1.0 / issue #1023). Do not
162
+ rely on it to keep one project's retrieval clean of another's.
105
163
 
106
- ```
107
- datasets (work / cognee_db)
108
- ├── aiobi-mail (14 docs, project knowledge)
109
- ├── claude_code_memory (per-client session memory)
110
- └── <next-project> (created on its first ingest)
111
- ```
164
+ Real isolation is **physical — separate stores**:
112
165
 
113
- In the curated store there is one dataset (`curated_library`) — the cross-project
114
- reference seed.
166
+ - Curated is a separate store from work (**D-007**).
167
+ - Each project is a separate work store from every other (**D-012** — per-project
168
+ embedded Ladybug containers).
115
169
 
116
- To query a specific dataset, pass `datasets=[...]` to `cognee.search`. This is
117
- the only way to keep retrieval *logically* clean within a store; the *physical*
118
- isolation between work and curated is by separate Neo4j containers.
170
+ So with per-project stores, a project's `cognee` store contains only that
171
+ project's data; the dataset label is cosmetic and a cross-project read is
172
+ impossible by construction. The curated store holds one dataset
173
+ (`curated_library`), the cross-project reference seed.
119
174
 
120
175
  ## Visualising the graph
121
176
 
122
177
  Two ways for each store:
123
178
 
179
+ Per-project work stores use an embedded Ladybug graph (no Neo4j, no UI).
180
+ The Neo4j-backed pillars (`cognee-memory` and `cognee-curated`) have UIs:
181
+
124
182
  ### Cognee Graph Explorer UI
125
183
 
126
- - **Work**: `http://localhost:7724`, backend `:7737`.
184
+ - **cognee-memory** (`:7777`, session memory): `http://localhost:7724`, backend `:7737`.
127
185
  Login = `DEFAULT_USER_EMAIL` / `DEFAULT_USER_PASSWORD` from `.env`.
128
- - **Curated**: `http://localhost:7734`, backend `:7733` (added in commit `751f95e`).
186
+ - **cognee-curated** (`:7730`, reference library): `http://localhost:7734`, backend `:7733`
187
+ (added in commit `751f95e`).
129
188
  Login = `DEFAULT_USER_EMAIL` / `DEFAULT_USER_PASSWORD` from `.env.curated`.
130
189
 
131
190
  ### Neo4j Browser (raw graph)
132
191
 
133
- | Store | HTTP | Bolt | Credentials |
192
+ | Pillar | HTTP | Bolt | Credentials |
134
193
  |---|---|---|---|
135
- | Work | `http://localhost:7716` | `bolt://localhost:7709` | `neo4j` + work `GRAPH_DATABASE_PASSWORD` |
136
- | Curated | `http://localhost:7731` | `bolt://localhost:7732` | `neo4j` + curated `GRAPH_DATABASE_PASSWORD` |
194
+ | cognee-memory (`:7777`) | `http://localhost:7716` | `bolt://localhost:7709` | `neo4j` + work `GRAPH_DATABASE_PASSWORD` |
195
+ | cognee-curated (`:7730`) | `http://localhost:7731` | `bolt://localhost:7732` | `neo4j` + curated `GRAPH_DATABASE_PASSWORD` |
137
196
 
138
197
  Important: use the `bolt://` scheme in the browser's connect URL, **not**
139
198
  `neo4j://`. The `neo4j://` scheme triggers routing discovery that fails over an
@@ -163,6 +222,12 @@ Only what you want to look at. The MCP itself doesn't need a tunnel — the
163
222
  agent runs on the host where cognee is, and the cognee MCP listens on
164
223
  `127.0.0.1` already.
165
224
 
225
+ **Per-project work stores have no web UI to tunnel.** They use an embedded
226
+ Ladybug graph (a file, no server) — there is no Graph Explorer or Neo4j browser
227
+ for them. The ports above are the shared **`cognee-memory`** (`:77xx`) and
228
+ **`cognee-curated`** (`:773x`) UIs only. To inspect a project's own graph, use
229
+ cognee's static-HTML `visualize_graph("./graph.html")` export on the host.
230
+
166
231
  ## What gets written when, and what to back up
167
232
 
168
233
  | Layer | Where | Persistence | Back up? |
@@ -177,8 +242,9 @@ volume lives. Standard restic / rsync covers it.
177
242
 
178
243
  ## Configuration anchors
179
244
 
180
- - Work box env: `~/.claude/mishkan/cognee/.env` (gitignored, mode 600).
245
+ - `cognee-memory` box (`:7777`) env: `~/.claude/mishkan/cognee/.env` (gitignored, mode 600).
181
246
  - Curated box env: `~/.claude/mishkan/cognee/.env.curated` (gitignored, mode 600).
247
+ - Per-project work store provisioner: `scripts/ensure-work-store.sh` (run at `/mishkan-init`; idempotent).
182
248
  - Compose entrypoint: `docker-compose.yml` + overlays (`hardening`, `selfhosted`,
183
249
  `ui`, `curated`, `curated-ui`).
184
250
  - Curated singleton helper: `scripts/ensure-curated-box.sh` (idempotent).
@@ -187,8 +253,8 @@ volume lives. Standard restic / rsync covers it.
187
253
 
188
254
  ## See also
189
255
 
190
- - The two-store rationale: [D-007](../design/MISHKAN_decisions.md) and
191
- commit `418d10a`.
256
+ - The three-pillar rationale: [D-007](../design/MISHKAN_decisions.md) (work vs curated split),
257
+ [D-012](../design/MISHKAN_decisions.md) (per-project work stores), commit `418d10a`.
192
258
  - Curated UI overlay: commit `751f95e`.
193
259
  - Storage persistence fix: commit `e24fabf`.
194
260
  - Curated structured ingestion (low-level): commit `086e80e`,
@@ -25,7 +25,8 @@ provider choices.
25
25
 
26
26
  | Store | Contains | Provider recommendation | Why |
27
27
  |---|---|---|---|
28
- | **Work** (`:7777`) | project knowledge, may contain PII | **Local Ollama LLM** (private, no quota), or paid/no-train cloud, or a free cloud you accept training on | every free cloud tier trains on prompts; PII shouldn't leak |
28
+ | **Per-project work store** (per-project port, Ladybug) | project knowledge, may contain PII | **Local Ollama LLM** (private, no quota), or paid/no-train cloud, or a free cloud you accept training on | every free cloud tier trains on prompts; PII shouldn't leak |
29
+ | **cognee-memory** (`:7777`, session memory) | per-client session memory (`claude_code_memory`) | Local Ollama recommended; treat the same as project work — may reference PII indirectly | cross-session memory accumulates project context |
29
30
  | **Curated** (`:7730`) | public reference resources, no PII | Any free cloud (Gemini, NVIDIA catalog, OpenRouter named-free) is fine | nothing sensitive |
30
31
 
31
32
  **Embeddings should be local** in both stores. Bulk ingest fires many embedding
@@ -137,6 +137,29 @@ docker run --rm -u 0 -v mishkan-cognee_cognee_data:/v busybox \
137
137
  docker compose ... up -d --force-recreate cognee-mcp
138
138
  ```
139
139
 
140
+ ## Per-project work store (Ladybug) — reset or stuck pipeline
141
+
142
+ The cognify / lock / Postgres fixes above target the shared **`cognee-memory`**
143
+ box (`mishkan-cognee-*`, Postgres-backed). **Per-project work stores (ADR D-012)
144
+ are different:** each is a `mishkan-work-<slug>` container with an embedded
145
+ Ladybug graph + SQLite + LanceDB — no Postgres, no shared graph. So:
146
+
147
+ - The Postgres `DATASET_PROCESSING_STARTED` lock fix does **not** apply — a stuck
148
+ per-project pipeline lives in that store's own SQLite, inside its own volume.
149
+ - To **reset** a project's work store cleanly (it is re-derivable from tagged
150
+ docs), tear it down, drop its volume, re-provision, and re-ingest:
151
+
152
+ ```bash
153
+ cd ~/theY4NN/<project>
154
+ docker compose -p mishkan-work-<slug> \
155
+ -f ~/.claude/mishkan/cognee/docker-compose.work.yml down -v # -v drops the volume
156
+ WORK_PORT=$(~/.claude/mishkan/scripts/ensure-work-store.sh) # re-provision (fresh graph)
157
+ bash ~/.claude/mishkan/scripts/mishkan-ingest.sh --tagged-only # re-ingest
158
+ ```
159
+
160
+ Never do this to `cognee-memory` (`:7777`) — `claude_code_memory` is **not**
161
+ re-derivable (D-012).
162
+
140
163
  ## Curated library is showing inside the work UI
141
164
 
142
165
  **Symptom** — the Cognee UI at `:7724` (work backend) shows `CuratedResource`
@@ -108,9 +108,17 @@ Each stage is a single-purpose agent. The pipeline is also a skill
108
108
  | Haiku | 14 | QA, Reporters, pure advisors, research summarise/evaluate/report |
109
109
  | **Total** | **45** | |
110
110
 
111
- The mapping is authoritative in
111
+ > *Fable tier dormant.* Claude Fable 5 was briefly assigned to the 8 Migdal+Mishmar
112
+ > specialists (D-002 amend, 2026-06-11) but **suspended 2026-06-12** by an
113
+ > export-control directive; those agents reverted to Sonnet. The routing layer still
114
+ > accepts `fable` as a valid value, so it can be re-enabled if access is restored.
115
+
116
+ The shipped mapping lives in
112
117
  [`payload/mishkan/config/model-routing.yaml`](../../payload/mishkan/config/model-routing.yaml);
113
118
  the hook `payload/mishkan/hooks/model-route.py` injects it at delegation time.
119
+ **You can re-tier any agent** with `mishkan model set <agent|team|all> <tier>` (D-017) —
120
+ overrides land in a `model-routing.local.yaml` overlay that survives updates, take effect
121
+ on the next delegation, and win over the shipped defaults above.
114
122
 
115
123
  ## Key terms
116
124
 
@@ -123,9 +131,10 @@ the hook `payload/mishkan/hooks/model-route.py` injects it at delegation time.
123
131
  | **Cognify** | the LLM-heavy step that extracts entities + relationships from a document and writes them into the graph. |
124
132
  | **Memify** | the enrichment step that runs after cognify and embeds the triplet/edge layer into the vector store. |
125
133
  | **Search** | cognee's retrieval, exposed via MCP. Always pass `datasets=[...]` to scope it. |
126
- | **Work store** | the per-project cognee box (`cognee`, `:7777`). |
134
+ | **Work store** | a per-project cognee-mcp container running an embedded Ladybug graph (no Neo4j), on its own port and volume (`mishkan-work-<slug>`). Provisioned by `ensure-work-store.sh` at `/mishkan-init`. Reached via the `cognee` MCP alias. NOT port `:7777`. |
127
135
  | **Curated store** | the cross-project reference cognee box (`cognee-curated`, `:7730`). Read-mostly. |
128
- | **`claude_code_memory`** | the per-client memory dataset auto-created by cognee-mcp when Claude Code connects to the work store. Never prune it. |
136
+ | **`cognee-memory` (`:7777`)** | the kept Neo4j-backed cognee box repurposed (D-012) to hold only `claude_code_memory` shared per-client session memory. Reached via the `cognee-memory` MCP alias. |
137
+ | **`claude_code_memory`** | the per-client memory dataset held in the `cognee-memory` (`:7777`) box. Shared across all projects; never prune it. |
129
138
  | **`mishkan: ingest`** | the YAML frontmatter tag that marks a doc as eligible for the work store. |
130
139
  | **Throttle** | the in-process LLM rate limiter (`LLM_RATE_LIMIT_*` in `.env`). Per-minute only; does not help with daily caps. |
131
140
  | **Asymmetric delegation** | the rule that stateful ops (`git push`, `ssh`, `sudo`, production `docker exec`, schema migrations, log forensics) stop at the engineer's hands — never executed by an agent. |
@@ -1,7 +1,7 @@
1
1
  # 09 — Dynamic Workflows
2
2
 
3
3
  > Goal: explain when MISHKAN reaches for a dynamic workflow vs ordinary
4
- > Task delegation, the **18 workflows shipped** (10 org-level + 8 team-
4
+ > Task delegation, the **20 workflows shipped** (10 org-level + 10 team-
5
5
  > level), the cost gate, and how dynamic workflows relate to the 30+
6
6
  > craft skills that drive each team's day-to-day work.
7
7
 
@@ -18,8 +18,9 @@ wrong tools at the wrong level:
18
18
 
19
19
  **Dynamic workflows** orchestrate dozens of agents in parallel with
20
20
  typed contracts, adversarial verification, and judge panels. They are
21
- expensive and bounded — hard cap at 10 in the harness. The gate to
22
- add one is high (recurrence + parallelism + repeatable shape).
21
+ expensive and bounded — **20 shipped today: 10 org-level (hard cap 10)
22
+ + 10 team-level (cap 4 per team, 6 teams)**. The gate to add one is high
23
+ (recurrence + parallelism + repeatable shape).
23
24
 
24
25
  **Craft skills** are how each agent ships day-to-day work. They are
25
26
  markdown documents loaded into the agent's context. There are 30+ of
@@ -33,6 +34,17 @@ review, escalation, sprint contribution).
33
34
  workflow nor a craft skill captures the shape, the main session
34
35
  spawns one or more subagents inline.
35
36
 
37
+ ```mermaid
38
+ graph TD
39
+ MS[["main session"]]
40
+ MS --> WF["Dynamic workflows · org + team<br/>workflows/*.js — 20 shipped (10 org + 10 team)<br/>dozens of agents · typed contracts"]
41
+ MS --> CS["Craft skills · team / agent-level<br/>(agent)-(role)-craft/ — 30+<br/>how each agent ships day-to-day"]
42
+ MS --> TD["Task delegation · single shot<br/>inline Task() — the one-off escape hatch"]
43
+ ```
44
+
45
+ *Three orchestration layers — reach for a dynamic workflow only when recurrence +
46
+ parallelism + a repeatable shape all hold; otherwise a craft skill or a one-off Task.*
47
+
36
48
  If you find yourself asking "is THIS a workflow?", apply the gate
37
49
  below. If it doesn't pass all three, it's probably a craft skill
38
50
  addition or a one-shot Task.