@x12i/memorix-docs 1.0.2 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (236) hide show
  1. package/dist/generated/bundle.d.ts +4849 -3
  2. package/dist/generated/bundle.d.ts.map +1 -1
  3. package/dist/generated/bundle.js +7580 -45
  4. package/dist/generated/bundle.js.map +1 -1
  5. package/dist/index.d.ts +2 -2
  6. package/dist/index.d.ts.map +1 -1
  7. package/dist/index.js +6 -4
  8. package/dist/index.js.map +1 -1
  9. package/files/INDEX.md +84 -0
  10. package/files/SEARCH.json +3087 -0
  11. package/files/agent-manifest.json +1785 -0
  12. package/files/apis/memorix-records/overview.md +22 -0
  13. package/files/concepts/abstracts-bindings.md +27 -0
  14. package/files/concepts/agents-inheritance.md +28 -0
  15. package/files/concepts/analysis-objects.md +26 -0
  16. package/files/concepts/composition.md +27 -0
  17. package/files/concepts/decisions-content.md +27 -0
  18. package/files/concepts/effective-metadata.md +27 -0
  19. package/files/concepts/explicit-metadata-installation.md +26 -0
  20. package/files/concepts/intelligence-proposals.md +27 -0
  21. package/files/concepts/mappings-transforms.md +26 -0
  22. package/files/concepts/object-content-category.md +26 -0
  23. package/files/concepts/org-agent-scope.md +27 -0
  24. package/files/concepts/people-union.md +27 -0
  25. package/files/concepts/raw-records.md +26 -0
  26. package/files/concepts/record-revisions.md +26 -0
  27. package/files/concepts/relationship-definitions.md +26 -0
  28. package/files/concepts/relationship-references.md +27 -0
  29. package/files/concepts/resolved-unresolved.md +27 -0
  30. package/files/concepts/sources-and-memory.md +26 -0
  31. package/files/concepts/traversal.md +27 -0
  32. package/files/diagrams/agent-inheritance.md +34 -0
  33. package/files/diagrams/intelligence-journey.md +38 -0
  34. package/files/diagrams/memorix-central-journey.md +51 -0
  35. package/files/diagrams/metadata-lifecycle.md +40 -0
  36. package/files/diagrams/people-union-diagram.md +36 -0
  37. package/files/diagrams/record-content-model.md +43 -0
  38. package/files/diagrams/relationship-traversal.md +40 -0
  39. package/files/diagrams/revision-safe-write.md +39 -0
  40. package/files/examples/analysis-decision-records.md +11 -0
  41. package/files/examples/memorix-record-envelope.md +11 -0
  42. package/files/examples/metadata-pack-install.md +11 -0
  43. package/files/examples/people-abstract-query.md +11 -0
  44. package/files/examples/raw-vs-composed.md +11 -0
  45. package/files/examples/relationship-def-and-ref.md +11 -0
  46. package/files/examples/revision-conflict-response.md +11 -0
  47. package/files/examples/revision-safe-write.md +11 -0
  48. package/files/examples/traversal-resolved-unresolved.md +11 -0
  49. package/files/glossary/abstract.md +9 -0
  50. package/files/glossary/agent.md +9 -0
  51. package/files/glossary/analysis.md +9 -0
  52. package/files/glossary/binding.md +9 -0
  53. package/files/glossary/composition.md +9 -0
  54. package/files/glossary/content-type.md +9 -0
  55. package/files/glossary/data-category.md +9 -0
  56. package/files/glossary/decision.md +9 -0
  57. package/files/glossary/effective-metadata.md +9 -0
  58. package/files/glossary/intelligence-proposal.md +9 -0
  59. package/files/glossary/mapping.md +9 -0
  60. package/files/glossary/memory.md +9 -0
  61. package/files/glossary/object-type.md +9 -0
  62. package/files/glossary/relationship-definition.md +9 -0
  63. package/files/glossary/relationship-reference.md +9 -0
  64. package/files/glossary/revision.md +9 -0
  65. package/files/glossary/scope.md +9 -0
  66. package/files/glossary/source.md +9 -0
  67. package/files/glossary/traversal.md +9 -0
  68. package/files/guides/apply-scope.md +40 -0
  69. package/files/guides/choose-access-method.md +24 -0
  70. package/files/guides/compose-subject-ui.md +24 -0
  71. package/files/guides/define-traverse-relationship.md +26 -0
  72. package/files/guides/diagnose-unresolved.md +24 -0
  73. package/files/guides/evidence-to-decision.md +25 -0
  74. package/files/guides/first-revision-safe-write.md +30 -0
  75. package/files/guides/inspect-effective-metadata.md +30 -0
  76. package/files/guides/query-people-abstract.md +25 -0
  77. package/files/guides/reload-retry-conflict.md +32 -0
  78. package/files/guides/validate-install-pack.md +32 -0
  79. package/files/indexes/books.json +95 -0
  80. package/files/indexes/books.md +17 -0
  81. package/files/indexes/concepts.json +119 -0
  82. package/files/indexes/concepts.md +21 -0
  83. package/files/indexes/diagrams.json +53 -0
  84. package/files/indexes/diagrams.md +10 -0
  85. package/files/indexes/guides.json +71 -0
  86. package/files/indexes/guides.md +13 -0
  87. package/files/indexes/scenarios.json +41 -0
  88. package/files/indexes/scenarios.md +8 -0
  89. package/files/indexes/tutorials.json +47 -0
  90. package/files/indexes/tutorials.md +9 -0
  91. package/files/indexes/use-cases.json +239 -0
  92. package/files/indexes/use-cases.md +41 -0
  93. package/files/knowledge.json +2940 -0
  94. package/files/packs/concept/abstracts-bindings/PACK.md +11 -0
  95. package/files/packs/concept/abstracts-bindings/pack.json +13 -0
  96. package/files/packs/concept/agents-inheritance/PACK.md +11 -0
  97. package/files/packs/concept/agents-inheritance/pack.json +13 -0
  98. package/files/packs/concept/analysis-objects/PACK.md +11 -0
  99. package/files/packs/concept/analysis-objects/pack.json +14 -0
  100. package/files/packs/concept/composition/PACK.md +11 -0
  101. package/files/packs/concept/composition/pack.json +13 -0
  102. package/files/packs/concept/decisions-content/PACK.md +11 -0
  103. package/files/packs/concept/decisions-content/pack.json +13 -0
  104. package/files/packs/concept/effective-metadata/PACK.md +11 -0
  105. package/files/packs/concept/effective-metadata/pack.json +13 -0
  106. package/files/packs/concept/explicit-metadata-installation/PACK.md +11 -0
  107. package/files/packs/concept/explicit-metadata-installation/pack.json +14 -0
  108. package/files/packs/concept/intelligence-proposals/PACK.md +11 -0
  109. package/files/packs/concept/intelligence-proposals/pack.json +13 -0
  110. package/files/packs/concept/mappings-transforms/PACK.md +11 -0
  111. package/files/packs/concept/mappings-transforms/pack.json +13 -0
  112. package/files/packs/concept/object-content-category/PACK.md +11 -0
  113. package/files/packs/concept/object-content-category/pack.json +14 -0
  114. package/files/packs/concept/org-agent-scope/PACK.md +11 -0
  115. package/files/packs/concept/org-agent-scope/pack.json +14 -0
  116. package/files/packs/concept/people-union/PACK.md +11 -0
  117. package/files/packs/concept/people-union/pack.json +13 -0
  118. package/files/packs/concept/raw-records/PACK.md +11 -0
  119. package/files/packs/concept/raw-records/pack.json +14 -0
  120. package/files/packs/concept/record-revisions/PACK.md +11 -0
  121. package/files/packs/concept/record-revisions/pack.json +13 -0
  122. package/files/packs/concept/relationship-definitions/PACK.md +11 -0
  123. package/files/packs/concept/relationship-definitions/pack.json +14 -0
  124. package/files/packs/concept/relationship-references/PACK.md +11 -0
  125. package/files/packs/concept/relationship-references/pack.json +13 -0
  126. package/files/packs/concept/resolved-unresolved/PACK.md +11 -0
  127. package/files/packs/concept/resolved-unresolved/pack.json +13 -0
  128. package/files/packs/concept/sources-and-memory/PACK.md +11 -0
  129. package/files/packs/concept/sources-and-memory/pack.json +13 -0
  130. package/files/packs/concept/traversal/PACK.md +11 -0
  131. package/files/packs/concept/traversal/pack.json +13 -0
  132. package/files/packs/guide/apply-scope/PACK.md +11 -0
  133. package/files/packs/guide/apply-scope/pack.json +13 -0
  134. package/files/packs/guide/choose-access-method/PACK.md +11 -0
  135. package/files/packs/guide/choose-access-method/pack.json +15 -0
  136. package/files/packs/guide/compose-subject-ui/PACK.md +11 -0
  137. package/files/packs/guide/compose-subject-ui/pack.json +13 -0
  138. package/files/packs/guide/define-traverse-relationship/PACK.md +11 -0
  139. package/files/packs/guide/define-traverse-relationship/pack.json +14 -0
  140. package/files/packs/guide/diagnose-unresolved/PACK.md +11 -0
  141. package/files/packs/guide/diagnose-unresolved/pack.json +13 -0
  142. package/files/packs/guide/evidence-to-decision/PACK.md +11 -0
  143. package/files/packs/guide/evidence-to-decision/pack.json +14 -0
  144. package/files/packs/guide/first-revision-safe-write/PACK.md +11 -0
  145. package/files/packs/guide/first-revision-safe-write/pack.json +13 -0
  146. package/files/packs/guide/inspect-effective-metadata/PACK.md +11 -0
  147. package/files/packs/guide/inspect-effective-metadata/pack.json +13 -0
  148. package/files/packs/guide/query-people-abstract/PACK.md +11 -0
  149. package/files/packs/guide/query-people-abstract/pack.json +13 -0
  150. package/files/packs/guide/reload-retry-conflict/PACK.md +11 -0
  151. package/files/packs/guide/reload-retry-conflict/pack.json +13 -0
  152. package/files/packs/guide/validate-install-pack/PACK.md +11 -0
  153. package/files/packs/guide/validate-install-pack/pack.json +13 -0
  154. package/files/packs/scenario/choose-subject-view/PACK.md +11 -0
  155. package/files/packs/scenario/choose-subject-view/pack.json +11 -0
  156. package/files/packs/scenario/pack-valid-not-installed/PACK.md +11 -0
  157. package/files/packs/scenario/pack-valid-not-installed/pack.json +11 -0
  158. package/files/packs/scenario/scope-mismatch/PACK.md +11 -0
  159. package/files/packs/scenario/scope-mismatch/pack.json +11 -0
  160. package/files/packs/scenario/stale-revision-conflict/PACK.md +11 -0
  161. package/files/packs/scenario/stale-revision-conflict/pack.json +11 -0
  162. package/files/packs/scenario/traversal-direction/PACK.md +11 -0
  163. package/files/packs/scenario/traversal-direction/pack.json +11 -0
  164. package/files/packs/scenario/unresolved-relationship/PACK.md +11 -0
  165. package/files/packs/scenario/unresolved-relationship/pack.json +11 -0
  166. package/files/packs/tutorial/analysis-decision-lab/PACK.md +11 -0
  167. package/files/packs/tutorial/analysis-decision-lab/pack.json +14 -0
  168. package/files/packs/tutorial/compare-views/PACK.md +11 -0
  169. package/files/packs/tutorial/compare-views/pack.json +14 -0
  170. package/files/packs/tutorial/conflict-lab/PACK.md +11 -0
  171. package/files/packs/tutorial/conflict-lab/pack.json +13 -0
  172. package/files/packs/tutorial/first-revision-safe-record/PACK.md +11 -0
  173. package/files/packs/tutorial/first-revision-safe-record/pack.json +13 -0
  174. package/files/packs/tutorial/install-pack-lab/PACK.md +11 -0
  175. package/files/packs/tutorial/install-pack-lab/pack.json +13 -0
  176. package/files/packs/tutorial/relationships-lab/PACK.md +11 -0
  177. package/files/packs/tutorial/relationships-lab/pack.json +13 -0
  178. package/files/packs/tutorial/start-with-sample-data/PACK.md +11 -0
  179. package/files/packs/tutorial/start-with-sample-data/pack.json +13 -0
  180. package/files/sample-data/memorix-synthetic-basic.md +13 -0
  181. package/files/scenarios/choose-subject-view.md +36 -0
  182. package/files/scenarios/pack-valid-not-installed.md +33 -0
  183. package/files/scenarios/scope-mismatch.md +44 -0
  184. package/files/scenarios/stale-revision-conflict.md +38 -0
  185. package/files/scenarios/traversal-direction.md +30 -0
  186. package/files/scenarios/unresolved-relationship.md +31 -0
  187. package/files/tutorials/analysis-decision-lab.md +18 -0
  188. package/files/tutorials/compare-views.md +22 -0
  189. package/files/tutorials/conflict-lab.md +18 -0
  190. package/files/tutorials/first-revision-safe-record.md +18 -0
  191. package/files/tutorials/install-pack-lab.md +22 -0
  192. package/files/tutorials/relationships-lab.md +18 -0
  193. package/files/tutorials/start-with-sample-data.md +18 -0
  194. package/files/ui/clients/memorix-studio.md +9 -0
  195. package/files/use-cases/abstract-people-list.md +237 -0
  196. package/files/use-cases/ai-pipeline-fail-closed.md +117 -0
  197. package/files/use-cases/author-pipeline-e2e.md +339 -0
  198. package/files/use-cases/build-callable-service.md +347 -0
  199. package/files/use-cases/builder-acceptance-ops.md +112 -0
  200. package/files/use-cases/choose-access-method.md +111 -0
  201. package/files/use-cases/client-first-read-write.md +374 -0
  202. package/files/use-cases/compose-for-ui.md +97 -0
  203. package/files/use-cases/custom-kind-descriptors.md +137 -0
  204. package/files/use-cases/data-journey-evidence.md +115 -0
  205. package/files/use-cases/discriminated-content-writes.md +152 -0
  206. package/files/use-cases/embed-studio-host.md +165 -0
  207. package/files/use-cases/four-field-links.md +246 -0
  208. package/files/use-cases/http-process-compliance.md +225 -0
  209. package/files/use-cases/install-metadata-pack.md +180 -0
  210. package/files/use-cases/intelligence-propose-install.md +163 -0
  211. package/files/use-cases/knowx-claims-review.md +396 -0
  212. package/files/use-cases/learn-record-envelope.md +233 -0
  213. package/files/use-cases/lists-cursors-pagination.md +196 -0
  214. package/files/use-cases/local-acceptance-g12ops.md +160 -0
  215. package/files/use-cases/memory-hourly-ingest.md +276 -0
  216. package/files/use-cases/metadata-before-write.md +145 -0
  217. package/files/use-cases/migrate-associated-star.md +159 -0
  218. package/files/use-cases/openapi-typed-client.md +78 -0
  219. package/files/use-cases/ops-pack-fixtures.md +168 -0
  220. package/files/use-cases/orient-platform.md +172 -0
  221. package/files/use-cases/pipeline-retry-evidence.md +121 -0
  222. package/files/use-cases/relationship-rebuild-verify.md +128 -0
  223. package/files/use-cases/revision-stale-conflicts.md +194 -0
  224. package/files/use-cases/start-from-zero.md +171 -0
  225. package/files/use-cases/studio-design-types.md +122 -0
  226. package/files/use-cases/studio-explore-subject.md +89 -0
  227. package/files/use-cases/studio-operate-pipelines.md +100 -0
  228. package/files/use-cases/studio-review-intelligence.md +74 -0
  229. package/files/use-cases/studio-scope-isolation.md +67 -0
  230. package/files/use-cases/studio-stale-conflict-ui.md +54 -0
  231. package/files/use-cases/troubleshoot-http-matrix.md +164 -0
  232. package/files/use-cases/trusted-in-process-embed.md +94 -0
  233. package/files/use-cases/workflow-sibling-api.md +212 -0
  234. package/files/visuals/revision-conflict.md +19 -0
  235. package/files/visuals/scope-apply.md +19 -0
  236. package/package.json +4 -3
@@ -0,0 +1,276 @@
1
+ # Ingest an external system on a schedule
2
+
3
+ > **Use case id:** `memory-hourly-ingest`
4
+ > **Goal:** Land exact provider payloads in Memory, then map/write via pipeline.
5
+ > **Audiences:** developers
6
+ > **Tags:** memory, ingest, pipelines
7
+
8
+ ## Reading path
9
+
10
+ 1. **Relationships & Memory** (developers) → chapters: `2-memory-exact-source-landing`
11
+ 1. **Building Pipelines** (developers) → chapters: `1-pipeline-definition`, `13-complete-ops-tutorial`
12
+ 1. **Data Access Patterns** (developers) → chapters: `13-memory-as-the-preferred-ingestion-mechanism`
13
+
14
+ ## From: Relationships & Memory — 2. Memory — exact source landing
15
+
16
+ **Package:** `@x12i/memorix-memory`
17
+ **HTTP:** `/api/memory`
18
+
19
+ Memory is a **collector**, not a promoter. It pulls from declared sources and lands **exact** provider payloads as `<objectType>-raw` through `@x12i/memorix-data`.
20
+
21
+ ### Knowledge vs context
22
+
23
+ | Mode | Retention | Landing | Typical use |
24
+ |---|---|---|---|
25
+ | `knowledge` | Persistent; content-hash dedup | `<objectType>-raw` | Employees, work-items, procedures from ops-source |
26
+ | `context` | Work-scoped TTL / discard | `<objectType>-raw` | Short-lived work-item context pulls |
27
+
28
+ Both modes land **raw**. Promotion to snapshots (or other subject content) is a **pipeline** concern, not Memory.
29
+
30
+ ### Boundaries (non-goals)
31
+
32
+ Memory:
33
+
34
+ - Is **read-only** toward sources (connectors do not mutate the provider).
35
+ - Accepts secrets via **`credentialRef` only** — never raw secrets in request bodies.
36
+ - **Stops at raw handoff** — no promote, no analysis, no relationship discovery inside Memory.
37
+ - Writes operational ledgers in `{orgId}_operational`:
38
+ - `memory-pull-runs`
39
+ - `memory-pull-cursors`
40
+ - `memory-quarantine`
41
+
42
+ ```mermaid
43
+ flowchart LR
44
+ Src[ops-source] -->|credentialRef| Pull[Memory pull]
45
+ Pull --> Raw["*-raw via memorix-data"]
46
+ Pull --> Ledger[operational ledgers]
47
+ Raw --> Pipeline[Pipelines consume]
48
+ ```
49
+
50
+ ### Source declaration (ops pack)
51
+
52
+ Conceptual source shape:
53
+
54
+ ```json
55
+ {
56
+ "id": "ops-source",
57
+ "name": "Ops fixture source",
58
+ "kind": "rest-fixture",
59
+ "defaultBaseUrl": "http://127.0.0.1:5102",
60
+ "credentialRef": "ops-source-local",
61
+ "endpoints": [
62
+ "/employees",
63
+ "/contractors",
64
+ "/work-items",
65
+ "/activity-events",
66
+ "/procedures",
67
+ "/work-items/:id/context"
68
+ ]
69
+ }
70
+ ```
71
+
72
+ Studio Design / Operate surfaces sources and pulls:
73
+
74
+ ![Sources / Memory](../screenshots/03-sources-memory.png)
75
+
76
+ ### credentialRef
77
+
78
+ | Do | Do not |
79
+ |---|---|
80
+ | Reference a configured credential id (`ops-source-local`) | Paste API keys into pull bodies |
81
+ | Rotate credentials in the secret store | Embed secrets in metadata packs checked into git |
82
+ | Fail closed when the ref is missing | Fall back to “temporary plaintext for debugging” in shared orgs |
83
+
84
+ ### Raw landing example
85
+
86
+ After a knowledge pull for employees, expect content addressed roughly as:
87
+
88
+ ```text
89
+ objectType: employees
90
+ contentType: raw
91
+ recordId: <stable subject id from source mapping, e.g. alex’s id>
92
+ data: { …exact provider payload fields per mapping… }
93
+ ```
94
+
95
+ The **exact** bytes (or canonical structured raw) matter later for provenance (`docHash` in KnowX) and for proving what arrived.
96
+
97
+ ### Failure modes
98
+
99
+ | Symptom | Likely cause | Fix |
100
+ |---|---|---|
101
+ | Pull rejected for secrets | Raw secret in body | Use `credentialRef` only |
102
+ | Quarantine entries | Mapping / identity failure on land | Fix source mapping; retry pull |
103
+ | No `*-raw` after “success” | Wrong org/agent scope | Confirm `g12ops*` + `ops` |
104
+ | Expecting snapshots from Memory | Misunderstood boundary | Run the promote pipeline |
105
+
106
+ ### Tutorial-shaped flow
107
+
108
+ See [documentation/tutorials/04-memory-and-pipeline.md](../../../documentation/tutorials/04-memory-and-pipeline.md):
109
+
110
+ ```text
111
+ Source → /api/memory → *-raw → /api/pipelines → snapshots → /api/data
112
+ ```
113
+
114
+ ---
115
+
116
+ ## From: Building Pipelines — 1. Pipeline definition
117
+
118
+ ### Identity
119
+
120
+ - Stable pipeline metadata ID (`pipelineId`) belongs to an agent (e.g. `ops`).
121
+ - Inheritance and child replacement follow multi-agent metadata rules.
122
+ - Pipeline ID is **not** a run ID.
123
+ - Metadata does not hold runtime state; fingerprints and build evidence live on the run.
124
+ - Do not invent unsupported metadata-version machinery beyond what the runtime captures.
125
+
126
+ ### Scope
127
+
128
+ Every run uses:
129
+
130
+ ```ts
131
+ {
132
+ orgId: string; // e.g. g12ops7a3c — complete data universe
133
+ agentIds: string[]; // e.g. ["ops"] — effective metadata only
134
+ }
135
+ ```
136
+
137
+ Scope is captured on the run. Every step receives the same canonical scope. Steps must not reinterpret scope. `agentIds` does not filter organization data.
138
+
139
+ ```http
140
+ x-memorix-org-id: g12ops7a3c
141
+ x-memorix-agent-ids: ops
142
+ ```
143
+
144
+ ### Definition fields (runtime)
145
+
146
+ Core runtime shape (`PipelineDefinition` in `@x12i/memorix-pipeline`):
147
+
148
+ ```ts
149
+ type PipelineDefinition = {
150
+ pipelineId: string;
151
+ version: string;
152
+ description?: string;
153
+ steps: StepDefinition[];
154
+ services?: ServiceRef[];
155
+ schedules?: ScheduleDefinition[];
156
+ retryPolicy?: RetryPolicy;
157
+ dataRefThresholdBytes?: number; // default 1_048_576
158
+ metadata?: Record<string, unknown>;
159
+ };
160
+
161
+ type StepDefinition = {
162
+ stepId: string;
163
+ stepKind: StepKind;
164
+ label?: string;
165
+ dependsOn?: string[];
166
+ inputs?: DataRefExpr[];
167
+ outputKey?: string;
168
+ retryPolicy?: RetryPolicy;
169
+ idempotencyKeyExpr?: string; // required for mutating steps
170
+ port?: string;
171
+ portConfig?: unknown;
172
+ serviceId?: string;
173
+ batchSize?: number;
174
+ atomicBatch?: boolean; // opt-in; rejected unless adapter supports
175
+ timeoutMs?: number;
176
+ disabled?: boolean;
177
+ inputSchema?: unknown;
178
+ outputSchema?: unknown;
179
+ };
180
+ ```
181
+
182
+ Agent-pack metadata may use a lighter authoring shape (`id`, `name`, `trigger`, `mode`, `steps[]` with `kind` / `serviceId` / `mappingId`). The runtime compiles or binds that into the executable definition. Always validate packs with `@x12i/memorix-metadata` before install.
183
+
184
+ ### Inputs and outputs
185
+
186
+ **Inputs:** small inline JSON, DataRef/artifact, record/content address, list/query, abstract reference, relationship reference, Memory source/pull reference, captured context, batch manifest, workflow reference, external trigger payload.
187
+
188
+ **Outputs:** inline result, DataRef, record address/revision, relationship run result, partial batch manifest, external action receipt, workflow/analysis/decision/narrative references, diagnostics.
189
+
190
+ Prefer references to stored results over copying large payloads into the run ledger. Default DataRef threshold: **1 MiB**.
191
+
192
+ ### DAG and dependencies
193
+
194
+ Rules:
195
+
196
+ - Unique `stepId`s
197
+ - `dependsOn` edges; topological execution
198
+ - Fan-out / fan-in and parallelism for independent branches
199
+ - Cycle rejection; unreachable steps rejected at validate
200
+ - Output-to-input via `inputs` DataRef expressions (`fromStep`, `key`, `path`)
201
+ - Disabled steps skipped
202
+
203
+ Loops are not a graph feature. Repeated work uses batches, retries, schedules, or an explicit redesign — not hidden cycles.
204
+
205
+ ### Control flow (supported)
206
+
207
+ Sequential execution · parallel independent branches · per-item batch work · short-circuit on terminal failure · continue-on-item-failure for batches · optional/disabled steps · cancellation.
208
+
209
+ **Unsupported as global features:** cross-service distributed transactions, hidden graph cycles, silent compensation without an explicit step. Atomic batch is **opt-in** and only when the backing port declares support.
210
+
211
+ ---
212
+
213
+ ## From: Building Pipelines — 13. Complete ops tutorial
214
+
215
+ End-to-end non-cyber drill. Use a **fresh** `g12ops*` org every acceptance run.
216
+
217
+ 1. **Define procedure source** in ops metadata (`ops-source`).
218
+ 2. **Define connector** (Memory collect; `credentialRef` only).
219
+ 3. **Define knowledge-ingestion pipeline** (`memory-pull` → raw → map → validate → write → relationships).
220
+ 4. **Pull procedures**; confirm raw for **PROC-2**.
221
+ 5. **Inspect raw** in Explore / data API (`procedures-raw`).
222
+ 6. **Promote** selected procedure snapshot via map/validate/`write-record` — never automatic.
223
+ 7. **Pull work-item context** for **WI-100** (context mode as configured).
224
+ 8. **Normalize** employee/contractor data (mapping and/or normalize service).
225
+ 9. **List People** abstract (employees + contractors including **Alex**).
226
+ 10. **Materialize assignments** (relationship discover + write).
227
+ 11. **Create work-item ↔ procedure relationship** (WI-100 ↔ PROC-2).
228
+ 12. **Run enrichment** (priority completion / enrich service).
229
+ 13. **Produce analysis** (`analyze` + `analysisType`).
230
+ 14. **Produce decision** (`decide` + `decisionType`).
231
+ 15. **Generate narrative** (recorded AI adapter acceptable).
232
+ 16. **Update workflow** sibling with exact revision.
233
+ 17. **Trigger controlled relationship failure** (e.g. break a definition temporarily).
234
+ 18. **Retry relationship step only**; confirm records not duplicated.
235
+ 19. **Restart runtime** during another run.
236
+ 20. **Resume** without repeating completed work.
237
+ 21. **Inspect captured context** artifacts vs current data.
238
+ 22. **Inspect artifacts** (hashes, sizes, step association).
239
+ 23. **Inspect final records/links** in Explore — four-field links, no copied payloads.
240
+ 24. **Run scheduled** invocation; confirm schedule history.
241
+ 25. **Run on-demand** invocation with explicit input; confirm distinct `runId`.
242
+
243
+ Tutorial companion narrative: [documentation/tutorials/04-memory-and-pipeline.md](../../../documentation/tutorials/04-memory-and-pipeline.md).
244
+
245
+ ---
246
+
247
+ ## From: Data Access Patterns — 13. Memory as the preferred ingestion mechanism
248
+
249
+ When data comes from an external source, land it through Memory first.
250
+
251
+ ```text
252
+ source (ops-source)
253
+ → Memory pull (credentialRef, read-only connector)
254
+ → exact raw V2 (*-raw)
255
+ → downstream pipeline maps / validates / writes subject content
256
+ → optional relationship materialization
257
+ ```
258
+
259
+ **Not:** connector → snapshot collection directly.
260
+
261
+ Covered concerns: scheduled knowledge pulls, on-demand context pulls, source cursor, exact raw payload, content-hash dedup, TTL/discard for context mode, quarantine, failure followed by successful later pull, `credentialRef` only, read-only source interaction.
262
+
263
+ Memory stops at raw handoff. Promote, analysis, and relationship discovery belong to pipelines and the relationship engine — not the connector.
264
+
265
+ **Best:** Memory pull.
266
+ **Forbidden:** writing `*-raw` or snapshots from a random script that skips Memory and directory rules; secrets in the request body.
267
+
268
+ ---
269
+
270
+ ## Also see
271
+
272
+ - **Building Services** (`07-services`) — When to build a service, handlers, pipeline invoke, and core-service / ports-manager / api-live-view.
273
+
274
+ ---
275
+
276
+ _Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._
@@ -0,0 +1,145 @@
1
+ # Discover effective metadata before writing
2
+
3
+ > **Use case id:** `metadata-before-write`
4
+ > **Goal:** Load views/writes from /api/metadata so clients do not invent shapes.
5
+ > **Audiences:** developers
6
+ > **Tags:** metadata, http
7
+
8
+ ## Reading path
9
+
10
+ 1. **Runtime Data Client** (developers) → chapters: `3-metadata-discovery-for-runtime-clients`
11
+ 1. **Metadata, Agents, Abstracts & Mappings** (developers) → chapters: `7-views-writes-and-the-client-effective-metadata-pattern`, `9-install-and-publish-never-auto`
12
+
13
+ ## From: Runtime Data Client — 3. Metadata discovery for runtime clients
14
+
15
+ Before constructing reads and writes, discover **effective** metadata for your scope.
16
+
17
+ Clients typically need:
18
+
19
+ - Effective object types and content types
20
+ - Data categories and routing implications
21
+ - Discriminator metadata (path, required/default)
22
+ - Property / schema / view descriptors for UI and validation
23
+ - Lifecycle rules where exposed
24
+ - Abstract types and bindings
25
+ - Relationship definitions
26
+ - Mapping / write capabilities relevant to your job
27
+
28
+ ### Raw vs effective
29
+
30
+ | Kind | Meaning for clients |
31
+ |---|---|
32
+ | Raw pack metadata | What was installed/published |
33
+ | **Effective** metadata | What this `orgId` + `agentIds[]` resolves to *now* |
34
+
35
+ Prefer effective endpoints for runtime addressing. Inheritance provenance and multi-agent conflict resolution are server concerns; clients should treat conflicts as metadata errors to surface, not silently pick a winner.
36
+
37
+ ### Caching guidance
38
+
39
+ - **You may** cache effective metadata for the duration of one user operation or short request batch.
40
+ - **You must** invalidate when the org’s metadata fingerprint/checksum changes, or after install/publish.
41
+ - **You must not** hardcode Mongo collection names derived from types.
42
+ - On metadata validation errors, fail closed — do not guess addresses.
43
+
44
+ ### Practical flow
45
+
46
+ 1. `GET /api/openapi.json` — learn operation IDs and schemas.
47
+ 2. `GET /api/metadata/...` — resolve kinds, object/content types, abstracts, views (exact paths in OpenAPI).
48
+ 3. Confirm discriminator defaults before writing multi-instance content.
49
+ 4. Drive list columns and forms from view/property descriptors — do not invent host-side schemas.
50
+
51
+ ### Ops example
52
+
53
+ For `g12opsabc123` + `ops`, discover that `work-items` exposes `snapshots`, `workflow`, `analysis`, `decisions`; that `People` is an abstract over `employees` ∪ `contractors`; that relationship definition `employee-owns-work-item` exists. Only then address `wi-100` or Alex.
54
+
55
+ ```bash
56
+ curl -sS \
57
+ -H "x-memorix-org-id: $ORG" \
58
+ -H "x-memorix-agent-ids: $AGENTS" \
59
+ "$BASE/api/metadata/kinds" | head
60
+ ```
61
+
62
+ ---
63
+
64
+ ## From: Metadata, Agents, Abstracts & Mappings — 7. Views, writes, and the client effective-metadata pattern
65
+
66
+ Integrators should not invent list columns or write forms in the host. Load effective metadata, then drive UI from descriptors.
67
+
68
+ ### Pattern
69
+
70
+ ```text
71
+ 1. Resolve effective metadata for (orgId, agentIds[])
72
+ 2. Read views / writes / objectTypes / abstracts / custom kinds
73
+ 3. Render UI columns and forms from those descriptors
74
+ 4. Address data only with concrete types (or abstract reverse-write)
75
+ 5. Never bypass the service with direct Mongo domain writes
76
+ ```
77
+
78
+ Example client sketch:
79
+
80
+ ```ts
81
+ const effective = await fetch("/api/metadata/effective", {
82
+ headers: {
83
+ "x-memorix-org-id": orgId, // g12ops…
84
+ "x-memorix-agent-ids": "ops",
85
+ },
86
+ }).then((r) => r.json());
87
+
88
+ const employeeSnapshotWrite = effective.writes.find(
89
+ (w) => w.id === "ops-employees-snapshots",
90
+ );
91
+
92
+ // Drive form identity / required fields from employeeSnapshotWrite
93
+ // Then POST/PATCH via /api/data against objectType: "employees", contentType: "snapshots"
94
+ ```
95
+
96
+ ### Built-in views / writes vs custom kinds
97
+
98
+ Prefer built-in `views` and `writes` when the need is first-class Memorix shape or Studio-compatible UI composition. Use **custom kinds** (next chapter) when another client owns a descriptor vocabulary the platform should only store and resolve.
99
+
100
+ ---
101
+
102
+ ## From: Metadata, Agents, Abstracts & Mappings — 9. Install and publish — never auto
103
+
104
+ Hard product behavior: **Intelligence may propose metadata. Install/publish is always an explicit human or API step.**
105
+
106
+ ![Explicit metadata installation](../screenshots/19-explicit-metadata-installation.png)
107
+
108
+ ### Why never auto-install
109
+
110
+ - Metadata changes change what engines will run next.
111
+ - Accidental install on a shared org is a production incident.
112
+ - Proposals can be wrong, incomplete, or unsafe — review is the product.
113
+
114
+ ### Routes
115
+
116
+ | Route | Meaning |
117
+ |---|---|
118
+ | `POST /api/metadata/validate` | Check a pack without making it live |
119
+ | `POST /api/metadata/install` | Install into the org metadata DB for the scoped agents |
120
+ | `POST /api/metadata/publish` | Publish / promote a validated set per product policy |
121
+ | `GET /api/metadata/packs` | List known packs |
122
+
123
+ `@x12i/memorix-metadata-runtime` is the in-process helper behind those routes. Host apps that only talk HTTP never need to import it.
124
+
125
+ ### Ops acceptance flow
126
+
127
+ 1. Start fresh `g12ops*` + `agentIds: ["ops"]`.
128
+ 2. Validate the ops pack (or the subset you are changing).
129
+ 3. Install explicitly.
130
+ 4. Fetch `/api/metadata/effective` and confirm `People`, `work-items`, relationship definitions, sources.
131
+ 5. Only then run Memory / pipelines / relationship verify.
132
+
133
+ If an intelligence proposal looks wrong, **reject it**. Do not install “to see what happens” on a shared org.
134
+
135
+ Full propose / profile / artifact flow (how proposals are produced and reviewed): [Structural Intelligence](../../10-intelligence/developers/BOOK.md). This book only owns the **install boundary**: proposals are never live until an explicit install/publish.
136
+
137
+ ---
138
+
139
+ ## Also see
140
+
141
+ - **Structural Intelligence** (`10-intelligence`) — Profile data, proposals with evidence, human install boundary.
142
+
143
+ ---
144
+
145
+ _Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._
@@ -0,0 +1,159 @@
1
+ # Migrate associated* present format to V2
2
+
3
+ > **Use case id:** `migrate-associated-star`
4
+ > **Goal:** Reject associated* payloads and migrate envelopes with memorix-format tooling.
5
+ > **Audiences:** developers
6
+ > **Tags:** migration, format
7
+
8
+ ## Reading path
9
+
10
+ 1. **Migration & Troubleshooting** (developers) → chapters: `1-present-memorix-record-2`, `2-associated-rejection-what-v2-forbids`, `12-associated-migration-table-detail`, `14-sample-migrate-cli-session`
11
+
12
+ ## From: Migration & Troubleshooting — 1. Present → memorix-record/2
13
+
14
+ ### CLI
15
+
16
+ ```bash
17
+ npx memorix-format migrate-to-v2 ./present.json \
18
+ --object-type=work-items \
19
+ --content-type=snapshots \
20
+ --data-category=entity \
21
+ --issuer=ops-source
22
+ ```
23
+
24
+ ### Library
25
+
26
+ `migratePresentToV2(present, options) → { ok, v2, extractedRelationships, warnings }`
27
+
28
+ | Output | Meaning |
29
+ |---|---|
30
+ | `v2` | Clean V2 envelope — **no** copied associated payloads |
31
+ | `extractedRelationships` | Out-of-band candidates — **not** auto-persisted |
32
+ | `warnings` | Timestamp defaults, dropped roots, quarantine notes, … |
33
+ | `ok` | A V2 envelope was produced (may still need validate + link materialization) |
34
+
35
+ Always:
36
+
37
+ ```bash
38
+ npx memorix-format validate-v2 ./out-v2.json
39
+ # or validateMemorixRecordV2(result.v2)
40
+ ```
41
+
42
+ ### Behavior highlights (Format §§18–19)
43
+
44
+ - Copied related payload removed (`droppedPayloadKeys` on extracted candidates).
45
+ - Resolvable `recordId` links preserved out-of-band (`mapped` / pending metadata).
46
+ - Unresolvable associations quarantined or dropped with status — not silently invented as subjects.
47
+ - `_extIdentifier` → primary identifier (`kind` + `value`).
48
+ - Missing timestamps default with warnings.
49
+ - Subject `_graphRuns` / similar dropped (ops ledgers own history).
50
+ - Present format remains **migration input**, not target truth.
51
+
52
+ Optional map when you know definition ids:
53
+
54
+ ```ts
55
+ associatedPropertyMap: {
56
+ associatedPeople: "people-assigned-to-work-items",
57
+ }
58
+ ```
59
+
60
+ ---
61
+
62
+ ## From: Migration & Troubleshooting — 2. `associated*` rejection (what V2 forbids)
63
+
64
+ ### Present world (legacy)
65
+
66
+ Root buckets like `associatedPeople` often held **copied** related records (names, titles, nested `.data`). That caused staleness and unsafe writes.
67
+
68
+ ### V2 / production writer world
69
+
70
+ | Shape | Format validate | Production writer |
71
+ |---|---|---|
72
+ | Copied payloads under `associated*` | **Reject** (`V2-ASSOC-COPIED`) | Reject (`WRITE_G4_*`) |
73
+ | Lightweight refs only (`relationshipDefinitionId` + `recordId`) | May be format-valid (Option A / migration compat) | **Not** authoritative storage — prefer central links |
74
+ | Four-field relationship link in category store | N/A (separate collection) | **Authoritative** navigation |
75
+
76
+ ```json
77
+ // BAD — copied payload (rejected)
78
+ {
79
+ "associatedPeople": [
80
+ {
81
+ "recordId": "employee-10",
82
+ "title": "Alex",
83
+ "data": { "email": "alex@example.com" }
84
+ }
85
+ ]
86
+ }
87
+ ```
88
+
89
+ ```json
90
+ // GOOD — central link (conceptual)
91
+ {
92
+ "relationshipId": "…deterministic…",
93
+ "relationshipDefinitionId": "people-assigned-to-work-items",
94
+ "fromRecordId": "wi-100",
95
+ "toRecordId": "employee-10"
96
+ }
97
+ ```
98
+
99
+ Migration extracts candidates **out-of-band**. You must still run relationship discover/verify/rebuild (or pipeline relationship-write) to persist links. Do not paste candidates back onto the V2 snapshot as fat `associated*`.
100
+
101
+ ---
102
+
103
+ ## From: Migration & Troubleshooting — 12. Associated* migration table (detail)
104
+
105
+ | Input | V2 record | `extractedRelationships` |
106
+ |---|---|---|
107
+ | Item with `recordId` (+ copied fields) | `associated*` removed | status `mapped` or `pending-metadata-resolution`; `droppedPayloadKeys` lists copies |
108
+ | Item without target `recordId` | removed | `quarantined-missing-recordId` |
109
+ | Non-array `associated*` | removed | warning; skipped |
110
+
111
+ Copied payload is **never** written onto V2. Resolvable links are **never** silently discarded — they leave as out-of-band candidates.
112
+
113
+ Success vs failure:
114
+
115
+ - Migration can succeed **with warnings** when a V2 envelope is produced (`ok: true`) even if relationships are pending/quarantined.
116
+ - Callers must still `validateMemorixRecordV2(result.v2)`.
117
+ - Migration does not throw merely because an association target is missing; it quarantines.
118
+
119
+ Gate G4 reminder: authoritative physical relationship storage is the centralized org-prefixed per-category `relationships` collection — not Option A roots on snapshots.
120
+
121
+ ---
122
+
123
+ ## From: Migration & Troubleshooting — 14. Sample migrate CLI session
124
+
125
+ ```bash
126
+ # 1) migrate
127
+ npx memorix-format migrate-to-v2 ./fixtures/present-wi-100.json \
128
+ --object-type=work-items \
129
+ --content-type=snapshots \
130
+ --data-category=entity \
131
+ --issuer=ops-source \
132
+ > ./out/migrate-wi-100.json
133
+
134
+ # 2) extract v2 document from tool output (shape depends on CLI pretty-print)
135
+ # validate the v2 object:
136
+ npx memorix-format validate-v2 ./out/wi-100-v2.json
137
+
138
+ # 3) write via service into fresh org
139
+ ORG=g12opsmig01
140
+ H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: ops")
141
+ curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
142
+ -H "content-type: application/json" \
143
+ -d '{"packId":"ops","confirm":true}'
144
+ curl -s "${H[@]}" -X PUT "$SERVICE/api/data/content" \
145
+ -H "content-type: application/json" \
146
+ -d @./out/wi-100-write-body.json
147
+ ```
148
+
149
+ Do not skip validate. Do not write present envelopes to the data plane as if they were V2.
150
+
151
+ ---
152
+
153
+ ## Also see
154
+
155
+ - **Format for Dummies** (`01-format-for-dummies`) — Mental model of the record envelope — OT/CT, composition, discriminators, and what is not stored on the record.
156
+
157
+ ---
158
+
159
+ _Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._
@@ -0,0 +1,78 @@
1
+ # Orient on OpenAPI / typed clients
2
+
3
+ > **Use case id:** `openapi-typed-client`
4
+ > **Goal:** Find OpenAPI generation workflow and keep clients aligned with the service.
5
+ > **Audiences:** developers
6
+ > **Tags:** openapi, client
7
+
8
+ ## Reading path
9
+
10
+ 1. **Workflow & Studio** (developers) → chapters: `8-relation-to-data-plane-and-openapi`, `12-openapi-generation-workflow`
11
+ 1. **Platform Overview** (developers) → chapters: `4-api-families`
12
+
13
+ ## From: Workflow & Studio — 8. Relation to data plane and OpenAPI
14
+
15
+ | Need | Prefer |
16
+ |---|---|
17
+ | Generic content CRUD / list / compose | `/api/data/*` |
18
+ | Operator workflow edit | `/api/workflow` |
19
+ | Metadata design | `/api/metadata/*` |
20
+ | Runs / attempts | `/api/pipelines/*` |
21
+ | Contract discovery | `GET /api/openapi.json` |
22
+
23
+ Workflow writes still obey V2 validation, directory identity, and revision preconditions. The dedicated route does not create a side channel around the data plane — it is a **narrow façade** for a common operator content type.
24
+
25
+ ---
26
+
27
+ ## From: Workflow & Studio — 12. OpenAPI generation workflow
28
+
29
+ 1. Run service locally (`dev-stack` or `memorix-service` alone).
30
+ 2. Fetch `GET /api/openapi.json`.
31
+ 3. Generate TypeScript (or other) clients with your org’s generator.
32
+ 4. Prefer generated `workflowGet` / `workflowPut` operationIds over hand-rolled paths.
33
+ 5. Regenerate when service OpenAPI changes — do not forever-pin a stale client against Studio.
34
+
35
+ Smoke after regenerate:
36
+
37
+ ```bash
38
+ curl -s "$SERVICE/api/openapi.json" | jq '.paths["/api/workflow"] | keys'
39
+ ```
40
+
41
+ Expect `get` and `put`. Treat other workflow-adjacent needs (list by assignee, bulk assign) as **product features on top of** sibling content — not as a reason to invent a second store.
42
+
43
+ ---
44
+
45
+ ## From: Platform Overview — 4. API families
46
+
47
+ Base: running `memorix-service`. OpenAPI: `GET /api/openapi.json`.
48
+
49
+ | Prefix | Purpose |
50
+ |---|---|
51
+ | `/api/metadata` | Agents, types, abstracts, install/publish, effective resolution |
52
+ | `/api/data` | Content get/list/compose, abstract get/list/patch, writes |
53
+ | `/api/relationships` | Verify/rebuild/query (no ad-hoc link PUT/DELETE) |
54
+ | `/api/memory` | Pulls, cursors, quarantine (`credentialRef` only for secrets) |
55
+ | `/api/pipelines` | Definitions, runs, retries, artifacts |
56
+ | `/api/intelligence` | Profiles and proposals (never auto-installs) |
57
+ | `/api/workflow` | Workflow sibling content + assignments |
58
+ | `/api/operations` | Operational health / ops helpers |
59
+
60
+ ### Writes
61
+
62
+ Require scope + revision precondition (`absent` or exact `revision`). Stale revision → conflict (`409`).
63
+
64
+ ### Prefer HTTP over libraries
65
+
66
+ Remote hosts should call **service HTTP**. Install `@x12i/memorix-data` / relationships / pipeline only inside the same trusted process boundary as the platform. Never write domain records with raw Mongo drivers.
67
+
68
+ Deep dive: [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md) · [Data Access Patterns](../../04-data-access-patterns/developers/BOOK.md).
69
+
70
+ ---
71
+
72
+ ## Also see
73
+
74
+ - **Runtime Data Client** (`02-runtime-data-client`) — Connection, scope, exact reads, writes, compose, abstracts, and client anti-patterns through Memorix Service / OpenAPI.
75
+
76
+ ---
77
+
78
+ _Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._