enigma-memory 0.1.11 → 0.1.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -0
- package/apps/cli/bin/enigma.mjs +362 -10
- package/deploy/SIMULATION.md +152 -0
- package/deploy/docker-compose.local-production-simulation.yml +237 -0
- package/deploy/docker-compose.production.example.yml +19 -0
- package/deploy/kms-mock.mjs +64 -0
- package/deploy/nginx.local-production-simulation.conf +33 -0
- package/deploy/siem-mock.mjs +50 -0
- package/docs/benchmark-attestation-network.md +488 -0
- package/docs/benchmark-reproducibility.md +19 -2
- package/docs/blockchain-only-mechanisms.md +388 -0
- package/docs/client-connectors.md +512 -0
- package/docs/demo-proof-network.md +275 -0
- package/docs/developer-ecosystem.md +47 -4
- package/docs/developer-proof-quickstart.md +325 -0
- package/docs/enigma-memory-ready-conformance.md +376 -0
- package/docs/enterprise-proof-control-plane.md +365 -0
- package/docs/install-anywhere.md +517 -0
- package/docs/market-category-narrative.md +398 -0
- package/docs/memory-drive-health-model.md +649 -0
- package/docs/memory-drive-strategy.md +458 -0
- package/docs/memory-passport-standard.md +445 -0
- package/docs/novelty-invention-candidates.md +161 -0
- package/docs/privacy-ledger-model.md +229 -0
- package/docs/proof-network-build-notes.md +240 -0
- package/docs/proof-network-claim-boundaries.md +318 -0
- package/docs/proof-network-dashboard-spec.md +773 -0
- package/docs/proof-network-glossary.md +27 -0
- package/docs/proof-network-launch-plan.md +421 -0
- package/docs/proof-network-operator-protocol.md +432 -0
- package/docs/proof-network-roadmap.md +431 -0
- package/docs/proof-network-test-plan.md +216 -0
- package/docs/proof-network-threat-model.md +373 -0
- package/docs/proof-network.md +257 -0
- package/docs/sdk-api.md +132 -10
- package/docs/solana-devnet-acceptance.md +226 -0
- package/docs/solana-proof-rail.md +453 -0
- package/examples/ci/github-actions.yml +6 -3
- package/examples/proof-network-anchor.json +37 -0
- package/examples/proof-network-attestation.json +35 -0
- package/examples/proof-network-grant.json +27 -0
- package/examples/proof-network-packet.json +71 -0
- package/package.json +42 -3
- package/packages/mcp-server/src/index.js +1 -1
- package/packages/proof-network/src/index.js +570 -0
- package/scripts/build-hosted-api-key-lifecycle.mjs +1 -1
- package/scripts/build-hosted-customer-lifecycle.mjs +1 -1
- package/scripts/build-installer-assets.mjs +1 -1
- package/scripts/build-proof-network-packet.mjs +213 -0
- package/scripts/run-standard-memory-benchmarks.mjs +1 -1
- package/scripts/simulate-production-env.mjs +210 -0
- package/scripts/verify-registry-install.mjs +1 -0
- package/scripts/wait-for-backend-ready.mjs +101 -0
- package/specs/goal-completion-audit-v1.schema.json +1 -0
- package/specs/proof-network-anchor-batch-v1.schema.json +125 -0
- package/specs/proof-network-benchmark-attestation-v1.schema.json +103 -0
- package/specs/proof-network-capability-grant-v1.schema.json +132 -0
- package/specs/proof-network-packet-v1.schema.json +171 -0
|
@@ -0,0 +1,649 @@
|
|
|
1
|
+
# Memory drive health model
|
|
2
|
+
|
|
3
|
+
This spec defines a SMART-style health model for an AI memory drive. It turns local memory operations into public-safe health signals an operator can inspect, alert on, and optionally commit into proof-network receipts without exposing private payloads, connector bodies, identity labels, secret material, or provider bodies.
|
|
4
|
+
|
|
5
|
+
## Product goal
|
|
6
|
+
|
|
7
|
+
Give every memory drive a compact health panel with the same operational feel as SSD SMART:
|
|
8
|
+
|
|
9
|
+
- identify silent decay before retrieval quality drops;
|
|
10
|
+
- separate correct forgetting from accidental loss;
|
|
11
|
+
- show when derived indexes, summaries, receipts, or connector sync state are stale;
|
|
12
|
+
- quantify token savings without storing private text;
|
|
13
|
+
- provide enough public-safe evidence to anchor health snapshots later.
|
|
14
|
+
|
|
15
|
+
## SMART analogy
|
|
16
|
+
|
|
17
|
+
The model borrows SSD SMART's product pattern, not its hardware fields. Each attribute is a bounded, locally measured warning signal with a status, observed value, threshold, evidence ref, and repair action.
|
|
18
|
+
|
|
19
|
+
| SSD SMART idea | Memory-drive analogue | Failure mode it catches |
|
|
20
|
+
| --- | --- | --- |
|
|
21
|
+
| Wear and age indicators | Freshness | Active memories have not been revalidated or touched inside policy. |
|
|
22
|
+
| Reallocated or duplicate sectors | Duplicate rate | Repeated facts inflate context and create retrieval ambiguity. |
|
|
23
|
+
| Pending sector errors | Tombstone risk | Deleted refs can still be replayed, derived, exported, or retrieved. |
|
|
24
|
+
| Offline uncorrectable sectors | Stale derived artifacts | Indexes, summaries, exports, or proof packets are built from old roots. |
|
|
25
|
+
| Read error rate | Retrieval hit rate | Expected public-safe evidence refs are not found at serving top-k. |
|
|
26
|
+
| Throughput efficiency | Token reduction | Context compression is weak or harms retrieval quality. |
|
|
27
|
+
| Media integrity scan | Leakage scan | Forbidden payload fields or secret-like values enter artifacts. |
|
|
28
|
+
| Power-on audit trail | Receipt coverage | Active refs lack receipt lineage or inclusion evidence. |
|
|
29
|
+
| Interface CRC errors | Connector health | Sync cursors, imports, or exports are lagging or discontinuous. |
|
|
30
|
+
| Unsafe shutdown / split brain | Sync fork risk | Replicas disagree on the current root or capability state.
|
|
31
|
+
|
|
32
|
+
## Non-goals and privacy boundaries
|
|
33
|
+
|
|
34
|
+
- Do not store private payloads, connector bodies, identity labels, secret material, provider bodies, or other denylisted values in a health report.
|
|
35
|
+
- Do not require network calls. Health reports are computed from local metadata, receipts, hashes, counters, timestamps, connector cursors, and benchmark summaries already available on disk.
|
|
36
|
+
- Do not claim provider deletion, model forgetting, compliance certification, or live-chain settlement from this report alone.
|
|
37
|
+
- Do not put customer-specific names in examples. Use stable public-safe refs such as `drive_ref`, `namespace_ref`, `connector_ref`, and SHA-256 roots.
|
|
38
|
+
|
|
39
|
+
## Health score shape
|
|
40
|
+
|
|
41
|
+
A memory drive reports one top-level score plus individual SMART-like attributes. The top-level score is intentionally conservative: a critical metric caps the whole drive score even if other metrics are healthy.
|
|
42
|
+
|
|
43
|
+
| Status | Score range | Meaning | Operator action |
|
|
44
|
+
| --- | ---: | --- | --- |
|
|
45
|
+
| `healthy` | 90-100 | No active corrective work required. | Keep normal monitoring cadence. |
|
|
46
|
+
| `watch` | 75-89 | Degradation is visible but not urgent. | Schedule compaction, sync, or benchmark review. |
|
|
47
|
+
| `degraded` | 50-74 | User-visible retrieval or governance risk is likely. | Run targeted repair within the current maintenance window. |
|
|
48
|
+
| `critical` | 0-49 | Trust boundary may be broken or retrieval quality may be materially wrong. | Stop publishing proof artifacts for the affected scope until remediated. |
|
|
49
|
+
|
|
50
|
+
Recommended aggregate formula:
|
|
51
|
+
|
|
52
|
+
1. Compute every metric as a normalized `score` from 0 to 100.
|
|
53
|
+
2. Set `overall_score` to the minimum of:
|
|
54
|
+
- weighted average across metrics;
|
|
55
|
+
- `70` if any metric is `degraded`;
|
|
56
|
+
- `49` if any metric is `critical`.
|
|
57
|
+
3. Report `overall_status` from the final `overall_score`.
|
|
58
|
+
|
|
59
|
+
Default weights should favor trust and retrieval quality:
|
|
60
|
+
|
|
61
|
+
| Metric | Default weight |
|
|
62
|
+
| --- | ---: |
|
|
63
|
+
| Freshness | 10 |
|
|
64
|
+
| Duplicate rate | 8 |
|
|
65
|
+
| Tombstone risk | 12 |
|
|
66
|
+
| Stale derived artifacts | 10 |
|
|
67
|
+
| Retrieval hit rate | 14 |
|
|
68
|
+
| Token reduction | 8 |
|
|
69
|
+
| Leakage scan | 14 |
|
|
70
|
+
| Receipt coverage | 10 |
|
|
71
|
+
| Connector health | 8 |
|
|
72
|
+
| Sync fork risk | 6 |
|
|
73
|
+
|
|
74
|
+
## Metric definitions
|
|
75
|
+
|
|
76
|
+
Each metric emits:
|
|
77
|
+
|
|
78
|
+
- `status`: `healthy`, `watch`, `degraded`, or `critical`;
|
|
79
|
+
- `score`: normalized 0-100 value;
|
|
80
|
+
- `observed`: public-safe measurement fields;
|
|
81
|
+
- `thresholds`: the policy used to classify the metric;
|
|
82
|
+
- `evidence_refs`: hashes, roots, report refs, or cursor refs supporting the measurement;
|
|
83
|
+
- `recommended_actions`: concrete next steps.
|
|
84
|
+
|
|
85
|
+
### 1. Freshness
|
|
86
|
+
|
|
87
|
+
**Question:** Are active memories and indexes current enough for the drive's stated retention and retrieval policy?
|
|
88
|
+
|
|
89
|
+
Suggested measurements:
|
|
90
|
+
|
|
91
|
+
- `oldest_unrefreshed_age_hours`: age of the oldest active memory that has not been revalidated, summarized, retrieved, or policy-checked within the freshness window;
|
|
92
|
+
- `p95_active_age_hours`: p95 age of active memory records since last validation touch;
|
|
93
|
+
- `fresh_records_ratio`: active records touched within the policy window divided by active records.
|
|
94
|
+
|
|
95
|
+
Suggested status policy:
|
|
96
|
+
|
|
97
|
+
| Status | Rule |
|
|
98
|
+
| --- | --- |
|
|
99
|
+
| `healthy` | `fresh_records_ratio >= 0.95` and p95 age is inside policy. |
|
|
100
|
+
| `watch` | `fresh_records_ratio >= 0.85`. |
|
|
101
|
+
| `degraded` | `fresh_records_ratio >= 0.70` or oldest stale record exceeds 2x policy. |
|
|
102
|
+
| `critical` | `fresh_records_ratio < 0.70` or freshness metadata is missing for active records. |
|
|
103
|
+
|
|
104
|
+
Recommended actions:
|
|
105
|
+
|
|
106
|
+
- run local revalidation for stale namespaces;
|
|
107
|
+
- rebuild retrieval index for stale partitions;
|
|
108
|
+
- lower TTL or change compaction cadence if stale records recur.
|
|
109
|
+
|
|
110
|
+
### 2. Duplicate rate
|
|
111
|
+
|
|
112
|
+
**Question:** Is the memory drive wasting storage and context budget on repeated facts?
|
|
113
|
+
|
|
114
|
+
Suggested measurements:
|
|
115
|
+
|
|
116
|
+
- `duplicate_candidate_count`: count of active records grouped by semantic or canonical hash collision;
|
|
117
|
+
- `duplicate_rate`: duplicate candidates divided by active records;
|
|
118
|
+
- `dedupe_savings_estimated_tokens`: estimated context tokens avoided if duplicates are consolidated.
|
|
119
|
+
|
|
120
|
+
Status policy:
|
|
121
|
+
|
|
122
|
+
| Status | Rule |
|
|
123
|
+
| --- | --- |
|
|
124
|
+
| `healthy` | duplicate rate `< 2%`. |
|
|
125
|
+
| `watch` | `2%-5%`. |
|
|
126
|
+
| `degraded` | `5%-12%`. |
|
|
127
|
+
| `critical` | `> 12%` or duplicate bursts create retrieval ambiguity. |
|
|
128
|
+
|
|
129
|
+
Recommended actions:
|
|
130
|
+
|
|
131
|
+
- merge duplicate candidates using local canonical refs;
|
|
132
|
+
- preserve receipt lineage for merged records;
|
|
133
|
+
- tune connector import idempotency when duplicates cluster by connector.
|
|
134
|
+
|
|
135
|
+
### 3. Tombstone risk
|
|
136
|
+
|
|
137
|
+
**Question:** Could deleted or revoked memories accidentally reappear through replicas, derived artifacts, or connector replay?
|
|
138
|
+
|
|
139
|
+
Suggested measurements:
|
|
140
|
+
|
|
141
|
+
- `tombstone_count`: count of public-safe deletion markers;
|
|
142
|
+
- `unsettled_tombstone_count`: tombstones without receipt or replication acknowledgement;
|
|
143
|
+
- `derived_artifacts_referencing_tombstones`: count of summaries, indexes, packs, or exports whose dependency roots include tombstoned refs;
|
|
144
|
+
- `tombstone_replay_window_hours`: maximum age of connector cursor lag for deleted refs.
|
|
145
|
+
|
|
146
|
+
Status policy:
|
|
147
|
+
|
|
148
|
+
| Status | Rule |
|
|
149
|
+
| --- | --- |
|
|
150
|
+
| `healthy` | all tombstones are acknowledged and no live derived artifact references tombstoned refs. |
|
|
151
|
+
| `watch` | acknowledgement lag is inside policy but non-zero. |
|
|
152
|
+
| `degraded` | derived artifacts reference tombstones or acknowledgement lag exceeds policy. |
|
|
153
|
+
| `critical` | tombstoned refs are eligible for retrieval/export or connector replay can resurrect them. |
|
|
154
|
+
|
|
155
|
+
Recommended actions:
|
|
156
|
+
|
|
157
|
+
- invalidate derived artifacts that depend on tombstoned refs;
|
|
158
|
+
- rebuild context packs and exports from active refs only;
|
|
159
|
+
- pause connector replay for affected cursor ranges until nullifiers are applied.
|
|
160
|
+
|
|
161
|
+
### 4. Stale derived artifacts
|
|
162
|
+
|
|
163
|
+
**Question:** Are indexes, summaries, context packs, receipts, benchmarks, or exports built from older roots than the current memory state?
|
|
164
|
+
|
|
165
|
+
Suggested measurements:
|
|
166
|
+
|
|
167
|
+
- `stale_artifact_count`: derived artifacts whose `source_root` differs from the current namespace root;
|
|
168
|
+
- `artifact_types_stale`: unique stale artifact types, for example `retrieval_index`, `summary`, `context_pack`, `benchmark_report`, `proof_packet`;
|
|
169
|
+
- `max_artifact_lag_versions`: largest version distance from current root;
|
|
170
|
+
- `max_artifact_lag_hours`: wall-clock lag since source root changed.
|
|
171
|
+
|
|
172
|
+
Status policy:
|
|
173
|
+
|
|
174
|
+
| Status | Rule |
|
|
175
|
+
| --- | --- |
|
|
176
|
+
| `healthy` | no stale artifacts in active serving paths. |
|
|
177
|
+
| `watch` | stale artifacts exist only outside serving paths. |
|
|
178
|
+
| `degraded` | active retrieval or benchmark artifacts lag by one or more versions. |
|
|
179
|
+
| `critical` | stale artifacts can publish, anchor, export, or serve recalled/tombstoned content. |
|
|
180
|
+
|
|
181
|
+
Recommended actions:
|
|
182
|
+
|
|
183
|
+
- rebuild active derived artifacts from the current root;
|
|
184
|
+
- quarantine proof packets made from stale roots;
|
|
185
|
+
- require artifact builders to declare `source_root` and `artifact_root`.
|
|
186
|
+
|
|
187
|
+
### 5. Retrieval hit rate
|
|
188
|
+
|
|
189
|
+
**Question:** Does retrieval find the expected public-safe evidence refs for benchmark or probe queries?
|
|
190
|
+
|
|
191
|
+
Suggested measurements:
|
|
192
|
+
|
|
193
|
+
- `probe_count`: number of local probes or benchmark items;
|
|
194
|
+
- `hit_at_k`: fraction with at least one expected evidence ref in top-k;
|
|
195
|
+
- `exact_coverage`: fraction where all expected public-safe evidence refs are retrieved;
|
|
196
|
+
- `abstention_correctness`: fraction of no-answer probes where retrieval returns no evidence above threshold.
|
|
197
|
+
|
|
198
|
+
Privacy rule: probes may be counted and hashed, but private text, answers, and raw evidence bodies must not appear in the report.
|
|
199
|
+
|
|
200
|
+
Status policy:
|
|
201
|
+
|
|
202
|
+
| Status | Rule |
|
|
203
|
+
| --- | --- |
|
|
204
|
+
| `healthy` | hit@k and abstention are at or above drive policy. |
|
|
205
|
+
| `watch` | one retrieval metric is within 5 percentage points of policy floor. |
|
|
206
|
+
| `degraded` | any retrieval metric is below policy floor. |
|
|
207
|
+
| `critical` | retrieval returns tombstoned refs, private refs outside scope, or cannot run. |
|
|
208
|
+
|
|
209
|
+
Recommended actions:
|
|
210
|
+
|
|
211
|
+
- rebuild retrieval indexes;
|
|
212
|
+
- inspect namespace filters and capability scopes;
|
|
213
|
+
- compare against last healthy benchmark report hash.
|
|
214
|
+
|
|
215
|
+
### 6. Token reduction
|
|
216
|
+
|
|
217
|
+
**Question:** Is the memory drive reducing context size without dropping required evidence refs?
|
|
218
|
+
|
|
219
|
+
Suggested measurements:
|
|
220
|
+
|
|
221
|
+
- `baseline_estimated_tokens`: estimated tokens for full active context count, not raw text;
|
|
222
|
+
- `selected_estimated_tokens`: estimated tokens for retrieval-selected refs;
|
|
223
|
+
- `token_reduction_ratio`: `1 - selected_estimated_tokens / baseline_estimated_tokens`;
|
|
224
|
+
- `quality_guard_passed`: whether retrieval hit policy passed for the same report.
|
|
225
|
+
|
|
226
|
+
Status policy:
|
|
227
|
+
|
|
228
|
+
| Status | Rule |
|
|
229
|
+
| --- | --- |
|
|
230
|
+
| `healthy` | token reduction meets policy and retrieval guard passes. |
|
|
231
|
+
| `watch` | token reduction is low but retrieval guard passes. |
|
|
232
|
+
| `degraded` | token reduction meets policy only by failing retrieval guard. |
|
|
233
|
+
| `critical` | token reporting uses private text or estimated token fields are missing. |
|
|
234
|
+
|
|
235
|
+
Recommended actions:
|
|
236
|
+
|
|
237
|
+
- tune ranking thresholds only if retrieval guard remains satisfied;
|
|
238
|
+
- dedupe before reducing top-k;
|
|
239
|
+
- never optimize tokens by suppressing required evidence refs.
|
|
240
|
+
|
|
241
|
+
### 7. Leakage scan
|
|
242
|
+
|
|
243
|
+
**Question:** Are reports, receipts, exports, proof packets, and derived artifacts free of private payload fields and secret-like values?
|
|
244
|
+
|
|
245
|
+
Suggested measurements:
|
|
246
|
+
|
|
247
|
+
- `scanned_artifact_count`: count of artifacts scanned;
|
|
248
|
+
- `forbidden_payload_key_hits`: number of denylisted payload key names detected;
|
|
249
|
+
- `secret_value_hits`: number of secret-like values detected by local heuristics;
|
|
250
|
+
- `unsafe_artifact_refs`: hashes or refs of artifacts requiring quarantine.
|
|
251
|
+
|
|
252
|
+
Status policy:
|
|
253
|
+
|
|
254
|
+
| Status | Rule |
|
|
255
|
+
| --- | --- |
|
|
256
|
+
| `healthy` | zero forbidden payload key hits and zero secret value hits. |
|
|
257
|
+
| `watch` | scanner could not inspect a non-published optional artifact. |
|
|
258
|
+
| `degraded` | forbidden payload keys or secret-like values appear in unpublished local artifacts. |
|
|
259
|
+
| `critical` | unsafe artifacts are published, exported, or eligible for anchoring. |
|
|
260
|
+
|
|
261
|
+
Recommended actions:
|
|
262
|
+
|
|
263
|
+
- quarantine unsafe artifacts;
|
|
264
|
+
- regenerate reports from public-safe fields only;
|
|
265
|
+
- block proof-network packet creation until leakage scan is clean.
|
|
266
|
+
|
|
267
|
+
### 8. Receipt coverage
|
|
268
|
+
|
|
269
|
+
**Question:** Can active memory state be traced to public-safe receipts without gaps?
|
|
270
|
+
|
|
271
|
+
Suggested measurements:
|
|
272
|
+
|
|
273
|
+
- `active_ref_count`: count of active memory refs;
|
|
274
|
+
- `covered_ref_count`: count with valid receipt refs or inclusion roots;
|
|
275
|
+
- `receipt_coverage_ratio`: covered active refs divided by active refs;
|
|
276
|
+
- `invalid_receipt_count`: malformed, stale, or mismatched receipt refs;
|
|
277
|
+
- `latest_anchor_batch_ref`: public-safe proof-network anchor batch ref, if one exists.
|
|
278
|
+
|
|
279
|
+
Status policy:
|
|
280
|
+
|
|
281
|
+
| Status | Rule |
|
|
282
|
+
| --- | --- |
|
|
283
|
+
| `healthy` | coverage `>= 0.99` and invalid receipt count is zero. |
|
|
284
|
+
| `watch` | coverage `>= 0.95`. |
|
|
285
|
+
| `degraded` | coverage `>= 0.85` or invalid receipts exist. |
|
|
286
|
+
| `critical` | coverage `< 0.85`, receipt roots mismatch memory roots, or unsafe receipt payload keys exist. |
|
|
287
|
+
|
|
288
|
+
Recommended actions:
|
|
289
|
+
|
|
290
|
+
- issue local receipts for uncovered active refs;
|
|
291
|
+
- regenerate inclusion roots after compaction;
|
|
292
|
+
- verify anchor batches before external publication.
|
|
293
|
+
|
|
294
|
+
### 9. Connector health
|
|
295
|
+
|
|
296
|
+
**Question:** Are connectors importing, exporting, and syncing without private-data leaks or cursor gaps?
|
|
297
|
+
|
|
298
|
+
Suggested measurements:
|
|
299
|
+
|
|
300
|
+
- `connector_count`: configured connector refs;
|
|
301
|
+
- `healthy_connector_count`: connectors with current cursor, no error burst, and clean leakage scan;
|
|
302
|
+
- `lagging_connector_count`: connectors behind policy lag;
|
|
303
|
+
- `error_rate_24h`: connector errors divided by operations over 24 hours;
|
|
304
|
+
- `cursor_gap_count`: count of discontinuities in public-safe cursor sequence.
|
|
305
|
+
|
|
306
|
+
Status policy:
|
|
307
|
+
|
|
308
|
+
| Status | Rule |
|
|
309
|
+
| --- | --- |
|
|
310
|
+
| `healthy` | all enabled connectors are current and cursor-contiguous. |
|
|
311
|
+
| `watch` | lag exists but is inside replay policy. |
|
|
312
|
+
| `degraded` | error bursts or lag exceed policy. |
|
|
313
|
+
| `critical` | cursor gaps, unscanned payload export, or tombstone replay risk exists. |
|
|
314
|
+
|
|
315
|
+
Recommended actions:
|
|
316
|
+
|
|
317
|
+
- pause unhealthy connector scopes before replay;
|
|
318
|
+
- repair cursor gaps from receipt refs, not raw provider payloads;
|
|
319
|
+
- require connector imports to emit idempotency refs.
|
|
320
|
+
|
|
321
|
+
### 10. Sync fork risk
|
|
322
|
+
|
|
323
|
+
**Question:** Could two replicas, devices, or connector lanes believe different roots are current for the same namespace?
|
|
324
|
+
|
|
325
|
+
Suggested measurements:
|
|
326
|
+
|
|
327
|
+
- `replica_count`: count of known replica refs;
|
|
328
|
+
- `root_disagreement_count`: replicas reporting a non-current namespace root;
|
|
329
|
+
- `max_root_lag_versions`: maximum version lag across replicas;
|
|
330
|
+
- `unmerged_branch_count`: unresolved branch/nullifier roots;
|
|
331
|
+
- `conflicting_capability_count`: grants or revocations with incompatible active scopes.
|
|
332
|
+
|
|
333
|
+
Status policy:
|
|
334
|
+
|
|
335
|
+
| Status | Rule |
|
|
336
|
+
| --- | --- |
|
|
337
|
+
| `healthy` | all replicas agree on current root or have an accepted merge root. |
|
|
338
|
+
| `watch` | lagging replicas are read-only and inside sync policy. |
|
|
339
|
+
| `degraded` | active replicas disagree but no conflicting writes are observed. |
|
|
340
|
+
| `critical` | conflicting writes, grants, revocations, or tombstones are active across forks. |
|
|
341
|
+
|
|
342
|
+
Recommended actions:
|
|
343
|
+
|
|
344
|
+
- freeze write grants for forked namespaces;
|
|
345
|
+
- merge by public-safe root/ref lineage and tombstone nullifiers;
|
|
346
|
+
- publish only the post-merge root after validation.
|
|
347
|
+
|
|
348
|
+
## Suggested JSON model
|
|
349
|
+
|
|
350
|
+
The health report schema name should be `enigma.memory_drive_health_report.v1`. It should be deterministic JSON: sorted keys before hashing, ISO-8601 timestamps, integer counts, numeric ratios from `0` to `1`, and no raw private payload fields.
|
|
351
|
+
|
|
352
|
+
```json
|
|
353
|
+
{
|
|
354
|
+
"schema": "enigma.memory_drive_health_report.v1",
|
|
355
|
+
"report_ref": "health_report_sha256:4b1f...",
|
|
356
|
+
"created_at": "2026-06-25T00:00:00.000Z",
|
|
357
|
+
"drive_ref": "drive_sha256:9c2a...",
|
|
358
|
+
"namespace_ref": "namespace_sha256:1f08...",
|
|
359
|
+
"source_root": "memory_root_sha256:7e44...",
|
|
360
|
+
"policy_ref": "memory_health_policy_sha256:44aa...",
|
|
361
|
+
"overall_status": "watch",
|
|
362
|
+
"overall_score": 84,
|
|
363
|
+
"transaction_submitted": false,
|
|
364
|
+
"raw_memory_on_chain": false,
|
|
365
|
+
"privacy_boundaries": {
|
|
366
|
+
"private_payloads_included": false,
|
|
367
|
+
"connector_bodies_included": false,
|
|
368
|
+
"identity_labels_included": false,
|
|
369
|
+
"secret_material_included": false,
|
|
370
|
+
"provider_bodies_included": false
|
|
371
|
+
},
|
|
372
|
+
"metrics": {
|
|
373
|
+
"freshness": {
|
|
374
|
+
"status": "healthy",
|
|
375
|
+
"score": 94,
|
|
376
|
+
"observed": {
|
|
377
|
+
"active_ref_count": 1200,
|
|
378
|
+
"fresh_records_ratio": 0.97,
|
|
379
|
+
"p95_active_age_hours": 18,
|
|
380
|
+
"oldest_unrefreshed_age_hours": 31
|
|
381
|
+
},
|
|
382
|
+
"thresholds": {
|
|
383
|
+
"fresh_records_ratio_watch_floor": 0.85,
|
|
384
|
+
"fresh_records_ratio_degraded_floor": 0.7,
|
|
385
|
+
"policy_window_hours": 48
|
|
386
|
+
},
|
|
387
|
+
"evidence_refs": ["freshness_scan_sha256:aa01..."],
|
|
388
|
+
"recommended_actions": []
|
|
389
|
+
},
|
|
390
|
+
"duplicate_rate": {
|
|
391
|
+
"status": "watch",
|
|
392
|
+
"score": 86,
|
|
393
|
+
"observed": {
|
|
394
|
+
"active_ref_count": 1200,
|
|
395
|
+
"duplicate_candidate_count": 38,
|
|
396
|
+
"duplicate_rate": 0.0317,
|
|
397
|
+
"dedupe_savings_estimated_tokens": 4200
|
|
398
|
+
},
|
|
399
|
+
"thresholds": {
|
|
400
|
+
"watch_floor": 0.02,
|
|
401
|
+
"degraded_floor": 0.05,
|
|
402
|
+
"critical_floor": 0.12
|
|
403
|
+
},
|
|
404
|
+
"evidence_refs": ["dedupe_scan_sha256:bb02..."],
|
|
405
|
+
"recommended_actions": ["Review duplicate clusters before the next context-pack build."]
|
|
406
|
+
},
|
|
407
|
+
"tombstone_risk": {
|
|
408
|
+
"status": "healthy",
|
|
409
|
+
"score": 100,
|
|
410
|
+
"observed": {
|
|
411
|
+
"tombstone_count": 44,
|
|
412
|
+
"unsettled_tombstone_count": 0,
|
|
413
|
+
"derived_artifacts_referencing_tombstones": 0,
|
|
414
|
+
"tombstone_replay_window_hours": 0
|
|
415
|
+
},
|
|
416
|
+
"thresholds": {
|
|
417
|
+
"max_ack_lag_hours": 24
|
|
418
|
+
},
|
|
419
|
+
"evidence_refs": ["tombstone_scan_sha256:cc03..."],
|
|
420
|
+
"recommended_actions": []
|
|
421
|
+
},
|
|
422
|
+
"stale_derived_artifacts": {
|
|
423
|
+
"status": "degraded",
|
|
424
|
+
"score": 68,
|
|
425
|
+
"observed": {
|
|
426
|
+
"stale_artifact_count": 2,
|
|
427
|
+
"artifact_types_stale": ["retrieval_index", "benchmark_report"],
|
|
428
|
+
"max_artifact_lag_versions": 1,
|
|
429
|
+
"max_artifact_lag_hours": 6
|
|
430
|
+
},
|
|
431
|
+
"thresholds": {
|
|
432
|
+
"serving_path_stale_artifacts_allowed": 0
|
|
433
|
+
},
|
|
434
|
+
"evidence_refs": ["artifact_inventory_sha256:dd04..."],
|
|
435
|
+
"recommended_actions": ["Rebuild retrieval indexes from the current source root."]
|
|
436
|
+
},
|
|
437
|
+
"retrieval_hit_rate": {
|
|
438
|
+
"status": "healthy",
|
|
439
|
+
"score": 92,
|
|
440
|
+
"observed": {
|
|
441
|
+
"probe_count": 200,
|
|
442
|
+
"top_k": 5,
|
|
443
|
+
"hit_at_k": 0.94,
|
|
444
|
+
"exact_coverage": 0.88,
|
|
445
|
+
"abstention_correctness": 0.97
|
|
446
|
+
},
|
|
447
|
+
"thresholds": {
|
|
448
|
+
"hit_at_k_floor": 0.9,
|
|
449
|
+
"exact_coverage_floor": 0.85,
|
|
450
|
+
"abstention_correctness_floor": 0.95
|
|
451
|
+
},
|
|
452
|
+
"evidence_refs": ["benchmark_report_sha256:ee05..."],
|
|
453
|
+
"recommended_actions": []
|
|
454
|
+
},
|
|
455
|
+
"token_reduction": {
|
|
456
|
+
"status": "healthy",
|
|
457
|
+
"score": 91,
|
|
458
|
+
"observed": {
|
|
459
|
+
"baseline_estimated_tokens": 180000,
|
|
460
|
+
"selected_estimated_tokens": 39000,
|
|
461
|
+
"token_reduction_ratio": 0.7833,
|
|
462
|
+
"quality_guard_passed": true
|
|
463
|
+
},
|
|
464
|
+
"thresholds": {
|
|
465
|
+
"token_reduction_floor": 0.5,
|
|
466
|
+
"quality_guard_required": true
|
|
467
|
+
},
|
|
468
|
+
"evidence_refs": ["optimizer_report_sha256:ff06..."],
|
|
469
|
+
"recommended_actions": []
|
|
470
|
+
},
|
|
471
|
+
"leakage_scan": {
|
|
472
|
+
"status": "healthy",
|
|
473
|
+
"score": 100,
|
|
474
|
+
"observed": {
|
|
475
|
+
"scanned_artifact_count": 18,
|
|
476
|
+
"forbidden_payload_key_hits": 0,
|
|
477
|
+
"secret_value_hits": 0,
|
|
478
|
+
"unsafe_artifact_refs": []
|
|
479
|
+
},
|
|
480
|
+
"thresholds": {
|
|
481
|
+
"forbidden_payload_key_hits_allowed": 0,
|
|
482
|
+
"secret_value_hits_allowed": 0
|
|
483
|
+
},
|
|
484
|
+
"evidence_refs": ["leakage_scan_sha256:1107..."],
|
|
485
|
+
"recommended_actions": []
|
|
486
|
+
},
|
|
487
|
+
"receipt_coverage": {
|
|
488
|
+
"status": "watch",
|
|
489
|
+
"score": 88,
|
|
490
|
+
"observed": {
|
|
491
|
+
"active_ref_count": 1200,
|
|
492
|
+
"covered_ref_count": 1164,
|
|
493
|
+
"receipt_coverage_ratio": 0.97,
|
|
494
|
+
"invalid_receipt_count": 0,
|
|
495
|
+
"latest_anchor_batch_ref": "anchor_batch_sha256:2208..."
|
|
496
|
+
},
|
|
497
|
+
"thresholds": {
|
|
498
|
+
"healthy_floor": 0.99,
|
|
499
|
+
"watch_floor": 0.95,
|
|
500
|
+
"degraded_floor": 0.85
|
|
501
|
+
},
|
|
502
|
+
"evidence_refs": ["receipt_inventory_sha256:3309..."],
|
|
503
|
+
"recommended_actions": ["Issue local receipts for uncovered active refs before anchoring the next batch."]
|
|
504
|
+
},
|
|
505
|
+
"connector_health": {
|
|
506
|
+
"status": "healthy",
|
|
507
|
+
"score": 96,
|
|
508
|
+
"observed": {
|
|
509
|
+
"connector_count": 4,
|
|
510
|
+
"healthy_connector_count": 4,
|
|
511
|
+
"lagging_connector_count": 0,
|
|
512
|
+
"error_rate_24h": 0,
|
|
513
|
+
"cursor_gap_count": 0
|
|
514
|
+
},
|
|
515
|
+
"thresholds": {
|
|
516
|
+
"cursor_gap_count_allowed": 0,
|
|
517
|
+
"max_error_rate_24h": 0.01
|
|
518
|
+
},
|
|
519
|
+
"evidence_refs": ["connector_inventory_sha256:440a..."],
|
|
520
|
+
"recommended_actions": []
|
|
521
|
+
},
|
|
522
|
+
"sync_fork_risk": {
|
|
523
|
+
"status": "healthy",
|
|
524
|
+
"score": 98,
|
|
525
|
+
"observed": {
|
|
526
|
+
"replica_count": 3,
|
|
527
|
+
"root_disagreement_count": 0,
|
|
528
|
+
"max_root_lag_versions": 0,
|
|
529
|
+
"unmerged_branch_count": 0,
|
|
530
|
+
"conflicting_capability_count": 0
|
|
531
|
+
},
|
|
532
|
+
"thresholds": {
|
|
533
|
+
"active_root_disagreement_allowed": 0,
|
|
534
|
+
"conflicting_capability_count_allowed": 0
|
|
535
|
+
},
|
|
536
|
+
"evidence_refs": ["replica_roots_sha256:550b..."],
|
|
537
|
+
"recommended_actions": []
|
|
538
|
+
}
|
|
539
|
+
},
|
|
540
|
+
"proof_network_ready": {
|
|
541
|
+
"eligible_for_anchor_batch": false,
|
|
542
|
+
"blocking_reasons": [
|
|
543
|
+
"stale_derived_artifacts.status is degraded",
|
|
544
|
+
"receipt_coverage.status is watch"
|
|
545
|
+
],
|
|
546
|
+
"public_payload_only": true,
|
|
547
|
+
"suggested_anchor_fields": {
|
|
548
|
+
"artifact_type": "memory_drive_health_report",
|
|
549
|
+
"artifact_schema": "enigma.memory_drive_health_report.v1",
|
|
550
|
+
"artifact_root": "health_report_sha256:4b1f...",
|
|
551
|
+
"source_root": "memory_root_sha256:7e44...",
|
|
552
|
+
"counts": {
|
|
553
|
+
"active_ref_count": 1200,
|
|
554
|
+
"scanned_artifact_count": 18,
|
|
555
|
+
"connector_count": 4
|
|
556
|
+
}
|
|
557
|
+
}
|
|
558
|
+
}
|
|
559
|
+
}
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
## Computation pipeline
|
|
563
|
+
|
|
564
|
+
1. Load local public-safe metadata inventories: memory refs, tombstones, receipt refs, connector cursor refs, artifact roots, benchmark report hashes, and optimizer summaries.
|
|
565
|
+
2. Reject any candidate field whose key or value matches the private payload denylist.
|
|
566
|
+
3. Compute metric observations from counts, timestamps, hashes, roots, and refs only.
|
|
567
|
+
4. Classify each metric with the active policy.
|
|
568
|
+
5. Compute `overall_score` and `overall_status` with critical caps.
|
|
569
|
+
6. Emit deterministic JSON and calculate `report_ref` from canonical JSON.
|
|
570
|
+
7. Run leakage scan over the final JSON before allowing export or proof-network packet creation.
|
|
571
|
+
|
|
572
|
+
## Operator workflow
|
|
573
|
+
|
|
574
|
+
- **Daily:** inspect `overall_status`, `leakage_scan`, `connector_health`, and `sync_fork_risk`.
|
|
575
|
+
- **Before export or anchoring:** require `leakage_scan.status === "healthy"`, `tombstone_risk.status !== "critical"`, `sync_fork_risk.status !== "critical"`, and no stale serving-path artifacts.
|
|
576
|
+
- **Before benchmark claims:** require `retrieval_hit_rate.status === "healthy"` or explicitly publish the lower status with the benchmark report hash and policy.
|
|
577
|
+
- **After forgetting or revocation:** rerun tombstone, stale artifact, receipt coverage, connector cursor, and sync fork checks.
|
|
578
|
+
|
|
579
|
+
## Product surface
|
|
580
|
+
|
|
581
|
+
A CLI or UI should show a concise SMART table first, then drill-down evidence:
|
|
582
|
+
|
|
583
|
+
| Attribute | Status | Score | Primary action |
|
|
584
|
+
| --- | --- | ---: | --- |
|
|
585
|
+
| Freshness | Healthy | 94 | None |
|
|
586
|
+
| Duplicate rate | Watch | 86 | Review duplicate clusters |
|
|
587
|
+
| Tombstone risk | Healthy | 100 | None |
|
|
588
|
+
| Stale derived artifacts | Degraded | 68 | Rebuild retrieval index |
|
|
589
|
+
| Retrieval hit rate | Healthy | 92 | None |
|
|
590
|
+
| Token reduction | Healthy | 91 | None |
|
|
591
|
+
| Leakage scan | Healthy | 100 | None |
|
|
592
|
+
| Receipt coverage | Watch | 88 | Issue local receipts |
|
|
593
|
+
| Connector health | Healthy | 96 | None |
|
|
594
|
+
| Sync fork risk | Healthy | 98 | None |
|
|
595
|
+
|
|
596
|
+
The product should default to safe blocking language: `proof_network_ready.eligible_for_anchor_batch` is false unless every public-safety and source-root condition passes. This makes health reports usable as local operational evidence now and as future proof-network anchor inputs without changing the privacy model.
|
|
597
|
+
|
|
598
|
+
## Implementation requirements
|
|
599
|
+
|
|
600
|
+
The first implementation should treat health reporting as a pure local planner:
|
|
601
|
+
|
|
602
|
+
- input: public-safe metadata inventories, local benchmark summaries, connector cursor summaries, receipt inventories, artifact inventories, and operator policy thresholds;
|
|
603
|
+
- output: one deterministic `enigma.memory_drive_health_report.v1` JSON object;
|
|
604
|
+
- side effects: none, except writing the operator-requested report file when exposed through a CLI;
|
|
605
|
+
- network behavior: none;
|
|
606
|
+
- chain behavior: none; if a later proof-network command consumes the report, it anchors only the report hash/root and counts.
|
|
607
|
+
|
|
608
|
+
Required validation:
|
|
609
|
+
|
|
610
|
+
1. Reject unknown top-level schemas unless the caller opts into a future schema version.
|
|
611
|
+
2. Reject negative counts, ratios outside `0..1`, timestamps that are not ISO-8601 strings, and missing metric statuses.
|
|
612
|
+
3. Reject any artifact whose private-payload denylist scan is not clean.
|
|
613
|
+
4. Require every metric to include at least one public-safe evidence ref unless the metric is explicitly `critical` because evidence is missing.
|
|
614
|
+
5. Require `transaction_submitted:false` and `raw_memory_on_chain:false` for every exported report.
|
|
615
|
+
|
|
616
|
+
## Policy knobs
|
|
617
|
+
|
|
618
|
+
Operators should be able to tune policy without changing report shape:
|
|
619
|
+
|
|
620
|
+
| Policy field | Default | Why it exists |
|
|
621
|
+
| --- | ---: | --- |
|
|
622
|
+
| `freshness_window_hours` | 48 | Different teams have different expectations for active memory revalidation. |
|
|
623
|
+
| `duplicate_rate_watch_floor` | 0.02 | High-volume connector imports may need earlier duplicate warnings. |
|
|
624
|
+
| `tombstone_ack_window_hours` | 24 | Regulated teams may require faster deletion propagation evidence. |
|
|
625
|
+
| `retrieval_top_k` | 5 | Retrieval probes should match the product's serving configuration. |
|
|
626
|
+
| `hit_at_k_floor` | 0.90 | Keeps retrieval quality policy explicit rather than hidden in copy. |
|
|
627
|
+
| `token_reduction_floor` | 0.50 | Prevents token savings claims that do not meet the operator's target. |
|
|
628
|
+
| `receipt_coverage_healthy_floor` | 0.99 | Sets how complete receipt evidence must be before publishing. |
|
|
629
|
+
| `connector_max_error_rate_24h` | 0.01 | Distinguishes transient connector noise from degraded sync. |
|
|
630
|
+
| `sync_max_read_only_lag_versions` | 1 | Allows read-only lag without allowing forked writes. |
|
|
631
|
+
|
|
632
|
+
Policy refs should be hashed into `policy_ref` so two health reports can be compared only when their thresholds match.
|
|
633
|
+
|
|
634
|
+
## Alerting rules
|
|
635
|
+
|
|
636
|
+
Recommended alert behavior:
|
|
637
|
+
|
|
638
|
+
- page an operator only for `critical` leakage scan, tombstone risk, sync fork risk, or connector cursor gaps;
|
|
639
|
+
- open a maintenance ticket for `degraded` stale artifacts, retrieval hit rate, or receipt coverage;
|
|
640
|
+
- suppress duplicate alerts while `source_root`, `policy_ref`, and metric status are unchanged;
|
|
641
|
+
- include only metric names, statuses, counts, refs, and recommended actions in alert payloads.
|
|
642
|
+
|
|
643
|
+
## Release acceptance checklist
|
|
644
|
+
|
|
645
|
+
- The report can be generated from public-safe local metadata without provider credentials or network access.
|
|
646
|
+
- Each required SMART-like metric is present and independently classifiable.
|
|
647
|
+
- The JSON example remains parseable and contains only public-safe hashes, roots, refs, counts, booleans, statuses, timestamps, thresholds, and signatures.
|
|
648
|
+
- Proof-network readiness is conservative: degraded trust or stale-source conditions block anchoring eligibility.
|
|
649
|
+
- Product copy describes the report as local operational evidence, not as provider deletion proof, model forgetting proof, compliance certification, or live-chain settlement.
|