@llblab/pi-state-flow 0.13.4 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +9 -9
- package/BACKLOG.md +0 -13
- package/CHANGELOG.md +11 -0
- package/README.md +7 -6
- package/dist/lib/config.d.ts +5 -2
- package/dist/lib/config.js +17 -16
- package/dist/lib/context.d.ts +8 -0
- package/dist/lib/context.js +24 -0
- package/dist/lib/continuation.js +4 -3
- package/dist/lib/durable.d.ts +1 -0
- package/dist/lib/durable.js +8 -3
- package/dist/lib/episode.d.ts +2 -0
- package/dist/lib/episode.js +4 -0
- package/dist/lib/extension.d.ts +5 -0
- package/dist/lib/extension.js +93 -35
- package/dist/lib/git.js +17 -14
- package/dist/lib/history.js +8 -4
- package/dist/lib/json.js +29 -3
- package/dist/lib/migration.js +34 -18
- package/dist/lib/protocol.js +14 -16
- package/dist/lib/query.d.ts +27 -1
- package/dist/lib/query.js +182 -10
- package/dist/lib/runtime.d.ts +2 -0
- package/dist/lib/runtime.js +34 -5
- package/dist/lib/snapshot.d.ts +5 -5
- package/dist/lib/snapshot.js +20 -24
- package/dist/lib/state.d.ts +6 -3
- package/dist/lib/state.js +5 -3
- package/dist/lib/status.js +1 -1
- package/dist/lib/storage.js +13 -12
- package/dist/lib/telegram.d.ts +8 -6
- package/dist/lib/telegram.js +10 -3
- package/dist/lib/transition.js +7 -3
- package/dist/package.json +1 -1
- package/dist/skills/state-flow-memory/SKILL.md +11 -3
- package/docs/README.md +1 -0
- package/docs/architecture.md +5 -3
- package/docs/filesystem-recovery.md +3 -2
- package/docs/lazy-state.md +504 -0
- package/docs/usage.md +11 -8
- package/lib/config.ts +18 -15
- package/lib/context.ts +24 -0
- package/lib/continuation.ts +4 -3
- package/lib/durable.ts +8 -4
- package/lib/episode.ts +5 -0
- package/lib/extension.ts +86 -34
- package/lib/git.ts +18 -14
- package/lib/history.ts +8 -4
- package/lib/json.ts +31 -3
- package/lib/migration.ts +33 -16
- package/lib/protocol.ts +14 -16
- package/lib/query.ts +173 -9
- package/lib/runtime.ts +32 -4
- package/lib/snapshot.ts +22 -25
- package/lib/state.ts +11 -6
- package/lib/status.ts +1 -1
- package/lib/storage.ts +14 -11
- package/lib/telegram.ts +19 -6
- package/lib/transition.ts +7 -3
- package/package.json +1 -1
- package/skills/state-flow-memory/SKILL.md +11 -3
package/dist/package.json
CHANGED
|
@@ -46,6 +46,8 @@ Separate a binding requirement from the method currently proposed to satisfy it.
|
|
|
46
46
|
|
|
47
47
|
When it affects continuation, retain what was proposed, accepted, rejected, corrected, explained, or left unresolved, and what the next response or action must address. Preserve enough referents for pending follow-ups to make sense.
|
|
48
48
|
|
|
49
|
+
Treat every completed work slice as a possible restart boundary. Its checkpoint should let a fresh executor recover the achieved outcome, surviving evidence, active commitments, decision-relevant uncertainty, and exact continuation without replaying the prior reasoning trajectory. Optimize decomposition for resumability as well as functional completion; if the next safe action depends on transient context that will disappear, the slice is not yet at a sufficient boundary.
|
|
50
|
+
|
|
49
51
|
Keep consequences, not a transcript or a personality dossier. Do not invent shared history or claim subjective continuity. A fresh run should not unnecessarily reopen a settled exchange or treat an unanswered proposal as approved.
|
|
50
52
|
|
|
51
53
|
### Preserve learning at its demonstrated boundary
|
|
@@ -70,11 +72,17 @@ A locator supports later retrieval; it does not replace content needed for the n
|
|
|
70
72
|
|
|
71
73
|
Do not rerun the underlying project merely to curate its memory. Leave an exact unresolved check when verification falls outside the requested boundary.
|
|
72
74
|
|
|
73
|
-
###
|
|
75
|
+
### Preserve priority and keep Lazy shallow
|
|
76
|
+
|
|
77
|
+
Treat array order in Lazy as semantic priority: earlier entries are higher priority. Preserve that order deliberately; do not reorder entries for aesthetics, incidental grouping, or normalization.
|
|
78
|
+
|
|
79
|
+
Minimize Lazy nesting, especially for top-level collections. Keep a top-level collection as a direct array when its members are the domain values. Represent a standalone item directly, normally as a string; use an object only when that item genuinely owns structured or nested fields. Do not add `items`, `owner`, `source`, or similar wrapper objects merely to describe the collection, and do not introduce nested arrays unless the domain itself requires a matrix or grouped sequence.
|
|
80
|
+
|
|
81
|
+
### Compact without flattening meaning
|
|
74
82
|
|
|
75
83
|
Merge redundant fragments and remove obsolete scaffolding, repeated argumentation, and routine progress. Do not rewrite unchanged state merely to normalize wording.
|
|
76
84
|
|
|
77
|
-
Do not erase a meaningful correction, uncertainty, commitment, negative result, or continuation dependency to make state shorter. Do not retain the previous chain of reasoning solely to steer the next run toward the same method.
|
|
85
|
+
Do not erase a meaningful correction, uncertainty, commitment, negative result, priority order, or continuation dependency to make state shorter. Do not retain the previous chain of reasoning solely to steer the next run toward the same method.
|
|
78
86
|
|
|
79
87
|
### Reconcile phase boundaries
|
|
80
88
|
|
|
@@ -90,7 +98,7 @@ Effective state does not prove which scope owns a value. When ownership matters
|
|
|
90
98
|
|
|
91
99
|
Before writing, review the proposed changes once within the requested boundary:
|
|
92
100
|
|
|
93
|
-
- Would a fresh executor know what must still hold, what changed, what remains unresolved, and
|
|
101
|
+
- Would a fresh executor know what must still hold, what changed, what remains unresolved, and the exact next action without replaying the prior cognitive trajectory?
|
|
94
102
|
- Could an omission cause a known failed attempt, an unnecessary repeated explanation, or loss of an active commitment?
|
|
95
103
|
- Could a retained claim impose an unapproved method, overgeneralize a result, or hide a live alternative?
|
|
96
104
|
|
package/docs/README.md
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
- [Usage and recovery](usage.md): Configuration, new/resumed sessions, Start/Stop, diagnostics, privacy, and storage recovery.
|
|
4
4
|
- [Architecture](architecture.md): Semantic state, temporal algebra, Pi lifecycle, storage, publication, artifacts, and embedding contracts.
|
|
5
|
+
- [Lazy state](lazy-state.md): Implemented ordinary-JSON lazy planes, an effective-by-default `lazy` read path, pure state/patch snapshots, narrow structural `meta` + `keys`, and recursive indexed array patches.
|
|
5
6
|
- [Filesystem recovery](filesystem-recovery.md): Cohort-wide absence, partial-presence, malformed-evidence, repair-authority, and transaction rules.
|
|
6
7
|
- [Temporal acceptance](temporal-acceptance.md): The twenty required temporal properties and their executable witnesses.
|
|
7
8
|
- [SDK compatibility](compatibility.md): Tested dependency stacks, public lifecycle seams, isolated validation, and host limits.
|
package/docs/architecture.md
CHANGED
|
@@ -100,6 +100,7 @@ The default store is `<agentDir>/state-flow`, independent from Markdown discover
|
|
|
100
100
|
Owned paths are:
|
|
101
101
|
|
|
102
102
|
```text
|
|
103
|
+
config.json
|
|
103
104
|
checkpoint.json
|
|
104
105
|
patches.jsonl
|
|
105
106
|
meta.json
|
|
@@ -108,13 +109,14 @@ meta.json
|
|
|
108
109
|
<cwd-key>/meta.json
|
|
109
110
|
<cwd-key>/<session-key>/checkpoint.json
|
|
110
111
|
<cwd-key>/<session-key>/patches.jsonl
|
|
111
|
-
<cwd-key>/<session-key>/config.json
|
|
112
112
|
<cwd-key>/<session-key>/meta.json
|
|
113
|
+
<cwd-key>/<session-key>/config.json
|
|
114
|
+
<cwd-key>/<session-key>/runtime.json
|
|
113
115
|
```
|
|
114
116
|
|
|
115
117
|
CWD and session keys mirror Pi's native encoding. The Pi UUID remains authoritative; readable directory keys never replace identity validation.
|
|
116
118
|
|
|
117
|
-
`checkpoint.json` is only the canonical materialized semantic state, and each nonblank `patches.jsonl` line is only one semantic patch.
|
|
119
|
+
Root `config.json` is the read-only operator configuration shared by every session in the repository; it never participates in semantic overlay. `checkpoint.json` is only the canonical materialized semantic state, and each nonblank `patches.jsonl` line is only one semantic patch. Every scope's `meta.json` symmetrically owns checkpoint/tail boundaries and artifact provenance, with CWD ownership added where applicable. Session `config.json` owns behavior; session `runtime.json` asymmetrically owns lineage, counters, session identity, publication provenance, remote-publication policy, and the full specification only while a run is unfinished. The predecessor combined session `meta.json` remains readable as migration input and is separated by the next normal CAS publication. Metadata writers replace only their owned leaves and preserve JSON-safe unknown siblings. Pi checkpoints retain only an exact Git revision, an exact `file:<hash>` cohort reference, or a proven ordinary-disabled marker.
|
|
118
120
|
|
|
119
121
|
All owned writes use same-directory atomic replacement, regular-file and symlink checks, prepared byte receipts and CAS validation. Unrelated files and detected concurrent bytes are preserved; Git staging follows the acceptance contract below. Rollback restores only bytes still matching the failed publisher's output.
|
|
120
122
|
|
|
@@ -122,7 +124,7 @@ All owned writes use same-directory atomic replacement, regular-file and symlink
|
|
|
122
124
|
|
|
123
125
|
If Git is unavailable specifically through executable `ENOENT`, State Flow uses file-only persistence. File mode retains exact current materialization and proven hot history but offers no arbitrary cold revisions.
|
|
124
126
|
|
|
125
|
-
With Git, each effective semantic cohort creates one local commit immediately through an isolated index that stages the complete non-ignored worktree delta before overlaying the exact prepared State Flow outputs; the caller-visible index is synchronized to the committed tree afterward. Each prepared content is still hashed separately from its supplied bytes, never substituted by mutable worktree reads or filtered staging. Prepared blobs enter the isolated index through one NUL-delimited `update-index --index-info` batch, preserving literal path characters; explicit removals retain their existing path. Any failed batch aborts before reference publication and follows the same exact-output rollback and temporary-index cleanup. Activation returns after local runtime acceptance for normal `turn-end`/`off` policy, skips full predecessor migration planning only when legacy snapshots and complete predecessor checkpoint/tail envelopes are absent, and defers Markdown discovery until the next enabled inference. State Flow-owned active files keep compare-and-swap protection, and `.gitignore` stays authoritative. Git supplies cold history and exact branch restoration. Runtime `revision: "self"` resolves to the commit that owns the runtime record, never arbitrary `HEAD`. Runtime-only writes may use `temporalRevision` to select older semantic streams and their matching artifact provenance. They update the current session's config
|
|
127
|
+
With Git, each effective semantic cohort creates one local commit immediately through an isolated index that stages the complete non-ignored worktree delta before overlaying the exact prepared State Flow outputs; the caller-visible index is synchronized to the committed tree afterward. Each prepared content is still hashed separately from its supplied bytes, never substituted by mutable worktree reads or filtered staging. Prepared blobs enter the isolated index through one NUL-delimited `update-index --index-info` batch, preserving literal path characters; explicit removals retain their existing path. Any failed batch aborts before reference publication and follows the same exact-output rollback and temporary-index cleanup. Activation returns after local runtime acceptance for normal `turn-end`/`off` policy, skips full predecessor migration planning only when legacy snapshots and complete predecessor checkpoint/tail envelopes are absent, and defers Markdown discovery until the next enabled inference. State Flow-owned active files keep compare-and-swap protection, and `.gitignore` stays authoritative. Git supplies cold history and exact branch restoration. Runtime `revision: "self"` resolves to the commit that owns the runtime record, never arbitrary `HEAD`. Runtime-only writes may use `temporalRevision` to select older semantic streams and their matching artifact provenance. They update the current session's `config.json`/`runtime.json` without rewriting live shared checkpoints, tails, or provenance.
|
|
126
128
|
|
|
127
129
|
Branch recovery validates immutable selection before live publication acquisition. `TemporalRuntime.prepareRestore` returns a detached snapshot and an instance-bound, single-use restoration closure. For an exact matching Git owner, that closure reuses the validated cohort and provenance rather than decoding them twice; it still captures the current publication basis under exclusion before installing any runtime fields. Expired file cohorts, legacy snapshot fallbacks, and references redirected to another runtime owner take the fresh-read path. A consumed or failed preparation cannot be replayed, and neither the mutable inspection snapshot nor an old publication basis can become restore authority. This is bounded reuse within one selection, not a cross-session revision cache.
|
|
128
130
|
|
|
@@ -8,7 +8,8 @@ State Flow classifies absence separately from partial or malformed evidence. Rec
|
|
|
8
8
|
| CWD `checkpoint.json` + `patches.jsonl` | State Flow; authoritative shared semantics with CWD identity | Same as global; selected values are not resurrected | Same as global; owner mismatch also fails closed | Normal CAS publication may materialize the complete empty pair |
|
|
9
9
|
| Session `checkpoint.json` + `patches.jsonl` | State Flow; authoritative private semantics | Fresh lifecycle origin may initialize; an existing selected session requires exact retained authority | Partial or malformed pair fails closed | Fresh initialization or exact selected-revision recovery only |
|
|
10
10
|
| Global/CWD `meta.json` | State Flow; temporal boundaries, CWD identity, and artifact provenance | Missing metadata removes temporal authority and fails closed; only an omitted `artifacts` leaf degrades provenance to `{}` | Malformed metadata or semantic/boundary mismatch fails closed | Normal CAS publication from a complete proven cohort |
|
|
11
|
-
| Session `
|
|
11
|
+
| Session `meta.json` | State Flow; session temporal boundaries and artifact provenance | Fresh origin may initialize; selected sessions recover only from exact scope authority | Partial, malformed, or contradictory boundary evidence fails closed | Canonical scope publication from the selected temporal state |
|
|
12
|
+
| Session `config.json` + `runtime.json` | State Flow; behavior plus authoritative runtime identity, lineage, counters, and publication recovery | Fresh origin may initialize; selected sessions recover only from exact authority | Partial, malformed, contradictory identity, lineage, or revision fails closed | Canonical runtime publication from proven lifecycle/selected state; predecessor combined `meta.json` is migration input only |
|
|
12
13
|
| Unsupported `state.json`, hashed layouts, or semantic Pi checkpoints | No current authority | Ignored | Presence never becomes recovery or migration input | Operator-managed removal or external conversion only |
|
|
13
14
|
| Selected Git revision blobs/modes | Git object database; immutable cold authority | A required blob/revision is unavailable | Mode, owner, hash, or cohort contradiction fails closed | Read-only reconstruction; never checkout/reset the live worktree |
|
|
14
15
|
| File-only revision pointer/cohort | State Flow/Pi entry; current exact authority only | No cold history can be invented | Any identity mismatch or incomplete retained cohort fails closed | Exact current cohort only; normal locked publication writes repairs |
|
|
@@ -16,7 +17,7 @@ State Flow classifies absence separately from partial or malformed evidence. Rec
|
|
|
16
17
|
| Worker lease | State Flow; operational ownership | Unclaimed | Malformed/foreign live evidence is preserved; live owner excludes peers | Existing dead-process reclamation protocol only |
|
|
17
18
|
| Publication locks | State Flow; mutual exclusion | Unlocked | Present lock excludes publishers, including interrupted owners | Current owner releases; no opportunistic deletion |
|
|
18
19
|
| Temporary queue files / isolated Git index | Creating State Flow operation; transient | No pending preparation | Unknown surviving files grant no authority | Creating operation cleans its own temporary path; fatal residue is not adopted |
|
|
19
|
-
|
|
|
20
|
+
| Repository-root `config.json` | Operator; optional global configuration, versioned with the store | Built-in defaults | Present unreadable/malformed/unknown settings fail extension configuration | State Flow never creates or rewrites it; ordinary repository publication preserves and versions operator edits |
|
|
20
21
|
| Knowledge root and Markdown | External Knowledge owner | Freshness unavailable; durable semantic state remains | Unsafe paths or malformed/unreadable sources disable acquisition locally | Never create; semantic removal only under existing confirmed ownership rules |
|
|
21
22
|
| Skill and external artifact sources | External package/user owner | Freshness unavailable unless ownership proves removal semantics | Unsafe/non-regular/unreadable sources disable acquisition locally | Never create or fabricate source/provenance |
|
|
22
23
|
| Pi State Flow entries and diagnostics | Pi session log / State Flow entry owner | Missing optional diagnostics provide no evidence; missing required selected pointer blocks that restore | Malformed or contradictory owner/version/pointer fails the dependent restore | Append through Pi entry APIs only; no standalone diagnostics file exists |
|
|
@@ -0,0 +1,504 @@
|
|
|
1
|
+
# Lazy state through progressive `read_state`
|
|
2
|
+
|
|
3
|
+
**Status**: Implemented architecture for the next minor release. Final release validation and publication remain open.
|
|
4
|
+
|
|
5
|
+
## Thesis
|
|
6
|
+
|
|
7
|
+
State Flow should add `lazy` as a fifth semantic plane in every scope. Lazy values are ordinary JSON: durable and versioned with the same causal lineage as hot state, but excluded from ordinary baseline hydration.
|
|
8
|
+
|
|
9
|
+
The model-facing surface remains small:
|
|
10
|
+
|
|
11
|
+
- `read_state` reads one path or an ordered list of paths using one explicit projection.
|
|
12
|
+
- `value` and `patch` return only the requested semantic snapshot or patch; `keys` returns narrowly bounded structural `meta` followed by `keys`.
|
|
13
|
+
- `patch_state` remains a recursive semantic patch, not an edit-command language.
|
|
14
|
+
- Array index selectors extend ordinary patch addressing; there are no `insert`, `remove`, `move`, or generalized query operations.
|
|
15
|
+
|
|
16
|
+
> Retention is not activation. Hot state carries what must matter now; lazy state preserves what may matter later; an explicit read activates only the current trajectory.
|
|
17
|
+
|
|
18
|
+
## Goals
|
|
19
|
+
|
|
20
|
+
- Preserve large or infrequently needed semantic state without linearly growing baseline context.
|
|
21
|
+
- Keep values domain-native: strings remain strings, arrays of strings remain arrays of strings, and objects exist only when the domain needs objects.
|
|
22
|
+
- Navigate current, scoped, effective, and retained historical state through one read protocol.
|
|
23
|
+
- Make every successful read exact: return everything requested or fail, never silently truncate.
|
|
24
|
+
- Keep runtime revision, CAS, and publication mechanics out of model-facing results.
|
|
25
|
+
- Preserve one lock/CAS publication cohort and one causal lineage across hot and lazy mutations.
|
|
26
|
+
|
|
27
|
+
## Non-goals
|
|
28
|
+
|
|
29
|
+
- Mandatory entry objects, stable IDs, timestamps, or provenance fields inside lazy values.
|
|
30
|
+
- A generalized typed query language, free-text search, tags, ranking, or a separate memory subsystem.
|
|
31
|
+
- Product-level element limits, pagination, cursors, or partial successful range reads.
|
|
32
|
+
- A separate array-edit algebra or mutation tool.
|
|
33
|
+
- Automatic promotion of retrieved values into future baseline context.
|
|
34
|
+
|
|
35
|
+
## Semantic model
|
|
36
|
+
|
|
37
|
+
Each scope may contain five semantic planes:
|
|
38
|
+
|
|
39
|
+
```text
|
|
40
|
+
global | CWD | session
|
|
41
|
+
├── artifacts
|
|
42
|
+
├── contract
|
|
43
|
+
├── working
|
|
44
|
+
├── response (session-owned where applicable)
|
|
45
|
+
└── lazy
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
`artifacts`, `contract`, `working`, and `response` remain hot. `lazy` differs only in projection policy:
|
|
49
|
+
|
|
50
|
+
- It is canonical semantic JSON, validated and versioned with its owning scope.
|
|
51
|
+
- It is excluded from the ordinary baseline effective-state body.
|
|
52
|
+
- It becomes model-visible only through bounded baseline navigation hints or explicit `read_state` output.
|
|
53
|
+
- Reading it does not mutate state, freshness, usage metadata, history, or future context.
|
|
54
|
+
|
|
55
|
+
Valid lazy values include:
|
|
56
|
+
|
|
57
|
+
```json
|
|
58
|
+
["important thought", "next thought"]
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
```json
|
|
62
|
+
{
|
|
63
|
+
"decisions": ["keep one publication barrier"],
|
|
64
|
+
"openQuestions": ["large-value storage layout"]
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
```json
|
|
69
|
+
42
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
State Flow does not inject IDs, provenance, revisions, range descriptors, or truncation fields into those values.
|
|
73
|
+
|
|
74
|
+
### Effective lazy overlay
|
|
75
|
+
|
|
76
|
+
The explicit `effective.lazy` path recursively overlays `global.lazy → cwd.lazy → session.lazy` using the existing scope precedence and conflict semantics. It is read-only as an effective view and is not inserted into ordinary hot baseline state.
|
|
77
|
+
|
|
78
|
+
Explicit paths such as `global.lazy`, `cwd.lazy`, and `session.lazy` preserve direct ownership access. Deleting a key from an upper scope reveals a lower-scope value exactly as ordinary scope-local deletion does. Hiding an inherited value without changing its owner is not a separate lazy feature.
|
|
79
|
+
|
|
80
|
+
Active obligations, current constraints, unresolved next actions, and facts required for the next correct action remain hot.
|
|
81
|
+
|
|
82
|
+
## Progressive `read_state`
|
|
83
|
+
|
|
84
|
+
### Request shape
|
|
85
|
+
|
|
86
|
+
`read_state` accepts either one path or an ordered path list:
|
|
87
|
+
|
|
88
|
+
```json
|
|
89
|
+
{
|
|
90
|
+
"path": "cwd.lazy.memory[0..10]",
|
|
91
|
+
"projection": "value"
|
|
92
|
+
}
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
```json
|
|
96
|
+
{
|
|
97
|
+
"paths": [
|
|
98
|
+
"effective.lazy.rules",
|
|
99
|
+
"cwd.lazy.memory[0..10]",
|
|
100
|
+
"session.working.nextAction"
|
|
101
|
+
],
|
|
102
|
+
"projection": "value"
|
|
103
|
+
}
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
`projection` defaults to `value`. A batch uses one projection for every path, evaluates every path against one captured state view, and returns results in request order. Duplicate paths remain duplicate results. If any path is invalid, the whole read fails; there is no mixed partial result.
|
|
107
|
+
|
|
108
|
+
The legacy single `path` form remains first-class rather than mere compatibility syntax. Existing `offset`/`scope` input may remain temporarily during migration but cannot combine with `path` or `paths`.
|
|
109
|
+
|
|
110
|
+
### Response shape
|
|
111
|
+
|
|
112
|
+
There are three projections:
|
|
113
|
+
|
|
114
|
+
| Projection | Response shape | Meaning |
|
|
115
|
+
| --- | --- | --- |
|
|
116
|
+
| `value` | `{ "value": ... }` | Exact selected semantic snapshot |
|
|
117
|
+
| `keys` | `{ "meta": ..., "keys": ... }` | Minimal structural facts followed by immediate keys |
|
|
118
|
+
| `patch` | `{ "patch": ... }` | Historical semantic patch at the selected boundary |
|
|
119
|
+
|
|
120
|
+
A single-path request returns one payload. A multi-path request returns positionally aligned arrays under the same projection fields.
|
|
121
|
+
|
|
122
|
+
`value` deliberately mirrors the effective-state snapshot injected at iteration start: it is semantic state without revision, provenance, range, transport, or storage fields. `patch` likewise contains only the selected semantic patch. Only structural discovery earns `meta`, and `keys` remains the final and most valuable field in that response.
|
|
123
|
+
|
|
124
|
+
The response does not repeat the requested path or projection and does not return an internal revision. Runtime owns revision selection, locking, CAS, and publication; the model cannot improve correctness by echoing that machinery.
|
|
125
|
+
|
|
126
|
+
Errors use the normal tool-error channel rather than successful JSON containing an `error` field.
|
|
127
|
+
|
|
128
|
+
### Path and range model
|
|
129
|
+
|
|
130
|
+
Every path has an explicit root and addresses:
|
|
131
|
+
|
|
132
|
+
- `effective` for the current composed overlay or an indexed historical effective root.
|
|
133
|
+
- `global`, `cwd`, and `session` for explicit current or historical scopes.
|
|
134
|
+
- `lazy` beneath `effective` or an explicit scope.
|
|
135
|
+
- Existing retained temporal selectors using accepted-transition semantics.
|
|
136
|
+
- Object members and array indices.
|
|
137
|
+
- Canonical half-open array ranges `[start:end]`, selecting indices `start <= i < end`; `[start..end]` is an accepted fallback spelling.
|
|
138
|
+
|
|
139
|
+
Examples:
|
|
140
|
+
|
|
141
|
+
```text
|
|
142
|
+
effective.lazy
|
|
143
|
+
effective.working.nextAction
|
|
144
|
+
cwd.lazy.memory
|
|
145
|
+
cwd.lazy.memory[4]
|
|
146
|
+
cwd.lazy.memory[10:20]
|
|
147
|
+
session[3].lazy.investigation
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Indices are zero-based. Negative indices, open-ended ranges, steps, predicates, wildcards, unions, and cross-array expressions are rejected in the first version.
|
|
151
|
+
|
|
152
|
+
A range must fit entirely within the current array. If an array has length 10, `[0:10]` and `[10:10]` are valid, while `[0:15]` and `[11:11]` fail. The fallback `..` spelling has identical semantics. A successful result always contains exactly the requested range. State Flow never returns a shorter successful range with truncation metadata.
|
|
153
|
+
|
|
154
|
+
The implementation must reuse or compatibly extend existing member escaping rather than inventing a second object-path language.
|
|
155
|
+
|
|
156
|
+
## Projections
|
|
157
|
+
|
|
158
|
+
### `value`
|
|
159
|
+
|
|
160
|
+
`value` returns the exact selected scalar, object, array, item, or range:
|
|
161
|
+
|
|
162
|
+
```json
|
|
163
|
+
{
|
|
164
|
+
"path": "cwd.lazy.memory[0..3]"
|
|
165
|
+
}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
```json
|
|
169
|
+
{
|
|
170
|
+
"value": [
|
|
171
|
+
"first thought",
|
|
172
|
+
"second thought",
|
|
173
|
+
"third thought"
|
|
174
|
+
]
|
|
175
|
+
}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
A whole-value request returns the whole value. State Flow imposes no semantic element-count limit, pagination, cursor, or silent truncation. Host, model-context, and transport ceilings remain external operational constraints and surface as ordinary failures rather than partial semantic success.
|
|
179
|
+
|
|
180
|
+
A multi-path value response is positional:
|
|
181
|
+
|
|
182
|
+
```json
|
|
183
|
+
{
|
|
184
|
+
"value": [
|
|
185
|
+
{"ranges": "Ranges are half-open."},
|
|
186
|
+
["first thought", "second thought"],
|
|
187
|
+
"Implement progressive reads."
|
|
188
|
+
]
|
|
189
|
+
}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
### `keys`
|
|
193
|
+
|
|
194
|
+
`keys` combines the minimum structural facts needed to navigate with the complete immediate named keys and child kinds:
|
|
195
|
+
|
|
196
|
+
```json
|
|
197
|
+
{
|
|
198
|
+
"path": "effective.lazy",
|
|
199
|
+
"projection": "keys"
|
|
200
|
+
}
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
```json
|
|
204
|
+
{
|
|
205
|
+
"meta": {
|
|
206
|
+
"type": "object",
|
|
207
|
+
"size": 3,
|
|
208
|
+
"sources": ["global", "cwd"]
|
|
209
|
+
},
|
|
210
|
+
"keys": {
|
|
211
|
+
"memory": "array",
|
|
212
|
+
"rules": "object",
|
|
213
|
+
"soul": "array"
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
For an explicitly scoped object, redundant ownership is omitted:
|
|
219
|
+
|
|
220
|
+
```json
|
|
221
|
+
{
|
|
222
|
+
"meta": {
|
|
223
|
+
"type": "object",
|
|
224
|
+
"size": 2
|
|
225
|
+
},
|
|
226
|
+
"keys": {
|
|
227
|
+
"ranges": "string",
|
|
228
|
+
"publication": "object"
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Arrays and scalars have no named child keys. Their `keys` result stays structurally uniform while `meta` supplies the only useful navigation fact:
|
|
234
|
+
|
|
235
|
+
```json
|
|
236
|
+
{
|
|
237
|
+
"meta": {
|
|
238
|
+
"type": "array",
|
|
239
|
+
"length": 10000
|
|
240
|
+
},
|
|
241
|
+
"keys": []
|
|
242
|
+
}
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
```json
|
|
246
|
+
{
|
|
247
|
+
"meta": {
|
|
248
|
+
"type": "string",
|
|
249
|
+
"length": 18420
|
|
250
|
+
},
|
|
251
|
+
"keys": []
|
|
252
|
+
}
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
```json
|
|
256
|
+
{
|
|
257
|
+
"meta": {
|
|
258
|
+
"type": "number"
|
|
259
|
+
},
|
|
260
|
+
"keys": []
|
|
261
|
+
}
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
Progressive discovery comes from selecting a deeper path, not from key pagination or enumerating every array index. State Flow does not add `limit`, `cursor`, or `truncated` fields.
|
|
265
|
+
|
|
266
|
+
The read-level `meta` object is intentionally narrow and unrelated to persisted scope `meta.json` except for the generic word “metadata.” Its closed initial schema contains only:
|
|
267
|
+
|
|
268
|
+
- `type` always.
|
|
269
|
+
- `size` for objects, meaning immediate named-key count.
|
|
270
|
+
- `length` for arrays and strings.
|
|
271
|
+
- `sources` only when an effective view actually combines or resolves scope ownership.
|
|
272
|
+
|
|
273
|
+
It never exposes temporal boundaries, runtime identity, revisions, publication state, provenance registries, timestamps, encoded sizes, cache/index details, diagnostics, or arbitrary copied fields from `meta.json`. Adding a metadata field requires evidence that it changes the agent's next read decision; diagnostic convenience alone is insufficient.
|
|
274
|
+
|
|
275
|
+
A multi-path `keys` response keeps both arrays positional and places `keys` last:
|
|
276
|
+
|
|
277
|
+
```json
|
|
278
|
+
{
|
|
279
|
+
"meta": [
|
|
280
|
+
{"type": "array", "length": 10000},
|
|
281
|
+
{"type": "object", "size": 2}
|
|
282
|
+
],
|
|
283
|
+
"keys": [
|
|
284
|
+
[],
|
|
285
|
+
{"ranges": "string", "publication": "object"}
|
|
286
|
+
]
|
|
287
|
+
}
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
### `patch`
|
|
291
|
+
|
|
292
|
+
`patch` explains change rather than materialized state:
|
|
293
|
+
|
|
294
|
+
```json
|
|
295
|
+
{
|
|
296
|
+
"path": "cwd[3].lazy.memory",
|
|
297
|
+
"projection": "patch"
|
|
298
|
+
}
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
An indexed change:
|
|
302
|
+
|
|
303
|
+
```json
|
|
304
|
+
{
|
|
305
|
+
"patch": {
|
|
306
|
+
"[1]": "corrected thought"
|
|
307
|
+
}
|
|
308
|
+
}
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
A whole-array replacement:
|
|
312
|
+
|
|
313
|
+
```json
|
|
314
|
+
{
|
|
315
|
+
"patch": [
|
|
316
|
+
"new first thought",
|
|
317
|
+
"new second thought"
|
|
318
|
+
]
|
|
319
|
+
}
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
No change to the selected path at that boundary:
|
|
323
|
+
|
|
324
|
+
```json
|
|
325
|
+
{
|
|
326
|
+
"patch": {}
|
|
327
|
+
}
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
Deletion of the selected object key:
|
|
331
|
+
|
|
332
|
+
```json
|
|
333
|
+
{
|
|
334
|
+
"patch": null
|
|
335
|
+
}
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
This is unambiguous because empty supplied semantic patches are invalid no-ops, while `null` is patch deletion syntax and cannot be retained semantic state. History before the active origin or outside the retained hot window fails explicitly.
|
|
339
|
+
|
|
340
|
+
## Baseline lazy hint
|
|
341
|
+
|
|
342
|
+
Baseline inference receives ordinary hot effective state plus a fixed-shape lazy availability hint. The hint contains navigation only, not lazy bodies, a catalog, pagination state, or per-value metadata.
|
|
343
|
+
|
|
344
|
+
It may identify:
|
|
345
|
+
|
|
346
|
+
- Whether effective lazy state exists.
|
|
347
|
+
- Its immediate top-level keys and kinds.
|
|
348
|
+
- The exact `read_state` path for deeper inspection.
|
|
349
|
+
|
|
350
|
+
The hint is emitted as `lazy_navigation` with `available` and the exact root `path`. For an object root it includes the complete immediate `keys` → child-kind map only when there are at most 32 keys and its canonical JSON is at most 1,024 characters. Otherwise it omits `keys` entirely rather than presenting a partial catalog. This keeps the hint bounded while `read_state` itself returns every explicitly requested value or key set; omission from the hint never makes lazy state unreachable.
|
|
351
|
+
|
|
352
|
+
## Activation and promotion
|
|
353
|
+
|
|
354
|
+
A lazy read enriches only the current tool-result trajectory. It does not:
|
|
355
|
+
|
|
356
|
+
- Enter ordinary effective baseline state.
|
|
357
|
+
- Persist in later baseline inference automatically.
|
|
358
|
+
- Become authoritative merely because it was retrieved.
|
|
359
|
+
- Update semantic freshness, usage counters, or history.
|
|
360
|
+
|
|
361
|
+
When a retrieved fact becomes necessary for future correctness, the agent promotes a distilled consequence through an ordinary hot `patch_state` mutation:
|
|
362
|
+
|
|
363
|
+
- Durable requirement or decision → `contract`.
|
|
364
|
+
- Current fact, uncertainty, or continuation → `working`.
|
|
365
|
+
- Source-addressed reusable compilation → `artifacts`.
|
|
366
|
+
- Historical support with no current consequence → remain under `lazy`.
|
|
367
|
+
|
|
368
|
+
## `patch_state` and arrays
|
|
369
|
+
|
|
370
|
+
`patch_state` remains the sole semantic mutation and publication barrier. There is no `edits` array and no operation vocabulary such as `replace`, `insert`, `remove`, or `move`.
|
|
371
|
+
|
|
372
|
+
Existing recursive semantics continue:
|
|
373
|
+
|
|
374
|
+
- An object recursively patches an object.
|
|
375
|
+
- A scalar replaces the selected scalar/value.
|
|
376
|
+
- An array replaces the selected array as a whole.
|
|
377
|
+
- `null` deletes an object key and remains invalid as retained semantic state.
|
|
378
|
+
|
|
379
|
+
Array index selectors extend recursive addressing:
|
|
380
|
+
|
|
381
|
+
```json
|
|
382
|
+
{
|
|
383
|
+
"cwd": {
|
|
384
|
+
"lazy": {
|
|
385
|
+
"memory": {
|
|
386
|
+
"[1]": "corrected second thought",
|
|
387
|
+
"[4]": "corrected fifth thought"
|
|
388
|
+
}
|
|
389
|
+
}
|
|
390
|
+
},
|
|
391
|
+
"final": true
|
|
392
|
+
}
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
Normative index behavior:
|
|
396
|
+
|
|
397
|
+
- `"[N]"` addresses an existing zero-based element of the retained array.
|
|
398
|
+
- Every addressed index must exist against the one captured publication basis.
|
|
399
|
+
- If any index is invalid, the entire `patch_state` call fails and publishes nothing.
|
|
400
|
+
- An indexed scalar or array value is replaced.
|
|
401
|
+
- An indexed object receives the ordinary recursive object patch; the materialized element is the resulting replacement value at that index.
|
|
402
|
+
- Multiple indexed and object changes in one call share one validation and one lock/CAS publication cohort.
|
|
403
|
+
- Index syntax is reserved and distinct from an ordinary object key such as `"1"`.
|
|
404
|
+
|
|
405
|
+
Nested addressing remains ordinary patch structure:
|
|
406
|
+
|
|
407
|
+
```json
|
|
408
|
+
{
|
|
409
|
+
"cwd": {
|
|
410
|
+
"lazy": {
|
|
411
|
+
"groups": {
|
|
412
|
+
"[0]": {
|
|
413
|
+
"notes": {
|
|
414
|
+
"[1]": "corrected note"
|
|
415
|
+
}
|
|
416
|
+
}
|
|
417
|
+
}
|
|
418
|
+
}
|
|
419
|
+
},
|
|
420
|
+
"final": true
|
|
421
|
+
}
|
|
422
|
+
```
|
|
423
|
+
|
|
424
|
+
Local insertion, removal with shifting, movement, predicates, and ID addressing are intentionally absent. A caller that needs structural array changes reads the array, constructs the desired ordinary JSON value, and replaces the array. A richer mutation language is considered only if measured real workloads prove whole-array replacement inadequate.
|
|
425
|
+
|
|
426
|
+
## Publication and storage
|
|
427
|
+
|
|
428
|
+
Lazy mutations inherit existing guarantees:
|
|
429
|
+
|
|
430
|
+
- One accepted transition identity across all changed scopes and planes.
|
|
431
|
+
- One lock/CAS publication cohort.
|
|
432
|
+
- Atomic hot-plus-lazy multi-scope changes.
|
|
433
|
+
- Scope-local deletion and effective revelation semantics.
|
|
434
|
+
- Exact revision selection on restore and branch navigation.
|
|
435
|
+
- Read-only discovery with no commit, timestamp update, or transition.
|
|
436
|
+
|
|
437
|
+
The first implementation keeps lazy trees co-located in the existing scope semantic files. A local Git-backed probe with incompressible 1 KiB, 100 KiB, and 1 MiB lazy payloads observed 0.33–0.42 s publication, 0.24–0.30 s cold restoration, and approximately linear loose-store growth; the 1 MiB case occupied about 2.2 MiB including the worktree and loose Git history. This does not justify sharding before real workload evidence. A future path-sharded or content-addressed optimization must expose one logical State Flow revision, preserve symlink and ownership safety, and keep normalized semantic JSON authoritative while indexes and caches remain rebuildable projections.
|
|
438
|
+
|
|
439
|
+
## Failure semantics
|
|
440
|
+
|
|
441
|
+
- A nonexistent path, wrong target kind, malformed selector, or out-of-bounds index/range is a tool error.
|
|
442
|
+
- One invalid member of a path batch fails the entire read before returning partial success.
|
|
443
|
+
- One invalid indexed patch fails the entire mutation before publication.
|
|
444
|
+
- A malformed lazy subtree fails closed at the smallest affected path and reports that path.
|
|
445
|
+
- Missing or corrupt optional indexes cannot make canonical lazy JSON disappear.
|
|
446
|
+
- Read failures create no semantic transition and do not affect ordinary hot state.
|
|
447
|
+
- Mechanical index rebuilds create no semantic transition.
|
|
448
|
+
|
|
449
|
+
## Normative invariants
|
|
450
|
+
|
|
451
|
+
1. **Ordinary JSON**: Lazy values contain domain semantics, never mandatory State Flow record wrappers.
|
|
452
|
+
2. **Semantic snapshots**: `value` contains only the selected state snapshot and `patch` only the selected semantic patch; `keys` alone adds closed structural `meta` before `keys`.
|
|
453
|
+
3. **Exact success**: A successful read returns everything requested; it never truncates or paginates silently.
|
|
454
|
+
4. **Runtime-owned concurrency**: Revisions, locks, and CAS remain internal unless explicitly needed for diagnostics.
|
|
455
|
+
5. **Hot-state safety**: Anything required for the next correct action remains hot.
|
|
456
|
+
6. **Retention/activation independence**: Growing lazy state does not hydrate its body into baseline context.
|
|
457
|
+
7. **One read protocol**: Current values, effective lazy values, structure, metadata, and retained patches share one address model.
|
|
458
|
+
8. **No activation by side effect**: Reads cannot change future inference context.
|
|
459
|
+
9. **Explicit frontier crossing**: Only a visible hot-state patch promotes a lazy consequence.
|
|
460
|
+
10. **Patch remains patch**: Array indices extend recursive addressing without introducing an edit-command language.
|
|
461
|
+
11. **Index safety**: Array indices are interpreted only against one captured basis under lock/CAS.
|
|
462
|
+
12. **Failure isolation**: Lazy corruption or unavailable indexes do not damage valid hot state.
|
|
463
|
+
|
|
464
|
+
## Validation contract
|
|
465
|
+
|
|
466
|
+
Before release, implementation evidence must prove:
|
|
467
|
+
|
|
468
|
+
- Lazy bodies do not enter ordinary baseline effective state.
|
|
469
|
+
- Baseline lazy navigation remains bounded as lazy body size grows.
|
|
470
|
+
- Single and ordered multi-path reads preserve exact request order and one captured state view.
|
|
471
|
+
- A failing path produces no partial batch result.
|
|
472
|
+
- Every projection returns exactly its matching top-level key.
|
|
473
|
+
- `value` returns complete selected values and exact valid ranges without product pagination or truncation.
|
|
474
|
+
- Out-of-bounds indices and ranges fail, including batch and boundary cases.
|
|
475
|
+
- `keys` returns minimal closed-schema `meta` followed by complete immediate named-key structure without descendant values or array-index enumeration.
|
|
476
|
+
- Read-level `meta` cannot leak or copy persisted `meta.json`, runtime revision, temporal, publication, provenance, cache, or diagnostic fields.
|
|
477
|
+
- `patch` selects the same causal boundary as existing historical reads and distinguishes no-change from deletion.
|
|
478
|
+
- `effective.lazy` follows global → CWD → session overlay while explicit scope paths preserve ownership.
|
|
479
|
+
- Reads create no semantic transition, Git commit, publication, freshness update, or future activation.
|
|
480
|
+
- Whole-array replacement and indexed scalar, array, object, nested, multi-index, and stale-basis patches remain atomic.
|
|
481
|
+
- Restore, fork, file-only, and Git-backed paths select lazy state from the same owning State Flow revision.
|
|
482
|
+
- Corrupt lazy data or optional indexes do not damage ordinary hot State Flow.
|
|
483
|
+
- Legacy stores and legacy `read_state` inputs either migrate deterministically or fail actionably.
|
|
484
|
+
|
|
485
|
+
## Remaining evolution decisions
|
|
486
|
+
|
|
487
|
+
- Exact member escaping beyond the current strict grammar for names containing separators or brackets.
|
|
488
|
+
- A measured real-workload threshold that would justify replacing the initial co-located semantic layout.
|
|
489
|
+
- Cold Git-history access beyond retained hot history.
|
|
490
|
+
|
|
491
|
+
These decisions cannot introduce mandatory record objects, default metadata envelopes, pagination, typed queries, search, ranking, or a second mutation language without a new design decision.
|
|
492
|
+
|
|
493
|
+
## Next minor release sequence
|
|
494
|
+
|
|
495
|
+
1. Extend the existing path parser with canonical escaping, ordered batches, strict indices, and half-open ranges.
|
|
496
|
+
2. Implement semantic-snapshot `value`, structural `meta` + `keys`, and semantic `patch` reads over current hot state first.
|
|
497
|
+
3. Add indexed recursive array patching through the existing lock/CAS barrier.
|
|
498
|
+
4. Add `lazy` to scope validation, persistence, history, restore, and explicit scoped reads.
|
|
499
|
+
5. Add the read-only `effective.lazy` overlay and bounded baseline navigation hint.
|
|
500
|
+
6. Add migration, corruption, stale-basis, restore, fork, file-only, Git-backed, and concurrency coverage.
|
|
501
|
+
7. Measure repository growth, publication latency, restoration latency, and package/store size before selecting any sharded layout.
|
|
502
|
+
8. Update runtime protocol and user documentation, run full validation, and release through the repository's guarded minor-release flow.
|
|
503
|
+
|
|
504
|
+
The stopping rule is conceptual economy: ordinary JSON, one effective lazy overlay, pure state and patch snapshots, one narrow structural `meta` + `keys` projection, recursive patches with indexed array addressing, and one mutation/publication barrier.
|