@untangle-code/storage-local 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +661 -0
- package/NOTICE +90 -0
- package/README.md +226 -2
- package/THIRD_PARTY_LICENSES.md +24 -0
- package/dist/abort.d.ts +18 -0
- package/dist/abort.d.ts.map +1 -0
- package/dist/abort.js +14 -0
- package/dist/abort.js.map +1 -0
- package/dist/analysis-cache.d.ts +81 -0
- package/dist/analysis-cache.d.ts.map +1 -0
- package/dist/analysis-cache.js +322 -0
- package/dist/analysis-cache.js.map +1 -0
- package/dist/checkpoint-store.d.ts +22 -0
- package/dist/checkpoint-store.d.ts.map +1 -0
- package/dist/checkpoint-store.js +123 -0
- package/dist/checkpoint-store.js.map +1 -0
- package/dist/embedding-cache.d.ts +38 -0
- package/dist/embedding-cache.d.ts.map +1 -0
- package/dist/embedding-cache.js +229 -0
- package/dist/embedding-cache.js.map +1 -0
- package/dist/fs-errors.d.ts +4 -0
- package/dist/fs-errors.d.ts.map +1 -0
- package/dist/fs-errors.js +14 -0
- package/dist/fs-errors.js.map +1 -0
- package/dist/index.d.ts +15 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +23 -0
- package/dist/index.js.map +1 -0
- package/dist/layout.d.ts +50 -0
- package/dist/layout.d.ts.map +1 -0
- package/dist/layout.js +99 -0
- package/dist/layout.js.map +1 -0
- package/dist/local-file-system.d.ts +37 -0
- package/dist/local-file-system.d.ts.map +1 -0
- package/dist/local-file-system.js +193 -0
- package/dist/local-file-system.js.map +1 -0
- package/dist/lock-provider.d.ts +59 -0
- package/dist/lock-provider.d.ts.map +1 -0
- package/dist/lock-provider.js +318 -0
- package/dist/lock-provider.js.map +1 -0
- package/dist/real-time.d.ts +16 -0
- package/dist/real-time.d.ts.map +1 -0
- package/dist/real-time.js +19 -0
- package/dist/real-time.js.map +1 -0
- package/dist/search-journey-store.d.ts +34 -0
- package/dist/search-journey-store.d.ts.map +1 -0
- package/dist/search-journey-store.js +530 -0
- package/dist/search-journey-store.js.map +1 -0
- package/dist/storage.d.ts +42 -0
- package/dist/storage.d.ts.map +1 -0
- package/dist/storage.js +34 -0
- package/dist/storage.js.map +1 -0
- package/package.json +60 -3
- package/src/abort.ts +26 -0
- package/src/analysis-cache.ts +404 -0
- package/src/checkpoint-store.ts +155 -0
- package/src/embedding-cache.ts +282 -0
- package/src/fs-errors.ts +19 -0
- package/src/index.ts +30 -0
- package/src/layout.ts +149 -0
- package/src/local-file-system.ts +232 -0
- package/src/lock-provider.ts +381 -0
- package/src/real-time.ts +18 -0
- package/src/search-journey-store.ts +638 -0
- package/src/storage.ts +98 -0
package/NOTICE
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# NOTICE
|
|
2
|
+
|
|
3
|
+
Copyright (C) 2026 Luke Bölling and contributors.
|
|
4
|
+
|
|
5
|
+
This attribution covers original Untangle work. Contributors retain copyright in
|
|
6
|
+
their contributions. Third-party packages, grammars and other materials retain
|
|
7
|
+
their own copyright holders, licenses and notices; this line does not replace them.
|
|
8
|
+
|
|
9
|
+
## License — binding decision
|
|
10
|
+
|
|
11
|
+
Untangle is licensed under the **GNU Affero General Public License, Version 3
|
|
12
|
+
(AGPL-3.0-only)**. See [`LICENSE`](./LICENSE).
|
|
13
|
+
|
|
14
|
+
This decision is **final, not provisional**, and was explicitly instructed by
|
|
15
|
+
the project owner. No contributor, agent or tool may change it without a new
|
|
16
|
+
explicit instruction.
|
|
17
|
+
|
|
18
|
+
The repository-wide license covers Untangle's first-party source code, scripts,
|
|
19
|
+
tests, fixtures, configuration, documentation, website and brand artwork, including
|
|
20
|
+
the logo and 3D yarn illustrations. Every workspace `package.json` declares
|
|
21
|
+
`"license": "AGPL-3.0-only"`. There is no separately licensed enterprise component
|
|
22
|
+
or separate permissive license for documentation or brand assets.
|
|
23
|
+
|
|
24
|
+
Third-party code, embedded grammars, dependency assets and their notices retain
|
|
25
|
+
their original licenses. Their inclusion does not relicense those works. The
|
|
26
|
+
repository license does not grant trademark rights or change the license of a
|
|
27
|
+
repository analyzed with Untangle. See the repository's `docs/LICENSING.md` for
|
|
28
|
+
scope and verification details.
|
|
29
|
+
|
|
30
|
+
This resolves the license question left open in `vision.md` §14 and completes
|
|
31
|
+
`product.md` §16, item 2.
|
|
32
|
+
|
|
33
|
+
## Rationale
|
|
34
|
+
|
|
35
|
+
- **Network copyleft fits the intended product model.** AGPL §13 sets source
|
|
36
|
+
availability requirements for modified versions used through a computer network.
|
|
37
|
+
The current local CLI/stdio-MCP entry and future shared-server modes share the
|
|
38
|
+
same license.
|
|
39
|
+
- **Local-first analysis has a clear boundary.** Running Untangle against a codebase
|
|
40
|
+
does not replace that codebase's own license with Untangle's license.
|
|
41
|
+
- **First-party components share one license.** Implemented components and future
|
|
42
|
+
official plugins use this repository's licensing policy. Planned integrations do
|
|
43
|
+
not establish implemented or published products.
|
|
44
|
+
|
|
45
|
+
## Deliberately accepted consequence
|
|
46
|
+
|
|
47
|
+
The project deliberately chooses AGPL rather than a permissive embedding license.
|
|
48
|
+
Redistribution, modifications and combined works must be assessed against the
|
|
49
|
+
actual terms in `LICENSE`, including applicable Corresponding Source requirements.
|
|
50
|
+
No permissive exception, commercial dual license or special contract-only package
|
|
51
|
+
license is granted or promised by this notice.
|
|
52
|
+
|
|
53
|
+
## Dependencies
|
|
54
|
+
|
|
55
|
+
Third-party runtime dependencies are checked against the allowlist in
|
|
56
|
+
`scripts/check-licenses.mjs`. Their licenses and copyright notices remain unchanged
|
|
57
|
+
and traceable through package `THIRD_PARTY_LICENSES.md` files, grammar provenance,
|
|
58
|
+
the lockfile and release SBOM. The allowlist includes MIT / Apache-2.0 / BSD / ISC
|
|
59
|
+
and other explicitly listed licenses; it is not a license grant for dependencies.
|
|
60
|
+
|
|
61
|
+
The allowlist in `scripts/check-licenses.mjs` applies to **dependencies**, not
|
|
62
|
+
workspace packages. Untangle's own packages carry AGPL-3.0-only and are excluded
|
|
63
|
+
from that check.
|
|
64
|
+
|
|
65
|
+
## Clean room
|
|
66
|
+
|
|
67
|
+
Untangle is an independent implementation with its own data models, APIs, tool
|
|
68
|
+
names, tests and sources. Feature ideas may be derived from publicly observable
|
|
69
|
+
behavior and generally known techniques. Third-party source code is not copied,
|
|
70
|
+
including from AGPL projects even where the license would permit it.
|
|
71
|
+
|
|
72
|
+
## Contribution terms
|
|
73
|
+
|
|
74
|
+
Pull requests are welcome under the repository's existing AGPL-3.0-only license.
|
|
75
|
+
No separate CLA, copyright assignment or DCO sign-off is required. Contributors
|
|
76
|
+
must have the right to submit their work under those terms. See CONTRIBUTING.md.
|
|
77
|
+
|
|
78
|
+
## Release decisions
|
|
79
|
+
|
|
80
|
+
- [x] Copyright attribution: Luke Bölling, with contributors retaining their rights.
|
|
81
|
+
- [x] Contributions use AGPL-3.0-only without an additional CLA or DCO gate.
|
|
82
|
+
|
|
83
|
+
## Open legal items before public beta
|
|
84
|
+
|
|
85
|
+
- [ ] Trademark search for the product name `Untangle`.
|
|
86
|
+
- [ ] Registration and legal review of the npm scope `@untangle-code`.
|
|
87
|
+
- [ ] Verify source availability and notices for each released network daemon/server
|
|
88
|
+
mode (WP-59). The current stdio MCP `untangle_about` already reports the license,
|
|
89
|
+
source repository and build commit when supplied; metadata alone does not prove
|
|
90
|
+
that a concrete deployed version's source is available.
|
package/README.md
CHANGED
|
@@ -1,3 +1,227 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @untangle-code/storage-local
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Untangle local state (WP-12/WP-250): the `.untangle/` layout, a persistent size-bounded embedding
|
|
4
|
+
cache, an atomic checkpoint mirror, a file-based `LockProvider`, and local Search Journey
|
|
5
|
+
persistence. Public package (`private: false`).
|
|
6
|
+
|
|
7
|
+
Normative sources: [`architecture.md`](../../architecture.md) §12 (layout), §21.1–§21.2 (lock
|
|
8
|
+
architecture), [`spec.md`](../../spec.md) §11.2, §12.3, §13.7, §24 (takes precedence in a conflict),
|
|
9
|
+
[`docs/BUILD_PLAN.md`](../../docs/BUILD_PLAN.md) (WP-12/WP-250).
|
|
10
|
+
|
|
11
|
+
Dependencies: **only** `@untangle-code/domain` (`workspace:*`) at runtime;
|
|
12
|
+
`@untangle-code/testkit` (`workspace:*`) as a development-only dependency for port contract suites
|
|
13
|
+
and deterministic Clock/Random test doubles. No other Untangle packages or external runtime
|
|
14
|
+
dependencies.
|
|
15
|
+
|
|
16
|
+
<a id="modulübersicht"></a>
|
|
17
|
+
|
|
18
|
+
## Module overview
|
|
19
|
+
|
|
20
|
+
| Module | Contents | Source |
|
|
21
|
+
|---|---|---|
|
|
22
|
+
| `layout.ts` | `StateDirLayout`, `buildStateDirLayout()`, `ensureStateDirLayout()`: architecture.md §12 `.untangle/` directory tree | arch §12 |
|
|
23
|
+
| `local-file-system.ts` | `LocalFileSystem`: `FileSystemPort` adapter over `node:fs/promises`, path sandbox through domain `assertWithinWorkspace` plus symlink recheck | arch §2.1, §23.1 |
|
|
24
|
+
| `lock-provider.ts` | `FileLockProvider`: file-based `LockProvider`, one directory per lock key as an atomic exclusivity primitive, `record.json` written through temp+rename | arch §21.1–§21.2 |
|
|
25
|
+
| `checkpoint-store.ts` | `LocalCheckpointStore`: `CheckpointStorePort`, atomic writes through temp+rename | arch §12, §13.3 |
|
|
26
|
+
| `embedding-cache.ts` | `LocalEmbeddingCache`, `buildEmbeddingCacheKey()`: persistent size-bounded embedding cache with LRU eviction | spec §9.4 |
|
|
27
|
+
| `search-journey-store.ts` | `LocalSearchJourneyStore`: deeply validated, atomic, revision-protected persistence under `.untangle/product-complexity/sessions/`, plus separate HMAC key `.untangle/product-complexity/fingerprint.key` | spec §13.7, §24; WP-250 |
|
|
28
|
+
| `storage.ts` | `openLocalStorage()`: wires all the above adapters to a shared `FileSystemPort` | WP-12 decision |
|
|
29
|
+
| `abort.ts`, `fs-errors.ts`, `real-time.ts` | internal helpers (AbortSignal handling, Node error-code checks, the one isolated real-time source) | internal |
|
|
30
|
+
|
|
31
|
+
<a id="verwendung"></a>
|
|
32
|
+
|
|
33
|
+
## Usage
|
|
34
|
+
|
|
35
|
+
```ts
|
|
36
|
+
import { openLocalStorage } from "@untangle-code/storage-local";
|
|
37
|
+
|
|
38
|
+
const storage = await openLocalStorage("/abs/path/to/workspace/.untangle", {
|
|
39
|
+
clock, // Clock (domain port) — real SystemClock in production
|
|
40
|
+
random, // Random (domain port) — real SystemRandom in production
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
await storage.embeddingCache.set(profileId, preparedText, vector);
|
|
44
|
+
await storage.checkpoints.save(checkpoint);
|
|
45
|
+
const journey = await storage.searchJourneys.get(sessionId);
|
|
46
|
+
const handle = await storage.locks.acquire(`repo:${repoId}:${refId}:index`, { waitMs: 5_000 });
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
<a id="entscheidungen"></a>
|
|
50
|
+
|
|
51
|
+
## Decisions
|
|
52
|
+
|
|
53
|
+
1. **Lock exclusivity through `FileSystemPort.mkdir` without `recursive`, not a separate locking
|
|
54
|
+
library.** Node's `mkdir` atomically fails with `EEXIST` if the target directory exists: a real,
|
|
55
|
+
operating-system-guaranteed exclusivity check directly underlying `FileLockProvider.tryAcquire()`
|
|
56
|
+
(one directory per `sha256(key)`, containing `record.json`). No extra lock-on-lock mechanism is
|
|
57
|
+
needed. The test “two provider instances racing for the same fresh lock” proves this with two
|
|
58
|
+
genuinely concurrent `FileLockProvider` instances in one process. The “lock is free” case is
|
|
59
|
+
entirely race-free because no takeover step is involved.
|
|
60
|
+
|
|
61
|
+
2. **Stale-lease takeover is deliberately NOT entirely race-free across processes.** A documented,
|
|
62
|
+
accepted MVP limit (architecture.md §21.2 itself explicitly calls file locks “MVP”, with a real
|
|
63
|
+
distributed lease only in “Remote Team Mode”). The takeover window: between “lease recognized
|
|
64
|
+
as expired” and “orphaned directory removed + recreated” is a TOCTOU window where a holder
|
|
65
|
+
renewing its lease *at that exact moment* could theoretically be overwritten. The following
|
|
66
|
+
cases are **not** affected and are therefore tested strictly: (a) contention for a **free** lock
|
|
67
|
+
(decision 1, fully atomic); (b) distinguishing “directory exists, `record.json` not yet present”
|
|
68
|
+
(= *busy*, never treated as *stale*, or a competitor still creating its valid lock would lose
|
|
69
|
+
it to another contender's takeover race); and (c), since the WP-80 fix, “expired” vs. “corrupt”,
|
|
70
|
+
now two distinct cases that must never be confused (see decision 2a).
|
|
71
|
+
|
|
72
|
+
2a. **`record.json` is written atomically through a temporary file plus `rename`, both on initial
|
|
73
|
+
`tryAcquire()` and every `renew()`.** The exact technique already used by this package's
|
|
74
|
+
`checkpoint-store.ts` (“every write is temporary-file-then-`rename`”). Before this fix (WP-80),
|
|
75
|
+
initial acquire and renew wrote directly to the live path using `fs.writeFile`. A reader could
|
|
76
|
+
theoretically see a torn, partially written record; `JSON.parse` failed, and `readRecordState()`
|
|
77
|
+
treated EVERY parse error as immediately eligible for takeover, regardless of the actual
|
|
78
|
+
holder's lease (“expired OR corrupt = stale”). Process A writing via `renew()` could be
|
|
79
|
+
overtaken by B, interpreting the torn intermediate state as “corrupt, therefore free”; both
|
|
80
|
+
then held the same lock, and A still reported success on its next `renew()`. After the fix,
|
|
81
|
+
only a single atomic `rename` reaches the live path: readers always see either the old complete
|
|
82
|
+
record or the new complete record, never a mixture. This guarantee also applies alongside the
|
|
83
|
+
separate documented TOCTOU gap of stale takeover itself (decision 2). That gap concerns
|
|
84
|
+
*removing* someone else's apparently stale lock directory, not *writing* one's own record; it
|
|
85
|
+
remains a separate, previously documented MVP risk.
|
|
86
|
+
|
|
87
|
+
**Since the fix, “expired” and “corrupt” are deliberately separate, no longer one “stale”
|
|
88
|
+
category:**
|
|
89
|
+
- An **expired lease** (`leaseExpiresAt` in the past) remains immediately eligible for
|
|
90
|
+
takeover: the lease itself authorizes when the holder is considered to have given up.
|
|
91
|
+
- An **unreadable (`corrupt`) record** is an ERROR STATE, not a normal lifecycle state. Because
|
|
92
|
+
every write is now atomic, readers fundamentally cannot see torn records. An actually
|
|
93
|
+
unparsable `record.json` means external influence (manual editing, disk fault, a record
|
|
94
|
+
written by an incompatible future format), no longer “caught a write in progress”. A corrupt
|
|
95
|
+
record also lacks its own trustworthy `leaseExpiresAt`, so unlike an expired lease there is
|
|
96
|
+
no information on how long it has been invalid. Immediately freeing it would repeat WP-80
|
|
97
|
+
with a different cause (“I cannot explain it” ≠ “safe to take over”). Selected rule: a
|
|
98
|
+
**grace period** (`DEFAULT_CORRUPT_GRACE_MS`, `lock-provider.ts`, 5 minutes, substantially
|
|
99
|
+
longer than any expected lease/write duration), measured from `record.json` file `mtime`.
|
|
100
|
+
Until that period expires, `tryAcquire()` treats a corrupt record exactly like a validly
|
|
101
|
+
held lock (`null`, no throw, no takeover attempt). Afterwards it automatically becomes
|
|
102
|
+
eligible for takeover like an expired lease. This self-heals without operator action while
|
|
103
|
+
giving a possibly unusually slow legitimate write, or a person investigating the finding,
|
|
104
|
+
ample time before anybody treats it as abandoned. A `FileLockProvider` constructor parameter
|
|
105
|
+
can override the grace period, mainly for tests that check the boundary without waiting
|
|
106
|
+
several real minutes.
|
|
107
|
+
|
|
108
|
+
3. **Stale detection uses lease expiry only, never PID existence**, exactly as required by
|
|
109
|
+
architecture.md §21.2. Besides mandatory domain `LockRecordV1` fields, `record.json` contains
|
|
110
|
+
internal `processStartedAtEpochMs` (the “start time” required by architecture.md §21.2). It never
|
|
111
|
+
contributes to stale decisions: it is diagnostic information for manually inspecting
|
|
112
|
+
`record.json`, and is therefore absent from `LockHandle.record` (whose shape is exactly
|
|
113
|
+
`LockRecordV1`; the domain contract register allows no extension). The test “PID reuse does not
|
|
114
|
+
cause false takeover” proves this directly: a manually written expired record with
|
|
115
|
+
`pid: process.pid` (the current test environment apparently “is” the holder) is still correctly
|
|
116
|
+
taken over because only `leaseExpiresAt` matters.
|
|
117
|
+
|
|
118
|
+
4. **`renew()` deliberately creates a new `processInstanceId`** rather than retaining the old one.
|
|
119
|
+
A consistency decision mirroring `InMemoryLockProvider` from `@untangle-code/testkit`. From the
|
|
120
|
+
record's perspective, renewal is a new “term” of the same holder, identified by the handle
|
|
121
|
+
itself (which updates its local `current` reference after every `renew()`), not an ID stable
|
|
122
|
+
throughout the entire holding period.
|
|
123
|
+
|
|
124
|
+
5. **`LocalFileSystem` symlink checks also cover target paths that do not yet exist**, e.g. a
|
|
125
|
+
`writeFile()` destination. They start at the nearest *existing* ancestor and resolve it against
|
|
126
|
+
`assertWithinWorkspace`, rather than checking only existing final paths as a minimal approach
|
|
127
|
+
would. This catches a *directory* segment symlinking outside the root even when the target file
|
|
128
|
+
does not yet exist (“rejects writing into a new file under a symlinked directory that escapes
|
|
129
|
+
the root”). Known limit: a TOCTOU gap remains between the check and actual `node:fs` call. An
|
|
130
|
+
attacker with filesystem write access could replace a symlink between check and operation.
|
|
131
|
+
Acceptable for the local single-user MVP described in architecture.md §1, not a sufficient
|
|
132
|
+
guarantee for multi-user/sandbox isolation.
|
|
133
|
+
|
|
134
|
+
6. **Checkpoint corruption is reported loudly (`INDEX_INCOMPLETE`), embedding-cache corruption
|
|
135
|
+
silently degrades to a miss.** Both follow spec.md §25.2 for opposite reasons. A checkpoint is
|
|
136
|
+
the *only* resume record for an ongoing indexing job. Ignoring corruption would invisibly
|
|
137
|
+
disguise a real storage error as “no checkpoint exists” (write-path safety reasoning applied to
|
|
138
|
+
a read path because the underlying issue is consistency, not existence). The embedding cache,
|
|
139
|
+
by contrast, is explicitly “deletable and reconstructable” under architecture.md §12: a missing
|
|
140
|
+
or corrupt entry only means “embed this text again”, never data loss. The checkpoint mirror's
|
|
141
|
+
pointer index (`jobs/latest/<hash>.json`, “which job is latest for this repo+ref”) sits between
|
|
142
|
+
these and is treated like the cache. It is merely an accelerator over the actual separately
|
|
143
|
+
integrity-checked checkpoint files; losing it means “no known latest state”, never actual data
|
|
144
|
+
loss.
|
|
145
|
+
|
|
146
|
+
7. **The embedding-cache manifest is rewritten completely on every mutation** (`O(number of
|
|
147
|
+
entries)` per `set()`/`get()` hit), rather than incrementally updating a genuine index. Sufficient
|
|
148
|
+
at MVP scale (a local workspace cache) and substantially easier to keep correct than a
|
|
149
|
+
concurrency-safe append log. For very large caches (millions of entries), a real embedded index
|
|
150
|
+
would be the appropriate next step; deliberately not brought forward because it is outside
|
|
151
|
+
this assignment.
|
|
152
|
+
|
|
153
|
+
8. **Two tsconfigs**, as in domain/scanner: `tsconfig.json` (composite, emit to `dist/`, no tests)
|
|
154
|
+
for the build; `tsconfig.typecheck.json` (`noEmit`, includes tests) for `pnpm run typecheck`.
|
|
155
|
+
Composite projects do not permit `noEmit`.
|
|
156
|
+
|
|
157
|
+
9. **Comparable journeys are durable learning artifacts, not telemetry.** Search Journeys exist
|
|
158
|
+
only after an explicit `start` in the calling service and remain in the workspace-local state
|
|
159
|
+
directory. The adapter serializes mutations per session, checks a compare-and-swap revision,
|
|
160
|
+
writes through a temporary file plus atomic `rename`, and deeply validates every record on write
|
|
161
|
+
and read. The fingerprint-key lock and both the outer creation lock and inner per-session lock
|
|
162
|
+
use a 30-second lease with non-overlapping renewal attempts every 10 seconds. Immediately before
|
|
163
|
+
an atomic publish, every held lock is owner-checked and a bounded verification loop requires a
|
|
164
|
+
shared window with at least 10 seconds remaining on all of them. Losing ownership or failing to
|
|
165
|
+
converge within eight rounds aborts before `rename` with `LOCK_HELD`. A separate creation lock
|
|
166
|
+
enforces the 10,000-record ceiling across processes; one record is limited to 32 MiB. Before a list/report parses any
|
|
167
|
+
record, it builds a file-and-size manifest for the complete snapshot against a 128 MiB
|
|
168
|
+
cumulative byte budget. Every read rechecks its actual UTF-8 byte count and post-read file size
|
|
169
|
+
before parsing; disappearance or a size/type change fails the whole report visibly—Untangle
|
|
170
|
+
never emits a partial cohort as if it were complete. A same-size atomic replacement cannot be
|
|
171
|
+
distinguished without a versioned file-read primitive and remains a documented local-filesystem
|
|
172
|
+
snapshot boundary. Corrupt or oversized records likewise fail closed instead of entering a
|
|
173
|
+
Product-Complexity cohort. Each exact per-write temporary path is removed best-effort after
|
|
174
|
+
success, cancellation, or I/O failure without reusing the aborted caller signal or masking the
|
|
175
|
+
primary error.
|
|
176
|
+
|
|
177
|
+
The persisted contract is privacy-minimized, not anonymous or encrypted. With the default
|
|
178
|
+
`retainText: false`, goal/query text and an optional feature ID declared stable by the caller are represented by
|
|
179
|
+
workspace-local, domain-separated HMAC-SHA-256 fingerprints and no chunk or source content is
|
|
180
|
+
stored. Untagged legacy journeys are never assigned a feature by the adapter. The random key lives outside
|
|
181
|
+
session records at `.untangle/product-complexity/fingerprint.key`; anyone with the complete local
|
|
182
|
+
state and key can test guesses. On POSIX systems, both the key and newly written journey records
|
|
183
|
+
are created with mode `0600`. Result locators, ranks, scores, explicit tool metadata, a controlled
|
|
184
|
+
abandonment reason code, and other lifecycle metadata remain. Callers that opt into
|
|
185
|
+
`retainText: true` additionally persist goal, query, failed-search, and free abandonment text.
|
|
186
|
+
|
|
187
|
+
New observations persist measurement context v2. Successful observations do not flatten their
|
|
188
|
+
profile provenance: embedding profile, generation set, and immutable
|
|
189
|
+
`corpusRevisionFingerprint` stay bound to the profile that produced them. The revision originates
|
|
190
|
+
from the completed index's full scanned File-ID/content-hash manifest, including successful
|
|
191
|
+
zero-change incremental runs. The adapter keeps additive legacy records readable and does not
|
|
192
|
+
fabricate a revision for them; report derivation leaves them visible but never
|
|
193
|
+
`sufficientEvidence: true`. Reports are derived rather than stored:
|
|
194
|
+
`sessionCounts.featureTaggedCompleted` counts completed records with an explicit feature
|
|
195
|
+
fingerprint, and feature mappings preserve separate Search Journey and accepted-target
|
|
196
|
+
code-complexity axes.
|
|
197
|
+
|
|
198
|
+
<a id="bekannte-grenzen-siehe-openissues-im-task-report"></a>
|
|
199
|
+
|
|
200
|
+
## Known limits (see `openIssues` in the task report)
|
|
201
|
+
|
|
202
|
+
- No cross-process lock on the embedding cache itself: two processes writing the same key
|
|
203
|
+
simultaneously overwrite each other without error (last writer wins). Unproblematic for the
|
|
204
|
+
intended case: one indexing job per ref, already serialized by `LockProvider`.
|
|
205
|
+
- Stale-lock takeover (decision 2) is not covered by real multi-process fuzz testing, only the
|
|
206
|
+
documented race window and same-process tests. A separate multi-process stress test would be a
|
|
207
|
+
useful addition but is outside this assignment.
|
|
208
|
+
- **Search Journey records currently have no TTL or delete operation in the port.** `abandon` ends
|
|
209
|
+
measurement but does not remove its record. Files remain local until workspace state is
|
|
210
|
+
deliberately cleaned up.
|
|
211
|
+
|
|
212
|
+
<a id="lizenz"></a>
|
|
213
|
+
|
|
214
|
+
## License
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
**AGPL-3.0-only.** This package includes the complete license in [`LICENSE`](./LICENSE), the
|
|
218
|
+
license-choice rationale in [`NOTICE`](./NOTICE), and licenses for all third-party production
|
|
219
|
+
dependencies in [`THIRD_PARTY_LICENSES.md`](./THIRD_PARTY_LICENSES.md). All three ship in the npm
|
|
220
|
+
tarball: AGPL-3.0-only §4 and §5 require recipients to receive the license; a README alone is not
|
|
221
|
+
a copy of the license.
|
|
222
|
+
|
|
223
|
+
Local use alone does not trigger an obligation. Copyleft obligations apply to redistribution and
|
|
224
|
+
providing a network service (AGPL §13).
|
|
225
|
+
|
|
226
|
+
Corresponding TypeScript source ships under `src/` in the same tarball; the complete repository
|
|
227
|
+
is available at <https://github.com/lboel/untangle>.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Third-party licenses — @untangle-code/storage-local
|
|
2
|
+
|
|
3
|
+
Generated by `pnpm third-party-licenses` from this package's installed
|
|
4
|
+
production dependency tree: transitive `dependencies`, across
|
|
5
|
+
workspace edges, with no `devDependencies` included. The generator reads
|
|
6
|
+
each resolved package's `package.json` in `node_modules`; nothing is maintained by hand
|
|
7
|
+
or estimated.
|
|
8
|
+
|
|
9
|
+
Do not edit manually: `pnpm third-party-licenses --check` fails on drift,
|
|
10
|
+
and `pnpm check-publishable` rejects packages without a current file.
|
|
11
|
+
|
|
12
|
+
This file exists because `docs/BUILD_PLAN.md`, risk R6, makes
|
|
13
|
+
per-package `THIRD_PARTY_LICENSES.md` a release gate for npm publication.
|
|
14
|
+
|
|
15
|
+
Untangle itself is **AGPL-3.0-only** (see `LICENSE` and `NOTICE` in
|
|
16
|
+
this package). The packages below are third-party works; their own
|
|
17
|
+
licenses and copyright notices continue to apply unchanged. Workspace
|
|
18
|
+
packages (`@untangle-code/*`) are not listed here; they belong to the same
|
|
19
|
+
work and use the same AGPL-3.0-only license.
|
|
20
|
+
|
|
21
|
+
## Result: no third-party dependencies
|
|
22
|
+
|
|
23
|
+
This package has no production dependencies outside the workspace.
|
|
24
|
+
The list is empty because none exist, not because they were not checked.
|
package/dist/abort.d.ts
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Shared `AbortSignal` handling (spec.md §22: "public APIs MUST support
|
|
3
|
+
* abort signals"; rule 9: every long-running method respects `AbortSignal`
|
|
4
|
+
* and aborts with `JOB_CANCELLED`).
|
|
5
|
+
*
|
|
6
|
+
* Every port method in this package is `async`, so throwing synchronously
|
|
7
|
+
* here still surfaces as a rejected Promise to the caller — unlike
|
|
8
|
+
* `@untangle-code/testkit`'s `abortedRejection` (which exists specifically
|
|
9
|
+
* for a few NON-`async` methods where a synchronous throw would be a bug).
|
|
10
|
+
*/
|
|
11
|
+
import type { PortCallOptions } from "@untangle-code/domain";
|
|
12
|
+
/** Throws `UntangleError("JOB_CANCELLED", …)` when `options.signal` is already aborted. */
|
|
13
|
+
export declare function assertNotAborted(options: PortCallOptions | undefined, operation: string): void;
|
|
14
|
+
/** Builds a `PortCallOptions` from an optional `AbortSignal` without ever setting
|
|
15
|
+
* `signal: undefined` explicitly — required because domain's `PortCallOptions` is
|
|
16
|
+
* checked under `exactOptionalPropertyTypes`. */
|
|
17
|
+
export declare function portOptions(signal: AbortSignal | undefined): PortCallOptions | undefined;
|
|
18
|
+
//# sourceMappingURL=abort.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"abort.d.ts","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AACH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAG7D,2FAA2F;AAC3F,wBAAgB,gBAAgB,CAAC,OAAO,EAAE,eAAe,GAAG,SAAS,EAAE,SAAS,EAAE,MAAM,GAAG,IAAI,CAI9F;AAED;;iDAEiD;AACjD,wBAAgB,WAAW,CAAC,MAAM,EAAE,WAAW,GAAG,SAAS,GAAG,eAAe,GAAG,SAAS,CAExF"}
|
package/dist/abort.js
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
import { UntangleError } from "@untangle-code/domain";
|
|
2
|
+
/** Throws `UntangleError("JOB_CANCELLED", …)` when `options.signal` is already aborted. */
|
|
3
|
+
export function assertNotAborted(options, operation) {
|
|
4
|
+
if (options?.signal?.aborted) {
|
|
5
|
+
throw new UntangleError("JOB_CANCELLED", `"${operation}" was aborted`, { operation });
|
|
6
|
+
}
|
|
7
|
+
}
|
|
8
|
+
/** Builds a `PortCallOptions` from an optional `AbortSignal` without ever setting
|
|
9
|
+
* `signal: undefined` explicitly — required because domain's `PortCallOptions` is
|
|
10
|
+
* checked under `exactOptionalPropertyTypes`. */
|
|
11
|
+
export function portOptions(signal) {
|
|
12
|
+
return signal === undefined ? undefined : { signal };
|
|
13
|
+
}
|
|
14
|
+
//# sourceMappingURL=abort.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"abort.js","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAWA,OAAO,EAAE,aAAa,EAAE,MAAM,uBAAuB,CAAC;AAEtD,2FAA2F;AAC3F,MAAM,UAAU,gBAAgB,CAAC,OAAoC,EAAE,SAAiB;IACtF,IAAI,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC;QAC7B,MAAM,IAAI,aAAa,CAAC,eAAe,EAAE,IAAI,SAAS,eAAe,EAAE,EAAE,SAAS,EAAE,CAAC,CAAC;IACxF,CAAC;AACH,CAAC;AAED;;iDAEiD;AACjD,MAAM,UAAU,WAAW,CAAC,MAA+B;IACzD,OAAO,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC;AACvD,CAAC"}
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
import type { AnalysisCacheKind, Clock, FileSystemPort, PortCallOptions, Random } from "@untangle-code/domain";
|
|
2
|
+
export { ANALYSIS_CACHE_KINDS, buildChunkCacheKey, buildMetricsCacheKey, buildParseCacheKey, type AnalysisCacheKind, type AnalysisIdentityKeyPartsV1, } from "@untangle-code/domain";
|
|
3
|
+
/** 256 MiB je Cache — eine Untangle-Vorgabe, keine Spec-Zahl. */
|
|
4
|
+
export declare const DEFAULT_ANALYSIS_CACHE_MAX_BYTES: number;
|
|
5
|
+
/** Nach so vielen Mutationen wird das Manifest geschrieben. */
|
|
6
|
+
export declare const MANIFEST_FLUSH_INTERVAL = 64;
|
|
7
|
+
export interface ContentCacheStatsV1 {
|
|
8
|
+
entryCount: number;
|
|
9
|
+
totalBytes: number;
|
|
10
|
+
/** Treffer seit dem Oeffnen dieses Cache-Objekts. */
|
|
11
|
+
hits: number;
|
|
12
|
+
/** Fehlschlaege seit dem Oeffnen dieses Cache-Objekts. */
|
|
13
|
+
misses: number;
|
|
14
|
+
/** Trefferquote in [0,1]; 0, wenn nichts nachgeschlagen wurde. */
|
|
15
|
+
hitRate: number;
|
|
16
|
+
/** Wie viele Eintraege die Groessengrenze verdraengt hat. */
|
|
17
|
+
evictions: number;
|
|
18
|
+
}
|
|
19
|
+
/**
|
|
20
|
+
* Ein inhaltsadressierter JSON-Cache mit Groessengrenze, Verdraengung und
|
|
21
|
+
* Trefferquote. Der Schluessel ist eine undurchsichtige Zeichenkette; wer ihn
|
|
22
|
+
* baut, verantwortet seine Vollstaendigkeit (siehe Kopf dieser Datei).
|
|
23
|
+
*/
|
|
24
|
+
export declare class LocalContentCache {
|
|
25
|
+
private readonly fs;
|
|
26
|
+
private readonly dir;
|
|
27
|
+
private readonly clock;
|
|
28
|
+
private readonly random;
|
|
29
|
+
private readonly maxBytes;
|
|
30
|
+
private manifest;
|
|
31
|
+
private dirty;
|
|
32
|
+
private hits;
|
|
33
|
+
private misses;
|
|
34
|
+
private evictions;
|
|
35
|
+
constructor(fs: FileSystemPort, dir: string, clock: Clock, random: Random, options?: {
|
|
36
|
+
maxBytes?: number;
|
|
37
|
+
});
|
|
38
|
+
private manifestPath;
|
|
39
|
+
private entryFileName;
|
|
40
|
+
private entryPath;
|
|
41
|
+
private loadManifest;
|
|
42
|
+
private persistManifest;
|
|
43
|
+
private touched;
|
|
44
|
+
/** Schreibt das Manifest, falls Mutationen ausstehen. Am Ende eines Laufs aufrufen. */
|
|
45
|
+
flush(options?: PortCallOptions): Promise<void>;
|
|
46
|
+
/** Der Wert, oder `null` fuer "nicht im Cache". Wirft nicht fuer kaputte Eintraege. */
|
|
47
|
+
get(logicalKey: string, options?: PortCallOptions): Promise<unknown | null>;
|
|
48
|
+
set(logicalKey: string, value: unknown, options?: PortCallOptions): Promise<void>;
|
|
49
|
+
private evictUntilWithinLimit;
|
|
50
|
+
stats(options?: PortCallOptions): Promise<ContentCacheStatsV1>;
|
|
51
|
+
/** Loescht ausschliesslich das eigene Verzeichnis dieses Caches. */
|
|
52
|
+
clear(options?: PortCallOptions): Promise<void>;
|
|
53
|
+
}
|
|
54
|
+
/** Trefferquoten aller drei Caches, einzeln und zusammen. */
|
|
55
|
+
export interface AnalysisCacheStatsV1 {
|
|
56
|
+
parse: ContentCacheStatsV1;
|
|
57
|
+
chunk: ContentCacheStatsV1;
|
|
58
|
+
metrics: ContentCacheStatsV1;
|
|
59
|
+
totalHits: number;
|
|
60
|
+
totalMisses: number;
|
|
61
|
+
/** Trefferquote ueber alle drei Caches in [0,1]. */
|
|
62
|
+
hitRate: number;
|
|
63
|
+
}
|
|
64
|
+
/**
|
|
65
|
+
* Die drei Caches als ein Gegenstand — genau die Form, die
|
|
66
|
+
* `@untangle-code/core` als `AnalysisCacheLike` erwartet.
|
|
67
|
+
*/
|
|
68
|
+
export declare class LocalAnalysisCache {
|
|
69
|
+
readonly parse: LocalContentCache;
|
|
70
|
+
readonly chunk: LocalContentCache;
|
|
71
|
+
readonly metrics: LocalContentCache;
|
|
72
|
+
constructor(parse: LocalContentCache, chunk: LocalContentCache, metrics: LocalContentCache);
|
|
73
|
+
private of;
|
|
74
|
+
get(kind: AnalysisCacheKind, key: string, options?: PortCallOptions): Promise<unknown | null>;
|
|
75
|
+
set(kind: AnalysisCacheKind, key: string, value: unknown, options?: PortCallOptions): Promise<void>;
|
|
76
|
+
/** Schreibt die Manifeste aller drei Caches. */
|
|
77
|
+
flush(options?: PortCallOptions): Promise<void>;
|
|
78
|
+
stats(options?: PortCallOptions): Promise<AnalysisCacheStatsV1>;
|
|
79
|
+
clear(options?: PortCallOptions): Promise<void>;
|
|
80
|
+
}
|
|
81
|
+
//# sourceMappingURL=analysis-cache.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"analysis-cache.d.ts","sourceRoot":"","sources":["../src/analysis-cache.ts"],"names":[],"mappings":"AAsCA,OAAO,KAAK,EACV,iBAAiB,EACjB,KAAK,EACL,cAAc,EACd,eAAe,EACf,MAAM,EACP,MAAM,uBAAuB,CAAC;AAW/B,OAAO,EACL,oBAAoB,EACpB,kBAAkB,EAClB,oBAAoB,EACpB,kBAAkB,EAClB,KAAK,iBAAiB,EACtB,KAAK,0BAA0B,GAChC,MAAM,uBAAuB,CAAC;AAE/B,iEAAiE;AACjE,eAAO,MAAM,gCAAgC,QAAoB,CAAC;AAElE,+DAA+D;AAC/D,eAAO,MAAM,uBAAuB,KAAK,CAAC;AAI1C,MAAM,WAAW,mBAAmB;IAClC,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,MAAM,CAAC;IACnB,qDAAqD;IACrD,IAAI,EAAE,MAAM,CAAC;IACb,0DAA0D;IAC1D,MAAM,EAAE,MAAM,CAAC;IACf,kEAAkE;IAClE,OAAO,EAAE,MAAM,CAAC;IAChB,6DAA6D;IAC7D,SAAS,EAAE,MAAM,CAAC;CACnB;AAgDD;;;;GAIG;AACH,qBAAa,iBAAiB;IAS1B,OAAO,CAAC,QAAQ,CAAC,EAAE;IACnB,OAAO,CAAC,QAAQ,CAAC,GAAG;IACpB,OAAO,CAAC,QAAQ,CAAC,KAAK;IACtB,OAAO,CAAC,QAAQ,CAAC,MAAM;IAXzB,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAA2B;IAC3C,OAAO,CAAC,KAAK,CAAK;IAClB,OAAO,CAAC,IAAI,CAAK;IACjB,OAAO,CAAC,MAAM,CAAK;IACnB,OAAO,CAAC,SAAS,CAAK;gBAGH,EAAE,EAAE,cAAc,EAClB,GAAG,EAAE,MAAM,EACX,KAAK,EAAE,KAAK,EACZ,MAAM,EAAE,MAAM,EAC/B,OAAO,CAAC,EAAE;QAAE,QAAQ,CAAC,EAAE,MAAM,CAAA;KAAE;IAKjC,OAAO,CAAC,YAAY;IAIpB,OAAO,CAAC,aAAa;IAKrB,OAAO,CAAC,SAAS;YAIH,YAAY;YA2BZ,eAAe;YAYf,OAAO;IAOrB,uFAAuF;IAC1E,KAAK,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAM5D,uFAAuF;IAC1E,GAAG,CAAC,UAAU,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,OAAO,GAAG,IAAI,CAAC;IAmC3E,GAAG,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;YAwBhF,qBAAqB;IA0BtB,KAAK,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,mBAAmB,CAAC;IAc3E,oEAAoE;IACvD,KAAK,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;CAM7D;AAED,6DAA6D;AAC7D,MAAM,WAAW,oBAAoB;IACnC,KAAK,EAAE,mBAAmB,CAAC;IAC3B,KAAK,EAAE,mBAAmB,CAAC;IAC3B,OAAO,EAAE,mBAAmB,CAAC;IAC7B,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,oDAAoD;IACpD,OAAO,EAAE,MAAM,CAAC;CACjB;AAED;;;GAGG;AACH,qBAAa,kBAAkB;aAEX,KAAK,EAAE,iBAAiB;aACxB,KAAK,EAAE,iBAAiB;aACxB,OAAO,EAAE,iBAAiB;gBAF1B,KAAK,EAAE,iBAAiB,EACxB,KAAK,EAAE,iBAAiB,EACxB,OAAO,EAAE,iBAAiB;IAG5C,OAAO,CAAC,EAAE;IAMG,GAAG,CACd,IAAI,EAAE,iBAAiB,EACvB,GAAG,EAAE,MAAM,EACX,OAAO,CAAC,EAAE,eAAe,GACxB,OAAO,CAAC,OAAO,GAAG,IAAI,CAAC;IAIb,GAAG,CACd,IAAI,EAAE,iBAAiB,EACvB,GAAG,EAAE,MAAM,EACX,KAAK,EAAE,OAAO,EACd,OAAO,CAAC,EAAE,eAAe,GACxB,OAAO,CAAC,IAAI,CAAC;IAIhB,gDAAgD;IACnC,KAAK,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAM/C,KAAK,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,oBAAoB,CAAC;IAiB/D,KAAK,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;CAK7D"}
|