@untangle-code/jobs 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +661 -0
- package/NOTICE +90 -0
- package/README.md +138 -2
- package/THIRD_PARTY_LICENSES.md +24 -0
- package/dist/abort.d.ts +14 -0
- package/dist/abort.d.ts.map +1 -0
- package/dist/abort.js +35 -0
- package/dist/abort.js.map +1 -0
- package/dist/bounded-queue.d.ts +200 -0
- package/dist/bounded-queue.d.ts.map +1 -0
- package/dist/bounded-queue.js +338 -0
- package/dist/bounded-queue.js.map +1 -0
- package/dist/checkpoint-store.d.ts +30 -0
- package/dist/checkpoint-store.d.ts.map +1 -0
- package/dist/checkpoint-store.js +131 -0
- package/dist/checkpoint-store.js.map +1 -0
- package/dist/events.d.ts +29 -0
- package/dist/events.d.ts.map +1 -0
- package/dist/events.js +66 -0
- package/dist/events.js.map +1 -0
- package/dist/index.d.ts +22 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +30 -0
- package/dist/index.js.map +1 -0
- package/dist/resume.d.ts +45 -0
- package/dist/resume.d.ts.map +1 -0
- package/dist/resume.js +60 -0
- package/dist/resume.js.map +1 -0
- package/dist/runner.d.ts +148 -0
- package/dist/runner.d.ts.map +1 -0
- package/dist/runner.js +686 -0
- package/dist/runner.js.map +1 -0
- package/dist/stages.d.ts +146 -0
- package/dist/stages.d.ts.map +1 -0
- package/dist/stages.js +2 -0
- package/dist/stages.js.map +1 -0
- package/dist/state-machine.d.ts +52 -0
- package/dist/state-machine.d.ts.map +1 -0
- package/dist/state-machine.js +114 -0
- package/dist/state-machine.js.map +1 -0
- package/package.json +61 -3
- package/src/abort.ts +37 -0
- package/src/bounded-queue.ts +462 -0
- package/src/checkpoint-store.ts +171 -0
- package/src/events.ts +71 -0
- package/src/index.ts +37 -0
- package/src/resume.ts +106 -0
- package/src/runner.ts +916 -0
- package/src/stages.ts +167 -0
- package/src/state-machine.ts +132 -0
package/NOTICE
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# NOTICE
|
|
2
|
+
|
|
3
|
+
Copyright (C) 2026 Luke Bölling and contributors.
|
|
4
|
+
|
|
5
|
+
This attribution covers original Untangle work. Contributors retain copyright in
|
|
6
|
+
their contributions. Third-party packages, grammars and other materials retain
|
|
7
|
+
their own copyright holders, licenses and notices; this line does not replace them.
|
|
8
|
+
|
|
9
|
+
## License — binding decision
|
|
10
|
+
|
|
11
|
+
Untangle is licensed under the **GNU Affero General Public License, Version 3
|
|
12
|
+
(AGPL-3.0-only)**. See [`LICENSE`](./LICENSE).
|
|
13
|
+
|
|
14
|
+
This decision is **final, not provisional**, and was explicitly instructed by
|
|
15
|
+
the project owner. No contributor, agent or tool may change it without a new
|
|
16
|
+
explicit instruction.
|
|
17
|
+
|
|
18
|
+
The repository-wide license covers Untangle's first-party source code, scripts,
|
|
19
|
+
tests, fixtures, configuration, documentation, website and brand artwork, including
|
|
20
|
+
the logo and 3D yarn illustrations. Every workspace `package.json` declares
|
|
21
|
+
`"license": "AGPL-3.0-only"`. There is no separately licensed enterprise component
|
|
22
|
+
or separate permissive license for documentation or brand assets.
|
|
23
|
+
|
|
24
|
+
Third-party code, embedded grammars, dependency assets and their notices retain
|
|
25
|
+
their original licenses. Their inclusion does not relicense those works. The
|
|
26
|
+
repository license does not grant trademark rights or change the license of a
|
|
27
|
+
repository analyzed with Untangle. See the repository's `docs/LICENSING.md` for
|
|
28
|
+
scope and verification details.
|
|
29
|
+
|
|
30
|
+
This resolves the license question left open in `vision.md` §14 and completes
|
|
31
|
+
`product.md` §16, item 2.
|
|
32
|
+
|
|
33
|
+
## Rationale
|
|
34
|
+
|
|
35
|
+
- **Network copyleft fits the intended product model.** AGPL §13 sets source
|
|
36
|
+
availability requirements for modified versions used through a computer network.
|
|
37
|
+
The current local CLI/stdio-MCP entry and future shared-server modes share the
|
|
38
|
+
same license.
|
|
39
|
+
- **Local-first analysis has a clear boundary.** Running Untangle against a codebase
|
|
40
|
+
does not replace that codebase's own license with Untangle's license.
|
|
41
|
+
- **First-party components share one license.** Implemented components and future
|
|
42
|
+
official plugins use this repository's licensing policy. Planned integrations do
|
|
43
|
+
not establish implemented or published products.
|
|
44
|
+
|
|
45
|
+
## Deliberately accepted consequence
|
|
46
|
+
|
|
47
|
+
The project deliberately chooses AGPL rather than a permissive embedding license.
|
|
48
|
+
Redistribution, modifications and combined works must be assessed against the
|
|
49
|
+
actual terms in `LICENSE`, including applicable Corresponding Source requirements.
|
|
50
|
+
No permissive exception, commercial dual license or special contract-only package
|
|
51
|
+
license is granted or promised by this notice.
|
|
52
|
+
|
|
53
|
+
## Dependencies
|
|
54
|
+
|
|
55
|
+
Third-party runtime dependencies are checked against the allowlist in
|
|
56
|
+
`scripts/check-licenses.mjs`. Their licenses and copyright notices remain unchanged
|
|
57
|
+
and traceable through package `THIRD_PARTY_LICENSES.md` files, grammar provenance,
|
|
58
|
+
the lockfile and release SBOM. The allowlist includes MIT / Apache-2.0 / BSD / ISC
|
|
59
|
+
and other explicitly listed licenses; it is not a license grant for dependencies.
|
|
60
|
+
|
|
61
|
+
The allowlist in `scripts/check-licenses.mjs` applies to **dependencies**, not
|
|
62
|
+
workspace packages. Untangle's own packages carry AGPL-3.0-only and are excluded
|
|
63
|
+
from that check.
|
|
64
|
+
|
|
65
|
+
## Clean room
|
|
66
|
+
|
|
67
|
+
Untangle is an independent implementation with its own data models, APIs, tool
|
|
68
|
+
names, tests and sources. Feature ideas may be derived from publicly observable
|
|
69
|
+
behavior and generally known techniques. Third-party source code is not copied,
|
|
70
|
+
including from AGPL projects even where the license would permit it.
|
|
71
|
+
|
|
72
|
+
## Contribution terms
|
|
73
|
+
|
|
74
|
+
Pull requests are welcome under the repository's existing AGPL-3.0-only license.
|
|
75
|
+
No separate CLA, copyright assignment or DCO sign-off is required. Contributors
|
|
76
|
+
must have the right to submit their work under those terms. See CONTRIBUTING.md.
|
|
77
|
+
|
|
78
|
+
## Release decisions
|
|
79
|
+
|
|
80
|
+
- [x] Copyright attribution: Luke Bölling, with contributors retaining their rights.
|
|
81
|
+
- [x] Contributions use AGPL-3.0-only without an additional CLA or DCO gate.
|
|
82
|
+
|
|
83
|
+
## Open legal items before public beta
|
|
84
|
+
|
|
85
|
+
- [ ] Trademark search for the product name `Untangle`.
|
|
86
|
+
- [ ] Registration and legal review of the npm scope `@untangle-code`.
|
|
87
|
+
- [ ] Verify source availability and notices for each released network daemon/server
|
|
88
|
+
mode (WP-59). The current stdio MCP `untangle_about` already reports the license,
|
|
89
|
+
source repository and build commit when supplied; metadata alone does not prove
|
|
90
|
+
that a concrete deployed version's source is available.
|
package/README.md
CHANGED
|
@@ -1,3 +1,139 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @untangle-code/jobs
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The Untangle indexer's job core (WP-13): state machine, typed job events, checkpoints, resume
|
|
4
|
+
planning, generation commit, bounded queues with backpressure and cancellation through `AbortSignal`.
|
|
5
|
+
|
|
6
|
+
Normative sources: `spec.md` §10.5, §11, §12.2, §22; `architecture.md` §2.2, §6.14, §13.1–§13.4,
|
|
7
|
+
§21.3–§21.4, §27.5.
|
|
8
|
+
|
|
9
|
+
<a id="inhalt"></a>
|
|
10
|
+
|
|
11
|
+
## Contents
|
|
12
|
+
|
|
13
|
+
- **`JobStateMachine`** (`state-machine.ts`): exactly the state machine from spec.md §11.1,
|
|
14
|
+
including `failed`, `cancel_requested`, `cancelled`. Illegal transitions are rejected with typed
|
|
15
|
+
errors, never silently ignored.
|
|
16
|
+
- **`AsyncEventChannel`** (`events.ts`): exposes the typed job events from architecture.md §2.2
|
|
17
|
+
(types in `@untangle-code/domain`) as `AsyncIterable<JobEvent>` (spec.md §22).
|
|
18
|
+
- **`BoundedQueue`** (`bounded-queue.ts`): bounded queues from architecture.md §13.2. A full queue
|
|
19
|
+
blocks `push()`, pausing the previous stage. Limits apply to count AND bytes; high-water marks
|
|
20
|
+
are visible through `stats()`.
|
|
21
|
+
- **`FileCheckpointStore`** (`checkpoint-store.ts`): `CheckpointStorePort` using `FileSystemPort`,
|
|
22
|
+
atomic through temp+rename, with the exact `IndexCheckpointV1` shape (§13.3).
|
|
23
|
+
- **`planResume`** (`resume.ts`): reconstructs remaining work from the last acknowledged checkpoint
|
|
24
|
+
(spec.md §11.2).
|
|
25
|
+
- **Stage interfaces** (`stages.ts`): narrow injectable contracts for discovery, analysis,
|
|
26
|
+
embedding, storage, graphs, context and generation commit. This package builds pipeline
|
|
27
|
+
MECHANICS, not domain stages.
|
|
28
|
+
- **`IndexJobRunner`** (`runner.ts`): orchestrates events, queues, checkpoints only after
|
|
29
|
+
acknowledged upserts (spec.md §10.5), cancellation only at safe batch boundaries
|
|
30
|
+
(architecture.md §21.3), and `completed` only after complete generation commit (§13.4, enforced
|
|
31
|
+
by domain's `assertGenerationComplete`). On success its terminal observer also publishes the
|
|
32
|
+
immutable `corpusRevisionFingerprint` of the fully scanned corpus manifest.
|
|
33
|
+
|
|
34
|
+
<a id="entscheidungen"></a>
|
|
35
|
+
|
|
36
|
+
## Decisions
|
|
37
|
+
|
|
38
|
+
Deliberate choices left open by source documents:
|
|
39
|
+
|
|
40
|
+
1. **State-machine transition relation.** spec.md §11.1 lists the pipeline chain and error states,
|
|
41
|
+
but not every edge. Only the immediately next stage is reachable along the chain; `failed` and
|
|
42
|
+
`cancel_requested` are reachable from every nonterminal pipeline stage; `cancel_requested`
|
|
43
|
+
leads only to `cancelled` (or `failed` if cancellation cleanup itself fails); `completed`,
|
|
44
|
+
`failed`, `cancelled` are terminal; self-transitions are not transitions.
|
|
45
|
+
2. **Stage as a high-water mark.** The pipeline overlaps work, using backpressure rather than
|
|
46
|
+
phase barriers. The state machine therefore advances as a high-water mark (`advanceTo`) when
|
|
47
|
+
the first work unit reaches a phase. Every individual movement is a legal chain transition;
|
|
48
|
+
it never moves backward. A run without remaining work also crosses intermediate stages legally
|
|
49
|
+
with no work.
|
|
50
|
+
3. **Error code for illegal transitions and untyped stage errors.** The closed catalog
|
|
51
|
+
(spec.md §25.1) has no "internal state error" code. Illegal transitions or untyped stage errors
|
|
52
|
+
leave the index incomplete, so they use `INDEX_INCOMPLETE` (`retryable: false`), never a guessed
|
|
53
|
+
infrastructure code (spec.md §25.2). A dedicated code would be clearer and is recorded as an
|
|
54
|
+
additive proposal for the domain owner.
|
|
55
|
+
4. **Corrupt checkpoint is not missing checkpoint.** `load`/`loadLatest` return `null` only for
|
|
56
|
+
absence. A corrupt checkpoint throws `INDEX_INCOMPLETE`; silently ignoring it would discard
|
|
57
|
+
acknowledged work and silently restart, prohibited by spec.md §25.2 and E4.
|
|
58
|
+
5. **Resume validity.** A checkpoint is accepted only when workspace, repository, ref AND
|
|
59
|
+
`effectiveProfileId` agree (incompatible profiles require a new generation, spec.md §11.2).
|
|
60
|
+
An acknowledged file is skipped only when its content hash also remains unchanged (hash is
|
|
61
|
+
the source of truth, spec.md §12.1). Changed or no-longer-planned files leave the adopted state
|
|
62
|
+
and require fresh acknowledgement; they are never counted twice.
|
|
63
|
+
6. **Event-stream cancellation semantics.** `run()` does NOT throw for job failure or
|
|
64
|
+
cancellation; it ends the stream with exactly one terminal event
|
|
65
|
+
(`job_completed` | `job_failed` | `job_cancelled`), the consumable form of returning `cancelled`
|
|
66
|
+
(architecture.md §21.3). Inside the package, long-running methods abort with
|
|
67
|
+
`UntangleError("JOB_CANCELLED")` (spec.md §22). A consumer stopping iteration stops the job
|
|
68
|
+
through internal abort; without external abort, this counts as failure rather than user cancellation.
|
|
69
|
+
7. **Memory budget.** One budget (`memoryBudgetBytes`, default 64 MiB) is split equally between
|
|
70
|
+
analysis and upsert queues and appears alongside queue depths in `status()`
|
|
71
|
+
(architecture.md §13.2). `queueCapacities`, `embedBatchSize` and `memoryBudgetBytes` are validated
|
|
72
|
+
when constructing the runner (`CONFIG_INVALID`), not halfway through a run.
|
|
73
|
+
|
|
74
|
+
The **only** exception is an item that **alone** exceeds the entire byte budget. Rejecting it
|
|
75
|
+
would silently drop data; waiting would hang rather than apply backpressure because no pull
|
|
76
|
+
could ever create enough room. The exception is narrow:
|
|
77
|
+
- It applies to the ITEM (`size > maxBytes`), not queue occupancy;
|
|
78
|
+
- It applies only to an empty queue so the violation never accumulates with other items;
|
|
79
|
+
- It has a quota (`maxOverBudgetAdmissions`, default 1); once exhausted, `push()` fails with
|
|
80
|
+
`INDEX_INCOMPLETE` rather than permanently exceeding the budget;
|
|
81
|
+
- Every admission is counted (`stats().overBudgetAdmissions`), reported
|
|
82
|
+
(`onOverBudgetAdmission`, logged by the runner as `queue_over_budget_admission`) and reflected
|
|
83
|
+
in high-water marks. Budget violations are never silent (E4).
|
|
84
|
+
|
|
85
|
+
Before WP-116, the byte branch depended on `slots.length > 0`. With `capacity: 1`, the queue is
|
|
86
|
+
empty before every push, so the exception applied to EVERY push: 20 × 50 MB crossed a 100-byte
|
|
87
|
+
budget without a single blocked push, despite fully valid configuration.
|
|
88
|
+
8. **Checkpoint batch equals embedding batch.** `embedBatchSize` defines the configured file
|
|
89
|
+
batch in spec.md §11.2: exactly one checkpoint is written after each acknowledged batch
|
|
90
|
+
upsert, never before (spec.md §10.5).
|
|
91
|
+
9. **Commit order.** Before generation commit, the runner prechecks through
|
|
92
|
+
`validateGenerationCompleteness` with assumed metadata status; AFTER commit,
|
|
93
|
+
`assertGenerationComplete` also enforces the actually reached final metadata state. An
|
|
94
|
+
incomplete generation is never submitted for commit.
|
|
95
|
+
10. **Optional injected locks.** When optional `lockProvider` is set, the job holds
|
|
96
|
+
`repo:<repoId>:<refId>:index` (architecture.md §21.1). Validated `lockLeaseMs` defaults to
|
|
97
|
+
30 seconds; the runner renews without overlap no later than one third of its duration.
|
|
98
|
+
Renewal failure aborts the internal pipeline as `LOCK_HELD`. A final renewal proves ownership
|
|
99
|
+
again before the terminal observer; a runner detecting ownership loss does not invoke it.
|
|
100
|
+
Adapter-owned current pointers additionally need serialization per repository/ref, ownership
|
|
101
|
+
checks immediately before publication and generation-CAS replacement. Stale detection and the
|
|
102
|
+
owner-checked renewal operation itself belong to the provider.
|
|
103
|
+
11. **Unbounded event channel.** Job events are small and finite per batch; bounded WORK queues,
|
|
104
|
+
rather than the event stream, limit pipeline memory pressure. Slow event consumption therefore
|
|
105
|
+
does not delay the job.
|
|
106
|
+
12. **Corpus revision from the complete scan, not the changed subset.** Discovery returns the
|
|
107
|
+
complete mapping of file IDs to content hashes. Only after successful generation commit does
|
|
108
|
+
the runner hash this canonical manifest and pass its immutable `corpusRevisionFingerprint`
|
|
109
|
+
to `onGenerationFinished`. This also applies to incremental runs with zero changed files.
|
|
110
|
+
Failed or cancelled runs publish no corpus revision; reused generations therefore cannot merge
|
|
111
|
+
different corpus snapshots in downstream Search Journeys.
|
|
112
|
+
|
|
113
|
+
## Tests
|
|
114
|
+
|
|
115
|
+
Co-located `src/**/*.test.ts`, using only `@untangle-code/testkit` (FakeClock, DeterministicRandom,
|
|
116
|
+
MemoryFileSystem, InMemoryLockProvider), with no network or Docker. Coverage includes the complete
|
|
117
|
+
transition relation (all 15×15 pairs), demonstrated backpressure (a blocked storage stage limits
|
|
118
|
+
analysis to queue capacity plus in-flight work), cancellation producing valid checkpoints and
|
|
119
|
+
`cancelled`, exact checkpoint resume without repeating acknowledged work, incomplete generations
|
|
120
|
+
never reaching `completed`, successful zero-change runs producing corpus revisions from complete
|
|
121
|
+
manifests, and fault injection from architecture.md §27.5 at points 1 (after embedding/before
|
|
122
|
+
upsert), 3 (after upsert/before checkpoint) and 5 (after checkpoint/before generation commit), with
|
|
123
|
+
post-restart consistency evidence.
|
|
124
|
+
|
|
125
|
+
<a id="lizenz"></a>
|
|
126
|
+
|
|
127
|
+
## License
|
|
128
|
+
|
|
129
|
+
**AGPL-3.0-only.** This package includes the complete license in [`LICENSE`](./LICENSE), the
|
|
130
|
+
license-choice rationale in [`NOTICE`](./NOTICE), and licenses for all third-party production
|
|
131
|
+
dependencies in [`THIRD_PARTY_LICENSES.md`](./THIRD_PARTY_LICENSES.md). All three ship in the npm
|
|
132
|
+
tarball: AGPL-3.0-only §4 and §5 require recipients to receive the license; a README alone is not
|
|
133
|
+
a copy of the license.
|
|
134
|
+
|
|
135
|
+
Local use alone does not trigger an obligation. Copyleft obligations apply to redistribution and
|
|
136
|
+
providing a network service (AGPL §13).
|
|
137
|
+
|
|
138
|
+
Corresponding TypeScript source ships under `src/` in the same tarball; the complete repository
|
|
139
|
+
is available at <https://github.com/lboel/untangle>.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Third-party licenses — @untangle-code/jobs
|
|
2
|
+
|
|
3
|
+
Generated by `pnpm third-party-licenses` from this package's installed
|
|
4
|
+
production dependency tree: transitive `dependencies`, across
|
|
5
|
+
workspace edges, with no `devDependencies` included. The generator reads
|
|
6
|
+
each resolved package's `package.json` in `node_modules`; nothing is maintained by hand
|
|
7
|
+
or estimated.
|
|
8
|
+
|
|
9
|
+
Do not edit manually: `pnpm third-party-licenses --check` fails on drift,
|
|
10
|
+
and `pnpm check-publishable` rejects packages without a current file.
|
|
11
|
+
|
|
12
|
+
This file exists because `docs/BUILD_PLAN.md`, risk R6, makes
|
|
13
|
+
per-package `THIRD_PARTY_LICENSES.md` a release gate for npm publication.
|
|
14
|
+
|
|
15
|
+
Untangle itself is **AGPL-3.0-only** (see `LICENSE` and `NOTICE` in
|
|
16
|
+
this package). The packages below are third-party works; their own
|
|
17
|
+
licenses and copyright notices continue to apply unchanged. Workspace
|
|
18
|
+
packages (`@untangle-code/*`) are not listed here; they belong to the same
|
|
19
|
+
work and use the same AGPL-3.0-only license.
|
|
20
|
+
|
|
21
|
+
## Result: no third-party dependencies
|
|
22
|
+
|
|
23
|
+
This package has no production dependencies outside the workspace.
|
|
24
|
+
The list is empty because none exist, not because they were not checked.
|
package/dist/abort.d.ts
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Throws `UntangleError("JOB_CANCELLED", …)` when `signal` is already
|
|
3
|
+
* aborted. Called at every safe batch boundary (architecture.md §21.3).
|
|
4
|
+
*/
|
|
5
|
+
export declare function throwIfAborted(signal: AbortSignal | undefined, operation: string): void;
|
|
6
|
+
/** True when `error` is an `UntangleError` with code `JOB_CANCELLED`. */
|
|
7
|
+
export declare function isJobCancelledError(error: unknown): boolean;
|
|
8
|
+
/**
|
|
9
|
+
* Resolves when `signal` aborts. The listener is removed as soon as
|
|
10
|
+
* `cleanup` aborts, so racing this against another promise does not leak
|
|
11
|
+
* one listener per wait.
|
|
12
|
+
*/
|
|
13
|
+
export declare function abortedPromise(signal: AbortSignal, cleanup: AbortSignal): Promise<void>;
|
|
14
|
+
//# sourceMappingURL=abort.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"abort.d.ts","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAQA;;;GAGG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,WAAW,GAAG,SAAS,EAAE,SAAS,EAAE,MAAM,GAAG,IAAI,CAIvF;AAED,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,KAAK,EAAE,OAAO,GAAG,OAAO,CAE3D;AAED;;;;GAIG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,WAAW,EAAE,OAAO,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC,CAQvF"}
|
package/dist/abort.js
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* AbortSignal helpers shared by every long-running method in this package
|
|
3
|
+
* (spec.md §22: public APIs MUST support abort signals; cancellation
|
|
4
|
+
* surfaces as `UntangleError` with code `JOB_CANCELLED`).
|
|
5
|
+
*/
|
|
6
|
+
import { isUntangleError } from "@untangle-code/domain";
|
|
7
|
+
import { jobCancelledError } from "@untangle-code/observability";
|
|
8
|
+
/**
|
|
9
|
+
* Throws `UntangleError("JOB_CANCELLED", …)` when `signal` is already
|
|
10
|
+
* aborted. Called at every safe batch boundary (architecture.md §21.3).
|
|
11
|
+
*/
|
|
12
|
+
export function throwIfAborted(signal, operation) {
|
|
13
|
+
if (signal?.aborted) {
|
|
14
|
+
throw jobCancelledError(`"${operation}" was aborted`, { operation });
|
|
15
|
+
}
|
|
16
|
+
}
|
|
17
|
+
/** True when `error` is an `UntangleError` with code `JOB_CANCELLED`. */
|
|
18
|
+
export function isJobCancelledError(error) {
|
|
19
|
+
return isUntangleError(error) && error.code === "JOB_CANCELLED";
|
|
20
|
+
}
|
|
21
|
+
/**
|
|
22
|
+
* Resolves when `signal` aborts. The listener is removed as soon as
|
|
23
|
+
* `cleanup` aborts, so racing this against another promise does not leak
|
|
24
|
+
* one listener per wait.
|
|
25
|
+
*/
|
|
26
|
+
export function abortedPromise(signal, cleanup) {
|
|
27
|
+
return new Promise((resolve) => {
|
|
28
|
+
if (signal.aborted) {
|
|
29
|
+
resolve();
|
|
30
|
+
return;
|
|
31
|
+
}
|
|
32
|
+
signal.addEventListener("abort", () => resolve(), { once: true, signal: cleanup });
|
|
33
|
+
});
|
|
34
|
+
}
|
|
35
|
+
//# sourceMappingURL=abort.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"abort.js","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AACxD,OAAO,EAAE,iBAAiB,EAAE,MAAM,8BAA8B,CAAC;AAEjE;;;GAGG;AACH,MAAM,UAAU,cAAc,CAAC,MAA+B,EAAE,SAAiB;IAC/E,IAAI,MAAM,EAAE,OAAO,EAAE,CAAC;QACpB,MAAM,iBAAiB,CAAC,IAAI,SAAS,eAAe,EAAE,EAAE,SAAS,EAAE,CAAC,CAAC;IACvE,CAAC;AACH,CAAC;AAED,yEAAyE;AACzE,MAAM,UAAU,mBAAmB,CAAC,KAAc;IAChD,OAAO,eAAe,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC,IAAI,KAAK,eAAe,CAAC;AAClE,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,cAAc,CAAC,MAAmB,EAAE,OAAoB;IACtE,OAAO,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE;QACnC,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;YACnB,OAAO,EAAE,CAAC;YACV,OAAO;QACT,CAAC;QACD,MAAM,CAAC,gBAAgB,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC,CAAC;IACrF,CAAC,CAAC,CAAC;AACL,CAAC"}
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Bounded FIFO queue with backpressure (architecture.md §13.1–§13.2).
|
|
3
|
+
*
|
|
4
|
+
* A full queue makes `push()` WAIT — that is the mechanism by which a slow
|
|
5
|
+
* downstream stage pauses the previous stage instead of letting it
|
|
6
|
+
* accumulate unbounded ASTs/vectors in memory. Bounds are enforced on both
|
|
7
|
+
* item count (`capacity`) and, optionally, estimated bytes (`maxBytes` with
|
|
8
|
+
* a `sizeOf` cost function), so the pipeline's memory budget is a hard
|
|
9
|
+
* limit, not a hope. All observed values are exposed via `stats()` for the
|
|
10
|
+
* job status output (spec-driven visibility, architecture.md §13.2).
|
|
11
|
+
*
|
|
12
|
+
* Die EINZIGE Ausnahme vom Byte-Budget ist ein Element, das das ganze Budget
|
|
13
|
+
* allein uebersteigt: fuer so ein Element kann Warten nie Platz schaffen, es
|
|
14
|
+
* wuerde die Queue also nicht bremsen, sondern haengen lassen. Diese Ausnahme
|
|
15
|
+
* ist eng gefasst (leere Queue, kontingentiert, gezaehlt, gemeldet) — sie war
|
|
16
|
+
* frueher an den Fuellstand geknuepft und damit bei `capacity: 1` ein
|
|
17
|
+
* Dauerzustand, der die Byte-Grenze mit gueltigen Werten abschaltete (WP-116).
|
|
18
|
+
*/
|
|
19
|
+
import type { PortCallOptions } from "@untangle-code/domain";
|
|
20
|
+
/**
|
|
21
|
+
* Eine Aufnahme, die das Byte-Budget verletzt hat, weil das Element ALLEIN
|
|
22
|
+
* groesser als das gesamte Budget ist. Wird gezaehlt und gemeldet — eine
|
|
23
|
+
* Budgetverletzung darf nie stillschweigend passieren (E4).
|
|
24
|
+
*/
|
|
25
|
+
export interface OverBudgetAdmissionV1 {
|
|
26
|
+
/** Groesse des aufgenommenen Elements in Bytes (> `maxBytes`). */
|
|
27
|
+
itemBytes: number;
|
|
28
|
+
/** Das Budget, das dieses Element allein ueberschreitet. */
|
|
29
|
+
maxBytes: number;
|
|
30
|
+
/** Die wievielte Ausnahme dieser Queue (1-basiert). */
|
|
31
|
+
ordinal: number;
|
|
32
|
+
/** Wie viele solcher Ausnahmen diese Queue insgesamt zulaesst. */
|
|
33
|
+
allowance: number;
|
|
34
|
+
}
|
|
35
|
+
export interface BoundedQueueOptions<T> {
|
|
36
|
+
/** Maximum number of buffered items; must be a safe integer >= 1. */
|
|
37
|
+
capacity: number;
|
|
38
|
+
/** Optional byte budget across all buffered items. Requires `sizeOf`. */
|
|
39
|
+
maxBytes?: number;
|
|
40
|
+
/** Estimated size of one item in bytes; required when `maxBytes` is set. */
|
|
41
|
+
sizeOf?: (item: T) => number;
|
|
42
|
+
/**
|
|
43
|
+
* Wie viele Elemente, die das Byte-Budget ALLEIN uebersteigen, darf diese
|
|
44
|
+
* Queue insgesamt aufnehmen? Vorgabe: 1.
|
|
45
|
+
*
|
|
46
|
+
* Die Ausnahme existiert nur, damit EIN einzelnes uebergrosses Element die
|
|
47
|
+
* Queue nicht dauerhaft blockiert (warten kann fuer so ein Element nie
|
|
48
|
+
* Platz schaffen). Sie ist ausdruecklich kein Dauerzustand: ist das
|
|
49
|
+
* Kontingent erschoepft, scheitert `push()` mit einem benannten Fehler,
|
|
50
|
+
* statt das Budget weiter stillschweigend zu ueberschreiten. `0` lehnt
|
|
51
|
+
* schon das erste uebergrosse Element ab.
|
|
52
|
+
*/
|
|
53
|
+
maxOverBudgetAdmissions?: number;
|
|
54
|
+
/**
|
|
55
|
+
* Wird bei JEDER Ausnahme gerufen — der vorgesehene Weg, die
|
|
56
|
+
* Budgetverletzung in Log, Diagnostics oder Skip-Ledger sichtbar zu
|
|
57
|
+
* machen. Wirft der Rueckruf, scheitert der `push()`.
|
|
58
|
+
*/
|
|
59
|
+
onOverBudgetAdmission?: (admission: OverBudgetAdmissionV1) => void;
|
|
60
|
+
}
|
|
61
|
+
/** Snapshot of one queue for status output and tests. */
|
|
62
|
+
export interface QueueStatsV1 {
|
|
63
|
+
depth: number;
|
|
64
|
+
capacity: number;
|
|
65
|
+
bytes: number;
|
|
66
|
+
maxBytes: number | null;
|
|
67
|
+
/** High-water marks since construction — prove boundedness in tests. */
|
|
68
|
+
maxDepthObserved: number;
|
|
69
|
+
maxBytesObserved: number;
|
|
70
|
+
/**
|
|
71
|
+
* Anzahl der Elemente, die trotz Budgetueberschreitung aufgenommen wurden,
|
|
72
|
+
* weil sie allein groesser als das gesamte Budget sind. `> 0` ist die
|
|
73
|
+
* Erklaerung dafuer, dass `maxBytesObserved > maxBytes` sein kann — ohne
|
|
74
|
+
* diesen Zaehler waere die Budgetverletzung unsichtbar (E4).
|
|
75
|
+
*/
|
|
76
|
+
overBudgetAdmissions: number;
|
|
77
|
+
/** Wie viele solcher Ausnahmen diese Queue insgesamt zulaesst. */
|
|
78
|
+
overBudgetAllowance: number;
|
|
79
|
+
/** Groesstes je aufgenommenes Einzelelement in Bytes. */
|
|
80
|
+
maxItemBytesObserved: number;
|
|
81
|
+
}
|
|
82
|
+
export type QueuePullResult<T> = {
|
|
83
|
+
done: true;
|
|
84
|
+
} | {
|
|
85
|
+
done: false;
|
|
86
|
+
value: T;
|
|
87
|
+
};
|
|
88
|
+
export declare class BoundedQueue<T> {
|
|
89
|
+
private readonly slots;
|
|
90
|
+
private readonly capacity;
|
|
91
|
+
private readonly maxBytes;
|
|
92
|
+
private readonly sizeOf;
|
|
93
|
+
private readonly overBudgetAllowance;
|
|
94
|
+
private readonly onOverBudgetAdmission;
|
|
95
|
+
private overBudgetAdmissions;
|
|
96
|
+
private maxItemBytesObserved;
|
|
97
|
+
private bytes;
|
|
98
|
+
private closed;
|
|
99
|
+
private failure;
|
|
100
|
+
private wakers;
|
|
101
|
+
private maxDepthObserved;
|
|
102
|
+
private maxBytesObserved;
|
|
103
|
+
constructor(options: BoundedQueueOptions<T>);
|
|
104
|
+
/**
|
|
105
|
+
* Rejects a caller-supplied byte size (a per-item `sizeOf(item)` result or
|
|
106
|
+
* the constructor's `maxBytes`) that is not finite and >= 0.
|
|
107
|
+
*
|
|
108
|
+
* This is deliberately a hard rejection, never a clamp to some
|
|
109
|
+
* conservative substitute: the whole point of `maxBytes`/`sizeOf` is that
|
|
110
|
+
* the QUEUE trusts the caller's cost estimate to keep the pipeline's
|
|
111
|
+
* memory bound truthful (architecture.md §13.2, "a hard limit, not a
|
|
112
|
+
* hope"). A `NaN`/`Infinity`/negative size means that trust is already
|
|
113
|
+
* broken — the caller's cost model produced a value with no relationship
|
|
114
|
+
* to real memory use — so any number the queue might invent instead
|
|
115
|
+
* (0, the budget itself, the last good size, …) would just re-establish
|
|
116
|
+
* that same false trust one layer down and let corrupted accounting
|
|
117
|
+
* accumulate exactly as the adversarial repro showed (`Infinity` on an
|
|
118
|
+
* empty queue turns `bytes` into `NaN` on the very next `pull()`, which
|
|
119
|
+
* then compares `false` against every future budget check forever). A
|
|
120
|
+
* caller that cannot produce a valid size has a bug that must surface as
|
|
121
|
+
* a failed job (spec.md §25.2: write-side operations fail closed when
|
|
122
|
+
* consistency cannot be guaranteed), not as a silently-disabled budget.
|
|
123
|
+
*/
|
|
124
|
+
private static validateSize;
|
|
125
|
+
/**
|
|
126
|
+
* Defense-in-depth invariant on the running `bytes` counter itself, on
|
|
127
|
+
* top of `validateSize()` at every entry point: `bytes` is a sum of
|
|
128
|
+
* values that already passed `validateSize()`, so it should never go
|
|
129
|
+
* non-finite or negative — but "should never" is not a guarantee, and a
|
|
130
|
+
* counter that silently drifts into `NaN`/negative is exactly as
|
|
131
|
+
* dangerous as an unchecked input (it disables the same budget check the
|
|
132
|
+
* same way). Called after every mutation of `bytes`; poisons the queue
|
|
133
|
+
* and throws rather than letting `push()`/`pull()` keep computing on a
|
|
134
|
+
* counter that can no longer prove the budget (spec.md §25.2, fail
|
|
135
|
+
* closed).
|
|
136
|
+
*/
|
|
137
|
+
private assertByteCounterIntact;
|
|
138
|
+
private notify;
|
|
139
|
+
private waitForChange;
|
|
140
|
+
/**
|
|
141
|
+
* Entscheidet ueber eine Aufnahme, ohne etwas zu veraendern.
|
|
142
|
+
*
|
|
143
|
+
* - `"admit"` — Element passt in beide Grenzen.
|
|
144
|
+
* - `"wait"` — kein Platz; Warten kann Platz schaffen (Rueckstau).
|
|
145
|
+
* - `"over-budget"` — Element ist ALLEIN groesser als das ganze Budget:
|
|
146
|
+
* Warten kann fuer dieses Element nie Platz schaffen, die Ausnahme greift.
|
|
147
|
+
*
|
|
148
|
+
* WP-116: Frueher haengte der Byte-Zweig an `slots.length > 0`. Auf einer
|
|
149
|
+
* leeren Queue ist `bytes === 0`, also fiel „Element passt nicht ins
|
|
150
|
+
* Budget" mit „Element ist allein zu gross" nur zufaellig zusammen — und
|
|
151
|
+
* bei `capacity: 1` ist die Queue vor JEDEM Push leer, womit die Ausnahme
|
|
152
|
+
* zum Dauerzustand wurde und die Byte-Grenze mit voellig gueltigen Werten
|
|
153
|
+
* abgeschaltet war (belegt: 20 x 50 MB durch ein 100-Byte-Budget). Die
|
|
154
|
+
* Bedingung steht jetzt dort, wo sie hingehoert: am ELEMENT
|
|
155
|
+
* (`size > maxBytes`), nicht am Fuellstand.
|
|
156
|
+
*/
|
|
157
|
+
private admissionVerdict;
|
|
158
|
+
/**
|
|
159
|
+
* Invariante der leeren Queue: ohne gepufferte Elemente MUSS `bytes` 0
|
|
160
|
+
* sein. Frueher hat der Byte-Zweig diese Annahme stillschweigend benutzt
|
|
161
|
+
* (`slots.length > 0` als Ersatz fuer `bytes === 0`); jetzt wird sie
|
|
162
|
+
* geprueft, damit eine kuenftige Abweichung als Fehler auffaellt und nicht
|
|
163
|
+
* als lautlos abgeschaltete Grenze.
|
|
164
|
+
*/
|
|
165
|
+
private assertEmptyStateCounter;
|
|
166
|
+
/**
|
|
167
|
+
* Bucht eine Ausnahme. Ist das Kontingent erschoepft, scheitert der Push
|
|
168
|
+
* mit benanntem, nicht wiederholbarem Fehler — fail closed (spec.md
|
|
169
|
+
* §25.2) statt einer Budgetverletzung, die niemand sieht. Absichtlich
|
|
170
|
+
* KEIN Warten: fuer ein Element groesser als das Budget kann Warten nie
|
|
171
|
+
* Platz schaffen, ein Warten waere ein Haenger statt eines Rueckstaus.
|
|
172
|
+
*/
|
|
173
|
+
private admitOverBudget;
|
|
174
|
+
/**
|
|
175
|
+
* Enqueues `item`, waiting while the queue is full (count or byte bound) —
|
|
176
|
+
* this wait IS the backpressure that pauses the previous stage. Rejects
|
|
177
|
+
* with `JOB_CANCELLED` when `options.signal` aborts, and with the failure
|
|
178
|
+
* error after `fail()`.
|
|
179
|
+
*/
|
|
180
|
+
push(item: T, options?: PortCallOptions): Promise<void>;
|
|
181
|
+
/**
|
|
182
|
+
* Dequeues the oldest item, waiting while the queue is empty. Returns
|
|
183
|
+
* `{ done: true }` once the queue is closed AND drained. Rejects with
|
|
184
|
+
* `JOB_CANCELLED` when `options.signal` aborts, and with the failure
|
|
185
|
+
* error after `fail()` (buffered items are NOT delivered after a failure —
|
|
186
|
+
* the pipeline is broken and must not continue on stale work).
|
|
187
|
+
*/
|
|
188
|
+
pull(options?: PortCallOptions): Promise<QueuePullResult<T>>;
|
|
189
|
+
/** Non-blocking pull: `null` when nothing is buffered right now. */
|
|
190
|
+
tryPull(): QueuePullResult<T> | null;
|
|
191
|
+
/** No more pushes; pulls drain the remaining items, then report done. */
|
|
192
|
+
close(): void;
|
|
193
|
+
/**
|
|
194
|
+
* Poisons the queue: every pending and future `push`/`pull` rejects with
|
|
195
|
+
* `error`. Used to unblock sibling pipeline workers when one stage fails.
|
|
196
|
+
*/
|
|
197
|
+
fail(error: unknown): void;
|
|
198
|
+
stats(): QueueStatsV1;
|
|
199
|
+
}
|
|
200
|
+
//# sourceMappingURL=bounded-queue.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"bounded-queue.d.ts","sourceRoot":"","sources":["../src/bounded-queue.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AACH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAY7D;;;;GAIG;AACH,MAAM,WAAW,qBAAqB;IACpC,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAC;IAClB,4DAA4D;IAC5D,QAAQ,EAAE,MAAM,CAAC;IACjB,uDAAuD;IACvD,OAAO,EAAE,MAAM,CAAC;IAChB,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,mBAAmB,CAAC,CAAC;IACpC,qEAAqE;IACrE,QAAQ,EAAE,MAAM,CAAC;IACjB,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,4EAA4E;IAC5E,MAAM,CAAC,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,MAAM,CAAC;IAC7B;;;;;;;;;;OAUG;IACH,uBAAuB,CAAC,EAAE,MAAM,CAAC;IACjC;;;;OAIG;IACH,qBAAqB,CAAC,EAAE,CAAC,SAAS,EAAE,qBAAqB,KAAK,IAAI,CAAC;CACpE;AAED,yDAAyD;AACzD,MAAM,WAAW,YAAY;IAC3B,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,wEAAwE;IACxE,gBAAgB,EAAE,MAAM,CAAC;IACzB,gBAAgB,EAAE,MAAM,CAAC;IACzB;;;;;OAKG;IACH,oBAAoB,EAAE,MAAM,CAAC;IAC7B,kEAAkE;IAClE,mBAAmB,EAAE,MAAM,CAAC;IAC5B,yDAAyD;IACzD,oBAAoB,EAAE,MAAM,CAAC;CAC9B;AAED,MAAM,MAAM,eAAe,CAAC,CAAC,IAAI;IAAE,IAAI,EAAE,IAAI,CAAA;CAAE,GAAG;IAAE,IAAI,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE,CAAC,CAAA;CAAE,CAAC;AAO5E,qBAAa,YAAY,CAAC,CAAC;IACzB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAiB;IACvC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAgB;IACzC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA+B;IACtD,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,OAAO,CAAC,QAAQ,CAAC,qBAAqB,CAAsD;IAC5F,OAAO,CAAC,oBAAoB,CAAK;IACjC,OAAO,CAAC,oBAAoB,CAAK;IACjC,OAAO,CAAC,KAAK,CAAK;IAClB,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,OAAO,CAAmC;IAClD,OAAO,CAAC,MAAM,CAAyB;IACvC,OAAO,CAAC,gBAAgB,CAAK;IAC7B,OAAO,CAAC,gBAAgB,CAAK;gBAEV,OAAO,EAAE,mBAAmB,CAAC,CAAC,CAAC;IAwClD;;;;;;;;;;;;;;;;;;;OAmBG;IACH,OAAO,CAAC,MAAM,CAAC,YAAY;IAY3B;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,uBAAuB;IAiB/B,OAAO,CAAC,MAAM;YAQA,aAAa;IAc3B;;;;;;;;;;;;;;;;OAgBG;IACH,OAAO,CAAC,gBAAgB;IAuBxB;;;;;;OAMG;IACH,OAAO,CAAC,uBAAuB;IAiB/B;;;;;;OAMG;IACH,OAAO,CAAC,eAAe;IAgCvB;;;;;OAKG;IACU,IAAI,CAAC,IAAI,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAmCpE;;;;;;OAMG;IACU,IAAI,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;IAoBzE,oEAAoE;IAC7D,OAAO,IAAI,eAAe,CAAC,CAAC,CAAC,GAAG,IAAI;IAiB3C,yEAAyE;IAClE,KAAK,IAAI,IAAI;IAKpB;;;OAGG;IACI,IAAI,CAAC,KAAK,EAAE,OAAO,GAAG,IAAI;IAQ1B,KAAK,IAAI,YAAY;CAa7B"}
|