@untangle-code/jobs 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/LICENSE +661 -0
  2. package/NOTICE +90 -0
  3. package/README.md +138 -2
  4. package/THIRD_PARTY_LICENSES.md +24 -0
  5. package/dist/abort.d.ts +14 -0
  6. package/dist/abort.d.ts.map +1 -0
  7. package/dist/abort.js +35 -0
  8. package/dist/abort.js.map +1 -0
  9. package/dist/bounded-queue.d.ts +200 -0
  10. package/dist/bounded-queue.d.ts.map +1 -0
  11. package/dist/bounded-queue.js +338 -0
  12. package/dist/bounded-queue.js.map +1 -0
  13. package/dist/checkpoint-store.d.ts +30 -0
  14. package/dist/checkpoint-store.d.ts.map +1 -0
  15. package/dist/checkpoint-store.js +131 -0
  16. package/dist/checkpoint-store.js.map +1 -0
  17. package/dist/events.d.ts +29 -0
  18. package/dist/events.d.ts.map +1 -0
  19. package/dist/events.js +66 -0
  20. package/dist/events.js.map +1 -0
  21. package/dist/index.d.ts +22 -0
  22. package/dist/index.d.ts.map +1 -0
  23. package/dist/index.js +30 -0
  24. package/dist/index.js.map +1 -0
  25. package/dist/resume.d.ts +45 -0
  26. package/dist/resume.d.ts.map +1 -0
  27. package/dist/resume.js +60 -0
  28. package/dist/resume.js.map +1 -0
  29. package/dist/runner.d.ts +148 -0
  30. package/dist/runner.d.ts.map +1 -0
  31. package/dist/runner.js +686 -0
  32. package/dist/runner.js.map +1 -0
  33. package/dist/stages.d.ts +146 -0
  34. package/dist/stages.d.ts.map +1 -0
  35. package/dist/stages.js +2 -0
  36. package/dist/stages.js.map +1 -0
  37. package/dist/state-machine.d.ts +52 -0
  38. package/dist/state-machine.d.ts.map +1 -0
  39. package/dist/state-machine.js +114 -0
  40. package/dist/state-machine.js.map +1 -0
  41. package/package.json +61 -3
  42. package/src/abort.ts +37 -0
  43. package/src/bounded-queue.ts +462 -0
  44. package/src/checkpoint-store.ts +171 -0
  45. package/src/events.ts +71 -0
  46. package/src/index.ts +37 -0
  47. package/src/resume.ts +106 -0
  48. package/src/runner.ts +916 -0
  49. package/src/stages.ts +167 -0
  50. package/src/state-machine.ts +132 -0
package/NOTICE ADDED
@@ -0,0 +1,90 @@
1
+ # NOTICE
2
+
3
+ Copyright (C) 2026 Luke Bölling and contributors.
4
+
5
+ This attribution covers original Untangle work. Contributors retain copyright in
6
+ their contributions. Third-party packages, grammars and other materials retain
7
+ their own copyright holders, licenses and notices; this line does not replace them.
8
+
9
+ ## License — binding decision
10
+
11
+ Untangle is licensed under the **GNU Affero General Public License, Version 3
12
+ (AGPL-3.0-only)**. See [`LICENSE`](./LICENSE).
13
+
14
+ This decision is **final, not provisional**, and was explicitly instructed by
15
+ the project owner. No contributor, agent or tool may change it without a new
16
+ explicit instruction.
17
+
18
+ The repository-wide license covers Untangle's first-party source code, scripts,
19
+ tests, fixtures, configuration, documentation, website and brand artwork, including
20
+ the logo and 3D yarn illustrations. Every workspace `package.json` declares
21
+ `"license": "AGPL-3.0-only"`. There is no separately licensed enterprise component
22
+ or separate permissive license for documentation or brand assets.
23
+
24
+ Third-party code, embedded grammars, dependency assets and their notices retain
25
+ their original licenses. Their inclusion does not relicense those works. The
26
+ repository license does not grant trademark rights or change the license of a
27
+ repository analyzed with Untangle. See the repository's `docs/LICENSING.md` for
28
+ scope and verification details.
29
+
30
+ This resolves the license question left open in `vision.md` §14 and completes
31
+ `product.md` §16, item 2.
32
+
33
+ ## Rationale
34
+
35
+ - **Network copyleft fits the intended product model.** AGPL §13 sets source
36
+ availability requirements for modified versions used through a computer network.
37
+ The current local CLI/stdio-MCP entry and future shared-server modes share the
38
+ same license.
39
+ - **Local-first analysis has a clear boundary.** Running Untangle against a codebase
40
+ does not replace that codebase's own license with Untangle's license.
41
+ - **First-party components share one license.** Implemented components and future
42
+ official plugins use this repository's licensing policy. Planned integrations do
43
+ not establish implemented or published products.
44
+
45
+ ## Deliberately accepted consequence
46
+
47
+ The project deliberately chooses AGPL rather than a permissive embedding license.
48
+ Redistribution, modifications and combined works must be assessed against the
49
+ actual terms in `LICENSE`, including applicable Corresponding Source requirements.
50
+ No permissive exception, commercial dual license or special contract-only package
51
+ license is granted or promised by this notice.
52
+
53
+ ## Dependencies
54
+
55
+ Third-party runtime dependencies are checked against the allowlist in
56
+ `scripts/check-licenses.mjs`. Their licenses and copyright notices remain unchanged
57
+ and traceable through package `THIRD_PARTY_LICENSES.md` files, grammar provenance,
58
+ the lockfile and release SBOM. The allowlist includes MIT / Apache-2.0 / BSD / ISC
59
+ and other explicitly listed licenses; it is not a license grant for dependencies.
60
+
61
+ The allowlist in `scripts/check-licenses.mjs` applies to **dependencies**, not
62
+ workspace packages. Untangle's own packages carry AGPL-3.0-only and are excluded
63
+ from that check.
64
+
65
+ ## Clean room
66
+
67
+ Untangle is an independent implementation with its own data models, APIs, tool
68
+ names, tests and sources. Feature ideas may be derived from publicly observable
69
+ behavior and generally known techniques. Third-party source code is not copied,
70
+ including from AGPL projects even where the license would permit it.
71
+
72
+ ## Contribution terms
73
+
74
+ Pull requests are welcome under the repository's existing AGPL-3.0-only license.
75
+ No separate CLA, copyright assignment or DCO sign-off is required. Contributors
76
+ must have the right to submit their work under those terms. See CONTRIBUTING.md.
77
+
78
+ ## Release decisions
79
+
80
+ - [x] Copyright attribution: Luke Bölling, with contributors retaining their rights.
81
+ - [x] Contributions use AGPL-3.0-only without an additional CLA or DCO gate.
82
+
83
+ ## Open legal items before public beta
84
+
85
+ - [ ] Trademark search for the product name `Untangle`.
86
+ - [ ] Registration and legal review of the npm scope `@untangle-code`.
87
+ - [ ] Verify source availability and notices for each released network daemon/server
88
+ mode (WP-59). The current stdio MCP `untangle_about` already reports the license,
89
+ source repository and build commit when supplied; metadata alone does not prove
90
+ that a concrete deployed version's source is available.
package/README.md CHANGED
@@ -1,3 +1,139 @@
1
- # Temporary Holding Version
1
+ # @untangle-code/jobs
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ The Untangle indexer's job core (WP-13): state machine, typed job events, checkpoints, resume
4
+ planning, generation commit, bounded queues with backpressure and cancellation through `AbortSignal`.
5
+
6
+ Normative sources: `spec.md` §10.5, §11, §12.2, §22; `architecture.md` §2.2, §6.14, §13.1–§13.4,
7
+ §21.3–§21.4, §27.5.
8
+
9
+ <a id="inhalt"></a>
10
+
11
+ ## Contents
12
+
13
+ - **`JobStateMachine`** (`state-machine.ts`): exactly the state machine from spec.md §11.1,
14
+ including `failed`, `cancel_requested`, `cancelled`. Illegal transitions are rejected with typed
15
+ errors, never silently ignored.
16
+ - **`AsyncEventChannel`** (`events.ts`): exposes the typed job events from architecture.md §2.2
17
+ (types in `@untangle-code/domain`) as `AsyncIterable<JobEvent>` (spec.md §22).
18
+ - **`BoundedQueue`** (`bounded-queue.ts`): bounded queues from architecture.md §13.2. A full queue
19
+ blocks `push()`, pausing the previous stage. Limits apply to count AND bytes; high-water marks
20
+ are visible through `stats()`.
21
+ - **`FileCheckpointStore`** (`checkpoint-store.ts`): `CheckpointStorePort` using `FileSystemPort`,
22
+ atomic through temp+rename, with the exact `IndexCheckpointV1` shape (§13.3).
23
+ - **`planResume`** (`resume.ts`): reconstructs remaining work from the last acknowledged checkpoint
24
+ (spec.md §11.2).
25
+ - **Stage interfaces** (`stages.ts`): narrow injectable contracts for discovery, analysis,
26
+ embedding, storage, graphs, context and generation commit. This package builds pipeline
27
+ MECHANICS, not domain stages.
28
+ - **`IndexJobRunner`** (`runner.ts`): orchestrates events, queues, checkpoints only after
29
+ acknowledged upserts (spec.md §10.5), cancellation only at safe batch boundaries
30
+ (architecture.md §21.3), and `completed` only after complete generation commit (§13.4, enforced
31
+ by domain's `assertGenerationComplete`). On success its terminal observer also publishes the
32
+ immutable `corpusRevisionFingerprint` of the fully scanned corpus manifest.
33
+
34
+ <a id="entscheidungen"></a>
35
+
36
+ ## Decisions
37
+
38
+ Deliberate choices left open by source documents:
39
+
40
+ 1. **State-machine transition relation.** spec.md §11.1 lists the pipeline chain and error states,
41
+ but not every edge. Only the immediately next stage is reachable along the chain; `failed` and
42
+ `cancel_requested` are reachable from every nonterminal pipeline stage; `cancel_requested`
43
+ leads only to `cancelled` (or `failed` if cancellation cleanup itself fails); `completed`,
44
+ `failed`, `cancelled` are terminal; self-transitions are not transitions.
45
+ 2. **Stage as a high-water mark.** The pipeline overlaps work, using backpressure rather than
46
+ phase barriers. The state machine therefore advances as a high-water mark (`advanceTo`) when
47
+ the first work unit reaches a phase. Every individual movement is a legal chain transition;
48
+ it never moves backward. A run without remaining work also crosses intermediate stages legally
49
+ with no work.
50
+ 3. **Error code for illegal transitions and untyped stage errors.** The closed catalog
51
+ (spec.md §25.1) has no "internal state error" code. Illegal transitions or untyped stage errors
52
+ leave the index incomplete, so they use `INDEX_INCOMPLETE` (`retryable: false`), never a guessed
53
+ infrastructure code (spec.md §25.2). A dedicated code would be clearer and is recorded as an
54
+ additive proposal for the domain owner.
55
+ 4. **Corrupt checkpoint is not missing checkpoint.** `load`/`loadLatest` return `null` only for
56
+ absence. A corrupt checkpoint throws `INDEX_INCOMPLETE`; silently ignoring it would discard
57
+ acknowledged work and silently restart, prohibited by spec.md §25.2 and E4.
58
+ 5. **Resume validity.** A checkpoint is accepted only when workspace, repository, ref AND
59
+ `effectiveProfileId` agree (incompatible profiles require a new generation, spec.md §11.2).
60
+ An acknowledged file is skipped only when its content hash also remains unchanged (hash is
61
+ the source of truth, spec.md §12.1). Changed or no-longer-planned files leave the adopted state
62
+ and require fresh acknowledgement; they are never counted twice.
63
+ 6. **Event-stream cancellation semantics.** `run()` does NOT throw for job failure or
64
+ cancellation; it ends the stream with exactly one terminal event
65
+ (`job_completed` | `job_failed` | `job_cancelled`), the consumable form of returning `cancelled`
66
+ (architecture.md §21.3). Inside the package, long-running methods abort with
67
+ `UntangleError("JOB_CANCELLED")` (spec.md §22). A consumer stopping iteration stops the job
68
+ through internal abort; without external abort, this counts as failure rather than user cancellation.
69
+ 7. **Memory budget.** One budget (`memoryBudgetBytes`, default 64 MiB) is split equally between
70
+ analysis and upsert queues and appears alongside queue depths in `status()`
71
+ (architecture.md §13.2). `queueCapacities`, `embedBatchSize` and `memoryBudgetBytes` are validated
72
+ when constructing the runner (`CONFIG_INVALID`), not halfway through a run.
73
+
74
+ The **only** exception is an item that **alone** exceeds the entire byte budget. Rejecting it
75
+ would silently drop data; waiting would hang rather than apply backpressure because no pull
76
+ could ever create enough room. The exception is narrow:
77
+ - It applies to the ITEM (`size > maxBytes`), not queue occupancy;
78
+ - It applies only to an empty queue so the violation never accumulates with other items;
79
+ - It has a quota (`maxOverBudgetAdmissions`, default 1); once exhausted, `push()` fails with
80
+ `INDEX_INCOMPLETE` rather than permanently exceeding the budget;
81
+ - Every admission is counted (`stats().overBudgetAdmissions`), reported
82
+ (`onOverBudgetAdmission`, logged by the runner as `queue_over_budget_admission`) and reflected
83
+ in high-water marks. Budget violations are never silent (E4).
84
+
85
+ Before WP-116, the byte branch depended on `slots.length > 0`. With `capacity: 1`, the queue is
86
+ empty before every push, so the exception applied to EVERY push: 20 × 50 MB crossed a 100-byte
87
+ budget without a single blocked push, despite fully valid configuration.
88
+ 8. **Checkpoint batch equals embedding batch.** `embedBatchSize` defines the configured file
89
+ batch in spec.md §11.2: exactly one checkpoint is written after each acknowledged batch
90
+ upsert, never before (spec.md §10.5).
91
+ 9. **Commit order.** Before generation commit, the runner prechecks through
92
+ `validateGenerationCompleteness` with assumed metadata status; AFTER commit,
93
+ `assertGenerationComplete` also enforces the actually reached final metadata state. An
94
+ incomplete generation is never submitted for commit.
95
+ 10. **Optional injected locks.** When optional `lockProvider` is set, the job holds
96
+ `repo:<repoId>:<refId>:index` (architecture.md §21.1). Validated `lockLeaseMs` defaults to
97
+ 30 seconds; the runner renews without overlap no later than one third of its duration.
98
+ Renewal failure aborts the internal pipeline as `LOCK_HELD`. A final renewal proves ownership
99
+ again before the terminal observer; a runner detecting ownership loss does not invoke it.
100
+ Adapter-owned current pointers additionally need serialization per repository/ref, ownership
101
+ checks immediately before publication and generation-CAS replacement. Stale detection and the
102
+ owner-checked renewal operation itself belong to the provider.
103
+ 11. **Unbounded event channel.** Job events are small and finite per batch; bounded WORK queues,
104
+ rather than the event stream, limit pipeline memory pressure. Slow event consumption therefore
105
+ does not delay the job.
106
+ 12. **Corpus revision from the complete scan, not the changed subset.** Discovery returns the
107
+ complete mapping of file IDs to content hashes. Only after successful generation commit does
108
+ the runner hash this canonical manifest and pass its immutable `corpusRevisionFingerprint`
109
+ to `onGenerationFinished`. This also applies to incremental runs with zero changed files.
110
+ Failed or cancelled runs publish no corpus revision; reused generations therefore cannot merge
111
+ different corpus snapshots in downstream Search Journeys.
112
+
113
+ ## Tests
114
+
115
+ Co-located `src/**/*.test.ts`, using only `@untangle-code/testkit` (FakeClock, DeterministicRandom,
116
+ MemoryFileSystem, InMemoryLockProvider), with no network or Docker. Coverage includes the complete
117
+ transition relation (all 15×15 pairs), demonstrated backpressure (a blocked storage stage limits
118
+ analysis to queue capacity plus in-flight work), cancellation producing valid checkpoints and
119
+ `cancelled`, exact checkpoint resume without repeating acknowledged work, incomplete generations
120
+ never reaching `completed`, successful zero-change runs producing corpus revisions from complete
121
+ manifests, and fault injection from architecture.md §27.5 at points 1 (after embedding/before
122
+ upsert), 3 (after upsert/before checkpoint) and 5 (after checkpoint/before generation commit), with
123
+ post-restart consistency evidence.
124
+
125
+ <a id="lizenz"></a>
126
+
127
+ ## License
128
+
129
+ **AGPL-3.0-only.** This package includes the complete license in [`LICENSE`](./LICENSE), the
130
+ license-choice rationale in [`NOTICE`](./NOTICE), and licenses for all third-party production
131
+ dependencies in [`THIRD_PARTY_LICENSES.md`](./THIRD_PARTY_LICENSES.md). All three ship in the npm
132
+ tarball: AGPL-3.0-only §4 and §5 require recipients to receive the license; a README alone is not
133
+ a copy of the license.
134
+
135
+ Local use alone does not trigger an obligation. Copyleft obligations apply to redistribution and
136
+ providing a network service (AGPL §13).
137
+
138
+ Corresponding TypeScript source ships under `src/` in the same tarball; the complete repository
139
+ is available at <https://github.com/lboel/untangle>.
@@ -0,0 +1,24 @@
1
+ # Third-party licenses — @untangle-code/jobs
2
+
3
+ Generated by `pnpm third-party-licenses` from this package's installed
4
+ production dependency tree: transitive `dependencies`, across
5
+ workspace edges, with no `devDependencies` included. The generator reads
6
+ each resolved package's `package.json` in `node_modules`; nothing is maintained by hand
7
+ or estimated.
8
+
9
+ Do not edit manually: `pnpm third-party-licenses --check` fails on drift,
10
+ and `pnpm check-publishable` rejects packages without a current file.
11
+
12
+ This file exists because `docs/BUILD_PLAN.md`, risk R6, makes
13
+ per-package `THIRD_PARTY_LICENSES.md` a release gate for npm publication.
14
+
15
+ Untangle itself is **AGPL-3.0-only** (see `LICENSE` and `NOTICE` in
16
+ this package). The packages below are third-party works; their own
17
+ licenses and copyright notices continue to apply unchanged. Workspace
18
+ packages (`@untangle-code/*`) are not listed here; they belong to the same
19
+ work and use the same AGPL-3.0-only license.
20
+
21
+ ## Result: no third-party dependencies
22
+
23
+ This package has no production dependencies outside the workspace.
24
+ The list is empty because none exist, not because they were not checked.
@@ -0,0 +1,14 @@
1
+ /**
2
+ * Throws `UntangleError("JOB_CANCELLED", …)` when `signal` is already
3
+ * aborted. Called at every safe batch boundary (architecture.md §21.3).
4
+ */
5
+ export declare function throwIfAborted(signal: AbortSignal | undefined, operation: string): void;
6
+ /** True when `error` is an `UntangleError` with code `JOB_CANCELLED`. */
7
+ export declare function isJobCancelledError(error: unknown): boolean;
8
+ /**
9
+ * Resolves when `signal` aborts. The listener is removed as soon as
10
+ * `cleanup` aborts, so racing this against another promise does not leak
11
+ * one listener per wait.
12
+ */
13
+ export declare function abortedPromise(signal: AbortSignal, cleanup: AbortSignal): Promise<void>;
14
+ //# sourceMappingURL=abort.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"abort.d.ts","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAQA;;;GAGG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,WAAW,GAAG,SAAS,EAAE,SAAS,EAAE,MAAM,GAAG,IAAI,CAIvF;AAED,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,KAAK,EAAE,OAAO,GAAG,OAAO,CAE3D;AAED;;;;GAIG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,WAAW,EAAE,OAAO,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC,CAQvF"}
package/dist/abort.js ADDED
@@ -0,0 +1,35 @@
1
+ /**
2
+ * AbortSignal helpers shared by every long-running method in this package
3
+ * (spec.md §22: public APIs MUST support abort signals; cancellation
4
+ * surfaces as `UntangleError` with code `JOB_CANCELLED`).
5
+ */
6
+ import { isUntangleError } from "@untangle-code/domain";
7
+ import { jobCancelledError } from "@untangle-code/observability";
8
+ /**
9
+ * Throws `UntangleError("JOB_CANCELLED", …)` when `signal` is already
10
+ * aborted. Called at every safe batch boundary (architecture.md §21.3).
11
+ */
12
+ export function throwIfAborted(signal, operation) {
13
+ if (signal?.aborted) {
14
+ throw jobCancelledError(`"${operation}" was aborted`, { operation });
15
+ }
16
+ }
17
+ /** True when `error` is an `UntangleError` with code `JOB_CANCELLED`. */
18
+ export function isJobCancelledError(error) {
19
+ return isUntangleError(error) && error.code === "JOB_CANCELLED";
20
+ }
21
+ /**
22
+ * Resolves when `signal` aborts. The listener is removed as soon as
23
+ * `cleanup` aborts, so racing this against another promise does not leak
24
+ * one listener per wait.
25
+ */
26
+ export function abortedPromise(signal, cleanup) {
27
+ return new Promise((resolve) => {
28
+ if (signal.aborted) {
29
+ resolve();
30
+ return;
31
+ }
32
+ signal.addEventListener("abort", () => resolve(), { once: true, signal: cleanup });
33
+ });
34
+ }
35
+ //# sourceMappingURL=abort.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"abort.js","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AACxD,OAAO,EAAE,iBAAiB,EAAE,MAAM,8BAA8B,CAAC;AAEjE;;;GAGG;AACH,MAAM,UAAU,cAAc,CAAC,MAA+B,EAAE,SAAiB;IAC/E,IAAI,MAAM,EAAE,OAAO,EAAE,CAAC;QACpB,MAAM,iBAAiB,CAAC,IAAI,SAAS,eAAe,EAAE,EAAE,SAAS,EAAE,CAAC,CAAC;IACvE,CAAC;AACH,CAAC;AAED,yEAAyE;AACzE,MAAM,UAAU,mBAAmB,CAAC,KAAc;IAChD,OAAO,eAAe,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC,IAAI,KAAK,eAAe,CAAC;AAClE,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,cAAc,CAAC,MAAmB,EAAE,OAAoB;IACtE,OAAO,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE;QACnC,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;YACnB,OAAO,EAAE,CAAC;YACV,OAAO;QACT,CAAC;QACD,MAAM,CAAC,gBAAgB,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC,CAAC;IACrF,CAAC,CAAC,CAAC;AACL,CAAC"}
@@ -0,0 +1,200 @@
1
+ /**
2
+ * Bounded FIFO queue with backpressure (architecture.md §13.1–§13.2).
3
+ *
4
+ * A full queue makes `push()` WAIT — that is the mechanism by which a slow
5
+ * downstream stage pauses the previous stage instead of letting it
6
+ * accumulate unbounded ASTs/vectors in memory. Bounds are enforced on both
7
+ * item count (`capacity`) and, optionally, estimated bytes (`maxBytes` with
8
+ * a `sizeOf` cost function), so the pipeline's memory budget is a hard
9
+ * limit, not a hope. All observed values are exposed via `stats()` for the
10
+ * job status output (spec-driven visibility, architecture.md §13.2).
11
+ *
12
+ * Die EINZIGE Ausnahme vom Byte-Budget ist ein Element, das das ganze Budget
13
+ * allein uebersteigt: fuer so ein Element kann Warten nie Platz schaffen, es
14
+ * wuerde die Queue also nicht bremsen, sondern haengen lassen. Diese Ausnahme
15
+ * ist eng gefasst (leere Queue, kontingentiert, gezaehlt, gemeldet) — sie war
16
+ * frueher an den Fuellstand geknuepft und damit bei `capacity: 1` ein
17
+ * Dauerzustand, der die Byte-Grenze mit gueltigen Werten abschaltete (WP-116).
18
+ */
19
+ import type { PortCallOptions } from "@untangle-code/domain";
20
+ /**
21
+ * Eine Aufnahme, die das Byte-Budget verletzt hat, weil das Element ALLEIN
22
+ * groesser als das gesamte Budget ist. Wird gezaehlt und gemeldet — eine
23
+ * Budgetverletzung darf nie stillschweigend passieren (E4).
24
+ */
25
+ export interface OverBudgetAdmissionV1 {
26
+ /** Groesse des aufgenommenen Elements in Bytes (> `maxBytes`). */
27
+ itemBytes: number;
28
+ /** Das Budget, das dieses Element allein ueberschreitet. */
29
+ maxBytes: number;
30
+ /** Die wievielte Ausnahme dieser Queue (1-basiert). */
31
+ ordinal: number;
32
+ /** Wie viele solcher Ausnahmen diese Queue insgesamt zulaesst. */
33
+ allowance: number;
34
+ }
35
+ export interface BoundedQueueOptions<T> {
36
+ /** Maximum number of buffered items; must be a safe integer >= 1. */
37
+ capacity: number;
38
+ /** Optional byte budget across all buffered items. Requires `sizeOf`. */
39
+ maxBytes?: number;
40
+ /** Estimated size of one item in bytes; required when `maxBytes` is set. */
41
+ sizeOf?: (item: T) => number;
42
+ /**
43
+ * Wie viele Elemente, die das Byte-Budget ALLEIN uebersteigen, darf diese
44
+ * Queue insgesamt aufnehmen? Vorgabe: 1.
45
+ *
46
+ * Die Ausnahme existiert nur, damit EIN einzelnes uebergrosses Element die
47
+ * Queue nicht dauerhaft blockiert (warten kann fuer so ein Element nie
48
+ * Platz schaffen). Sie ist ausdruecklich kein Dauerzustand: ist das
49
+ * Kontingent erschoepft, scheitert `push()` mit einem benannten Fehler,
50
+ * statt das Budget weiter stillschweigend zu ueberschreiten. `0` lehnt
51
+ * schon das erste uebergrosse Element ab.
52
+ */
53
+ maxOverBudgetAdmissions?: number;
54
+ /**
55
+ * Wird bei JEDER Ausnahme gerufen — der vorgesehene Weg, die
56
+ * Budgetverletzung in Log, Diagnostics oder Skip-Ledger sichtbar zu
57
+ * machen. Wirft der Rueckruf, scheitert der `push()`.
58
+ */
59
+ onOverBudgetAdmission?: (admission: OverBudgetAdmissionV1) => void;
60
+ }
61
+ /** Snapshot of one queue for status output and tests. */
62
+ export interface QueueStatsV1 {
63
+ depth: number;
64
+ capacity: number;
65
+ bytes: number;
66
+ maxBytes: number | null;
67
+ /** High-water marks since construction — prove boundedness in tests. */
68
+ maxDepthObserved: number;
69
+ maxBytesObserved: number;
70
+ /**
71
+ * Anzahl der Elemente, die trotz Budgetueberschreitung aufgenommen wurden,
72
+ * weil sie allein groesser als das gesamte Budget sind. `> 0` ist die
73
+ * Erklaerung dafuer, dass `maxBytesObserved > maxBytes` sein kann — ohne
74
+ * diesen Zaehler waere die Budgetverletzung unsichtbar (E4).
75
+ */
76
+ overBudgetAdmissions: number;
77
+ /** Wie viele solcher Ausnahmen diese Queue insgesamt zulaesst. */
78
+ overBudgetAllowance: number;
79
+ /** Groesstes je aufgenommenes Einzelelement in Bytes. */
80
+ maxItemBytesObserved: number;
81
+ }
82
+ export type QueuePullResult<T> = {
83
+ done: true;
84
+ } | {
85
+ done: false;
86
+ value: T;
87
+ };
88
+ export declare class BoundedQueue<T> {
89
+ private readonly slots;
90
+ private readonly capacity;
91
+ private readonly maxBytes;
92
+ private readonly sizeOf;
93
+ private readonly overBudgetAllowance;
94
+ private readonly onOverBudgetAdmission;
95
+ private overBudgetAdmissions;
96
+ private maxItemBytesObserved;
97
+ private bytes;
98
+ private closed;
99
+ private failure;
100
+ private wakers;
101
+ private maxDepthObserved;
102
+ private maxBytesObserved;
103
+ constructor(options: BoundedQueueOptions<T>);
104
+ /**
105
+ * Rejects a caller-supplied byte size (a per-item `sizeOf(item)` result or
106
+ * the constructor's `maxBytes`) that is not finite and >= 0.
107
+ *
108
+ * This is deliberately a hard rejection, never a clamp to some
109
+ * conservative substitute: the whole point of `maxBytes`/`sizeOf` is that
110
+ * the QUEUE trusts the caller's cost estimate to keep the pipeline's
111
+ * memory bound truthful (architecture.md §13.2, "a hard limit, not a
112
+ * hope"). A `NaN`/`Infinity`/negative size means that trust is already
113
+ * broken — the caller's cost model produced a value with no relationship
114
+ * to real memory use — so any number the queue might invent instead
115
+ * (0, the budget itself, the last good size, …) would just re-establish
116
+ * that same false trust one layer down and let corrupted accounting
117
+ * accumulate exactly as the adversarial repro showed (`Infinity` on an
118
+ * empty queue turns `bytes` into `NaN` on the very next `pull()`, which
119
+ * then compares `false` against every future budget check forever). A
120
+ * caller that cannot produce a valid size has a bug that must surface as
121
+ * a failed job (spec.md §25.2: write-side operations fail closed when
122
+ * consistency cannot be guaranteed), not as a silently-disabled budget.
123
+ */
124
+ private static validateSize;
125
+ /**
126
+ * Defense-in-depth invariant on the running `bytes` counter itself, on
127
+ * top of `validateSize()` at every entry point: `bytes` is a sum of
128
+ * values that already passed `validateSize()`, so it should never go
129
+ * non-finite or negative — but "should never" is not a guarantee, and a
130
+ * counter that silently drifts into `NaN`/negative is exactly as
131
+ * dangerous as an unchecked input (it disables the same budget check the
132
+ * same way). Called after every mutation of `bytes`; poisons the queue
133
+ * and throws rather than letting `push()`/`pull()` keep computing on a
134
+ * counter that can no longer prove the budget (spec.md §25.2, fail
135
+ * closed).
136
+ */
137
+ private assertByteCounterIntact;
138
+ private notify;
139
+ private waitForChange;
140
+ /**
141
+ * Entscheidet ueber eine Aufnahme, ohne etwas zu veraendern.
142
+ *
143
+ * - `"admit"` — Element passt in beide Grenzen.
144
+ * - `"wait"` — kein Platz; Warten kann Platz schaffen (Rueckstau).
145
+ * - `"over-budget"` — Element ist ALLEIN groesser als das ganze Budget:
146
+ * Warten kann fuer dieses Element nie Platz schaffen, die Ausnahme greift.
147
+ *
148
+ * WP-116: Frueher haengte der Byte-Zweig an `slots.length > 0`. Auf einer
149
+ * leeren Queue ist `bytes === 0`, also fiel „Element passt nicht ins
150
+ * Budget" mit „Element ist allein zu gross" nur zufaellig zusammen — und
151
+ * bei `capacity: 1` ist die Queue vor JEDEM Push leer, womit die Ausnahme
152
+ * zum Dauerzustand wurde und die Byte-Grenze mit voellig gueltigen Werten
153
+ * abgeschaltet war (belegt: 20 x 50 MB durch ein 100-Byte-Budget). Die
154
+ * Bedingung steht jetzt dort, wo sie hingehoert: am ELEMENT
155
+ * (`size > maxBytes`), nicht am Fuellstand.
156
+ */
157
+ private admissionVerdict;
158
+ /**
159
+ * Invariante der leeren Queue: ohne gepufferte Elemente MUSS `bytes` 0
160
+ * sein. Frueher hat der Byte-Zweig diese Annahme stillschweigend benutzt
161
+ * (`slots.length > 0` als Ersatz fuer `bytes === 0`); jetzt wird sie
162
+ * geprueft, damit eine kuenftige Abweichung als Fehler auffaellt und nicht
163
+ * als lautlos abgeschaltete Grenze.
164
+ */
165
+ private assertEmptyStateCounter;
166
+ /**
167
+ * Bucht eine Ausnahme. Ist das Kontingent erschoepft, scheitert der Push
168
+ * mit benanntem, nicht wiederholbarem Fehler — fail closed (spec.md
169
+ * §25.2) statt einer Budgetverletzung, die niemand sieht. Absichtlich
170
+ * KEIN Warten: fuer ein Element groesser als das Budget kann Warten nie
171
+ * Platz schaffen, ein Warten waere ein Haenger statt eines Rueckstaus.
172
+ */
173
+ private admitOverBudget;
174
+ /**
175
+ * Enqueues `item`, waiting while the queue is full (count or byte bound) —
176
+ * this wait IS the backpressure that pauses the previous stage. Rejects
177
+ * with `JOB_CANCELLED` when `options.signal` aborts, and with the failure
178
+ * error after `fail()`.
179
+ */
180
+ push(item: T, options?: PortCallOptions): Promise<void>;
181
+ /**
182
+ * Dequeues the oldest item, waiting while the queue is empty. Returns
183
+ * `{ done: true }` once the queue is closed AND drained. Rejects with
184
+ * `JOB_CANCELLED` when `options.signal` aborts, and with the failure
185
+ * error after `fail()` (buffered items are NOT delivered after a failure —
186
+ * the pipeline is broken and must not continue on stale work).
187
+ */
188
+ pull(options?: PortCallOptions): Promise<QueuePullResult<T>>;
189
+ /** Non-blocking pull: `null` when nothing is buffered right now. */
190
+ tryPull(): QueuePullResult<T> | null;
191
+ /** No more pushes; pulls drain the remaining items, then report done. */
192
+ close(): void;
193
+ /**
194
+ * Poisons the queue: every pending and future `push`/`pull` rejects with
195
+ * `error`. Used to unblock sibling pipeline workers when one stage fails.
196
+ */
197
+ fail(error: unknown): void;
198
+ stats(): QueueStatsV1;
199
+ }
200
+ //# sourceMappingURL=bounded-queue.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"bounded-queue.d.ts","sourceRoot":"","sources":["../src/bounded-queue.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AACH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAY7D;;;;GAIG;AACH,MAAM,WAAW,qBAAqB;IACpC,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAC;IAClB,4DAA4D;IAC5D,QAAQ,EAAE,MAAM,CAAC;IACjB,uDAAuD;IACvD,OAAO,EAAE,MAAM,CAAC;IAChB,kEAAkE;IAClE,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,mBAAmB,CAAC,CAAC;IACpC,qEAAqE;IACrE,QAAQ,EAAE,MAAM,CAAC;IACjB,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,4EAA4E;IAC5E,MAAM,CAAC,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,MAAM,CAAC;IAC7B;;;;;;;;;;OAUG;IACH,uBAAuB,CAAC,EAAE,MAAM,CAAC;IACjC;;;;OAIG;IACH,qBAAqB,CAAC,EAAE,CAAC,SAAS,EAAE,qBAAqB,KAAK,IAAI,CAAC;CACpE;AAED,yDAAyD;AACzD,MAAM,WAAW,YAAY;IAC3B,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,wEAAwE;IACxE,gBAAgB,EAAE,MAAM,CAAC;IACzB,gBAAgB,EAAE,MAAM,CAAC;IACzB;;;;;OAKG;IACH,oBAAoB,EAAE,MAAM,CAAC;IAC7B,kEAAkE;IAClE,mBAAmB,EAAE,MAAM,CAAC;IAC5B,yDAAyD;IACzD,oBAAoB,EAAE,MAAM,CAAC;CAC9B;AAED,MAAM,MAAM,eAAe,CAAC,CAAC,IAAI;IAAE,IAAI,EAAE,IAAI,CAAA;CAAE,GAAG;IAAE,IAAI,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE,CAAC,CAAA;CAAE,CAAC;AAO5E,qBAAa,YAAY,CAAC,CAAC;IACzB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAiB;IACvC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAgB;IACzC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA+B;IACtD,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,OAAO,CAAC,QAAQ,CAAC,qBAAqB,CAAsD;IAC5F,OAAO,CAAC,oBAAoB,CAAK;IACjC,OAAO,CAAC,oBAAoB,CAAK;IACjC,OAAO,CAAC,KAAK,CAAK;IAClB,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,OAAO,CAAmC;IAClD,OAAO,CAAC,MAAM,CAAyB;IACvC,OAAO,CAAC,gBAAgB,CAAK;IAC7B,OAAO,CAAC,gBAAgB,CAAK;gBAEV,OAAO,EAAE,mBAAmB,CAAC,CAAC,CAAC;IAwClD;;;;;;;;;;;;;;;;;;;OAmBG;IACH,OAAO,CAAC,MAAM,CAAC,YAAY;IAY3B;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,uBAAuB;IAiB/B,OAAO,CAAC,MAAM;YAQA,aAAa;IAc3B;;;;;;;;;;;;;;;;OAgBG;IACH,OAAO,CAAC,gBAAgB;IAuBxB;;;;;;OAMG;IACH,OAAO,CAAC,uBAAuB;IAiB/B;;;;;;OAMG;IACH,OAAO,CAAC,eAAe;IAgCvB;;;;;OAKG;IACU,IAAI,CAAC,IAAI,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAmCpE;;;;;;OAMG;IACU,IAAI,CAAC,OAAO,CAAC,EAAE,eAAe,GAAG,OAAO,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;IAoBzE,oEAAoE;IAC7D,OAAO,IAAI,eAAe,CAAC,CAAC,CAAC,GAAG,IAAI;IAiB3C,yEAAyE;IAClE,KAAK,IAAI,IAAI;IAKpB;;;OAGG;IACI,IAAI,CAAC,KAAK,EAAE,OAAO,GAAG,IAAI;IAQ1B,KAAK,IAAI,YAAY;CAa7B"}