@untangle-code/chunker 0.0.0-stage → 0.1.0

package/NOTICE

@@ -0,0 +1,90 @@
+# NOTICE
+
+Copyright (C) 2026 Luke Bölling and contributors.
+
+This attribution covers original Untangle work. Contributors retain copyright in
+their contributions. Third-party packages, grammars and other materials retain
+their own copyright holders, licenses and notices; this line does not replace them.
+
+## License — binding decision
+
+Untangle is licensed under the **GNU Affero General Public License, Version 3
+(AGPL-3.0-only)**. See [`LICENSE`](./LICENSE).
+
+This decision is **final, not provisional**, and was explicitly instructed by
+the project owner. No contributor, agent or tool may change it without a new
+explicit instruction.
+
+The repository-wide license covers Untangle's first-party source code, scripts,
+tests, fixtures, configuration, documentation, website and brand artwork, including
+the logo and 3D yarn illustrations. Every workspace `package.json` declares
+`"license": "AGPL-3.0-only"`. There is no separately licensed enterprise component
+or separate permissive license for documentation or brand assets.
+
+Third-party code, embedded grammars, dependency assets and their notices retain
+their original licenses. Their inclusion does not relicense those works. The
+repository license does not grant trademark rights or change the license of a
+repository analyzed with Untangle. See the repository's `docs/LICENSING.md` for
+scope and verification details.
+
+This resolves the license question left open in `vision.md` §14 and completes
+`product.md` §16, item 2.
+
+## Rationale
+
+- **Network copyleft fits the intended product model.** AGPL §13 sets source
+  availability requirements for modified versions used through a computer network.
+  The current local CLI/stdio-MCP entry and future shared-server modes share the
+  same license.
+- **Local-first analysis has a clear boundary.** Running Untangle against a codebase
+  does not replace that codebase's own license with Untangle's license.
+- **First-party components share one license.** Implemented components and future
+  official plugins use this repository's licensing policy. Planned integrations do
+  not establish implemented or published products.
+
+## Deliberately accepted consequence
+
+The project deliberately chooses AGPL rather than a permissive embedding license.
+Redistribution, modifications and combined works must be assessed against the
+actual terms in `LICENSE`, including applicable Corresponding Source requirements.
+No permissive exception, commercial dual license or special contract-only package
+license is granted or promised by this notice.
+
+## Dependencies
+
+Third-party runtime dependencies are checked against the allowlist in
+`scripts/check-licenses.mjs`. Their licenses and copyright notices remain unchanged
+and traceable through package `THIRD_PARTY_LICENSES.md` files, grammar provenance,
+the lockfile and release SBOM. The allowlist includes MIT / Apache-2.0 / BSD / ISC
+and other explicitly listed licenses; it is not a license grant for dependencies.
+
+The allowlist in `scripts/check-licenses.mjs` applies to **dependencies**, not
+workspace packages. Untangle's own packages carry AGPL-3.0-only and are excluded
+from that check.
+
+## Clean room
+
+Untangle is an independent implementation with its own data models, APIs, tool
+names, tests and sources. Feature ideas may be derived from publicly observable
+behavior and generally known techniques. Third-party source code is not copied,
+including from AGPL projects even where the license would permit it.
+
+## Contribution terms
+
+Pull requests are welcome under the repository's existing AGPL-3.0-only license.
+No separate CLA, copyright assignment or DCO sign-off is required. Contributors
+must have the right to submit their work under those terms. See CONTRIBUTING.md.
+
+## Release decisions
+
+- [x] Copyright attribution: Luke Bölling, with contributors retaining their rights.
+- [x] Contributions use AGPL-3.0-only without an additional CLA or DCO gate.
+
+## Open legal items before public beta
+
+- [ ] Trademark search for the product name `Untangle`.
+- [ ] Registration and legal review of the npm scope `@untangle-code`.
+- [ ] Verify source availability and notices for each released network daemon/server
+  mode (WP-59). The current stdio MCP `untangle_about` already reports the license,
+  source repository and build commit when supplied; metadata alone does not prove
+  that a concrete deployed version's source is available.

package/README.md

@@ -1,3 +1,207 @@
-# Temporary Holding Version
+# @untangle-code/chunker
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Lossless token-budgeted AST chunking for Untangle (work following spec.md §8, architecture.md §6.6
+and §14). Internal private package, not published to npm.
+
+Normative sources: [`spec.md`](../../spec.md) §8 (takes precedence),
+[`architecture.md`](../../architecture.md) §6.6, §14, and [`vision.md`](../../vision.md) principles
+8.4 ("no silent data loss") and 8.5 ("stable identity over source position").
+
+Its only runtime dependency is `@untangle-code/domain` (`workspace:*`). Production imports neither
+`parser-registry` nor other Untangle packages; it consumes only language-neutral `LanguageAdapter`
+output (`ParseResult`, `AstRegion[]`, `ExtractedSymbol[]`). The seam's region contract is documented
+normatively on `AstRegion` in `packages/domain/src/analysis.ts`. The sole exception is
+`@untangle-code/parser-registry` as a **devDependency** for a real seam integration test
+(`src/integration.test.ts`) that parses actual TypeScript through the adapter rather than using
+hand-built region trees.
+
+<a id="was-dieses-package-tut"></a>
+
+## Responsibilities
+
+`chunkFile()` (the main entry point in `src/index.ts`) receives file identity, raw content and
+language-adapter output, returning `CodeChunkDraft[]` and a complete coverage ledger.
+Three strategies follow architecture.md §6.6:
+
+```text
+AstChunkStrategy      — spec.md §8.3 steps 1–8 (normal case, successfully parsed file)
+LineFallbackStrategy  — spec.md §8.3 step 9 (parse error / no usable regions)
+MinifiedStrategy      — spec.md §8.3 step 10 (minified content)
+```
+
+All use the same `BuildContext`/`emitChunk` core (`src/context.ts`), producing every `CodeChunkDraft`
+with identical `chunkId` derivation (`domain.buildChunkId`), prepared-text construction and token
+estimation. After each strategy, the orchestrator (`src/chunker.ts`) fills remaining byte gaps
+with `fallback_lines` and checks the lossless invariant again through `domain.validateCoverage`.
+This is defense in depth; a correct strategy must never leave a gap.
+
+<a id="modulübersicht"></a>
+
+## Modules
+
+| Module | Contents |
+|---|---|
+| `chunker.ts` | `chunkFile()`: strategy selection, gap filling, final coverage check |
+| `config.ts` | Defaults and `resolveChunkerConfig()` |
+| `context.ts` | `BuildContext`, sole `CodeChunkDraft` factory `emitChunk()`, `estimateTokensWithHeader()` |
+| `byte-text.ts` | Byte-exact slicing and line indexes: `LineIndex`, `sliceBytes`, `safeUtf8Boundary` |
+| `split.ts` | `flattenToAtoms()` (AST block splitting and statement grouping, steps 6–7), `splitByTokenBoundary()` (safe boundaries, step 8) |
+| `token-estimator.ts` | Conservative offline `estimateTokens()` (architecture.md §14.4) |
+| `minified-detection.ts` | Step-10 heuristic `detectMinifiedContent()` |
+| `prepared-text.ts` | `buildPreparedDenseText()`/`buildBm25Text()` (spec.md §8.6) |
+| `coverage.ts` | `chunkedEntry()`/`excludedEntry()`/`assertLosslessCoverage()`; the validator itself is `domain.validateCoverage()` |
+| `strategies/ast-chunk-strategy.ts` | Classes, symbols and fragments (spec.md §8.3 steps 1–8) |
+| `strategies/line-fallback-strategy.ts` | Overlapping line windows for parse failures |
+| `strategies/minified-strategy.ts` | Token/byte-safe splitting without AST structure |
+| `types.ts` | `ChunkerConfig`, `ChunkFileParams`, `ChunkFileResult` and related types |
+
+`src/index.ts` is the only public export (rule 10). Internal modules (`context.ts`, `split.ts`,
+`test-helpers.ts`, …) are not public API and are imported directly only by co-located tests.
+
+<a id="nutzung"></a>
+
+## Usage
+
+```ts
+import { chunkFile } from "@untangle-code/chunker";
+
+const result = chunkFile({
+  identity: { workspaceId, repoId, refId, fileId, filePath, languageId },
+  content: sourceText,
+  parseResult,          // from LanguageAdapter.parse()
+  regions,               // from LanguageAdapter.extractChunks(ast)
+  symbols,               // from LanguageAdapter.extractSymbols(ast)
+  config: {
+    preparedText: { documentPrefix, repoLabel, refName }, // Required; other settings have defaults
+  },
+});
+// result.drafts: CodeChunkDraft[] (including preparedDenseText/bm25Text/tokenEstimate)
+// result.coverage: CoverageEntry[] (Lossless evidence, spec.md §8.4)
+// result.strategy: "ast" | "line_fallback" | "minified"
+```
+
+<a id="entscheidungen"></a>
+
+## Decisions
+
+Choices left open by spec.md/architecture.md (rule 9); downstream packages should extend additively
+rather than redefine contracts locally:
+
+1. **`AstRegion.scopeId` is ignored; the chunker calculates `symbolId`.** During
+   `LanguageAdapter.extractChunks`, an adapter knows only `SourceFile` (fileId/repoId/refId), not
+   `workspaceId`, and therefore cannot derive a spec-conformant `symbol_id` (spec.md §5.5 requires
+   four components including `workspace_id`). The chunker receives `workspaceId` through
+   `ChunkFileIdentity` and uses `domain.buildSymbolId` with the region's `ExtractedSymbol`, linked
+   by `AstRegion.symbolFqn === ExtractedSymbol.qualifiedName`. Here `AstRegion.scopeId` is a
+   required adapter-populated field with no local semantic meaning.
+
+2. **Minified versus parse-error order.** spec.md §8.3 lists step 9 (parse failure) before step
+   10 (minified), without defining priority when either is used as fallback only if the other is
+   absent. `chunkFile()` checks minification FIRST from content, independently of parse success.
+   Minified files can parse successfully yet yield unusable structure, such as one huge top-level
+   statement; AST chunking would be pointless even without parser errors.
+
+3. **`class_overview` content is synthesized (`synthesized: true`) and proves NO range coverage
+   beyond the header.** To expose member names without method bodies,
+   `buildClassOverviewContent()` builds documentation comment, class signature and member
+   names/signatures, never bodies. Draft `range` retains the entire class for navigation;
+   `content_hash` describes synthesized text. Coverage declares ONLY the header (class start to
+   first member) as "chunked", because the overview actually carries that documentation/signature.
+   Remaining class bytes not covered by member chunks—inter-method comments, unmodeled constructs,
+   delimiters—are explicit: meaningful gaps become real class `symbol_fragment` chunks with
+   `symbolId`, `fragmentIndex`/`fragmentCount` and signature headers (spec.md §8.5); pure
+   delimiter/whitespace gaps are logged as "excluded" with reasons (spec.md §8.4). Synthesized
+   chunks must never hide gaps: the wave-0 defect (finding 2) is now protected by
+   `src/integration.test.ts`. If the member list itself exceeds the token budget, it is capped
+   with "... and N more members", never silently emitted over budget. A class without member
+   regions becomes a REAL byte chunk rather than synthesized content.
+
+4. **Small properties are grouped only across consecutive `property_group` child regions.**
+   Adapters mark each field/property individually with `chunkKind: "property_group"`. The chunker
+   greedily merges consecutive runs under the token budget (spec.md §8.3 step 5). Properties
+   interrupted by other members stay in separate groups; source is not reordered across the file.
+
+5. **`metricsScope`/`metricsScopeId` per chunk kind**, not prescribed normatively:
+   `class_overview`, `function`/`method`/`constructor`/`interface`/`type`/`enum`/`symbol_fragment`
+   use `"symbol"` (the single associated symbol ID); `module`/`file_preamble`/`comment`/
+   `configuration` without associated symbols use `"file"`; `property_group`/`fallback_lines`/
+   `minified_fragment` use `"chunk"` because no single symbol owns the chunk.
+
+6. **`chunk_ordinal`** uses one counter per `(scopeId, chunkKind)` pair (`nextOrdinal` in
+   `context.ts`). For `symbol_fragment`, it equals `fragmentIndex` because a symbol's fragments
+   are emitted consecutively in source order under one `scopeId`. This also explains stable
+   `chunk_id` under line-only shifts (spec.md §5.6): `symbolId` does not change (lines are not
+   identity inputs, §5.5), and fragment structure depends on symbol *content*, not absolute position.
+
+7. **Token estimate:** 3 characters/token (rather than the more realistic code range ~3.5–4.5),
+   plus small per-line overhead. Deliberate overestimation (architecture.md §14.4), documented in
+   `token-estimator.ts`; no network tokenizer, entirely offline and deterministic.
+
+8. **Minification detection** is a pure content heuristic using file size and mean/maximum line
+   length, all configurable. It does not rely on conventions such as `.min.js`, which generated
+   code does not always follow.
+
+9. **`LineFallbackStrategy` overlap** uses line windows with `linesPerChunk`/`overlapLines`, default
+   60/8, as explicitly requested. `MinifiedStrategy` also supports configurable `overlapBytes`,
+   default 0, because the spec does not explicitly require overlap there; it adds retrieval context.
+
+10. **Safe split boundaries** prefer a newline within a 120-byte lookback, never shrinking a
+    fragment by more than half to do so, then fall back to UTF-8-safe byte boundaries—never
+    continuation bytes. A dedicated multibyte-character test verifies this.
+
+11. **`module` regions are ALWAYS unpacked**, never emitted wholesale as single-file chunks.
+    Adapters provide one `module` root region (the `AstRegion` contract in domain's `analysis.ts`);
+    `dispatchRegion` handles `chunkKind` at every depth. Symbol chunks (spec.md §8.3 steps 3–5)
+    therefore exist even below the token budget. Module bytes between symbol regions, including
+    top-level statements and free comments, become real `module` chunks; pure whitespace gaps
+    are excluded with reasons. Only a `module` without children—a statement-only file—is emitted
+    as a complete content chunk or split into fragments.
+
+<a id="bekannte-grenzen--offene-punkte"></a>
+
+## Known limits / open issues
+
+See `openIssues` in the work unit's completion report.
+
+<a id="e4-ehrlichkeitspruefung-wp-165"></a>
+
+## E4 honesty check (WP-165)
+
+`src/honesty.test.ts` runs `assertHonestUnderDamage` from testkit against the real
+parser-registry → `chunkFile` seam. Six damage kinds—truncation, broken structure, removed required
+fields, size beyond every limit, concurrency and abort signals—must each produce either a visible
+finding or a demonstrably unchanged result.
+
+**Found and fixed: new `ChunkFileResult.degradations`.** When an adapter returned NO regions after
+`parseQuality: "complete"`, `chunkFile` silently used line fallback. Byte coverage stayed complete
+(`assertLosslessCoverage` passed), and quality stayed `"complete"`, yet no chunk had a symbol
+assignment. Callers inferred structural analysis that never happened: "completeness claimed,
+subtree discarded". Every deviation from the AST strategy now produces a `ChunkDegradation` with
+`ast_regions_missing`, `parse_failed` or `minified_content`. Regression tests are in `src/honesty.test.ts`.
+
+<a id="bekannte-grenzen-e4"></a>
+
+### Known limits (E4)
+
+- **`chunkFile()` accepts no `AbortSignal`.** It is synchronous and pure; spec.md §22 requires
+  cancellation support for public APIs. Damage kind `abort_signal` is therefore explicitly
+  waived for this reason in the honesty check and recorded in its skip ledger, never silently
+  omitted. Cancellation would be meaningful only between strategy steps and would change the
+  signature, remaining a separate task.
+
+<a id="lizenz"></a>
+
+## License
+
+
+**AGPL-3.0-only.** This package includes the complete license in [`LICENSE`](./LICENSE), the
+license-choice rationale in [`NOTICE`](./NOTICE), and licenses for all third-party production
+dependencies in [`THIRD_PARTY_LICENSES.md`](./THIRD_PARTY_LICENSES.md). All three ship in the npm
+tarball: AGPL-3.0-only §4 and §5 require recipients to receive the license; a README alone is not
+a copy of the license.
+
+Local use alone does not trigger an obligation. Copyleft obligations apply to redistribution and
+providing a network service (AGPL §13).
+
+Corresponding TypeScript source ships under `src/` in the same tarball; the complete repository
+is available at <https://github.com/lboel/untangle>.

package/THIRD_PARTY_LICENSES.md

@@ -0,0 +1,24 @@
+# Third-party licenses — @untangle-code/chunker
+
+Generated by `pnpm third-party-licenses` from this package's installed
+production dependency tree: transitive `dependencies`, across
+workspace edges, with no `devDependencies` included. The generator reads
+each resolved package's `package.json` in `node_modules`; nothing is maintained by hand
+or estimated.
+
+Do not edit manually: `pnpm third-party-licenses --check` fails on drift,
+and `pnpm check-publishable` rejects packages without a current file.
+
+This file exists because `docs/BUILD_PLAN.md`, risk R6, makes
+per-package `THIRD_PARTY_LICENSES.md` a release gate for npm publication.
+
+Untangle itself is **AGPL-3.0-only** (see `LICENSE` and `NOTICE` in
+this package). The packages below are third-party works; their own
+licenses and copyright notices continue to apply unchanged. Workspace
+packages (`@untangle-code/*`) are not listed here; they belong to the same
+work and use the same AGPL-3.0-only license.
+
+## Result: no third-party dependencies
+
+This package has no production dependencies outside the workspace.
+The list is empty because none exist, not because they were not checked.

package/dist/byte-text.d.ts

@@ -0,0 +1,39 @@
+/**
+ * Byte-accurate text slicing and line/byte conversion (spec.md §7.1's
+ * `SourceRange` convention: byte offsets are 0-based half-open, line numbers
+ * 1-based inclusive).
+ *
+ * The chunker MUST slice chunk content by byte offset, not by JS string
+ * (UTF-16) index, so that `CodeChunkPayloadV1.start_byte`/`end_byte` are
+ * exact and so that content is byte-identical to the source file on disk.
+ * Everything here operates on a `Buffer` built once per file.
+ */
+import type { SourceRange } from "@untangle-code/domain";
+/** Precomputed line-start byte offsets for one file's content. */
+export interface LineIndex {
+    buffer: Buffer;
+    /** Byte offset of the first byte of each line; `lineStarts[0] === 0`. */
+    lineStarts: number[];
+}
+/** Builds a `LineIndex` from full file content (decoded UTF-8 string). */
+export declare function buildLineIndex(content: string): LineIndex;
+/** 1-based line number containing the given 0-based byte offset. */
+export declare function lineNumberForByteOffset(index: LineIndex, byteOffset: number): number;
+/**
+ * Builds a `SourceRange` for a byte span, deriving line numbers from the
+ * index. `endByte` is exclusive; for a non-empty span the end line is the
+ * line containing the last included byte (`endByte - 1`).
+ */
+export declare function rangeForByteSpan(index: LineIndex, startByte: number, endByte: number): SourceRange;
+/** Decodes the UTF-8 bytes `[startByte, endByte)` of the file as a string. */
+export declare function sliceBytes(index: LineIndex, startByte: number, endByte: number): string;
+/** True if `byte` is a UTF-8 continuation byte (`10xxxxxx`). */
+export declare function isUtf8ContinuationByte(byte: number): boolean;
+/**
+ * Returns the largest offset `<= target` (and `>= 0`) that does not fall in
+ * the middle of a multi-byte UTF-8 sequence, i.e. `buffer[offset]` (if any)
+ * is not a continuation byte. Used to keep token-boundary splits byte-safe
+ * (architecture.md §14.4) even when the file mixes multi-byte characters.
+ */
+export declare function safeUtf8Boundary(buffer: Buffer, target: number): number;
+//# sourceMappingURL=byte-text.d.ts.map

package/dist/byte-text.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"byte-text.d.ts","sourceRoot":"","sources":["../src/byte-text.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AACH,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC;AAEzD,kEAAkE;AAClE,MAAM,WAAW,SAAS;IACxB,MAAM,EAAE,MAAM,CAAC;IACf,yEAAyE;IACzE,UAAU,EAAE,MAAM,EAAE,CAAC;CACtB;AAED,0EAA0E;AAC1E,wBAAgB,cAAc,CAAC,OAAO,EAAE,MAAM,GAAG,SAAS,CASzD;AAED,oEAAoE;AACpE,wBAAgB,uBAAuB,CAAC,KAAK,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,GAAG,MAAM,CAapF;AAED;;;;GAIG;AACH,wBAAgB,gBAAgB,CAC9B,KAAK,EAAE,SAAS,EAChB,SAAS,EAAE,MAAM,EACjB,OAAO,EAAE,MAAM,GACd,WAAW,CAIb;AAED,8EAA8E;AAC9E,wBAAgB,UAAU,CAAC,KAAK,EAAE,SAAS,EAAE,SAAS,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,MAAM,CAEvF;AAED,gEAAgE;AAChE,wBAAgB,sBAAsB,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAE5D;AAED;;;;;GAKG;AACH,wBAAgB,gBAAgB,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,MAAM,CAMvE"}

package/dist/byte-text.js

@@ -0,0 +1,59 @@
+/** Builds a `LineIndex` from full file content (decoded UTF-8 string). */
+export function buildLineIndex(content) {
+    const buffer = Buffer.from(content, "utf8");
+    const lineStarts = [0];
+    for (let i = 0; i < buffer.length; i += 1) {
+        if (buffer[i] === 0x0a) {
+            lineStarts.push(i + 1);
+        }
+    }
+    return { buffer, lineStarts };
+}
+/** 1-based line number containing the given 0-based byte offset. */
+export function lineNumberForByteOffset(index, byteOffset) {
+    const { lineStarts } = index;
+    let lo = 0;
+    let hi = lineStarts.length - 1;
+    while (lo < hi) {
+        const mid = Math.ceil((lo + hi) / 2);
+        if (lineStarts[mid] <= byteOffset) {
+            lo = mid;
+        }
+        else {
+            hi = mid - 1;
+        }
+    }
+    return lo + 1;
+}
+/**
+ * Builds a `SourceRange` for a byte span, deriving line numbers from the
+ * index. `endByte` is exclusive; for a non-empty span the end line is the
+ * line containing the last included byte (`endByte - 1`).
+ */
+export function rangeForByteSpan(index, startByte, endByte) {
+    const startLine = lineNumberForByteOffset(index, startByte);
+    const endLine = endByte > startByte ? lineNumberForByteOffset(index, endByte - 1) : startLine;
+    return { startLine, endLine, startByte, endByte };
+}
+/** Decodes the UTF-8 bytes `[startByte, endByte)` of the file as a string. */
+export function sliceBytes(index, startByte, endByte) {
+    return index.buffer.toString("utf8", startByte, endByte);
+}
+/** True if `byte` is a UTF-8 continuation byte (`10xxxxxx`). */
+export function isUtf8ContinuationByte(byte) {
+    return (byte & 0b1100_0000) === 0b1000_0000;
+}
+/**
+ * Returns the largest offset `<= target` (and `>= 0`) that does not fall in
+ * the middle of a multi-byte UTF-8 sequence, i.e. `buffer[offset]` (if any)
+ * is not a continuation byte. Used to keep token-boundary splits byte-safe
+ * (architecture.md §14.4) even when the file mixes multi-byte characters.
+ */
+export function safeUtf8Boundary(buffer, target) {
+    let offset = Math.max(0, Math.min(target, buffer.length));
+    while (offset > 0 && offset < buffer.length && isUtf8ContinuationByte(buffer[offset])) {
+        offset -= 1;
+    }
+    return offset;
+}
+//# sourceMappingURL=byte-text.js.map

package/dist/byte-text.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"byte-text.js","sourceRoot":"","sources":["../src/byte-text.ts"],"names":[],"mappings":"AAmBA,0EAA0E;AAC1E,MAAM,UAAU,cAAc,CAAC,OAAe;IAC5C,MAAM,MAAM,GAAG,MAAM,CAAC,IAAI,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;IAC5C,MAAM,UAAU,GAAa,CAAC,CAAC,CAAC,CAAC;IACjC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;QAC1C,IAAI,MAAM,CAAC,CAAC,CAAC,KAAK,IAAI,EAAE,CAAC;YACvB,UAAU,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QACzB,CAAC;IACH,CAAC;IACD,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,CAAC;AAChC,CAAC;AAED,oEAAoE;AACpE,MAAM,UAAU,uBAAuB,CAAC,KAAgB,EAAE,UAAkB;IAC1E,MAAM,EAAE,UAAU,EAAE,GAAG,KAAK,CAAC;IAC7B,IAAI,EAAE,GAAG,CAAC,CAAC;IACX,IAAI,EAAE,GAAG,UAAU,CAAC,MAAM,GAAG,CAAC,CAAC;IAC/B,OAAO,EAAE,GAAG,EAAE,EAAE,CAAC;QACf,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,EAAE,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;QACrC,IAAI,UAAU,CAAC,GAAG,CAAE,IAAI,UAAU,EAAE,CAAC;YACnC,EAAE,GAAG,GAAG,CAAC;QACX,CAAC;aAAM,CAAC;YACN,EAAE,GAAG,GAAG,GAAG,CAAC,CAAC;QACf,CAAC;IACH,CAAC;IACD,OAAO,EAAE,GAAG,CAAC,CAAC;AAChB,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,gBAAgB,CAC9B,KAAgB,EAChB,SAAiB,EACjB,OAAe;IAEf,MAAM,SAAS,GAAG,uBAAuB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;IAC5D,MAAM,OAAO,GAAG,OAAO,GAAG,SAAS,CAAC,CAAC,CAAC,uBAAuB,CAAC,KAAK,EAAE,OAAO,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IAC9F,OAAO,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAAE,OAAO,EAAE,CAAC;AACpD,CAAC;AAED,8EAA8E;AAC9E,MAAM,UAAU,UAAU,CAAC,KAAgB,EAAE,SAAiB,EAAE,OAAe;IAC7E,OAAO,KAAK,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,EAAE,SAAS,EAAE,OAAO,CAAC,CAAC;AAC3D,CAAC;AAED,gEAAgE;AAChE,MAAM,UAAU,sBAAsB,CAAC,IAAY;IACjD,OAAO,CAAC,IAAI,GAAG,WAAW,CAAC,KAAK,WAAW,CAAC;AAC9C,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,gBAAgB,CAAC,MAAc,EAAE,MAAc;IAC7D,IAAI,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IAC1D,OAAO,MAAM,GAAG,CAAC,IAAI,MAAM,GAAG,MAAM,CAAC,MAAM,IAAI,sBAAsB,CAAC,MAAM,CAAC,MAAM,CAAE,CAAC,EAAE,CAAC;QACvF,MAAM,IAAI,CAAC,CAAC;IACd,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC"}

package/dist/chunker.d.ts

@@ -0,0 +1,11 @@
+import type { ChunkFileParams, ChunkFileResult } from "./types.js";
+/**
+ * Chunks one file per spec.md §8.3: minified detection takes priority over
+ * AST chunking (a minified file may still parse "successfully" but produces
+ * no meaningful symbol structure), then a failed/empty parse falls back to
+ * line windows, and otherwise the AST strategy runs (README "Entscheidungen"
+ * documents this ordering as an Untangle decision — the spec numbers 9/10
+ * as trailing fallback steps without fixing their relative priority).
+ */
+export declare function chunkFile(params: ChunkFileParams): ChunkFileResult;
+//# sourceMappingURL=chunker.d.ts.map

package/dist/chunker.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"chunker.d.ts","sourceRoot":"","sources":["../src/chunker.ts"],"names":[],"mappings":"AAmBA,OAAO,KAAK,EAAoB,eAAe,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AA+BrF;;;;;;;GAOG;AACH,wBAAgB,SAAS,CAAC,MAAM,EAAE,eAAe,GAAG,eAAe,CAmFlE"}

package/dist/chunker.js

@@ -0,0 +1,112 @@
+/**
+ * Top-level chunking entry point (spec.md §8.3): picks a strategy, runs it,
+ * and — regardless of which strategy ran — guarantees the lossless coverage
+ * invariant (spec.md §8.4) before returning: any residual gap is filled with
+ * `fallback_lines` chunks, then the result is validated one last time and
+ * `ANALYSIS_INCOMPLETE` is thrown if that somehow still fails (defense in
+ * depth; a correct strategy should never leave a gap).
+ */
+import { classifyTextFormat, validateCoverage } from "@untangle-code/domain";
+import { resolveChunkerConfig } from "./config.js";
+import { contentOf, createBuildContext, emitChunk } from "./context.js";
+import { assertLosslessCoverage, assertTruthfulLedger, chunkedEntry } from "./coverage.js";
+import { detectMinifiedContent } from "./minified-detection.js";
+import { splitByTokenBoundary } from "./split.js";
+import { runAstChunkStrategy } from "./strategies/ast-chunk-strategy.js";
+import { runLineFallbackStrategy } from "./strategies/line-fallback-strategy.js";
+import { runMinifiedStrategy } from "./strategies/minified-strategy.js";
+const GAP_HEADER = { signature: "fallback_lines" };
+/** Fills any residual byte gap with `fallback_lines` chunks (defense in depth for every strategy). */
+function fillGaps(ctx, drafts, coverage) {
+    const gaps = validateCoverage(coverage, ctx.fileRange).gaps;
+    for (const gap of gaps) {
+        const range = { ...gap, startLine: -1, endLine: -1 };
+        for (const piece of splitByTokenBoundary(ctx, range, GAP_HEADER)) {
+            const draft = emitChunk(ctx, {
+                range: piece,
+                content: contentOf(ctx, piece),
+                scopeId: ctx.identity.fileId,
+                chunkKind: "fallback_lines",
+                header: {
+                    signature: "fallback_lines",
+                    metricsScope: "chunk",
+                    metricsScopeId: ctx.identity.fileId,
+                },
+            });
+            drafts.push(draft);
+            coverage.push(chunkedEntry(draft.range, draft.chunkId));
+        }
+    }
+}
+/**
+ * Chunks one file per spec.md §8.3: minified detection takes priority over
+ * AST chunking (a minified file may still parse "successfully" but produces
+ * no meaningful symbol structure), then a failed/empty parse falls back to
+ * line windows, and otherwise the AST strategy runs (README "Entscheidungen"
+ * documents this ordering as an Untangle decision — the spec numbers 9/10
+ * as trailing fallback steps without fixing their relative priority).
+ */
+export function chunkFile(params) {
+    const config = resolveChunkerConfig(params.config);
+    const ctx = createBuildContext(params.identity, params.content, config, params.parseResult.parseQuality);
+    const isMinified = params.isMinified ??
+        detectMinifiedContent(params.content, config.minifiedDetection, classifyTextFormat(params.identity.filePath));
+    const hasUsableAst = params.parseResult.parseQuality !== "failed" && params.regions.length > 0;
+    let drafts;
+    let coverage;
+    let strategy;
+    // E4 (WP-165): jede Abweichung von der AST-Strategie bekommt hier einen
+    // benannten Grund. Ohne ihn ist die Abstufung unsichtbar -- die Bytes sind
+    // lueckenlos erfasst, `parseQuality` kann `"complete"` lauten, und trotzdem
+    // hat kein Chunk eine Symbolzuordnung.
+    const degradations = [];
+    if (isMinified) {
+        const result = runMinifiedStrategy(ctx);
+        drafts = result.drafts;
+        coverage = result.coverage;
+        strategy = "minified";
+        degradations.push({
+            reason: "minified_content",
+            strategy: "minified",
+            detail: `content was classified as minified (${params.isMinified === undefined ? "auto-detected" : "declared by the caller"}), so it was cut into byte windows; no symbol association exists for this file`,
+        });
+    }
+    else if (!hasUsableAst) {
+        const result = runLineFallbackStrategy(ctx);
+        drafts = result.drafts;
+        coverage = result.coverage;
+        strategy = "line_fallback";
+        degradations.push(params.parseResult.parseQuality === "failed"
+            ? {
+                reason: "parse_failed",
+                strategy: "line_fallback",
+                detail: `the parse failed (${String(params.parseResult.errorNodeCount)} error node(s), ${String(params.parseResult.missingNodeCount)} missing node(s)), so the file was cut into line windows; no symbol association exists for this file`,
+            }
+            : {
+                reason: "ast_regions_missing",
+                strategy: "line_fallback",
+                detail: `the parse reported "${params.parseResult.parseQuality}" but the language adapter delivered no chunk regions, so the file was cut into line windows; no symbol association exists for this file although the parse claims success`,
+            });
+    }
+    else {
+        const result = runAstChunkStrategy(ctx, params.regions, params.symbols);
+        drafts = result.drafts;
+        coverage = result.coverage;
+        strategy = "ast";
+    }
+    fillGaps(ctx, drafts, coverage);
+    // Two-sided machine check (spec.md §8.4): the ledger must be COMPLETE
+    // (assertLosslessCoverage) and every claim in it must be TRUE
+    // (assertTruthfulLedger) — a synthesized chunk may never stand in for
+    // bytes its content does not literally carry.
+    assertTruthfulLedger(ctx.lineIndex, drafts, coverage, params.identity.filePath);
+    assertLosslessCoverage(coverage, ctx.fileRange, params.identity.filePath);
+    return {
+        drafts,
+        coverage,
+        strategy,
+        parseQuality: params.parseResult.parseQuality,
+        degradations,
+    };
+}
+//# sourceMappingURL=chunker.js.map

package/dist/chunker.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"chunker.js","sourceRoot":"","sources":["../src/chunker.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AACH,OAAO,EAAE,kBAAkB,EAAE,gBAAgB,EAAE,MAAM,uBAAuB,CAAC;AAG7E,OAAO,EAAE,oBAAoB,EAAE,MAAM,aAAa,CAAC;AACnD,OAAO,EAAE,SAAS,EAAE,kBAAkB,EAAE,SAAS,EAAE,MAAM,cAAc,CAAC;AACxE,OAAO,EAAE,sBAAsB,EAAE,oBAAoB,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC3F,OAAO,EAAE,qBAAqB,EAAE,MAAM,yBAAyB,CAAC;AAChE,OAAO,EAAE,oBAAoB,EAAE,MAAM,YAAY,CAAC;AAClD,OAAO,EAAE,mBAAmB,EAAE,MAAM,oCAAoC,CAAC;AACzE,OAAO,EAAE,uBAAuB,EAAE,MAAM,wCAAwC,CAAC;AACjF,OAAO,EAAE,mBAAmB,EAAE,MAAM,mCAAmC,CAAC;AAGxE,MAAM,UAAU,GAAG,EAAE,SAAS,EAAE,gBAAgB,EAAW,CAAC;AAE5D,sGAAsG;AACtG,SAAS,QAAQ,CACf,GAA0C,EAC1C,MAAwB,EACxB,QAAyB;IAEzB,MAAM,IAAI,GAAG,gBAAgB,CAAC,QAAQ,EAAE,GAAG,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC;IAC5D,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,KAAK,GAAgB,EAAE,GAAG,GAAG,EAAE,SAAS,EAAE,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC;QAClE,KAAK,MAAM,KAAK,IAAI,oBAAoB,CAAC,GAAG,EAAE,KAAK,EAAE,UAAU,CAAC,EAAE,CAAC;YACjE,MAAM,KAAK,GAAG,SAAS,CAAC,GAAG,EAAE;gBAC3B,KAAK,EAAE,KAAK;gBACZ,OAAO,EAAE,SAAS,CAAC,GAAG,EAAE,KAAK,CAAC;gBAC9B,OAAO,EAAE,GAAG,CAAC,QAAQ,CAAC,MAAM;gBAC5B,SAAS,EAAE,gBAAgB;gBAC3B,MAAM,EAAE;oBACN,SAAS,EAAE,gBAAgB;oBAC3B,YAAY,EAAE,OAAO;oBACrB,cAAc,EAAE,GAAG,CAAC,QAAQ,CAAC,MAAM;iBACpC;aACF,CAAC,CAAC;YACH,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACnB,QAAQ,CAAC,IAAI,CAAC,YAAY,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC;AACH,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,SAAS,CAAC,MAAuB;IAC/C,MAAM,MAAM,GAAG,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;IACnD,MAAM,GAAG,GAAG,kBAAkB,CAC5B,MAAM,CAAC,QAAQ,EACf,MAAM,CAAC,OAAO,EACd,MAAM,EACN,MAAM,CAAC,WAAW,CAAC,YAAY,CAChC,CAAC;IAEF,MAAM,UAAU,GACd,MAAM,CAAC,UAAU;QACjB,qBAAqB,CACnB,MAAM,CAAC,OAAO,EACd,MAAM,CAAC,iBAAiB,EACxB,kBAAkB,CAAC,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAC7C,CAAC;IACJ,MAAM,YAAY,GAAG,MAAM,CAAC,WAAW,CAAC,YAAY,KAAK,QAAQ,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC;IAE/F,IAAI,MAAwB,CAAC;IAC7B,IAAI,QAAyB,CAAC;IAC9B,IAAI,QAAqC,CAAC;IAC1C,wEAAwE;IACxE,2EAA2E;IAC3E,4EAA4E;IAC5E,uCAAuC;IACvC,MAAM,YAAY,GAAuB,EAAE,CAAC;IAE5C,IAAI,UAAU,EAAE,CAAC;QACf,MAAM,MAAM,GAAG,mBAAmB,CAAC,GAAG,CAAC,CAAC;QACxC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC;QACvB,QAAQ,GAAG,MAAM,CAAC,QAAQ,CAAC;QAC3B,QAAQ,GAAG,UAAU,CAAC;QACtB,YAAY,CAAC,IAAI,CAAC;YAChB,MAAM,EAAE,kBAAkB;YAC1B,QAAQ,EAAE,UAAU;YACpB,MAAM,EAAE,uCACN,MAAM,CAAC,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,wBACtD,gFAAgF;SACjF,CAAC,CAAC;IACL,CAAC;SAAM,IAAI,CAAC,YAAY,EAAE,CAAC;QACzB,MAAM,MAAM,GAAG,uBAAuB,CAAC,GAAG,CAAC,CAAC;QAC5C,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC;QACvB,QAAQ,GAAG,MAAM,CAAC,QAAQ,CAAC;QAC3B,QAAQ,GAAG,eAAe,CAAC;QAC3B,YAAY,CAAC,IAAI,CACf,MAAM,CAAC,WAAW,CAAC,YAAY,KAAK,QAAQ;YAC1C,CAAC,CAAC;gBACE,MAAM,EAAE,cAAc;gBACtB,QAAQ,EAAE,eAAe;gBACzB,MAAM,EAAE,qBAAqB,MAAM,CACjC,MAAM,CAAC,WAAW,CAAC,cAAc,CAClC,mBAAmB,MAAM,CACxB,MAAM,CAAC,WAAW,CAAC,gBAAgB,CACpC,sGAAsG;aACxG;YACH,CAAC,CAAC;gBACE,MAAM,EAAE,qBAAqB;gBAC7B,QAAQ,EAAE,eAAe;gBACzB,MAAM,EAAE,uBAAuB,MAAM,CAAC,WAAW,CAAC,YAAY,4KAA4K;aAC3O,CACN,CAAC;IACJ,CAAC;SAAM,CAAC;QACN,MAAM,MAAM,GAAG,mBAAmB,CAAC,GAAG,EAAE,MAAM,CAAC,OAAO,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC;QACxE,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC;QACvB,QAAQ,GAAG,MAAM,CAAC,QAAQ,CAAC;QAC3B,QAAQ,GAAG,KAAK,CAAC;IACnB,CAAC;IAED,QAAQ,CAAC,GAAG,EAAE,MAAM,EAAE,QAAQ,CAAC,CAAC;IAChC,sEAAsE;IACtE,8DAA8D;IAC9D,sEAAsE;IACtE,8CAA8C;IAC9C,oBAAoB,CAAC,GAAG,CAAC,SAAS,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;IAChF,sBAAsB,CAAC,QAAQ,EAAE,GAAG,CAAC,SAAS,EAAE,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;IAE1E,OAAO;QACL,MAAM;QACN,QAAQ;QACR,QAAQ;QACR,YAAY,EAAE,MAAM,CAAC,WAAW,CAAC,YAAY;QAC7C,YAAY;KACb,CAAC;AACJ,CAAC"}

package/dist/config.d.ts

@@ -0,0 +1,17 @@
+import type { ChunkerConfig, ChunkerConfigInput } from "./types.js";
+/**
+ * Untangle decision (README "Entscheidungen"): 800 tokens is the default
+ * per-chunk budget — generous enough for a typical method or a small class
+ * overview, small enough that four to six chunks fit a model's context
+ * alongside a query and other results.
+ */
+export declare const DEFAULT_MAX_TOKENS_PER_CHUNK = 800;
+export declare const DEFAULT_LINE_FALLBACK_CONFIG: {
+    linesPerChunk: number;
+    overlapLines: number;
+};
+export declare const DEFAULT_MINIFIED_STRATEGY_CONFIG: {
+    overlapBytes: number;
+};
+export declare function resolveChunkerConfig(input: ChunkerConfigInput): ChunkerConfig;
+//# sourceMappingURL=config.d.ts.map

package/dist/config.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAE,aAAa,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;AAEpE;;;;;GAKG;AACH,eAAO,MAAM,4BAA4B,MAAM,CAAC;AAEhD,eAAO,MAAM,4BAA4B;;;CAAyC,CAAC;AACnF,eAAO,MAAM,gCAAgC;;CAAsB,CAAC;AAEpE,wBAAgB,oBAAoB,CAAC,KAAK,EAAE,kBAAkB,GAAG,aAAa,CAW7E"}

package/dist/config.js

@@ -0,0 +1,28 @@
+/**
+ * Default chunker configuration and resolution of caller overrides.
+ */
+import { CHUNKER_VERSION, PARSER_VERSION } from "@untangle-code/domain";
+import { DEFAULT_MINIFIED_DETECTION_CONFIG } from "./minified-detection.js";
+import { DEFAULT_TOKEN_ESTIMATOR_CONFIG } from "./token-estimator.js";
+/**
+ * Untangle decision (README "Entscheidungen"): 800 tokens is the default
+ * per-chunk budget — generous enough for a typical method or a small class
+ * overview, small enough that four to six chunks fit a model's context
+ * alongside a query and other results.
+ */
+export const DEFAULT_MAX_TOKENS_PER_CHUNK = 800;
+export const DEFAULT_LINE_FALLBACK_CONFIG = { linesPerChunk: 60, overlapLines: 8 };
+export const DEFAULT_MINIFIED_STRATEGY_CONFIG = { overlapBytes: 0 };
+export function resolveChunkerConfig(input) {
+    return {
+        chunkerVersion: input.chunkerVersion ?? CHUNKER_VERSION,
+        parserVersion: input.parserVersion ?? PARSER_VERSION,
+        maxTokensPerChunk: input.maxTokensPerChunk ?? DEFAULT_MAX_TOKENS_PER_CHUNK,
+        tokenEstimator: input.tokenEstimator ?? DEFAULT_TOKEN_ESTIMATOR_CONFIG,
+        preparedText: input.preparedText,
+        minifiedDetection: input.minifiedDetection ?? DEFAULT_MINIFIED_DETECTION_CONFIG,
+        lineFallback: input.lineFallback ?? DEFAULT_LINE_FALLBACK_CONFIG,
+        minifiedStrategy: input.minifiedStrategy ?? DEFAULT_MINIFIED_STRATEGY_CONFIG,
+    };
+}
+//# sourceMappingURL=config.js.map

package/dist/config.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA;;GAEG;AACH,OAAO,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AAExE,OAAO,EAAE,iCAAiC,EAAE,MAAM,yBAAyB,CAAC;AAC5E,OAAO,EAAE,8BAA8B,EAAE,MAAM,sBAAsB,CAAC;AAGtE;;;;;GAKG;AACH,MAAM,CAAC,MAAM,4BAA4B,GAAG,GAAG,CAAC;AAEhD,MAAM,CAAC,MAAM,4BAA4B,GAAG,EAAE,aAAa,EAAE,EAAE,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC;AACnF,MAAM,CAAC,MAAM,gCAAgC,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC;AAEpE,MAAM,UAAU,oBAAoB,CAAC,KAAyB;IAC5D,OAAO;QACL,cAAc,EAAE,KAAK,CAAC,cAAc,IAAI,eAAe;QACvD,aAAa,EAAE,KAAK,CAAC,aAAa,IAAI,cAAc;QACpD,iBAAiB,EAAE,KAAK,CAAC,iBAAiB,IAAI,4BAA4B;QAC1E,cAAc,EAAE,KAAK,CAAC,cAAc,IAAI,8BAA8B;QACtE,YAAY,EAAE,KAAK,CAAC,YAAY;QAChC,iBAAiB,EAAE,KAAK,CAAC,iBAAiB,IAAI,iCAAiC;QAC/E,YAAY,EAAE,KAAK,CAAC,YAAY,IAAI,4BAA4B;QAChE,gBAAgB,EAAE,KAAK,CAAC,gBAAgB,IAAI,gCAAgC;KAC7E,CAAC;AACJ,CAAC"}