@memberjunction/ai-segmentation 0.0.0 → 5.51.0

package/README.md

@@ -1,45 +1,179 @@
 # @memberjunction/ai-segmentation
 
-## ⚠️ IMPORTANT NOTICE ⚠️
-
-**This package is created solely for the purpose of setting up OIDC (OpenID Connect) trusted publishing with npm.**
-
-This is **NOT** a functional package and contains **NO** code or functionality beyond the OIDC setup configuration.
-
-## Purpose
-
-This package exists to:
-1. Configure OIDC trusted publishing for the package name `@memberjunction/ai-segmentation`
-2. Enable secure, token-less publishing from CI/CD workflows
-3. Establish provenance for packages published under this name
-
-## What is OIDC Trusted Publishing?
-
-OIDC trusted publishing allows package maintainers to publish packages directly from their CI/CD workflows without needing to manage npm access tokens. Instead, it uses OpenID Connect to establish trust between the CI/CD provider (like GitHub Actions) and npm.
-
-## Setup Instructions
-
-To properly configure OIDC trusted publishing for this package:
-
-1. Go to [npmjs.com](https://www.npmjs.com/) and navigate to your package settings
-2. Configure the trusted publisher (e.g., GitHub Actions)
-3. Specify the repository and workflow that should be allowed to publish
-4. Use the configured workflow to publish your actual package
-
-## DO NOT USE THIS PACKAGE
-
-This package is a placeholder for OIDC configuration only. It:
-- Contains no executable code
-- Provides no functionality
-- Should not be installed as a dependency
-- Exists only for administrative purposes
-
-## More Information
-
-For more details about npm's trusted publishing feature, see:
-- [npm Trusted Publishing Documentation](https://docs.npmjs.com/generating-provenance-statements)
-- [GitHub Actions OIDC Documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
-
----
-
-**Maintained for OIDC setup purposes only**
+Pluggable **content segmentation** for MemberJunction's RAG ingestion — the step that decides *what*
+gets embedded, before an embedding model decides *how*.
+
+## Why this package exists
+
+Retrieval quality is capped by segmentation quality. An embedding is only as good as the span of
+content it represents: split a document on an arbitrary token boundary and you get vectors that
+straddle two topics and match neither query well. Feed a 60-minute recording to a multimodal model in
+one call and you get a mush vector, because those models sample a bounded window regardless of clip
+length.
+
+Before this package, chunking was a private helper inside whichever pipeline needed it — which meant
+every new strategy would have been another bespoke branch in someone else's engine. Segmentation is
+now a **registered, swappable strategy**, selected the same way `BaseEmbeddings` and `VectorDBBase`
+providers already are.
+
+## Layering
+
+```
+@memberjunction/ai-vectors        TextChunker  — "split this string to fit a token budget"
+            ▲
+            │ uses
+@memberjunction/ai-segmentation   BaseSegmenter — "what are the meaningful units of this content"
+            ▲
+            │ consumed by
+   ingestion pipelines (content autotagging, vector sync, knowledge pipeline)
+```
+
+`TextChunker` stays a low-level string primitive in `ai-vectors`. Segmenters live one layer up,
+where `@memberjunction/ai-prompts` is available — which is what lets `SemanticText` run a real,
+tracked `MJ: AI Prompt Run` rather than an untracked side-channel LLM call. (`ai-vectors` cannot
+depend on `ai-prompts`: `ai-prompts → templates → ai-provider-bundle → ai-vectors-pinecone →
+ai-vectors` would make it circular.)
+
+## Built-in segmenters
+
+| Key | Class | Best for |
+|---|---|---|
+| `StructuralText` | `StructuralTextSegmenter` | Documents with headings — markdown, HTML, converted PDFs. **Recommended text default.** |
+| `SemanticText` | `SemanticTextSegmenter` | Prose with no structure — transcripts, reports, long articles. Uses an LLM to find latent topic boundaries. |
+| `Transcript` | `TranscriptSegmenter` | Audio/video with a timed transcript. Produces time-windowed **chapters** (and optional per-speaker sub-chapters). |
+| `FixedWindow` | `FixedWindowSegmenter` | Universal fallback — token windows for text, fixed-duration windows for untranscribed media. |
+
+The ordering in `SuggestSegmenterKey` encodes the quality hierarchy: a real transcript beats document
+structure, which beats uniform windows.
+
+## Usage
+
+```typescript
+import { ResolveSegmenter, SuggestSegmenterKey } from '@memberjunction/ai-segmentation';
+
+const params = { Text: extractedText, MimeType: 'text/markdown', ContextUser: contextUser };
+const segmenter = ResolveSegmenter(contentType.SegmenterKey, SuggestSegmenterKey(params));
+
+const result = await segmenter.Segment({ ...params, Options: { MaxSegmentTokens: 512 } });
+if (!result.Success) {
+    LogError(`Segmentation failed: ${result.ErrorMessage}`);
+    return;
+}
+
+for (const segment of result.Segments) {
+    // segment.Text        -> embed into the text index
+    // segment.Media       -> embed via EmbedContent into a multimodal index
+    // segment.StartMs/EndMs, StartOffset/EndOffset -> persist as chunk provenance
+    // segment.ParentSequence -> chapter / sub-chapter hierarchy
+}
+```
+
+Segmenters **never throw** for content-shaped problems; they return `Success: false` with an
+`ErrorMessage`, matching the convention used by `RunView` and `BaseEntity.Save()`.
+
+### Bootstrap
+
+Segmenters are resolved dynamically through the class factory, so bundlers may tree-shake them. Call
+the load-prevention export once from your application bootstrap:
+
+```typescript
+import { LoadContentSegmenters } from '@memberjunction/ai-segmentation';
+LoadContentSegmenters();
+```
+
+Skipping this doesn't produce an error — `ResolveSegmenter` degrades to the fallback strategy — so
+content still gets chunked, just by the wrong strategy and silently.
+
+### Audio/video chapters
+
+```typescript
+const result = await ResolveSegmenter('Transcript').Segment({
+    Media: { URL: 'https://cdn/session-1428.mp4', MimeType: 'video/mp4' },
+    Cues: timedTranscriptCues,          // from ASR, or MJ realtime session capture
+    Options: { MaxChapterMs: 300_000, BoundaryGapMs: 4_000 },
+});
+```
+
+Each chapter carries **both** a media reference with a `StartMs`/`EndMs` window *and* the transcript
+text for that window. That dual payload is the point: the media reference is what a multimodal model
+embeds for retrieval, while the transcript is what an agent can read, what a cross-encoder can rerank,
+and what keyword search can match.
+
+This is **one vector per chunk**, not two — the native multimodal vector — with the description and
+transcript stored as text alongside it. A short summary plus structured fields (`SegmentTitle`,
+`StartMs`/`EndMs`, `Speaker`, `Modality`) may be mirrored into the vector record's metadata for
+filtering and display; a full transcript should not be, since metadata is size-capped and is
+filter/payload rather than ranked full-text.
+
+### Making media findable from a text query
+
+| Path | Strong at | Weak at |
+|---|---|---|
+| Native text→media similarity | visual/audio semantics | runs colder than text→text |
+| Lexical (FTS/BM25) over the description | names, acronyms, jargon, titles | paraphrase |
+| A description *vector* (text→text) | paraphrase | rare proper nouns |
+
+Ship the first two; add a description vector selectively, once measurement shows paraphrase misses.
+When you do, add it as a **sibling text chunk row** pointing at the media chunk rather than a second
+vector on the same row — that keeps one vector per row and lets retrieval collapse duplicates on the
+parent key. For speech-dominant archives, also price transcript-only ingestion first: text chapters
+alone give full semantic search at zero multimodal embedding spend.
+
+See the [Content Segmentation Guide](../../../guides/CONTENT_SEGMENTATION_GUIDE.md) for the full
+rationale and the cross-package picture.
+
+## Adding a new strategy
+
+Implement one method and register the class:
+
+```typescript
+@RegisterClass(BaseSegmenter, 'MyStrategy')
+export class MySegmenter extends BaseSegmenter {
+    public get Key(): string { return 'MyStrategy'; }
+    public get SupportedModalities(): ContentModality[] { return ['text']; }
+
+    protected async SegmentCore(params: SegmentationParams): Promise<RawSegment[]> {
+        return findBoundaries(params.Text ?? '').map(b => ({
+            Modality: 'text', Text: b.Text, StartOffset: b.Start, EndOffset: b.End,
+        }));
+    }
+}
+```
+
+`BaseSegmenter` handles everything else: input validation, enforcing the token ceiling (splitting
+oversized segments while preserving titles and rebasing offsets), merging undersized segments,
+sequence numbering, remapping `ParentIndex` → `ParentSequence` after splits, cycle-safe depth
+calculation, and provenance stamping. Subclasses reference sibling segments by their index in the
+array they just returned and never reason about post-split numbering.
+
+## Options
+
+Common to every segmenter (`SegmentationOptions`):
+
+| Option | Default | Purpose |
+|---|---|---|
+| `MaxSegmentTokens` | 512 | Hard ceiling; the base class splits anything larger. |
+| `OverlapTokens` | 10% of max | Overlap applied when an oversized segment is split. |
+| `MinSegmentTokens` | 0 (off) | Merge adjacent text segments below this size. |
+
+Each segmenter extends these with its own strongly-typed options — see
+`StructuralTextSegmentationOptions`, `SemanticTextSegmentationOptions`,
+`TranscriptSegmentationOptions`, and `FixedWindowSegmentationOptions`.
+
+## Cost posture
+
+Segmentation runs on every ingested item, so the defaults are deliberately cheap:
+
+- `StructuralText` and `FixedWindow` make **no** model calls.
+- `SemanticText` skips the LLM entirely for documents under `MinTokensForLLM` (default 750 tokens),
+  truncates each block to a preview before prompting, asks the model for **block indices** rather than
+  character offsets (models are unreliable at arithmetic over long strings, and a bad offset would
+  silently corrupt chunk provenance), and degrades to `StructuralText` on any failure.
+- `Transcript` emits sub-chapters only when `EmitSubChapters` is enabled, since they double the
+  embedding count for a chapter.
+
+## Testing
+
+```bash
+cd packages/AI/Segmentation && npm run test
+```

package/dist/generic/AdaptiveBoundarySegmenter.d.ts

@@ -0,0 +1,98 @@
+/**
+ * @fileoverview Target-size segmenter with an escalating boundary preference.
+ *
+ * @module @memberjunction/ai-segmentation
+ */
+import { BaseSegmenter } from './BaseSegmenter.js';
+import { ContentModality, RawSegment, SegmentationOptions, SegmentationParams } from './Segmentation.types.js';
+/** Registration key for {@link AdaptiveBoundarySegmenter}. */
+export declare const ADAPTIVE_BOUNDARY_SEGMENTER_KEY = "AdaptiveBoundary";
+/** Options specific to {@link AdaptiveBoundarySegmenter}. */
+export interface AdaptiveBoundarySegmentationOptions extends SegmentationOptions {
+    /**
+     * Desired segment size in tokens.
+     *
+     * **Size this to your queries, not to your embedding model.** The model's context
+     * window is an upper bound, not a target — a chunk should be about as much content as
+     * a good answer to a typical query, so that a matching chunk is mostly signal. If
+     * queries are short paraphrases, smaller chunks retrieve better; if downstream
+     * summarization wants context, larger ones do. Default: 512.
+     */
+    TargetTokens?: number;
+    /**
+     * How far *below* target (percent) the segmenter may close on a good boundary.
+     * Entering this band is what makes segment sizes vary in service of clean breaks.
+     * Default: 20.
+     */
+    UndershootPercent?: number;
+    /**
+     * How far *above* target (percent) it keeps looking for a sentence or word boundary
+     * before giving up and cutting at the hard ceiling. Default: 20.
+     */
+    OvershootPercent?: number;
+    /**
+     * If the whole text is within this percent above target, emit it as ONE segment
+     * rather than splitting it into a large piece plus a small remainder. Default: 40.
+     */
+    NoSplitPercent?: number;
+}
+/**
+ * Splits text toward a **target** size, closing on the best available natural boundary
+ * near that target rather than cutting at a fixed offset.
+ *
+ * ## Why this beats a fixed window
+ *
+ * A fixed window cuts wherever the budget runs out, which routinely lands mid-paragraph
+ * — the chunk then straddles two ideas and matches neither query well. This segmenter
+ * treats the target as a goal with a tolerance band and escalates through boundary
+ * quality as it goes:
+ *
+ * 1. Once within `UndershootPercent` of target, close on a **paragraph** break.
+ * 2. Past target, accept a **sentence** break.
+ * 3. Past `OvershootPercent`, accept a **word** break.
+ * 4. Failing all of those, cut at the hard `MaxSegmentTokens` ceiling.
+ *
+ * Segment sizes therefore vary — deliberately. A slightly short segment that ends at a
+ * paragraph is worth more at retrieval time than an exactly-sized one that ends mid-clause.
+ *
+ * It also declines to split at all when the whole text is only modestly over target
+ * (`NoSplitPercent`), which avoids the common pathology of one full-size chunk followed by
+ * a runt carrying two sentences and no context.
+ *
+ * This is the recommended default for prose when document structure isn't available;
+ * prefer `StructuralText` when the content has headings, since an authored boundary beats
+ * an inferred one.
+ */
+export declare class AdaptiveBoundarySegmenter extends BaseSegmenter {
+    get Key(): string;
+    get SupportedModalities(): ContentModality[];
+    protected SegmentCore(params: SegmentationParams<AdaptiveBoundarySegmentationOptions>): RawSegment[];
+    /** True when the whole text is close enough to target that splitting would only make a runt. */
+    private fitsWithoutSplitting;
+    /** Walk the text emitting one segment per located boundary. */
+    private walk;
+    /**
+     * Locate this segment's end by escalating through boundary quality.
+     * Ranges are half-open [from, to).
+     */
+    private findBoundary;
+    /**
+     * End offset of the last match of `pattern` starting within [from, to), or null.
+     * The returned offset is the END of the match, so the delimiter stays with the
+     * segment it terminates rather than opening the next one.
+     */
+    private lastMatch;
+    /** Merge caller options with adaptive-specific defaults. */
+    private resolveAdaptiveOptions;
+    /** Keep a percentage option inside a sane range. */
+    private clampPercent;
+    /** Target size expressed in characters (TextChunker's ~4 chars/token estimate). */
+    private targetChars;
+    /** Hard ceiling expressed in characters. */
+    private hardChars;
+    /** Overlap in characters, capped at half the target so segments always advance. */
+    private overlapChars;
+    /** Estimated tokens for a string — exposed for callers reasoning about sizing. */
+    EstimateTokens(text: string): number;
+}
+//# sourceMappingURL=AdaptiveBoundarySegmenter.d.ts.map

package/dist/generic/AdaptiveBoundarySegmenter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"AdaptiveBoundarySegmenter.d.ts","sourceRoot":"","sources":["../../src/generic/AdaptiveBoundarySegmenter.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAIH,OAAO,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAChD,OAAO,EAAE,eAAe,EAAE,UAAU,EAAE,mBAAmB,EAAE,kBAAkB,EAAE,MAAM,sBAAsB,CAAC;AAE5G,8DAA8D;AAC9D,eAAO,MAAM,+BAA+B,qBAAqB,CAAC;AAElE,6DAA6D;AAC7D,MAAM,WAAW,mCAAoC,SAAQ,mBAAmB;IAC5E;;;;;;;;OAQG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B;;;OAGG;IACH,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B;;;OAGG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;CAC3B;AAUD;;;;;;;;;;;;;;;;;;;;;;;;;;GA0BG;AACH,qBACa,yBAA0B,SAAQ,aAAa;IACxD,IAAW,GAAG,IAAI,MAAM,CAEvB;IAED,IAAW,mBAAmB,IAAI,eAAe,EAAE,CAElD;IAED,SAAS,CAAC,WAAW,CAAC,MAAM,EAAE,kBAAkB,CAAC,mCAAmC,CAAC,GAAG,UAAU,EAAE;IAYpG,gGAAgG;IAChG,OAAO,CAAC,oBAAoB;IAK5B,+DAA+D;IAC/D,OAAO,CAAC,IAAI;IAoBZ;;;OAGG;IACH,OAAO,CAAC,YAAY;IA6BpB;;;;OAIG;IACH,OAAO,CAAC,SAAS;IAmBjB,4DAA4D;IAC5D,OAAO,CAAC,sBAAsB;IAiB9B,oDAAoD;IACpD,OAAO,CAAC,YAAY;IAOpB,mFAAmF;IACnF,OAAO,CAAC,WAAW;IAInB,4CAA4C;IAC5C,OAAO,CAAC,SAAS;IAIjB,mFAAmF;IACnF,OAAO,CAAC,YAAY;IAIpB,kFAAkF;IAC3E,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM;CAG9C"}

package/dist/generic/AdaptiveBoundarySegmenter.js

@@ -0,0 +1,177 @@
+/**
+ * @fileoverview Target-size segmenter with an escalating boundary preference.
+ *
+ * @module @memberjunction/ai-segmentation
+ */
+var __decorate = (this && this.__decorate) || function (decorators, target, key, desc) {
+    var c = arguments.length, r = c < 3 ? target : desc === null ? desc = Object.getOwnPropertyDescriptor(target, key) : desc, d;
+    if (typeof Reflect === "object" && typeof Reflect.decorate === "function") r = Reflect.decorate(decorators, target, key, desc);
+    else for (var i = decorators.length - 1; i >= 0; i--) if (d = decorators[i]) r = (c < 3 ? d(r) : c > 3 ? d(target, key, r) : d(target, key)) || r;
+    return c > 3 && r && Object.defineProperty(target, key, r), r;
+};
+import { RegisterClass } from '@memberjunction/global';
+import { TextChunker } from '@memberjunction/ai-vectors';
+import { BaseSegmenter } from './BaseSegmenter.js';
+/** Registration key for {@link AdaptiveBoundarySegmenter}. */
+export const ADAPTIVE_BOUNDARY_SEGMENTER_KEY = 'AdaptiveBoundary';
+/**
+ * Splits text toward a **target** size, closing on the best available natural boundary
+ * near that target rather than cutting at a fixed offset.
+ *
+ * ## Why this beats a fixed window
+ *
+ * A fixed window cuts wherever the budget runs out, which routinely lands mid-paragraph
+ * — the chunk then straddles two ideas and matches neither query well. This segmenter
+ * treats the target as a goal with a tolerance band and escalates through boundary
+ * quality as it goes:
+ *
+ * 1. Once within `UndershootPercent` of target, close on a **paragraph** break.
+ * 2. Past target, accept a **sentence** break.
+ * 3. Past `OvershootPercent`, accept a **word** break.
+ * 4. Failing all of those, cut at the hard `MaxSegmentTokens` ceiling.
+ *
+ * Segment sizes therefore vary — deliberately. A slightly short segment that ends at a
+ * paragraph is worth more at retrieval time than an exactly-sized one that ends mid-clause.
+ *
+ * It also declines to split at all when the whole text is only modestly over target
+ * (`NoSplitPercent`), which avoids the common pathology of one full-size chunk followed by
+ * a runt carrying two sentences and no context.
+ *
+ * This is the recommended default for prose when document structure isn't available;
+ * prefer `StructuralText` when the content has headings, since an authored boundary beats
+ * an inferred one.
+ */
+let AdaptiveBoundarySegmenter = class AdaptiveBoundarySegmenter extends BaseSegmenter {
+    get Key() {
+        return ADAPTIVE_BOUNDARY_SEGMENTER_KEY;
+    }
+    get SupportedModalities() {
+        return ['text'];
+    }
+    SegmentCore(params) {
+        const text = params.Text ?? '';
+        if (text.trim().length === 0) {
+            return [];
+        }
+        const settings = this.resolveAdaptiveOptions(params.Options);
+        if (this.fitsWithoutSplitting(text, settings)) {
+            return [{ Modality: 'text', Text: text.trim(), StartOffset: 0, EndOffset: text.length }];
+        }
+        return this.walk(text, settings);
+    }
+    /** True when the whole text is close enough to target that splitting would only make a runt. */
+    fitsWithoutSplitting(text, settings) {
+        const limit = this.targetChars(settings) * (1 + settings.NoSplitPercent / 100);
+        return text.length <= limit;
+    }
+    /** Walk the text emitting one segment per located boundary. */
+    walk(text, settings) {
+        const segments = [];
+        const overlapChars = this.overlapChars(settings);
+        let cursor = 0;
+        while (cursor < text.length) {
+            const boundary = this.findBoundary(text, cursor, settings);
+            const body = text.slice(cursor, boundary.End).trim();
+            if (body.length > 0) {
+                segments.push({ Modality: 'text', Text: body, StartOffset: cursor, EndOffset: boundary.End });
+            }
+            if (boundary.End >= text.length) {
+                break;
+            }
+            // Always advance, even when the overlap would otherwise stall the cursor.
+            cursor = Math.max(boundary.End - overlapChars, cursor + 1);
+        }
+        return segments;
+    }
+    /**
+     * Locate this segment's end by escalating through boundary quality.
+     * Ranges are half-open [from, to).
+     */
+    findBoundary(text, cursor, settings) {
+        const target = this.targetChars(settings);
+        const softMin = cursor + Math.floor(target * (1 - settings.UndershootPercent / 100));
+        const softMax = cursor + Math.floor(target * (1 + settings.OvershootPercent / 100));
+        const hardMax = cursor + this.hardChars(settings);
+        if (text.length <= softMax) {
+            return { End: text.length, Kind: 'end' };
+        }
+        const paragraph = this.lastMatch(text, /\n\s*\n/g, softMin, softMax);
+        if (paragraph !== null) {
+            return { End: paragraph, Kind: 'paragraph' };
+        }
+        const sentence = this.lastMatch(text, /[.!?]["')\]]?\s/g, softMin, softMax);
+        if (sentence !== null) {
+            return { End: sentence, Kind: 'sentence' };
+        }
+        const word = this.lastMatch(text, /\s+/g, softMin, Math.min(hardMax, text.length));
+        if (word !== null) {
+            return { End: word, Kind: 'word' };
+        }
+        return { End: Math.min(hardMax, text.length), Kind: 'hard' };
+    }
+    /**
+     * End offset of the last match of `pattern` starting within [from, to), or null.
+     * The returned offset is the END of the match, so the delimiter stays with the
+     * segment it terminates rather than opening the next one.
+     */
+    lastMatch(text, pattern, from, to) {
+        if (to <= from) {
+            return null;
+        }
+        const scan = new RegExp(pattern.source, 'g');
+        scan.lastIndex = Math.max(from, 0);
+        let found = null;
+        let match = scan.exec(text);
+        while (match !== null && match.index < to) {
+            found = match.index + match[0].length;
+            match = scan.exec(text);
+        }
+        return found;
+    }
+    // ─────────────────────────────────────────────
+    // Settings
+    // ─────────────────────────────────────────────
+    /** Merge caller options with adaptive-specific defaults. */
+    resolveAdaptiveOptions(options) {
+        const base = this.resolveOptions(options);
+        const target = Math.max(options?.TargetTokens ?? base.MaxSegmentTokens, 1);
+        return {
+            ...base,
+            // The hard ceiling can never sit below the target, or every segment would be
+            // cut by the ceiling before a boundary was ever considered.
+            MaxSegmentTokens: Math.max(base.MaxSegmentTokens, target),
+            TargetTokens: target,
+            UndershootPercent: this.clampPercent(options?.UndershootPercent ?? 20, 0, 90),
+            OvershootPercent: this.clampPercent(options?.OvershootPercent ?? 20, 0, 200),
+            NoSplitPercent: this.clampPercent(options?.NoSplitPercent ?? 40, 0, 500),
+        };
+    }
+    /** Keep a percentage option inside a sane range. */
+    clampPercent(value, min, max) {
+        if (!Number.isFinite(value)) {
+            return min;
+        }
+        return Math.min(Math.max(value, min), max);
+    }
+    /** Target size expressed in characters (TextChunker's ~4 chars/token estimate). */
+    targetChars(settings) {
+        return settings.TargetTokens * 4;
+    }
+    /** Hard ceiling expressed in characters. */
+    hardChars(settings) {
+        return settings.MaxSegmentTokens * 4;
+    }
+    /** Overlap in characters, capped at half the target so segments always advance. */
+    overlapChars(settings) {
+        return Math.min(settings.OverlapTokens * 4, Math.floor(this.targetChars(settings) / 2));
+    }
+    /** Estimated tokens for a string — exposed for callers reasoning about sizing. */
+    EstimateTokens(text) {
+        return TextChunker.EstimateTokenCount(text);
+    }
+};
+AdaptiveBoundarySegmenter = __decorate([
+    RegisterClass(BaseSegmenter, ADAPTIVE_BOUNDARY_SEGMENTER_KEY)
+], AdaptiveBoundarySegmenter);
+export { AdaptiveBoundarySegmenter };
+//# sourceMappingURL=AdaptiveBoundarySegmenter.js.map

package/dist/generic/AdaptiveBoundarySegmenter.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"AdaptiveBoundarySegmenter.js","sourceRoot":"","sources":["../../src/generic/AdaptiveBoundarySegmenter.ts"],"names":[],"mappings":"AAAA;;;;GAIG;;;;;;;AAEH,OAAO,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAC;AACvD,OAAO,EAAE,WAAW,EAAE,MAAM,4BAA4B,CAAC;AACzD,OAAO,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAGhD,8DAA8D;AAC9D,MAAM,CAAC,MAAM,+BAA+B,GAAG,kBAAkB,CAAC;AAwClE;;;;;;;;;;;;;;;;;;;;;;;;;;GA0BG;AAEI,IAAM,yBAAyB,GAA/B,MAAM,yBAA0B,SAAQ,aAAa;IACxD,IAAW,GAAG;QACV,OAAO,+BAA+B,CAAC;IAC3C,CAAC;IAED,IAAW,mBAAmB;QAC1B,OAAO,CAAC,MAAM,CAAC,CAAC;IACpB,CAAC;IAES,WAAW,CAAC,MAA+D;QACjF,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;QAC/B,IAAI,IAAI,CAAC,IAAI,EAAE,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC3B,OAAO,EAAE,CAAC;QACd,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,sBAAsB,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QAC7D,IAAI,IAAI,CAAC,oBAAoB,CAAC,IAAI,EAAE,QAAQ,CAAC,EAAE,CAAC;YAC5C,OAAO,CAAC,EAAE,QAAQ,EAAE,MAAM,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,EAAE,WAAW,EAAE,CAAC,EAAE,SAAS,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;QAC7F,CAAC;QACD,OAAO,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACrC,CAAC;IAED,gGAAgG;IACxF,oBAAoB,CAAC,IAAY,EAAE,QAAuD;QAC9F,MAAM,KAAK,GAAG,IAAI,CAAC,WAAW,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,GAAG,QAAQ,CAAC,cAAc,GAAG,GAAG,CAAC,CAAC;QAC/E,OAAO,IAAI,CAAC,MAAM,IAAI,KAAK,CAAC;IAChC,CAAC;IAED,+DAA+D;IACvD,IAAI,CAAC,IAAY,EAAE,QAAuD;QAC9E,MAAM,QAAQ,GAAiB,EAAE,CAAC;QAClC,MAAM,YAAY,GAAG,IAAI,CAAC,YAAY,CAAC,QAAQ,CAAC,CAAC;QACjD,IAAI,MAAM,GAAG,CAAC,CAAC;QAEf,OAAO,MAAM,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC;YAC1B,MAAM,QAAQ,GAAG,IAAI,CAAC,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,CAAC,CAAC;YAC3D,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,QAAQ,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;YACrD,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAClB,QAAQ,CAAC,IAAI,CAAC,EAAE,QAAQ,EAAE,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,WAAW,EAAE,MAAM,EAAE,SAAS,EAAE,QAAQ,CAAC,GAAG,EAAE,CAAC,CAAC;YAClG,CAAC;YACD,IAAI,QAAQ,CAAC,GAAG,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;gBAC9B,MAAM;YACV,CAAC;YACD,0EAA0E;YAC1E,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,GAAG,GAAG,YAAY,EAAE,MAAM,GAAG,CAAC,CAAC,CAAC;QAC/D,CAAC;QACD,OAAO,QAAQ,CAAC;IACpB,CAAC;IAED;;;OAGG;IACK,YAAY,CAChB,IAAY,EACZ,MAAc,EACd,QAAuD;QAEvD,MAAM,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC,QAAQ,CAAC,CAAC;QAC1C,MAAM,OAAO,GAAG,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,GAAG,QAAQ,CAAC,iBAAiB,GAAG,GAAG,CAAC,CAAC,CAAC;QACrF,MAAM,OAAO,GAAG,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,GAAG,QAAQ,CAAC,gBAAgB,GAAG,GAAG,CAAC,CAAC,CAAC;QACpF,MAAM,OAAO,GAAG,MAAM,GAAG,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QAElD,IAAI,IAAI,CAAC,MAAM,IAAI,OAAO,EAAE,CAAC;YACzB,OAAO,EAAE,GAAG,EAAE,IAAI,CAAC,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;QAC7C,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,UAAU,EAAE,OAAO,EAAE,OAAO,CAAC,CAAC;QACrE,IAAI,SAAS,KAAK,IAAI,EAAE,CAAC;YACrB,OAAO,EAAE,GAAG,EAAE,SAAS,EAAE,IAAI,EAAE,WAAW,EAAE,CAAC;QACjD,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,kBAAkB,EAAE,OAAO,EAAE,OAAO,CAAC,CAAC;QAC5E,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC;YACpB,OAAO,EAAE,GAAG,EAAE,QAAQ,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC;QAC/C,CAAC;QACD,MAAM,IAAI,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC;QACnF,IAAI,IAAI,KAAK,IAAI,EAAE,CAAC;YAChB,OAAO,EAAE,GAAG,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;QACvC,CAAC;QACD,OAAO,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,IAAI,CAAC,MAAM,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;IACjE,CAAC;IAED;;;;OAIG;IACK,SAAS,CAAC,IAAY,EAAE,OAAe,EAAE,IAAY,EAAE,EAAU;QACrE,IAAI,EAAE,IAAI,IAAI,EAAE,CAAC;YACb,OAAO,IAAI,CAAC;QAChB,CAAC;QACD,MAAM,IAAI,GAAG,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC7C,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;QACnC,IAAI,KAAK,GAAkB,IAAI,CAAC;QAChC,IAAI,KAAK,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QAC5B,OAAO,KAAK,KAAK,IAAI,IAAI,KAAK,CAAC,KAAK,GAAG,EAAE,EAAE,CAAC;YACxC,KAAK,GAAG,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC;YACtC,KAAK,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QAC5B,CAAC;QACD,OAAO,KAAK,CAAC;IACjB,CAAC;IAED,gDAAgD;IAChD,WAAW;IACX,gDAAgD;IAEhD,4DAA4D;IACpD,sBAAsB,CAC1B,OAA6C;QAE7C,MAAM,IAAI,GAAG,IAAI,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;QAC1C,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,YAAY,IAAI,IAAI,CAAC,gBAAgB,EAAE,CAAC,CAAC,CAAC;QAC3E,OAAO;YACH,GAAG,IAAI;YACP,6EAA6E;YAC7E,4DAA4D;YAC5D,gBAAgB,EAAE,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,gBAAgB,EAAE,MAAM,CAAC;YACzD,YAAY,EAAE,MAAM;YACpB,iBAAiB,EAAE,IAAI,CAAC,YAAY,CAAC,OAAO,EAAE,iBAAiB,IAAI,EAAE,EAAE,CAAC,EAAE,EAAE,CAAC;YAC7E,gBAAgB,EAAE,IAAI,CAAC,YAAY,CAAC,OAAO,EAAE,gBAAgB,IAAI,EAAE,EAAE,CAAC,EAAE,GAAG,CAAC;YAC5E,cAAc,EAAE,IAAI,CAAC,YAAY,CAAC,OAAO,EAAE,cAAc,IAAI,EAAE,EAAE,CAAC,EAAE,GAAG,CAAC;SAC3E,CAAC;IACN,CAAC;IAED,oDAAoD;IAC5C,YAAY,CAAC,KAAa,EAAE,GAAW,EAAE,GAAW;QACxD,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC;YAC1B,OAAO,GAAG,CAAC;QACf,CAAC;QACD,OAAO,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC;IAC/C,CAAC;IAED,mFAAmF;IAC3E,WAAW,CAAC,QAAuD;QACvE,OAAO,QAAQ,CAAC,YAAY,GAAG,CAAC,CAAC;IACrC,CAAC;IAED,4CAA4C;IACpC,SAAS,CAAC,QAAuD;QACrE,OAAO,QAAQ,CAAC,gBAAgB,GAAG,CAAC,CAAC;IACzC,CAAC;IAED,mFAAmF;IAC3E,YAAY,CAAC,QAAuD;QACxE,OAAO,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,aAAa,GAAG,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,WAAW,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;IAC5F,CAAC;IAED,kFAAkF;IAC3E,cAAc,CAAC,IAAY;QAC9B,OAAO,WAAW,CAAC,kBAAkB,CAAC,IAAI,CAAC,CAAC;IAChD,CAAC;CACJ,CAAA;AAtJY,yBAAyB;IADrC,aAAa,CAAC,aAAa,EAAE,+BAA+B,CAAC;GACjD,yBAAyB,CAsJrC"}

package/dist/generic/BaseContentCleaner.d.ts

@@ -0,0 +1,101 @@
+/**
+ * @fileoverview Pluggable content cleaning — the stage that runs before segmentation.
+ *
+ * @module @memberjunction/ai-segmentation
+ */
+/** Options common to every cleaner. */
+export interface ContentCleaningOptions {
+    /**
+     * CSS selectors whose content is the ONLY content to keep. When set, everything
+     * outside these selectors is discarded before any exclusion is applied.
+     *
+     * This is the highest-leverage knob for messy sources: pointing at `.article-body`
+     * removes navigation, sidebars, and advertising in one stroke, without having to
+     * enumerate every block you *don't* want.
+     */
+    IncludeSelectors?: string[];
+    /**
+     * CSS selectors to remove. Applied after `IncludeSelectors`. Use for the blocks that
+     * survive the include (inline ad slots, share widgets, cookie banners).
+     */
+    ExcludeSelectors?: string[];
+    /** Collapse runs of whitespace and blank lines. Default: true. */
+    NormalizeWhitespace?: boolean;
+    /** Maximum characters to retain; content beyond this is dropped. Unset = no limit. */
+    MaxLength?: number;
+}
+/** Input to a cleaning pass. */
+export interface ContentCleaningParams<TOptions extends ContentCleaningOptions = ContentCleaningOptions> {
+    /** Raw content — markup or plain text. */
+    Content: string;
+    /** Mime type of `Content`, when known; lets a cleaner pick its parser. */
+    MimeType?: string;
+    /** Cleaner-specific options. */
+    Options?: TOptions;
+}
+/** Result of a cleaning pass. Never throws for content-shaped problems. */
+export interface ContentCleaningResult {
+    Success: boolean;
+    /** Cleaned text. Equal to the input when `Success` is false. */
+    Content: string;
+    /** Registration key of the cleaner that ran. */
+    CleanerKey: string;
+    ErrorMessage?: string;
+    Warnings: string[];
+    /** Characters removed by cleaning — a cheap signal that a selector is wrong. */
+    CharactersRemoved: number;
+}
+/**
+ * Base class for content cleaning strategies.
+ *
+ * Cleaning is deliberately a **separate stage from segmentation**, and a separate
+ * plug-in point. The two answer different questions — cleaning asks *which text is
+ * actually content*, segmentation asks *where that content divides* — and they change for
+ * different reasons: a new CMS template needs new selectors, not a new chunking strategy.
+ * Splitting them also means the cleaning rules apply once and benefit every downstream
+ * consumer (embedding chunks, tagging chunks, full-text indexing) instead of being
+ * reimplemented per pipeline.
+ *
+ * Garbage that survives this stage is expensive: it gets embedded, stored, retrieved, and
+ * eventually shown to a user or an agent. Navigation chrome repeated across a thousand
+ * pages produces a thousand near-identical vectors that crowd out real answers.
+ *
+ * ```typescript
+ * @RegisterClass(BaseContentCleaner, 'MyCleaner')
+ * export class MyCleaner extends BaseContentCleaner {
+ *     public get Key(): string { return 'MyCleaner'; }
+ *     protected CleanCore(params: ContentCleaningParams): string { return strip(params.Content); }
+ * }
+ * ```
+ */
+export declare abstract class BaseContentCleaner {
+    /** Registration key; must match the key passed to `@RegisterClass`. */
+    abstract get Key(): string;
+    /** Perform the cleaning. The base class handles validation, whitespace, and truncation. */
+    protected abstract CleanCore(params: ContentCleaningParams): string;
+    /**
+     * Clean content ahead of segmentation.
+     *
+     * Never throws for content-shaped problems — inspect `Success`/`ErrorMessage`. On
+     * failure the ORIGINAL content is returned rather than an empty string, so a bad
+     * selector degrades to "not cleaned" instead of silently discarding the document.
+     */
+    Clean(params: ContentCleaningParams): ContentCleaningResult;
+    /**
+     * Resolve a registered cleaner by key.
+     *
+     * Uses `TryCreateInstance` because `CreateInstance` never returns null for an unknown
+     * key — it silently yields a hollow base instance whose abstract members are undefined.
+     */
+    static Resolve(key: string): BaseContentCleaner | null;
+    /** Whitespace normalization and truncation, shared by every cleaner. */
+    protected applyCommonRules(content: string, options?: ContentCleaningOptions, warnings?: string[]): string;
+    /**
+     * Collapse horizontal whitespace and runs of blank lines, while preserving the single
+     * blank line that marks a paragraph break — segmenters rely on it as a boundary signal.
+     */
+    protected normalizeWhitespace(content: string): string;
+    /** Build a successful result with the removal delta computed. */
+    private buildResult;
+}
+//# sourceMappingURL=BaseContentCleaner.d.ts.map

package/dist/generic/BaseContentCleaner.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"BaseContentCleaner.d.ts","sourceRoot":"","sources":["../../src/generic/BaseContentCleaner.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAKH,uCAAuC;AACvC,MAAM,WAAW,sBAAsB;IACnC;;;;;;;OAOG;IACH,gBAAgB,CAAC,EAAE,MAAM,EAAE,CAAC;IAC5B;;;OAGG;IACH,gBAAgB,CAAC,EAAE,MAAM,EAAE,CAAC;IAC5B,kEAAkE;IAClE,mBAAmB,CAAC,EAAE,OAAO,CAAC;IAC9B,sFAAsF;IACtF,SAAS,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,gCAAgC;AAChC,MAAM,WAAW,qBAAqB,CAAC,QAAQ,SAAS,sBAAsB,GAAG,sBAAsB;IACnG,0CAA0C;IAC1C,OAAO,EAAE,MAAM,CAAC;IAChB,0EAA0E;IAC1E,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,gCAAgC;IAChC,OAAO,CAAC,EAAE,QAAQ,CAAC;CACtB;AAED,2EAA2E;AAC3E,MAAM,WAAW,qBAAqB;IAClC,OAAO,EAAE,OAAO,CAAC;IACjB,gEAAgE;IAChE,OAAO,EAAE,MAAM,CAAC;IAChB,gDAAgD;IAChD,UAAU,EAAE,MAAM,CAAC;IACnB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,gFAAgF;IAChF,iBAAiB,EAAE,MAAM,CAAC;CAC7B;AAED;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,8BAAsB,kBAAkB;IACpC,uEAAuE;IACvE,aAAoB,GAAG,IAAI,MAAM,CAAC;IAElC,2FAA2F;IAC3F,SAAS,CAAC,QAAQ,CAAC,SAAS,CAAC,MAAM,EAAE,qBAAqB,GAAG,MAAM;IAEnE;;;;;;OAMG;IACI,KAAK,CAAC,MAAM,EAAE,qBAAqB,GAAG,qBAAqB;IA6BlE;;;;;OAKG;WACW,OAAO,CAAC,GAAG,EAAE,MAAM,GAAG,kBAAkB,GAAG,IAAI;IAW7D,wEAAwE;IACxE,SAAS,CAAC,gBAAgB,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,sBAAsB,EAAE,QAAQ,CAAC,EAAE,MAAM,EAAE,GAAG,MAAM;IAY1G;;;OAGG;IACH,SAAS,CAAC,mBAAmB,CAAC,OAAO,EAAE,MAAM,GAAG,MAAM;IAQtD,iEAAiE;IACjE,OAAO,CAAC,WAAW;CAStB"}