flappa-doormal 2.14.1 → 2.15.0

package/AGENTS.md

@@ -604,6 +604,24 @@ bunx biome lint .
 
 48. **The "adding content changes behavior" smell**: If adding unrelated content (like a second page) dramatically changes how the first page is processed, suspect incorrect span/window calculations. The fix pattern: ensure window calculations are scoped to the CURRENT context (current page) not the ORIGINAL context (all remaining content).
 
+49. **Use `trimStart()` not `trim()` for user-provided patterns with semantic whitespace**: When processing user-provided patterns like the `words` field, only strip leading whitespace (likely accidental). Trailing whitespace may be intentional for whole-word matching (e.g., `'بل '` should match only the standalone word, not words starting with `بل` like `بلغ`). **Bug symptom**: `words: ['بل ']` matched `بلغ` because `.trim()` stripped the trailing space to just `بل`. **Fix**: Use `.trimStart()` to preserve trailing whitespace.
+
+50. **When expected boundaries exceed segment length, trust content matches**: If `expectedBoundary >= remainingContent.length`, any deviation-based validation is meaningless. In this case the boundary search must scan the full segment content and rank candidates without a distance constraint. Otherwise early valid page starts can be missed and page attribution will drift.
+
+51. **Infer start offset from the first boundary when necessary**: If the initial boundary search fails right after a structural split, rerun a relaxed search to find the true page start and infer `startOffsetInFromPage`. This corrects the baseline for all subsequent boundary estimates.
+
+52. **Windowed boundary searches can be wrong when offsets drift**: If the approximate offset is skewed (e.g., repeated line-start markers), a windowed scan may miss the real boundary. A full-content scan is required to recover early matches.
+
+53. **Harden maxPages=0 with targeted tests**: Add tests that hit the failure modes: segment starts at page boundary with repeated marker, very short pages (< 100 chars), minimal prefix lengths (15 chars), multiple candidate prefixes in content, tiny tail segments after structural splits, and fast-path threshold transitions (999 vs 1000 pages).
+
+54. **Beware trusting `segment.to` in validation**: When validating `maxPages` violations, do NOT rely solely on `segment.to` if it exists. A segment might claim to end on page X (via `segment.to`) but its content physically matches text on page Y. If `maxPages=0`, trusting `segment.to` hides the violation. Always check the physical match location (`actualToId`) against the constraint, regardless of what the segment claims.
+
+55. **Duplicate `case` labels in manual merges**: When applying fixes suggested by AI or manual merges, check surrounding code for duplicate `case` statements. JavaScript switch statements with duplicate cases are syntax errors (strict mode) or unreachable code. Validation errors usually catch this, but careful reading prevents it.
+
+56. **Linting vs Checks**: `bunx biome check` is strict. Complexity limits (max 15/18) force you to decompose functions. If you receive a complexity error, extract the complex logic (e.g., switch cases, specific validation checks) into standalone helper functions.
+
+57. **Validation Hints Specificity**: Generic error hints like "Check segmenter.ts" are unhelpful. Provide specific file names and logical components (e.g., "Check maxPages windowing in breakpoint-processor.ts"). User-friendly validation reports guide debugging much faster than "Something is wrong".
+
 ### Process Template (Multi-agent design review, TDD-first)
 
 If you want to repeat the “write a plan → get multiple AI critiques → synthesize → update plan → implement TDD-first” workflow, use:
@@ -746,6 +764,24 @@ See README.md for complete examples.
 
 ## Debugging Tips
 
+### Reading Validation Reports
+
+`validateSegments(pages, options, segments)` returns a structured report for attribution and `maxPages` issues. Use it to quickly localize bugs without re-running segmentation.
+
+**Key fields:**
+- `summary.errors` / `summary.warnings`: Use to decide if the bug is a hard failure or a suspected edge case.
+- `issues[].type`: The failure class (see below).
+- `issues[].segmentIndex`: Which segment to inspect in output.
+- `issues[].expected` / `issues[].actual`: Where attribution diverged.
+- `issues[].pageContext`: Matching page preview + `matchIndex` (if found).
+- `issues[].hint`: Direct pointer to likely culprit code path.
+
+**Issue types and next steps:**
+- `max_pages_violation`: Segment spans too many pages. Check breakpoint windowing in `breakpoint-processor.ts` and boundary logic in `breakpoint-utils.ts`.
+- `page_attribution_mismatch`: Content matched a different page than `segment.from`. Focus on `buildBoundaryPositions()` and `findPageStartNearExpectedBoundary()`.
+- `content_not_found`: Segment content not found in any page. Compare preprocessing, `pageJoiner`, and trimming behavior.
+- `page_not_found`: Segment `from` is not in input pages; validate page IDs and input ordering.
+
 ### Page Boundary Detection Issues
 
 If `maxPages=0` produces merged segments when pages have identical prefixes or duplicated content:
@@ -761,3 +797,8 @@ Key functions: `applyBreakpoints()` → `processOversizedSegment()` → `findBre
 - Check `boundaryPositions built` log for page boundary byte offsets
 - Check `iteration=N` logs for `currentFromIdx`, `cursorPos`, `windowEndPosition` per loop
 
+## Known Issues
+
+- **Binary Search Gap (Theoretical)**: `findBoundaryIdForOffset` returns `undefined` if the search offset falls exactly on a joiner character (e.g., a space or newline) between two pages. This is mathematically correct (the gap belongs to neither page) but may cause validation errors if a segment consists _only_ of such a gap or matches content starting/ending strictly within the gap. We have marked this as "accept" behavior for now, with a documented skipped test case.
+
+

package/README.md

@@ -92,6 +92,35 @@ const segments = segmentPages(pages, {
 // ]
 ```
 
+## Segment Validation
+
+Use `validateSegments()` to sanity-check segmentation output against the input pages and options. This is useful for detecting page attribution issues or maxPages violations before sending segments to downstream systems.
+
+```typescript
+import { segmentPages, validateSegments } from 'flappa-doormal';
+
+const segments = segmentPages(pages, { rules, maxPages: 0 });
+const report = validateSegments(pages, { rules, maxPages: 0 }, segments);
+
+if (!report.ok) {
+  console.log(report.summary);
+  console.log(report.issues[0]);
+}
+```
+
+Example issue entry (truncated):
+
+```json
+{
+  "type": "page_attribution_mismatch",
+  "severity": "error",
+  "segmentIndex": 2,
+  "expected": { "from": 5 },
+  "actual": { "from": 4 },
+  "evidence": "Content found in page 5, but segment.from=4."
+}
+```
+
 ## Features
 
 ### 1. Template Tokens
@@ -947,6 +976,57 @@ const segments = segmentPages(pages, { rules });
 // ]
 ```
 
+## Advanced: Metadata Extraction & Data Migration
+
+If you already have pre-segmented data (e.g., records from a database or JSON file) and want to use **flappa-doormal's** token system to extract metadata and clean the content without further splitting, you can use the **Metadata Extraction** pattern.
+
+By setting `maxPages: 0`, you guarantee a **1:1 mapping**: each input page produces exactly one output segment, regardless of how much text is on the page.
+
+### Example: Extracting multiple fields from pre-split records
+
+```typescript
+import { segmentPages, type Page } from 'flappa-doormal';
+
+const excerpts = [
+  { nass: '٧٠١٦ - ١ - ١ - فَقَصَّتْهَا حَفْصَةُ', id: 1 },
+  { nass: '٧٠١٧ (أ) - بَابُ الْقَيْدِ', id: 2 },
+  { nass: 'باب الصلاة - الفصل الأول', id: 3 },
+];
+
+// Convert your data to the Page format
+const pages: Page[] = excerpts.map(e => ({ content: e.nass, id: e.id }));
+
+const result = segmentPages(pages, {
+  maxPages: 0, // IMPORTANT: Guarantees each page stays isolated (no merging/splitting)
+  rules: [
+    // 1. Extract triple numbers: ٧٠١٦ - ١ - ١
+    {
+      lineStartsAfter: ['{{raqms:num}} {{dash}} {{raqms:num2}} {{dash}} {{raqms:num3}} '],
+    },
+    // 2. Extract number + indicator: ٧٠١٧ (أ)
+    {
+      lineStartsAfter: ['{{raqms:num}} ({{harf:indicator}}) {{dash}} '],
+    },
+    // 3. Mark chapters using fuzzy tokens
+    {
+      fuzzy: true,
+      lineStartsWith: ['{{bab}} '],
+      meta: { type: 'Chapter' },
+    },
+  ],
+});
+
+// Segment 0: { content: 'فَقَصَّتْهَا حَفْصَةُ', meta: { num: '٧٠١٦', num2: '١', num3: '١' }, ... }
+// Segment 1: { content: 'بَابُ الْقَيْدِ', meta: { num: '٧٠١٧', indicator: 'أ' }, ... }
+// Segment 2: { content: 'باب الصلاة - الفصل الأول', meta: { type: 'Chapter' }, ... }
+```
+
+### Why use this?
+- **Pattern Robustness**: Use `{{raqms}}`, `{{dash}}`, and `{{harf}}` instead of writing raw regex for every edge case.
+- **Prefix Cleaning**: `lineStartsAfter` automatically removes the matched pattern, leaving only the clean text.
+- **Deduplication**: Named captures like `{{raqms:num}}` automatically populate the `meta` object.
+- **Fuzzy Headers**: Use `fuzzy: true` to match headers like "Book" or "Chapter" regardless of Arabic diacritics.
+
 ## Rule Optimization
 
 Use `optimizeRules()` to automatically merge compatible rules, remove duplicate patterns, and sort rules by specificity (longest patterns first):
@@ -1335,6 +1415,38 @@ Notes:
 - Recovery is **explicitly scoped** by the `selector`; it will not “guess” which rules are mistaken.
 - If your segments were heavily post-processed (trimmed/normalized/reordered), recovery may return unresolved items; see the report for details.
 
+#### `recoverMistakenMarkersForRuns(runs, opts?)`
+
+Batch version of `recoverMistakenLineStartsAfterMarkers`. Processes multiple independent segmentation runs (e.g. from different books) and returns a consolidated report.
+
+```typescript
+import { recoverMistakenMarkersForRuns } from 'flappa-doormal';
+
+const results = recoverMistakenMarkersForRuns([
+  { pages: pages1, segments: segments1, options: options1, selector: selector1 },
+  { pages: pages2, segments: segments2, options: options2, selector: selector2 },
+]);
+```
+
+### `validateSegments(pages, options, segments, validationOptions?)`
+
+Validates that segments correctly map back to the source pages and adhere to constraints.
+
+```typescript
+import { validateSegments } from 'flappa-doormal';
+
+const report = validateSegments(pages, options, segments, {
+  // Optional: Max content length to search before falling back (default: 500)
+  // Segments longer than this are checked via fast path unless issues are found.
+  fullSearchThreshold: 1000, 
+});
+```
+
+Returns a `SegmentValidationReport` containing:
+- `ok`: boolean
+- `summary`: counts of errors/warnings
+- `issues`: detailed list of problems (page attribution mismatch, maxPages violation, etc.)
+
 ### `stripHtmlTags(html)`
 
 Remove all HTML tags from content, keeping only text.
@@ -1668,7 +1780,3 @@ bun run deploy
 ## License
 
 MIT
-
-## Inspiration
-
-The name of the project is from Asmāʾ, it seems to be some sort of gymnastic move.

package/dist/index.d.mts

@@ -150,8 +150,7 @@ type Breakpoint = string | BreakpointRule;
  * { regex: '^[٠-٩]+ - (.*)', split: 'at' }
  */
 type RegexPattern = {
-  /** Raw regex pattern string (no token expansion, no auto-escaping) */
-  regex: string;
+  /** Raw regex pattern string (no token expansion, no auto-escaping) */regex: string;
 };
 /**
  * Template pattern rule - expands `{{tokens}}` before compiling to regex.
@@ -177,8 +176,7 @@ type RegexPattern = {
  * @see TOKEN_PATTERNS for available tokens
  */
 type TemplatePattern = {
-  /** Template string with `{{token}}` or `{{token:name}}` placeholders. Brackets `()[]` are auto-escaped. */
-  template: string;
+  /** Template string with `{{token}}` or `{{token:name}}` placeholders. Brackets `()[]` are auto-escaped. */template: string;
 };
 /**
  * Line-start pattern rule - matches lines starting with any of the given patterns.
@@ -206,8 +204,7 @@ type TemplatePattern = {
  * { lineStartsWith: ['({{harf}}) '], split: 'at' }
  */
 type LineStartsWithPattern = {
-  /** Array of patterns that mark line beginnings (marker included in content). Brackets `()[]` are auto-escaped. */
-  lineStartsWith: string[];
+  /** Array of patterns that mark line beginnings (marker included in content). Brackets `()[]` are auto-escaped. */lineStartsWith: string[];
 };
 /**
  * Line-start-after pattern rule - matches lines starting with patterns,
@@ -238,8 +235,7 @@ type LineStartsWithPattern = {
  * { lineStartsAfter: ['({{harf}}): '], split: 'at' }
  */
 type LineStartsAfterPattern = {
-  /** Array of patterns that mark line beginnings (marker excluded from content). Brackets `()[]` are auto-escaped. */
-  lineStartsAfter: string[];
+  /** Array of patterns that mark line beginnings (marker excluded from content). Brackets `()[]` are auto-escaped. */lineStartsAfter: string[];
 };
 /**
  * Line-end pattern rule - matches lines ending with any of the given patterns.
@@ -261,8 +257,7 @@ type LineStartsAfterPattern = {
  * { lineEndsWith: ['(انتهى)'], split: 'after' }
  */
 type LineEndsWithPattern = {
-  /** Array of patterns that mark line endings. Brackets `()[]` are auto-escaped. */
-  lineEndsWith: string[];
+  /** Array of patterns that mark line endings. Brackets `()[]` are auto-escaped. */lineEndsWith: string[];
 };
 /**
  * Union of all pattern types for split rules.
@@ -593,9 +588,7 @@ type SegmentationOptions = {
    * `meta._flappa.rule` and/or `meta._flappa.breakpoint`.
    */
   debug?: boolean | {
-    /** Where to store provenance in meta. @default '_flappa' */
-    metaKey?: string;
-    /** Which kinds of provenance to include. @default ['rule','breakpoint'] */
+    /** Where to store provenance in meta. @default '_flappa' */metaKey?: string; /** Which kinds of provenance to include. @default ['rule','breakpoint'] */
     include?: Array<'rule' | 'breakpoint'>;
   };
   /**
@@ -734,6 +727,46 @@ type SegmentationOptions = {
   preprocess?: PreprocessTransform[];
 };
 //#endregion
+//#region src/types/validation.d.ts
+type SegmentValidationIssueSeverity = 'error' | 'warn';
+type SegmentValidationIssueType = 'max_pages_violation' | 'page_attribution_mismatch' | 'content_not_found' | 'page_not_found';
+type SegmentValidationIssue = {
+  type: SegmentValidationIssueType;
+  severity: SegmentValidationIssueSeverity;
+  segmentIndex: number;
+  segment: {
+    from: number;
+    to?: number;
+    contentPreview: string;
+  };
+  expected?: {
+    from?: number;
+    to?: number;
+  };
+  actual?: {
+    from?: number;
+    to?: number;
+  };
+  pageContext?: {
+    pageId: number;
+    pagePreview: string;
+    matchIndex?: number;
+  };
+  evidence?: string;
+  hint?: string;
+};
+type SegmentValidationReport = {
+  ok: boolean;
+  summary: {
+    segmentCount: number;
+    pageCount: number;
+    issues: number;
+    errors: number;
+    warnings: number;
+  };
+  issues: SegmentValidationIssue[];
+};
+//#endregion
 //#region src/types/index.d.ts
 /**
  * Output segment produced by `segmentPages()`.
@@ -935,13 +968,9 @@ declare const analyzeRepeatingSequences: (pages: Page[], options?: RepeatingSequ
  * Result of detecting a token pattern in text
  */
 type DetectedPattern = {
-  /** Token name from TOKEN_PATTERNS (e.g., 'raqms', 'dash') */
-  token: string;
-  /** The matched text */
-  match: string;
-  /** Start index in the original text */
-  index: number;
-  /** End index (exclusive) */
+  /** Token name from TOKEN_PATTERNS (e.g., 'raqms', 'dash') */token: string; /** The matched text */
+  match: string; /** Start index in the original text */
+  index: number; /** End index (exclusive) */
   endIndex: number;
 };
 /**
@@ -1004,9 +1033,7 @@ declare const analyzeTextForRule: (text: string) => {
  * Result from optimizing rules.
  */
 type OptimizeResult = {
-  /** Optimized rules (merged and sorted by specificity) */
-  rules: SplitRule[];
-  /** Number of rules that were merged into existing rules */
+  /** Optimized rules (merged and sorted by specificity) */rules: SplitRule[]; /** Number of rules that were merged into existing rules */
   mergedCount: number;
 };
 declare const optimizeRules: (rules: SplitRule[]) => {
@@ -1014,6 +1041,47 @@ declare const optimizeRules: (rules: SplitRule[]) => {
   rules: SplitRule[];
 };
 //#endregion
+//#region src/preprocessing/transforms.d.ts
+/**
+ * Remove zero-width control characters from text.
+ *
+ * @param text - Input text
+ * @param mode - 'strip' (default) removes entirely, 'space' replaces with space
+ * @returns Text with zero-width characters removed or replaced
+ */
+declare const removeZeroWidth: (text: string, mode?: "strip" | "space") => string;
+/**
+ * Condense multiple periods (...) into ellipsis character (…).
+ *
+ * Prevents `{{tarqim}}` from false-matching inside ellipsis since
+ * the `.` in tarqim matches individual periods.
+ *
+ * @param text - Input text
+ * @returns Text with period sequences replaced by ellipsis
+ */
+declare const condenseEllipsis: (text: string) => string;
+/**
+ * Join trailing و (waw) to the next word.
+ *
+ * Fixes OCR/digitization artifacts: ' و ' → ' و' (waw joined to next word)
+ *
+ * @param text - Input text
+ * @returns Text with trailing waw joined to following word
+ */
+declare const fixTrailingWaw: (text: string) => string;
+/**
+ * Apply preprocessing transforms to a page's content.
+ *
+ * Transforms run in array order. Each can be limited to specific pages
+ * via `min`/`max` constraints.
+ *
+ * @param content - Page content to transform
+ * @param pageId - Page ID for constraint checking
+ * @param transforms - Array of transforms to apply
+ * @returns Transformed content
+ */
+declare const applyPreprocessToPage: (content: string, pageId: number, transforms: PreprocessTransform[]) => string;
+//#endregion
 //#region src/recovery.d.ts
 type MarkerRecoverySelector = {
   type: 'rule_indices';
@@ -1114,10 +1182,8 @@ type ValidationIssueType = 'missing_braces' | 'unknown_token' | 'duplicate' | 'e
 type ValidationIssue = {
   type: ValidationIssueType;
   message: string;
-  suggestion?: string;
-  /** The token name involved in the issue (for unknown_token / missing_braces) */
-  token?: string;
-  /** The specific pattern involved (for duplicate) */
+  suggestion?: string; /** The token name involved in the issue (for unknown_token / missing_braces) */
+  token?: string; /** The specific pattern involved (for duplicate) */
   pattern?: string;
 };
 /**
@@ -1213,39 +1279,22 @@ declare const segmentPages: (pages: Page[], options: SegmentationOptions) => Seg
 //#region src/segmentation/tokens.d.ts
 /** Pre-defined token constants for use in patterns. */
 declare const Token: {
-  /** Chapter marker - باب */
-  readonly BAB: "{{bab}}";
-  /** Basmala - بسم الله */
-  readonly BASMALAH: "{{basmalah}}";
-  /** Bullet point variants */
-  readonly BULLET: "{{bullet}}";
-  /** Dash variants (hyphen, en-dash, em-dash, tatweel) */
-  readonly DASH: "{{dash}}";
-  /** Section marker - فصل / مسألة */
-  readonly FASL: "{{fasl}}";
-  /** Single Arabic letter */
-  readonly HARF: "{{harf}}";
-  /** Multiple Arabic letters separated by spaces */
-  readonly HARFS: "{{harfs}}";
-  /** Book marker - كتاب */
-  readonly KITAB: "{{kitab}}";
-  /** Hadith transmission phrases */
-  readonly NAQL: "{{naql}}";
-  /** Newline character (for breakpoints) */
-  readonly NEWLINE: "{{newline}}";
-  /** Single ASCII digit */
-  readonly NUM: "{{num}}";
-  /** Composite: {{raqms}} {{dash}} (space) */
-  readonly NUMBERED: "{{numbered}}";
-  /** One or more ASCII digits */
-  readonly NUMS: "{{nums}}";
-  /** Single Arabic-Indic digit */
-  readonly RAQM: "{{raqm}}";
-  /** One or more Arabic-Indic digits */
-  readonly RAQMS: "{{raqms}}";
-  /** Source abbreviations (rijāl/takhrīj) */
-  readonly RUMUZ: "{{rumuz}}";
-  /** Punctuation marks */
+  /** Chapter marker - باب */readonly BAB: "{{bab}}"; /** Basmala - بسم الله */
+  readonly BASMALAH: "{{basmalah}}"; /** Bullet point variants */
+  readonly BULLET: "{{bullet}}"; /** Dash variants (hyphen, en-dash, em-dash, tatweel) */
+  readonly DASH: "{{dash}}"; /** Section marker - فصل / مسألة */
+  readonly FASL: "{{fasl}}"; /** Single Arabic letter */
+  readonly HARF: "{{harf}}"; /** Multiple Arabic letters separated by spaces */
+  readonly HARFS: "{{harfs}}"; /** Book marker - كتاب */
+  readonly KITAB: "{{kitab}}"; /** Hadith transmission phrases */
+  readonly NAQL: "{{naql}}"; /** Newline character (for breakpoints) */
+  readonly NEWLINE: "{{newline}}"; /** Single ASCII digit */
+  readonly NUM: "{{num}}"; /** Composite: {{raqms}} {{dash}} (space) */
+  readonly NUMBERED: "{{numbered}}"; /** One or more ASCII digits */
+  readonly NUMS: "{{nums}}"; /** Single Arabic-Indic digit */
+  readonly RAQM: "{{raqm}}"; /** One or more Arabic-Indic digits */
+  readonly RAQMS: "{{raqms}}"; /** Source abbreviations (rijāl/takhrīj) */
+  readonly RUMUZ: "{{rumuz}}"; /** Punctuation marks */
   readonly TARQIM: "{{tarqim}}";
 };
 /**
@@ -1493,47 +1542,6 @@ declare const applyTokenMappings: (template: string, mappings: TokenMapping[]) =
  */
 declare const stripTokenMappings: (template: string) => string;
 //#endregion
-//#region src/preprocessing/transforms.d.ts
-/**
- * Remove zero-width control characters from text.
- *
- * @param text - Input text
- * @param mode - 'strip' (default) removes entirely, 'space' replaces with space
- * @returns Text with zero-width characters removed or replaced
- */
-declare const removeZeroWidth: (text: string, mode?: "strip" | "space") => string;
-/**
- * Condense multiple periods (...) into ellipsis character (…).
- *
- * Prevents `{{tarqim}}` from false-matching inside ellipsis since
- * the `.` in tarqim matches individual periods.
- *
- * @param text - Input text
- * @returns Text with period sequences replaced by ellipsis
- */
-declare const condenseEllipsis: (text: string) => string;
-/**
- * Join trailing و (waw) to the next word.
- *
- * Fixes OCR/digitization artifacts: ' و ' → ' و' (waw joined to next word)
- *
- * @param text - Input text
- * @returns Text with trailing waw joined to following word
- */
-declare const fixTrailingWaw: (text: string) => string;
-/**
- * Apply preprocessing transforms to a page's content.
- *
- * Transforms run in array order. Each can be limited to specific pages
- * via `min`/`max` constraints.
- *
- * @param content - Page content to transform
- * @param pageId - Page ID for constraint checking
- * @param transforms - Array of transforms to apply
- * @returns Transformed content
- */
-declare const applyPreprocessToPage: (content: string, pageId: number, transforms: PreprocessTransform[]) => string;
-//#endregion
 //#region src/utils/textUtils.d.ts
 /**
  * Escapes regex metacharacters (parentheses and brackets) in template patterns,
@@ -1575,5 +1583,30 @@ declare const escapeTemplateBrackets: (pattern: string) => string;
 declare const escapeRegex: (s: string) => string;
 declare const makeDiacriticInsensitive: (text: string) => string;
 //#endregion
-export { type Breakpoint, type BreakpointRule, type CommonLineStartPattern, type CondenseEllipsisRule, type DetectedPattern, type ExpandResult, type FixTrailingWawRule, type LineStartAnalysisOptions, type LineStartPatternExample, type Logger, type MarkerRecoveryReport, type MarkerRecoveryRun, type MarkerRecoverySelector, type OptimizeResult, PATTERN_TYPE_KEYS, type Page, type PageRange, type PageRangeConstraint, type PageRangeConstraintWithExclude, type PatternProcessor, type PatternTypeKey, type PreprocessTransform, type RemoveZeroWidthRule, type RepeatingSequenceExample, type RepeatingSequenceOptions, type RepeatingSequencePattern, type RuleValidationResult, type Segment, type SegmentationOptions, type SplitRule, TOKEN_PATTERNS, Token, type TokenKey, type TokenMapping, type ValidationIssue, type ValidationIssueType, analyzeCommonLineStarts, analyzeRepeatingSequences, analyzeTextForRule, applyPreprocessToPage, applyTokenMappings, condenseEllipsis, containsTokens, detectTokenPatterns, escapeRegex, escapeTemplateBrackets, escapeWordsOutsideTokens, expandCompositeTokensInTemplate, expandTokens, expandTokensWithCaptures, fixTrailingWaw, formatValidationReport, generateTemplateFromText, getAvailableTokens, getTokenPattern, makeDiacriticInsensitive, optimizeRules, recoverMistakenLineStartsAfterMarkers, recoverMistakenMarkersForRuns, removeZeroWidth, segmentPages, shouldDefaultToFuzzy, stripTokenMappings, suggestPatternConfig, templateToRegex, validateRules, withCapture };
+//#region src/validation/validate-segments.d.ts
+type ValidationOptions = {
+  /**
+   * Threshold for short segment content (characters).
+   * Segments shorter than this will trigger a full-document search fallback.
+   * @default 500
+   */
+  fullSearchThreshold?: number;
+};
+/**
+ * Validates a list of segments against the source pages.
+ * checks for:
+ * - Page existence (invalid IDs)
+ * - Content fidelity (content must exist in pages)
+ * - Page attribution (from/to must match content location)
+ * - Page constraints (maxPages violations)
+ *
+ * @param pages Input pages used for segmentation
+ * @param options Operations used during segmentation (for preprocessing/joining consistency)
+ * @param segments The output segments to validate
+ * @param validationOptions Optional settings for validation behavior
+ * @returns A detailed validation report
+ */
+declare const validateSegments: (pages: Page[], options: SegmentationOptions, segments: Segment[], validationOptions?: ValidationOptions) => SegmentValidationReport;
+//#endregion
+export { type Breakpoint, type BreakpointRule, type CommonLineStartPattern, type CondenseEllipsisRule, type DetectedPattern, type ExpandResult, type FixTrailingWawRule, type LineStartAnalysisOptions, type LineStartPatternExample, type Logger, type MarkerRecoveryReport, type MarkerRecoveryRun, type MarkerRecoverySelector, type OptimizeResult, PATTERN_TYPE_KEYS, type Page, type PageRange, type PageRangeConstraint, type PageRangeConstraintWithExclude, type PatternProcessor, type PatternTypeKey, type PreprocessTransform, type RemoveZeroWidthRule, type RepeatingSequenceExample, type RepeatingSequenceOptions, type RepeatingSequencePattern, type RuleValidationResult, type Segment, type SegmentValidationIssue, type SegmentValidationIssueSeverity, type SegmentValidationIssueType, type SegmentValidationReport, type SegmentationOptions, type SplitRule, TOKEN_PATTERNS, Token, type TokenKey, type TokenMapping, type ValidationIssue, type ValidationIssueType, type ValidationOptions, analyzeCommonLineStarts, analyzeRepeatingSequences, analyzeTextForRule, applyPreprocessToPage, applyTokenMappings, condenseEllipsis, containsTokens, detectTokenPatterns, escapeRegex, escapeTemplateBrackets, escapeWordsOutsideTokens, expandCompositeTokensInTemplate, expandTokens, expandTokensWithCaptures, fixTrailingWaw, formatValidationReport, generateTemplateFromText, getAvailableTokens, getTokenPattern, makeDiacriticInsensitive, optimizeRules, recoverMistakenLineStartsAfterMarkers, recoverMistakenMarkersForRuns, removeZeroWidth, segmentPages, shouldDefaultToFuzzy, stripTokenMappings, suggestPatternConfig, templateToRegex, validateRules, validateSegments, withCapture };
 //# sourceMappingURL=index.d.mts.map

package/dist/index.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.mts","names":[],"sources":["../src/types/breakpoints.ts","../src/types/rules.ts","../src/types/options.ts","../src/types/index.ts","../src/analysis/line-starts.ts","../src/analysis/repeating-sequences.ts","../src/detection.ts","../src/optimization/optimize-rules.ts","../src/recovery.ts","../src/segmentation/breakpoint-utils.ts","../src/segmentation/pattern-validator.ts","../src/segmentation/segmenter.ts","../src/segmentation/tokens.ts","../src/preprocessing/transforms.ts","../src/utils/textUtils.ts"],"sourcesContent":[],"mappings":";AAqBA;AA2GA;;;;AChIiE;AA4BhD;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAqCxB;AAOA;AAUK,KDhLO,cAAA,GAAiB,8BCgLX,GAAA;EA2Cb;AAiEL;;;;;;;;ACvRA;AAgCA;EAyBY,OAAA,CAAA,EAAA,MAAA;EAsBA;;;;;AAgCZ;AA6CA;;;;;;;;;;ACxKA;AA4CA;AA2BA;AAgBA;AAwBA;;;;AC1GA;AAcA;AAEA;AA4PA;;;;;;;;ACtQA;AAaA;AAOA;AAiOA;;;;;;;;ACzQA;AA+EA;AAgEA;AAuBA;AAiCA;;;;ECpMY,KAAA,CAAA,EAAA,IAAA,GAAA,OAAc;EAsCb;;;;ACvCb;AAKA;;;;;;AAOA;AA2BE;AAknBF;;;;;EAO2B,QAAA,CAAA,EAAA,MAAA;CAEd;;;AA6Cb;;;;;;;;;ACrrBA;AA0KA;;KT3EY,UAAA,YAAsB;;;AA3GlC;AA2GA;;;;AChIiE;AA4BhD;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAqCxB;AAOA;AAAgE;AAU9C;AA4GlB;;;;;KArRK,YAAA;;;ACFL,CAAA;AAgCA;AAyBA;AAsBA;;;;;AAgCA;AA6CA;;;;;;;;;;ACxKA;AA4CA;AA2BA;AAgBA;AAwBA;KFrEK,eAAA;;;AGrCL,CAAA;AAcA;AAEA;AA4PA;;;;;;;;ACtQA;AAaA;AAOA;AAiOA;;;;;;;;ACzQA;AA+EA;AAgEA;AAuBA;AAiCA,KLtHK,qBAAA,GKyIJ;;;;ACvND;AAsCA;;;;ACvCA;AAKA;;;;;;AAOA;AA2BE;AAknBF;;;;;;;;;AAsDA;;;;;KP/lBK,sBAAA,GOkmB+C;;;;ACxrBpD;AA0KA;;;;AC9LA;AAKA;AAcA;;;;;;AA2FA;AAsCA;;;;ACiJA;KVnKK,mBAAA,GUmK+B;EAAiB;EAAmB,YAAA,EAAA,MAAA,EAAA;CAAA;;;;ACxMxE;AAwCA;AAGA;AAiBA;AAgDA;AA2CA;AAWA;AAkJA,KXhQK,WAAA,GACC,YWwRL,GXvRK,eWuRL,GXtRK,qBWsRL,GXrRK,sBWqRL,GXpRK,mBWoRL;AAqBD;AAuBA;AAqBA;AAgBA;AA8BA;AAWA;AAoBA;AA6BA;;;;AClkBA;AAgCA;AAUA;AA2CA;cZ+Da;;;Aa/Ib;AAsDA;AAeA;KbiFY,cAAA,WAAyB;;;;;;;KAUhC,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA2CA,eAAA,GAAkB;;;;;;;;;;;;SAYZ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqDC,SAAA,GAAY,cAAc,gBAAgB;;;;;;AAjTW;AA4BhD;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAqCxB;AAOA;AAAgE;AAU9C;AA4GlB;;;AAAsD,KCvR1C,mBAAA,GAAsB,mBDuRoB,GAAA;EAAe,IAAA,EAAA,iBAAA;;;;ACvRrE;AAgCA;AAyBA;AAsBA;EAIM,IAAA,CAAA,EAAA,OAAA,GAAA,OAAA;CACA;;;AA2BN;AA6CA;;;;;;;;;;ACxKA;AA4CA;AA2BA;AAgBA;AAwBA;;;KDnEY,oBAAA,GAAuB;EEvCvB,IAAA,EAAA,kBAAA;AAcZ,CAAA;AAEA;AA4PA;;;;;;;;ACtQA;AAaA;AAOA;AAiOA;;;;;;;;ACzQA;AA+Ea,KJDD,kBAAA,GAAqB,mBICe,GAAA;EAgEnC,IAAA,EAAA,gBAAA;AAuBb,CAAA;AAiCA;;;;ACpMA;AAsCA;;;;ACvCA;AAKA;;;;;;AAOA;AA2BE;AAknBc,KNvjBJ,mBAAA,GMujBI,iBAAqC,GAAA,kBAAA,GAAA,gBAAA,GNnjB/C,mBMmjB+C,GNljB/C,oBMkjB+C,GNjjB/C,kBMijB+C;;;;;;;;;AAsDrD;;;;;;;;;ACrrBA;AA0KA;;;;AC9LA;AAKA;AAcY,URyGK,MAAA,CQzGL;EACU;EACC,KAAA,CAAA,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,IAAA,EAAA,OAAA,EAAA,EAAA,GAAA,IAAA;EACH;EACL,KAAA,CAAA,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,IAAA,EAAA,OAAA,EAAA,EAAA,GAAA,IAAA;EAAe;EAuFjB,IAAA,CAAA,EAAA,CAAA,OAAA,EAwBP,MAAA,EAAA,GAxB+B,IAAA,EAAA,OAAW,EAAA,EAAA,GAAA,IAAA;EAsCnC;;;;ACiJb;;;;;;;;ACxMA;AAwCA;AAGA;AAiBA;AAgDA;AA2CA;AAWA;AAkJA;AA8CA;AAuBA;AAqBA;AAgBA;AA8BA;AAWA;AAoBA;AA6BA;;;;AClkBA;AAgCA;AAUA;AA2CA;;;KXiEY,mBAAA;EYjJC;AAsDb;AAeA;;;;;UZoFY;;;;;;;;;;;;;;cAiBY;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA4DN;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;WAwDL;;;;;;;;;;;;;;;;;;;;;;;;;eA0BI;;;;;AFxUjB;AA2GA;;;;AChIiE;AA4BhD;AA4BG;AA8BM;AAiCC;AAwBH;;;AAkBlB,KEnJM,OAAA,GFmJN;EACA;;;AAkBN;AAOA;AAAgE;EAqD3D,OAAA,EAAA,MAAA;EAiEO;;;EAA0C,IAAA,EAAA,MAAA;EAAe;;;;ACvRrE;AAgCA;EAyBY,EAAA,CAAA,EAAA,MAAA;EAsBA;;;;;AAgCZ;AA6CA;EAQY,IAAA,CAAA,ECnJD,MDmJC,CAAA,MAAA,EAAA,OAAA,CAAA;CAiBY;;;;;;;;ACjMxB;AA4CA;AA2BA;AAgBA;AAwBA;KAnEY,IAAA;;;ACvCZ;AAcA;AAEA;EA4Pa,EAAA,EAAA,MAAA;EACF;;;;;;;ACvQX,CAAA;AAaA;AAOA;AAiOA;;;;;;;;ACzQY,KHgFA,SAAA,GGhFe,MAAA,GAAA,CAAA,MAAA,EAAA,MAAA,CAAA;AA+E3B;AAgEA;AAuBA;AAiCA;;;;ACpMA;AAsCA;;;;ACvCA;AAKA;AACa,KLwFD,mBAAA,GKxFC;EACF;;;;EAKC,GAAA,CAAA,EAAA,MAAA;EA6BP;AAgnBL;;;EAGa,GAAA,CAAA,EAAA,MAAA;CACC;;;;;AAkDd;;;;;;KLzlBY,8BAAA,GAAiC;;;AM5F7C;AA0KA;;;;AC9LA;AAKA;AAcA;;;;;;EA2Fa,OAAA,CAAA,EPkBC,SOMR,EAAA;AAcN,CAAA;;;AV5IY,KIFA,wBAAA,GJEiB;EA2GjB,IAAA,CAAA,EAAA,MAAU;;;;ECpGjB,WAAA,CAAA,EAAA,MAAY;EA4BZ,wBAAe,CAAA,EAAA,OAAA;EA8Bf,yBAAqB,CAAA,EAAA,OAAA;EAiCrB,MAAA,CAAA,EAAA,aAAA,GAAsB,OAAA;EAwBtB,UAAA,CAAA,EAAA,CAAA,IAAA,EAAA,MAAmB,EAAA,MAAA,EAAA,MAAA,EAAA,GAAA,OAAA;EAenB,cAAW,CAAA,EGjIK,MHiIL,EAAA;EACV,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;CACA;AACA,KGhIM,uBAAA,GHgIN;EACA,IAAA,EAAA,MAAA;EACA,MAAA,EAAA,MAAA;CAAmB;AAiBZ,KGjJD,sBAAA,GHiJwG;EAOxG,OAAA,EAAA,MAAA;EAUP,KAAA,EAAA,MAAA;EA2CA,QAAA,EG1MS,uBH0MS,EAAA;AAiEvB,CAAA;;;;AAAqE,cGlBxD,uBHkBwD,EAAA,CAAA,KAAA,EGjB1D,IHiB0D,EAAA,EAAA,OAAA,CAAA,EGhBxD,wBHgBwD,EAAA,GGflE,sBHekE,EAAA;;;AAlKhE,KItHO,wBAAA,GJsHY;EAenB,WAAA,CAAA,EAAW,MAAA;EACV,WAAA,CAAA,EAAA,MAAA;EACA,QAAA,CAAA,EAAA,MAAA;EACA,IAAA,CAAA,EAAA,MAAA;EACA,yBAAA,CAAA,EAAA,OAAA;EACA,YAAA,CAAA,EAAA,OAAA;EAAmB,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;EAiBZ,WAAA,CAAA,EAAA,MAAuG;EAOxG,YAAA,CAAA,EAAA,MAAc;EAUrB,iBAAa,CAAA,EAAA,MAAA;AAAA,CAAA;AA4GN,KI3QA,wBAAA,GJ2QS;EAAG,IAAA,EAAA,MAAA;EAAc,OAAA,EAAA,MAAA;EAAgB,MAAA,EAAA,MAAA;EAAe,YAAA,EAAA,MAAA,EAAA;;KIpQzD,wBAAA;;EHnBA,KAAA,EAAA,MAAA;EAgCA,QAAA,EGVE,wBHUkB,EAAG;AAyBnC,CAAA;;;;AAsDA;AA6CA;;AAyBwB,cG+DX,yBH/DW,EAAA,CAAA,KAAA,EGgEb,IHhEa,EAAA,EAAA,OAAA,CAAA,EGiEV,wBHjEU,EAAA,GGkErB,wBHlEqB,EAAA;;;;AF1LxB;AA2GA;KM3HY,eAAA;;;ELuBP;EA4BA,KAAA,EAAA,MAAA;EA8BA;EAiCA,KAAA,EAAA,MAAA;EAwBA;EAeA,QAAA,EAAA,MAAW;CACV;;;;;;AAqBN;AAOA;AAAgE;AAU9C;AA4GlB;;;;;;cK7Na,uCAAmC;;AJ1DhD;AAgCA;AAyBA;AAsBA;;;;;AAgCA;AA6CA;;;AAqFkB,cIvHL,wBJuHK,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,QAAA,EIvH+C,eJuH/C,EAAA,EAAA,GAAA,MAAA;;;;;;;AC7PN,cG6JC,oBHhII,EAAA,CAAA,QAAA,EGiIH,eHjIG,EAAA,EAAA,GAAA;EAeL,WAAI,EAAA,gBAAA,GAAA,iBAAA;EA2BJ,KAAA,EAAA,OAAS;EAgBT,QAAA,CAAA,EAAA,MAAA;AAwBZ,CAAA;;;;AC1GA;AAcA;AAEA;AA4Pa,cEnFA,kBFsGZ,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA;EAlBU,QAAA,EAAA,MAAA;EACE,WAAA,EAAA,gBAAA,GAAA,iBAAA;EACV,KAAA,EAAA,OAAA;EAAsB,QAAA,CAAA,EAAA,MAAA;YE/EX;;;;AN9Ld;AA2GA;;KOxHY,cAAA;;ENoBP,KAAA,EMlBM,SNkBM,EAAA;EA4BZ;EA8BA,WAAA,EAAA,MAAA;AAAqB,CAAA;AAyDrB,cMjGQ,aNiGW,EAAA,CAAA,KAAA,EMjGa,SNiGb,EAAA,EAAA,GAAA;EAenB,WAAA,EAAA,MAAW;EACV,KAAA,EMjH+B,SNiH/B,EAAA;CACA;;;KOzJM,sBAAA;;;APPqD,CAAA,GA4B5D;EA4BA,IAAA,EAAA,0BAAe;EA8Bf,KAAA,CAAA,EAAA,OAAA,GAAA,YAAqB;EAiCrB,QAAA,EAAA,MAAA,EAAA;AAAsB,CAAA,GAwBtB;EAeA,IAAA,EAAA,WAAW;EACV,SAAA,EAAA,CAAA,IAAA,EOrJuC,SPqJvC,EAAA,KAAA,EAAA,MAAA,EAAA,GAAA,OAAA;CACA;AACA,KOrJM,iBAAA,GPqJN;EACA,OAAA,EOrJO,mBPqJP;EACA,KAAA,EOrJK,IPqJL,EAAA;EAAmB,QAAA,EOpJX,OPoJW,EAAA;EAiBZ,QAAA,EOpKC,sBPoKsG;AAOpH,CAAA;AAUK,KOlLO,oBAAA,GPkLM;EA2Cb,OAAA,EAAA;IAiEO,IAAA,EAAA,YAAS,GAAA,wBAAA;IAAG,SAAA,EAAA,MAAA;IAAc,aAAA,EAAA,MAAA;IAAgB,SAAA,EAAA,MAAA;IAAe,UAAA,EAAA,MAAA;;UOtRzD;;INDA,QAAA,EAAA,MAAA;IAgCA,aAAA,EAAA,MAAoB;IAyBpB,UAAA,EAAA,MAAkB;EAsBlB,CAAA,CAAA;EAIN,OAAA,EM5EO,KN4EP,CAAA;IACA,IAAA,EAAA,MAAA;IACA,oBAAA,EAAA,MAAA;IAAkB,sBAAA,CAAA,EAAA,MAAA;IA0BP,qBAAM,CAAA,EAAA,MAAA;IA6CX,YAAA,EAAA,MAAmB;IAQnB,MAAA,EAAA,WAAA,GAAA,oBAAA,GAAA,WAAA,GAAA,sBAAA,GAAA,qBAAA;IAiBY,QAAA,EAAA,OAAA,GAAA,QAAA,GAAA,MAAA;IA4DN,EAAA,CAAA,EAAA,MAAA;IAwDL,KAAA,CAAA,EAAA,MAAA,EAAA;EA0BI,CAAA,CAAA;EAAmB,MAAA,EAAA,MAAA,EAAA;;;KM7S/B,oBAAA;ALlCO,iBKkpBI,qCAAA,CLrnBC,KAAA,EKsnBN,ILtnBM,EAAA,EAAA,QAAA,EKunBH,OLvnBG,EAAA,EAAA,OAAA,EKwnBJ,mBLxnBI,EAAA,QAAA,EKynBH,sBLznBG,EAAA,IA0DjB,CA1DiB,EAAA;EAeL,IAAA,CAAA,EAAI,YAAA,GAAA,wBAAA;EA2BJ,gBAAS,CAAA,EKklBM,oBLllBN;AAgBrB,CAAA,CAAA,EAAY;EAwBA,MAAA,EK4iBC,oBL5iBD;YK4iBiC;;iBA6C7B,6BAAA,OACN,yBJprBV;EAhBY,IAAA,CAAA,EAAA,YAAA,GAAA,wBAUS;EAIT,gBAAA,CAAA,EIurBoE,oBJvrB7C;AAEnC,CAAA,CAAA,EAAY;EA4PC,MAAA,EI0bA,oBJvaZ;EAlBU,QAAA,EIybkC,OJzblC,EAAA;CACE;;;;;;AHjSoD;AA4BhD;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAqCxB;AAOA;AAAgE;AAU9C;AA4GlB;;;AAAsD,cQhRzC,wBRgRyC,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;AS7QxB,KDuKlB,gBAAA,GCvKkB,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;ATRb;AA4BG;AA8BM;AAyDrB,KSlIO,mBAAA,GTkIY,gBAAA,GAAA,eAAA,GAAA,WAAA,GAAA,eAAA;AAAA;;;AAkBlB,KS/IM,eAAA,GT+IN;EACA,IAAA,ES/II,mBT+IJ;EACA,OAAA,EAAA,MAAA;EAAmB,UAAA,CAAA,EAAA,MAAA;EAiBZ;EAOD,KAAA,CAAA,EAAA,MAAA;EAUP;EA2CA,OAAA,CAAA,EAAA,MAAA;AAiEL,CAAA;;;;;KSjRY,oBAAA;oBACU;qBACC;ERRX,YAAA,CAAA,EAAA,CQSQ,eRTW,GAAG,SAAA,CAAA,EAAA;EAgCtB,QAAA,CAAA,EQtBG,eRsBiB;AAyBhC,CAAA;AAsBA;;;;;AAgCA;AA6CA;;;;;;;;;;ACxKA;AA4CA;AA2BA;AAgBY,cOsBC,aPtBkB,EAAA,CAAA,KAAA,EOsBM,SPtBN,EAAA,EAAA,GAAA,COsBiB,oBPtBjB,GAAA,SAAA,CAAA,EAAA;AAwB/B;;;;AC1GA;AAcA;AAEA;AA4PA;;;;;;cM9Ha,mCAAoC;;;;;ATmBjD;AAOA;AAAgE;AAU9C;AA4GlB;;;;;;;;ACvRA;AAgCA;AAyBA;AAsBA;;;;;AAgCA;AA6CA;;;;;;;;;;ACxKA;AA4CA;AA2BA;AAgBA;AAwBA;;;;AC1GA;AAcY,cOiRC,YPjRsB,EAAA,CAAA,KAAA,EOiRC,IPjRD,EAAA,EAAA,OAAA,EOiRkB,mBPjRlB,EAAA,GOiRqC,OPjRrC,EAAA;;;;AJZvB,cYqFC,KZrFa,EAAA;EA2Gd;;;;ECpGP;EA4BA,SAAA,MAAA,EAAA,YAAe;EA8Bf;EAiCA,SAAA,IAAA,EAAA,UAAsB;EAwBtB;EAeA,SAAA,IAAA,EAAW,UAAA;EACV;EACA,SAAA,IAAA,EAAA,UAAA;EACA;EACA,SAAA,KAAA,EAAA,WAAA;EACA;EAAmB,SAAA,KAAA,EAAA,WAAA;EAiBZ;EAOD,SAAA,IAAA,EAAA,UAAc;EAUrB;EA2CA,SAAA,OAAA,EAAe,aAAG;EAiEX;EAAY,SAAA,GAAA,EAAA,SAAA;EAAc;EAAgB,SAAA,QAAA,EAAA,cAAA;EAAe;;;;ECvRzD;EAgCA,SAAA,KAAA,EAAA,WAAoB;EAyBpB;EAsBA,SAAA,KAAA,EAAA,WAAmB;EAIzB;EACA,SAAA,MAAA,EAAA,YAAA;CACA;;AA0BN;AA6CA;AAQY,KU5CA,QAAA,GV4CA,MAAA,OU5CwB,KV4CxB;;AA6EM,cUtHL,WVsHK,EAAA,CAAA,KAAA,EAAA,MAAA,EAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;AAkFD,cUvLJ,+BVuLI,EAAA,CAAA,QAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;AC/UjB;AA4CA;AA2BA;AAgBA;AAwBA;;;;AC1GA;AAcA;AAEA;AA4PA;;;;;;;;ACtQA;AAaA;AAOY,cOyKC,cPzKuB,EOyKP,MPtKf,CAAA,MAAA,EAAA,MAAA,CAAA;AA8Nd;;;;;;;;ACzQA;AA+EA;AAgEA;AAuBA;AAiCA;;;cMqDa;ALzPb;AAsCA;;;;ACvCA;AAKY,KIgQA,YAAA,GJhQiB;EAChB;;;;;EAMD,OAAA,EAAA,MAAA;EA6BP;AAgnBL;;;;EAIc,YAAA,EAAA,MAAA,EAAA;EAGa;;;;AA+C3B;EACU,WAAA,EAAA,OAAA;CACsE;;;;;;;ACvrBhF;AA0KA;;;;AC9LA;AAKA;AAcA;;;;;;AA2FA;AAsCA;;;;ACiJA;;;;;;;;ACxMA;AAwCA;AAGA;AAiBA;AAgDa,cAwMA,wBAxMsB,EAAA,CAAA,KAAA,EAAA,MAAA,EAAA,cAAA,CAAA,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA,EAAA,aAAA,CAAA,EAAA,MAAA,EAAA,GAAA;EA2CtB,YAAA,EAAA,MAGZ,EAAA;EAQW,WAAA,EAAA,OAAY;EAkJX,OAAA,EAAA,MAAA;AA8Cb,CAAA;AAuBA;AAqBA;AAgBA;AA8BA;AAWA;AAoBA;AA6BA;;;;AClkBA;AAgCA;AAUA;AA2CA;;;;AChFA;AAsDA;AAea,cFkWA,YE7VZ,EAAA,CAAA,KAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;;;;;;;;;;;;;;;;;;cFoXY,uCAAmC;;;;;;;;;;;;;cAqBnC;;;;;;;;;;;;;;;cAgBA;;;;;;;;;;;;;;;cA8BA;;;;KAWD,YAAA;;;;;;;;;;;;;;;;;;;;;;cAoBC,iDAAkD;;;;;;;;;;;;;;;;cA6BlD;;;AXndW;;;;;;;AAqCX,cYpJA,eZoJuG,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,IAAA,CAAA,EAAA,OAAA,GAAA,OAAA,EAAA,GAAA,MAAA;AAOpH;AAAgE;AAU9C;AA4GlB;;;;;;cYjPa;;AXtCb;AAgCA;AAyBA;AAsBA;;;;AAMwB,cWrCX,cXqCW,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AA0BxB;AA6CA;;;;;;;;;;ACxKY,cUuGC,qBV1EI,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,MAAA,EAAA,MAAA,EAAA,UAAA,EU0EkE,mBV1ElE,EAAA,EAAA,GAAA,MAAA;;;AF4EU;AAwBH;;;;;;;AAqCxB;AAOA;AAAgE;AAU9C;AA4GlB;;;;;;;;ACvRA;AAgCA;AAyBA;AAsBY,cYpEC,sBZoEkB,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;AAgC/B;AA6CA;;;;;;;;cY3Fa;cAeA"}
+{"version":3,"file":"index.d.mts","names":[],"sources":["../src/types/breakpoints.ts","../src/types/rules.ts","../src/types/options.ts","../src/types/validation.ts","../src/types/index.ts","../src/analysis/line-starts.ts","../src/analysis/repeating-sequences.ts","../src/detection.ts","../src/optimization/optimize-rules.ts","../src/preprocessing/transforms.ts","../src/recovery.ts","../src/segmentation/breakpoint-utils.ts","../src/segmentation/pattern-validator.ts","../src/segmentation/segmenter.ts","../src/segmentation/tokens.ts","../src/utils/textUtils.ts","../src/validation/validate-segments.ts"],"mappings":";AAqBA;;;;;;;;;;;;AA2GA;;;;;;;AA3GA,KAAY,cAAA,GAAiB,8BAAA;ECOxB;;;;;AAEI;;;;;AA4BG;EDzBR,OAAA;;;;ACuDc;;;;;AAiCC;;;;;AAwBH;EDhGZ,KAAA;;;;;;;;;;;;;;;;ACmIJ;;;;;AAOA;;;;;AAAgE;;;ED5G5D,KAAA;EC6HA;;;;;AAyBK;;;;;ED1IL,KAAA;ECiKO;;;;AAqDX;;;;;;;;;;;;;;EDlMI,QAAA;AAAA;AErFJ;;;;;;;;;AAgCA;;;;;AAhCA,KFsGY,UAAA,YAAsB,cAAA;;;AA3GlC;;;;;;;;;;;;AA2GA;;;;;;;;AChIiE;;;;;AA8BxD;ADTT,KCOK,YAAA;wEAED,KAAA;AAAA;AA4BQ;;;;;AA8BM;;;;;AAiCC;;;;;AAwBH;;;;;;;;AAvFJ,KAFP,eAAA;EA2GoB,2GAzGrB,QAAA;AAAA;;;;;;AA0HJ;;;;;AAOA;;;;;AAAgE;;;;;;;;;AA0CvD;KA/IJ,qBAAA;oHAED,cAAA;AAAA;;;;;;AAyNJ;;;;;;;;;;;;;;;;ACvRA;;;;;;;KD6FK,sBAAA;ECpFG,oHDsFJ,eAAA;AAAA;;;;ACtCJ;;;;;AAsBA;;;;;;;;;;;KDsCK,mBAAA;EChCmB,kFDkCpB,YAAA;AAAA;;;;;;;;;;;KAaC,WAAA,GACC,YAAA,GACA,eAAA,GACA,qBAAA,GACA,sBAAA,GACA,mBAAA;;;;;;;;;ACmBN;;;;;;;cDFa,iBAAA;;;;;;KAOD,cAAA,WAAyB,iBAAA;;;;;;;KAUhC,aAAA;EC6FD;;;;;;EDtFA,KAAA;;;;AE5MJ;;;;;EFsNI,UAAA;EEpNkC;;;;AAMtC;;;;;;;;;EF6NI,KAAA;AAAA;;;;;;;;;KAWC,eAAA,GAAkB,8BAAA;EEtNf;;;;;;;AAQR;;;;EF0NI,IAAA,GAAO,MAAA;EExNP;;;;;;;;;;;;;ACtBJ;;;;;;;EHoQI,cAAA;AAAA;;;AGxNJ;;;;;AA2BA;;;;;AAgBA;;;;;AAwBA;;;;;;;;;KHoLY,SAAA,GAAY,WAAA,GAAc,aAAA,GAAgB,eAAA;;;;;;;;;;;;;ADjLtD;;;;;;;;AChIiE;;;;KC0BrD,mBAAA,GAAsB,mBAAA;EAC9B,IAAA;ED6BgB;;;;AAER;;;ECvBR,IAAA;AAAA;ADqDc;;;;;AAiCC;;;;;AAwBH;;;;;;;;;;AAzDE,KC9BN,oBAAA,GAAuB,mBAAA;EAC/B,IAAA;AAAA;;;;;ADyHJ;;;;;AAOA;;;;;AAAgE;;;;;;;KCxGpD,kBAAA,GAAqB,mBAAA;EAC7B,IAAA;AAAA;;;;;;;;;;AD6NJ;;;;;;;;;KCxMY,mBAAA,+DAIN,mBAAA,GACA,oBAAA,GACA,kBAAA;;;;;;;AArFN;;;;;;;;;AAgCA;;;;;AAyBA;;;;UAsDiB,MAAA;EAhCL;EAkCR,KAAA,IAAS,OAAA,aAAoB,IAAA;;EAE7B,KAAA,IAAS,OAAA,aAAoB,IAAA;EA/B3B;EAiCF,IAAA,IAAQ,OAAA,aAAoB,IAAA;EAhCR;EAkCpB,KAAA,IAAS,OAAA,aAAoB,IAAA;EApC3B;EAsCF,IAAA,IAAQ,OAAA,aAAoB,IAAA;AAAA;;;AAVhC;;;;;;;;;;;;;;;;;;;;;AA6CA;;;;;;;;;KAAY,mBAAA;EAQR;;;;;;;EAAA,KAAA,GAAQ,SAAA;EA6ER;;;;;;;;;EAlEA,KAAA;gEAIU,OAAA;IAEA,OAAA,GAAU,KAAA;EAAA;;;;AC7MxB;;;;;AAMA;;;;;;EDwNI,QAAA;ECtNU;;;;;;;;;;;EDmOV,gBAAA;ECpNA;;;;;;;;AASJ;;;;;;;;;;;;;;;;;;ACpBA;;EF6PI,WAAA,GAAc,UAAA;EEhOD;;;;;;;EFyOb,MAAA;EE1NQ;;;;;AA2BZ;;;;;AAgBA;;EF6LI,UAAA;EExLA;;AAmBJ;;;;;;;;;;;;AC1GA;;;;;;;;;;;;;;;;;EHgTI,MAAA,GAAS,MAAA;EGrSC;;AAGd;;;;;AAEA;;;;;;;;;;AA4PA;;;;;;;EH8DI,UAAA,GAAa,mBAAA;AAAA;;;KC7VL,8BAAA;AAAA,KAEA,0BAAA;AAAA,KAMA,sBAAA;EACR,IAAA,EAAM,0BAAA;EACN,QAAA,EAAU,8BAAA;EACV,YAAA;EACA,OAAA;IACI,IAAA;IACA,EAAA;IACA,cAAA;EAAA;EAEJ,QAAA;IACI,IAAA;IACA,EAAA;EAAA;EAEJ,MAAA;IACI,IAAA;IACA,EAAA;EAAA;EAEJ,WAAA;IACI,MAAA;IACA,WAAA;IACA,UAAA;EAAA;EAEJ,QAAA;EACA,IAAA;AAAA;AAAA,KAGQ,uBAAA;EACR,EAAA;EACA,OAAA;IACI,YAAA;IACA,SAAA;IACA,MAAA;IACA,MAAA;IACA,QAAA;EAAA;EAEJ,MAAA,EAAQ,sBAAA;AAAA;;;;AHtBZ;;;;;;;;;;;;AA2GA;KIlHY,OAAA;;;;;;;EAOR,OAAA;EHOa;;;EGFb,IAAA;EH8BC;;;;;AAEO;EGxBR,EAAA;;;;AHsDc;;;;EG7Cd,IAAA,GAAO,MAAA;AAAA;;;;;AHsGK;;;;;;;;KGvFJ,IAAA;EHyGa;;;;;EGnGrB,EAAA;EHmGqB;;AAiBzB;;;;EG5GI,OAAA;AAAA;;;;;AHmH4D;;;;;;KGtGpD,SAAA;;;AHgJH;;;;;;;;;;AA4ET;;KG5MY,mBAAA;EH4MY;;;;EGvMpB,GAAA;EHuMoB;;;;EGjMpB,GAAA;AAAA;;;AFtFJ;;;;;;;;KEmGY,8BAAA,GAAiC,mBAAA;EFnEjC;;;;;AAyBZ;;;;;AAsBA;;;;;EEoCI,OAAA,GAAU,SAAA;AAAA;;;KC1HF,wBAAA;EACR,IAAA;EACA,WAAA;EACA,aAAA;EACA,QAAA;EACA,WAAA;EACA,wBAAA;EACA,yBAAA;EACA,MAAA;EACA,UAAA,IAAc,IAAA,UAAc,MAAA;EAC5B,cAAA,GAAiB,MAAA;EACjB,UAAA;AAAA;AAAA,KAGQ,uBAAA;EAA4B,IAAA;EAAc,MAAA;AAAA;AAAA,KAE1C,sBAAA;EACR,OAAA;EACA,KAAA;EACA,QAAA,EAAU,uBAAA;AAAA;;;;cAyPD,uBAAA,GACT,KAAA,EAAO,IAAA,IACP,OAAA,GAAS,wBAAA,KACV,sBAAA;;;KCzQS,wBAAA;EACR,WAAA;EACA,WAAA;EACA,QAAA;EACA,IAAA;EACA,yBAAA;EACA,YAAA;EACA,UAAA;EACA,WAAA;EACA,YAAA;EACA,iBAAA;AAAA;AAAA,KAGQ,wBAAA;EACR,IAAA;EACA,OAAA;EACA,MAAA;EACA,YAAA;AAAA;AAAA,KAGQ,wBAAA;EACR,OAAA;EACA,KAAA;EACA,QAAA,EAAU,wBAAA;AAAA;;ALwCI;;;;;cKsLL,yBAAA,GACT,KAAA,EAAO,IAAA,IACP,OAAA,GAAU,wBAAA,KACX,wBAAA;;;;AN5PH;;KOhBY,eAAA;EPgB+C,6DOdvD,KAAA,UP0BA;EOxBA,KAAA,UPsEA;EOpEA,KAAA,UPoGA;EOlGA,QAAA;AAAA;APmHJ;;;;;;;;AChIiE;;;;;AA8BxD;;ADkGT,cO5Ca,mBAAA,GAAuB,IAAA,aAAY,eAAA;;;AN1BpC;;;;;AA8BM;;;;;AAiCC;cM2BN,wBAAA,GAA4B,IAAA,UAAc,QAAA,EAAU,eAAA;;;;ANHjD;;;cM0BH,oBAAA,GACT,QAAA,EAAU,eAAA;EACT,WAAA;EAAmD,KAAA;EAAgB,QAAA;AAAA;;;;;;;cA+B3D,kBAAA,GACT,IAAA;EAEA,QAAA;EACA,WAAA;EACA,KAAA;EACA,QAAA;EACA,QAAA,EAAU,eAAA;AAAA;;;AP9Ld;;;AAAA,KQbY,cAAA;ERaiB,yDQXzB,KAAA,EAAO,SAAA,IRuCP;EQrCA,WAAA;AAAA;AAAA,cAkCS,aAAA,GAAiB,KAAA,EAAO,SAAA;;SAAA,SAAA;AAAA;;;;;ARkFrC;;;;;cShGa,eAAA,GAAmB,IAAA,UAAc,IAAA;;;ARhCmB;;;;;AA8BxD;;cQkCI,gBAAA,GAAoB,IAAA;;;ARNrB;;;;;AA8BM;cQdL,cAAA,GAAkB,IAAA;;;;AR+CZ;;;;;AAwBH;;;cQ5BH,qBAAA,GAAyB,OAAA,UAAiB,MAAA,UAAgB,UAAA,EAAY,mBAAA;;;KC9GvE,sBAAA;EACJ,IAAA;EAAsB,OAAA;AAAA;EACtB,IAAA;EAAkC,KAAA;EAAgC,QAAA;AAAA;EAClE,IAAA;EAAmB,SAAA,GAAY,IAAA,EAAM,SAAA,EAAW,KAAA;AAAA;AAAA,KAE5C,iBAAA;EACR,OAAA,EAAS,mBAAA;EACT,KAAA,EAAO,IAAA;EACP,QAAA,EAAU,OAAA;EACV,QAAA,EAAU,sBAAA;AAAA;AAAA,KAGF,oBAAA;EACR,OAAA;IACI,IAAA;IACA,SAAA;IACA,aAAA;IACA,SAAA;IACA,UAAA;EAAA;EAEJ,KAAA,GAAQ,KAAA;IACJ,SAAA;IACA,QAAA;IACA,aAAA;IACA,UAAA;EAAA;EAEJ,OAAA,EAAS,KAAA;IACL,IAAA;IACA,oBAAA;IACA,sBAAA;IACA,qBAAA;IACA,YAAA;IACA,MAAA;IACA,QAAA;IACA,EAAA;IACA,KAAA;EAAA;EAEJ,MAAA;EACA,QAAA;AAAA;AAAA,KAGC,oBAAA;AAAA,iBAgnBW,qCAAA,CACZ,KAAA,EAAO,IAAA,IACP,QAAA,EAAU,OAAA,IACV,OAAA,EAAS,mBAAA,EACT,QAAA,EAAU,sBAAA,EACV,IAAA;EACI,IAAA;EACA,gBAAA,GAAmB,oBAAA;AAAA;EAEtB,MAAA,EAAQ,oBAAA;EAAsB,QAAA,EAAU,OAAA;AAAA;AAAA,iBA6C7B,6BAAA,CACZ,IAAA,EAAM,iBAAA,IACN,IAAA;EAAS,IAAA;EAAgD,gBAAA,GAAmB,oBAAA;AAAA;EAC3E,MAAA,EAAQ,oBAAA;EAAsB,QAAA,EAAU,OAAA;AAAA;;;;;;;;;;;;;AVzlB7C;;;;;;;;AChIiE;;;;cUoCpD,wBAAA,GAA4B,IAAA;;KA0K7B,gBAAA,IAAoB,OAAA;;;;;;KCjMpB,mBAAA;;;AZmHZ;KY9GY,eAAA;EACR,IAAA,EAAM,mBAAA;EACN,OAAA;EACA,UAAA;EAEA,KAAA;EAEA,OAAA;AAAA;;;;AXKK;KWEG,oBAAA;EACR,cAAA,IAAkB,eAAA;EAClB,eAAA,IAAmB,eAAA;EACnB,YAAA,IAAgB,eAAA;EAChB,QAAA,GAAW,eAAA;AAAA;;;;AXoDG;;;;;AAiCC;;;;;AAwBH;;;;;;cWtBH,aAAA,GAAiB,KAAA,EAAO,SAAA,QAAW,oBAAA;;;;;;;;;;;AXyDhD;;;cWnBa,sBAAA,GAA0B,OAAA,GAAU,oBAAA;;;;;;;;AXjKgB;;;;;AA8BxD;;;;;AA4BG;;;;;AA8BM;;;;;AAiCC;;;;;AAwBH;;;;;;;;;;;;cYiKH,YAAA,GAAgB,KAAA,EAAO,IAAA,IAAQ,OAAA,EAAS,mBAAA,KAAmB,OAAA;;;;cCxM3D,KAAA;EdrFa,oDAAiC;EAAA,mCAYvD;EAAA,+BA8CA;EAAA,2BAgCA;EAAA,2BAAQ;EAAA,2BAiBU;EAAA,6BAAY;EAAA;;mCCpGjB;EAAA,yBAEb;EAAA,mCA0BC;EAAA;6BAEO;EAAA,6BA4Bc;EAAA,6BAEtB;EAAA;;;;;Ka0DQ,QAAA,gBAAwB,KAAA;AbzBjB;AAAA,ca4BN,WAAA,GAAe,KAAA,UAAe,IAAA;;cAiB9B,+BAAA,GAAmC,QAAA;;AbrBhC;;;;;;;;;;;;;;;;;AAmChB;;;;;AAOA;;ca2Ba,cAAA,EAAgB,MAAA;;;Ab3BmC;;;;;;;;;AA0CvD;;;;ca4BI,cAAA,GAAkB,KAAA;;;;;;AbgD/B;KarCY,YAAA;;;;;;EAMR,OAAA;Eb+BoB;;;;;EaxBpB,YAAA;;;AZ/PJ;;;EYsQI,WAAA;AAAA;;;;;AZtOJ;;;;;AAyBA;;;;;AAsBA;;;;;;;;;;;;;AAgCA;;;;;;;;;cYqRa,wBAAA,GACT,KAAA,UACA,cAAA,IAAkB,OAAA,qBAClB,aAAA;;;;;;;;;;;;AZ3OJ;;;;;;;;;;;;cYsRa,YAAA,GAAgB,KAAA;;;;;;;;;;;;;;;;;;;AX5c7B;;;cWmea,eAAA,GAAmB,QAAA,aAAgB,MAAA;;AXjehD;;;;;AAMA;;;;;;cWgfa,kBAAA;;;;;;;;;;;;;;;cAgBA,eAAA,GAAmB,SAAA;;;;;;AXtehC;;;;;;;;;cWogBa,oBAAA,GAAwB,QAAA;;;;KAWzB,YAAA;EAAiB,KAAA;EAAe,IAAA;AAAA;;AVniB5C;;;;;;;;;;;AA4CA;;;;;AA2BA;cUgfa,kBAAA,GAAsB,QAAA,UAAkB,QAAA,EAAU,YAAA;;;;AVhe/D;;;;;AAwBA;;;;;;;cUqea,kBAAA,GAAsB,QAAA;;;;;;AdlenC;;;;;;;;AChIiE;;;;;AA8BxD;;;;;AA4BG;;ccrBC,sBAAA,GAA0B,OAAA;;;AdmDrB;;;;;AAiCC;;;;;AAwBH;cctDH,WAAA,GAAe,CAAA;AAAA,cAef,wBAAA,GAA4B,IAAA;;;KCT7B,iBAAA;EhB5E+C;;;;;EgBkFvD,mBAAA;AAAA;;;AhByBJ;;;;;;;;AChIiE;;;;ce2iBpD,gBAAA,GACT,KAAA,EAAO,IAAA,IACP,OAAA,EAAS,mBAAA,EACT,QAAA,EAAU,OAAA,IACV,iBAAA,GAAoB,iBAAA,KACrB,uBAAA"}