flappa-doormal 2.8.0 → 2.9.0

package/AGENTS.md

@@ -26,8 +26,14 @@ Traditional Arabic text segmentation requires:
 ```text
 src/
 ├── index.ts                    # Main entry point and exports
-├── pattern-detection.ts        # Token detection for auto-generating rules (NEW)
-├── pattern-detection.test.ts   # Pattern detection tests (22 tests)
+├── analysis/                   # Analysis helpers module
+│   ├── index.ts                # Barrel exports for analysis functions
+│   ├── shared.ts               # Shared utilities for analysis
+│   ├── line-starts.ts          # analyzeCommonLineStarts (line-based patterns)
+│   ├── repeating-sequences.ts  # analyzeRepeatingSequences (continuous text N-grams)
+│   └── *.test.ts               # Analysis tests
+├── pattern-detection.ts        # Token detection for auto-generating rules
+├── pattern-detection.test.ts   # Pattern detection tests
 ├── recovery.ts                 # Marker recovery utility (recover mistaken lineStartsAfter)
 ├── recovery.test.ts            # Marker recovery tests
 └── segmentation/
@@ -202,7 +208,7 @@ Raw `regex` patterns now support named capture groups for metadata extraction:
 
 ### Breakpoints Post-Processing Algorithm
 
-The `breakpoints` option provides a post-processing mechanism for limiting segment size. Unlike the deprecated `maxSpan` (which was per-rule), breakpoints runs AFTER all structural rules.
+The `breakpoints` option provides a post-processing mechanism for limiting segment size. Breakpoints runs AFTER all structural rules.
 
 **API Options:**
 ```typescript
@@ -244,7 +250,7 @@ segmentPages(pages, {
 - **`prefer: 'shorter'`**: Finds FIRST match (conservative)
 - **Recursive**: If split result still exceeds `maxPages`, breakpoints runs again
 
-> **Note**: The old `maxSpan` and `fallback` properties on `SplitRule` are deprecated and removed.
+> **Note**: Older per-rule span limiting approaches were removed in favor of post-processing `breakpoints`.
 
 ## Design Decisions
 
@@ -357,7 +363,7 @@ bunx biome lint .
 
 5. **Rule order matters for specificity**: When multiple rules can match the same position, put specific patterns BEFORE generic ones. Example: `## {{raqms:num}} {{dash}}` must come before `##` to capture the number.
 
-6. **Post-processing beats per-rule limits**: The `maxSpan` approach (per-rule page limits) caused premature splits. Moving to post-processing `breakpoints` preserves structural integrity while still limiting segment size.
+6. **Post-processing beats per-rule limits**: Per-rule span limiting caused premature splits. Moving to post-processing `breakpoints` preserves structural integrity while still limiting segment size.
 
 7. **Window padding matters**: When calculating approximate content windows, 50% padding is needed (not 20%) to ensure enough content is captured for `prefer: 'longer'` scenarios.
 
@@ -494,3 +500,32 @@ const quoted = analyzeCommonLineStarts(pages, {
 });
 ```
 
+## Repeating Sequence Analysis (`analyzeRepeatingSequences`)
+
+For continuous text **without line breaks** (prose-like content), use `analyzeRepeatingSequences(pages)`. It scans for commonly repeating word/token sequences (N-grams) across pages.
+
+Key options:
+- `minElements` / `maxElements`: N-gram size range (default 1-3)
+- `minCount`: Minimum occurrences to include (default 3)
+- `topK`: Maximum patterns to return (default 20)
+- `requireToken`: Only patterns containing `{{tokens}}` (default true)
+- `normalizeArabicDiacritics`: Ignore diacritics when matching (default true)
+
+Example:
+```typescript
+import { analyzeRepeatingSequences } from 'flappa-doormal';
+
+const patterns = analyzeRepeatingSequences(pages, { minCount: 3, topK: 20 });
+// [{ pattern: '{{naql}}', count: 42, examples: [...] }, ...]
+```
+
+## Analysis → Segmentation Workflow
+
+Use analysis functions to discover patterns, then pass to `segmentPages()`:
+
+1. **Continuous text**: `analyzeRepeatingSequences()` → build rules → `segmentPages()`
+2. **Structured text**: `analyzeCommonLineStarts()` → build rules → `segmentPages()`
+
+See README.md for complete examples.
+
+

package/README.md

@@ -228,7 +228,6 @@ Control which matches to use:
   lineEndsWith: ['\\.'],
   split: 'after',
   occurrence: 'last',  // Only split at LAST period on page
-  maxSpan: 1,          // Apply per-page
 }
 ```
 
@@ -406,8 +405,99 @@ Key options:
 - If you paste these signatures into `lineStartsWith` / `lineStartsAfter` / `template`, that’s fine: those template pattern types **auto-escape `()[]`** outside `{{tokens}}`.
 - If you paste them into a raw `regex` rule, you may need to escape literal brackets yourself.
 
+### Repeating Sequence Analysis (continuous text)
+
+For texts without line breaks (continuous prose), use `analyzeRepeatingSequences()`:
+
+```typescript
+import { analyzeRepeatingSequences } from 'flappa-doormal';
+
+const patterns = analyzeRepeatingSequences(pages, {
+  minElements: 2,
+  maxElements: 4,
+  minCount: 3,
+  topK: 20,
+});
+// [{ pattern: "{{naql}}\\s*{{harf}}", count: 42, examples: [...] }, ...]
+```
+
+Key options:
+- `minElements` / `maxElements`: N-gram size range (default 1-3)
+- `minCount`: Minimum occurrences to include (default 3)
+- `topK`: Maximum patterns to return (default 20)
+- `requireToken`: Only patterns containing `{{tokens}}` (default true)
+- `normalizeArabicDiacritics`: Ignore diacritics when matching (default true)
+
+## Analysis → Segmentation Workflow
+
+Use analysis functions to discover patterns, then pass to `segmentPages()`.
+
+### Example A: Continuous Text (No Punctuation)
+
+For prose-like text without structural line breaks:
+
+```typescript
+import { analyzeRepeatingSequences, segmentPages, type Page } from 'flappa-doormal';
+
+// Continuous Arabic text with narrator phrases
+const pages: Page[] = [
+  { id: 1, content: 'حدثنا أحمد بن محمد عن عمر قال سمعت النبي حدثنا خالد بن زيد عن علي' },
+  { id: 2, content: 'حدثنا سعيد بن جبير عن ابن عباس أخبرنا يوسف عن أنس' },
+];
+
+// Step 1: Discover repeating patterns
+const patterns = analyzeRepeatingSequences(pages, { minCount: 2, topK: 10 });
+// [{ pattern: '{{naql}}', count: 5, examples: [...] }, ...]
+
+// Step 2: Build rules from discovered patterns
+const rules = patterns.filter(p => p.count >= 3).map(p => ({
+  lineStartsWith: [p.pattern],
+  split: 'at' as const,
+  fuzzy: true,
+}));
+
+// Step 3: Segment
+const segments = segmentPages(pages, { rules });
+// [{ content: 'حدثنا أحمد بن محمد عن عمر قال سمعت النبي', from: 1 }, ...]
+```
+
+### Example B: Structured Text (With Numbering)
+
+For hadith-style numbered entries:
+
+```typescript
+import { analyzeCommonLineStarts, segmentPages, type Page } from 'flappa-doormal';
+
+// Numbered hadith text
+const pages: Page[] = [
+  { id: 1, content: '٦٦٩٦ - حَدَّثَنَا أَبُو بَكْرٍ عَنِ النَّبِيِّ\n٦٦٩٧ - أَخْبَرَنَا عُمَرُ قَالَ' },
+  { id: 2, content: '٦٦٩٨ - حَدَّثَنِي مُحَمَّدٌ عَنْ عَائِشَةَ' },
+];
+
+// Step 1: Discover common line-start patterns
+const patterns = analyzeCommonLineStarts(pages, { topK: 10, minCount: 2 });
+// [{ pattern: '{{raqms}}\\s*{{dash}}', count: 3, examples: [...] }, ...]
+
+// Step 2: Build rules (add named capture for hadith number)
+const topPattern = patterns[0]?.pattern ?? '{{raqms}} {{dash}} ';
+const rules = [{
+  lineStartsAfter: [topPattern.replace('{{raqms}}', '{{raqms:num}}')],
+  split: 'at' as const,
+  meta: { type: 'hadith' }
+}];
+
+// Step 3: Segment
+const segments = segmentPages(pages, { rules });
+// [
+//   { content: 'حَدَّثَنَا أَبُو بَكْرٍ...', from: 1, meta: { type: 'hadith', num: '٦٦٩٦' } },
+//   { content: 'أَخْبَرَنَا عُمَرُ قَالَ', from: 1, meta: { type: 'hadith', num: '٦٦٩٧' } },
+//   { content: 'حَدَّثَنِي مُحَمَّدٌ...', from: 2, meta: { type: 'hadith', num: '٦٦٩٨' } },
+// ]
+```
+
 ## Rule Validation
 
+
 Use `validateRules()` to detect common mistakes in rule patterns before running segmentation:
 
 ```typescript
@@ -619,32 +709,10 @@ const segments = segmentPages(pages, {
     lineEndsWith: ['\\.'],
     split: 'after',
     occurrence: 'last',
-    maxSpan: 1
-  }]
-});
-```
-
-### Page Fallback for Unmatched Content
-
-When using `maxSpan` to group matches per page, use `fallback: 'page'` to prevent unmatched pages from merging with adjacent segments:
-
-```typescript
-const segments = segmentPages(pages, {
-  rules: [{
-    template: '{{tarqim}}',  // Match punctuation marks
-    split: 'after',
-    occurrence: 'last',
-    maxSpan: 1,
-    fallback: 'page'  // If no punctuation found, segment the page anyway
   }]
 });
 ```
 
-**Without `fallback`**: Pages without matches merge into the next segment  
-**With `fallback: 'page'`**: Each page becomes its own segment even without matches
-
-> **Future extensions**: The `fallback` option may support additional values like `'skip'` (omit unmatched content) or `'line'` (split at line breaks) in future versions.
-
 ### Multiple Rules with Priority
 
 ```typescript
@@ -912,9 +980,7 @@ type SplitRule = {
   // Split behavior
   split?: 'at' | 'after';  // Default: 'at'
   occurrence?: 'first' | 'last' | 'all';
-  maxSpan?: number;
   fuzzy?: boolean;
-  fallback?: 'page';  // NEW: Page-boundary fallback
 
   // Constraints
   min?: number;
@@ -1042,28 +1108,6 @@ The library concatenates all pages into a single string for pattern matching acr
 
 For typical book processing (up to 6,000 pages), memory usage is well within Node.js defaults. For very large books (40,000+ pages), ensure adequate heap size.
 
-### `maxSpan` Sliding Window Behavior
-
-The `maxSpan` option uses a **sliding window algorithm** based on page ID difference:
-
-```typescript
-// maxSpan = maximum page ID difference when looking ahead for split points
-// Algorithm prefers LONGER segments by looking as far ahead as allowed
-
-// Pages [1, 2, 3, 4] with maxSpan: 1, occurrence: 'last'
-// Window from page 1: pages 1-2 (diff <= 1), splits at page 2's last match
-// Window from page 3: pages 3-4 (diff <= 1), splits at page 4's last match
-// Result: 2 segments spanning pages 1-2 and 3-4
-
-// Pages [1, 5, 10] with maxSpan: 1, occurrence: 'last'  
-// Window from page 1: only page 1 (5-1=4 > 1), splits at page 1
-// Window from page 5: only page 5 (10-5=5 > 1), splits at page 5
-// Window from page 10: only page 10, splits at page 10
-// Result: 3 segments (pages too far apart to merge)
-```
-
-This is intentional for books where page IDs represent actual page numbers. With `occurrence: 'last'`, the algorithm finds the last match within the lookahead window, creating longer segments where possible.
-
 ## For AI Agents
 
 See [AGENTS.md](./AGENTS.md) for:

package/dist/index.d.mts

@@ -238,38 +238,9 @@ type SplitBehavior = {
    * - `'first'`: Only split at the first match
    * - `'last'`: Only split at the last match
    *
-   * When `maxSpan` is set, occurrence filtering is applied per sliding
-   * window rather than globally. With `'last'`, the algorithm prefers
-   * longer segments by looking as far ahead as allowed before selecting
-   * the last match in the window.
-   *
    * @default 'all'
    */
   occurrence?: 'first' | 'last' | 'all';
-  /**
-   * Maximum page ID difference allowed when looking ahead for split points.
-   *
-   * Uses a sliding window algorithm that prefers longer segments:
-   * 1. Start from the first page of the current segment
-   * 2. Look for matches within pages where `pageId - startPageId <= maxSpan`
-   * 3. Apply occurrence filter (e.g., 'last') to select a match
-   * 4. Next window starts from the page after the match
-   *
-   * Examples:
-   * - `maxSpan: 1` = look 1 page ahead (segments span at most 2 pages)
-   * - `maxSpan: 2` = look 2 pages ahead (segments span at most 3 pages)
-   * - `undefined` = no limit (entire content treated as one group)
-   *
-   * Note: With non-consecutive page IDs, the algorithm uses actual ID
-   * difference, not array index. Pages 1 and 5 have a difference of 4.
-   *
-   * @example
-   * // Split at last period, looking up to 1 page ahead
-   * // Pages 1,2: split at page 2's last period
-   * // Page 3: split at page 3's last period
-   * { lineEndsWith: ['.'], split: 'after', occurrence: 'last', maxSpan: 1 }
-   */
-  maxSpan?: number;
   /**
    * Enable diacritic-insensitive matching for Arabic text.
    *
@@ -354,12 +325,6 @@ type RuleConstraints = {
    * { lineStartsWith: ['{{bab}}'], split: 'before', meta: { type: 'chapter' } }
    */
   meta?: Record<string, unknown>;
-  /**
-   * Fallback behavior when no matches are found within a maxSpan boundary.
-   * - 'page': Create split points at page boundaries
-   * - undefined: No fallback (current behavior)
-   */
-  fallback?: 'page';
   /**
    * Page-start guard: only allow this rule to match at the START of a page if the
    * previous page's last non-whitespace character matches this pattern.
@@ -388,7 +353,7 @@ type RuleConstraints = {
  * Each rule must specify:
  * - **Pattern** (exactly one): `regex`, `template`, `lineStartsWith`,
  *   `lineStartsAfter`, or `lineEndsWith`
- * - **Split behavior**: `split` (optional, defaults to `'at'`), `occurrence`, `maxSpan`, `fuzzy`
+ * - **Split behavior**: `split` (optional, defaults to `'at'`), `occurrence`, `fuzzy`
  * - **Constraints** (optional): `min`, `max`, `meta`
  *
  * @example
@@ -424,7 +389,6 @@ type SplitRule = PatternType & SplitBehavior & RuleConstraints;
 type Page = {
   /**
    * Unique page/entry ID used for:
-   * - `maxSpan` grouping (segments spanning multiple pages)
    * - `min`/`max` constraint filtering
    * - `from`/`to` tracking in output segments
    */
@@ -625,6 +589,21 @@ type SegmentationOptions = {
    * rule's metadata is used for each segment.
    */
   rules?: SplitRule[];
+  /**
+   * Attach debugging provenance into `segment.meta` indicating which rule and/or breakpoint
+   * created the segment boundary.
+   *
+   * This is opt-in because it increases output size.
+   *
+   * When enabled (default metaKey: `_flappa`), segments may include:
+   * `meta._flappa.rule` and/or `meta._flappa.breakpoint`.
+   */
+  debug?: boolean | {
+    /** Where to store provenance in meta. @default '_flappa' */
+    metaKey?: string;
+    /** Which kinds of provenance to include. @default ['rule','breakpoint'] */
+    include?: Array<'rule' | 'breakpoint'>;
+  };
   /**
    * Maximum pages per segment before breakpoints are applied.
    *
@@ -770,7 +749,7 @@ type Segment = {
 /**
  * Types of validation issues that can be detected.
  */
-type ValidationIssueType = 'missing_braces' | 'unknown_token' | 'duplicate';
+type ValidationIssueType = 'missing_braces' | 'unknown_token' | 'duplicate' | 'empty_pattern';
 /**
  * A validation issue found in a pattern.
  */

package/dist/index.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.mts","names":[],"sources":["../src/segmentation/fuzzy.ts","../src/segmentation/types.ts","../src/segmentation/pattern-validator.ts","../src/segmentation/replace.ts","../src/segmentation/segmenter.ts","../src/segmentation/tokens.ts","../src/analysis/line-starts.ts","../src/analysis/repeating-sequences.ts","../src/detection.ts","../src/recovery.ts"],"sourcesContent":[],"mappings":";;AAkEA;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EY,cD/bC,WC+bqB,EAAA,CAAA,CAAA,EAAA,MAAc,EAAA,GAAA,MAAA;AA8BhD;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;AC6SA;;AAAqD,cJnNxC,wBImNwC,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;AJlTrD;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA,KApXK,YAAA,GAoXW;EAqCJ;EA0EA,KAAA,EAAA,MAAU;AA8BtB,CAAA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA,KF8CK,eAAA,GE9CkB;EA2DV;EAA4B,QAAA,EAAA,MAAA;CAAgB;;;;;;AC6SzD;;;;;;;;AChTA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;;;KJlnBK,qBAAA;EKnEO;EAcA,cAAA,EAAA,MAAA,EAAA;AAEZ,CAAA;AAwQA;;;;;;;;AClRA;AAaA;AAOA;AA2OA;;;;;;;;AC9QA;AA+EA;AAgEA;AAuBA;AAiCA;;;;AC7MA;AAKA,KR8GK,sBAAA,GQ9GwB;EAChB;EACF,eAAA,EAAA,MAAA,EAAA;CACG;;;AAId;AA2BE;AAmnBF;;;;;;;;;AAsDA;;;;;;KRrkBK,mBAAA;;;;;;;;;;;;;;KAeA,WAAA,GACC,eACA,kBACA,wBACA,yBACA;;;;;;;KAYD,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+EO,SAAA;;;;;;;KAYP,eAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAyCS;;;;;;;;;;;;SAaH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA8DC,SAAA,GAAY,cAAc,gBAAgB;;;;;;;;;;;;;KAkB1C,IAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCA,cAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAqCE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCF,UAAA,YAAsB;;;;;;;;;;;;;;;;;;;;;;;;;UA8BjB,MAAA;;;;;;;;;;;;;;;;;;;;;;KAuBL,WAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+CA,mBAAA;;;;;;YAME;;;;;;;;UASF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA8CM;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;WAwDL;;;;;;;;;;;;;;;;KAiBD,OAAA;;;;;;;;;;;;;;;;;;;;;;;;;;SA6BD;;;;AA1uBM;AA4BG;AA8BM;AAyDrB,KCpIO,mBAAA,GDoIY,gBAAA,GAAA,eAAA,GAAA,WAAA;AAAA;;;AAkBlB,KCjJM,eAAA,GDiJN;EACA,IAAA,ECjJI,mBDiJJ;EACA,OAAA,EAAA,MAAA;EAAmB,UAAA,CAAA,EAAA,MAAA;AAAA,CAAA;AA2FzB;AAAkD;AAgIlD;;AAAsC,KCpW1B,oBAAA,GDoW0B;EAAgB,cAAA,CAAA,EAAA,CCnWhC,eDmWgC,GAAA,SAAA,CAAA,EAAA;EAAe,eAAA,CAAA,EAAA,CClW9C,eDkW8C,GAAA,SAAA,CAAA,EAAA;EAkBzD,YAAI,CAAA,EAAA,CCnXI,eDmXJ,GAAA,SAAA,CAAA,EAAA;EAqCJ,QAAA,CAAA,ECvZG,eDuZW;AA0E1B,CAAA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;AAI8B,cAoGjB,aApGiB,EAAA,CAAA,KAAA,EAoGO,SApGP,EAAA,EAAA,GAAA,CAoGsB,oBApGtB,GAAA,SAAA,CAAA,EAAA;;;AFkC9B;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAuCtB,KEpJO,WAAA,GAAc,WFoJV,CEpJsB,mBFoJtB,CAAA,SAAA,CAAA,CAAA,CAAA,MAAA,CAAA;;;;;;;AAKS;AA2FzB;AAAkD;AAgIlD;AAAwB,cEzTX,iBFyTW,EAAA,CAAA,KAAA,EEzTiB,IFyTjB,EAAA,EAAA,KAAA,CAAA,EEzTiC,WFyTjC,EAAA,EAAA,GEzTiD,IFyTjD,EAAA;;;;;AAkBxB;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;AC6SA;;;;;;;;AChTA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;cDtVa,sBAAuB,iBAAiB,wBAAmB;;;;AJlTxE;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;AC6SA;AAAoC,cChTvB,sBDgTuB,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;;;AChTpC;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBa,cA5WA,+BA4WsD,EAAA,CAAA,QAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AAgBnE;;;;ACrrBA;AAcA;AAEA;AAwQA;;;;;;;;AClRA;AAaA;AAOA;AA2OA;;;;;;;cF0Ga,gBAAgB;AGxX7B;AA+EA;AAgEA;AAuBA;AAiCA;;;;AC7MA;AAKA;;;;;;AAOY,cJ6ZC,cI7ZmB,EAAA,CAAA,KAQpB,EAAA,MAMC,EAAK,GAAA,OAAA;AAahB;AAmnBF;;;;;AAO2B,KJ7Of,YAAA,GI6Oe;EAEd;;;AA6Cb;;EAEgF,OAAA,EAAA,MAAA;EACnE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cJxHA,mHAIV;;;;;;;;;;;;;;;;;;;;cAyCU;;;;;;;;;;;;;;;;;;;;;;cAuBA,uCAAmC;;;;;;;;;;;;;cAqBnC;;;;;;;;;;;;;;;cAgBA;;;ALxoBA,KM7CD,wBAAA,GN6C8E;EA+F7E,IAAA,CAAA,EAAA,MAAA;;;;ECnIR,WAAA,CAAA,EAAA,MAAY;EA4BZ,wBAAe,CAAA,EAAA,OAAA;EA8Bf,yBAAqB,CAAA,EAAA,OAAA;EAiCrB,MAAA,CAAA,EAAA,aAAA,GAAsB,OAAA;EAwBtB,UAAA,CAAA,EAAA,CAAA,IAAA,EAAA,MAAmB,EAAA,MAAA,EAAA,MAAA,EAAA,GAAA,OAAA;EAenB,cAAW,CAAA,EKjIK,MLiIL,EAAA;EACV,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;CACA;AACA,KKhIM,uBAAA,GLgIN;EACA,IAAA,EAAA,MAAA;EACA,MAAA,EAAA,MAAA;CAAmB;AAYpB,KK5IO,sBAAA,GL4IM;EA+EN,OAAA,EAAA,MAAS;EAYhB,KAAA,EAAA,MAAA;EAoHO,QAAA,EKxVE,uBLwVO,EAAA;CAAG;;;;AAkBZ,cKrGC,uBLqGG,EAAA,CAAA,KAAA,EKpGL,ILoGK,EAAA,EAAA,OAAA,CAAA,EKnGH,wBLmGG,EAAA,GKlGb,sBLkGa,EAAA;;;AAjQX,KMtHO,wBAAA,GNsHY;EAenB,WAAA,CAAA,EAAW,MAAA;EACV,WAAA,CAAA,EAAA,MAAA;EACA,QAAA,CAAA,EAAA,MAAA;EACA,IAAA,CAAA,EAAA,MAAA;EACA,yBAAA,CAAA,EAAA,OAAA;EACA,YAAA,CAAA,EAAA,OAAA;EAAmB,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;EAYpB,WAAA,CAAA,EAAA,MAAa;EA+EN,YAAS,CAAA,EAAA,MAAA;EAYhB,iBAAA,CAAe,EAAA,MAAA;AAoHpB,CAAA;AAAwB,KMxVZ,wBAAA,GNwVY;EAAc,IAAA,EAAA,MAAA;EAAgB,OAAA,EAAA,MAAA;EAAe,MAAA,EAAA,MAAA;EAkBzD,YAAI,EAAA,MAAA,EAAA;AAqChB,CAAA;AA0EY,KMldA,wBAAA,GNkdsB;EA8BjB,OAAA,EAAM,MAAA;EAuBX,KAAA,EAAA,MAAA;EA+CA,QAAA,EMnjBE,wBNmjBiB,EAAA;CAMjB;;;AAgId;;;;AC9tBY,cK6QC,yBL7QkB,EAAA,CAAA,KAAA,EK8QpB,IL9QoB,EAAA,EAAA,OAAA,CAAA,EK+QjB,wBL/QiB,EAAA,GKgR5B,wBLhR4B,EAAA;;;;AFqD/B;AA+FA;;;;;ACnIiB;AA4BG;AA+Df,KO7GO,eAAA,GP6Ge;EAwBtB;EAeA,KAAA,EAAA,MAAA;EACC;EACA,KAAA,EAAA,MAAA;EACA;EACA,KAAA,EAAA,MAAA;EACA;EAAmB,QAAA,EAAA,MAAA;AAAA,CAAA;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;AA6DkB,cOvkBL,mBPukBK,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GOvkB8B,ePukB9B,EAAA;;;AAyElB;;;;AC9tBA;AAKA;AAUA;;;;;AAI8B,cM2HjB,wBN3HiB,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,QAAA,EM2HmC,eN3HnC,EAAA,EAAA,GAAA,MAAA;AAoG9B;;;;ACxHA;AA2DA;AAAyC,cK2G5B,oBL3G4B,EAAA,CAAA,QAAA,EK4G3B,eL5G2B,EAAA,EAAA,GAAA;EAAgB,WAAA,EAAA,gBAAA,GAAA,iBAAA;EAAgB,KAAA,EAAA,OAAA;EAAI,QAAA,CAAA,EAAA,MAAA;;;;AC6S7E;;;;AAAwE,cIjK3D,kBJiK2D,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA;;;;EChT3D,QAAA,CAAA,EAAA,MAAA;EA0QA,QAAA,EGpHC,eHoHD,EAAA;AAsDb,CAAA,GAAa,IAAA;;;ALlUA,KS5DD,sBAAA,GT4D8E;EA+F7E,IAAA,EAAA,cAAA;;;;ECnIR,KAAA,CAAA,EAAA,OAAY,GAAA,YAAA;EA4BZ,QAAA,EAAA,MAAA,EAAe;AAAA,CAAA,GA8Bf;EAiCA,IAAA,EAAA,WAAA;EAwBA,SAAA,EAAA,CAAA,IAAA,EQxIwC,SRwIrB,EAAA,KAAA,EAAA,MAAA,EAAA,GAAA,OAAA;AAAA,CAAA;AAgBlB,KQtJM,iBAAA,GRsJN;EACA,OAAA,EQtJO,mBRsJP;EACA,KAAA,EQtJK,IRsJL,EAAA;EACA,QAAA,EQtJQ,ORsJR,EAAA;EACA,QAAA,EQtJQ,sBRsJR;CAAmB;AAYpB,KQ/JO,oBAAA,GR+JM;EA+EN,OAAA,EAAA;IAYP,IAAA,EAAA,YAAe,GAAA,wBAsDH;IA8DL,SAAS,EAAA,MAAA;IAAG,aAAA,EAAA,MAAA;IAAc,SAAA,EAAA,MAAA;IAAgB,UAAA,EAAA,MAAA;EAAe,CAAA;EAkBzD,KAAA,CAAA,EQxXA,KRwXI,CAAA;IAqCJ,SAAA,EAAA,MAAc;IA0Ed,QAAA,EAAU,MAAA;IA8BL,aAAM,EAAA,MAAA;IAuBX,UAAW,EAAA,MAAA;EA+CX,CAAA,CAAA;EAME,OAAA,EQ3kBD,KR2kBC,CAAA;IASF,IAAA,EAAA,MAAA;IA8CM,oBAAA,EAAA,MAAA;IAwDL,sBAAA,CAAA,EAAA,MAAA;IAAM,qBAAA,CAAA,EAAA,MAAA;IAiBP,YAAO,EAAA,MA6BR;;;;IC3vBC,KAAA,CAAA,EAAA,MAAA,EAAA;EAKA,CAAA,CAAA;EAUA,MAAA,EAAA,MAAA,EAAA;EACU,QAAA,EAAA,MAAA,EAAA;CACC;KOiBlB,oBAAA,GPhBe,MAAA,GAAA,YAAA,GAAA,qBAAA;AACL,iBOgoBC,qCAAA,CPhoBD,KAAA,EOioBJ,IPjoBI,EAAA,EAAA,QAAA,EOkoBD,OPloBC,EAAA,EAAA,OAAA,EOmoBF,mBPnoBE,EAAA,QAAA,EOooBD,sBPpoBC,EAAA,KAAA,EAAA;EAAe,IAAA,CAAA,EAAA,YAAA,GAAA,wBAAA;EAoGjB,gBA0CZ,CAAA,EOyf0B,oBPniByB;;UOqiBvC;YAAgC;AN7pB7C,CAAA;AA2Da,iBM+oBG,6BAAA,CNznBf,IAAA,EM0nBS,iBN1nBT,EAAA,EAAA,IAtBwE,CAsBxE,EAAA;EAtBwC,IAAA,CAAA,EAAA,YAAA,GAAA,wBAAA;EAAgB,gBAAA,CAAA,EMipBuB,oBNjpBvB;CAAgB,CAAA,EAAA;EAAI,MAAA,EMkpBhE,oBNlpBgE;YMkpBhC"}
+{"version":3,"file":"index.d.mts","names":[],"sources":["../src/segmentation/fuzzy.ts","../src/segmentation/types.ts","../src/segmentation/pattern-validator.ts","../src/segmentation/replace.ts","../src/segmentation/segmenter.ts","../src/segmentation/tokens.ts","../src/analysis/line-starts.ts","../src/analysis/repeating-sequences.ts","../src/detection.ts","../src/recovery.ts"],"sourcesContent":[],"mappings":";;AAkEA;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA6DzB;AAAkD;AAyHlD;;;;;AAkBA;AAoCA;AA0EY,cDzZC,WCyZqB,EAAA,CAAA,CAAA,EAAA,MAAc,EAAA,GAAA,MAAA;AA8BhD;AAuBA;AA+CA;;;;;;;AAwJA;;;;AC1sBA;AAKA;AAUA;;;;;;AA2GA;;;;AC3HA;AA2DA;;;;;;;;AC+SA;AAAoC,cJrNvB,wBIqNuB,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;AJpTpC;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA6DzB;AAAkD;AAyHlD;;;;;AAkBA,KA/UK,YAAA,GA+UW;EAoCJ;EA0EA,KAAA,EAAA,MAAU;AA8BtB,CAAA;AAuBA;AA+CA;;;;;;;AAwJA;;;;AC1sBA;AAKA;AAUA;;;;;;AA2GA;;;KD7EK,eAAA;EE9CO;EA2DC,QAAA,EAAA,MAAA;CAA4B;;;;;;;AC+SzC;;;;;;;;AClTA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;;KJlnBK,qBAAA;;EKnEO,cAAA,EAAA,MAAA,EAAA;AAcZ,CAAA;AAEA;AAwQA;;;;;;;;AClRA;AAaA;AAOA;AA2OA;;;;;;;;AC9QA;AA+EA;AAgEA;AAuBA;AAiCA;;;;AC7MA,KRmHK,sBAAA,GQnH6B;EAKtB;EACC,eAAA,EAAA,MAAA,EAAA;CACF;;;;AAKX;AA2BE;AAmnBF;;;;;;;;;AAsDA;;;;;KRrkBK,mBAAA,GQwkB+C;;;;;;;;;;;;;;KRzjB/C,WAAA,GACC,eACA,kBACA,wBACA,yBACA;;;;;;;KAYD,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAiDO,SAAA;;;;;;;KAYP,eAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAyCS;;;;;;;;;;;;SAaH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAuDC,SAAA,GAAY,cAAc,gBAAgB;;;;;;;;;;;;;KAkB1C,IAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAoCA,cAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAqCE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCF,UAAA,YAAsB;;;;;;;;;;;;;;;;;;;;;;;;;UA8BjB,MAAA;;;;;;;;;;;;;;;;;;;;;;KAuBL,WAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+CA,mBAAA;;;;;;YAME;;;;;;;;UASF;;;;;;;;;;;;;;cAiBY;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA+CN;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;WAwDL;;;;;;;;;;;;;;;;KAiBD,OAAA;;;;;;;;;;;;;;;;;;;;;;;;;;SA6BD;;;;AAttBM;AA4BG;AA8BM;AAyDrB,KCpIO,mBAAA,GDoIY,gBAAA,GAAA,eAAA,GAAA,WAAA,GAAA,eAAA;AAAA;;;AAkBlB,KCjJM,eAAA,GDiJN;EACA,IAAA,ECjJI,mBDiJJ;EACA,OAAA,EAAA,MAAA;EAAmB,UAAA,CAAA,EAAA,MAAA;AAAA,CAAA;AA6DzB;AAAkD;AAyHlD;;AAAsC,KC/T1B,oBAAA,GD+T0B;EAAgB,cAAA,CAAA,EAAA,CC9ThC,eD8TgC,GAAA,SAAA,CAAA,EAAA;EAAe,eAAA,CAAA,EAAA,CC7T9C,eD6T8C,GAAA,SAAA,CAAA,EAAA;EAkBzD,YAAI,CAAA,EAAA,CC9UI,eD8UJ,GAAA,SAAA,CAAA,EAAA;EAoCJ,QAAA,CAAA,ECjXG,eDiXW;AA0E1B,CAAA;AA8BA;AAuBA;AA+CA;;;;;;;AAwJA;;;;AC1sBA;AAKA;AAUA;;;;AAIe,cAuGF,aAvGE,EAAA,CAAA,KAAA,EAuGsB,SAvGtB,EAAA,EAAA,GAAA,CAuGqC,oBAvGrC,GAAA,SAAA,CAAA,EAAA;;;AFkCf;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAuCtB,KEpJO,WAAA,GAAc,WFoJV,CEpJsB,mBFoJtB,CAAA,SAAA,CAAA,CAAA,CAAA,MAAA,CAAA;;;;;;;AAKS;AA6DzB;AAAkD;AAyHlD;AAAwB,cEpRX,iBFoRW,EAAA,CAAA,KAAA,EEpRiB,IFoRjB,EAAA,EAAA,KAAA,CAAA,EEpRiC,WFoRjC,EAAA,EAAA,GEpRiD,IFoRjD,EAAA;;;;;AAkBxB;AAoCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;;AAwJA;;;;AC1sBA;AAKA;AAUA;;;;;;AA2GA;;;;AC3HA;AA2DA;;;;;;;;AC+SA;;;;;;;;AClTA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBa,cDpVA,YCoVsF,EAAA,CAAA,KAAA,EDpV/D,ICoV+D,EAAA,EAAA,OAAA,EDpV9C,mBCoV8C,EAAA,GDpV3B,OCoV2B,EAAA;;;;ALxoBnG;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA6DzB;AAAkD;AAyHlD;;;;;AAkBA;AAoCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;;AAwJA;;;;AC1sBA;AAKA;AAUA;;;;;;AA2GA;;;;AC3HA;AA2DA;;;;;;;;AC+Sa,cClTA,sBDmXZ,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;;;;ACnXD;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBa,cAvVA,+BAuVmC,EAAA,CAAA,QAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AAqBhD;AAgBA;;;;ACrrBA;AAcA;AAEA;AAwQA;;;;;;;;AClRA;AAaA;AAOA;AA2OA;;;;;;cF0Ga,gBAAgB;;AGxX7B;AA+EA;AAgEA;AAuBA;AAiCA;;;;AC7MA;AAKA;;;;;AAIoC,cJgavB,cIhauB,EAAA,CAAA,KAAA,EAAA,MAAA,EAAA,GAAA,OAAA;AAGpC;AA2BE;AAmnBF;;;;AAIc,KJ1OF,YAAA,GI0OE;EAGa;;;;AA+C3B;EACU,OAAA,EAAA,MAAA;EACsE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cJvHnE,mHAIV;;;;;;;;;;;;;;;;;;;;cAyCU;;;;;;;;;;;;;;;;;;;;;;cAuBA,uCAAmC;;;;;;;;;;;;;cAqBnC;;;;;;;;;;;;;;;cAgBA;;;ALxoBA,KM7CD,wBAAA,GN6C8E;EA+F7E,IAAA,CAAA,EAAA,MAAA;;;;ECnIR,WAAA,CAAA,EAAA,MAAY;EA4BZ,wBAAe,CAAA,EAAA,OAAA;EA8Bf,yBAAqB,CAAA,EAAA,OAAA;EAiCrB,MAAA,CAAA,EAAA,aAAA,GAAsB,OAAA;EAwBtB,UAAA,CAAA,EAAA,CAAA,IAAA,EAAA,MAAmB,EAAA,MAAA,EAAA,MAAA,EAAA,GAAA,OAAA;EAenB,cAAW,CAAA,EKjIK,MLiIL,EAAA;EACV,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;CACA;AACA,KKhIM,uBAAA,GLgIN;EACA,IAAA,EAAA,MAAA;EACA,MAAA,EAAA,MAAA;CAAmB;AAYpB,KK5IO,sBAAA,GL4IM;EAiDN,OAAA,EAAA,MAAS;EAYhB,KAAA,EAAA,MAAA;EA6GO,QAAA,EKnTE,uBLmTO,EAAA;CAAG;;;;AAkBZ,cKhEC,uBLgEG,EAAA,CAAA,KAAA,EK/DL,IL+DK,EAAA,EAAA,OAAA,CAAA,EK9DH,wBL8DG,EAAA,GK7Db,sBL6Da,EAAA;;;AA5NX,KMtHO,wBAAA,GNsHY;EAenB,WAAA,CAAA,EAAW,MAAA;EACV,WAAA,CAAA,EAAA,MAAA;EACA,QAAA,CAAA,EAAA,MAAA;EACA,IAAA,CAAA,EAAA,MAAA;EACA,yBAAA,CAAA,EAAA,OAAA;EACA,YAAA,CAAA,EAAA,OAAA;EAAmB,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;EAYpB,WAAA,CAAA,EAAA,MAAa;EAiDN,YAAS,CAAA,EAAA,MAAA;EAYhB,iBAAA,CAAe,EAAA,MAAA;AA6GpB,CAAA;AAAwB,KMnTZ,wBAAA,GNmTY;EAAc,IAAA,EAAA,MAAA;EAAgB,OAAA,EAAA,MAAA;EAAe,MAAA,EAAA,MAAA;EAkBzD,YAAI,EAAA,MAAA,EAAA;AAoChB,CAAA;AA0EY,KM5aA,wBAAA,GN4asB;EA8BjB,OAAA,EAAM,MAAA;EAuBX,KAAA,EAAA,MAAA;EA+CA,QAAA,EM7gBE,wBN6gBiB,EAAA;CAMjB;;;;AAkJd;;;cM7ba,mCACF,kBACG,6BACX;;;;AP3NH;AA+FA;;;;;ACnIiB;AA4BG;AA+Df,KO7GO,eAAA,GP6Ge;EAwBtB;EAeA,KAAA,EAAA,MAAA;EACC;EACA,KAAA,EAAA,MAAA;EACA;EACA,KAAA,EAAA,MAAA;EACA;EAAmB,QAAA,EAAA,MAAA;AAAA,CAAA;AA6DzB;AAAkD;AAyHlD;;;;;AAkBA;AAoCA;AA0EA;AA8BA;AAuBA;AA+CA;;;AAgCwB,cOpgBX,mBPogBW,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GOpgBwB,ePogBxB,EAAA;;;;AAwHxB;;;;AC1sBA;AAKA;AAUA;;;;AAIe,cM2HF,wBN3HE,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,QAAA,EM2HkD,eN3HlD,EAAA,EAAA,GAAA,MAAA;;AAuGf;;;;AC3HA;AA2Da,cK2GA,oBLrFZ,EAAA,CAAA,QAAA,EKsFa,eLtFb,EAAA,EAAA,GAAA;EAtBwC,WAAA,EAAA,gBAAA,GAAA,iBAAA;EAAgB,KAAA,EAAA,OAAA;EAAgB,QAAA,CAAA,EAAA,MAAA;CAAI;;;;AC+S7E;;;AAAwE,cInK3D,kBJmK2D,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA;EAAA,QAAA,EAAA,MAAA;;;;EClT3D,QAAA,EGsJC,eH7Ib,EAAA;AAiQD,CAAA,GAAa,IAAA;;;AL5QA,KS5DD,sBAAA,GT4D8E;EA+F7E,IAAA,EAAA,cAAA;;;;ECnIR,KAAA,CAAA,EAAA,OAAY,GAAA,YAAA;EA4BZ,QAAA,EAAA,MAAA,EAAe;AAAA,CAAA,GA8Bf;EAiCA,IAAA,EAAA,WAAA;EAwBA,SAAA,EAAA,CAAA,IAAA,EQxIwC,SRwIrB,EAAA,KAAA,EAAA,MAAA,EAAA,GAAA,OAAA;AAAA,CAAA;AAgBlB,KQtJM,iBAAA,GRsJN;EACA,OAAA,EQtJO,mBRsJP;EACA,KAAA,EQtJK,IRsJL,EAAA;EACA,QAAA,EQtJQ,ORsJR,EAAA;EACA,QAAA,EQtJQ,sBRsJR;CAAmB;AAYpB,KQ/JO,oBAAA,GR+JM;EAiDN,OAAA,EAAA;IAYP,IAAA,EAAA,YAAe,GAAA,wBAsDH;IAuDL,SAAS,EAAA,MAAA;IAAG,aAAA,EAAA,MAAA;IAAc,SAAA,EAAA,MAAA;IAAgB,UAAA,EAAA,MAAA;EAAe,CAAA;EAkBzD,KAAA,CAAA,EQnVA,KRmVI,CAAA;IAoCJ,SAAA,EAAA,MAAc;IA0Ed,QAAA,EAAU,MAAA;IA8BL,aAAM,EAAA,MAAA;IAuBX,UAAW,EAAA,MAAA;EA+CX,CAAA,CAAA;EAME,OAAA,EQriBD,KRqiBC,CAAA;IASF,IAAA,EAAA,MAAA;IAiBY,oBAAA,EAAA,MAAA;IA+CN,sBAAA,CAAA,EAAA,MAAA;IAwDL,qBAAA,CAAA,EAAA,MAAA;IAAM,YAAA,EAAA,MAAA;IAiBP,MAAO,EAAA,WA6BR,GAAA,oBAAM,GAAA,WAAA,GAAA,sBAAA,GAAA,qBAAA;;;;ECvuBL,CAAA,CAAA;EAKA,MAAA,EAAA,MAAA,EAAA;EAUA,QAAA,EAAA,MAAA,EAAA;CACU;KOkBjB,oBAAA,GPjBkB,MAAA,GAAA,YAAA,GAAA,qBAAA;AACH,iBOioBJ,qCAAA,CPjoBI,KAAA,EOkoBT,IPloBS,EAAA,EAAA,QAAA,EOmoBN,OPnoBM,EAAA,EAAA,OAAA,EOooBP,mBPpoBO,EAAA,QAAA,EOqoBN,sBProBM,EAAA,IAwGpB,CAxGoB,EAAA;EACL,IAAA,CAAA,EAAA,YAAA,GAAA,wBAAA;EAAe,gBAAA,CAAA,EOuoBH,oBPvoBG;AAuG9B,CAAA,CAAA,EAAa;UOkiBA;YAAgC;;AN7pBjC,iBM0sBI,6BAAA,CN1sBU,IAAA,EM2sBhB,iBN3sB2B,EAAA,EAAA,IA2DoB,CA3DpB,EAAA;EA2DxB,IAAA,CAAA,EAAA,YAAA,GAsBZ,wBAAA;EAtBwC,gBAAA,CAAA,EMipBuC,oBNjpBvC;CAAgB,CAAA,EAAA;EAAgB,MAAA,EMkpB5D,oBNlpB4D;EAAI,QAAA,EMkpBhC,ONlpBgC,EAAA"}

package/dist/index.mjs

@@ -645,6 +645,10 @@ const buildBareTokenRegex = () => {
 * Validates a single pattern for common issues.
 */
 const validatePattern = (pattern, seenPatterns) => {
+	if (!pattern.trim()) return {
+		message: "Empty pattern is not allowed",
+		type: "empty_pattern"
+	};
 	if (seenPatterns.has(pattern)) return {
 		message: `Duplicate pattern: "${pattern}"`,
 		type: "duplicate"
@@ -727,7 +731,7 @@ const validateRules = (rules) => {
 				hasIssues = true;
 			}
 		}
-		if ("template" in rule && rule.template) {
+		if ("template" in rule && rule.template !== void 0) {
 			const seenPatterns = /* @__PURE__ */ new Set();
 			const issue = validatePattern(rule.template, seenPatterns);
 			if (issue) {
@@ -1245,16 +1249,71 @@ const handlePageBoundaryBreak = (remainingContent, windowEndIdx, windowEndPositi
 */
 const findBreakPosition = (remainingContent, currentFromIdx, toIdx, windowEndIdx, windowEndPosition, ctx) => {
 	const { pageIds, normalizedPages, expandedBreakpoints, prefer } = ctx;
-	for (const { rule, regex, excludeSet, skipWhenRegex } of expandedBreakpoints) {
+	for (let i = 0; i < expandedBreakpoints.length; i++) {
+		const { rule, regex, excludeSet, skipWhenRegex } = expandedBreakpoints[i];
 		if (!isInBreakpointRange(pageIds[currentFromIdx], rule)) continue;
 		if (hasExcludedPageInRange(excludeSet, pageIds, currentFromIdx, windowEndIdx)) continue;
 		if (skipWhenRegex?.test(remainingContent)) continue;
-		if (regex === null) return handlePageBoundaryBreak(remainingContent, windowEndIdx, windowEndPosition, toIdx, pageIds, normalizedPages);
+		if (regex === null) return {
+			breakpointIndex: i,
+			breakPos: handlePageBoundaryBreak(remainingContent, windowEndIdx, windowEndPosition, toIdx, pageIds, normalizedPages),
+			rule
+		};
 		const breakPos = findPatternBreakPosition(remainingContent.slice(0, Math.min(windowEndPosition, remainingContent.length)), regex, prefer);
-		if (breakPos > 0) return breakPos;
+		if (breakPos > 0) return {
+			breakpointIndex: i,
+			breakPos,
+			rule
+		};
 	}
-	return -1;
+	return null;
+};
+
+//#endregion
+//#region src/segmentation/debug-meta.ts
+const resolveDebugConfig = (debug) => {
+	if (!debug) return null;
+	if (debug === true) return {
+		includeBreakpoint: true,
+		includeRule: true,
+		metaKey: "_flappa"
+	};
+	if (typeof debug !== "object") return null;
+	const metaKey = debug.metaKey;
+	const include = debug.include;
+	const includeRule = Array.isArray(include) ? include.includes("rule") : true;
+	return {
+		includeBreakpoint: Array.isArray(include) ? include.includes("breakpoint") : true,
+		includeRule,
+		metaKey: typeof metaKey === "string" && metaKey ? metaKey : "_flappa"
+	};
+};
+const getRulePatternType = (rule) => {
+	if ("lineStartsWith" in rule) return "lineStartsWith";
+	if ("lineStartsAfter" in rule) return "lineStartsAfter";
+	if ("lineEndsWith" in rule) return "lineEndsWith";
+	if ("template" in rule) return "template";
+	return "regex";
+};
+const isPlainObject = (v) => Boolean(v) && typeof v === "object" && !Array.isArray(v);
+const mergeDebugIntoMeta = (meta, metaKey, patch) => {
+	const out = meta ? { ...meta } : {};
+	const existing = out[metaKey];
+	out[metaKey] = {
+		...isPlainObject(existing) ? existing : {},
+		...patch
+	};
+	return out;
 };
+const buildRuleDebugPatch = (ruleIndex, rule) => ({ rule: {
+	index: ruleIndex,
+	patternType: getRulePatternType(rule)
+} });
+const buildBreakpointDebugPatch = (breakpointIndex, rule) => ({ breakpoint: {
+	index: breakpointIndex,
+	kind: rule.pattern === "" ? "pageBoundary" : "pattern",
+	pattern: rule.pattern
+} });
 
 //#endregion
 //#region src/segmentation/breakpoint-processor.ts
@@ -1338,15 +1397,20 @@ const createPieceSegment = (pieceContent, actualStartIdx, actualEndIdx, pageIds,
 const findBreakOffsetForWindow = (remainingContent, currentFromIdx, windowEndIdx, toIdx, windowEndPosition, pageIds, expandedBreakpoints, cumulativeOffsets, normalizedPages, prefer) => {
 	if (hasAnyExclusionsInRange(expandedBreakpoints, pageIds, currentFromIdx, windowEndIdx)) {
 		const exclusionBreak = findExclusionBreakPosition(currentFromIdx, windowEndIdx, toIdx, pageIds, expandedBreakpoints, cumulativeOffsets);
-		if (exclusionBreak > 0) return exclusionBreak;
+		if (exclusionBreak > 0) return { breakOffset: exclusionBreak };
 	}
-	const patternBreak = findBreakPosition(remainingContent, currentFromIdx, toIdx, windowEndIdx, windowEndPosition, {
+	const patternMatch = findBreakPosition(remainingContent, currentFromIdx, toIdx, windowEndIdx, windowEndPosition, {
 		expandedBreakpoints,
 		normalizedPages,
 		pageIds,
 		prefer
 	});
-	return patternBreak > 0 ? patternBreak : windowEndPosition;
+	if (patternMatch && patternMatch.breakPos > 0) return {
+		breakOffset: patternMatch.breakPos,
+		breakpointIndex: patternMatch.breakpointIndex,
+		breakpointRule: patternMatch.rule
+	};
+	return { breakOffset: windowEndPosition };
 };
 /**
 * Advances cursor position past any leading whitespace.
@@ -1362,12 +1426,13 @@ const skipWhitespace$1 = (content, startPos) => {
 *
 * Uses precomputed boundary positions for O(log n) page attribution lookups.
 */
-const processOversizedSegment = (segment, fromIdx, toIdx, pageIds, normalizedPages, cumulativeOffsets, expandedBreakpoints, maxPages, prefer, logger) => {
+const processOversizedSegment = (segment, fromIdx, toIdx, pageIds, normalizedPages, cumulativeOffsets, expandedBreakpoints, maxPages, prefer, logger, debugMetaKey) => {
 	const result = [];
 	const fullContent = segment.content;
 	let cursorPos = 0;
 	let currentFromIdx = fromIdx;
 	let isFirstPiece = true;
+	let lastBreakpoint = null;
 	const boundaryPositions = buildBoundaryPositions(fullContent, fromIdx, toIdx, pageIds, normalizedPages, cumulativeOffsets);
 	logger?.debug?.("[breakpoints] boundaryPositions built", {
 		boundaryPositions,
@@ -1382,7 +1447,9 @@ const processOversizedSegment = (segment, fromIdx, toIdx, pageIds, normalizedPag
 		const remainingSpan = computeRemainingSpan(currentFromIdx, toIdx, pageIds);
 		const remainingHasExclusions = hasAnyExclusionsInRange(expandedBreakpoints, pageIds, currentFromIdx, toIdx);
 		if (remainingSpan <= maxPages && !remainingHasExclusions) {
-			const finalSeg = createFinalSegment(remainingContent, currentFromIdx, toIdx, pageIds, segment.meta, isFirstPiece);
+			const includeMeta = isFirstPiece || Boolean(debugMetaKey);
+			const meta = debugMetaKey && lastBreakpoint ? mergeDebugIntoMeta(includeMeta ? segment.meta : void 0, debugMetaKey, buildBreakpointDebugPatch(lastBreakpoint.breakpointIndex, lastBreakpoint.rule)) : includeMeta ? segment.meta : void 0;
+			const finalSeg = createFinalSegment(remainingContent, currentFromIdx, toIdx, pageIds, meta, includeMeta);
 			if (finalSeg) result.push(finalSeg);
 			break;
 		}
@@ -1393,8 +1460,12 @@ const processOversizedSegment = (segment, fromIdx, toIdx, pageIds, normalizedPag
 			cursorPos,
 			windowEndIdx
 		});
-		const breakOffset = findBreakOffsetForWindow(remainingContent, currentFromIdx, windowEndIdx, toIdx, windowEndPosition, pageIds, expandedBreakpoints, cumulativeOffsets, normalizedPages, prefer);
-		const breakPos = cursorPos + breakOffset;
+		const found = findBreakOffsetForWindow(remainingContent, currentFromIdx, windowEndIdx, toIdx, windowEndPosition, pageIds, expandedBreakpoints, cumulativeOffsets, normalizedPages, prefer);
+		if (found.breakpointIndex !== void 0 && found.breakpointRule) lastBreakpoint = {
+			breakpointIndex: found.breakpointIndex,
+			rule: found.breakpointRule
+		};
+		const breakPos = cursorPos + found.breakOffset;
 		const pieceContent = fullContent.slice(cursorPos, breakPos).trim();
 		const { actualEndIdx, actualStartIdx } = computePiecePages(cursorPos, breakPos, boundaryPositions, fromIdx, toIdx);
 		logger?.trace?.("[breakpoints] piece", {
@@ -1403,7 +1474,8 @@ const processOversizedSegment = (segment, fromIdx, toIdx, pageIds, normalizedPag
 			pieceLength: pieceContent.length
 		});
 		if (pieceContent) {
-			const pieceSeg = createPieceSegment(pieceContent, actualStartIdx, actualEndIdx, pageIds, segment.meta, isFirstPiece);
+			const includeMeta = isFirstPiece || Boolean(debugMetaKey);
+			const pieceSeg = createPieceSegment(pieceContent, actualStartIdx, actualEndIdx, pageIds, debugMetaKey && lastBreakpoint ? mergeDebugIntoMeta(includeMeta ? segment.meta : void 0, debugMetaKey, buildBreakpointDebugPatch(lastBreakpoint.breakpointIndex, lastBreakpoint.rule)) : includeMeta ? segment.meta : void 0, includeMeta);
 			if (pieceSeg) result.push(pieceSeg);
 		}
 		cursorPos = skipWhitespace$1(fullContent, breakPos);
@@ -1418,7 +1490,7 @@ const processOversizedSegment = (segment, fromIdx, toIdx, pageIds, normalizedPag
 *
 * Note: This is an internal engine used by `segmentPages()`.
 */
-const applyBreakpoints = (segments, pages, normalizedContent, maxPages, breakpoints, prefer, patternProcessor, logger, pageJoiner = "space") => {
+const applyBreakpoints = (segments, pages, normalizedContent, maxPages, breakpoints, prefer, patternProcessor, logger, pageJoiner = "space", debugMetaKey) => {
 	const pageIds = pages.map((p) => p.id);
 	const pageIdToIndex = buildPageIdToIndexMap(pageIds);
 	const normalizedPages = buildNormalizedPagesMap(pages, normalizedContent);
@@ -1446,7 +1518,7 @@ const applyBreakpoints = (segments, pages, normalizedContent, maxPages, breakpoi
 			result.push(segment);
 			continue;
 		}
-		const broken = processOversizedSegment(segment, fromIdx, toIdx, pageIds, normalizedPages, cumulativeOffsets, expandedBreakpoints, maxPages, prefer, logger);
+		const broken = processOversizedSegment(segment, fromIdx, toIdx, pageIds, normalizedPages, cumulativeOffsets, expandedBreakpoints, maxPages, prefer, logger, debugMetaKey);
 		result.push(...broken.map((s) => {
 			const segFromIdx = pageIdToIndex.get(s.from) ?? -1;
 			const segToIdx = s.to !== void 0 ? pageIdToIndex.get(s.to) ?? segFromIdx : segFromIdx;
@@ -2059,13 +2131,25 @@ const findMatchesInContent = (content, regex, usesCapture, captureNames) => {
 	}
 	return matches;
 };
-const applyOccurrenceFilter = (rules, splitPointsByRule) => {
+const applyOccurrenceFilter = (rules, splitPointsByRule, debugMetaKey) => {
 	const result = [];
 	rules.forEach((rule, index) => {
 		const points = splitPointsByRule.get(index);
 		if (!points?.length) return;
 		const filtered = rule.occurrence === "first" ? [points[0]] : rule.occurrence === "last" ? [points.at(-1)] : points;
-		result.push(...filtered);
+		if (!debugMetaKey) {
+			result.push(...filtered.map((p) => ({
+				...p,
+				ruleIndex: index
+			})));
+			return;
+		}
+		const debugPatch = buildRuleDebugPatch(index, rule);
+		result.push(...filtered.map((p) => ({
+			...p,
+			meta: mergeDebugIntoMeta(p.meta, debugMetaKey, debugPatch),
+			ruleIndex: index
+		})));
 	});
 	return result;
 };
@@ -2203,7 +2287,7 @@ const ensureFallbackSegment = (segments, pages, normalizedContent, pageJoiner) =
 	if (lastPage.id !== firstPage.id) initialSeg.to = lastPage.id;
 	return [initialSeg];
 };
-const collectSplitPointsFromRules = (rules, matchContent, pageMap, logger) => {
+const collectSplitPointsFromRules = (rules, matchContent, pageMap, debugMetaKey, logger) => {
 	logger?.debug?.("[segmenter] collecting split points from rules", {
 		contentLength: matchContent.length,
 		ruleCount: rules.length
@@ -2218,7 +2302,7 @@ const collectSplitPointsFromRules = (rules, matchContent, pageMap, logger) => {
 	const splitPointsByRule = collectFastFuzzySplitPoints(matchContent, pageMap, fastFuzzyRules, passesPageStartGuard);
 	if (combinableRules.length > 0) processCombinedMatches(matchContent, combinableRules, buildRuleRegexes(combinableRules), pageMap, passesPageStartGuard, splitPointsByRule, logger);
 	for (const rule of standaloneRules) processStandaloneRule(rule, rules.indexOf(rule), matchContent, pageMap, passesPageStartGuard, splitPointsByRule);
-	return applyOccurrenceFilter(rules, splitPointsByRule);
+	return applyOccurrenceFilter(rules, splitPointsByRule, debugMetaKey);
 };
 /**
 * Finds page breaks within a given offset range using binary search.
@@ -2321,6 +2405,8 @@ const convertPageBreaks = (content, startOffset, pageBreaks) => {
 */
 const segmentPages = (pages, options) => {
 	const { rules = [], maxPages = 0, breakpoints = [], prefer = "longer", pageJoiner = "space", logger } = options;
+	const debug = resolveDebugConfig(options.debug);
+	const debugMetaKey = debug?.includeRule ? debug.metaKey : void 0;
 	logger?.info?.("[segmenter] starting segmentation", {
 		breakpointCount: breakpoints.length,
 		maxPages,
@@ -2334,7 +2420,7 @@ const segmentPages = (pages, options) => {
 		pageIds: pageMap.pageIds,
 		totalContentLength: matchContent.length
 	});
-	const splitPoints = collectSplitPointsFromRules(rules, matchContent, pageMap, logger);
+	const splitPoints = collectSplitPointsFromRules(rules, matchContent, pageMap, debugMetaKey, logger);
 	const unique = dedupeSplitPoints(splitPoints);
 	logger?.debug?.("[segmenter] split points collected", {
 		rawSplitPoints: splitPoints.length,
@@ -2353,7 +2439,7 @@ const segmentPages = (pages, options) => {
 	if (maxPages >= 0 && breakpoints.length) {
 		logger?.debug?.("[segmenter] applying breakpoints to oversized segments");
 		const patternProcessor = (p) => processPattern(p, false).pattern;
-		const result = applyBreakpoints(segments, processedPages, normalizedContent, maxPages, breakpoints, prefer, patternProcessor, logger, pageJoiner);
+		const result = applyBreakpoints(segments, processedPages, normalizedContent, maxPages, breakpoints, prefer, patternProcessor, logger, pageJoiner, debug?.includeBreakpoint ? debug.metaKey : void 0);
 		logger?.info?.("[segmenter] segmentation complete (with breakpoints)", { finalSegmentCount: result.length });
 		return result;
 	}