flappa-doormal 2.7.0 → 2.8.0

package/AGENTS.md

@@ -28,6 +28,8 @@ src/
 ├── index.ts                    # Main entry point and exports
 ├── pattern-detection.ts        # Token detection for auto-generating rules (NEW)
 ├── pattern-detection.test.ts   # Pattern detection tests (22 tests)
+├── recovery.ts                 # Marker recovery utility (recover mistaken lineStartsAfter)
+├── recovery.test.ts            # Marker recovery tests
 └── segmentation/
     ├── types.ts                # TypeScript type definitions for rules/segments
     ├── segmenter.ts            # Core segmentation engine (segmentPages)
@@ -56,6 +58,11 @@ src/
    - Takes array of `{id, content}` pages and split rules
    - Returns array of `{content, from, to?, meta?}` segments
 
+1. **`recoverMistakenLineStartsAfterMarkers(pages, segments, options, selector)`** - Recovery helper
+   - Use when a client mistakenly used `lineStartsAfter` where `lineStartsWith` was intended
+   - Deterministic mode reruns segmentation with selected rules converted to `lineStartsWith` and merges recovered `content` back into the provided segments
+   - Optional `mode: 'best_effort_then_rerun'` attempts a conservative anchor-based recovery first, then falls back to rerun for unresolved segments
+
 2. **`tokens.ts`** - Template system
    - `TOKEN_PATTERNS` - Map of token names to regex patterns
    - `expandTokensWithCaptures()` - Expands `{{token:name}}` syntax
@@ -362,6 +369,39 @@ bunx biome lint .
 
 11. **Boundary-position algorithm improves page attribution**: Building a position map of page boundaries once per segment (O(n)) enables binary search for O(log n) lookups per piece. Key insight: when a segment starts mid-page (common after structural rules), expected boundary estimates must account for the offset into the starting page. Without this adjustment, position-based lookups can return the wrong page when pages have identical content prefixes.
 
+### For Future AI Agents (Recovery + Repo gotchas)
+
+1. **`lineStartsAfter` vs `lineStartsWith` is not “cosmetic”**: `lineStartsAfter` changes output by stripping the matched marker via an internal `contentStartOffset` during segment construction. If a client used it by accident, you cannot reconstruct the exact stripped prefix from output alone without referencing the original pages and re-matching the marker.
+
+2. **Recovery must mirror segmentation’s preprocessing**: If `SegmentationOptions.replace` was used, recovery must apply the same replacements (see `src/segmentation/replace.ts`) before attempting anchoring or rerun alignment, otherwise substring matching and page joins will drift.
+
+3. **Page joining differs between matching and output**:
+   - Matching always happens on pages concatenated with `\\n` separators.
+   - Output segments may normalize page boundaries (`pageJoiner: 'space' | 'newline'`) and breakpoints post-processing uses its own join normalization utilities.
+   Recovery code must be explicit about which representation it’s searching.
+
+4. **Breakpoints can produce “pieces” that were never marker-stripped**: When `maxPages` + `breakpoints` are enabled, only the piece that starts at the original structural boundary could have lost a marker due to `lineStartsAfter`. Mid-segment breakpoint pieces should not be “recovered” unless you can anchor them confidently.
+
+5. **Fuzzy defaults are easy to miss**: Some tokens auto-enable fuzzy matching unless `fuzzy: false` is set (`bab`, `basmalah`, `fasl`, `kitab`, `naql`). If you are validating markers or re-matching prefixes, use the same compilation path as segmentation (`buildRuleRegex` / `processPattern`) so diacritics and token expansion behave identically.
+
+6. **Auto-escaping applies to template-like patterns**: `lineStartsWith`, `lineStartsAfter`, `lineEndsWith`, and `template` auto-escape `()[]` outside `{{tokens}}`. Raw `regex` does not. If you compare patterns by string equality, be careful about escaping and whitespace.
+
+7. **TypeScript union pitfalls with `SplitRule`**: `SplitRule` is a union where only one pattern type should exist. Avoid mutating rules in-place with `delete` on fields (TS often narrows unions and then complains). Prefer rebuilding converted rules via destructuring (e.g. `{ lineStartsAfter, ...rest }` then create `{...rest, lineStartsWith: lineStartsAfter}`).
+
+8. **Biome lint constraints shape implementation**: The repo enforces low function complexity. Expect to extract helpers (alignment, selector resolution, anchoring) to keep Biome happy. Also, Biome can flag regex character-class usage as misleading; prefer alternation (e.g. `(?:\\u200C|\\u200D|\\uFEFF)`) when removing specific codepoints.
+
+9. **When debugging recovery, start here**:
+   - `src/segmentation/segmenter.ts` (how content is sliced/trimmed and how `from/to` are computed)
+   - `src/segmentation/rule-regex.ts` + `src/segmentation/tokens.ts` (token expansion + fuzzy behavior)
+   - `src/segmentation/replace.ts` (preprocessing parity)
+   - `src/recovery.ts` (recovery implementation)
+
+### Process Template (Multi-agent design review, TDD-first)
+
+If you want to repeat the “write a plan → get multiple AI critiques → synthesize → update plan → implement TDD-first” workflow, use:
+
+- `docs/ai-multi-agent-tdd-template.md`
+
 ### Architecture Insights
 
 - **Declarative > Imperative**: Users describe patterns, library handles regex

package/README.md

@@ -697,6 +697,53 @@ const options: SegmentationOptions = {
 const segments: Segment[] = segmentPages(pages, options);
 ```
 
+### Marker recovery (when `lineStartsAfter` was used by accident)
+
+If you accidentally used `lineStartsAfter` for markers that should have been preserved (e.g. Arabic connective phrases like `وروى` / `وذكر`), you can recover those missing prefixes from existing segments.
+
+#### `recoverMistakenLineStartsAfterMarkers(pages, segments, options, selector, opts?)`
+
+This function returns new segments with recovered `content` plus a `report` describing what happened.
+
+**Recommended (deterministic) mode**: rerun segmentation with selected rules converted to `lineStartsWith`, then merge recovered content back.
+
+```ts
+import { recoverMistakenLineStartsAfterMarkers, segmentPages } from 'flappa-doormal';
+
+const pages = [{ id: 1, content: 'وروى أحمد\nوذكر خالد' }];
+const options = { rules: [{ lineStartsAfter: ['وروى '] }, { lineStartsAfter: ['وذكر '] }] };
+
+const segments = segmentPages(pages, options);
+// segments[0].content === 'أحمد' (marker stripped)
+
+const { segments: recovered, report } = recoverMistakenLineStartsAfterMarkers(
+  pages,
+  segments,
+  options,
+  { type: 'rule_indices', indices: [0] }, // recover only the first rule
+);
+
+// recovered[0].content === 'وروى أحمد'
+// recovered[1].content === 'خالد'  (unchanged)
+console.log(report.summary);
+```
+
+**Optional**: best-effort anchoring mode attempts to recover without rerunning first, then falls back to rerun for unresolved segments:
+
+```ts
+const { segments: recovered } = recoverMistakenLineStartsAfterMarkers(
+  pages,
+  segments,
+  options,
+  { type: 'rule_indices', indices: [0] },
+  { mode: 'best_effort_then_rerun' }
+);
+```
+
+Notes:
+- Recovery is **explicitly scoped** by the `selector`; it will not “guess” which rules are mistaken.
+- If your segments were heavily post-processed (trimmed/normalized/reordered), recovery may return unresolved items; see the report for details.
+
 ### `stripHtmlTags(html)`
 
 Remove all HTML tags from content, keeping only text.

package/dist/index.d.mts

@@ -1149,70 +1149,18 @@ declare const getAvailableTokens: () => string[];
  */
 declare const getTokenPattern: (tokenName: string) => string | undefined;
 //#endregion
-//#region src/analysis.d.ts
+//#region src/analysis/line-starts.d.ts
 type LineStartAnalysisOptions = {
-  /** Return top K patterns (after filtering). Default: 20 */
   topK?: number;
-  /** Only consider the first N characters of each trimmed line. Default: 60 */
   prefixChars?: number;
-  /** Ignore lines shorter than this (after trimming). Default: 6 */
   minLineLength?: number;
-  /** Only include patterns that appear at least this many times. Default: 3 */
   minCount?: number;
-  /** Keep up to this many example lines per pattern. Default: 5 */
   maxExamples?: number;
-  /**
-   * If true, include a literal first word when no token match is found at the start.
-   * Default: true
-   */
   includeFirstWordFallback?: boolean;
-  /**
-   * If true, strip Arabic diacritics (harakat/tashkeel) for the purposes of matching tokens.
-   * This helps patterns like `وأَخْبَرَنَا` match the `{{naql}}` token (`وأخبرنا`).
-   *
-   * Note: examples are still stored in their original (unstripped) form.
-   *
-   * Default: true
-   */
   normalizeArabicDiacritics?: boolean;
-  /**
-   * How to sort patterns before applying `topK`.
-   *
-   * - `specificity` (default): prioritize more structured prefixes first (tokenCount, then literalLen), then count.
-   * - `count`: prioritize highest-frequency patterns first, then specificity.
-   */
   sortBy?: 'specificity' | 'count';
-  /**
-   * Optional filter to restrict which lines are analyzed.
-   *
-   * The `line` argument is the trimmed + whitespace-collapsed version of the line.
-   * Return `true` to include it, `false` to skip it.
-   *
-   * @example
-   * // Only analyze markdown H2 headings
-   * { lineFilter: (line) => line.startsWith('## ') }
-   */
   lineFilter?: (line: string, pageId: number) => boolean;
-  /**
-   * Optional list of prefix matchers to consume before tokenization.
-   *
-   * This is for "syntactic" prefixes that are common at line start but are not
-   * meaningful as tokens by themselves (e.g. markdown headings like `##`).
-   *
-   * Each matcher is applied at the current position. If it matches, the matched
-   * text is appended (escaped) to the signature and the scanner advances.
-   *
-   * @example
-   * // Support markdown blockquotes and headings
-   * { prefixMatchers: [/^>+/u, /^#+/u] }
-   */
   prefixMatchers?: RegExp[];
-  /**
-   * How to represent whitespace in returned `pattern` signatures.
-   *
-   * - `regex` (default): use `\\s*` placeholders between tokens (useful if you paste patterns into regex-ish templates).
-   * - `space`: use literal single spaces (`' '`) between tokens (safer if you don't want `\\s` to match newlines when reused as regex).
-   */
   whitespace?: 'regex' | 'space';
 };
 type LineStartPatternExample = {
@@ -1226,12 +1174,41 @@ type CommonLineStartPattern = {
 };
 /**
  * Analyze pages and return the most common line-start patterns (top K).
- *
- * This is a pure algorithmic heuristic: it tokenizes common prefixes into a stable
- * template-ish string using the library tokens (e.g., `{{bab}}`, `{{raqms}}`, `{{rumuz}}`).
  */
 declare const analyzeCommonLineStarts: (pages: Page[], options?: LineStartAnalysisOptions) => CommonLineStartPattern[];
 //#endregion
+//#region src/analysis/repeating-sequences.d.ts
+type RepeatingSequenceOptions = {
+  minElements?: number;
+  maxElements?: number;
+  minCount?: number;
+  topK?: number;
+  normalizeArabicDiacritics?: boolean;
+  requireToken?: boolean;
+  whitespace?: 'regex' | 'space';
+  maxExamples?: number;
+  contextChars?: number;
+  maxUniquePatterns?: number;
+};
+type RepeatingSequenceExample = {
+  text: string;
+  context: string;
+  pageId: number;
+  startIndices: number[];
+};
+type RepeatingSequencePattern = {
+  pattern: string;
+  count: number;
+  examples: RepeatingSequenceExample[];
+};
+/**
+ * Analyze pages for commonly repeating word sequences.
+ *
+ * Use for continuous text without line breaks. For line-based analysis,
+ * use `analyzeCommonLineStarts()` instead.
+ */
+declare const analyzeRepeatingSequences: (pages: Page[], options?: RepeatingSequenceOptions) => RepeatingSequencePattern[];
+//#endregion
 //#region src/detection.d.ts
 /**
  * Pattern detection utilities for recognizing template tokens in Arabic text.
@@ -1307,5 +1284,67 @@ declare const analyzeTextForRule: (text: string) => {
   detected: DetectedPattern[];
 } | null;
 //#endregion
-export { type Breakpoint, type BreakpointRule, type CommonLineStartPattern, type DetectedPattern, type ExpandResult, type LineStartAnalysisOptions, type LineStartPatternExample, type Logger, type Page, type PageRange, type ReplaceRule, type RuleValidationResult, type Segment, type SegmentationOptions, type SplitRule, TOKEN_PATTERNS, type ValidationIssue, type ValidationIssueType, analyzeCommonLineStarts, analyzeTextForRule, applyReplacements, containsTokens, detectTokenPatterns, escapeRegex, escapeTemplateBrackets, expandCompositeTokensInTemplate, expandTokens, expandTokensWithCaptures, generateTemplateFromText, getAvailableTokens, getTokenPattern, makeDiacriticInsensitive, segmentPages, suggestPatternConfig, templateToRegex, validateRules };
+//#region src/recovery.d.ts
+type MarkerRecoverySelector = {
+  type: 'rule_indices';
+  indices: number[];
+} | {
+  type: 'lineStartsAfter_patterns';
+  match?: 'exact' | 'normalized';
+  patterns: string[];
+} | {
+  type: 'predicate';
+  predicate: (rule: SplitRule, index: number) => boolean;
+};
+type MarkerRecoveryRun = {
+  options: SegmentationOptions;
+  pages: Page[];
+  segments: Segment[];
+  selector: MarkerRecoverySelector;
+};
+type MarkerRecoveryReport = {
+  summary: {
+    mode: 'rerun_only' | 'best_effort_then_rerun';
+    recovered: number;
+    totalSegments: number;
+    unchanged: number;
+    unresolved: number;
+  };
+  byRun?: Array<{
+    recovered: number;
+    runIndex: number;
+    totalSegments: number;
+    unresolved: number;
+  }>;
+  details: Array<{
+    from: number;
+    originalStartPreview: string;
+    recoveredPrefixPreview?: string;
+    recoveredStartPreview?: string;
+    segmentIndex: number;
+    status: 'recovered' | 'skipped_idempotent' | 'unchanged' | 'unresolved_alignment' | 'unresolved_selector';
+    strategy: 'rerun' | 'stage1' | 'none';
+    to?: number;
+    notes?: string[];
+  }>;
+  errors: string[];
+  warnings: string[];
+};
+type NormalizeCompareMode = 'none' | 'whitespace' | 'whitespace_and_nfkc';
+declare function recoverMistakenLineStartsAfterMarkers(pages: Page[], segments: Segment[], options: SegmentationOptions, selector: MarkerRecoverySelector, opts?: {
+  mode?: 'rerun_only' | 'best_effort_then_rerun';
+  normalizeCompare?: NormalizeCompareMode;
+}): {
+  report: MarkerRecoveryReport;
+  segments: Segment[];
+};
+declare function recoverMistakenMarkersForRuns(runs: MarkerRecoveryRun[], opts?: {
+  mode?: 'rerun_only' | 'best_effort_then_rerun';
+  normalizeCompare?: NormalizeCompareMode;
+}): {
+  report: MarkerRecoveryReport;
+  segments: Segment[];
+};
+//#endregion
+export { type Breakpoint, type BreakpointRule, type CommonLineStartPattern, type DetectedPattern, type ExpandResult, type LineStartAnalysisOptions, type LineStartPatternExample, type Logger, type MarkerRecoveryReport, type MarkerRecoveryRun, type MarkerRecoverySelector, type Page, type PageRange, type RepeatingSequenceExample, type RepeatingSequenceOptions, type RepeatingSequencePattern, type ReplaceRule, type RuleValidationResult, type Segment, type SegmentationOptions, type SplitRule, TOKEN_PATTERNS, type ValidationIssue, type ValidationIssueType, analyzeCommonLineStarts, analyzeRepeatingSequences, analyzeTextForRule, applyReplacements, containsTokens, detectTokenPatterns, escapeRegex, escapeTemplateBrackets, expandCompositeTokensInTemplate, expandTokens, expandTokensWithCaptures, generateTemplateFromText, getAvailableTokens, getTokenPattern, makeDiacriticInsensitive, recoverMistakenLineStartsAfterMarkers, recoverMistakenMarkersForRuns, segmentPages, suggestPatternConfig, templateToRegex, validateRules };
 //# sourceMappingURL=index.d.mts.map

package/dist/index.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.mts","names":[],"sources":["../src/segmentation/fuzzy.ts","../src/segmentation/types.ts","../src/segmentation/pattern-validator.ts","../src/segmentation/replace.ts","../src/segmentation/segmenter.ts","../src/segmentation/tokens.ts","../src/analysis.ts","../src/detection.ts"],"sourcesContent":[],"mappings":";;AAkEA;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EY,cD/bC,WC+bqB,EAAA,CAAA,CAAA,EAAA,MAAc,EAAA,GAAA,MAAA;AA8BhD;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;ACkcA;;AAAqD,cJxWxC,wBIwWwC,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;AJvcrD;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA,KApXK,YAAA,GAoXW;EAqCJ;EA0EA,KAAA,EAAA,MAAU;AA8BtB,CAAA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA,KF8CK,eAAA,GE9CkB;EA2DV;EAA4B,QAAA,EAAA,MAAA;CAAgB;;;;;;ACkczD;;;;;;;;ACrcA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;;;KJlnBK,qBAAA;EKpFO;EAkEA,cAAA,EAAA,MAAA,EAAA;AAEZ,CAAA;AAkSA;;;;;;;;AC9VA;AA+EA;AAgEA;AAuBA;AAiCA;;;;;;;;;;;;;;;;KN1FK,sBAAA;;;;;;;;;;;;;;;;;;;;;;;KAwBA,mBAAA;;;;;;;;;;;;;;KAeA,WAAA,GACC,eACA,kBACA,wBACA,yBACA;;;;;;;KAYD,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+EO,SAAA;;;;;;;KAYP,eAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAyCS;;;;;;;;;;;;SAaH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA8DC,SAAA,GAAY,cAAc,gBAAgB;;;;;;;;;;;;;KAkB1C,IAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCA,cAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAqCE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCF,UAAA,YAAsB;;;;;;;;;;;;;;;;;;;;;;;;;UA8BjB,MAAA;;;;;;;;;;;;;;;;;;;;;;KAuBL,WAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+CA,mBAAA;;;;;;YAME;;;;;;;;UASF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA8CM;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;WAwDL;;;;;;;;;;;;;;;;KAiBD,OAAA;;;;;;;;;;;;;;;;;;;;;;;;;;SA6BD;;;;AA1uBM;AA4BG;AA8BM;AAyDrB,KCpIO,mBAAA,GDoIY,gBAAA,GAAA,eAAA,GAAA,WAAA;AAAA;;;AAkBlB,KCjJM,eAAA,GDiJN;EACA,IAAA,ECjJI,mBDiJJ;EACA,OAAA,EAAA,MAAA;EAAmB,UAAA,CAAA,EAAA,MAAA;AAAA,CAAA;AA2FzB;AAAkD;AAgIlD;;AAAsC,KCpW1B,oBAAA,GDoW0B;EAAgB,cAAA,CAAA,EAAA,CCnWhC,eDmWgC,GAAA,SAAA,CAAA,EAAA;EAAe,eAAA,CAAA,EAAA,CClW9C,eDkW8C,GAAA,SAAA,CAAA,EAAA;EAkBzD,YAAI,CAAA,EAAA,CCnXI,eDmXJ,GAAA,SAAA,CAAA,EAAA;EAqCJ,QAAA,CAAA,ECvZG,eDuZW;AA0E1B,CAAA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;AAI8B,cAoGjB,aApGiB,EAAA,CAAA,KAAA,EAoGO,SApGP,EAAA,EAAA,GAAA,CAoGsB,oBApGtB,GAAA,SAAA,CAAA,EAAA;;;AFkC9B;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAuCtB,KEpJO,WAAA,GAAc,WFoJV,CEpJsB,mBFoJtB,CAAA,SAAA,CAAA,CAAA,CAAA,MAAA,CAAA;;;;;;;AAKS;AA2FzB;AAAkD;AAgIlD;AAAwB,cEzTX,iBFyTW,EAAA,CAAA,KAAA,EEzTiB,IFyTjB,EAAA,EAAA,KAAA,CAAA,EEzTiC,WFyTjC,EAAA,EAAA,GEzTiD,IFyTjD,EAAA;;;;;AAkBxB;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;ACkcA;;;;;;;;ACrcA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;cDjMa,sBAAuB,iBAAiB,wBAAsB;;;;AJvc3E;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;ACkcA;AAAoC,cCrcvB,sBDqcuB,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;;;ACrcpC;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBa,cA5WA,+BA4WsD,EAAA,CAAA,QAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AAgBnE;;;;ACtsBA;AAkEA;AAEA;AAkSA;;;;;;;;AC9VA;AA+EA;AAgEA;AAuBA;AAiCA;;;;;;cFiLa,gBAAgB;;;;;;;;;;;;;;;;cA2ChB;;;;;;;KAWD,YAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cAuKC,mHAIV;;;;;;;;;;;;;;;;;;;;cAyCU;;;;;;;;;;;;;;;;;;;;;;cAuBA,uCAAmC;;;;;;;;;;;;;cAqBnC;;;;;;;;;;;;;;;cAgBA;;;ALxoBA,KM9DD,wBAAA,GN8D8E;EA+F7E;;;;ECnIR;EA4BA,aAAA,CAAA,EAAA,MAAe;EA8Bf;EAiCA,QAAA,CAAA,EAAA,MAAA;EAwBA;EAeA,WAAA,CAAA,EAAW,MAAA;EACV;;;;EAIA,wBAAA,CAAA,EAAA,OAAA;EAAmB;AAAA;AA2FzB;AAAkD;AAgIlD;;;;EAAqE,yBAAA,CAAA,EAAA,OAAA;EAkBzD;AAqCZ;AA0EA;AA8BA;AAuBA;AA+CA;EAMc,MAAA,CAAA,EAAA,aAAA,GAAA,OAAA;EASF;;;;AAuHZ;;;;AC9tBA;AAKA;EAUY,UAAA,CAAA,EAAA,CAAA,IAAA,EAAA,MAAoB,EAAA,MAAA,EAAA,MAAA,EAAA,GAAA,OAAA;EACV;;;;;AAuGtB;;;;ACxHA;AA2DA;;;EAAyE,cAAA,CAAA,EGXpD,MHWoD,EAAA;EAAI;;;;ACkc7E;;EAAqD,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;CAAsB;AAAO,KEnctE,uBAAA,GFmcsE;;;;ACrcrE,KCID,sBAAA,GDKX;EAiQY,OAAA,EAAA,MAAA;EAsDA,KAAA,EAAA,MAAA;EA2CA,QAAA,ECpWC,uBDuWb,EAAA;AAQD,CAAA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;;cChWa,iCACF,kBACE,6BACV;;;;AN3SH;AA+FA;;;;;ACnIiB;AA4BG;AA+Df,KM7GO,eAAA,GN6Ge;EAwBtB;EAeA,KAAA,EAAA,MAAA;EACC;EACA,KAAA,EAAA,MAAA;EACA;EACA,KAAA,EAAA,MAAA;EACA;EAAmB,QAAA,EAAA,MAAA;AAAA,CAAA;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;AA6DkB,cMvkBL,mBNukBK,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GMvkB8B,eNukB9B,EAAA;;;AAyElB;;;;AC9tBA;AAKA;AAUA;;;;;AAI8B,cK2HjB,wBL3HiB,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,QAAA,EK2HmC,eL3HnC,EAAA,EAAA,GAAA,MAAA;AAoG9B;;;;ACxHA;AA2DA;AAAyC,cI2G5B,oBJ3G4B,EAAA,CAAA,QAAA,EI4G3B,eJ5G2B,EAAA,EAAA,GAAA;EAAgB,WAAA,EAAA,gBAAA,GAAA,iBAAA;EAAgB,KAAA,EAAA,OAAA;EAAI,QAAA,CAAA,EAAA,MAAA;;;;ACkc7E;;;;AAAkF,cGtTrE,kBHsTqE,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA;;;;ECrcrE,QAAA,CAAA,EAAA,MAAA;EA0QA,QAAA,EEpHC,eFoHD,EAAA;AAsDb,CAAA,GAAa,IAAA"}
+{"version":3,"file":"index.d.mts","names":[],"sources":["../src/segmentation/fuzzy.ts","../src/segmentation/types.ts","../src/segmentation/pattern-validator.ts","../src/segmentation/replace.ts","../src/segmentation/segmenter.ts","../src/segmentation/tokens.ts","../src/analysis/line-starts.ts","../src/analysis/repeating-sequences.ts","../src/detection.ts","../src/recovery.ts"],"sourcesContent":[],"mappings":";;AAkEA;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EY,cD/bC,WC+bqB,EAAA,CAAA,CAAA,EAAA,MAAc,EAAA,GAAA,MAAA;AA8BhD;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;AC6SA;;AAAqD,cJnNxC,wBImNwC,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;AJlTrD;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA,KApXK,YAAA,GAoXW;EAqCJ;EA0EA,KAAA,EAAA,MAAU;AA8BtB,CAAA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA,KF8CK,eAAA,GE9CkB;EA2DV;EAA4B,QAAA,EAAA,MAAA;CAAgB;;;;;;AC6SzD;;;;;;;;AChTA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;;;KJlnBK,qBAAA;EKnEO;EAcA,cAAA,EAAA,MAAA,EAAA;AAEZ,CAAA;AAwQA;;;;;;;;AClRA;AAaA;AAOA;AA2OA;;;;;;;;AC9QA;AA+EA;AAgEA;AAuBA;AAiCA;;;;AC7MA;AAKA,KR8GK,sBAAA,GQ9GwB;EAChB;EACF,eAAA,EAAA,MAAA,EAAA;CACG;;;AAId;AA2BE;AAmnBF;;;;;;;;;AAsDA;;;;;;KRrkBK,mBAAA;;;;;;;;;;;;;;KAeA,WAAA,GACC,eACA,kBACA,wBACA,yBACA;;;;;;;KAYD,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+EO,SAAA;;;;;;;KAYP,eAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAyCS;;;;;;;;;;;;SAaH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA8DC,SAAA,GAAY,cAAc,gBAAgB;;;;;;;;;;;;;KAkB1C,IAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCA,cAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAqCE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCF,UAAA,YAAsB;;;;;;;;;;;;;;;;;;;;;;;;;UA8BjB,MAAA;;;;;;;;;;;;;;;;;;;;;;KAuBL,WAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA+CA,mBAAA;;;;;;YAME;;;;;;;;UASF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA8CM;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;WAwDL;;;;;;;;;;;;;;;;KAiBD,OAAA;;;;;;;;;;;;;;;;;;;;;;;;;;SA6BD;;;;AA1uBM;AA4BG;AA8BM;AAyDrB,KCpIO,mBAAA,GDoIY,gBAAA,GAAA,eAAA,GAAA,WAAA;AAAA;;;AAkBlB,KCjJM,eAAA,GDiJN;EACA,IAAA,ECjJI,mBDiJJ;EACA,OAAA,EAAA,MAAA;EAAmB,UAAA,CAAA,EAAA,MAAA;AAAA,CAAA;AA2FzB;AAAkD;AAgIlD;;AAAsC,KCpW1B,oBAAA,GDoW0B;EAAgB,cAAA,CAAA,EAAA,CCnWhC,eDmWgC,GAAA,SAAA,CAAA,EAAA;EAAe,eAAA,CAAA,EAAA,CClW9C,eDkW8C,GAAA,SAAA,CAAA,EAAA;EAkBzD,YAAI,CAAA,EAAA,CCnXI,eDmXJ,GAAA,SAAA,CAAA,EAAA;EAqCJ,QAAA,CAAA,ECvZG,eDuZW;AA0E1B,CAAA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;AAI8B,cAoGjB,aApGiB,EAAA,CAAA,KAAA,EAoGO,SApGP,EAAA,EAAA,GAAA,CAoGsB,oBApGtB,GAAA,SAAA,CAAA,EAAA;;;AFkC9B;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAuCtB,KEpJO,WAAA,GAAc,WFoJV,CEpJsB,mBFoJtB,CAAA,SAAA,CAAA,CAAA,CAAA,MAAA,CAAA;;;;;;;AAKS;AA2FzB;AAAkD;AAgIlD;AAAwB,cEzTX,iBFyTW,EAAA,CAAA,KAAA,EEzTiB,IFyTjB,EAAA,EAAA,KAAA,CAAA,EEzTiC,WFyTjC,EAAA,EAAA,GEzTiD,IFyTjD,EAAA;;;;;AAkBxB;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;AC6SA;;;;;;;;AChTA;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBA;AAgBA;cDtVa,sBAAuB,iBAAiB,wBAAmB;;;;AJlTxE;AA+FA;;;;;ACnIiB;AA4BG;AA8BM;AAiCC;AAwBH;;;;;;;AAoBC;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;;;;AAsIA;;;;AC9tBA;AAKA;AAUA;;;;;;AAwGA;;;;ACxHA;AA2DA;;;;;;;;AC6SA;AAAoC,cChTvB,sBDgTuB,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;;;;AChTpC;AA0QA;AAsDA;AA2CA;AAWA;AAuKA;AA6CA;AAuBA;AAqBa,cA5WA,+BA4WsD,EAAA,CAAA,QAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AAgBnE;;;;ACrrBA;AAcA;AAEA;AAwQA;;;;;;;;AClRA;AAaA;AAOA;AA2OA;;;;;;;cF0Ga,gBAAgB;AGxX7B;AA+EA;AAgEA;AAuBA;AAiCA;;;;AC7MA;AAKA;;;;;;AAOY,cJ6ZC,cI7ZmB,EAAA,CAAA,KAQpB,EAAA,MAMC,EAAK,GAAA,OAAA;AAahB;AAmnBF;;;;;AAO2B,KJ7Of,YAAA,GI6Oe;EAEd;;;AA6Cb;;EAEgF,OAAA,EAAA,MAAA;EACnE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cJxHA,mHAIV;;;;;;;;;;;;;;;;;;;;cAyCU;;;;;;;;;;;;;;;;;;;;;;cAuBA,uCAAmC;;;;;;;;;;;;;cAqBnC;;;;;;;;;;;;;;;cAgBA;;;ALxoBA,KM7CD,wBAAA,GN6C8E;EA+F7E,IAAA,CAAA,EAAA,MAAA;;;;ECnIR,WAAA,CAAA,EAAA,MAAY;EA4BZ,wBAAe,CAAA,EAAA,OAAA;EA8Bf,yBAAqB,CAAA,EAAA,OAAA;EAiCrB,MAAA,CAAA,EAAA,aAAA,GAAsB,OAAA;EAwBtB,UAAA,CAAA,EAAA,CAAA,IAAA,EAAA,MAAmB,EAAA,MAAA,EAAA,MAAA,EAAA,GAAA,OAAA;EAenB,cAAW,CAAA,EKjIK,MLiIL,EAAA;EACV,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;CACA;AACA,KKhIM,uBAAA,GLgIN;EACA,IAAA,EAAA,MAAA;EACA,MAAA,EAAA,MAAA;CAAmB;AAYpB,KK5IO,sBAAA,GL4IM;EA+EN,OAAA,EAAA,MAAS;EAYhB,KAAA,EAAA,MAAA;EAoHO,QAAA,EKxVE,uBLwVO,EAAA;CAAG;;;;AAkBZ,cKrGC,uBLqGG,EAAA,CAAA,KAAA,EKpGL,ILoGK,EAAA,EAAA,OAAA,CAAA,EKnGH,wBLmGG,EAAA,GKlGb,sBLkGa,EAAA;;;AAjQX,KMtHO,wBAAA,GNsHY;EAenB,WAAA,CAAA,EAAW,MAAA;EACV,WAAA,CAAA,EAAA,MAAA;EACA,QAAA,CAAA,EAAA,MAAA;EACA,IAAA,CAAA,EAAA,MAAA;EACA,yBAAA,CAAA,EAAA,OAAA;EACA,YAAA,CAAA,EAAA,OAAA;EAAmB,UAAA,CAAA,EAAA,OAAA,GAAA,OAAA;EAYpB,WAAA,CAAA,EAAA,MAAa;EA+EN,YAAS,CAAA,EAAA,MAAA;EAYhB,iBAAA,CAAe,EAAA,MAAA;AAoHpB,CAAA;AAAwB,KMxVZ,wBAAA,GNwVY;EAAc,IAAA,EAAA,MAAA;EAAgB,OAAA,EAAA,MAAA;EAAe,MAAA,EAAA,MAAA;EAkBzD,YAAI,EAAA,MAAA,EAAA;AAqChB,CAAA;AA0EY,KMldA,wBAAA,GNkdsB;EA8BjB,OAAA,EAAM,MAAA;EAuBX,KAAA,EAAA,MAAA;EA+CA,QAAA,EMnjBE,wBNmjBiB,EAAA;CAMjB;;;AAgId;;;;AC9tBY,cK6QC,yBL7QkB,EAAA,CAAA,KAAA,EK8QpB,IL9QoB,EAAA,EAAA,OAAA,CAAA,EK+QjB,wBL/QiB,EAAA,GKgR5B,wBLhR4B,EAAA;;;;AFqD/B;AA+FA;;;;;ACnIiB;AA4BG;AA+Df,KO7GO,eAAA,GP6Ge;EAwBtB;EAeA,KAAA,EAAA,MAAA;EACC;EACA,KAAA,EAAA,MAAA;EACA;EACA,KAAA,EAAA,MAAA;EACA;EAAmB,QAAA,EAAA,MAAA;AAAA,CAAA;AA2FzB;AAAkD;AAgIlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAuBA;AA+CA;;;AA6DkB,cOvkBL,mBPukBK,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GOvkB8B,ePukB9B,EAAA;;;AAyElB;;;;AC9tBA;AAKA;AAUA;;;;;AAI8B,cM2HjB,wBN3HiB,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,QAAA,EM2HmC,eN3HnC,EAAA,EAAA,GAAA,MAAA;AAoG9B;;;;ACxHA;AA2DA;AAAyC,cK2G5B,oBL3G4B,EAAA,CAAA,QAAA,EK4G3B,eL5G2B,EAAA,EAAA,GAAA;EAAgB,WAAA,EAAA,gBAAA,GAAA,iBAAA;EAAgB,KAAA,EAAA,OAAA;EAAI,QAAA,CAAA,EAAA,MAAA;;;;AC6S7E;;;;AAAwE,cIjK3D,kBJiK2D,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA;;;;EChT3D,QAAA,CAAA,EAAA,MAAA;EA0QA,QAAA,EGpHC,eHoHD,EAAA;AAsDb,CAAA,GAAa,IAAA;;;ALlUA,KS5DD,sBAAA,GT4D8E;EA+F7E,IAAA,EAAA,cAAA;;;;ECnIR,KAAA,CAAA,EAAA,OAAY,GAAA,YAAA;EA4BZ,QAAA,EAAA,MAAA,EAAe;AAAA,CAAA,GA8Bf;EAiCA,IAAA,EAAA,WAAA;EAwBA,SAAA,EAAA,CAAA,IAAA,EQxIwC,SRwIrB,EAAA,KAAA,EAAA,MAAA,EAAA,GAAA,OAAA;AAAA,CAAA;AAgBlB,KQtJM,iBAAA,GRsJN;EACA,OAAA,EQtJO,mBRsJP;EACA,KAAA,EQtJK,IRsJL,EAAA;EACA,QAAA,EQtJQ,ORsJR,EAAA;EACA,QAAA,EQtJQ,sBRsJR;CAAmB;AAYpB,KQ/JO,oBAAA,GR+JM;EA+EN,OAAA,EAAA;IAYP,IAAA,EAAA,YAAe,GAAA,wBAsDH;IA8DL,SAAS,EAAA,MAAA;IAAG,aAAA,EAAA,MAAA;IAAc,SAAA,EAAA,MAAA;IAAgB,UAAA,EAAA,MAAA;EAAe,CAAA;EAkBzD,KAAA,CAAA,EQxXA,KRwXI,CAAA;IAqCJ,SAAA,EAAA,MAAc;IA0Ed,QAAA,EAAU,MAAA;IA8BL,aAAM,EAAA,MAAA;IAuBX,UAAW,EAAA,MAAA;EA+CX,CAAA,CAAA;EAME,OAAA,EQ3kBD,KR2kBC,CAAA;IASF,IAAA,EAAA,MAAA;IA8CM,oBAAA,EAAA,MAAA;IAwDL,sBAAA,CAAA,EAAA,MAAA;IAAM,qBAAA,CAAA,EAAA,MAAA;IAiBP,YAAO,EAAA,MA6BR;;;;IC3vBC,KAAA,CAAA,EAAA,MAAA,EAAA;EAKA,CAAA,CAAA;EAUA,MAAA,EAAA,MAAA,EAAA;EACU,QAAA,EAAA,MAAA,EAAA;CACC;KOiBlB,oBAAA,GPhBe,MAAA,GAAA,YAAA,GAAA,qBAAA;AACL,iBOgoBC,qCAAA,CPhoBD,KAAA,EOioBJ,IPjoBI,EAAA,EAAA,QAAA,EOkoBD,OPloBC,EAAA,EAAA,OAAA,EOmoBF,mBPnoBE,EAAA,QAAA,EOooBD,sBPpoBC,EAAA,KAAA,EAAA;EAAe,IAAA,CAAA,EAAA,YAAA,GAAA,wBAAA;EAoGjB,gBA0CZ,CAAA,EOyf0B,oBPniByB;;UOqiBvC;YAAgC;AN7pB7C,CAAA;AA2Da,iBM+oBG,6BAAA,CNznBf,IAAA,EM0nBS,iBN1nBT,EAAA,EAAA,IAtBwE,CAsBxE,EAAA;EAtBwC,IAAA,CAAA,EAAA,YAAA,GAAA,wBAAA;EAAgB,gBAAA,CAAA,EMipBuB,oBNjpBvB;CAAgB,CAAA,EAAA;EAAI,MAAA,EMkpBhE,oBNlpBgE;YMkpBhC"}