flappa-doormal 2.0.0 → 2.1.0

package/AGENTS.md

@@ -0,0 +1,325 @@
+# AGENTS.md
+
+## Project Overview
+
+**flappa-doormal** is a declarative text segmentation library for Arabic texts. It splits pages of content into logical segments (hadiths, chapters, verses) using pattern-based rules with a human-readable template syntax.
+
+### Why This Library Exists
+
+Traditional Arabic text segmentation requires:
+- Hand-crafted regular expressions with complex Unicode patterns
+- Deep knowledge of Arabic diacritics and character equivalences
+- Handling page boundaries and multi-page content spans
+- Manual capture group management for metadata extraction
+
+**flappa-doormal** solves these problems with:
+- **Declarative rules**: Describe *what* to match, not *how* to match it
+- **Template tokens**: `{{raqms}} {{dash}}` instead of `[\u0660-\u0669]+\s*[-–—ـ]`
+- **Named captures**: `{{raqms:hadithNum}}` → automatic `meta.hadithNum` extraction
+- **Fuzzy matching**: Diacritic-insensitive matching for harakat variations
+- **Page-aware**: Tracks which page each segment comes from
+
+## Architecture
+
+### Repository Structure
+
+```text
+src/
+├── index.ts                    # Main entry point and exports
+├── pattern-detection.ts        # Token detection for auto-generating rules (NEW)
+├── pattern-detection.test.ts   # Pattern detection tests (22 tests)
+└── segmentation/
+    ├── types.ts                # TypeScript type definitions for rules/segments
+    ├── segmenter.ts            # Core segmentation engine (segmentPages, applyBreakpoints)
+    ├── breakpoint-utils.ts     # Extracted breakpoint processing utilities
+    ├── tokens.ts               # Token definitions and expansion logic  
+    ├── fuzzy.ts                # Diacritic-insensitive matching utilities
+    ├── html.ts                 # HTML utilities (stripHtmlTags)
+    ├── textUtils.ts            # Text processing utilities
+    ├── match-utils.ts          # Extracted match processing utilities
+    ├── segmenter.test.ts       # Core test suite (150+ tests including breakpoints)
+    ├── segmenter.bukhari.test.ts # Real-world test cases
+    ├── breakpoint-utils.test.ts # Breakpoint utility tests (42 tests)
+    ├── tokens.test.ts          # Token expansion tests
+    ├── fuzzy.test.ts           # Fuzzy matching tests
+    ├── textUtils.test.ts       # Text utility tests
+    └── match-utils.test.ts     # Utility function tests
+
+test/
+├── 2576.json                   # Test data for book 2576 (Sahih Bukhari)
+└── 2588.json                   # Test data for book 2588 (Al-Mughni)
+
+docs/
+├── checkpoints/                # AI agent handoff documentation
+│   └── 2025-12-09-handoff.md
+└── reviews/                    # Performance analysis reports
+    └── 2025-12-10/
+```
+
+### Core Components
+
+1. **`segmentPages(pages, options)`** - Main entry point
+   - Takes array of `{id, content}` pages and split rules
+   - Returns array of `{content, from, to?, meta?}` segments
+
+2. **`tokens.ts`** - Template system
+   - `TOKEN_PATTERNS` - Map of token names to regex patterns
+   - `expandTokensWithCaptures()` - Expands `{{token:name}}` syntax
+   - Supports fuzzy transform for diacritic-insensitive matching
+
+3. **`match-utils.ts`** - Extracted utilities (for testability)
+   - `extractNamedCaptures()` - Get named groups from regex match
+   - `filterByConstraints()` - Apply min/max page filters
+   - `anyRuleAllowsId()` - Check if page passes rule constraints
+
+4. **`breakpoint-utils.ts`** - Breakpoint processing utilities (NEW)
+   - `normalizeBreakpoint()` - Convert string to BreakpointRule object
+   - `isPageExcluded()` - Check if page is in exclude list
+   - `isInBreakpointRange()` - Validate page against min/max/exclude constraints
+   - `buildExcludeSet()` - Create Set from PageRange[] for O(1) lookups
+   - `createSegment()` - Create segment with optional to/meta fields
+   - `expandBreakpoints()` - Expand patterns with pre-compiled regexes
+   - `findActualEndPage()` - Search backwards for ending page by content
+   - `findBreakPosition()` - Find break position using breakpoint patterns
+   - `hasExcludedPageInRange()` - Check if range contains excluded pages
+   - `findNextPagePosition()` - Find next page content position
+   - `findPatternBreakPosition()` - Find pattern match by preference
+
+5. **`types.ts`** - Type definitions
+   - `Logger` interface - Optional logging for debugging
+   - `SegmentationOptions` - Options with `logger` property
+   - Verbosity levels: `trace`, `debug`, `info`, `warn`, `error`
+
+6. **`fuzzy.ts`** - Arabic text normalization
+   - `makeDiacriticInsensitive()` - Generate regex that ignores diacritics
+
+7. **`pattern-detection.ts`** - Token auto-detection (NEW)
+   - `detectTokenPatterns()` - Detect tokens in text with positions
+   - `generateTemplateFromText()` - Convert text to template string
+   - `suggestPatternConfig()` - Suggest rule configuration
+   - `analyzeTextForRule()` - Complete analysis returning template + config
+
+## Key Algorithms
+
+### Token Expansion
+
+```
+Input:  "{{raqms:num}} {{dash}} {{:text}}"
+Output: "(?<num>[\u0660-\u0669]+) [-–—ـ] (?<text>.+)"
+```
+
+The expansion algorithm:
+1. Splits query into token and text segments
+2. Looks up token patterns from `TOKEN_PATTERNS`
+3. Wraps in named capture group if `:name` suffix present
+4. Applies fuzzy transform if enabled (before wrapping in groups)
+
+### Fuzzy Application Order
+
+**Critical design decision**: Fuzzy transforms are applied to raw token patterns and plain text *before* they're wrapped in regex groups.
+
+```
+WRONG:  makeDiacriticInsensitive("(?<name>حدثنا)")  // Breaks ( ? < > )
+RIGHT:  "(?<name>" + makeDiacriticInsensitive("حدثنا") + ")"
+```
+
+### lineStartsAfter Content Capture
+
+For patterns like `^٦٦٩٦ - (content)`, the content capture is the *last* positional group:
+
+```typescript
+// Pattern: ^(?:(?<num>[\u0660-\u0669]+) [-–—ـ] )(.*)
+// Match:   m[1] = named group value, m[2] = content
+// Solution: Iterate backward from m.length-1 to find last defined capture
+```
+
+### Breakpoints Post-Processing Algorithm
+
+The `breakpoints` option provides a post-processing mechanism for limiting segment size. Unlike the deprecated `maxSpan` (which was per-rule), breakpoints runs AFTER all structural rules.
+
+**API Options:**
+```typescript
+interface SegmentationOptions {
+  rules: SplitRule[];
+  maxPages?: number;           // Maximum pages a segment can span
+  breakpoints?: string[];      // Ordered array of regex patterns (supports token expansion)
+  prefer?: 'longer' | 'shorter'; // Select last or first match within window
+}
+```
+
+**How it works:**
+1. Structural rules run first, creating initial segments
+2. Breakpoints then processes any segment exceeding `maxPages`
+3. Patterns are tried in order until one matches
+4. Empty string `''` means "fall back to page boundary"
+
+**Example:**
+```typescript
+segmentPages(pages, {
+  rules: [
+    { lineStartsWith: ['{{basmalah}}'], split: 'at' },
+    { lineStartsWith: ['{{bab}}'], split: 'at', meta: { type: 'chapter' } },
+  ],
+  maxPages: 2,
+  breakpoints: ['{{tarqim}}\\s*', '\\n', ''],  // Try: punctuation → newline → page boundary
+  prefer: 'longer',  // Greedy: make segments as large as possible
+});
+```
+
+**Key behaviors:**
+- **Pattern order matters**: First matching pattern wins
+- **`prefer: 'longer'`**: Finds LAST match in window (greedy)
+- **`prefer: 'shorter'`**: Finds FIRST match (conservative)
+- **Recursive**: If split result still exceeds `maxPages`, breakpoints runs again
+
+> **Note**: The old `maxSpan` and `fallback` properties on `SplitRule` are deprecated and removed.
+
+## Design Decisions
+
+### 1. Why `{{double-braces}}`?
+
+- Single braces `{}` conflict with regex quantifiers `{n,m}`
+- Double braces are visually distinct and rarely appear in content
+- Consistent with template systems (Handlebars, Mustache)
+
+### 2. Why `lineStartsAfter` vs `lineStartsWith`?
+
+| Pattern | Marker in content? | Use case |
+|---------|-------------------|----------|
+| `lineStartsWith` | ✅ Yes | Keep marker, segment at boundary |
+| `lineStartsAfter` | ❌ No | Strip marker, capture only content |
+
+### 3. Why fuzzy transform at token level?
+
+Applying fuzzy globally would corrupt regex metacharacters. Instead:
+- Fuzzy is passed to `expandTokensWithCaptures()` 
+- Applied only to Arabic text portions
+- Preserves `(`, `)`, `|`, `?`, etc.
+
+### 4. Why extract match utilities?
+
+The original `segmentPages` had complexity 37 (max: 15). Extraction:
+- Creates independently testable units
+- Reduces main function complexity
+- Improves code readability
+
+## Working with the Codebase
+
+### Adding a New Token
+
+1. Add to `TOKEN_PATTERNS` in `tokens.ts`:
+   ```typescript
+   export const TOKEN_PATTERNS = {
+     // ...existing
+     verse: '﴿[^﴾]+﴾',  // Quranic verse markers
+   };
+   ```
+2. Add test cases in `segmenter.test.ts`
+3. Document in README.md
+
+### Adding a New Pattern Type
+
+1. Add type to union in `types.ts`:
+   ```typescript
+   type NewPattern = { newPatternField: string[] };
+   type PatternType = ... | NewPattern;
+   ```
+2. Handle in `buildRuleRegex()` in `segmenter.ts`
+3. Add comprehensive tests
+
+### Testing Strategy
+
+- **Unit tests**: Each utility function has dedicated tests
+- **Integration tests**: Full pipeline tests in `segmenter.test.ts`
+- **Real-world tests**: `segmenter.bukhari.test.ts` uses actual hadith data
+- Run: `bun test`
+
+## Code Quality Standards
+
+1. **TypeScript strict mode** - No `any` types
+2. **Biome linting** - Max complexity 15 per function (some exceptions exist)
+3. **JSDoc comments** - All exported functions documented
+4. **Test coverage** - 251 tests across 8 files
+
+## Dependencies
+
+### Development
+- **@biomejs/biome** - Linting and formatting
+- **tsdown** - Build tool (generates `.mjs` and `.d.mts`)
+- **Bun** - Runtime and test runner
+
+## Build & Release
+
+```bash
+# Install dependencies
+bun install
+
+# Run all tests
+bun test
+
+# Build distribution
+bun run build
+# Output: dist/index.mjs (~17 KB gzip ~5.7 KB)
+
+# Run performance test (generates 50K pages, measures segmentation speed/memory)
+bun run perf
+
+# Format code
+bunx biome format --write .
+
+# Lint code
+bunx biome lint .
+```
+
+## Lessons Learned
+
+### From Development
+
+1. **Named captures shift positional indices**: When `(?<name>…)` appears before `(.*)`, the content is at `m[2]` not `m[1]`. Solution: iterate backward to find last defined group.
+
+2. **Fuzzy + metacharacters don't mix**: `makeDiacriticInsensitive` expands Arabic to character classes. If applied to `(?<name>text)`, it corrupts the `(`, `?`, `<`, `>` characters.
+
+3. **Alternations need per-alternative fuzzy**: Token `narrated: 'حدثنا|أخبرنا'` requires splitting at `|`, applying fuzzy to each, then rejoining.
+
+4. **Complexity extraction works**: Pulling logic into `match-utils.ts` reduced main function complexity from 37 to 10 and made the code testable.
+
+5. **Rule order matters for specificity**: When multiple rules can match the same position, put specific patterns BEFORE generic ones. Example: `## {{raqms:num}} {{dash}}` must come before `##` to capture the number.
+
+6. **Post-processing beats per-rule limits**: The `maxSpan` approach (per-rule page limits) caused premature splits. Moving to post-processing `breakpoints` preserves structural integrity while still limiting segment size.
+
+7. **Window padding matters**: When calculating approximate content windows, 50% padding is needed (not 20%) to ensure enough content is captured for `prefer: 'longer'` scenarios.
+
+8. **Escaping in tests requires care**: TypeScript string `'\\.'` creates regex `\.`, but regex literal `/\./` is already escaped. Double-backslash in strings, single in literals.
+
+### Architecture Insights
+
+- **Declarative > Imperative**: Users describe patterns, library handles regex
+- **Composability**: Tokens can be combined freely with `:name` captures
+- **Fail gracefully**: Unknown tokens are left as-is, allowing partial templates
+- **Post-process > Inline**: Breakpoints runs after rules, avoiding conflicts
+- **Dependency injection for testability**: `breakpoint-utils.ts` accepts a `PatternProcessor` function instead of importing `processPattern` directly, enabling independent testing without mocking
+- **Optional logging**: Use optional chaining (`logger?.debug?.()`) for zero-overhead when no logger is provided. All log methods are optional, allowing clients to subscribe to only the verbosity levels they need.
+
+---
+
+## Token Reference
+
+| Token | Pattern Description | Example Match |
+|-------|---------------------|---------------|
+| `{{tarqim}}` | Arabic punctuation (. , ; ? ! ( ) etc.) | `؛` `،` `.` |
+| `{{basmalah}}` | "بِسْمِ اللَّهِ" patterns | بِسْمِ اللَّهِ الرَّحْمَنِ |
+| `{{bab}}` | "باب" (chapter) | بَابُ الإيمان |
+| `{{fasl}}` | "فصل" (section) | فصل: في الطهارة |
+| `{{kitab}}` | "كتاب" (book) | كتاب الصلاة |
+| `{{raqm}}` | Single Arabic-Indic numeral | ٥ |
+| `{{raqms}}` | Multiple Arabic-Indic numerals | ٧٥٦٣ |
+| `{{raqms:num}}` | Numerals with named capture | `meta.num = "٧٥٦٣"` |
+| `{{dash}}` | Various dash characters | - – — ـ |
+| `{{numbered}}` | Composite: `{{raqms}} {{dash}}` | ٧٥٦٣ - |
+
+**Named captures**: Add `:name` suffix to capture into `meta`:
+```typescript
+'{{raqms:hadithNum}} {{dash}}' 
+// → segment.meta.hadithNum = "٧٥٦٣"
+```
+

package/README.md

@@ -395,6 +395,32 @@ console.log(TOKEN_PATTERNS.narrated);
 // 'حدثنا|أخبرنا|حدثني|وحدثنا|أنبأنا|سمعت'
 ```
 
+### Pattern Detection
+
+Auto-detect tokens in Arabic text for building rules:
+
+```typescript
+import { detectTokenPatterns, analyzeTextForRule } from 'flappa-doormal';
+
+// Detect individual tokens
+const tokens = detectTokenPatterns('٣٤ - حدثنا');
+// [
+//   { token: 'raqms', match: '٣٤', index: 0, endIndex: 2 },
+//   { token: 'dash', match: '-', index: 3, endIndex: 4 },
+//   { token: 'naql', match: 'حدثنا', index: 5, endIndex: 10 }
+// ]
+
+// Get complete rule suggestion
+const rule = analyzeTextForRule('٣٤ - ');
+// {
+//   template: '{{raqms}} {{dash}} ',
+//   patternType: 'lineStartsAfter',
+//   fuzzy: false,
+//   metaType: 'hadith',
+//   detected: [...]
+// }
+```
+
 ## Types
 
 ### `SplitRule`
@@ -433,6 +459,56 @@ type Segment = {
 };
 ```
 
+### `Logger`
+
+Optional logging interface for debugging segmentation:
+
+```typescript
+interface Logger {
+  trace?: (message: string, ...args: unknown[]) => void;  // Per-iteration details
+  debug?: (message: string, ...args: unknown[]) => void;  // Detailed operations
+  info?: (message: string, ...args: unknown[]) => void;   // Key progress points
+  warn?: (message: string, ...args: unknown[]) => void;   // Potential issues
+  error?: (message: string, ...args: unknown[]) => void;  // Critical failures
+}
+```
+
+## Debugging
+
+### Using the Logger
+
+Pass a `logger` option to receive detailed information about the segmentation process:
+
+```typescript
+// Console logger for development
+const segments = segmentPages(pages, {
+  rules: [...],
+  logger: {
+    debug: console.debug,
+    info: console.info,
+    warn: console.warn,
+  }
+});
+
+// Production logger (only errors)
+const segments = segmentPages(pages, {
+  rules: [...],
+  logger: {
+    error: (msg, ...args) => myLoggingService.error(msg, args),
+  }
+});
+```
+
+**Verbosity levels:**
+- `trace` - Per-iteration loop details (very verbose)
+- `debug` - Segment processing, pattern matching
+- `info` - Start/completion of breakpoint processing
+- `warn` - Safety checks triggered
+- `error` - Infinite loop detection
+
+When no logger is provided, no logging overhead is incurred.
+
+
 ## Usage with Next.js / Node.js
 
 ```typescript
@@ -474,7 +550,7 @@ console.log(`Found ${segments.length} segments`);
 # Install dependencies
 bun install
 
-# Run tests (222 tests)
+# Run tests (251 tests)
 bun test
 
 # Build

package/dist/index.d.mts

@@ -460,6 +460,42 @@ type BreakpointRule = {
  * { pattern: '{{tarqim}}\\s*', min: 10 }
  */
 type Breakpoint = string | BreakpointRule;
+/**
+ * Logger interface for custom logging implementations.
+ *
+ * All methods are optional - only implement the verbosity levels you need.
+ * When no logger is provided, no logging overhead is incurred.
+ *
+ * Compatible with the Logger interface from ffmpeg-simplified and similar libraries.
+ *
+ * @example
+ * // Simple console logger
+ * const logger: Logger = {
+ *   debug: console.debug,
+ *   info: console.info,
+ *   warn: console.warn,
+ *   error: console.error,
+ * };
+ *
+ * @example
+ * // Production logger (only warnings and errors)
+ * const prodLogger: Logger = {
+ *   warn: (msg, ...args) => myLoggingService.warn(msg, args),
+ *   error: (msg, ...args) => myLoggingService.error(msg, args),
+ * };
+ */
+interface Logger {
+  /** Log a debug message (verbose debugging output) */
+  debug?: (message: string, ...args: unknown[]) => void;
+  /** Log an error message (critical failures) */
+  error?: (message: string, ...args: unknown[]) => void;
+  /** Log an informational message (key progress points) */
+  info?: (message: string, ...args: unknown[]) => void;
+  /** Log a trace message (extremely verbose, per-iteration details) */
+  trace?: (message: string, ...args: unknown[]) => void;
+  /** Log a warning message (potential issues) */
+  warn?: (message: string, ...args: unknown[]) => void;
+}
 /**
  * Segmentation options controlling how pages are split.
  *
@@ -480,6 +516,17 @@ type Breakpoint = string | BreakpointRule;
  *   breakpoints: ['{{tarqim}}\\s*', '\\n', ''],
  *   prefer: 'longer'
  * };
+ *
+ * @example
+ * // With custom logger for debugging
+ * const options: SegmentationOptions = {
+ *   rules: [...],
+ *   logger: {
+ *     debug: console.debug,
+ *     info: console.info,
+ *     warn: console.warn,
+ *   }
+ * };
  */
 type SegmentationOptions = {
   /**
@@ -542,6 +589,38 @@ type SegmentationOptions = {
    * @default 'longer'
    */
   prefer?: 'longer' | 'shorter';
+  /**
+   * Optional logger for debugging segmentation.
+   *
+   * Provide a logger to receive detailed information about the segmentation
+   * process. Useful for debugging pattern matching, page tracking, and
+   * breakpoint processing issues.
+   *
+   * When not provided, no logging overhead is incurred (methods are not called).
+   *
+   * Verbosity levels:
+   * - `trace`: Per-iteration details (very verbose)
+   * - `debug`: Detailed operation information
+   * - `info`: Key progress points
+   * - `warn`: Potential issues
+   * - `error`: Critical failures
+   *
+   * @example
+   * // Console logger for development
+   * logger: {
+   *   debug: console.debug,
+   *   info: console.info,
+   *   warn: console.warn,
+   * }
+   *
+   * @example
+   * // Custom logger integration
+   * logger: {
+   *   debug: (msg, ...args) => winston.debug(msg, { meta: args }),
+   *   error: (msg, ...args) => winston.error(msg, { meta: args }),
+   * }
+   */
+  logger?: Logger;
 };
 /**
  * Output segment produced by `segmentPages()`.
@@ -846,5 +925,80 @@ declare const getAvailableTokens: () => string[];
  */
 declare const getTokenPattern: (tokenName: string) => string | undefined;
 //#endregion
-export { type Breakpoint, type BreakpointRule, type ExpandResult, type Page, type PageRange, type Segment, type SegmentationOptions, type SplitRule, TOKEN_PATTERNS, containsTokens, escapeRegex, expandTokens, expandTokensWithCaptures, getAvailableTokens, getTokenPattern, makeDiacriticInsensitive, normalizeLineEndings, segmentPages, stripHtmlTags, templateToRegex };
+//#region src/pattern-detection.d.ts
+/**
+ * Pattern detection utilities for recognizing template tokens in Arabic text.
+ * Used to auto-detect patterns from user-highlighted text in the segmentation dialog.
+ *
+ * @module pattern-detection
+ */
+/**
+ * Result of detecting a token pattern in text
+ */
+type DetectedPattern = {
+  /** Token name from TOKEN_PATTERNS (e.g., 'raqms', 'dash') */
+  token: string;
+  /** The matched text */
+  match: string;
+  /** Start index in the original text */
+  index: number;
+  /** End index (exclusive) */
+  endIndex: number;
+};
+/**
+ * Analyzes text and returns all detected token patterns with their positions.
+ * Patterns are detected in priority order to avoid partial matches.
+ *
+ * @param text - The text to analyze for token patterns
+ * @returns Array of detected patterns sorted by position
+ *
+ * @example
+ * detectTokenPatterns("٣٤ - حدثنا")
+ * // Returns: [
+ * //   { token: 'raqms', match: '٣٤', index: 0, endIndex: 2 },
+ * //   { token: 'dash', match: '-', index: 3, endIndex: 4 },
+ * //   { token: 'naql', match: 'حدثنا', index: 5, endIndex: 10 }
+ * // ]
+ */
+declare const detectTokenPatterns: (text: string) => DetectedPattern[];
+/**
+ * Generates a template pattern from text using detected tokens.
+ * Replaces matched portions with {{token}} syntax.
+ *
+ * @param text - Original text
+ * @param detected - Array of detected patterns from detectTokenPatterns
+ * @returns Template string with tokens, e.g., "{{raqms}} {{dash}} "
+ *
+ * @example
+ * const detected = detectTokenPatterns("٣٤ - ");
+ * generateTemplateFromText("٣٤ - ", detected);
+ * // Returns: "{{raqms}} {{dash}} "
+ */
+declare const generateTemplateFromText: (text: string, detected: DetectedPattern[]) => string;
+/**
+ * Determines the best pattern type for auto-generated rules based on detected patterns.
+ *
+ * @param detected - Array of detected patterns
+ * @returns Suggested pattern type and whether to use fuzzy matching
+ */
+declare const suggestPatternConfig: (detected: DetectedPattern[]) => {
+  patternType: "lineStartsWith" | "lineStartsAfter";
+  fuzzy: boolean;
+  metaType?: string;
+};
+/**
+ * Analyzes text and generates a complete suggested rule configuration.
+ *
+ * @param text - Highlighted text from the page
+ * @returns Suggested rule configuration or null if no patterns detected
+ */
+declare const analyzeTextForRule: (text: string) => {
+  template: string;
+  patternType: "lineStartsWith" | "lineStartsAfter";
+  fuzzy: boolean;
+  metaType?: string;
+  detected: DetectedPattern[];
+} | null;
+//#endregion
+export { type Breakpoint, type BreakpointRule, type DetectedPattern, type ExpandResult, type Logger, type Page, type PageRange, type Segment, type SegmentationOptions, type SplitRule, TOKEN_PATTERNS, analyzeTextForRule, containsTokens, detectTokenPatterns, escapeRegex, expandTokens, expandTokensWithCaptures, generateTemplateFromText, getAvailableTokens, getTokenPattern, makeDiacriticInsensitive, normalizeLineEndings, segmentPages, stripHtmlTags, suggestPatternConfig, templateToRegex };
 //# sourceMappingURL=index.d.mts.map

package/dist/index.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.mts","names":[],"sources":["../src/segmentation/fuzzy.ts","../src/segmentation/types.ts","../src/segmentation/segmenter.ts","../src/segmentation/textUtils.ts","../src/segmentation/tokens.ts"],"sourcesContent":[],"mappings":";;AAkEA;AA+FA;;;;;AC9IiB;AAoBG;AAsBM;AAyBC;AAiBH;;;;;;;AAoBC;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EY,cDhYC,WCgYqB,EAAA,CAAA,CAAA,EAAA,MAAc,EAAA,GAAA,MAAA;AAuBhD;AAgFA;;;;ACoKA;;;;;;;;ACvsBA;AAaA;;;;AC6NA;AA2CA;AAWA;AA2DA;AAqGA;AAuBA;AAqBA;AAgBA;;;;;;;;;;cJjWa;;;;AA/Fb;AA+FA;;;;;AC9IiB;AAoBG;AAsBM;AAyBC;AAiBH;;;;KApFnB,YAAA,GAuGC;EACA;EAAmB,KAAA,EAAA,MAAA;AAAA,CAAA;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AAuBA;AAgFA;;;;ACoKA,KDtqBK,eAAA,GC6vBJ;EAvFmC;EAAiB,QAAA,EAAA,MAAA;CAAsB;;;;;ACvsB3E;AAaA;;;;AC6NA;AA2CA;AAWA;AA2DA;AAqGA;AAuBA;AAqBA;AAgBA;KHrcK,qBAAA;;;;;;;;;;;;;;;;;;;;;;;;KAyBA,sBAAA;;;;;;;;;;;;;;;;KAiBA,mBAAA;;;;;;;;;;;;;;KAeA,WAAA,GACC,eACA,kBACA,wBACA,yBACA;;;;;;;KAYD,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA8EO,SAAA;;;;;;;KAYP,eAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAyCS;;;;;;;;;;;;SAaH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA0CC,SAAA,GAAY,cAAc,gBAAgB;;;;;;;;;;;;;KAkB1C,IAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCA,cAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAqCE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCF,UAAA,YAAsB;;;;;;;;;;;;;;;;;;;;;;KAuBtB,mBAAA;;;;;;;;UAQA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA8CM;;;;;;;;;;;;;;;;;;;;;;;;KA0BN,OAAA;;;;;;;;;;;;;;;;;;;;;;;;;;SA6BD;;;;AAhfgB;AAiBH;;;;;;;AAoBC;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AAuBA;AAgFA;;;;ACoKA;;;;;;;;ACvsBA;AAaA;;;;AC6NA;AA2CA;AAWA;AA2DA;AAqGA;AAuBa,cFgPA,YEzOZ,EAPkD,CAAA,KAAM,EFgPrB,IEhPqB,EAAA,EAAA,OAAA,EFgPJ,mBEhPI,EAAA,GFgPkB,OEhPlB,EAAA;;;;AJ3ZzD;AA+FA;;;;AC9IK,cEbQ,aFaI,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AAAA;AAoBG;AAsBM;AAyBC;AAiBH;;;;;AAoBlB,cExGO,oBFwGP,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;ADzDN;AA+FA;;;;;AC9IiB;AAoBG;AAsBM;AAyBC;AAiBH;;;;;;;AAoBC;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AAuBA;AAgFA;;;;ACoKA;;;;;;;;ACvsBA;AAaA;cC6Na,gBAAgB;;;AAA7B;AA2CA;AAWA;AA2DA;AAqGA;AAuBA;AAqBA;AAgBA;;;;;;cAvOa;;;;;;;KAWD,YAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cA2DC,2FAA0F;;;;;;;;;;;;;;;;;;;;cAqG1F;;;;;;;;;;;;;;;;;;;;;;cAuBA,uCAAsC;;;;;;;;;;;;;cAqBtC;;;;;;;;;;;;;;;cAgBA"}
+{"version":3,"file":"index.d.mts","names":[],"sources":["../src/segmentation/fuzzy.ts","../src/segmentation/types.ts","../src/segmentation/segmenter.ts","../src/segmentation/textUtils.ts","../src/segmentation/tokens.ts","../src/pattern-detection.ts"],"sourcesContent":[],"mappings":";;AAkEA;AA+FA;;;;;AC9IiB;AAoBG;AAsBM;AAyBC;AAiBH;;;;;;;AAoBC;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EY,cDhYC,WCgYqB,EAAA,CAAA,CAAA,EAAA,MAAc,EAAA,GAAA,MAAA;AA8BhD;AAiDA;;;;;AAiHA;;;;AC2KA;;;;;;;;ACvyBA;AAaA;;;;AC6NA;AA2CA;AAWA;AA2DA;AAqGA;AAuBA;AAqBA;AAgBA;;;;ACtfA;AA0Da,cL2FA,wBK3FsC,EAAA,CAAA,IAAA,EAAe,MAAA,EAAA,GAAA,MAAA;;;;ALJlE;AA+FA;;;;;AC9IiB;AAoBG;AAsBM;AAyBC;AAiBH;;;;KApFnB,YAAA,GAuGC;EACA;EAAmB,KAAA,EAAA,MAAA;AAAA,CAAA;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAiDA;;;;KA1eK,eAAA,GA0kBc;EAiBP;;;;AC2KZ;;;;;;;;ACvyBA;AAaA;;;;AC6NA;AA2CA;AAWA;AA2DA,KHpSK,qBAAA,GGoSQ;EAqGA;EAuBA,cAAA,EAAA,MAOZ,EAAA;AAcD,CAAA;AAgBA;;;;ACtfA;AA0DA;AA4DA;AAuBA;AAiCA;;;;;;;;;;;;KJpGK,sBAAA;;;;;;;;;;;;;;;;KAiBA,mBAAA;;;;;;;;;;;;;;KAeA,WAAA,GACC,eACA,kBACA,wBACA,yBACA;;;;;;;KAYD,aAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA8EO,SAAA;;;;;;;KAYP,eAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAyCS;;;;;;;;;;;;SAaH;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA0CC,SAAA,GAAY,cAAc,gBAAgB;;;;;;;;;;;;;KAkB1C,IAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCA,cAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;YAqCE;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAqCF,UAAA,YAAsB;;;;;;;;;;;;;;;;;;;;;;;;;UA8BjB,MAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;KAiDL,mBAAA;;;;;;;;UAQA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;gBA8CM;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;WA0CL;;;;;;;;;;;;;;;;KAiBD,OAAA;;;;;;;;;;;;;;;;;;;;;;;;;;SA6BD;;;;AAzkBgB;AAiBH;;;;;;;AAoBC;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAiDA;;;;;AAiHA;;;;AC2KA;;;;;;;;ACvyBA;AAaA;;;;AC6Na,cF6jBA,YE7jBgB,EAAA,CAAA,KAAM,EF6jBC,IE7jBD,EAAA,EAAA,OAAA,EF6jBkB,mBE7jBlB,EAAA,GF6jBwC,OE7jBxC,EAAA;;;;AJ9KnC;AA+FA;;;;AC9IK,cEbQ,aFaI,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GAAA,MAAA;AAAA;AAoBG;AAsBM;AAyBC;AAiBH;;;;;AAoBlB,cExGO,oBFwGP,EAAA,CAAA,OAAA,EAAA,MAAA,EAAA,GAAA,MAAA;;;;ADzDN;AA+FA;;;;;AC9IiB;AAoBG;AAsBM;AAyBC;AAiBH;;;;;;;AAoBC;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAiDA;;;;;AAiHA;;;;AC2KA;;;;;cE7jBa,gBAAgB;;;AD1O7B;AAaA;;;;AC6NA;AA2CA;AAWA;AA2DA;AAqGA;AAuBA;AAqBA;AAgBA;cAvOa;;;AC/Qb;AA0DA;AA4DA;AAuBA;AAiCa,KD4GD,YAAA,GCzFX;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cDoJY,2FAA0F;;;;;;;;;;;;;;;;;;;;cAqG1F;;;;;;;;;;;;;;;;;;;;;;cAuBA,uCAAsC;;;;;;;;;;;;;cAqBtC;;;;;;;;;;;;;;;cAgBA;;;;AJhcb;AA+FA;;;;;AC9IiB;AAoBG;AA+Cf,KI1EO,eAAA,GJ0Ee;EAiBtB;EAeA,KAAA,EAAA,MAAA;EACC;EACA,KAAA,EAAA,MAAA;EACA;EACA,KAAA,EAAA,MAAA;EACA;EAAmB,QAAA,EAAA,MAAA;AAAA,CAAA;AA0FzB;AAAkD;AA4GlD;;;;;AAkBA;AAqCA;AA0EA;AA8BA;AAiDA;;;;AAgGmB,cI3iBN,mBJ2iBM,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,GI3iBgC,eJ2iBhC,EAAA;AAiBnB;;;;AC2KA;;;;;;;;ACvyBA;AAaa,cE+GA,wBF/G2E,EAAA,CAAA,IAAA,EAAA,MAAA,EAAA,QAAA,EE+GvB,eF/GuB,EAAA,EAAA,GAAA,MAAA;;;;AC6NxF;AA2CA;AAWA;AA2Da,cCxMA,oBDwRZ,EAhFsG,CAAA,QAAA,ECvMzF,eDuRb,EAAA,EAAA,GAAA;EAqBY,WAAA,EAAA,gBAAiF,GAAA,iBAAA;EAuBjF,KAAA,EAAA,OAAA;EAqBA,QAAA,CAAA,EAAA,MAAA;AAgBb,CAAA;;;;ACtfA;AA0DA;AA4DA;AAuBa,cAiCA,kBAhCC,EAAA,CAAA,IAAA,EAAA,MAAe,EAAA,GAAA;EAgChB,QAAA,EAAA,MAAA;;;;YAOC"}