fast-sentence-segment 1.1.8__tar.gz → 1.2.1__tar.gz

{fast_sentence_segment-1.1.8 → fast_sentence_segment-1.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fast-sentence-segment
-Version: 1.1.8
+Version: 1.2.1
 Summary: Fast and Efficient Sentence Segmentation
 License: MIT
 License-File: LICENSE
@@ -30,17 +30,25 @@ Description-Content-Type: text/markdown
 [![PyPI version](https://img.shields.io/pypi/v/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)
 [![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![spaCy](https://img.shields.io/badge/spaCy-3.5-blue.svg)](https://spacy.io/)
+[![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)
 
-Fast and efficient sentence segmentation using spaCy. Handles complex edge cases like abbreviations (Dr., Mr., etc.), quoted text, and multi-paragraph documents.
+Fast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.
+
+## Why This Library?
+
+1. **Keep it local**: LLM API calls cost money and send your data to third parties. Run sentence segmentation entirely on your machine.
+2. **spaCy perfected**: spaCy is a great local model, but it makes mistakes. This library fixes most of spaCy's shortcomings.
 
 ## Features
 
 - **Paragraph-aware segmentation**: Returns sentences grouped by paragraph
-- **Abbreviation handling**: Correctly handles "Dr.", "Mr.", "etc." without false splits
+- **Abbreviation handling**: Correctly handles "Dr.", "Mr.", "etc.", "p.m.", "a.m." without false splits
+- **Ellipsis preservation**: Keeps `...` intact while detecting sentence boundaries
+- **Question/exclamation splitting**: Properly splits on `?` and `!` followed by capital letters
 - **Cached processing**: LRU cache for repeated text processing
 - **Flexible output**: Nested lists (by paragraph) or flattened list of sentences
 - **Bullet point & numbered list normalization**: Cleans common list formats
+- **CLI tool**: Command-line interface for quick segmentation
 
 ## Installation
 
@@ -59,12 +67,21 @@ python -m spacy download en_core_web_sm
 ```python
 from fast_sentence_segment import segment_text
 
-text = "Here is a Dr. who says something. And then again, what else? I don't know. Do you?"
+text = "Do you like Dr. Who? I prefer Dr. Strange! Mr. T is also cool."
 
-results = segment_text(text)
-# Returns: [['Here is a Dr. who says something.', 'And then again, what else?', "I don't know.", 'Do you?']]
+results = segment_text(text, flatten=True)
 ```
 
+```json
+[
+  "Do you like Dr. Who?",
+  "I prefer Dr. Strange!",
+  "Mr. T is also cool."
+]
+```
+
+Notice how "Dr. Who?" stays together as a single sentence—the library correctly recognizes that a title followed by a single-word name ending in `?` or `!` is a name reference, not a sentence boundary.
+
 ## Usage
 
 ### Basic Segmentation
@@ -74,16 +91,24 @@ The `segment_text` function returns a list of lists, where each inner list repre
 ```python
 from fast_sentence_segment import segment_text
 
-text = """First paragraph here. It has two sentences.
+text = """Gandalf spoke softly. "All we have to decide is what to do with the time given us."
 
-Second paragraph starts here. This one also has multiple sentences. And a third."""
+Frodo nodded. The weight of the Ring pressed against his chest."""
 
 results = segment_text(text)
-# Returns:
-# [
-#     ['First paragraph here.', 'It has two sentences.'],
-#     ['Second paragraph starts here.', 'This one also has multiple sentences.', 'And a third.']
-# ]
+```
+
+```json
+[
+  [
+    "Gandalf spoke softly.",
+    "\"All we have to decide is what to do with the time given us.\"."
+  ],
+  [
+    "Frodo nodded.",
+    "The weight of the Ring pressed against his chest."
+  ]
+]
 ```
 
 ### Flattened Output
@@ -91,8 +116,17 @@ results = segment_text(text)
 If you don't need paragraph boundaries, use the `flatten` parameter:
 
 ```python
+text = "At 9 a.m. the hobbits set out. By 3 p.m. they reached Rivendell. Mr. Frodo was exhausted."
+
 results = segment_text(text, flatten=True)
-# Returns: ['First paragraph here.', 'It has two sentences.', 'Second paragraph starts here.', ...]
+```
+
+```json
+[
+  "At 9 a.m. the hobbits set out.",
+  "By 3 p.m. they reached Rivendell.",
+  "Mr. Frodo was exhausted."
+]
 ```
 
 ### Direct Segmenter Access
@@ -106,6 +140,36 @@ segmenter = Segmenter()
 results = segmenter.input_text("Your text here.")
 ```
 
+### Command Line Interface
+
+Segment text directly from the terminal:
+
+```bash
+# Direct text input
+echo "Have you seen Dr. Who? It's brilliant!" | segment
+```
+
+```
+Have you seen Dr. Who?
+It's brilliant!
+```
+
+```bash
+# Numbered output
+segment -n "Gandalf paused... You shall not pass! The Balrog roared."
+```
+
+```
+1. Gandalf paused...
+2. You shall not pass!
+3. The Balrog roared.
+```
+
+```bash
+# From file
+segment -f silmarillion.txt
+```
+
 ## API Reference
 
 | Function | Parameters | Returns | Description |
@@ -113,6 +177,14 @@ results = segmenter.input_text("Your text here.")
 | `segment_text()` | `input_text: str`, `flatten: bool = False` | `list` | Main entry point for segmentation |
 | `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |
 
+### CLI Options
+
+| Option | Description |
+|--------|-------------|
+| `text` | Text to segment (positional argument) |
+| `-f, --file` | Read text from file |
+| `-n, --numbered` | Number output lines |
+
 ## Why Nested Lists?
 
 The segmentation process preserves document structure by segmenting into both paragraphs and sentences. Each outer list represents a paragraph, and each inner list contains that paragraph's sentences. This is useful for:
@@ -125,10 +197,19 @@ Use `flatten=True` when you only need sentences without paragraph context.
 
 ## Requirements
 
-- Python 3.8.5+
-- spaCy 3.5.3
+- Python 3.9+
+- spaCy 3.8+
 - en_core_web_sm spaCy model
 
+## How It Works
+
+This library uses spaCy for initial sentence segmentation, then applies surgical post-processing fixes for cases where spaCy's default behavior is incorrect:
+
+1. **Pre-processing**: Normalize numbered lists, preserve ellipses with placeholders
+2. **spaCy segmentation**: Use spaCy's sentence boundary detection
+3. **Post-processing**: Split on abbreviation boundaries, handle `?`/`!` + capital patterns
+4. **Denormalization**: Restore placeholders to original text
+
 ## License
 
 MIT License - see [LICENSE](LICENSE) for details.

fast_sentence_segment-1.2.1/README.md

@@ -0,0 +1,199 @@
+# Fast Sentence Segmentation
+
+[![PyPI version](https://img.shields.io/pypi/v/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)
+[![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)
+
+Fast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.
+
+## Why This Library?
+
+1. **Keep it local**: LLM API calls cost money and send your data to third parties. Run sentence segmentation entirely on your machine.
+2. **spaCy perfected**: spaCy is a great local model, but it makes mistakes. This library fixes most of spaCy's shortcomings.
+
+## Features
+
+- **Paragraph-aware segmentation**: Returns sentences grouped by paragraph
+- **Abbreviation handling**: Correctly handles "Dr.", "Mr.", "etc.", "p.m.", "a.m." without false splits
+- **Ellipsis preservation**: Keeps `...` intact while detecting sentence boundaries
+- **Question/exclamation splitting**: Properly splits on `?` and `!` followed by capital letters
+- **Cached processing**: LRU cache for repeated text processing
+- **Flexible output**: Nested lists (by paragraph) or flattened list of sentences
+- **Bullet point & numbered list normalization**: Cleans common list formats
+- **CLI tool**: Command-line interface for quick segmentation
+
+## Installation
+
+```bash
+pip install fast-sentence-segment
+```
+
+After installation, download the spaCy model:
+
+```bash
+python -m spacy download en_core_web_sm
+```
+
+## Quick Start
+
+```python
+from fast_sentence_segment import segment_text
+
+text = "Do you like Dr. Who? I prefer Dr. Strange! Mr. T is also cool."
+
+results = segment_text(text, flatten=True)
+```
+
+```json
+[
+  "Do you like Dr. Who?",
+  "I prefer Dr. Strange!",
+  "Mr. T is also cool."
+]
+```
+
+Notice how "Dr. Who?" stays together as a single sentence—the library correctly recognizes that a title followed by a single-word name ending in `?` or `!` is a name reference, not a sentence boundary.
+
+## Usage
+
+### Basic Segmentation
+
+The `segment_text` function returns a list of lists, where each inner list represents a paragraph containing its sentences:
+
+```python
+from fast_sentence_segment import segment_text
+
+text = """Gandalf spoke softly. "All we have to decide is what to do with the time given us."
+
+Frodo nodded. The weight of the Ring pressed against his chest."""
+
+results = segment_text(text)
+```
+
+```json
+[
+  [
+    "Gandalf spoke softly.",
+    "\"All we have to decide is what to do with the time given us.\"."
+  ],
+  [
+    "Frodo nodded.",
+    "The weight of the Ring pressed against his chest."
+  ]
+]
+```
+
+### Flattened Output
+
+If you don't need paragraph boundaries, use the `flatten` parameter:
+
+```python
+text = "At 9 a.m. the hobbits set out. By 3 p.m. they reached Rivendell. Mr. Frodo was exhausted."
+
+results = segment_text(text, flatten=True)
+```
+
+```json
+[
+  "At 9 a.m. the hobbits set out.",
+  "By 3 p.m. they reached Rivendell.",
+  "Mr. Frodo was exhausted."
+]
+```
+
+### Direct Segmenter Access
+
+For more control, use the `Segmenter` class directly:
+
+```python
+from fast_sentence_segment import Segmenter
+
+segmenter = Segmenter()
+results = segmenter.input_text("Your text here.")
+```
+
+### Command Line Interface
+
+Segment text directly from the terminal:
+
+```bash
+# Direct text input
+echo "Have you seen Dr. Who? It's brilliant!" | segment
+```
+
+```
+Have you seen Dr. Who?
+It's brilliant!
+```
+
+```bash
+# Numbered output
+segment -n "Gandalf paused... You shall not pass! The Balrog roared."
+```
+
+```
+1. Gandalf paused...
+2. You shall not pass!
+3. The Balrog roared.
+```
+
+```bash
+# From file
+segment -f silmarillion.txt
+```
+
+## API Reference
+
+| Function | Parameters | Returns | Description |
+|----------|------------|---------|-------------|
+| `segment_text()` | `input_text: str`, `flatten: bool = False` | `list` | Main entry point for segmentation |
+| `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |
+
+### CLI Options
+
+| Option | Description |
+|--------|-------------|
+| `text` | Text to segment (positional argument) |
+| `-f, --file` | Read text from file |
+| `-n, --numbered` | Number output lines |
+
+## Why Nested Lists?
+
+The segmentation process preserves document structure by segmenting into both paragraphs and sentences. Each outer list represents a paragraph, and each inner list contains that paragraph's sentences. This is useful for:
+
+- Document structure analysis
+- Paragraph-level processing
+- Maintaining original text organization
+
+Use `flatten=True` when you only need sentences without paragraph context.
+
+## Requirements
+
+- Python 3.9+
+- spaCy 3.8+
+- en_core_web_sm spaCy model
+
+## How It Works
+
+This library uses spaCy for initial sentence segmentation, then applies surgical post-processing fixes for cases where spaCy's default behavior is incorrect:
+
+1. **Pre-processing**: Normalize numbered lists, preserve ellipses with placeholders
+2. **spaCy segmentation**: Use spaCy's sentence boundary detection
+3. **Post-processing**: Split on abbreviation boundaries, handle `?`/`!` + capital patterns
+4. **Denormalization**: Restore placeholders to original text
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Run tests (`make test`)
+4. Commit your changes
+5. Push to the branch
+6. Open a Pull Request

fast_sentence_segment-1.2.1/fast_sentence_segment/cli.py

@@ -0,0 +1,56 @@
+# -*- coding: UTF-8 -*-
+"""CLI for fast-sentence-segment."""
+
+import argparse
+import logging
+import sys
+
+from fast_sentence_segment import segment_text
+
+logging.disable(logging.CRITICAL)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="segment",
+        description="Segment text into sentences",
+    )
+    parser.add_argument(
+        "text",
+        nargs="?",
+        help="Text to segment (or use stdin)",
+    )
+    parser.add_argument(
+        "-f", "--file",
+        help="Read text from file",
+    )
+    parser.add_argument(
+        "-n", "--numbered",
+        action="store_true",
+        help="Number output lines",
+    )
+    args = parser.parse_args()
+
+    # Get input text
+    if args.file:
+        with open(args.file, "r", encoding="utf-8") as f:
+            text = f.read()
+    elif args.text:
+        text = args.text
+    elif not sys.stdin.isatty():
+        text = sys.stdin.read()
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+    # Segment and output
+    sentences = segment_text(text.strip(), flatten=True)
+    for i, sentence in enumerate(sentences, 1):
+        if args.numbered:
+            print(f"{i}. {sentence}")
+        else:
+            print(sentence)
+
+
+if __name__ == "__main__":
+    main()

fast_sentence_segment-1.2.1/fast_sentence_segment/dmo/__init__.py

@@ -0,0 +1,10 @@
+from .abbreviation_merger import AbbreviationMerger
+from .abbreviation_splitter import AbbreviationSplitter
+from .title_name_merger import TitleNameMerger
+from .bullet_point_cleaner import BulletPointCleaner
+from .ellipsis_normalizer import EllipsisNormalizer
+from .newlines_to_periods import NewlinesToPeriods
+from .post_process_sentences import PostProcessStructure
+from .question_exclamation_splitter import QuestionExclamationSplitter
+from .spacy_doc_segmenter import SpacyDocSegmenter
+from .numbered_list_normalizer import NumberedListNormalizer

fast_sentence_segment-1.2.1/fast_sentence_segment/dmo/abbreviation_merger.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python
+# -*- coding: UTF-8 -*-
+"""
+Merge sentences that spaCy incorrectly split at abbreviation boundaries.
+
+When spaCy incorrectly splits after an abbreviation (e.g., "ext. 5" becomes
+["ext.", "5. Ask for help."]), this component merges them back together
+using specific known patterns.
+
+Reference: https://github.com/craigtrim/fast-sentence-segment/issues/3
+"""
+
+import re
+from typing import List, Optional, Tuple
+
+from fast_sentence_segment.core import BaseObject
+
+
+# Patterns where spaCy incorrectly splits after an abbreviation.
+# Format: (ending_pattern, extract_pattern)
+#   - ending_pattern: regex to match end of current sentence
+#   - extract_pattern: regex to extract the portion to merge from next sentence
+#
+# The extract_pattern MUST have a capture group for the portion to merge.
+# Whatever is NOT captured remains as a separate sentence.
+
+MERGE_PATTERNS: List[Tuple[str, str]] = [
+
+    # ext. 5, Ext. 123, EXT. 42
+    (r"(?i)\bext\.$", r"^(\d+\.?)\s*"),
+
+    # no. 5, No. 42, NO. 100
+    (r"(?i)\bno\.$", r"^(\d+\.?)\s*"),
+
+    # vol. 3, Vol. 42, VOL. 1
+    (r"(?i)\bvol\.$", r"^(\d+\.?)\s*"),
+
+    # pt. 2, Pt. 1, PT. 3
+    (r"(?i)\bpt\.$", r"^(\d+\.?)\s*"),
+
+    # ch. 5, Ch. 10, CH. 3
+    (r"(?i)\bch\.$", r"^(\d+\.?)\s*"),
+
+    # sec. 3, Sec. 14, SEC. 2
+    (r"(?i)\bsec\.$", r"^(\d+(?:\.\d+)?\.?)\s*"),
+
+    # fig. 1, Fig. 3.2, FIG. 10
+    (r"(?i)\bfig\.$", r"^(\d+(?:\.\d+)?\.?)\s*"),
+
+    # p. 42, P. 100
+    (r"(?i)\bp\.$", r"^(\d+\.?)\s*"),
+
+    # pp. 42-50, PP. 100-110
+    (r"(?i)\bpp\.$", r"^(\d+(?:-\d+)?\.?)\s*"),
+
+    # art. 5, Art. 12, ART. 1
+    (r"(?i)\bart\.$", r"^(\d+\.?)\s*"),
+
+]
+
+
+class AbbreviationMerger(BaseObject):
+    """Merge sentences incorrectly split at abbreviation boundaries."""
+
+    def __init__(self):
+        """
+        Created:
+            27-Dec-2024
+            craigtrim@gmail.com
+        Reference:
+            https://github.com/craigtrim/fast-sentence-segment/issues/3
+        """
+        BaseObject.__init__(self, __name__)
+        # Compile patterns for efficiency
+        self._patterns = [
+            (re.compile(ending), re.compile(extract))
+            for ending, extract in MERGE_PATTERNS
+        ]
+
+    def _try_merge(self, current: str, next_sent: str) -> Optional[Tuple[str, str]]:
+        """Try to merge two sentences based on known patterns.
+
+        Args:
+            current: Current sentence
+            next_sent: Next sentence
+
+        Returns:
+            Tuple of (merged_sentence, remainder) if merge needed, else None
+        """
+        current = current.strip()
+        next_sent = next_sent.strip()
+
+        for ending_pattern, extract_pattern in self._patterns:
+            if ending_pattern.search(current):
+                match = extract_pattern.match(next_sent)
+                if match:
+                    # Extract the portion to merge
+                    extracted = match.group(1)
+                    # Get the remainder (everything after the match)
+                    remainder = next_sent[match.end():].strip()
+                    # Build merged sentence
+                    merged = current + " " + extracted
+                    return (merged, remainder)
+
+        return None
+
+    def process(self, sentences: List[str]) -> List[str]:
+        """Process a list of sentences, merging incorrectly split ones.
+
+        Args:
+            sentences: List of sentences from spaCy
+
+        Returns:
+            List of sentences with incorrect splits merged
+        """
+        if not sentences:
+            return sentences
+
+        result = []
+        i = 0
+
+        while i < len(sentences):
+            current = sentences[i]
+
+            # Check if we should merge with next sentence
+            if i + 1 < len(sentences):
+                next_sent = sentences[i + 1]
+                merge_result = self._try_merge(current, next_sent)
+
+                if merge_result:
+                    merged, remainder = merge_result
+                    result.append(merged)
+
+                    # If there's a remainder, it becomes a new sentence to process
+                    if remainder:
+                        # Insert remainder back for processing
+                        sentences = sentences[:i+2] + [remainder] + sentences[i+2:]
+                        sentences[i+1] = remainder
+
+                    i += 2
+                    continue
+
+            result.append(current)
+            i += 1
+
+        return result

fast_sentence_segment-1.2.1/fast_sentence_segment/dmo/abbreviation_splitter.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python
+# -*- coding: UTF-8 -*-
+"""
+Split sentences at abbreviation boundaries.
+
+When spaCy fails to detect a sentence boundary after an abbreviation
+(e.g., "I woke at 6 a.m. It was dark."), this component splits the
+sentence by detecting the pattern: abbreviation + space + Capital letter.
+
+Reference: https://github.com/craigtrim/fast-sentence-segment/issues/3
+"""
+
+import re
+from typing import List
+
+from fast_sentence_segment.core import BaseObject
+from fast_sentence_segment.dmo.abbreviations import SENTENCE_ENDING_ABBREVIATIONS
+
+
+class AbbreviationSplitter(BaseObject):
+    """Split sentences at abbreviation boundaries."""
+
+    def __init__(self):
+        """
+        Created:
+            27-Dec-2024
+            craigtrim@gmail.com
+        Reference:
+            https://github.com/craigtrim/fast-sentence-segment/issues/3
+        """
+        BaseObject.__init__(self, __name__)
+        self._pattern = self._build_pattern()
+
+    def _build_pattern(self) -> re.Pattern:
+        """Build regex pattern to match abbreviation + capital letter.
+
+        Pattern matches:
+        - A known sentence-ending abbreviation (escaped for regex)
+        - Followed by one or more spaces
+        - Followed by a capital letter (start of new sentence)
+
+        Note: Title abbreviations (Dr., Mr., etc.) are excluded because
+        they are typically followed by names, not new sentences.
+
+        Returns:
+            Compiled regex pattern
+        """
+        escaped_abbrevs = [re.escape(abbr) for abbr in SENTENCE_ENDING_ABBREVIATIONS]
+        abbrev_pattern = "|".join(escaped_abbrevs)
+        pattern = rf"({abbrev_pattern})\s+([A-Z])"
+        return re.compile(pattern)
+
+    def _split_sentence(self, sentence: str) -> List[str]:
+        """Split a single sentence at abbreviation boundaries.
+
+        Args:
+            sentence: A sentence that may contain abbreviation boundaries
+
+        Returns:
+            List of one or more sentences
+        """
+        results = []
+        remaining = sentence
+
+        while True:
+            match = self._pattern.search(remaining)
+            if not match:
+                if remaining.strip():
+                    results.append(remaining.strip())
+                break
+
+            split_pos = match.end(1)
+
+            before = remaining[:split_pos].strip()
+            if before:
+                results.append(before)
+
+            remaining = remaining[split_pos:].strip()
+
+        return results if results else [sentence]
+
+    def process(self, sentences: List[str]) -> List[str]:
+        """Process a list of sentences, splitting at abbreviation boundaries.
+
+        Args:
+            sentences: List of sentences from spaCy
+
+        Returns:
+            List of sentences with abbreviation boundaries properly split
+        """
+        result = []
+        for sentence in sentences:
+            split_sentences = self._split_sentence(sentence)
+            result.extend(split_sentences)
+        return result