fast-sentence-segment 1.2.1__tar.gz → 1.3.0__tar.gz

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/PKG-INFO

@@ -1,25 +1,29 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.1
 Name: fast-sentence-segment
-Version: 1.2.1
+Version: 1.3.0
 Summary: Fast and Efficient Sentence Segmentation
+Home-page: https://github.com/craigtrim/fast-sentence-segment
 License: MIT
-License-File: LICENSE
 Keywords: nlp,text,preprocess,segment
 Author: Craig Trim
 Author-email: craigtrim@gmail.com
 Maintainer: Craig Trim
 Maintainer-email: craigtrim@gmail.com
 Requires-Python: >=3.9,<4.0
-Classifier: Development Status :: 4 - Beta
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
 Requires-Dist: spacy (>=3.8.0,<4.0.0)
 Project-URL: Bug Tracker, https://github.com/craigtrim/fast-sentence-segment/issues
 Project-URL: Repository, https://github.com/craigtrim/fast-sentence-segment
@@ -31,6 +35,8 @@ Description-Content-Type: text/markdown
 [![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)
+[![Downloads](https://static.pepy.tech/badge/fast-sentence-segment)](https://pepy.tech/project/fast-sentence-segment)
+[![Downloads/Month](https://static.pepy.tech/badge/fast-sentence-segment/month)](https://pepy.tech/project/fast-sentence-segment)
 
 Fast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.
 
@@ -142,48 +148,36 @@ results = segmenter.input_text("Your text here.")
 
 ### Command Line Interface
 
-Segment text directly from the terminal:
-
 ```bash
-# Direct text input
-echo "Have you seen Dr. Who? It's brilliant!" | segment
-```
+# Inline text
+segment "Gandalf paused... You shall not pass! The Balrog roared."
 
-```
-Have you seen Dr. Who?
-It's brilliant!
-```
+# Pipe from stdin
+echo "Have you seen Dr. Who? It's brilliant!" | segment
 
-```bash
 # Numbered output
-segment -n "Gandalf paused... You shall not pass! The Balrog roared."
-```
+segment -n -f silmarillion.txt
 
-```
-1. Gandalf paused...
-2. You shall not pass!
-3. The Balrog roared.
-```
+# File-to-file (one sentence per line)
+segment-file --input-file book.txt --output-file sentences.txt
 
-```bash
-# From file
-segment -f silmarillion.txt
+# Unwrap hard-wrapped e-texts (Project Gutenberg, etc.)
+segment-file --input-file book.txt --output-file sentences.txt --unwrap
 ```
 
 ## API Reference
 
 | Function | Parameters | Returns | Description |
 |----------|------------|---------|-------------|
-| `segment_text()` | `input_text: str`, `flatten: bool = False` | `list` | Main entry point for segmentation |
+| `segment_text()` | `input_text: str`, `flatten: bool = False`, `unwrap: bool = False` | `list` | Main entry point for segmentation |
 | `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |
 
-### CLI Options
+### CLI Commands
 
-| Option | Description |
-|--------|-------------|
-| `text` | Text to segment (positional argument) |
-| `-f, --file` | Read text from file |
-| `-n, --numbered` | Number output lines |
+| Command | Description |
+|---------|-------------|
+| `segment [text]` | Segment text from argument, `-f FILE`, or stdin. Use `-n` for numbered output. |
+| `segment-file --input-file IN --output-file OUT [--unwrap]` | Segment a file and write one sentence per line. Use `--unwrap` for hard-wrapped e-texts. |
 
 ## Why Nested Lists?
 

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/README.md

@@ -4,6 +4,8 @@
 [![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)
+[![Downloads](https://static.pepy.tech/badge/fast-sentence-segment)](https://pepy.tech/project/fast-sentence-segment)
+[![Downloads/Month](https://static.pepy.tech/badge/fast-sentence-segment/month)](https://pepy.tech/project/fast-sentence-segment)
 
 Fast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.
 
@@ -115,48 +117,36 @@ results = segmenter.input_text("Your text here.")
 
 ### Command Line Interface
 
-Segment text directly from the terminal:
-
 ```bash
-# Direct text input
-echo "Have you seen Dr. Who? It's brilliant!" | segment
-```
+# Inline text
+segment "Gandalf paused... You shall not pass! The Balrog roared."
 
-```
-Have you seen Dr. Who?
-It's brilliant!
-```
+# Pipe from stdin
+echo "Have you seen Dr. Who? It's brilliant!" | segment
 
-```bash
 # Numbered output
-segment -n "Gandalf paused... You shall not pass! The Balrog roared."
-```
+segment -n -f silmarillion.txt
 
-```
-1. Gandalf paused...
-2. You shall not pass!
-3. The Balrog roared.
-```
+# File-to-file (one sentence per line)
+segment-file --input-file book.txt --output-file sentences.txt
 
-```bash
-# From file
-segment -f silmarillion.txt
+# Unwrap hard-wrapped e-texts (Project Gutenberg, etc.)
+segment-file --input-file book.txt --output-file sentences.txt --unwrap
 ```
 
 ## API Reference
 
 | Function | Parameters | Returns | Description |
 |----------|------------|---------|-------------|
-| `segment_text()` | `input_text: str`, `flatten: bool = False` | `list` | Main entry point for segmentation |
+| `segment_text()` | `input_text: str`, `flatten: bool = False`, `unwrap: bool = False` | `list` | Main entry point for segmentation |
 | `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |
 
-### CLI Options
+### CLI Commands
 
-| Option | Description |
-|--------|-------------|
-| `text` | Text to segment (positional argument) |
-| `-f, --file` | Read text from file |
-| `-n, --numbered` | Number output lines |
+| Command | Description |
+|---------|-------------|
+| `segment [text]` | Segment text from argument, `-f FILE`, or stdin. Use `-n` for numbered output. |
+| `segment-file --input-file IN --output-file OUT [--unwrap]` | Segment a file and write one sentence per line. Use `--unwrap` for hard-wrapped e-texts. |
 
 ## Why Nested Lists?
 

fast_sentence_segment-1.3.0/fast_sentence_segment/__init__.py

@@ -0,0 +1,51 @@
+from .bp import *
+from .svc import *
+from .dmo import *
+
+from .bp.segmenter import Segmenter
+from .dmo.unwrap_hard_wrapped_text import unwrap_hard_wrapped_text
+from .dmo.normalize_quotes import normalize_quotes
+
+segment = Segmenter().input_text
+
+
+def segment_text(
+    input_text: str,
+    flatten: bool = False,
+    unwrap: bool = False,
+    normalize: bool = True,
+) -> list:
+    """Segment text into sentences.
+
+    Args:
+        input_text: The text to segment.
+        flatten: If True, return a flat list of sentences instead of
+            nested paragraphs.
+        unwrap: If True, unwrap hard-wrapped lines (e.g., Project
+            Gutenberg e-texts) before segmenting.
+        normalize: If True (default), normalize unicode quote variants
+            to ASCII equivalents before segmenting. Ensures consistent
+            quote characters for downstream processing.
+
+    Returns:
+        List of sentences (if flatten=True) or list of paragraph
+        groups, each containing a list of sentences.
+
+    Related GitHub Issue:
+        #6 - Review findings from Issue #5
+        https://github.com/craigtrim/fast-sentence-segment/issues/6
+    """
+    if unwrap:
+        input_text = unwrap_hard_wrapped_text(input_text)
+
+    if normalize:
+        input_text = normalize_quotes(input_text)
+
+    results = segment(input_text)
+
+    if flatten:
+        flat = []
+        [[flat.append(y) for y in x] for x in results]
+        return flat
+
+    return results

fast_sentence_segment-1.3.0/fast_sentence_segment/cli.py

@@ -0,0 +1,144 @@
+# -*- coding: UTF-8 -*-
+"""CLI for fast-sentence-segment."""
+
+import argparse
+import logging
+import os
+import sys
+import time
+
+from fast_sentence_segment import segment_text
+from fast_sentence_segment.dmo.group_quoted_sentences import format_grouped_sentences
+
+logging.disable(logging.CRITICAL)
+
+# ANSI color codes
+BOLD = "\033[1m"
+DIM = "\033[2m"
+CYAN = "\033[36m"
+GREEN = "\033[32m"
+YELLOW = "\033[33m"
+RESET = "\033[0m"
+
+
+def _header(title: str):
+    print(f"\n{BOLD}{CYAN}{title}{RESET}")
+    print(f"{DIM}{'─' * 40}{RESET}")
+
+
+def _param(label: str, value: str):
+    print(f"  {DIM}{label}:{RESET} {value}")
+
+
+def _done(msg: str):
+    print(f"\n  {GREEN}✓{RESET} {msg}")
+
+
+def _file_size(path: str) -> str:
+    size = os.path.getsize(path)
+    if size < 1024:
+        return f"{size} B"
+    elif size < 1024 * 1024:
+        return f"{size / 1024:.1f} KB"
+    return f"{size / (1024 * 1024):.1f} MB"
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="segment",
+        description="Segment text into sentences",
+    )
+    parser.add_argument(
+        "text",
+        nargs="?",
+        help="Text to segment (or use stdin)",
+    )
+    parser.add_argument(
+        "-f", "--file",
+        help="Read text from file",
+    )
+    parser.add_argument(
+        "-n", "--numbered",
+        action="store_true",
+        help="Number output lines",
+    )
+    args = parser.parse_args()
+
+    # Get input text
+    if args.file:
+        with open(args.file, "r", encoding="utf-8") as f:
+            text = f.read()
+    elif args.text:
+        text = args.text
+    elif not sys.stdin.isatty():
+        text = sys.stdin.read()
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+    # Segment and output
+    sentences = segment_text(text.strip(), flatten=True)
+    for i, sentence in enumerate(sentences, 1):
+        if args.numbered:
+            print(f"{i}. {sentence}")
+        else:
+            print(sentence)
+
+
+def file_main():
+    parser = argparse.ArgumentParser(
+        prog="segment-file",
+        description="Segment a text file into sentences and write to an output file",
+    )
+    parser.add_argument(
+        "--input-file", required=True,
+        help="Path to input text file",
+    )
+    parser.add_argument(
+        "--output-file", required=True,
+        help="Path to output file",
+    )
+    parser.add_argument(
+        "--unwrap", action="store_true",
+        help="Unwrap hard-wrapped lines (e.g., Project Gutenberg e-texts)",
+    )
+    parser.add_argument(
+        "--no-normalize-quotes", action="store_true",
+        help="Disable unicode quote normalization to ASCII equivalents",
+    )
+    args = parser.parse_args()
+
+    _header("segment-file")
+    _param("Input", args.input_file)
+    _param("Output", args.output_file)
+    _param("Size", _file_size(args.input_file))
+    if args.unwrap:
+        _param("Unwrap", "enabled")
+
+    print(f"\n  {YELLOW}Segmenting...{RESET}", end="", flush=True)
+
+    with open(args.input_file, "r", encoding="utf-8") as f:
+        text = f.read()
+
+    start = time.perf_counter()
+    normalize = not args.no_normalize_quotes
+    sentences = segment_text(
+        text.strip(), flatten=True, unwrap=args.unwrap, normalize=normalize,
+    )
+    elapsed = time.perf_counter() - start
+
+    with open(args.output_file, "w", encoding="utf-8") as f:
+        if args.unwrap:
+            f.write(format_grouped_sentences(sentences) + "\n")
+        else:
+            for sentence in sentences:
+                f.write(sentence + "\n")
+
+    print(f"\r  {' ' * 20}\r", end="")
+    _done(f"{len(sentences):,} sentences in {elapsed:.2f}s")
+    _done(f"Written to {args.output_file}")
+    print()
+
+
+if __name__ == "__main__":
+    main()

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/fast_sentence_segment/dmo/__init__.py

@@ -8,3 +8,7 @@ from .post_process_sentences import PostProcessStructure
 from .question_exclamation_splitter import QuestionExclamationSplitter
 from .spacy_doc_segmenter import SpacyDocSegmenter
 from .numbered_list_normalizer import NumberedListNormalizer
+from .unwrap_hard_wrapped_text import unwrap_hard_wrapped_text
+from .normalize_quotes import normalize_quotes
+from .group_quoted_sentences import group_quoted_sentences, format_grouped_sentences
+from .strip_trailing_period_after_quote import StripTrailingPeriodAfterQuote

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/fast_sentence_segment/dmo/abbreviations.py

@@ -20,6 +20,8 @@ SENTENCE_ENDING_ABBREVIATIONS: List[str] = [
     # Common sentence-enders
     "etc.",
     "ext.",
+    "approx.",
+    "dept.",
 
     # Academic degrees (when at end of sentence)
     "Ph.D.",
@@ -32,6 +34,9 @@ SENTENCE_ENDING_ABBREVIATIONS: List[str] = [
     "J.D.",
     "D.D.S.",
     "R.N.",
+    "M.B.A.",
+    "LL.B.",
+    "LL.M.",
 
     # Business (when at end of sentence)
     "Inc.",
@@ -39,6 +44,14 @@ SENTENCE_ENDING_ABBREVIATIONS: List[str] = [
     "Ltd.",
     "Co.",
     "Bros.",
+    "LLC.",
+    "LLP.",
+
+    # Academic/legal citations (can end sentences)
+    "ibid.",
+    "Ibid.",
+    "cf.",
+    "Cf.",
 
     # Countries/Regions (when at end of sentence)
     "U.S.",
@@ -62,23 +75,59 @@ TITLE_ABBREVIATIONS: List[str] = [
     "Sr.",
     "Jr.",
     "Rev.",
+    "Hon.",
+    "Esq.",
+
+    # French/formal titles (common in translated literature)
+    "Mme.",
+    "Mlle.",
+    "Messrs.",
+
+    # Military ranks
     "Gen.",
     "Col.",
     "Capt.",
     "Lt.",
     "Sgt.",
+    "Maj.",
+    "Cpl.",
+    "Pvt.",
+    "Adm.",
+    "Cmdr.",
+
+    # Political titles
     "Rep.",
     "Sen.",
     "Gov.",
     "Pres.",
-    "Hon.",
+
+    # Ecclesiastical titles
+    "Fr.",
+    "Msgr.",
 
     # Geographic prefixes
     "St.",
     "Mt.",
     "Ft.",
+    "Ave.",
+    "Blvd.",
+    "Rd.",
 
-    # Other prefixes
+    # Latin terms (never end sentences -- always introduce clauses)
+    # Include common inconsistent forms: with/without internal periods,
+    # and with trailing comma (the most common real-world form)
+    "i.e.",
+    "i.e.,",
+    "ie.",
+    "ie.,",
+    "e.g.",
+    "e.g.,",
+    "eg.",
+    "eg.,",
+    "viz.",
+    "viz.,",
+
+    # Reference/numbering prefixes
     "Fig.",
     "fig.",
     "Sec.",
@@ -93,4 +142,8 @@ TITLE_ABBREVIATIONS: List[str] = [
     "no.",
     "Pt.",
     "pt.",
+
+    # Legal / adversarial
+    "vs.",
+    "Vs.",
 ]

fast_sentence_segment-1.3.0/fast_sentence_segment/dmo/group_quoted_sentences.py

@@ -0,0 +1,141 @@
+# -*- coding: UTF-8 -*-
+"""Group sentences that belong to the same open-quote span.
+
+When outputting segmented text with blank-line separators, sentences
+that open with a double quote but do not close it should be grouped
+with subsequent sentences (no blank line between them) until the
+closing quote is found.
+
+Related GitHub Issues:
+    #5 - Normalize quotes and group open-quote sentences in unwrap mode
+    https://github.com/craigtrim/fast-sentence-segment/issues/5
+
+    #6 - Review findings from Issue #5
+    https://github.com/craigtrim/fast-sentence-segment/issues/6
+"""
+
+from typing import List
+
+# Maximum number of sentences that can be grouped under a single
+# open-quote span before the quote state is forcibly reset. This
+# bounds the damage from a stray quote character (e.g., OCR artifact)
+# which would otherwise corrupt grouping for all subsequent sentences.
+#
+# A typical quoted passage in literature rarely exceeds 20 sentences.
+# This limit is deliberately generous to avoid false resets on
+# legitimately long quoted passages while still preventing runaway
+# grouping on malformed input.
+MAX_QUOTE_GROUP_SIZE = 20
+
+
+def group_quoted_sentences(sentences: List[str]) -> List[List[str]]:
+    """Group sentences into blocks based on open/close quote tracking.
+
+    Sentences within an unclosed double-quote span are grouped together
+    into the same block. Sentences outside of a quote span each form
+    their own block.
+
+    When rendered, each block is joined by newlines, and blocks are
+    separated by blank lines (double newlines).
+
+    The algorithm tracks the quote state by counting ASCII double quote
+    characters in each sentence. An odd count toggles the open/close
+    state. When a quote is open, subsequent sentences are appended to
+    the current group rather than starting a new one.
+
+    A safety limit (MAX_QUOTE_GROUP_SIZE) prevents a stray or malformed
+    quote from swallowing all remaining sentences into one group. When
+    the limit is reached, the current group is flushed and the quote
+    state is reset. This bounds corruption from OCR artifacts or
+    encoding errors to a bounded window rather than the entire document.
+
+    Args:
+        sentences: Flat list of segmented sentences.
+
+    Returns:
+        List of sentence groups. Each group is a list of sentences
+        that should be rendered together without blank-line separators.
+
+    Example:
+        >>> groups = group_quoted_sentences([
+        ...     '"The probability lies in that direction.',
+        ...     'And if we take this as a working hypothesis."',
+        ...     'He paused.',
+        ... ])
+        >>> groups
+        [['"The probability lies in that direction.',
+          'And if we take this as a working hypothesis."'],
+         ['He paused.']]
+
+    Related GitHub Issues:
+        #5 - Normalize quotes and group open-quote sentences in unwrap mode
+        https://github.com/craigtrim/fast-sentence-segment/issues/5
+
+        #6 - Review findings from Issue #5
+        https://github.com/craigtrim/fast-sentence-segment/issues/6
+    """
+    if not sentences:
+        return []
+
+    groups: List[List[str]] = []
+    current_group: List[str] = []
+    quote_open = False
+
+    for sentence in sentences:
+        quote_count = sentence.count('"')
+
+        if not quote_open:
+            # Starting a new group
+            if current_group:
+                groups.append(current_group)
+            current_group = [sentence]
+        else:
+            # Inside an open quote span -- append to current group
+            current_group.append(sentence)
+
+        # Toggle quote state on odd quote count
+        if quote_count % 2 == 1:
+            quote_open = not quote_open
+
+        # Safety: if a group grows beyond the limit, the quote is
+        # likely corrupted (stray quote character). Flush the group
+        # and reset state to prevent runaway grouping.
+        if quote_open and len(current_group) >= MAX_QUOTE_GROUP_SIZE:
+            groups.append(current_group)
+            current_group = []
+            quote_open = False
+
+    # Flush the final group
+    if current_group:
+        groups.append(current_group)
+
+    return groups
+
+
+def format_grouped_sentences(sentences: List[str]) -> str:
+    """Format sentences with quote-aware blank-line separation.
+
+    Sentences within the same quoted span are separated by single
+    newlines. Sentence groups are separated by blank lines (double
+    newlines).
+
+    Args:
+        sentences: Flat list of segmented sentences.
+
+    Returns:
+        Formatted string with appropriate line separation.
+
+    Example:
+        >>> text = format_grouped_sentences([
+        ...     '"The probability lies in that direction.',
+        ...     'And if we take this as a working hypothesis."',
+        ...     'He paused.',
+        ... ])
+        >>> print(text)
+        "The probability lies in that direction.
+        And if we take this as a working hypothesis."
+        <BLANKLINE>
+        He paused.
+    """
+    groups = group_quoted_sentences(sentences)
+    return '\n\n'.join('\n'.join(group) for group in groups)

fast_sentence_segment-1.3.0/fast_sentence_segment/dmo/normalize_quotes.py

@@ -0,0 +1,80 @@
+# -*- coding: UTF-8 -*-
+"""Normalize unicode quote variants to ASCII equivalents.
+
+E-texts use a variety of quote characters (curly/smart quotes, unicode
+variants, primes, guillemets). This module normalizes all quote variants
+to their standard ASCII equivalents: double quote (") and single
+quote/apostrophe (').
+
+Related GitHub Issues:
+    #5 - Normalize quotes and group open-quote sentences in unwrap mode
+    https://github.com/craigtrim/fast-sentence-segment/issues/5
+
+    #6 - Review findings from Issue #5
+    https://github.com/craigtrim/fast-sentence-segment/issues/6
+"""
+
+import re
+
+# Unicode double quote variants to normalize to ASCII " (U+0022).
+#
+# U+201C  " LEFT DOUBLE QUOTATION MARK
+# U+201D  " RIGHT DOUBLE QUOTATION MARK
+# U+201E  „ DOUBLE LOW-9 QUOTATION MARK
+# U+201F  ‟ DOUBLE HIGH-REVERSED-9 QUOTATION MARK
+# U+00AB  « LEFT-POINTING DOUBLE ANGLE QUOTATION MARK
+# U+00BB  » RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
+# U+2033  ″ DOUBLE PRIME
+# U+301D  〝 REVERSED DOUBLE PRIME QUOTATION MARK
+# U+301E  〞 DOUBLE PRIME QUOTATION MARK
+# U+301F  〟 LOW DOUBLE PRIME QUOTATION MARK
+# U+FF02  ＂ FULLWIDTH QUOTATION MARK
+DOUBLE_QUOTE_PATTERN = re.compile(
+    '[\u201c\u201d\u201e\u201f\u00ab\u00bb\u2033\u301d\u301e\u301f\uff02]'
+)
+
+# Unicode single quote variants to normalize to ASCII ' (U+0027).
+#
+# U+2018  ' LEFT SINGLE QUOTATION MARK
+# U+2019  ' RIGHT SINGLE QUOTATION MARK
+# U+201A  ‚ SINGLE LOW-9 QUOTATION MARK
+# U+201B  ‛ SINGLE HIGH-REVERSED-9 QUOTATION MARK
+# U+2039  ‹ SINGLE LEFT-POINTING ANGLE QUOTATION MARK
+# U+203A  › SINGLE RIGHT-POINTING ANGLE QUOTATION MARK
+# U+2032  ′ PRIME
+# U+FF07  ＇ FULLWIDTH APOSTROPHE
+# U+0060  ` GRAVE ACCENT (used as opening quote in some e-texts)
+# U+00B4  ´ ACUTE ACCENT (used as closing quote in some e-texts)
+SINGLE_QUOTE_PATTERN = re.compile(
+    '[\u2018\u2019\u201a\u201b\u2039\u203a\u2032\uff07\u0060\u00b4]'
+)
+
+
+def normalize_quotes(text: str) -> str:
+    """Replace all unicode quote variants with their ASCII equivalents.
+
+    Double quote variants are normalized to ASCII " (U+0022).
+    Single quote variants are normalized to ASCII ' (U+0027).
+
+    Args:
+        text: Input text potentially containing unicode quotes.
+
+    Returns:
+        Text with all quote variants replaced by ASCII equivalents.
+
+    Example:
+        >>> normalize_quotes('\u201cHello,\u201d she said.')
+        '"Hello," she said.'
+        >>> normalize_quotes('It\u2019s fine.')
+        "It's fine."
+
+    Related GitHub Issues:
+        #5 - Normalize quotes and group open-quote sentences in unwrap mode
+        https://github.com/craigtrim/fast-sentence-segment/issues/5
+
+        #6 - Review findings from Issue #5
+        https://github.com/craigtrim/fast-sentence-segment/issues/6
+    """
+    text = DOUBLE_QUOTE_PATTERN.sub('"', text)
+    text = SINGLE_QUOTE_PATTERN.sub("'", text)
+    return text

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/fast_sentence_segment/dmo/spacy_doc_segmenter.py

@@ -23,18 +23,31 @@ class SpacyDocSegmenter(BaseObject):
 
     @staticmethod
     def _append_period(a_sentence: str) -> str:
+        """Append a period if the sentence lacks terminal punctuation.
+
+        Checks for terminal punctuation (. ? ! :) after stripping any
+        trailing quote characters (" '). This prevents a spurious period
+        from being appended to sentences like:
+            'He said "Hello."'  ->  unchanged (not 'He said "Hello.".')
+
+        Related GitHub Issue:
+            #7 - Spurious trailing period appended after sentence-final
+                 closing quote
+            https://github.com/craigtrim/fast-sentence-segment/issues/7
+
+        Args:
+            a_sentence: A sentence that may or may not have terminal
+                punctuation.
+
+        Returns:
+            The sentence with a period appended if it lacked terminal
+            punctuation, otherwise unchanged.
         """
-        Purpose:
-            if the sentence is not terminated with a period, then add one
-        :return:
-            a sentence terminated by a period
-        """
-        __blacklist = [':', '?', '!']
-        if not a_sentence.strip().endswith('.'):
-            for ch in __blacklist:
-                if not a_sentence.endswith(ch):
-                    return f"{a_sentence}."
-        return a_sentence
+        # Strip trailing quotes to inspect the actual punctuation
+        stripped = a_sentence.strip().rstrip('"\'')
+        if stripped and stripped[-1] in '.?!:':
+            return a_sentence
+        return f"{a_sentence}."
 
     @staticmethod
     def _is_valid_sentence(a_sentence: str) -> bool:

fast_sentence_segment-1.3.0/fast_sentence_segment/dmo/strip_trailing_period_after_quote.py

@@ -0,0 +1,70 @@
+# -*- coding: UTF-8 -*-
+"""Strip spurious trailing periods appended after sentence-final closing quotes.
+
+The spaCy segmenter's _append_period method can produce sentences like:
+    'He said "Hello.".'   (spurious trailing period)
+    'She asked "Why?".'   (spurious trailing period)
+    'He yelled "Stop!".'  (spurious trailing period)
+
+This post-processor removes the trailing period when the sentence ends
+with a closing double quote preceded by terminal punctuation.
+
+Related GitHub Issue:
+    #7 - Spurious trailing period appended after sentence-final
+         closing quote
+    https://github.com/craigtrim/fast-sentence-segment/issues/7
+"""
+
+import re
+
+from fast_sentence_segment.core import BaseObject
+
+# Matches a sentence that ends with terminal punctuation (. ? !)
+# followed by a closing double quote, followed by a spurious period.
+# The fix strips the final period.
+_SPURIOUS_PERIOD_PATTERN = re.compile(r'([.?!]")\.$')
+
+
+class StripTrailingPeriodAfterQuote(BaseObject):
+    """Strip spurious trailing periods after sentence-final closing quotes.
+
+    Detects sentences ending with patterns like:
+        ."."  ->  ."
+        ?"."  ->  ?"
+        !"."  ->  !"
+
+    Applied as a post-processing step in the sentence segmentation
+    pipeline, after spaCy segmentation and after the existing
+    PostProcessStructure step.
+
+    Related GitHub Issue:
+        #7 - Spurious trailing period appended after sentence-final
+             closing quote
+        https://github.com/craigtrim/fast-sentence-segment/issues/7
+    """
+
+    def __init__(self):
+        """
+        Created:
+            29-Jan-2026
+        """
+        BaseObject.__init__(self, __name__)
+
+    def process(self, sentences: list) -> list:
+        """Remove spurious trailing periods after closing quotes.
+
+        Args:
+            sentences: List of segmented sentences.
+
+        Returns:
+            List of sentences with spurious trailing periods removed.
+
+        Example:
+            >>> proc = StripTrailingPeriodAfterQuote()
+            >>> proc.process(['He said "Hello.".', 'She waved.'])
+            ['He said "Hello."', 'She waved.']
+        """
+        return [
+            _SPURIOUS_PERIOD_PATTERN.sub(r'\1', sentence)
+            for sentence in sentences
+        ]

fast_sentence_segment-1.3.0/fast_sentence_segment/dmo/unwrap_hard_wrapped_text.py

@@ -0,0 +1,34 @@
+# -*- coding: UTF-8 -*-
+"""Unwrap hard-wrapped text (e.g., Project Gutenberg e-texts).
+
+Joins lines within paragraphs into continuous strings while
+preserving paragraph boundaries (blank lines).
+"""
+
+import re
+
+
+def unwrap_hard_wrapped_text(text: str) -> str:
+    """Unwrap hard-wrapped paragraphs into continuous lines.
+
+    Splits on blank lines to identify paragraphs, then joins
+    lines within each paragraph into a single string with
+    single spaces.
+
+    Args:
+        text: Raw text with hard-wrapped lines.
+
+    Returns:
+        Text with paragraphs unwrapped into continuous strings,
+        separated by double newlines.
+    """
+    blocks = re.split(r'\n\s*\n', text)
+    unwrapped = []
+
+    for block in blocks:
+        lines = block.splitlines()
+        joined = ' '.join(line.strip() for line in lines if line.strip())
+        if joined:
+            unwrapped.append(joined)
+
+    return '\n\n'.join(unwrapped)

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/fast_sentence_segment/svc/perform_sentence_segmentation.py

@@ -17,6 +17,7 @@ from fast_sentence_segment.dmo import NumberedListNormalizer
 from fast_sentence_segment.dmo import QuestionExclamationSplitter
 from fast_sentence_segment.dmo import SpacyDocSegmenter
 from fast_sentence_segment.dmo import PostProcessStructure
+from fast_sentence_segment.dmo import StripTrailingPeriodAfterQuote
 
 
 class PerformSentenceSegmentation(BaseObject):
@@ -55,6 +56,7 @@ class PerformSentenceSegmentation(BaseObject):
         self._question_exclamation_splitter = QuestionExclamationSplitter().process
         self._title_name_merger = TitleNameMerger().process
         self._post_process = PostProcessStructure().process
+        self._strip_trailing_period = StripTrailingPeriodAfterQuote().process
 
     def _denormalize(self, text: str) -> str:
         """ Restore normalized placeholders to original form """
@@ -129,6 +131,9 @@ class PerformSentenceSegmentation(BaseObject):
 
         sentences = self._post_process(sentences)
 
+        # Strip spurious trailing periods after closing quotes (issue #7)
+        sentences = self._strip_trailing_period(sentences)
+
         sentences = [
             self._normalize_numbered_lists(x, denormalize=True)
             for x in sentences

{fast_sentence_segment-1.2.1 → fast_sentence_segment-1.3.0}/pyproject.toml

@@ -11,14 +11,26 @@ description = "Fast and Efficient Sentence Segmentation"
 license = "MIT"
 name = "fast-sentence-segment"
 readme = "README.md"
-version = "1.2.1"
+version = "1.3.0"
 
 keywords = ["nlp", "text", "preprocess", "segment"]
 repository = "https://github.com/craigtrim/fast-sentence-segment"
 
 classifiers = [
-  "Development Status :: 4 - Beta",
+  "Development Status :: 5 - Production/Stable",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
   "Topic :: Software Development :: Libraries :: Python Modules",
+  "Topic :: Text Processing :: Linguistic",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Operating System :: OS Independent",
+  "Typing :: Typed",
 ]
 
 [tool.poetry.urls]
@@ -36,6 +48,7 @@ ruff = "*"
 
 [tool.poetry.scripts]
 segment = "fast_sentence_segment.cli:main"
+segment-file = "fast_sentence_segment.cli:file_main"
 
 [tool.poetry.build]
 generate-setup-file = true

fast_sentence_segment-1.3.0/setup.py

@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+from setuptools import setup
+
+packages = \
+['fast_sentence_segment',
+ 'fast_sentence_segment.bp',
+ 'fast_sentence_segment.core',
+ 'fast_sentence_segment.dmo',
+ 'fast_sentence_segment.svc']
+
+package_data = \
+{'': ['*']}
+
+install_requires = \
+['spacy>=3.8.0,<4.0.0']
+
+entry_points = \
+{'console_scripts': ['segment = fast_sentence_segment.cli:main',
+                     'segment-file = fast_sentence_segment.cli:file_main']}
+
+setup_kwargs = {
+    'name': 'fast-sentence-segment',
+    'version': '1.3.0',
+    'description': 'Fast and Efficient Sentence Segmentation',
+    'long_description': '# Fast Sentence Segmentation\n\n[![PyPI version](https://img.shields.io/pypi/v/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)\n[![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)\n[![Downloads](https://static.pepy.tech/badge/fast-sentence-segment)](https://pepy.tech/project/fast-sentence-segment)\n[![Downloads/Month](https://static.pepy.tech/badge/fast-sentence-segment/month)](https://pepy.tech/project/fast-sentence-segment)\n\nFast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.\n\n## Why This Library?\n\n1. **Keep it local**: LLM API calls cost money and send your data to third parties. Run sentence segmentation entirely on your machine.\n2. **spaCy perfected**: spaCy is a great local model, but it makes mistakes. This library fixes most of spaCy\'s shortcomings.\n\n## Features\n\n- **Paragraph-aware segmentation**: Returns sentences grouped by paragraph\n- **Abbreviation handling**: Correctly handles "Dr.", "Mr.", "etc.", "p.m.", "a.m." without false splits\n- **Ellipsis preservation**: Keeps `...` intact while detecting sentence boundaries\n- **Question/exclamation splitting**: Properly splits on `?` and `!` followed by capital letters\n- **Cached processing**: LRU cache for repeated text processing\n- **Flexible output**: Nested lists (by paragraph) or flattened list of sentences\n- **Bullet point & numbered list normalization**: Cleans common list formats\n- **CLI tool**: Command-line interface for quick segmentation\n\n## Installation\n\n```bash\npip install fast-sentence-segment\n```\n\nAfter installation, download the spaCy model:\n\n```bash\npython -m spacy download en_core_web_sm\n```\n\n## Quick Start\n\n```python\nfrom fast_sentence_segment import segment_text\n\ntext = "Do you like Dr. Who? I prefer Dr. Strange! Mr. T is also cool."\n\nresults = segment_text(text, flatten=True)\n```\n\n```json\n[\n  "Do you like Dr. Who?",\n  "I prefer Dr. Strange!",\n  "Mr. T is also cool."\n]\n```\n\nNotice how "Dr. Who?" stays together as a single sentence—the library correctly recognizes that a title followed by a single-word name ending in `?` or `!` is a name reference, not a sentence boundary.\n\n## Usage\n\n### Basic Segmentation\n\nThe `segment_text` function returns a list of lists, where each inner list represents a paragraph containing its sentences:\n\n```python\nfrom fast_sentence_segment import segment_text\n\ntext = """Gandalf spoke softly. "All we have to decide is what to do with the time given us."\n\nFrodo nodded. The weight of the Ring pressed against his chest."""\n\nresults = segment_text(text)\n```\n\n```json\n[\n  [\n    "Gandalf spoke softly.",\n    "\\"All we have to decide is what to do with the time given us.\\"."\n  ],\n  [\n    "Frodo nodded.",\n    "The weight of the Ring pressed against his chest."\n  ]\n]\n```\n\n### Flattened Output\n\nIf you don\'t need paragraph boundaries, use the `flatten` parameter:\n\n```python\ntext = "At 9 a.m. the hobbits set out. By 3 p.m. they reached Rivendell. Mr. Frodo was exhausted."\n\nresults = segment_text(text, flatten=True)\n```\n\n```json\n[\n  "At 9 a.m. the hobbits set out.",\n  "By 3 p.m. they reached Rivendell.",\n  "Mr. Frodo was exhausted."\n]\n```\n\n### Direct Segmenter Access\n\nFor more control, use the `Segmenter` class directly:\n\n```python\nfrom fast_sentence_segment import Segmenter\n\nsegmenter = Segmenter()\nresults = segmenter.input_text("Your text here.")\n```\n\n### Command Line Interface\n\n```bash\n# Inline text\nsegment "Gandalf paused... You shall not pass! The Balrog roared."\n\n# Pipe from stdin\necho "Have you seen Dr. Who? It\'s brilliant!" | segment\n\n# Numbered output\nsegment -n -f silmarillion.txt\n\n# File-to-file (one sentence per line)\nsegment-file --input-file book.txt --output-file sentences.txt\n\n# Unwrap hard-wrapped e-texts (Project Gutenberg, etc.)\nsegment-file --input-file book.txt --output-file sentences.txt --unwrap\n```\n\n## API Reference\n\n| Function | Parameters | Returns | Description |\n|----------|------------|---------|-------------|\n| `segment_text()` | `input_text: str`, `flatten: bool = False`, `unwrap: bool = False` | `list` | Main entry point for segmentation |\n| `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |\n\n### CLI Commands\n\n| Command | Description |\n|---------|-------------|\n| `segment [text]` | Segment text from argument, `-f FILE`, or stdin. Use `-n` for numbered output. |\n| `segment-file --input-file IN --output-file OUT [--unwrap]` | Segment a file and write one sentence per line. Use `--unwrap` for hard-wrapped e-texts. |\n\n## Why Nested Lists?\n\nThe segmentation process preserves document structure by segmenting into both paragraphs and sentences. Each outer list represents a paragraph, and each inner list contains that paragraph\'s sentences. This is useful for:\n\n- Document structure analysis\n- Paragraph-level processing\n- Maintaining original text organization\n\nUse `flatten=True` when you only need sentences without paragraph context.\n\n## Requirements\n\n- Python 3.9+\n- spaCy 3.8+\n- en_core_web_sm spaCy model\n\n## How It Works\n\nThis library uses spaCy for initial sentence segmentation, then applies surgical post-processing fixes for cases where spaCy\'s default behavior is incorrect:\n\n1. **Pre-processing**: Normalize numbered lists, preserve ellipses with placeholders\n2. **spaCy segmentation**: Use spaCy\'s sentence boundary detection\n3. **Post-processing**: Split on abbreviation boundaries, handle `?`/`!` + capital patterns\n4. **Denormalization**: Restore placeholders to original text\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Run tests (`make test`)\n4. Commit your changes\n5. Push to the branch\n6. Open a Pull Request\n',
+    'author': 'Craig Trim',
+    'author_email': 'craigtrim@gmail.com',
+    'maintainer': 'Craig Trim',
+    'maintainer_email': 'craigtrim@gmail.com',
+    'url': 'https://github.com/craigtrim/fast-sentence-segment',
+    'packages': packages,
+    'package_data': package_data,
+    'install_requires': install_requires,
+    'entry_points': entry_points,
+    'python_requires': '>=3.9,<4.0',
+}
+
+
+setup(**setup_kwargs)

fast_sentence_segment-1.2.1/fast_sentence_segment/__init__.py

@@ -1,18 +0,0 @@
-from .bp import *
-from .svc import *
-from .dmo import *
-
-from .bp.segmenter import Segmenter
-
-segment = Segmenter().input_text
-
-
-def segment_text(input_text: str, flatten: bool = False) -> list:
-    results = segment(input_text)
-
-    if flatten:
-        flat = []
-        [[flat.append(y) for y in x] for x in results]
-        return flat
-
-    return results

fast_sentence_segment-1.2.1/fast_sentence_segment/cli.py

@@ -1,56 +0,0 @@
-# -*- coding: UTF-8 -*-
-"""CLI for fast-sentence-segment."""
-
-import argparse
-import logging
-import sys
-
-from fast_sentence_segment import segment_text
-
-logging.disable(logging.CRITICAL)
-
-
-def main():
-    parser = argparse.ArgumentParser(
-        prog="segment",
-        description="Segment text into sentences",
-    )
-    parser.add_argument(
-        "text",
-        nargs="?",
-        help="Text to segment (or use stdin)",
-    )
-    parser.add_argument(
-        "-f", "--file",
-        help="Read text from file",
-    )
-    parser.add_argument(
-        "-n", "--numbered",
-        action="store_true",
-        help="Number output lines",
-    )
-    args = parser.parse_args()
-
-    # Get input text
-    if args.file:
-        with open(args.file, "r", encoding="utf-8") as f:
-            text = f.read()
-    elif args.text:
-        text = args.text
-    elif not sys.stdin.isatty():
-        text = sys.stdin.read()
-    else:
-        parser.print_help()
-        sys.exit(1)
-
-    # Segment and output
-    sentences = segment_text(text.strip(), flatten=True)
-    for i, sentence in enumerate(sentences, 1):
-        if args.numbered:
-            print(f"{i}. {sentence}")
-        else:
-            print(sentence)
-
-
-if __name__ == "__main__":
-    main()

fast_sentence_segment-1.2.1/setup.py

@@ -1,38 +0,0 @@
-# -*- coding: utf-8 -*-
-from setuptools import setup
-
-packages = \
-['fast_sentence_segment',
- 'fast_sentence_segment.bp',
- 'fast_sentence_segment.core',
- 'fast_sentence_segment.dmo',
- 'fast_sentence_segment.svc']
-
-package_data = \
-{'': ['*']}
-
-install_requires = \
-['spacy>=3.8.0,<4.0.0']
-
-entry_points = \
-{'console_scripts': ['segment = fast_sentence_segment.cli:main']}
-
-setup_kwargs = {
-    'name': 'fast-sentence-segment',
-    'version': '1.2.1',
-    'description': 'Fast and Efficient Sentence Segmentation',
-    'long_description': '# Fast Sentence Segmentation\n\n[![PyPI version](https://img.shields.io/pypi/v/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)\n[![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)\n\nFast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.\n\n## Why This Library?\n\n1. **Keep it local**: LLM API calls cost money and send your data to third parties. Run sentence segmentation entirely on your machine.\n2. **spaCy perfected**: spaCy is a great local model, but it makes mistakes. This library fixes most of spaCy\'s shortcomings.\n\n## Features\n\n- **Paragraph-aware segmentation**: Returns sentences grouped by paragraph\n- **Abbreviation handling**: Correctly handles "Dr.", "Mr.", "etc.", "p.m.", "a.m." without false splits\n- **Ellipsis preservation**: Keeps `...` intact while detecting sentence boundaries\n- **Question/exclamation splitting**: Properly splits on `?` and `!` followed by capital letters\n- **Cached processing**: LRU cache for repeated text processing\n- **Flexible output**: Nested lists (by paragraph) or flattened list of sentences\n- **Bullet point & numbered list normalization**: Cleans common list formats\n- **CLI tool**: Command-line interface for quick segmentation\n\n## Installation\n\n```bash\npip install fast-sentence-segment\n```\n\nAfter installation, download the spaCy model:\n\n```bash\npython -m spacy download en_core_web_sm\n```\n\n## Quick Start\n\n```python\nfrom fast_sentence_segment import segment_text\n\ntext = "Do you like Dr. Who? I prefer Dr. Strange! Mr. T is also cool."\n\nresults = segment_text(text, flatten=True)\n```\n\n```json\n[\n  "Do you like Dr. Who?",\n  "I prefer Dr. Strange!",\n  "Mr. T is also cool."\n]\n```\n\nNotice how "Dr. Who?" stays together as a single sentence—the library correctly recognizes that a title followed by a single-word name ending in `?` or `!` is a name reference, not a sentence boundary.\n\n## Usage\n\n### Basic Segmentation\n\nThe `segment_text` function returns a list of lists, where each inner list represents a paragraph containing its sentences:\n\n```python\nfrom fast_sentence_segment import segment_text\n\ntext = """Gandalf spoke softly. "All we have to decide is what to do with the time given us."\n\nFrodo nodded. The weight of the Ring pressed against his chest."""\n\nresults = segment_text(text)\n```\n\n```json\n[\n  [\n    "Gandalf spoke softly.",\n    "\\"All we have to decide is what to do with the time given us.\\"."\n  ],\n  [\n    "Frodo nodded.",\n    "The weight of the Ring pressed against his chest."\n  ]\n]\n```\n\n### Flattened Output\n\nIf you don\'t need paragraph boundaries, use the `flatten` parameter:\n\n```python\ntext = "At 9 a.m. the hobbits set out. By 3 p.m. they reached Rivendell. Mr. Frodo was exhausted."\n\nresults = segment_text(text, flatten=True)\n```\n\n```json\n[\n  "At 9 a.m. the hobbits set out.",\n  "By 3 p.m. they reached Rivendell.",\n  "Mr. Frodo was exhausted."\n]\n```\n\n### Direct Segmenter Access\n\nFor more control, use the `Segmenter` class directly:\n\n```python\nfrom fast_sentence_segment import Segmenter\n\nsegmenter = Segmenter()\nresults = segmenter.input_text("Your text here.")\n```\n\n### Command Line Interface\n\nSegment text directly from the terminal:\n\n```bash\n# Direct text input\necho "Have you seen Dr. Who? It\'s brilliant!" | segment\n```\n\n```\nHave you seen Dr. Who?\nIt\'s brilliant!\n```\n\n```bash\n# Numbered output\nsegment -n "Gandalf paused... You shall not pass! The Balrog roared."\n```\n\n```\n1. Gandalf paused...\n2. You shall not pass!\n3. The Balrog roared.\n```\n\n```bash\n# From file\nsegment -f silmarillion.txt\n```\n\n## API Reference\n\n| Function | Parameters | Returns | Description |\n|----------|------------|---------|-------------|\n| `segment_text()` | `input_text: str`, `flatten: bool = False` | `list` | Main entry point for segmentation |\n| `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |\n\n### CLI Options\n\n| Option | Description |\n|--------|-------------|\n| `text` | Text to segment (positional argument) |\n| `-f, --file` | Read text from file |\n| `-n, --numbered` | Number output lines |\n\n## Why Nested Lists?\n\nThe segmentation process preserves document structure by segmenting into both paragraphs and sentences. Each outer list represents a paragraph, and each inner list contains that paragraph\'s sentences. This is useful for:\n\n- Document structure analysis\n- Paragraph-level processing\n- Maintaining original text organization\n\nUse `flatten=True` when you only need sentences without paragraph context.\n\n## Requirements\n\n- Python 3.9+\n- spaCy 3.8+\n- en_core_web_sm spaCy model\n\n## How It Works\n\nThis library uses spaCy for initial sentence segmentation, then applies surgical post-processing fixes for cases where spaCy\'s default behavior is incorrect:\n\n1. **Pre-processing**: Normalize numbered lists, preserve ellipses with placeholders\n2. **spaCy segmentation**: Use spaCy\'s sentence boundary detection\n3. **Post-processing**: Split on abbreviation boundaries, handle `?`/`!` + capital patterns\n4. **Denormalization**: Restore placeholders to original text\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Run tests (`make test`)\n4. Commit your changes\n5. Push to the branch\n6. Open a Pull Request\n',
-    'author': 'Craig Trim',
-    'author_email': 'craigtrim@gmail.com',
-    'maintainer': 'Craig Trim',
-    'maintainer_email': 'craigtrim@gmail.com',
-    'url': 'https://github.com/craigtrim/fast-sentence-segment',
-    'packages': packages,
-    'package_data': package_data,
-    'install_requires': install_requires,
-    'entry_points': entry_points,
-    'python_requires': '>=3.9,<4.0',
-}
-
-
-setup(**setup_kwargs)