pystylometry 1.3.0__tar.gz → 1.3.5__tar.gz

pystylometry-1.3.5/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Craig Trim
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pystylometry-1.3.5/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.1
+Name: pystylometry
+Version: 1.3.5
+Summary: Comprehensive Python package for stylometric analysis
+License: MIT
+Keywords: stylometry,nlp,text-analysis,authorship,readability,lexical-diversity,readability-metrics
+Author: Craig Trim
+Author-email: craigtrim@gmail.com
+Requires-Python: >=3.9,<4.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Project-URL: Homepage, https://github.com/craigtrim/pystylometry
+Project-URL: Issues, https://github.com/craigtrim/pystylometry/issues
+Project-URL: Repository, https://github.com/craigtrim/pystylometry
+Description-Content-Type: text/markdown
+
+# pystylometry
+
+[![PyPI version](https://badge.fury.io/py/pystylometry.svg)](https://badge.fury.io/py/pystylometry)
+[![Downloads](https://static.pepy.tech/badge/pystylometry)](https://pepy.tech/project/pystylometry)
+[![Downloads/Month](https://static.pepy.tech/badge/pystylometry/month)](https://pepy.tech/project/pystylometry)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+[![Tests](https://img.shields.io/badge/tests-1022%20passed-brightgreen)]()
+
+Stylometric analysis and authorship attribution for Python. 50+ metrics across 11 modules, from vocabulary diversity to AI-generation detection.
+
+## Install
+
+```bash
+pip install pystylometry              # Core (lexical metrics)
+pip install pystylometry[all]         # Everything
+```
+
+## Modules
+
+| Module | Metrics | Description |
+|--------|---------|-------------|
+| [**lexical**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/lexical) | TTR, MTLD, Yule's K/I, Hapax, MATTR, VocD-D, HD-D, MSTTR, function words, word frequency | Vocabulary diversity and richness |
+| [**readability**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/readability) | Flesch, Flesch-Kincaid, SMOG, Gunning Fog, Coleman-Liau, ARI, Dale-Chall, Fry, FORCAST, Linsear Write, Powers-Sumner-Kearl | Grade-level and difficulty scoring |
+| [**syntactic**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/syntactic) | POS ratios, sentence types, parse tree depth, clausal density, passive voice, T-units, dependency distance | Sentence and parse structure (requires spaCy) |
+| [**authorship**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/authorship) | Burrows' Delta, Cosine Delta, Zeta, Kilgarriff chi-squared, MinMax, John's Delta, NCD | Author attribution and text comparison |
+| [**stylistic**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/stylistic) | Contractions, hedges, intensifiers, modals, punctuation, vocabulary overlap (Jaccard/Dice/Cosine/KL), cohesion, genre/register | Style markers and text similarity |
+| [**character**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/character) | Letter frequencies, digit/uppercase ratios, special characters, whitespace | Character-level fingerprinting |
+| [**ngrams**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/ngrams) | Word/character/POS n-grams, Shannon entropy, skipgrams | N-gram profiles and entropy |
+| [**dialect**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/dialect) | British/American classification, spelling/grammar/vocabulary markers, markedness | Regional dialect detection |
+| [**consistency**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/consistency) | Sliding-window chi-squared drift, pattern classification | Intra-document style analysis |
+| [**prosody**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/prosody) | Syllable stress, rhythm regularity | Prose rhythm (requires spaCy) |
+| [**viz**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/viz) | Timeline, scatter, report (PNG + interactive HTML) | Drift detection visualization |
+
+## Development
+
+```bash
+git clone https://github.com/craigtrim/pystylometry && cd pystylometry
+pip install -e ".[dev,all]"
+make test       # 1022 tests
+make lint       # ruff + mypy
+make all        # lint + test + build
+```
+
+## License
+
+MIT
+
+## Author
+
+Craig Trim -- craigtrim@gmail.com
+

pystylometry-1.3.5/README.md

@@ -0,0 +1,51 @@
+# pystylometry
+
+[![PyPI version](https://badge.fury.io/py/pystylometry.svg)](https://badge.fury.io/py/pystylometry)
+[![Downloads](https://static.pepy.tech/badge/pystylometry)](https://pepy.tech/project/pystylometry)
+[![Downloads/Month](https://static.pepy.tech/badge/pystylometry/month)](https://pepy.tech/project/pystylometry)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+[![Tests](https://img.shields.io/badge/tests-1022%20passed-brightgreen)]()
+
+Stylometric analysis and authorship attribution for Python. 50+ metrics across 11 modules, from vocabulary diversity to AI-generation detection.
+
+## Install
+
+```bash
+pip install pystylometry              # Core (lexical metrics)
+pip install pystylometry[all]         # Everything
+```
+
+## Modules
+
+| Module | Metrics | Description |
+|--------|---------|-------------|
+| [**lexical**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/lexical) | TTR, MTLD, Yule's K/I, Hapax, MATTR, VocD-D, HD-D, MSTTR, function words, word frequency | Vocabulary diversity and richness |
+| [**readability**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/readability) | Flesch, Flesch-Kincaid, SMOG, Gunning Fog, Coleman-Liau, ARI, Dale-Chall, Fry, FORCAST, Linsear Write, Powers-Sumner-Kearl | Grade-level and difficulty scoring |
+| [**syntactic**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/syntactic) | POS ratios, sentence types, parse tree depth, clausal density, passive voice, T-units, dependency distance | Sentence and parse structure (requires spaCy) |
+| [**authorship**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/authorship) | Burrows' Delta, Cosine Delta, Zeta, Kilgarriff chi-squared, MinMax, John's Delta, NCD | Author attribution and text comparison |
+| [**stylistic**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/stylistic) | Contractions, hedges, intensifiers, modals, punctuation, vocabulary overlap (Jaccard/Dice/Cosine/KL), cohesion, genre/register | Style markers and text similarity |
+| [**character**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/character) | Letter frequencies, digit/uppercase ratios, special characters, whitespace | Character-level fingerprinting |
+| [**ngrams**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/ngrams) | Word/character/POS n-grams, Shannon entropy, skipgrams | N-gram profiles and entropy |
+| [**dialect**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/dialect) | British/American classification, spelling/grammar/vocabulary markers, markedness | Regional dialect detection |
+| [**consistency**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/consistency) | Sliding-window chi-squared drift, pattern classification | Intra-document style analysis |
+| [**prosody**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/prosody) | Syllable stress, rhythm regularity | Prose rhythm (requires spaCy) |
+| [**viz**](https://github.com/craigtrim/pystylometry/tree/master/pystylometry/viz) | Timeline, scatter, report (PNG + interactive HTML) | Drift detection visualization |
+
+## Development
+
+```bash
+git clone https://github.com/craigtrim/pystylometry && cd pystylometry
+pip install -e ".[dev,all]"
+make test       # 1022 tests
+make lint       # ruff + mypy
+make all        # lint + test + build
+```
+
+## License
+
+MIT
+
+## Author
+
+Craig Trim -- craigtrim@gmail.com

{pystylometry-1.3.0 → pystylometry-1.3.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pystylometry"
-version = "1.3.0"
+version = "1.3.5"
 description = "Comprehensive Python package for stylometric analysis"
 authors = ["Craig Trim <craigtrim@gmail.com>"]
 readme = "README.md"
@@ -16,7 +16,7 @@ keywords = [
     "readability-metrics",
 ]
 classifiers = [
-    "Development Status :: 4 - Beta",
+    "Development Status :: 5 - Production/Stable",
     "Intended Audience :: Science/Research",
     "Intended Audience :: Developers",
     "License :: OSI Approved :: MIT License",
@@ -35,6 +35,8 @@ classifiers = [
 pystylometry-drift = "pystylometry.cli:drift_cli"
 # pystylometry-viewer drift_analyzer.html
 pystylometry-viewer = "pystylometry.cli:viewer_cli"
+# pystylometry-tokenize manuscript.txt --json --metadata
+pystylometry-tokenize = "pystylometry.cli:tokenize_cli"
 
 [tool.poetry.urls]
 Homepage = "https://github.com/craigtrim/pystylometry"
@@ -43,7 +45,6 @@ Issues = "https://github.com/craigtrim/pystylometry/issues"
 
 [tool.poetry.dependencies]
 python = "^3.9"
-stylometry-ttr = "^1.0.3"
 
 [tool.poetry.group.readability.dependencies]
 pronouncing = "^0.2.0"
@@ -61,7 +62,7 @@ spacy = "^3.8.0"
 # - True neologisms (not in WordNet, not in BNC)
 # - Rare words (in BNC but not WordNet, or vice versa)
 # - Common words (in both lexicons)
-bnc-lookup = "*"
+bnc-lookup = ">=1.3.0"
 wordnet-lookup = "*"
 
 [tool.poetry.group.viz.dependencies]
@@ -98,6 +99,7 @@ ignore = []
 "pystylometry/viz/jsx/*.py" = ["E501"]
 # Test files may have long assertions and test data
 "tests/*.py" = ["E501"]
+"tests/**/*.py" = ["E501", "E402"]
 
 [tool.mypy]
 python_version = "3.9"

{pystylometry-1.3.0 → pystylometry-1.3.5}/pystylometry/__init__.py

@@ -40,14 +40,13 @@ Usage:
     print(result.pattern_confidence)
 """
 
+from . import lexical  # noqa: E402
 from ._types import AnalysisResult
+from .tokenizer import TokenizationStats, Tokenizer, TokenMetadata
 
 # Version
 __version__ = "0.1.0"
 
-# Core exports - always available
-from . import lexical
-
 # Optional exports - may raise ImportError if dependencies not installed
 try:
     from . import readability  # noqa: F401
@@ -87,6 +86,41 @@ _CONSISTENCY_AVAILABLE = True
 _STYLISTIC_AVAILABLE = True
 
 
+def tokenize(text: str, **kwargs: object) -> list[str]:
+    """Tokenize text using the stylometric tokenizer.
+
+    Convenience wrapper around Tokenizer.tokenize(). All keyword arguments
+    are forwarded to the Tokenizer constructor.
+
+    Args:
+        text: Input text to tokenize.
+        **kwargs: Options forwarded to Tokenizer (lowercase, strip_numbers,
+            expand_contractions, etc.).
+
+    Returns:
+        List of token strings.
+
+    Example:
+        >>> from pystylometry import tokenize
+        >>> tokenize("Hello, world! It's a test.")
+        ['hello', 'world', "it's", 'a', 'test']
+    """
+    return Tokenizer(**kwargs).tokenize(text)  # type: ignore[arg-type]
+
+
+def tokenize_with_metadata(text: str, **kwargs: object) -> list[TokenMetadata]:
+    """Tokenize text and return tokens with positional and type metadata.
+
+    Args:
+        text: Input text to tokenize.
+        **kwargs: Options forwarded to Tokenizer.
+
+    Returns:
+        List of TokenMetadata objects.
+    """
+    return Tokenizer(**kwargs).tokenize_with_metadata(text)  # type: ignore[arg-type]
+
+
 def analyze(
     text: str,
     lexical_metrics: bool = True,
@@ -225,6 +259,11 @@ __all__ = [
     "__version__",
     "analyze",
     "get_available_modules",
+    "tokenize",
+    "tokenize_with_metadata",
+    "Tokenizer",
+    "TokenMetadata",
+    "TokenizationStats",
     "lexical",
 ]
 

{pystylometry-1.3.0 → pystylometry-1.3.5}/pystylometry/_types.py

@@ -23,7 +23,7 @@ from __future__ import annotations
 
 import statistics
 from dataclasses import dataclass
-from typing import Any
+from typing import Any, Optional
 
 # ===== Distribution and Chunking =====
 # Related to GitHub Issue #27: Native chunked analysis with Distribution dataclass
@@ -316,8 +316,8 @@ class HapaxLexiconResult:
 class TTRResult:
     """Result from Type-Token Ratio (TTR) analysis.
 
-    Wraps stylometry-ttr package functionality to measure vocabulary richness
-    through the ratio of unique words (types) to total words (tokens).
+    Measures vocabulary richness through the ratio of unique words (types)
+    to total words (tokens).
 
     All numeric metrics include both a mean value (convenient access) and
     a full distribution with per-chunk values and statistics.
@@ -370,6 +370,208 @@ class TTRResult:
     metadata: dict[str, Any]
 
 
+@dataclass
+class TTRAggregateResult:
+    """Aggregated TTR statistics for a collection of texts.
+
+    Computes group-level summary statistics (mean, std, min, max, median)
+    across multiple ``TTRResult`` objects.  Useful for comparative analysis
+    across authors, genres, or time periods.
+
+    Related GitHub Issue:
+        #43 - Inline stylometry-ttr into pystylometry (remove external dependency)
+        https://github.com/craigtrim/pystylometry/issues/43
+
+    Example:
+        >>> from pystylometry.lexical import compute_ttr, TTRAggregator
+        >>> results = [compute_ttr(t) for t in texts]
+        >>> agg = TTRAggregator()
+        >>> stats = agg.aggregate(results, group_id="Austen")
+        >>> stats.ttr_mean
+        0.412
+    """
+
+    group_id: str
+    text_count: int
+    total_words: int
+
+    # Raw TTR statistics
+    ttr_mean: float
+    ttr_std: float
+    ttr_min: float
+    ttr_max: float
+    ttr_median: float
+
+    # Root TTR (Guiraud's index) statistics
+    root_ttr_mean: float
+    root_ttr_std: float
+
+    # Log TTR (Herdan's C) statistics
+    log_ttr_mean: float
+    log_ttr_std: float
+
+    # STTR statistics (None if no texts had enough words for STTR)
+    sttr_mean: Optional[float]
+    sttr_std: Optional[float]
+
+    # Delta std mean (None if no texts had delta metrics)
+    delta_std_mean: Optional[float]
+
+    metadata: dict[str, Any]
+
+
+# ===== Repetition Detection Results =====
+# Related to GitHub Issue #28: Verbal tics detection for slop analysis
+# https://github.com/craigtrim/pystylometry/issues/28
+
+
+@dataclass
+class RepetitiveWord:
+    """A single word flagged as abnormally repetitive.
+
+    The repetition_score is the ratio of observed count to expected count
+    based on the word's frequency in the British National Corpus (BNC).
+    Higher scores indicate stronger overrepresentation.
+
+    Related GitHub Issue:
+        #28 - Verbal tics detection for slop analysis
+        https://github.com/craigtrim/pystylometry/issues/28
+
+    Attributes:
+        word: The flagged word (lowercased).
+        count: Observed count in the text.
+        expected_count: Expected count based on BNC relative frequency × text length.
+            0.0 if word not found in BNC.
+        repetition_score: count / expected_count. float('inf') if expected_count is 0.
+        bnc_bucket: BNC frequency bucket (1-100, 1=most frequent). None if not in BNC.
+        chunk_counts: Per-chunk occurrence counts (for distribution analysis).
+        distribution_entropy: Shannon entropy of the word's chunk distribution.
+            Low entropy = suspiciously even spread (model tic).
+            High entropy = clustered usage (human writing about a specific scene).
+        distribution_variance: Variance of per-chunk counts.
+    """
+
+    word: str
+    count: int
+    expected_count: float
+    repetition_score: float
+    bnc_bucket: int | None
+    chunk_counts: list[int]
+    distribution_entropy: float
+    distribution_variance: float
+
+
+@dataclass
+class RepetitiveUnigramsResult:
+    """Result from repetitive unigram detection.
+
+    Identifies content words that appear far more frequently than expected
+    based on their frequency in the British National Corpus (BNC, ~100M tokens).
+    This is a key indicator of AI-generated "slop" where models exhibit verbal
+    tics — repeating certain words with suspicious regularity.
+
+    Related GitHub Issue:
+        #28 - Verbal tics detection for slop analysis
+        https://github.com/craigtrim/pystylometry/issues/28
+
+    The slop_score provides a single aggregate metric:
+        slop_score = flagged_words_per_10k × mean_repetition_score
+
+    Where:
+        - flagged_words_per_10k = count of flagged words / (total content words / 10000)
+        - mean_repetition_score = mean repetition_score across all flagged words
+
+    Higher slop_score = more likely AI-generated verbal tics.
+
+    References:
+        British National Corpus Consortium. (2007). The British National Corpus,
+            version 3 (BNC XML Edition). http://www.natcorp.ox.ac.uk/
+
+    Example:
+        >>> result = compute_repetitive_unigrams(text)
+        >>> for w in result.repetitive_words[:5]:
+        ...     print(f"{w.word}: {w.count}x (expected {w.expected_count:.1f}, "
+        ...           f"score {w.repetition_score:.1f})")
+        shimmered: 23x (expected 0.1, score 266.2)
+        >>> result.slop_score
+        42.7
+    """
+
+    repetitive_words: list[RepetitiveWord]  # Sorted by repetition_score descending
+    total_content_words: int
+    flagged_count: int  # Number of words exceeding threshold
+    flagged_words_per_10k: float  # flagged_count / (total_content_words / 10000)
+    mean_repetition_score: float  # Mean score across flagged words
+    slop_score: float  # Aggregate: flagged_words_per_10k × mean_repetition_score
+    total_content_words_dist: Distribution
+    chunk_size: int
+    chunk_count: int
+    metadata: dict[str, Any]
+
+
+@dataclass
+class RepetitiveNgram:
+    """A single n-gram flagged as abnormally repetitive.
+
+    Content n-grams (bigrams, trigrams, etc.) should rarely repeat verbatim
+    in natural writing. N-grams that repeat beyond a length-scaled threshold
+    are flagged.
+
+    Related GitHub Issue:
+        #28 - Verbal tics detection for slop analysis
+        https://github.com/craigtrim/pystylometry/issues/28
+
+    Attributes:
+        ngram: The flagged n-gram as a tuple of words.
+        count: Observed count in the text.
+        frequency_per_10k: Occurrences per 10,000 n-grams.
+        chunk_counts: Per-chunk occurrence counts.
+        distribution_entropy: Shannon entropy of the n-gram's chunk distribution.
+        distribution_variance: Variance of per-chunk counts.
+    """
+
+    ngram: tuple[str, ...]
+    count: int
+    frequency_per_10k: float
+    chunk_counts: list[int]
+    distribution_entropy: float
+    distribution_variance: float
+
+
+@dataclass
+class RepetitiveNgramsResult:
+    """Result from repetitive n-gram detection.
+
+    Detects bigrams, trigrams, or higher-order n-grams that repeat more than
+    expected within the text. No external corpus is required — content n-grams
+    should not repeat verbatim often in natural writing.
+
+    N-grams composed entirely of function words (e.g., "of the", "in a") are
+    excluded since their repetition is expected.
+
+    Related GitHub Issue:
+        #28 - Verbal tics detection for slop analysis
+        https://github.com/craigtrim/pystylometry/issues/28
+
+    Example:
+        >>> result = compute_repetitive_ngrams(text, n=2)
+        >>> for ng in result.repetitive_ngrams[:5]:
+        ...     print(f"{' '.join(ng.ngram)}: {ng.count}x "
+        ...           f"({ng.frequency_per_10k:.1f} per 10k)")
+        uncomfortable truth: 8x (1.6 per 10k)
+    """
+
+    repetitive_ngrams: list[RepetitiveNgram]  # Sorted by count descending
+    n: int | tuple[int, ...]  # N-gram order(s) analyzed
+    total_ngrams: int
+    flagged_count: int
+    flagged_per_10k: float  # flagged_count / (total_ngrams / 10000)
+    total_ngrams_dist: Distribution
+    chunk_size: int
+    chunk_count: int
+    metadata: dict[str, Any]
+
+
 # ===== Readability Results =====