PyReprism 0.1.0__tar.gz → 0.2.0__tar.gz

{pyreprism-0.1.0 → pyreprism-0.2.0}/CHANGELOG.md

@@ -7,7 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-07-08
+
+This release adds a static code-analysis toolkit on top of the tokenizer:
+diff/PR analysis, complexity metrics, n-gram "naturalness", and clone/plagiarism
+similarity — plus a comprehensive per-language test suite that hardened comment
+handling across the board.
+
 ### Added
+- **Fingerprinting & similarity** (`PyReprism.fingerprints`): winnowing k-gram
+  fingerprints over the normalized token stream for clone/plagiarism detection —
+  `fingerprint()`, `similarity()`/`containment()` (rename-invariant by default),
+  and a `FingerprintIndex` for many-to-many detection over a corpus. CLI:
+  `pyreprism similarity a b` and `pyreprism clones DIR --threshold`.
+- **N-gram analysis & code naturalness** (`PyReprism.ngrams`): token/type n-gram
+  extraction and frequency counts, plus an `NgramModel` (add-k smoothing,
+  save/load) that measures cross-entropy / perplexity against a trained corpus
+  ("naturalness of software"). CLI: `pyreprism ngrams` and
+  `pyreprism perplexity --train`.
+- **Complexity metrics** computed from the token stream: `halstead()`
+  (volume/difficulty/effort/bugs), `cyclomatic_complexity()` (approximate McCabe),
+  `maintainability_index()` (0–100), `max_nesting_depth()`, and `code_metrics()`
+  which bundles them with the line/token stats. Exposed on the CLI via
+  `pyreprism stats --full`.
 - **Diff processing** (`PyReprism.diffs`): parse unified/`git` diffs and analyze
   the changed code per file in its own language. Includes churn metrics
   (`diff_stats`: added/removed split into code, comment and blank), cosmetic
@@ -17,6 +39,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `old_source` enables accurate full-file classification. New CLI command
   `pyreprism diff` (`--json` / `--csv` / `--per-file` / `--cosmetic`).
 
+### Changed
+- Adopted the standard `src/` layout with tests at the top level, consolidated
+  all dependencies into `pyproject.toml` extras, and rewrote the landing page.
+- Single-sourced the version from the package and automated releases: pushing a
+  `vX.Y.Z` tag now publishes to PyPI (Trusted Publishing) and creates a GitHub
+  Release. Added Python 3.13 to the CI matrix.
+- Added a comprehensive, data-driven per-language comment test suite that
+  requires every registered language to be covered.
+
+### Fixed
+- Fixed comment stripping in 8 more languages surfaced by the new test suite:
+  `smalltalk` (kept the comment and deleted the code) and
+  `eiffel`/`bro`/`coffeescript`/`io`/`nix`/`gherkin`/`gedcom` (trailing comments
+  not removed and newlines dropped).
+- Fixed a CLI argument-ordering incompatibility on Python ≤ 3.11 (options must
+  follow positionals).
+
 ## [0.1.0] - 2026-07-07
 
 This release turns PyReprism from a comment-removal helper into a full
@@ -73,6 +112,7 @@ ML-oriented normalization, an optional accurate backend, and batch processing.
 - Early beta releases: comment removal for an initial set of languages and the
   `Normalizer` whitespace helper.
 
-[Unreleased]: https://github.com/unlv-evol/PyReprism/compare/v0.1.0...HEAD
+[Unreleased]: https://github.com/unlv-evol/PyReprism/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/unlv-evol/PyReprism/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/unlv-evol/PyReprism/releases/tag/v0.1.0
 [0.0.4]: https://github.com/unlv-evol/PyReprism/releases/tag/v0.0.4

{pyreprism-0.1.0 → pyreprism-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PyReprism
-Version: 0.1.0
+Version: 0.2.0
 Summary: Framework for Source Code Preprocessing
 Project-URL: Homepage, https://github.com/unlv-evol/PyReprism
 Project-URL: Documentation, https://pyreprism.readthedocs.io
@@ -58,54 +58,101 @@ Description-Content-Type: text/x-rst
 
 .. _intro_toplevel:
 
-==================
-Overview / Install
-==================
+========
+Overview
+========
 
-PyReprism is a Python framework that helps researchers and developers the task of source code preprocessing. With PyReprism, you can easily match, extract, count, and remove comments, whitespaces, operators, numbers and other language specific constructs from over 150 programming languages and file extensions.
+**PyReprism** is a Python framework for source-code preprocessing. It lets you
+**match, extract, count, and remove** comments, strings, numbers, operators,
+keywords and other language-specific constructs across **145+ programming
+languages and file formats** — through a small high-level API, a command-line
+tool, code metrics, ML-oriented normalization, and diff/pull-request analysis.
 
+Highlights
+==========
+
+* One consistent API for every language: ``remove_*`` / ``extract_*`` /
+  ``count_*`` / ``match_*`` for each construct, plus a lossless ``tokenize()``.
+* Code metrics (``stats``) and ML canonicalization (``normalize``) for
+  clone/plagiarism detection and code-embedding pipelines.
+* Optional high-accuracy `Pygments`_ backend — with **zero required
+  dependencies** by default.
+* Batch/corpus analysis and unified/``git`` diff processing.
+* A ``pyreprism`` command-line tool (also ``python -m PyReprism``).
 
 Requirements
 ============
 
-* `Python`_ 3.6 or newer
-* `Git`_
+* `Python`_ 3.8 or newer
 
-.. _Python: https://www.python.org
-.. _Git: https://git-scm.com/
+Install
+=======
 
-Installing PyReprism
-====================
+.. code-block:: shell
 
-Installing PyReprism is easily done using `pip`_. Assuming it is installed, just run the following from the command-line:
+    pip install PyReprism
 
-.. _pip: https://pip.pypa.io/en/latest/installing.html
+For the higher-accuracy tokenizer backend:
 
-.. sourcecode:: none
+.. code-block:: shell
 
-    $ pip install PyReprism
+    pip install "PyReprism[accurate]"
 
-Source Code
+Quick start
 ===========
 
-PyReprism's git repo is available on GitHub, which can be browsed at:
+.. code-block:: python
+
+    import PyReprism as pr
+
+    source = """
+    # a comment
+    x = 5 + 6
+    print(x)  # inline
+    """
 
- * https://github.com/unlv-evol/PyReprism.git
+    pr.remove_comments(source, lang="python")   # code without comments
+    pr.extract_comments(source, lang="python")  # ['# a comment', '# inline']
+    pr.count_comments(source, lang="python")     # 2
 
-and cloned using::
+``lang`` accepts a language name (``"python"``), a file extension (``".py"``),
+or a language class. Canonicalize code for machine learning, or measure it:
+
+.. code-block:: python
+
+    pr.normalize("total = price * 42", lang="python")   # 'VAR1 = VAR2 * 0'
+    pr.stats(source, lang="python").comment_to_code_ratio
+
+Command line
+============
 
-    $ git clone https://github.com/unlv-evol/PyReprism.git
-    $ cd PyReprism
+.. code-block:: shell
 
-Optionally (but suggested), make use of virtual environment. Therefore, before installing the requirements run::
-    
-    $ python3 -m venv venv
-    $ source venv/bin/activate
+    pyreprism remove comments file.py
+    pyreprism scan myproject/ --csv          # aggregate code metrics over a tree
+    git diff | pyreprism diff --cosmetic     # flag comment/whitespace-only changes
 
-Install the requirements::
-    
-    $ pip install -r requirements.txt
+Documentation
+=============
 
-and run the tests using pytest::
+Full documentation, including the API reference and the list of supported
+languages, is available at https://pyreprism.readthedocs.io.
 
-    $ pytest
+Source code
+===========
+
+PyReprism is developed on GitHub at
+https://github.com/unlv-evol/PyReprism. To work on it locally:
+
+.. code-block:: shell
+
+    git clone https://github.com/unlv-evol/PyReprism.git
+    cd PyReprism
+    python3 -m venv venv && source venv/bin/activate
+    pip install -e ".[dev]"
+    pytest
+
+See ``CONTRIBUTING.md`` for contribution guidelines.
+
+.. _Python: https://www.python.org
+.. _Pygments: https://pygments.org

{pyreprism-0.1.0 → pyreprism-0.2.0}/README.md

@@ -87,6 +87,56 @@ s.comment_to_code_ratio, s.comment_density
 s.as_dict()   # includes per-token-type counts, ready for JSON / dataframes
 ```
 
+Complexity metrics (computed from the token stream):
+
+```python
+pr.halstead(source, lang="python").volume        # Halstead volume/difficulty/effort/bugs
+pr.cyclomatic_complexity(source, lang="python")  # approximate McCabe complexity
+pr.maintainability_index(source, lang="python")  # 0–100 (higher is better)
+pr.code_metrics(source, lang="python")           # everything above in one dict
+```
+
+On the CLI: `pyreprism stats --full file.py` (add `--json` for machine output).
+
+### Similarity & clone / plagiarism detection
+
+Winnowing k-gram fingerprints over the *normalized* token stream, so matches
+survive variable renaming and literal changes (Type-2 clones):
+
+```python
+from PyReprism import fingerprints as fp
+
+fp.similarity(code_a, code_b, "python")     # 0.0–1.0 (Jaccard of fingerprints)
+fp.containment(code_a, code_b, "python")    # how much of A appears in B
+
+index = fp.FingerprintIndex()               # many-to-many clone detection
+index.add_paths("submissions/")
+index.similar_pairs(threshold=0.7)          # -> [(file_a, file_b, score), ...]
+```
+
+On the CLI: `pyreprism similarity a.py b.py` and
+`pyreprism clones submissions/ --threshold 0.7`.
+
+### N-grams & code "naturalness"
+
+Token n-grams (over token text or, structurally, over token *types*) and an
+n-gram language model that measures how predictable/"natural" code is
+(Hindle et al.):
+
+```python
+from PyReprism import ngrams
+
+ngrams.ngram_counts(source, "python", n=3).most_common(10)
+ngrams.ngrams(source, "python", n=2, types=True)      # structural n-grams
+
+model = ngrams.train("corpus/", n=3)                  # train on a code corpus
+model.perplexity(ngrams.token_sequence(source, "python"))   # lower = more natural
+model.save("model.json")
+```
+
+On the CLI: `pyreprism ngrams file.py -n 3 --top 20` and
+`pyreprism perplexity --train corpus/ file.py`.
+
 ### Normalization for ML / clone detection
 
 Canonicalize code so that only its structure remains — rename identifiers to

pyreprism-0.2.0/docs/intro.rst

@@ -0,0 +1,100 @@
+.. _intro_toplevel:
+
+========
+Overview
+========
+
+**PyReprism** is a Python framework for source-code preprocessing. It lets you
+**match, extract, count, and remove** comments, strings, numbers, operators,
+keywords and other language-specific constructs across **145+ programming
+languages and file formats** — through a small high-level API, a command-line
+tool, code metrics, ML-oriented normalization, and diff/pull-request analysis.
+
+Highlights
+==========
+
+* One consistent API for every language: ``remove_*`` / ``extract_*`` /
+  ``count_*`` / ``match_*`` for each construct, plus a lossless ``tokenize()``.
+* Code metrics (``stats``) and ML canonicalization (``normalize``) for
+  clone/plagiarism detection and code-embedding pipelines.
+* Optional high-accuracy `Pygments`_ backend — with **zero required
+  dependencies** by default.
+* Batch/corpus analysis and unified/``git`` diff processing.
+* A ``pyreprism`` command-line tool (also ``python -m PyReprism``).
+
+Requirements
+============
+
+* `Python`_ 3.8 or newer
+
+Install
+=======
+
+.. code-block:: shell
+
+    pip install PyReprism
+
+For the higher-accuracy tokenizer backend:
+
+.. code-block:: shell
+
+    pip install "PyReprism[accurate]"
+
+Quick start
+===========
+
+.. code-block:: python
+
+    import PyReprism as pr
+
+    source = """
+    # a comment
+    x = 5 + 6
+    print(x)  # inline
+    """
+
+    pr.remove_comments(source, lang="python")   # code without comments
+    pr.extract_comments(source, lang="python")  # ['# a comment', '# inline']
+    pr.count_comments(source, lang="python")     # 2
+
+``lang`` accepts a language name (``"python"``), a file extension (``".py"``),
+or a language class. Canonicalize code for machine learning, or measure it:
+
+.. code-block:: python
+
+    pr.normalize("total = price * 42", lang="python")   # 'VAR1 = VAR2 * 0'
+    pr.stats(source, lang="python").comment_to_code_ratio
+
+Command line
+============
+
+.. code-block:: shell
+
+    pyreprism remove comments file.py
+    pyreprism scan myproject/ --csv          # aggregate code metrics over a tree
+    git diff | pyreprism diff --cosmetic     # flag comment/whitespace-only changes
+
+Documentation
+=============
+
+Full documentation, including the API reference and the list of supported
+languages, is available at https://pyreprism.readthedocs.io.
+
+Source code
+===========
+
+PyReprism is developed on GitHub at
+https://github.com/unlv-evol/PyReprism. To work on it locally:
+
+.. code-block:: shell
+
+    git clone https://github.com/unlv-evol/PyReprism.git
+    cd PyReprism
+    python3 -m venv venv && source venv/bin/activate
+    pip install -e ".[dev]"
+    pytest
+
+See ``CONTRIBUTING.md`` for contribution guidelines.
+
+.. _Python: https://www.python.org
+.. _Pygments: https://pygments.org

{pyreprism-0.1.0 → pyreprism-0.2.0}/src/PyReprism/__init__.py

@@ -17,11 +17,11 @@ import importlib
 import os
 from typing import List, Optional, Sequence, Type, Union
 
-from .metrics import CodeStats
+from .metrics import CodeStats, Halstead
 from .tokens import Token, TokenType
 from .utils.normalizer import Normalizer
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 LanguageLike = Union[str, "type"]
 
@@ -274,6 +274,46 @@ def stats(source: str, lang: LanguageLike, engine: str = 'regex') -> CodeStats:
     return _tokenops.stats(_engine_tokens(source, lang, engine), source)
 
 
+def halstead(source: str, lang: LanguageLike, engine: str = 'regex') -> Halstead:
+    """Return the :class:`Halstead` complexity measures for ``source``."""
+    if _is_regex(engine):
+        return _resolve(lang).halstead(source)
+    from . import _tokenops
+    return _tokenops.halstead(_engine_tokens(source, lang, engine))
+
+
+def cyclomatic_complexity(source: str, lang: LanguageLike, engine: str = 'regex') -> int:
+    """Approximate McCabe cyclomatic complexity (token-based)."""
+    if _is_regex(engine):
+        return _resolve(lang).cyclomatic_complexity(source)
+    from . import _tokenops
+    return _tokenops.cyclomatic(_engine_tokens(source, lang, engine))
+
+
+def maintainability_index(source: str, lang: LanguageLike, engine: str = 'regex') -> float:
+    """SEI-normalized Maintainability Index in ``[0, 100]`` (higher is better)."""
+    if _is_regex(engine):
+        return _resolve(lang).maintainability_index(source)
+    from . import _tokenops
+    tokens = _engine_tokens(source, lang, engine)
+    return _tokenops.maintainability_index(tokens, _tokenops.stats(tokens, source).code_lines)
+
+
+def code_metrics(source: str, lang: LanguageLike, engine: str = 'regex') -> dict:
+    """Return a combined metrics dict (line stats + Halstead + complexity + MI)."""
+    if _is_regex(engine):
+        return _resolve(lang).code_metrics(source)
+    from . import _tokenops
+    tokens = _engine_tokens(source, lang, engine)
+    stats_obj = _tokenops.stats(tokens, source)
+    data = stats_obj.as_dict()
+    data['halstead'] = _tokenops.halstead(tokens).as_dict()
+    data['cyclomatic_complexity'] = _tokenops.cyclomatic(tokens)
+    data['max_nesting_depth'] = _tokenops.max_nesting_depth(tokens)
+    data['maintainability_index'] = _tokenops.maintainability_index(tokens, stats_obj.code_lines)
+    return data
+
+
 def normalize(source: str, lang: LanguageLike, engine: str = 'regex', **options) -> str:
     """Return a canonicalized form of ``source`` for ML / clone detection.
 
@@ -329,7 +369,7 @@ def preprocess(source: str, lang: LanguageLike, steps: Sequence[str] = ('comment
 
 __all__ = [
     '__version__',
-    'Token', 'TokenType', 'CodeStats', 'Normalizer',
+    'Token', 'TokenType', 'CodeStats', 'Halstead', 'Normalizer',
     'get_language', 'detect_language',
     'remove_comments', 'extract_comments', 'count_comments', 'match_comments',
     'remove_keywords', 'extract_keywords', 'count_keywords',
@@ -338,5 +378,6 @@ __all__ = [
     'remove_strings', 'extract_strings',
     'extract_identifiers', 'remove_whitespaces',
     'blank_comments', 'stats', 'normalize',
+    'halstead', 'cyclomatic_complexity', 'maintainability_index', 'code_metrics',
     'tokenize', 'preprocess',
 ]

pyreprism-0.2.0/src/PyReprism/_tokenops.py

@@ -0,0 +1,153 @@
+"""Generic operations derived from a token stream.
+
+These work on any ``List[Token]`` regardless of which engine produced it, so the
+regex and pygments backends share one implementation of remove/extract/count,
+normalization and metrics.
+"""
+import math
+from collections import Counter
+from typing import List, Set
+
+from .metrics import CodeStats, Halstead
+from .tokens import Token, TokenType
+
+# Language-agnostic decision points used for the approximate cyclomatic complexity.
+DECISION_KEYWORDS = frozenset({
+    'if', 'elif', 'elseif', 'for', 'foreach', 'while', 'case', 'when', 'catch',
+    'except', 'and', 'or', 'unless',
+})
+# Substrings within operator tokens that add a decision (&&, ||, ternary ?).
+DECISION_OPERATORS = ('&&', '||', '?')
+
+_OPERAND_TYPES = (TokenType.IDENTIFIER, TokenType.NUMBER, TokenType.STRING)
+_OPEN_BRACKETS, _CLOSE_BRACKETS = set('([{'), set(')]}')
+
+
+def remove(tokens: List[Token], ttype: TokenType) -> str:
+    return ''.join(t.value for t in tokens if t.type is not ttype)
+
+
+def extract(tokens: List[Token], ttype: TokenType) -> List[str]:
+    return [t.value for t in tokens if t.type is ttype]
+
+
+def count(tokens: List[Token], ttype: TokenType) -> int:
+    return sum(1 for t in tokens if t.type is ttype)
+
+
+def normalize(tokens: List[Token], *, rename_identifiers: bool = True,
+              mask_numbers: bool = True, mask_strings: bool = True,
+              drop_comments: bool = True, collapse_whitespace: bool = False,
+              identifier_prefix: str = 'VAR', number_placeholder: str = '0',
+              string_placeholder: str = '"STR"') -> str:
+    mapping = {}
+    parts = []
+    for tok in tokens:
+        t = tok.type
+        if t is TokenType.COMMENT:
+            if not drop_comments:
+                parts.append(tok.value)
+        elif t is TokenType.STRING and mask_strings:
+            parts.append(string_placeholder)
+        elif t is TokenType.NUMBER and mask_numbers:
+            parts.append(number_placeholder)
+        elif t is TokenType.IDENTIFIER and rename_identifiers:
+            name = mapping.get(tok.value)
+            if name is None:
+                name = f"{identifier_prefix}{len(mapping) + 1}"
+                mapping[tok.value] = name
+            parts.append(name)
+        elif t is TokenType.WHITESPACE and collapse_whitespace:
+            parts.append(' ')
+        else:
+            parts.append(tok.value)
+    return ''.join(parts)
+
+
+def stats(tokens: List[Token], source: str) -> CodeStats:
+    counts = {t: 0 for t in TokenType}
+    code_lines = set()
+    comment_lines = set()
+    for tok in tokens:
+        counts[tok.type] += 1
+        if tok.type is TokenType.WHITESPACE:
+            continue
+        span = range(tok.line, tok.line + tok.value.count('\n') + 1)
+        target = comment_lines if tok.type is TokenType.COMMENT else code_lines
+        target.update(span)
+    total = len(source.splitlines())
+    code = len(code_lines)
+    comment = len(comment_lines - code_lines)
+    blank = max(total - code - comment, 0)
+    return CodeStats(
+        lines=total, code_lines=code, comment_lines=comment, blank_lines=blank,
+        characters=len(source),
+        comment_tokens=counts[TokenType.COMMENT],
+        string_tokens=counts[TokenType.STRING],
+        number_tokens=counts[TokenType.NUMBER],
+        keyword_tokens=counts[TokenType.KEYWORD],
+        identifier_tokens=counts[TokenType.IDENTIFIER],
+        operator_tokens=counts[TokenType.OPERATOR],
+    )
+
+
+def halstead(tokens: List[Token]) -> Halstead:
+    """Compute :class:`~PyReprism.metrics.Halstead` measures from ``tokens``."""
+    operators: Counter = Counter()
+    operands: Counter = Counter()
+    for tok in tokens:
+        if tok.type in (TokenType.OPERATOR, TokenType.KEYWORD):
+            operators[tok.value] += 1
+        elif tok.type is TokenType.OTHER and tok.value.strip():
+            operators[tok.value] += 1
+        elif tok.type in _OPERAND_TYPES:
+            operands[tok.value] += 1
+    return Halstead(
+        distinct_operators=len(operators), distinct_operands=len(operands),
+        total_operators=sum(operators.values()), total_operands=sum(operands.values()),
+    )
+
+
+def cyclomatic(tokens: List[Token], decision_keywords: Set[str] = DECISION_KEYWORDS) -> int:
+    """Approximate McCabe cyclomatic complexity: ``1 + decision points``.
+
+    Counts branch keywords plus ``&&``/``||``/``?`` operators. This is a
+    token-level approximation; an exact value needs a control-flow graph.
+    """
+    complexity = 1
+    for tok in tokens:
+        # Branch words are usually KEYWORD, but word-operators (and/or) can land
+        # as IDENTIFIER in the generic lexer, so check both.
+        if (tok.type in (TokenType.KEYWORD, TokenType.IDENTIFIER)
+                and tok.value.lower() in decision_keywords):
+            complexity += 1
+        elif tok.type is TokenType.OPERATOR:
+            for op in DECISION_OPERATORS:
+                complexity += tok.value.count(op)
+    return complexity
+
+
+def max_nesting_depth(tokens: List[Token]) -> int:
+    """Maximum bracket nesting depth (``()``/``[]``/``{}``), ignoring strings/comments."""
+    depth = deepest = 0
+    for tok in tokens:
+        if tok.type in (TokenType.STRING, TokenType.COMMENT):
+            continue
+        for ch in tok.value:
+            if ch in _OPEN_BRACKETS:
+                depth += 1
+                deepest = max(deepest, depth)
+            elif ch in _CLOSE_BRACKETS:
+                depth = max(0, depth - 1)
+    return deepest
+
+
+def maintainability_index(tokens: List[Token], code_lines: int) -> float:
+    """SEI-normalized Maintainability Index in ``[0, 100]`` (higher is better)."""
+    volume = halstead(tokens).volume
+    complexity = cyclomatic(tokens)
+    raw = (171
+           - 5.2 * math.log(max(volume, 1))
+           - 0.23 * complexity
+           - 16.2 * math.log(max(code_lines, 1)))
+    return round(max(0.0, min(100.0, raw * 100 / 171)), 2)