codevariability 0.2.0__tar.gz

codevariability-0.2.0/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+## 0.2.0 — 2026-10-03
+
+- Correct empty Python AST input scores, including comment-only source and
+  empty Python Markdown fences. Advance Python AST normalization to v3 while
+  preserving the v2 TED formula and nonempty-tree behavior.
+
+- Separate group means from statistic keys; expose `within_group_columns` for
+  collision-safe access and preserve legitimate group labels.
+- Replace the generic quoted-token fallback with linear scanning; keep token
+  output semantics and existing normalization IDs.
+- Reject optional JavaScript results when input hashes differ from the base
+  analysis. Correct source typing and add permanent adversarial/property tests.
+- Require scikit-learn >=1.5.0 and Pygments >=2.20.0 at runtime, setuptools
+  >=83.0.0 for builds, and pytest >=9.0.3 for development to exclude known
+  affected upstream versions. Add a source type-check gate.
+
+- Prepare the independent `codevariability` distribution with a `src` layout,
+  standalone tests, public examples, and explicit package contents.
+- Preserve the Python API and metric definitions from the historical 0.1.2
+  source. Keep `import codevariability` and the `codevariability` command.
+- Resolve the optional JavaScript adapter only through its installed command;
+  remove discovery of files outside the Python project.
+- Document installation, outputs, resource limits, and migration from the
+  historical local source. This is the first release from the independent
+  repository.

codevariability-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,23 @@
+# Contributing
+
+Use Python 3.10 or later. Create a virtual environment, install the development
+extra, then run the independent Python checks:
+
+```bash
+python -m pip install -e '.[dev]'
+python -m ruff check src tests examples
+python -m pytest
+python -m build
+```
+
+Node.js is optional. The JavaScript integration test skips when the separate
+adapter command is unavailable; all Python tests and builds run independently.
+
+For bug reports, include versions, selected metric names, and the smallest
+reproducible inputs that can be shared publicly. Keep credentials and private
+source code out of reports. Pull requests should describe observable behavior,
+include relevant regression tests, and update the documentation.
+
+Preserve metric identifiers when definitions remain equivalent. Changes to
+normalization, parsing, or score definitions need a new identifier and a
+migration note. See [release instructions](docs/RELEASING.md).

codevariability-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Otávio Gomes
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

codevariability-0.2.0/MANIFEST.in

@@ -0,0 +1,10 @@
+include LICENSE README.md CHANGELOG.md CONTRIBUTING.md PUBLICATION_CHECKLIST.md
+recursive-include src *.py
+recursive-include tests *.py
+recursive-include examples *.py
+recursive-include docs *.md
+global-exclude *.py[cod]
+global-exclude .DS_Store
+prune .github
+prune build
+prune dist

codevariability-0.2.0/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.4
+Name: codevariability
+Version: 0.2.0
+Summary: Source-code similarity matrices, representative rankings, and independent group comparison.
+Author: Otávio Gomes
+Maintainer: Otávio Gomes
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/otaviouss/codevariability-py
+Project-URL: Documentation, https://github.com/otaviouss/codevariability-py/tree/main/docs
+Project-URL: Source, https://github.com/otaviouss/codevariability-py
+Project-URL: Issues, https://github.com/otaviouss/codevariability-py/issues
+Keywords: code similarity,source code analysis,ast,software metrics
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=2.0
+Requires-Dist: scikit-learn>=1.5.0
+Requires-Dist: rapidfuzz>=3.0
+Requires-Dist: Pygments<3,>=2.20.0
+Requires-Dist: openpyxl>=3.1
+Provides-Extra: dev
+Requires-Dist: pytest>=9.0.3; extra == "dev"
+Requires-Dist: hypothesis>=6.168.3; extra == "dev"
+Requires-Dist: mypy>=2.4; extra == "dev"
+Requires-Dist: ruff>=0.14; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=6; extra == "dev"
+Dynamic: license-file
+
+# CodeVariability for Python
+
+CodeVariability compares UTF-8 source-code files and returns similarity
+matrices, descriptive statistics, representative rankings, and comparisons
+between independent groups. It helps you identify common structures and
+unusual variants in a collection without executing the submitted code.
+
+Version **0.2.0** is the first release of this independent repository. The API is alpha. The distribution name is `codevariability`;
+the import name and command are `codevariability`.
+
+## Installation
+
+Python 3.10 or later is required. Install from PyPI:
+
+```bash
+python -m pip install codevariability==0.2.0
+```
+
+For development, install a checkout with `python -m pip install .`.
+The historical 0.1.2 code is the implementation baseline; 0.2.0 is the first
+public PyPI release. Use a new virtual environment when migrating from a
+local historical installation.
+
+## Quick start
+
+This example creates its own inputs and works with an installed package:
+
+```python
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from codevariability import analyze
+
+with TemporaryDirectory() as directory:
+    folder = Path(directory)
+    (folder / "a.py").write_text("def total(a, b):\n    return a + b\n", encoding="utf-8")
+    (folder / "b.py").write_text("def sum_values(x, y):\n    return x + y\n", encoding="utf-8")
+    result = analyze(folder, metrics="all")
+    print(result.statistics)
+    print(result.ranking)
+    print(result.most_representative)
+```
+
+The runnable [example](https://github.com/otaviouss/codevariability-py/blob/main/examples/basic.py) also exports JSON, CSV, and Excel.
+For the bundled input files, the CLI is:
+
+```bash
+codevariability analyze examples/inputs --metrics all --output analysis-output
+codevariability analyze examples/inputs --metrics cosine jaccard --output analysis-output/report.xlsx
+```
+
+## Inputs and metrics
+
+Pass a directory or a list of file paths to `analyze()`. Directory scans are
+not recursive. File basenames must be unique. Files are decoded as UTF-8,
+including an optional BOM. Use `extensions="py"` or a list of extensions to
+filter a directory.
+
+| Metric | What it compares |
+| --- | --- |
+| `cosine` | Word frequencies in the full document. |
+| `jaccard` | Sets of words in the full document. |
+| `lcs` | The longest common subsequence of code tokens. |
+| `levenshtein` | Token sequences using unit edit costs. |
+| `ast_tree_edit_similarity` | Ordered, normalized Python syntax trees. |
+
+`metrics="all"` includes all applicable metrics. Python AST analysis requires
+Python inputs; explicit selection on incompatible inputs raises `AnalysisError`.
+Source files are read in full. For Markdown, token and AST metrics use the
+identified fenced code blocks; textual metrics include the full document.
+
+## Results and export
+
+`AnalysisResult` exposes `matrices`, `statistics`, `representativeness`,
+`ranking`, `rankings_by_metric`, `most_representative`, and `most_distinct`.
+Similarities range from 0 to 1. Statistics use each unique pair once. Rankings
+average within each available dimension and then give dimensions equal weight.
+
+Use `result.export(directory, formats=("json", "csv"))` for machine-readable
+results and tables, or `result.to_excel(path)` for a workbook. Outputs include
+metric identifiers, normalization and runtime versions, and input hashes.
+Spreadsheet exports escape formula-like labels; JSON retains the original
+labels. See [input/output details](https://github.com/otaviouss/codevariability-py/blob/main/docs/FORMATS.md).
+
+`compare_groups()` supports two groups or a mapping of two or more groups,
+file-label permutation tests, and Holm-adjusted p-values. Each group needs at
+least two files and groups must not share physical files. A complete example
+is in [examples/groups.py](https://github.com/otaviouss/codevariability-py/blob/main/examples/groups.py).
+
+## Optional JavaScript integration
+
+The Python library works without Node.js. To request structural JavaScript or
+TypeScript analysis through `compare_groups(..., include_ast=True)`, install
+the independent [codevariability-js](https://github.com/otaviouss/codevariability-js)
+package and make its command available on `PATH`. Alternatively, import a
+single-metric JSON with `load_matrix_json()` and attach it with `with_metric()`.
+The Python build and normal test suite require no JavaScript checkout.
+
+## Limits and errors
+
+Scores describe text, tokens, or syntax; they do not establish functional
+equivalence, correctness, authorship, or copied-code percentages. AST
+normalization removes concrete names and literal values. Different sources
+can therefore have structural similarity 1.
+
+Exact tree comparison can be expensive. The default `max_ted_cells=2_000_000`
+limits tables for each non-identical AST pair; exceeding it raises an error,
+without approximation. `None` removes the limit. It does not limit source
+file size, the full pairwise matrix, or total CPU time. Use trusted output
+directories and apply application-level resource limits for untrusted inputs.
+
+Expected input and serialization errors raise `AnalysisError`. JavaScript
+integration also supports `ast_timeout=120` seconds. See the
+[API](https://github.com/otaviouss/codevariability-py/blob/main/docs/API.md) and [metric definitions](https://github.com/otaviouss/codevariability-py/blob/main/docs/METRICS.md) for details.
+
+## Contributing and license
+
+See [CONTRIBUTING.md](https://github.com/otaviouss/codevariability-py/blob/main/CONTRIBUTING.md) for development and
+[PUBLICATION_CHECKLIST.md](https://github.com/otaviouss/codevariability-py/blob/main/PUBLICATION_CHECKLIST.md) for release preparation.
+The project uses the [MIT license](https://github.com/otaviouss/codevariability-py/blob/main/LICENSE), copyright 2026 Otávio Gomes.
+
+## Compatibility
+
+Arbitrary distinct nonempty group names are
+accepted; use `result.within_group_columns` to locate collision-safe within-group
+means. Keep files unchanged while an analysis runs; optional JavaScript results
+with different input hashes raise `AnalysisError`.
+
+Runtime minimums are scikit-learn 1.5.0 and Pygments 2.20.0; build minimum is
+setuptools 83.0.0 and development tests require pytest 9.0.3. These scopes are
+separate. See [API](docs/API.md) and [metric limits](docs/METRICS.md).
+
+Python AST normalization now uses v3 to identify corrected empty-program
+behavior. The TED formula ID remains v2; use compatible normalization IDs when
+comparing results across versions.

codevariability-0.2.0/PUBLICATION_CHECKLIST.md

@@ -0,0 +1,29 @@
+# Python publication checklist
+
+Prepared version: **0.2.0**. Distribution: **codevariability**.
+Import/CLI: **codevariability**. A checked item records an executed check;
+unchecked items are required before the first public release.
+
+- [x] Public source and selected behavior tests reviewed.
+- [x] README, API, metric, format, and release documentation reviewed.
+- [x] MIT license and original copyright preserved.
+- [x] Metadata, distribution name, import name, and single-source version checked.
+- [x] Dependencies resolved and checked for known vulnerabilities.
+- [x] Independent Python tests and lint passed.
+- [ ] Run repository CI on the final remediated commit (local runtime matrix is validated separately).
+- [x] Clean wheel and sdist builds completed.
+- [x] Every wheel and sdist member inspected.
+- [x] Wheel installed in a fresh environment; import and CLI checked.
+- [x] Sdist rebuilt independently.
+- [x] README code and all bundled examples executed.
+- [x] New files checked for secrets, local paths, and internal materials.
+- [x] No research inputs, manuscript files, caches, or generated results included.
+- [x] New repository URLs verified with authenticated access.
+- [x] PyPI project-name existence checked; recheck ownership/version before upload.
+- [x] Make the GitHub repository public and verify links without authentication.
+- [ ] Configure registry authentication or a trusted publisher for this new project.
+- [ ] Choose the release date, publish, and verify installation from the registry.
+
+The historical local implementation is the baseline; this distribution is
+the first public PyPI release. See
+[docs/RELEASING.md](docs/RELEASING.md).

codevariability-0.2.0/README.md

@@ -0,0 +1,134 @@
+# CodeVariability for Python
+
+CodeVariability compares UTF-8 source-code files and returns similarity
+matrices, descriptive statistics, representative rankings, and comparisons
+between independent groups. It helps you identify common structures and
+unusual variants in a collection without executing the submitted code.
+
+Version **0.2.0** is the first release of this independent repository. The API is alpha. The distribution name is `codevariability`;
+the import name and command are `codevariability`.
+
+## Installation
+
+Python 3.10 or later is required. Install from PyPI:
+
+```bash
+python -m pip install codevariability==0.2.0
+```
+
+For development, install a checkout with `python -m pip install .`.
+The historical 0.1.2 code is the implementation baseline; 0.2.0 is the first
+public PyPI release. Use a new virtual environment when migrating from a
+local historical installation.
+
+## Quick start
+
+This example creates its own inputs and works with an installed package:
+
+```python
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from codevariability import analyze
+
+with TemporaryDirectory() as directory:
+    folder = Path(directory)
+    (folder / "a.py").write_text("def total(a, b):\n    return a + b\n", encoding="utf-8")
+    (folder / "b.py").write_text("def sum_values(x, y):\n    return x + y\n", encoding="utf-8")
+    result = analyze(folder, metrics="all")
+    print(result.statistics)
+    print(result.ranking)
+    print(result.most_representative)
+```
+
+The runnable [example](https://github.com/otaviouss/codevariability-py/blob/main/examples/basic.py) also exports JSON, CSV, and Excel.
+For the bundled input files, the CLI is:
+
+```bash
+codevariability analyze examples/inputs --metrics all --output analysis-output
+codevariability analyze examples/inputs --metrics cosine jaccard --output analysis-output/report.xlsx
+```
+
+## Inputs and metrics
+
+Pass a directory or a list of file paths to `analyze()`. Directory scans are
+not recursive. File basenames must be unique. Files are decoded as UTF-8,
+including an optional BOM. Use `extensions="py"` or a list of extensions to
+filter a directory.
+
+| Metric | What it compares |
+| --- | --- |
+| `cosine` | Word frequencies in the full document. |
+| `jaccard` | Sets of words in the full document. |
+| `lcs` | The longest common subsequence of code tokens. |
+| `levenshtein` | Token sequences using unit edit costs. |
+| `ast_tree_edit_similarity` | Ordered, normalized Python syntax trees. |
+
+`metrics="all"` includes all applicable metrics. Python AST analysis requires
+Python inputs; explicit selection on incompatible inputs raises `AnalysisError`.
+Source files are read in full. For Markdown, token and AST metrics use the
+identified fenced code blocks; textual metrics include the full document.
+
+## Results and export
+
+`AnalysisResult` exposes `matrices`, `statistics`, `representativeness`,
+`ranking`, `rankings_by_metric`, `most_representative`, and `most_distinct`.
+Similarities range from 0 to 1. Statistics use each unique pair once. Rankings
+average within each available dimension and then give dimensions equal weight.
+
+Use `result.export(directory, formats=("json", "csv"))` for machine-readable
+results and tables, or `result.to_excel(path)` for a workbook. Outputs include
+metric identifiers, normalization and runtime versions, and input hashes.
+Spreadsheet exports escape formula-like labels; JSON retains the original
+labels. See [input/output details](https://github.com/otaviouss/codevariability-py/blob/main/docs/FORMATS.md).
+
+`compare_groups()` supports two groups or a mapping of two or more groups,
+file-label permutation tests, and Holm-adjusted p-values. Each group needs at
+least two files and groups must not share physical files. A complete example
+is in [examples/groups.py](https://github.com/otaviouss/codevariability-py/blob/main/examples/groups.py).
+
+## Optional JavaScript integration
+
+The Python library works without Node.js. To request structural JavaScript or
+TypeScript analysis through `compare_groups(..., include_ast=True)`, install
+the independent [codevariability-js](https://github.com/otaviouss/codevariability-js)
+package and make its command available on `PATH`. Alternatively, import a
+single-metric JSON with `load_matrix_json()` and attach it with `with_metric()`.
+The Python build and normal test suite require no JavaScript checkout.
+
+## Limits and errors
+
+Scores describe text, tokens, or syntax; they do not establish functional
+equivalence, correctness, authorship, or copied-code percentages. AST
+normalization removes concrete names and literal values. Different sources
+can therefore have structural similarity 1.
+
+Exact tree comparison can be expensive. The default `max_ted_cells=2_000_000`
+limits tables for each non-identical AST pair; exceeding it raises an error,
+without approximation. `None` removes the limit. It does not limit source
+file size, the full pairwise matrix, or total CPU time. Use trusted output
+directories and apply application-level resource limits for untrusted inputs.
+
+Expected input and serialization errors raise `AnalysisError`. JavaScript
+integration also supports `ast_timeout=120` seconds. See the
+[API](https://github.com/otaviouss/codevariability-py/blob/main/docs/API.md) and [metric definitions](https://github.com/otaviouss/codevariability-py/blob/main/docs/METRICS.md) for details.
+
+## Contributing and license
+
+See [CONTRIBUTING.md](https://github.com/otaviouss/codevariability-py/blob/main/CONTRIBUTING.md) for development and
+[PUBLICATION_CHECKLIST.md](https://github.com/otaviouss/codevariability-py/blob/main/PUBLICATION_CHECKLIST.md) for release preparation.
+The project uses the [MIT license](https://github.com/otaviouss/codevariability-py/blob/main/LICENSE), copyright 2026 Otávio Gomes.
+
+## Compatibility
+
+Arbitrary distinct nonempty group names are
+accepted; use `result.within_group_columns` to locate collision-safe within-group
+means. Keep files unchanged while an analysis runs; optional JavaScript results
+with different input hashes raise `AnalysisError`.
+
+Runtime minimums are scikit-learn 1.5.0 and Pygments 2.20.0; build minimum is
+setuptools 83.0.0 and development tests require pytest 9.0.3. These scopes are
+separate. See [API](docs/API.md) and [metric limits](docs/METRICS.md).
+
+Python AST normalization now uses v3 to identify corrected empty-program
+behavior. The TED formula ID remains v2; use compatible normalization IDs when
+comparing results across versions.

codevariability-0.2.0/docs/API.md

@@ -0,0 +1,101 @@
+# Python API
+
+Import the public API from `codevariability`. Submodules implement the metrics
+and validation; they are not a separately versioned extension API.
+
+## Analysis
+
+```text
+analyze(files_or_directory, metrics="all", *, extensions=None,
+        max_ted_cells=2_000_000) -> AnalysisResult
+```
+
+`files_or_directory` is a directory, an individual file, or a sequence of file
+paths. `extensions` filters directory inputs only. `metrics` accepts one name,
+a sequence, or `"all"`. Duplicate selected names are deduplicated.
+
+For repeated use, `CodeDataset.from_directory(directory, extensions=None)`
+and `CodeDataset.from_files(paths)` create an immutable mapping of labels to
+paths. `dataset.analyze(metrics="all", max_ted_cells=2_000_000)` reads the
+current file contents and returns a new result.
+
+| AnalysisResult member | Return value or action |
+| --- | --- |
+| `files` / `metrics` | File labels / tuple of selected metric names. |
+| `matrices` | Mapping from metric name to a square pandas DataFrame. |
+| `statistics` | One row per metric, using unique off-diagonal pairs. |
+| `representativeness` | Mean scores by metric and dimension, overall score, and row dispersion. |
+| `ranking` | Overall ranking: `rank`, `file`, and `score`. |
+| `rankings_by_metric` | One ranking DataFrame per metric. |
+| `most_representative` / `most_distinct` | Dictionaries containing `file`, `score`, and `rank`. |
+| `medoid(metric)` | The file with highest mean similarity for that metric. |
+| `composite(weights="equal")` | A separate weighted similarity DataFrame; it does not replace the normal ranking. |
+| `with_metric(name, matrix)` | A new result containing a validated external matrix. |
+| `export(directory, formats=("json", "csv"))` | Write JSON and/or CSV files. |
+| `to_excel(path)` | Write one workbook. |
+
+Custom composite weights must cover exactly the selected metrics, be finite
+and nonnegative, and have a positive sum. External matrices must have unique
+matching labels, finite real entries in [0, 1], symmetry, and diagonal 1.
+Boolean and string entries are rejected. Names and dimensions must not collide
+with reserved result columns. Derived tables are recalculated from the current
+matrices; augmented results copy their matrices and metadata.
+
+## Comparing groups
+
+```text
+compare_groups(group_a, group_b=None, *, metrics="all",
+               group_names=("group_a", "group_b"), permutations=10_000,
+               random_state=42, include_ast=False, alpha=0.05,
+               progress=False, cache_dir=None, ast_timeout=120.0,
+               max_ted_cells=2_000_000)
+```
+
+Pass two inputs (paths or lists of files) for a `GroupComparisonResult`, or a
+mapping from group name to input for a `MultiGroupComparisonResult`. The mapping
+form requires at least two groups. Each group needs at least two files; groups
+must not share files, including through hardlinks. `progress` accepts a boolean
+or a callback receiving messages.
+
+The two-group result exposes `summary`, `overview`, `interpretation`,
+`by_metric`, `by_dimension`, `within_groups`, `between_groups`, and `p_values`.
+The mapping result exposes `global_test`, `pairwise`, `within_groups`,
+`between_groups`, `overview`, `pairwise_overview`, and `interpretation`.
+Both support `print_report()`, `export(directory)`, and `to_excel(path)`.
+
+Group names must be distinct, nonempty strings. They are labels, not statistic
+keys. For two groups, `within_group_columns` returns the two summary columns
+in group order and is also recorded in metadata. Usually these are
+`within_<name>`. If a label would collide with a statistic or another label,
+`__group` is appended until the column is unique. Thus a group named
+`difference` remains valid without replacing the `within_difference` contrast.
+Use this property when reading arbitrary user-supplied group names.
+
+Tests permute file-level group labels with fixed group sizes and use the
+Monte Carlo correction `(extremes + 1)/(permutations + 1)`. Two-group
+homogeneity and separation tests are two-sided. Global multigroup homogeneity
+uses the upper tail of dispersion among within-group means. Holm correction
+is applied separately by hypothesis family; pairwise correction includes all
+reported pairs and measures. Group weighting is equal. These tests assume
+exchangeable, independent files; repeated or paired observations need another
+permutation design. See [examples/groups.py](../examples/groups.py).
+
+## JavaScript integration and errors
+
+`include_ast=True` adds structural JavaScript/TypeScript analysis only when
+Python AST analysis is not already present. The installed `codevariability-js`
+command must be on PATH. No sibling directory or another checkout is searched.
+`ast_timeout` must be a positive finite number of seconds; timeout or adapter
+failure raises `AnalysisError`. `cache_dir` is passed to the optional adapter.
+
+Keep inputs unchanged throughout group analysis, including progress callbacks.
+The optional adapter's raw input hashes must match those used by the base
+analysis; a changed snapshot raises `AnalysisError` rather than combining
+results from different file contents.
+
+`load_matrix_json(path)` returns `(metric_name, pandas.DataFrame)` for the
+single-metric schema in [FORMATS.md](FORMATS.md). Attach it with `with_metric()`.
+
+Expected invalid input, incompatible metric, parsing, resource-budget, or
+serialization failures raise `AnalysisError`. Filesystem failures may also
+raise `OSError`. The CLI reports expected failures with a nonzero exit status.

codevariability-0.2.0/docs/FORMATS.md

@@ -0,0 +1,43 @@
+# Inputs and outputs
+
+Source files use UTF-8, optionally with BOM. Directory discovery is nonrecursive
+and uses natural filename ordering with deterministic ties. All regular files
+are considered unless an extension filter is supplied. Explicit file lists
+must have unique basenames. No submitted program is executed.
+
+## Analysis files
+
+`export()` writes `analysis.json`, `statistics.csv`, `representativeness.csv`,
+`ranking.csv`, `<metric>_similarity_matrix.csv`, and `<metric>_ranking.csv`.
+Select only JSON or CSV with the `formats` parameter. JSON contains file names,
+metrics, metadata, matrices, statistics, and rankings. Non-simple external
+metric names receive a safe filename containing a hash suffix; the original
+name is retained inside the result and JSON.
+
+The workbook contains Summary, Metadata, Representativeness, Ranking,
+Statistics, and one sheet per metric. Group exports have separate group
+statistics files and an overview workbook.
+
+File and group labels resembling formulas are escaped in CSV and Excel,
+including row and column labels. Escaped CSV labels have a leading tab inside
+quoted fields; workbook labels use a leading apostrophe. JSON keeps exact
+labels and is the preferred interchange format when labels matter.
+
+Individual output files are replaced atomically. Existing destination symlinks
+are rejected. Multi-file export is not a transaction over the entire directory.
+Output/cache directories should be trusted; the library does not confine all
+filesystem operations to a sandbox root. Input hashes identify file bytes and
+are included with runtime, metric, normalization, and aggregation metadata.
+
+## External matrix JSON
+
+The supported schema is `codevariability.matrix.v1`. It requires `metric`,
+unique nonempty `files`, and a square real-number `matrix`. Optional `metric_id`
+and `metadata` identify the adapter, normalization, and metric dimension.
+The matrix must be symmetric, have diagonal 1, and contain finite values in
+[0, 1]. Duplicate JSON keys, booleans, implicit numeric strings, and nonfinite
+numbers are rejected. Multi-metric `codevariability.analysis.v1` output from
+JavaScript is not accepted by `load_matrix_json()`; export one metric instead.
+
+A self-contained loading and attachment example is in
+[examples/interop.py](../examples/interop.py).

codevariability-0.2.0/docs/METRICS.md

@@ -0,0 +1,68 @@
+# Metric definitions and limits
+
+Text normalization applies Unicode NFC, case folding, and Unicode word
+extraction to the entire document. `cosine` uses raw word-count vectors,
+not TF-IDF; an empty vector has zero similarity to other files, even another
+empty document. `jaccard` uses word sets, returning 1 for two empty sets.
+All result matrices explicitly have diagonal 1.
+
+For code-token metrics, `.py` and other source files are read in full.
+Markdown uses identified fenced blocks; backtick and tilde fences are supported,
+with the first word of the opening information string naming the language.
+Unclosed fences raise an error. Markdown without code blocks has an empty code
+sequence. Pygments handles known languages; unknown languages use a generic
+deterministic tokenizer. Comments and whitespace are excluded.
+
+`lcs` divides longest common subsequence length by the longer token sequence.
+`levenshtein` uses unit insertion, deletion, and substitution costs and returns
+`1 - distance / max_length`. Two empty sequences have similarity 1; an empty
+and a nonempty sequence have similarity 0.
+
+Python AST normalization preserves ordered syntax nodes, operators, literal
+categories, and structural fields. Concrete identifiers, literal values,
+comments, formatting, and locations are removed. Identified Python Markdown
+fragments appear in order under a synthetic root. Empty or comment-only
+fragments are ignored. Python grammar depends on the running interpreter.
+
+`ast_tree_edit_similarity` uses exact ordered Zhang–Shasha tree edit distance
+with unit insertion, deletion, and relabeling costs. For nonempty normalized
+trees with `n` and `m` nodes:
+
+```text
+similarity = 1 - tree_edit_distance / (n + m - 1)
+```
+
+The denominator is a valid upper bound: remove non-root nodes, relabel the
+root if necessary, and insert the target's non-root nodes. Empty/empty has
+similarity 1; empty/nonempty has similarity 0. The score is symmetric and in
+[0, 1], but does not establish semantic or functional equivalence.
+
+Exact TED uses O(nm) table memory and shape-dependent computation. Its default
+budget is 2,000,000 cells `(n+1)*(m+1)` for a non-identical pair. Equal trees
+are checked first without those tables. `max_ted_cells=None` removes the
+budget. Parsing, full matrices, file sizes, and permutations have separate
+resource costs that the caller must manage.
+
+Rankings first average metrics within each dimension (`textual`,
+`syntactic_token_sequence`, `structural_ast`), then average available dimensions
+equally. If an external AST node-frequency baseline and TED are both present,
+only TED enters the structural dimension. `overall_dispersion` is the sample
+standard deviation of a file's off-diagonal values in the equally weighted
+dimension-average matrix; it is 0 when fewer than two values exist.
+
+Internal IDs and normalization versions are recorded in metadata. Python AST
+uses `ast_tree_edit_similarity_v2` and `python_normalized_ast_tree_v3`.
+Compare results only when metric IDs, normalization, runtime/parser versions,
+and selected dimensions are compatible.
+
+The generic fallback scans quoted tokens in linear time with linear auxiliary
+storage; token output semantics and `pygments_lexemes_with_generic_fallback_v2`
+remain unchanged.
+
+Version 0.2.0 advances Python AST normalization to v3 so that
+empty/comment-only source and empty Python fences follow the documented scores:
+empty/empty is 1 and empty/nonempty is 0. Nonempty trees and the TED formula
+remain unchanged. The normalization helper retains its synthetic container;
+the public metric treats a container without fragments as an empty tree.
+This improvement does not make LCS, exact TED, permutation tests, or complete
+pairwise matrices linear. Callers must still budget total work and input sizes.