mlrepromutate 0.1.1__tar.gz

mlrepromutate-0.1.1/.gitignore

@@ -0,0 +1,29 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+
+# Virtual environments
+.venv/
+venv/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Ruff
+.ruff_cache/
+
+# IDE
+.vscode/
+.idea/
+
+# OS
+.DS_Store
+
+# Temporary experiment data
+.tmp/
+tmp/

mlrepromutate-0.1.1/CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog
+
+All notable changes to MLReproMutate will be documented in this file.
+
+MLReproMutate is currently pre-1.0 software; its public API may change between
+releases.
+
+## [0.1.1] - 2026-08-28
+
+### Fixed
+
+- Report the installed MLReproMutate package version from distribution
+  metadata instead of a hard-coded development placeholder.
+- Resolve validation commands using either `python` or `python3` when one of
+  those common executable aliases is unavailable.
+
+### Added
+
+- Interactive command-line setup for guided project, operator, validation,
+  execution-mode, and candidate selection.
+- Static `detect` command for previewing mutation candidates without executing
+  project code.
+- `in-place` execution mode for disposable, CI, container, or otherwise safely
+  resettable workspaces.
+- Repeatable project-relative `--exclude` options for omitting unnecessary
+  paths from sandbox copies.
+- User-facing quick-start and command-line documentation.
+- CI usage documentation for reproducibility mutation checks.
+- Expanded worked examples for the supported mutation operators.
+- Public citation and link for the accompanying empirical study,
+  arXiv:2608.27100.
+- GitHub Actions workflow for PyPI Trusted Publishing.
+
+### Changed
+
+- Require explicit selection of a mutation operator instead of defaulting to
+  `dependency-pin`.
+- Keep `sandbox` as the safe default execution mode while allowing users to
+  avoid full project copies with explicit `in-place` execution.
+- Expanded installation, usage, interpretation, isolation, and contribution
+  guidance.
+
+
+## [0.1.0] - 2026-08-27
+### Changed
+
+- Reuse a successful baseline validation across all mutation candidates in a single orchestration run.
+
+### Added
+
+- Initial public repository structure.
+- Initial project scope and research question.
+- Initial threat-model documentation.
+- Continuous integration setup.
+- Core mutation outcome, candidate, and result models.
+- Base interface for reproducibility mutation operators.
+- Isolated temporary project sandbox for safe mutation execution.
+- Experiment command runner with captured output, exit status, runtime, and timeout handling.
+- Dependency mutation operator for relaxing exact pins in requirements files.
+- Mutation evaluation pipeline with baseline validation and KILLED, SURVIVED, and TIMEOUT classification.
+- End-to-end dependency mutation fixture demonstrating both surviving and killed reproducibility mutations.
+- Mutation orchestration layer for detecting and evaluating operator candidates.
+- Initial `mlrepromutate run` command for end-to-end dependency mutation evaluation.
+- Added incremental CLI progress reporting for baseline validation and individual mutation evaluation.
+- Added workflow-aware dependency mutation scoping through `--requirements-file`.
+- Mutation progress output now includes the source file and line number.
+- Added machine-readable JSON reports containing validation provenance,
+  mutation metadata, outcomes, execution durations, and Git revisions.
+- Added an isolated virtual-environment resolver foundation for future
+  dependency re-resolution experiments.
+- Added resolved dependency evaluation with explicit `INVALID`,
+  `EQUIVALENT`, `SURVIVED`, `KILLED`, and `TIMEOUT` semantics.
+- Added a controlled offline resolved-dependency fixture demonstrating a real
+  dependency transition from version 1.0.0 to 1.1.0.
+- Added AST-based detection and mutation of literal Python random seeds for
+  `random.seed`, NumPy seed calls, and `torch.manual_seed`.
+- Added a controlled random-seed fixture demonstrating both survived and
+  killed reproducibility mutations.
+- Added the `data-split` mutation operator for removing explicit
+  `train_test_split` stratification.
+- Added the `cv-fold-count` mutation operator for changing explicit
+  cross-validation fold counts.

mlrepromutate-0.1.1/CITATION.cff

@@ -0,0 +1,18 @@
+cff-version: 1.2.0
+message: "If you use MLReproMutate in research, please cite this software."
+type: software
+title: "MLReproMutate"
+authors:
+  - family-names: "Shulepov"
+    given-names: "Ilya"
+    orcid: "https://orcid.org/0009-0001-1348-9576"
+version: 0.1.1
+date-released: 2026-08-28
+repository-code: "https://github.com/ilyuka/MLReproMutate"
+url: "https://github.com/ilyuka/MLReproMutate"
+license: MIT
+keywords:
+  - mutation testing
+  - reproducibility
+  - machine learning
+  - research software

mlrepromutate-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ilya Shulepov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mlrepromutate-0.1.1/PKG-INFO

@@ -0,0 +1,413 @@
+Metadata-Version: 2.5
+Name: mlrepromutate
+Version: 0.1.1
+Summary: Mutation testing for reproducibility safeguards in machine-learning experiments
+Project-URL: Homepage, https://github.com/ilyuka/MLReproMutate
+Project-URL: Repository, https://github.com/ilyuka/MLReproMutate
+Project-URL: Issues, https://github.com/ilyuka/MLReproMutate/issues
+Project-URL: Documentation, https://github.com/ilyuka/MLReproMutate/tree/main/docs
+Project-URL: Changelog, https://github.com/ilyuka/MLReproMutate/blob/main/CHANGELOG.md
+Project-URL: Preprint, https://arxiv.org/abs/2608.27100
+Author: Ilya Shulepov
+License: MIT
+License-File: LICENSE
+Keywords: machine learning,mutation testing,reproducibility,research software
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# MLReproMutate
+
+MLReproMutate is research software for mutation testing of
+reproducibility-relevant safeguards in machine-learning research software.
+
+It introduces controlled changes to experimental and environment choices and
+evaluates whether validation workflows already present in a repository detect
+those changes.
+
+MLReproMutate is intended for empirical software-engineering research,
+machine-learning reproducibility studies, and developers who want to assess
+whether existing validation workflows constrain reproducibility-relevant
+experimental choices.
+
+## Installation
+
+MLReproMutate requires Python 3.11 or newer.
+
+Clone the repository and install the package:
+
+```bash
+git clone https://github.com/ilyuka/MLReproMutate.git
+cd MLReproMutate
+
+python -m venv .venv
+source .venv/bin/activate
+
+python -m pip install .
+```
+
+Confirm the installation:
+
+```bash
+mlrepromutate version
+mlrepromutate --help
+```
+
+For a guided first run, start the interactive setup:
+
+```bash
+mlrepromutate
+```
+
+The wizard guides you through project selection, mutation operator,
+validation command, execution mode, candidate preview, and confirmation.
+
+For development dependencies:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+## Quick start
+
+The repository includes small CPU-runnable fixtures demonstrating the mutation
+workflow.
+
+The `random-seed` fixture can be run without additional machine-learning
+dependencies:
+
+```bash
+mlrepromutate run examples/random-seed \
+  --command "python validate_unguarded.py" \
+  --operator random-seed \
+  --python-file experiment.py
+```
+
+MLReproMutate first validates the unmodified baseline. It then detects the
+supported mutation candidate, applies the mutation in the selected execution
+workspace, runs the same validation command, and reports whether the workflow
+detected the change. The default `sandbox` execution mode uses temporary
+isolated project copies.
+
+For the unguarded fixture, the random-seed mutation survives because the
+validation workflow checks only that the experiment completes successfully.
+
+The same mutation can be evaluated against a workflow containing an explicit
+reproducibility safeguard:
+
+```bash
+mlrepromutate run examples/random-seed \
+  --command "python validate_guarded.py" \
+  --operator random-seed \
+  --python-file experiment.py
+```
+
+For this fixture, the mutation is killed because the guarded workflow checks the
+deterministic output associated with the original seed.
+
+See the [quick-start guide](docs/quickstart.md) for the complete walkthrough.
+
+## Mutation operators
+
+The current release implements four reproducibility-relevant mutation classes.
+
+### `random-seed`
+
+Changes a supported literal random seed from `N` to `N + 1`.
+
+Supported seed-setting forms include calls such as:
+
+```python
+random.seed(42)
+np.random.seed(42)
+numpy.random.seed(42)
+torch.manual_seed(42)
+```
+
+### `dependency-pin`
+
+Relaxes an exact dependency constraint:
+
+```text
+package==version
+```
+
+to:
+
+```text
+package>=version
+```
+
+The resolved-dependency evaluation mode can additionally determine whether the
+changed specification actually resolves to a different installed version of the
+target dependency.
+
+### `data-split`
+
+Changes a supported `train_test_split` call containing an explicit non-`None`
+`stratify` argument:
+
+```python
+stratify=<expression>
+```
+
+to:
+
+```python
+stratify=None
+```
+
+### `cv-fold-count`
+
+Changes an explicit literal cross-validation fold count from `N` to `N + 1`.
+
+Supported splitter classes include `KFold`, `StratifiedKFold`,
+`RepeatedKFold`, and `RepeatedStratifiedKFold`.
+
+## Evaluation model
+
+Mutation evaluation is baseline-first.
+
+The unmodified project is evaluated before any mutation outcome is interpreted.
+A baseline failure or timeout is therefore kept separate from a mutation
+result.
+
+After a successful baseline, selected mutations are evaluated in isolated
+project workspaces using the same validation command.
+
+At the execution level:
+
+- `KILLED` means that the selected validation workflow returned a non-zero
+  status after the mutation was applied.
+- `SURVIVED` means that the selected validation workflow completed successfully
+  after the mutation was applied.
+- baseline failures and validation timeouts are represented separately.
+- additional semantic states are used where necessary, including dependency
+  equivalence handling during resolved evaluation.
+
+A survived mutation does **not** by itself establish that a repository or its
+scientific results are irreproducible. It shows only that the selected
+validation workflow did not detect that particular controlled change.
+
+## Candidate preview
+
+Mutation candidates can be inspected without executing project code:
+
+```bash
+mlrepromutate detect examples/random-seed \
+  --operator random-seed \
+  --python-file experiment.py
+```
+
+`detect` performs candidate detection only. It does not run the baseline,
+validation command, or mutants.
+
+## Execution modes
+
+The default `sandbox` mode evaluates the project in temporary copies so that
+mutation targets in the original project are not modified:
+
+```bash
+mlrepromutate run PROJECT \
+  --operator random-seed \
+  --command "pytest -q" \
+  --execution-mode sandbox
+```
+
+Large directories that are not needed by the validation workflow can be omitted
+from sandbox copies with repeatable project-relative `--exclude` options:
+
+```bash
+mlrepromutate run PROJECT \
+  --operator random-seed \
+  --command "pytest -q" \
+  --execution-mode sandbox \
+  --exclude data \
+  --exclude checkpoints
+```
+
+Exclusion paths are interpreted relative to the project root. Absolute paths
+and parent-directory traversal are rejected.
+
+The `in-place` mode avoids copying the project and is intended for disposable
+or version-controlled workspaces such as CI checkouts:
+
+```bash
+mlrepromutate run PROJECT \
+  --operator random-seed \
+  --command "pytest -q" \
+  --execution-mode in-place
+```
+
+MLReproMutate restores its mutation target after each in-place evaluation,
+including failed or timed-out validations. However, arbitrary side effects
+created by the validation command itself are not reverted. For that reason,
+`in-place` should be used only in workspaces where such side effects are safe.
+
+`dependency-pin --dependency-mode resolved` currently requires `sandbox` mode.
+
+## Continuous integration
+
+MLReproMutate can be used as a CI validation step. A disposable CI checkout is
+a natural fit for `in-place` execution because no full project copy is needed:
+
+```yaml
+- uses: actions/checkout@v4
+
+- name: Install MLReproMutate
+  run: python3 -m pip install mlrepromutate
+
+- name: Check reproducibility safeguards
+  run: |
+    mlrepromutate run . \
+      --operator random-seed \
+      --python-file experiment.py \
+      --execution-mode in-place \
+      --command "pytest -q"
+```
+
+For human use, `mlrepromutate` provides an interactive setup. For scripts, CI,
+and reproducible research workflows, prefer explicit `detect` and `run`
+commands.
+
+## Python validation commands
+
+For validation commands whose executable is exactly `python` or `python3`,
+MLReproMutate resolves the requested executable from `PATH` and falls back to
+the other common alias when necessary. Other executables such as `pytest`,
+`bash`, and `make` are not rewritten.
+
+## Machine-readable reports
+
+Use `--json-out` to write a structured report:
+
+```bash
+mlrepromutate run examples/random-seed \
+  --command "python validate_unguarded.py" \
+  --operator random-seed \
+  --python-file experiment.py \
+  --json-out report.json
+```
+
+Reports contain software and project metadata, validation information, mutation
+candidate metadata, outcomes, and execution details.
+
+## Documentation
+
+User documentation:
+
+- [Quick start](docs/quickstart.md)
+- [Command-line interface](docs/cli.md)
+- [Examples](examples/README.md)
+
+Research and design materials:
+
+- [Threat model](docs/threat-model.md)
+- [Research log](docs/research-log.md)
+- [Related-work notes](docs/related-work.md)
+
+Command-line help for the installed version is also available with:
+
+```bash
+mlrepromutate run --help
+```
+
+## Examples
+
+The `examples/` directory contains small fixtures for exercising individual
+mutation concepts and evaluation behavior.
+
+These fixtures are separate from the frozen empirical corpus and are intended
+for documentation, testing, and software evaluation.
+
+See [examples/README.md](examples/README.md).
+
+## Empirical research
+
+MLReproMutate has been used as the experimental instrument in an empirical
+study of reproducibility-relevant safeguards in machine-learning research
+software.
+
+The repository contains the frozen machine-readable evidence underlying that
+study, including corpus records, restoration evidence, study metadata,
+generated accounting tables, and provenance information.
+
+The empirical corpus is frozen and is not expanded or modified in response to
+observed mutation outcomes.
+
+The software release and frozen empirical artifacts are archived on Zenodo:
+
+**MLReproMutate v0.1.0**
+DOI: https://doi.org/10.5281/zenodo.22126120
+
+The accompanying empirical study is available as a preprint:
+
+**Ilya Shulepov. _Mutation Testing for Reproducibility Safeguards in Machine
+Learning Research Software: An Empirical Study._ arXiv:2608.27100, 2026.**
+
+https://arxiv.org/abs/2608.27100
+
+## Development
+
+Install the development environment:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+Run the test suite:
+
+```bash
+python -m pytest -q
+```
+
+Run static checks:
+
+```bash
+ruff check src tests
+```
+
+The continuous-integration workflow runs the software checks across supported
+Python versions.
+
+## Reporting problems and contributing
+
+Bug reports, usability feedback, research use cases, documentation
+improvements, and suggestions for reproducibility mutation operators are
+welcome.
+
+Please use the GitHub issue tracker for reproducible bugs or usability
+problems:
+
+https://github.com/ilyuka/MLReproMutate/issues
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development and contribution
+guidelines.
+
+## Citation
+
+Citation metadata is provided in [CITATION.cff](CITATION.cff).
+
+The archived software release can be cited using:
+
+> Shulepov, Ilya. MLReproMutate, version 0.1.0. Zenodo, 2026.
+> https://doi.org/10.5281/zenodo.22126120
+
+## License
+
+MLReproMutate is released under the [MIT License](LICENSE).