llm-code-validator 0.1.0__tar.gz

llm_code_validator-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Felix Mathew
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_code_validator-0.1.0/PKG-INFO

@@ -0,0 +1,220 @@
+Metadata-Version: 2.4
+Name: llm-code-validator
+Version: 0.1.0
+Summary: CLI guardrail for catching stale Python APIs before runtime.
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# llm-code-validator
+
+Python CLI for detecting stale or version-incompatible third-party API usage in Python source code.
+
+## Installation
+
+Install from the repository:
+
+```bash
+git clone https://github.com/mathew-felix/llm-code-validator
+cd llm-code-validator
+pip install -e .
+```
+
+Install with test dependencies:
+
+```bash
+pip install -e ".[dev]"
+```
+
+After the package is published to PyPI:
+
+```bash
+pip install llm-code-validator
+```
+
+## Usage
+
+Check one file:
+
+```bash
+llm-code-validator check file.py
+```
+
+Check a directory:
+
+```bash
+llm-code-validator check src/
+```
+
+Check standard input:
+
+```bash
+llm-code-validator check - < snippet.py
+```
+
+Check staged Git files:
+
+```bash
+llm-code-validator check --staged
+```
+
+Use an explicit dependency file:
+
+```bash
+llm-code-validator check --requirements requirements.txt src/
+```
+
+Show low-confidence diagnostics:
+
+```bash
+llm-code-validator check --show-low-confidence src/
+```
+
+Exit codes:
+
+- `0`: no diagnostics
+- `1`: diagnostics found
+- `2`: tool error
+
+## Output
+
+Text output:
+
+```bash
+llm-code-validator check src/
+```
+
+JSON output:
+
+```bash
+llm-code-validator check src/ --format json
+```
+
+GitHub Actions annotation output:
+
+```bash
+llm-code-validator check src/ --format github
+```
+
+## Fixes
+
+Preview fixes:
+
+```bash
+llm-code-validator fix file.py
+```
+
+Apply safe fixes:
+
+```bash
+llm-code-validator fix file.py --write
+```
+
+Only rules marked `safe_fix` are written. Rules marked `suggested_fix` or `no_fix` are reported but not changed.
+
+Current rule safety counts:
+
+- `safe_fix`: 15 rules
+- `suggested_fix`: 51 rules
+- `no_fix`: 2 rules
+
+## Signature Database
+
+Validate the rule database:
+
+```bash
+llm-code-validator validate-signatures
+```
+
+The source rule database is:
+
+```text
+data/library_signatures.json
+```
+
+The packaged rule database is:
+
+```text
+llm_code_validator/library_signatures.json
+```
+
+Current rule count:
+
+- 68 API-drift rules
+
+## Benchmarks
+
+Run the CLI benchmark dataset:
+
+```bash
+python -m llm_code_validator.benchmark --dataset validation_dataset/cli_benchmark_cases.json
+```
+
+Run the AI-stack benchmark dataset:
+
+```bash
+python -m llm_code_validator.benchmark --dataset validation_dataset/ai_stack_benchmark_cases.json
+```
+
+Current saved benchmark results:
+
+- CLI dataset: precision `1.0`, recall `1.0`, p50 `0.243ms`, p95 `6.199ms`
+- AI-stack dataset: precision `1.0`, recall `1.0`, p50 `0.444ms`, p95 `4.939ms`
+
+## Pre-Commit
+
+`.pre-commit-hooks.yaml` is included:
+
+```yaml
+repos:
+  - repo: https://github.com/mathew-felix/llm-code-validator
+    rev: v0.1.0
+    hooks:
+      - id: llm-code-validator
+```
+
+## GitHub Actions
+
+Example workflow:
+
+```yaml
+name: API Drift Check
+
+on:
+  pull_request:
+
+jobs:
+  api-drift:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install llm-code-validator
+      - run: llm-code-validator check . --format github
+```
+
+## Testing
+
+Run the test suite:
+
+```bash
+pytest -q
+```
+
+Current local result:
+
+```text
+72 passed
+```
+
+## Documentation
+
+- `docs/demo.md`: example check and fix workflow
+- `docs/rules.md`: rule database notes
+- `docs/release.md`: package release steps
+- `PROJECT_REPORT.md`: project report

llm_code_validator-0.1.0/README.md

@@ -0,0 +1,209 @@
+# llm-code-validator
+
+Python CLI for detecting stale or version-incompatible third-party API usage in Python source code.
+
+## Installation
+
+Install from the repository:
+
+```bash
+git clone https://github.com/mathew-felix/llm-code-validator
+cd llm-code-validator
+pip install -e .
+```
+
+Install with test dependencies:
+
+```bash
+pip install -e ".[dev]"
+```
+
+After the package is published to PyPI:
+
+```bash
+pip install llm-code-validator
+```
+
+## Usage
+
+Check one file:
+
+```bash
+llm-code-validator check file.py
+```
+
+Check a directory:
+
+```bash
+llm-code-validator check src/
+```
+
+Check standard input:
+
+```bash
+llm-code-validator check - < snippet.py
+```
+
+Check staged Git files:
+
+```bash
+llm-code-validator check --staged
+```
+
+Use an explicit dependency file:
+
+```bash
+llm-code-validator check --requirements requirements.txt src/
+```
+
+Show low-confidence diagnostics:
+
+```bash
+llm-code-validator check --show-low-confidence src/
+```
+
+Exit codes:
+
+- `0`: no diagnostics
+- `1`: diagnostics found
+- `2`: tool error
+
+## Output
+
+Text output:
+
+```bash
+llm-code-validator check src/
+```
+
+JSON output:
+
+```bash
+llm-code-validator check src/ --format json
+```
+
+GitHub Actions annotation output:
+
+```bash
+llm-code-validator check src/ --format github
+```
+
+## Fixes
+
+Preview fixes:
+
+```bash
+llm-code-validator fix file.py
+```
+
+Apply safe fixes:
+
+```bash
+llm-code-validator fix file.py --write
+```
+
+Only rules marked `safe_fix` are written. Rules marked `suggested_fix` or `no_fix` are reported but not changed.
+
+Current rule safety counts:
+
+- `safe_fix`: 15 rules
+- `suggested_fix`: 51 rules
+- `no_fix`: 2 rules
+
+## Signature Database
+
+Validate the rule database:
+
+```bash
+llm-code-validator validate-signatures
+```
+
+The source rule database is:
+
+```text
+data/library_signatures.json
+```
+
+The packaged rule database is:
+
+```text
+llm_code_validator/library_signatures.json
+```
+
+Current rule count:
+
+- 68 API-drift rules
+
+## Benchmarks
+
+Run the CLI benchmark dataset:
+
+```bash
+python -m llm_code_validator.benchmark --dataset validation_dataset/cli_benchmark_cases.json
+```
+
+Run the AI-stack benchmark dataset:
+
+```bash
+python -m llm_code_validator.benchmark --dataset validation_dataset/ai_stack_benchmark_cases.json
+```
+
+Current saved benchmark results:
+
+- CLI dataset: precision `1.0`, recall `1.0`, p50 `0.243ms`, p95 `6.199ms`
+- AI-stack dataset: precision `1.0`, recall `1.0`, p50 `0.444ms`, p95 `4.939ms`
+
+## Pre-Commit
+
+`.pre-commit-hooks.yaml` is included:
+
+```yaml
+repos:
+  - repo: https://github.com/mathew-felix/llm-code-validator
+    rev: v0.1.0
+    hooks:
+      - id: llm-code-validator
+```
+
+## GitHub Actions
+
+Example workflow:
+
+```yaml
+name: API Drift Check
+
+on:
+  pull_request:
+
+jobs:
+  api-drift:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install llm-code-validator
+      - run: llm-code-validator check . --format github
+```
+
+## Testing
+
+Run the test suite:
+
+```bash
+pytest -q
+```
+
+Current local result:
+
+```text
+72 passed
+```
+
+## Documentation
+
+- `docs/demo.md`: example check and fix workflow
+- `docs/rules.md`: rule database notes
+- `docs/release.md`: package release steps
+- `PROJECT_REPORT.md`: project report

llm_code_validator-0.1.0/llm_code_validator/__init__.py

@@ -0,0 +1,3 @@
+"""Deterministic API-drift checker for Python source code."""
+
+__version__ = "0.1.0"

llm_code_validator-0.1.0/llm_code_validator/benchmark.py

@@ -0,0 +1,141 @@
+from __future__ import annotations
+
+import argparse
+import json
+import platform
+import statistics
+import time
+import tracemalloc
+from datetime import date
+from pathlib import Path
+
+from .core import check_file, check_source, iter_python_files
+from .versioning import build_version_context
+
+
+def run_benchmark(paths: list[str]) -> dict[str, object]:
+    files = iter_python_files(paths)
+    version_context = build_version_context(paths)
+    timings: list[float] = []
+    diagnostics = 0
+    tracemalloc.start()
+    start = time.perf_counter()
+    for path in files:
+        file_start = time.perf_counter()
+        result = check_file(path, version_context)
+        timings.append(time.perf_counter() - file_start)
+        diagnostics += len(result.diagnostics)
+    total = time.perf_counter() - start
+    _, peak = tracemalloc.get_traced_memory()
+    tracemalloc.stop()
+
+    p50 = statistics.median(timings) if timings else 0.0
+    p95 = statistics.quantiles(timings, n=20)[18] if len(timings) >= 20 else (max(timings) if timings else 0.0)
+    files_per_second = len(files) / total if total else 0.0
+    return {
+        "files": len(files),
+        "diagnostics": diagnostics,
+        "total_seconds": total,
+        "p50_ms": p50 * 1000,
+        "p95_ms": p95 * 1000,
+        "files_per_second": files_per_second,
+        "peak_ram_mb": peak / (1024 * 1024),
+        "hardware": platform.machine(),
+        "os": platform.platform(),
+        "python_version": platform.python_version(),
+        "precision": None,
+        "recall": None,
+        "false_positives": None,
+        "false_negatives": None,
+    }
+
+
+def run_labeled_benchmark(dataset_path: str | Path) -> dict[str, object]:
+    dataset_file = Path(dataset_path)
+    cases = json.loads(dataset_file.read_text(encoding="utf-8"))
+    timings: list[float] = []
+    true_positives = 0
+    false_positives = 0
+    false_negatives = 0
+    false_positive_examples: list[dict[str, str]] = []
+    false_negative_examples: list[dict[str, str]] = []
+    total_expected = 0
+    total_diagnostics = 0
+
+    tracemalloc.start()
+    start = time.perf_counter()
+    for case in cases:
+        case_start = time.perf_counter()
+        result = check_source(case["code"], case.get("path") or f"{case['id']}.py")
+        timings.append(time.perf_counter() - case_start)
+        expected = {(item["library"], item["symbol"]) for item in case.get("expected_diagnostics", [])}
+        actual = {(diagnostic.library, diagnostic.symbol) for diagnostic in result.diagnostics}
+        total_expected += len(expected)
+        total_diagnostics += len(actual)
+        true_positives += len(expected & actual)
+        case_false_positives = actual - expected
+        case_false_negatives = expected - actual
+        false_positives += len(case_false_positives)
+        false_negatives += len(case_false_negatives)
+        for library, symbol in sorted(case_false_positives):
+            false_positive_examples.append(
+                {"case_id": case["id"], "library": library, "symbol": symbol, "reason": "unexpected diagnostic"}
+            )
+        for library, symbol in sorted(case_false_negatives):
+            false_negative_examples.append(
+                {"case_id": case["id"], "library": library, "symbol": symbol, "reason": "missing rule or extraction gap"}
+            )
+    total = time.perf_counter() - start
+    _, peak = tracemalloc.get_traced_memory()
+    tracemalloc.stop()
+
+    precision = true_positives / (true_positives + false_positives) if true_positives + false_positives else 1.0
+    recall = true_positives / (true_positives + false_negatives) if true_positives + false_negatives else 1.0
+    p50 = statistics.median(timings) if timings else 0.0
+    p95 = statistics.quantiles(timings, n=20)[18] if len(timings) >= 20 else (max(timings) if timings else 0.0)
+    return {
+        "dataset": str(dataset_file),
+        "benchmark_date": date.today().isoformat(),
+        "cases": len(cases),
+        "files": len(cases),
+        "diagnostics": total_diagnostics,
+        "expected_diagnostics": total_expected,
+        "true_positives": true_positives,
+        "false_positives": false_positives,
+        "false_negatives": false_negatives,
+        "false_positive_examples": false_positive_examples,
+        "false_negative_examples": false_negative_examples,
+        "precision": precision,
+        "recall": recall,
+        "total_seconds": total,
+        "p50_ms": p50 * 1000,
+        "p95_ms": p95 * 1000,
+        "files_per_second": len(cases) / total if total else 0.0,
+        "peak_ram_mb": peak / (1024 * 1024),
+        "hardware": platform.machine(),
+        "os": platform.platform(),
+        "python_version": platform.python_version(),
+    }
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="python -m llm_code_validator.benchmark")
+    parser.add_argument("paths", nargs="*")
+    parser.add_argument("--dataset", help="Run a labeled benchmark dataset JSON file.")
+    parser.add_argument("--output", help="Write JSON benchmark output to a file.")
+    args = parser.parse_args(argv)
+    if args.dataset:
+        payload = run_labeled_benchmark(args.dataset)
+    elif args.paths:
+        payload = run_benchmark(args.paths)
+    else:
+        parser.error("provide one or more paths or --dataset")
+    output = json.dumps(payload, indent=2, sort_keys=True)
+    if args.output:
+        Path(args.output).write_text(output + "\n", encoding="utf-8")
+    print(output)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

llm_code_validator-0.1.0/llm_code_validator/cli.py

@@ -0,0 +1,105 @@
+from __future__ import annotations
+
+import argparse
+import sys
+
+from .core import CheckResult, check_paths, check_stdin, staged_python_files
+from .fixes import fix_file
+from .formatting import format_github, format_json, format_text
+from .signatures import validate_signature_database
+from .versioning import build_version_context
+
+
+def check_staged(requirements: str | None = None, python_version: str | None = None) -> CheckResult:
+    files = staged_python_files()
+    return check_paths(files, requirements=requirements, python_version=python_version) if files else check_paths([])
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="llm-code-validator")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    check = subparsers.add_parser("check", help="Check Python files for known API drift.")
+    check.add_argument("paths", nargs="*", help="Files or directories to scan. Use '-' for stdin.")
+    check.add_argument("--staged", action="store_true", help="Check staged Python files from git.")
+    check.add_argument("--format", choices=["text", "json", "github"], default="text")
+    check.add_argument("--requirements", help="Requirements file used for version assumptions.")
+    check.add_argument("--python-version", help="Target Python version label for result context.")
+    check.add_argument("--show-low-confidence", action="store_true", help="Show lower-confidence diagnostics.")
+
+    fix = subparsers.add_parser("fix", help="Preview or apply deterministic safe fixes.")
+    fix.add_argument("paths", nargs="+", help="Python files to fix.")
+    fix.add_argument("--write", action="store_true", help="Write safe fixes to disk.")
+    fix.add_argument("--requirements", help="Requirements file used for version assumptions.")
+    fix.add_argument("--python-version", help="Target Python version label for result context.")
+
+    validate = subparsers.add_parser("validate-signatures", help="Validate the signature database.")
+    validate.add_argument("--path", help="Path to library_signatures.json.")
+    validate.add_argument(
+        "--require-official-evidence",
+        action="store_true",
+        help="Require diagnostic rules to use source_url or release_note instead of generic notes.",
+    )
+    return parser
+
+
+def _render(result: CheckResult, output_format: str) -> str:
+    if output_format == "json":
+        return format_json(result)
+    if output_format == "github":
+        return format_github(result)
+    return format_text(result)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+
+    try:
+        if args.command == "check":
+            if args.staged:
+                result = check_paths(
+                    staged_python_files(),
+                    requirements=args.requirements,
+                    python_version=args.python_version,
+                    show_low_confidence=args.show_low_confidence,
+                )
+            elif args.paths == ["-"]:
+                result = check_stdin(args.requirements, args.python_version, args.show_low_confidence)
+            elif args.paths:
+                result = check_paths(args.paths, args.requirements, args.python_version, args.show_low_confidence)
+            else:
+                parser.error("check requires a path, '-', or --staged")
+            output = _render(result, args.format)
+            if output:
+                print(output)
+            return 1 if result.diagnostics else 0
+        if args.command == "fix":
+            version_context = build_version_context(args.paths, args.requirements, args.python_version)
+            exit_code = 0
+            for path in args.paths:
+                result = fix_file(path, write=args.write, version_context=version_context)
+                for preview in result.previews:
+                    print(preview)
+                for skipped in result.skipped:
+                    print(skipped)
+                if result.skipped:
+                    exit_code = 1
+            return exit_code
+        if args.command == "validate-signatures":
+            errors = validate_signature_database(args.path, args.require_official_evidence)
+            if errors:
+                for error in errors:
+                    print(error, file=sys.stderr)
+                return 1
+            print("OK: signature database is valid")
+            return 0
+    except Exception as exc:
+        print(f"llm-code-validator: {exc}", file=sys.stderr)
+        return 2
+
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())