llm-code-validator 0.1.0__tar.gz → 0.1.1__tar.gz

llm_code_validator-0.1.1/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.4
+Name: llm-code-validator
+Version: 0.1.1
+Summary: CLI guardrail for catching stale Python APIs before runtime.
+Author: Felix Mathew
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/mathew-felix/llm-code-validator
+Project-URL: Repository, https://github.com/mathew-felix/llm-code-validator
+Project-URL: Issues, https://github.com/mathew-felix/llm-code-validator/issues
+Project-URL: PyPI, https://pypi.org/project/llm-code-validator/
+Keywords: api-drift,static-analysis,llm,python,ci
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# llm-code-validator
+
+Python CLI for checking dependency-heavy Python projects for stale or version-incompatible third-party API usage before commit or CI.
+
+It parses Python files with `ast`, checks imports and calls against a maintained API-drift rule database, and reports issues before runtime.
+
+Default checks are local-only. No OpenAI, Anthropic, or other LLM API key is required, and the tool does not make network calls in normal use.
+
+Current local validation: 74 tests passing, 68 API-drift rules, and PyPI install verified.
+
+PyPI: https://pypi.org/project/llm-code-validator/
+
+![Terminal demo showing API drift diagnostics and safe fix preview](docs/demo.gif)
+
+## Install
+
+```bash
+pip install llm-code-validator
+```
+
+For local development:
+
+```bash
+git clone https://github.com/mathew-felix/llm-code-validator
+cd llm-code-validator
+pip install -e ".[dev]"
+```
+
+## Quick Use
+
+```bash
+llm-code-validator check file.py
+llm-code-validator check src/
+llm-code-validator check --staged
+llm-code-validator check src/ --format json
+llm-code-validator check src/ --format github
+```
+
+Exit codes:
+
+- `0`: no diagnostics
+- `1`: diagnostics found
+- `2`: tool error
+
+## Example
+
+```python
+from sqlalchemy.ext.declarative import declarative_base
+
+Base = declarative_base()
+```
+
+```bash
+llm-code-validator check app.py
+```
+
+```text
+app.py:1 LCV001 warning sqlalchemy.declarative_base sqlalchemy.declarative_base is incompatible with sqlalchemy>=2.0.0
+  fix: from sqlalchemy.orm import declarative_base
+```
+
+Preview or apply safe fixes:
+
+```bash
+llm-code-validator fix app.py
+llm-code-validator fix app.py --write
+```
+
+## What It Checks
+
+Current rule database:
+
+- 68 API-drift rules
+- 15 safe fixes
+- Rules for OpenAI, Anthropic, LangChain, LangGraph, LlamaIndex, Pinecone, ChromaDB, FastAPI, Pydantic, pandas, NumPy, SQLAlchemy, Torch, and Transformers
+
+Validate the rule database:
+
+```bash
+llm-code-validator validate-signatures
+```
+
+This checks source-level API migration patterns. It does not replace Ruff for linting, mypy for type checking, pip-audit for vulnerability checks, or Dependabot for dependency updates.
+
+## Security Model
+
+By default, `llm-code-validator` reads local Python files, parses them with Python's built-in `ast` module, and compares imports and calls with the bundled rule database. It does not send source code, dependency files, environment variables, or secrets to any external service.
+
+If optional AI-assisted review is added in the future, it should remain explicit opt-in and should minimize and redact any code snippets before a provider request.
+
+## Rule Maintenance
+
+Public rules are reviewed before release. New rules should be added to `data/library_signatures.json`, backed by official evidence such as migration guides, release notes, official docs, or maintainer discussions, and covered by a test or benchmark case.
+
+The packaged PyPI wheel includes `llm_code_validator/library_signatures.json`, so users receive reviewed rule updates by upgrading the package:
+
+```bash
+pip install --upgrade llm-code-validator
+```
+
+Use `docs/rules.md` for the contribution workflow and `docs/release.md` for release verification.
+
+## Limitations
+
+- Detects known API-drift rules only.
+- Does not detect every possible Python, dependency, security, or runtime issue.
+- Does not prove full program correctness.
+- Complex dynamic imports may be missed.
+- Dependency checks depend on available project metadata.
+- Suggested fixes require review before applying.
+- External repository findings are treated as candidates until manually reviewed.
+
+## Integrations
+
+Pre-commit:
+
+```yaml
+repos:
+  - repo: https://github.com/mathew-felix/llm-code-validator
+    rev: v0.1.0
+    hooks:
+      - id: llm-code-validator
+```
+
+GitHub Actions:
+
+```yaml
+- run: pip install llm-code-validator
+- run: llm-code-validator check . --format github
+```
+
+## Development
+
+Run tests:
+
+```bash
+pytest -q
+```
+
+Current local result:
+
+```text
+74 passed
+```
+
+Run benchmarks:
+
+```bash
+python -m llm_code_validator.benchmark --dataset validation_dataset/cli_benchmark_cases.json
+python -m llm_code_validator.benchmark --dataset validation_dataset/ai_stack_benchmark_cases.json
+```
+
+## More Details
+
+- `docs/demo.md`: command walkthrough
+- `docs/accuracy.md`: benchmark and external-review notes
+- `docs/rules.md`: rule database notes
+- `docs/security.md`: local-only, AI-review, and policy controls
+- `docs/ai-review.md`: optional AI-review roadmap and candidate-rule workflow
+- `docs/release.md`: release steps

llm_code_validator-0.1.1/README.md

@@ -0,0 +1,160 @@
+# llm-code-validator
+
+Python CLI for checking dependency-heavy Python projects for stale or version-incompatible third-party API usage before commit or CI.
+
+It parses Python files with `ast`, checks imports and calls against a maintained API-drift rule database, and reports issues before runtime.
+
+Default checks are local-only. No OpenAI, Anthropic, or other LLM API key is required, and the tool does not make network calls in normal use.
+
+Current local validation: 74 tests passing, 68 API-drift rules, and PyPI install verified.
+
+PyPI: https://pypi.org/project/llm-code-validator/
+
+![Terminal demo showing API drift diagnostics and safe fix preview](docs/demo.gif)
+
+## Install
+
+```bash
+pip install llm-code-validator
+```
+
+For local development:
+
+```bash
+git clone https://github.com/mathew-felix/llm-code-validator
+cd llm-code-validator
+pip install -e ".[dev]"
+```
+
+## Quick Use
+
+```bash
+llm-code-validator check file.py
+llm-code-validator check src/
+llm-code-validator check --staged
+llm-code-validator check src/ --format json
+llm-code-validator check src/ --format github
+```
+
+Exit codes:
+
+- `0`: no diagnostics
+- `1`: diagnostics found
+- `2`: tool error
+
+## Example
+
+```python
+from sqlalchemy.ext.declarative import declarative_base
+
+Base = declarative_base()
+```
+
+```bash
+llm-code-validator check app.py
+```
+
+```text
+app.py:1 LCV001 warning sqlalchemy.declarative_base sqlalchemy.declarative_base is incompatible with sqlalchemy>=2.0.0
+  fix: from sqlalchemy.orm import declarative_base
+```
+
+Preview or apply safe fixes:
+
+```bash
+llm-code-validator fix app.py
+llm-code-validator fix app.py --write
+```
+
+## What It Checks
+
+Current rule database:
+
+- 68 API-drift rules
+- 15 safe fixes
+- Rules for OpenAI, Anthropic, LangChain, LangGraph, LlamaIndex, Pinecone, ChromaDB, FastAPI, Pydantic, pandas, NumPy, SQLAlchemy, Torch, and Transformers
+
+Validate the rule database:
+
+```bash
+llm-code-validator validate-signatures
+```
+
+This checks source-level API migration patterns. It does not replace Ruff for linting, mypy for type checking, pip-audit for vulnerability checks, or Dependabot for dependency updates.
+
+## Security Model
+
+By default, `llm-code-validator` reads local Python files, parses them with Python's built-in `ast` module, and compares imports and calls with the bundled rule database. It does not send source code, dependency files, environment variables, or secrets to any external service.
+
+If optional AI-assisted review is added in the future, it should remain explicit opt-in and should minimize and redact any code snippets before a provider request.
+
+## Rule Maintenance
+
+Public rules are reviewed before release. New rules should be added to `data/library_signatures.json`, backed by official evidence such as migration guides, release notes, official docs, or maintainer discussions, and covered by a test or benchmark case.
+
+The packaged PyPI wheel includes `llm_code_validator/library_signatures.json`, so users receive reviewed rule updates by upgrading the package:
+
+```bash
+pip install --upgrade llm-code-validator
+```
+
+Use `docs/rules.md` for the contribution workflow and `docs/release.md` for release verification.
+
+## Limitations
+
+- Detects known API-drift rules only.
+- Does not detect every possible Python, dependency, security, or runtime issue.
+- Does not prove full program correctness.
+- Complex dynamic imports may be missed.
+- Dependency checks depend on available project metadata.
+- Suggested fixes require review before applying.
+- External repository findings are treated as candidates until manually reviewed.
+
+## Integrations
+
+Pre-commit:
+
+```yaml
+repos:
+  - repo: https://github.com/mathew-felix/llm-code-validator
+    rev: v0.1.0
+    hooks:
+      - id: llm-code-validator
+```
+
+GitHub Actions:
+
+```yaml
+- run: pip install llm-code-validator
+- run: llm-code-validator check . --format github
+```
+
+## Development
+
+Run tests:
+
+```bash
+pytest -q
+```
+
+Current local result:
+
+```text
+74 passed
+```
+
+Run benchmarks:
+
+```bash
+python -m llm_code_validator.benchmark --dataset validation_dataset/cli_benchmark_cases.json
+python -m llm_code_validator.benchmark --dataset validation_dataset/ai_stack_benchmark_cases.json
+```
+
+## More Details
+
+- `docs/demo.md`: command walkthrough
+- `docs/accuracy.md`: benchmark and external-review notes
+- `docs/rules.md`: rule database notes
+- `docs/security.md`: local-only, AI-review, and policy controls
+- `docs/ai-review.md`: optional AI-review roadmap and candidate-rule workflow
+- `docs/release.md`: release steps

{llm_code_validator-0.1.0 → llm_code_validator-0.1.1}/llm_code_validator/__init__.py

@@ -1,3 +1,3 @@
 """Deterministic API-drift checker for Python source code."""
 
-__version__ = "0.1.0"
+__version__ = "0.1.1"

llm_code_validator-0.1.1/llm_code_validator/ai_review.py

@@ -0,0 +1,153 @@
+from __future__ import annotations
+
+import json
+import os
+import re
+from datetime import datetime, timezone
+from dataclasses import dataclass
+from pathlib import Path
+
+from .core import EXCLUDED_DIR_NAMES, iter_python_files
+
+
+SECRET_FILE_NAMES = {
+    ".env",
+    ".env.local",
+    ".env.production",
+    "id_rsa",
+    "id_dsa",
+    "id_ecdsa",
+    "id_ed25519",
+}
+SECRET_NAME_PARTS = {"secret", "secrets", "credential", "credentials", "token", "private-key"}
+
+SECRET_PATTERNS = [
+    re.compile(r"sk-[A-Za-z0-9_-]{12,}"),
+    re.compile(r"(?i)(api[_-]?key|secret|token|password)\s*=\s*['\"][^'\"]+['\"]"),
+    re.compile(r"(?i)(authorization:\s*bearer\s+)[A-Za-z0-9._~+/=-]+"),
+]
+
+
+@dataclass(frozen=True)
+class ProviderConfig:
+    provider: str
+    api_key_env: str | None
+    endpoint: str | None = None
+
+    @property
+    def configured(self) -> bool:
+        if self.provider == "local":
+            return bool(self.endpoint)
+        return bool(self.api_key_env and os.getenv(self.api_key_env))
+
+
+def default_key_env(provider: str) -> str | None:
+    if provider == "openai":
+        return "OPENAI_API_KEY"
+    if provider == "anthropic":
+        return "ANTHROPIC_API_KEY"
+    if provider == "azure-openai":
+        return "AZURE_OPENAI_API_KEY"
+    return None
+
+
+def redact_secrets(text: str) -> str:
+    redacted = text
+    for pattern in SECRET_PATTERNS:
+        redacted = pattern.sub(_redaction, redacted)
+    return redacted
+
+
+def _redaction(match: re.Match[str]) -> str:
+    value = match.group(0)
+    if value.lower().startswith("authorization:"):
+        return f"{match.group(1)}[REDACTED]"
+    if "=" in value:
+        name = value.split("=", 1)[0].strip()
+        return f"{name} = \"[REDACTED]\""
+    return "[REDACTED]"
+
+
+def is_secret_path(path: Path) -> bool:
+    lowered = {part.lower() for part in path.parts}
+    if lowered.intersection(EXCLUDED_DIR_NAMES):
+        return True
+    if path.name.lower() in SECRET_FILE_NAMES:
+        return True
+    return bool(lowered.intersection(SECRET_NAME_PARTS))
+
+
+def _extract_relevant_lines(source: str, max_snippet_lines: int) -> list[str]:
+    relevant: list[str] = []
+    for line in source.splitlines():
+        stripped = line.strip()
+        if (
+            stripped.startswith("import ")
+            or stripped.startswith("from ")
+            or any(name in stripped.lower() for name in ("key", "secret", "token", "password", "authorization"))
+            or "(" in stripped
+            or "." in stripped
+            or "@" in stripped
+        ):
+            relevant.append(line)
+        if len(relevant) >= max_snippet_lines:
+            break
+    return relevant
+
+
+def build_ai_payload(
+    paths: list[str],
+    *,
+    max_snippet_lines: int = 30,
+    redact: bool = True,
+) -> dict[str, object]:
+    files = []
+    for file_path in iter_python_files(paths):
+        if is_secret_path(file_path):
+            continue
+        try:
+            source = file_path.read_text(encoding="utf-8")
+        except UnicodeDecodeError:
+            source = file_path.read_text(encoding="utf-8", errors="replace")
+        snippet = "\n".join(_extract_relevant_lines(source, max_snippet_lines))
+        if redact:
+            snippet = redact_secrets(snippet)
+        files.append({"path": str(file_path), "snippet": snippet})
+    return {
+        "purpose": "advisory API-drift review",
+        "files": files,
+        "instructions": (
+            "Review only the provided minimized snippets for stale third-party API usage. "
+            "Return advisory findings and candidate rules; do not assume full program context."
+        ),
+    }
+
+
+def render_ai_payload(payload: dict[str, object]) -> str:
+    return json.dumps(payload, indent=2, sort_keys=True)
+
+
+def write_ai_audit_log(path: str | Path, provider: str, payload: dict[str, object]) -> None:
+    files = payload.get("files", [])
+    file_count = len(files) if isinstance(files, list) else 0
+    record = {
+        "timestamp": datetime.now(timezone.utc).isoformat(),
+        "provider": provider,
+        "file_count": file_count,
+        "payload_type": payload.get("purpose", "advisory API-drift review"),
+        "contains_source_snippets": False,
+    }
+    target = Path(path)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    with target.open("a", encoding="utf-8") as handle:
+        handle.write(json.dumps(record, sort_keys=True) + "\n")
+
+
+def validate_ai_provider(config: ProviderConfig, no_network: bool) -> None:
+    if no_network:
+        raise RuntimeError("--no-network prevents AI review provider calls")
+    if not config.configured:
+        if config.provider == "local":
+            raise RuntimeError("--ai-provider local requires --ai-endpoint")
+        env_name = config.api_key_env or "provider API key environment variable"
+        raise RuntimeError(f"--ai-review requires {env_name} to be set")