python-harness 0.0.1__tar.gz

python_harness-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mingli Yuan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

python_harness-0.0.1/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.4
+Name: python-harness
+Version: 0.0.1
+Summary: An agentic codebase evaluation and evolution tool for Python projects.
+Author-email: Mingli Yuan <mingli.yuan@gmail.com>
+License: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer>=0.9.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: anthropic>=0.18.0
+Requires-Dist: tenacity>=8.2.0
+Requires-Dist: tiktoken>=0.6.0
+Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: ruff>=0.3.0; extra == "dev"
+Requires-Dist: mypy>=1.9.0; extra == "dev"
+Requires-Dist: ty>=0.0.1; extra == "dev"
+Requires-Dist: radon>=6.0.1; extra == "dev"
+Dynamic: license-file
+
+# Python Harness
+
+An agentic codebase evaluation and evolution tool for Python projects.
+
+`python-harness` is designed to be a universal standard tool—just like `pytest` or `ruff`—but instead of just checking syntax or running tests, it evaluates the **architecture, readability, and governance** of your codebase using both static analysis and LLMs (DeepSeek/OpenAI).
+
+## Features
+
+1. **Hard Evaluation (First Fence)**: Enforces strict rules using `ruff`, `mypy`, and `ty`. Evaluates Cyclomatic Complexity (CC) and Maintainability Index (MI) via `radon`.
+2. **Governance QC (Second Fence)**: Checks if the changes violate core project governance or attempt to bypass the evaluation rules themselves.
+3. **Soft Evaluation (Third Fence)**:
+   - Calculates architecture metrics like Fan-out (coupling).
+   - Generates a holistic package understanding using LLMs.
+   - Performs "Blind QA": Randomly samples functions/classes and tests the LLM's ability to understand them without context.
+4. **Actionable Output**: Synthesizes the evaluation into a final `Pass/Fail` verdict with exactly 3 concrete, actionable refactoring suggestions.
+
+## Installation
+
+You can install `python-harness` using `uv` or `pip`:
+
+```bash
+uv pip install python-harness
+```
+
+## Configuration
+
+`python-harness` requires an LLM to perform its soft evaluation. Create a `.env` file in the root of your project:
+
+```env
+LLM_API_KEY=your_api_key_here
+LLM_BASE_URL=https://api.deepseek.com/v1
+LLM_MODEL_NAME=deepseek-reasoner
+LLM_MINI_MODEL_NAME=deepseek-chat
+```
+
+*(Note: If you don't provide an API key, the harness will safely run in Mock mode).*
+
+## Usage
+
+### 1. Measure
+
+To evaluate your codebase, run the `measure` command in your project directory:
+
+```bash
+harness measure .
+```
+
+This will run the full 3-fence evaluation and output a report with a final verdict and top 3 improvement suggestions.
+
+### 2. Refine (Evolution Loop - WIP)
+
+The `refine` command is an Agentic Edit-Test-Improve loop. It takes the suggestions generated by `measure`, automatically creates branches (variants), applies the changes, runs the tests (`pytest`), and picks the best variant.
+
+```bash
+harness refine . --steps 1 --max-retries 3
+```
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for more details.
+
+A harness toolkit for Python projects

python_harness-0.0.1/README.md

@@ -0,0 +1,62 @@
+# Python Harness
+
+An agentic codebase evaluation and evolution tool for Python projects.
+
+`python-harness` is designed to be a universal standard tool—just like `pytest` or `ruff`—but instead of just checking syntax or running tests, it evaluates the **architecture, readability, and governance** of your codebase using both static analysis and LLMs (DeepSeek/OpenAI).
+
+## Features
+
+1. **Hard Evaluation (First Fence)**: Enforces strict rules using `ruff`, `mypy`, and `ty`. Evaluates Cyclomatic Complexity (CC) and Maintainability Index (MI) via `radon`.
+2. **Governance QC (Second Fence)**: Checks if the changes violate core project governance or attempt to bypass the evaluation rules themselves.
+3. **Soft Evaluation (Third Fence)**:
+   - Calculates architecture metrics like Fan-out (coupling).
+   - Generates a holistic package understanding using LLMs.
+   - Performs "Blind QA": Randomly samples functions/classes and tests the LLM's ability to understand them without context.
+4. **Actionable Output**: Synthesizes the evaluation into a final `Pass/Fail` verdict with exactly 3 concrete, actionable refactoring suggestions.
+
+## Installation
+
+You can install `python-harness` using `uv` or `pip`:
+
+```bash
+uv pip install python-harness
+```
+
+## Configuration
+
+`python-harness` requires an LLM to perform its soft evaluation. Create a `.env` file in the root of your project:
+
+```env
+LLM_API_KEY=your_api_key_here
+LLM_BASE_URL=https://api.deepseek.com/v1
+LLM_MODEL_NAME=deepseek-reasoner
+LLM_MINI_MODEL_NAME=deepseek-chat
+```
+
+*(Note: If you don't provide an API key, the harness will safely run in Mock mode).*
+
+## Usage
+
+### 1. Measure
+
+To evaluate your codebase, run the `measure` command in your project directory:
+
+```bash
+harness measure .
+```
+
+This will run the full 3-fence evaluation and output a report with a final verdict and top 3 improvement suggestions.
+
+### 2. Refine (Evolution Loop - WIP)
+
+The `refine` command is an Agentic Edit-Test-Improve loop. It takes the suggestions generated by `measure`, automatically creates branches (variants), applies the changes, runs the tests (`pytest`), and picks the best variant.
+
+```bash
+harness refine . --steps 1 --max-retries 3
+```
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for more details.
+
+A harness toolkit for Python projects

python_harness-0.0.1/pyproject.toml

@@ -0,0 +1,65 @@
+[project]
+name = "python-harness"
+version = "0.0.1"
+description = "An agentic codebase evaluation and evolution tool for Python projects."
+requires-python = ">=3.11"
+readme = "README.md"
+authors = [
+    {name = "Mingli Yuan", email = "mingli.yuan@gmail.com"}
+]
+license = {text = "MIT"}
+dependencies = [
+    "typer>=0.9.0",
+    "rich>=13.0.0",
+    "pydantic>=2.0.0",
+    "openai>=1.0.0",
+    "anthropic>=0.18.0",
+    "tenacity>=8.2.0",
+    "tiktoken>=0.6.0",
+    "python-dotenv>=1.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-cov>=4.1.0",
+    "ruff>=0.3.0",
+    "mypy>=1.9.0",
+    "ty>=0.0.1", # Assuming ty is available or will be replaced with actual LSP integration
+    "radon>=6.0.1",
+]
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+include = ["python_harness", "python_harness.*"]
+
+[project.scripts]
+harness = "python_harness.cli:app"
+
+[tool.ruff]
+line-length = 88
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+ignore = []
+
+[tool.mypy]
+python_version = "3.11"
+strict = true
+warn_return_any = true
+warn_unused_configs = true
+exclude = [
+    "vendors/.*"
+]
+
+[tool.pytest.ini_options]
+minversion = "8.0"
+addopts = "-ra -q --cov=python_harness --cov-report=term-missing --cov-report=html"
+testpaths = [
+    "tests",
+]
+

python_harness-0.0.1/python_harness/__init__.py

@@ -0,0 +1,5 @@
+"""
+Python Harness - An agentic evaluation tool for codebases.
+"""
+
+__version__ = "0.0.1"

python_harness-0.0.1/python_harness/cli.py

@@ -0,0 +1,207 @@
+"""
+Command-line interface for python-harness.
+"""
+
+import os
+import sys
+
+import typer
+from dotenv import load_dotenv
+from rich.console import Console
+
+from python_harness.evaluator import Evaluator
+
+# Try to find .env file explicitly before anything else executes
+env_path = os.path.join(os.getcwd(), '.env')
+if os.path.exists(env_path):
+    load_dotenv(dotenv_path=env_path)
+else:
+    load_dotenv() # Fallback to default search
+
+app = typer.Typer(help="Agentic harness tool for universal Python codebase evaluation.")
+console = Console()
+
+
+@app.command()
+def refine(
+    path: str = typer.Argument(".", help="The path to evaluate and evolve"),
+    steps: int = typer.Option(1, help="Number of evolution steps to perform"),
+    max_retries: int = typer.Option(3, help="Maximum retries per variant if tests fail")
+) -> None:
+    """
+    Refine the codebase through an agentic Edit-Test-Improve loop.
+    Generates variants based on suggestions, tests them, and picks the best.
+    """
+    console.print(
+        f"[bold magenta]Starting evolution loop for path:[/bold magenta] {path} "
+        f"[dim](steps={steps}, max_retries={max_retries})[/dim]"
+    )
+    
+    # 1. First, run a baseline evaluation to get suggestions
+    evaluator = Evaluator(path)
+    console.print("[cyan]Running baseline evaluation...[/cyan]")
+    hard_results = evaluator.hard_evaluator.evaluate()
+    soft_results = evaluator.soft_evaluator.evaluate()
+    baseline_report = evaluator.soft_evaluator.generate_final_report(
+        hard_results, soft_results
+    )
+    
+    suggestions = baseline_report.get("suggestions", [])
+    if not suggestions:
+        console.print("[yellow]No suggestions found to evolve. Exiting.[/yellow]")
+        return
+        
+    console.print(
+        f"[green]Found {len(suggestions)} suggestions. "
+        f"Starting evolution branches...[/green]"
+    )
+    
+    # TODO: Implement the Git branching and Agent modification logic here.
+    # The loop will be:
+    # for step in range(steps):
+    #   for suggestion in suggestions:
+    #     checkout new branch variant-X
+    #     for retry in range(max_retries):
+    #       ask LLM to apply suggestion to code
+    #       run pytest
+    #       if pytest passes:
+    #         run harness . to get new score
+    #         break
+    #       else:
+    #         feed error back to LLM for retry
+    #   compare all variants and checkout the best one
+    
+    console.print(
+        "[yellow]Evolution engine skeleton ready. "
+        "Actual git mutation logic pending.[/yellow]"
+    )
+@app.command()
+def measure(path: str = typer.Argument(".", help="The path to evaluate")) -> None:
+    """
+    Measure the codebase against hard, soft, and governance constraints.
+    Outputs a final report with scores and actionable improvement suggestions.
+    """
+    console.print(
+        f"[bold green]Starting harness measurement for path:[/bold green] {path}"
+    )
+    
+    evaluator = Evaluator(path)
+    
+    # 1. Hard Evaluation Gate (First Fence)
+    console.print("[bold blue]Running Hard Evaluation (ruff, mypy)...[/bold blue]")
+    hard_results = evaluator.hard_evaluator.evaluate()
+    
+    if not hard_results["all_passed"]:
+        console.print("[bold red]Hard Evaluation Failed! Exiting.[/bold red]")
+        if hard_results["ruff"]["status"] != "success":
+            console.print("[red]Ruff issues found.[/red]")
+        if hard_results["mypy"]["status"] != "success":
+            output = hard_results["mypy"].get("output", "")
+            console.print(f"[red]Mypy issues found:[/red]\n{output}")
+        if hard_results["ty"]["status"] != "success":
+            output = hard_results["ty"].get("output", "")
+            console.print(f"[red]Ty issues found:[/red]\n{output}")
+        if hard_results["radon_cc"]["status"] != "success":
+            issues = hard_results["radon_cc"].get("issues", [])
+            console.print(
+                f"[red]Cyclomatic Complexity too high "
+                f"({len(issues)} functions > 15):[/red]"
+            )
+            for issue in issues:
+                console.print(
+                    f"  - {issue['file']}: {issue['type']} '{issue['name']}' "
+                    f"has CC {issue['complexity']}"
+                )
+        sys.exit(1)
+        
+    console.print("[bold green]Hard Evaluation Passed![/bold green]")
+    
+    # Print Maintainability Index scorecard
+    mi_scores = hard_results.get("radon_mi", {}).get("mi_scores", {})
+    if mi_scores:
+        avg_mi = sum(mi_scores.values()) / len(mi_scores)
+        color = "green" if avg_mi > 50 else "yellow" if avg_mi > 20 else "red"
+        console.print(
+            f"[{color}]Average Maintainability Index: {avg_mi:.1f}/100[/{color}]"
+        )
+
+    # 2. Governance/QC Evaluation (Second Fence)
+    console.print("\n[bold blue]Running Governance QC (Second Fence)...[/bold blue]")
+    qc_results = evaluator.qc_evaluator.evaluate()
+    
+    if not qc_results["all_passed"]:
+        console.print("[bold red]Governance QC Failed! Exiting.[/bold red]")
+        console.print(
+            "[red]The proposed changes violate governance constraints "
+            "or lack sufficient evidence.[/red]"
+        )
+        for failure in qc_results["failures"]:
+            console.print(f"[red]- {failure}[/red]")
+        sys.exit(1)
+        
+    console.print(
+        "[bold green]Governance QC Passed! (Change is admissible)[/bold green]"
+    )
+
+    # 3. Soft Evaluation/Readability (Third Fence)
+    console.print(
+        "[bold blue]Running Soft Evaluation "
+        "(Readability & Understandability)...[/bold blue]"
+    )
+    soft_results = evaluator.soft_evaluator.evaluate()
+
+    pkg_summary = soft_results["package_summary"]
+    console.print(
+        f"[green]Analyzed {pkg_summary['total_files']} files with a total of "
+        f"{pkg_summary['total_tokens']} tokens.[/green]"
+    )
+    console.print(
+        f"[magenta]Agent's Understanding of the Package:[/magenta]\n"
+        f"{pkg_summary['package_understanding']}"
+    )
+    
+    console.print(
+        f"\n[cyan]Overall Understandability Score:[/cyan] "
+        f"{soft_results['understandability_score']:.1f}/100"
+    )
+    
+    qa_results = soft_results.get("qa_results", {}).get("sampled_entities", [])
+    if qa_results:
+        console.print("\n[bold yellow]Blind QA Sampling Results:[/bold yellow]")
+        for qa in qa_results:
+            color = "green" if qa['score'] >= 80 else "red"
+            console.print(f"  - [{color}]{qa['entity']}: Score {qa['score']}[/{color}]")
+            console.print(f"    [dim]Feedback: {qa['feedback']}[/dim]")
+
+    console.print("\n[yellow]Evaluation completed. Generating report...[/yellow]\n")
+    
+    # Generate Final Report
+    final_report = evaluator.soft_evaluator.generate_final_report(
+        hard_results, soft_results
+    )
+    
+    if final_report:
+        verdict = final_report.get("verdict", "Unknown")
+        verdict_color = "bold green" if "Pass" in verdict else "bold red"
+        
+        console.print(
+            f"[{verdict_color}]=== FINAL VERDICT: {verdict} ===[/{verdict_color}]"
+        )
+        console.print(f"[bold]Summary:[/bold] {final_report.get('summary', '')}\n")
+        
+        suggestions = final_report.get("suggestions", [])
+        if suggestions:
+            console.print("[bold cyan]Top 3 Improvement Suggestions:[/bold cyan]")
+            for i, sug in enumerate(suggestions, 1):
+                console.print(
+                    f"  {i}. [bold]{sug.get('title', 'Suggestion')}[/bold] "
+                    f"(Target: [yellow]{sug.get('target_file', 'unknown')}[/yellow])"
+                )
+                console.print(f"     [dim]{sug.get('description', '')}[/dim]")
+        
+        if "Fail" in verdict:
+            sys.exit(1)
+
+
+if __name__ == "__main__":
+    app()

python_harness-0.0.1/python_harness/evaluator.py

@@ -0,0 +1,42 @@
+"""
+Core module for integrating all evaluations and producing the final report.
+"""
+
+from typing import Any
+
+from python_harness.hard_evaluator import HardEvaluator
+from python_harness.qc_evaluator import QCEvaluator
+from python_harness.soft_evaluator import SoftEvaluator
+
+
+class Evaluator:
+    """
+    Main evaluator coordinating hard, QC, and soft assessments.
+    """
+
+    def __init__(self, target_path: str):
+        self.target_path = target_path
+        self.hard_evaluator = HardEvaluator(target_path)
+        self.qc_evaluator = QCEvaluator(target_path)
+        self.soft_evaluator = SoftEvaluator(target_path)
+
+    def run(self) -> dict[str, Any]:
+        """
+        Run the complete evaluation process.
+        """
+        hard_results = self.hard_evaluator.evaluate()
+        qc_results = self.qc_evaluator.evaluate()
+        soft_results = self.soft_evaluator.evaluate()
+
+        # Generate Final Synthesized Report with 3 Suggestions
+        final_report = self.soft_evaluator.generate_final_report(
+            hard_results, soft_results
+        )
+
+        return {
+            "hard_evaluation": hard_results,
+            "qc_evaluation": qc_results,
+            "soft_evaluation": soft_results,
+            "final_report": final_report,
+            "overall_status": "success",
+        }

python_harness-0.0.1/python_harness/hard_evaluator.py

@@ -0,0 +1,200 @@
+"""
+Core module for integrating hard evaluation tools like ruff, mypy, and pytest.
+"""
+
+import json
+import subprocess
+from pathlib import Path
+from typing import Any
+
+from rich.console import Console
+
+console = Console()
+
+class HardEvaluator:
+    """
+    Evaluator for collecting structural code quality metrics.
+    """
+
+    def __init__(self, target_path: str):
+        self.target_path = Path(target_path).resolve()
+
+    def run_ruff(self) -> dict[str, Any]:
+        """
+        Run Ruff linter and return results.
+        """
+        try:
+            result = subprocess.run(
+                ["ruff", "check", str(self.target_path), "--output-format", "json"],
+                capture_output=True,
+                text=True,
+                check=False
+            )
+            issues = json.loads(result.stdout) if result.stdout else []
+            status = "success" if result.returncode == 0 else "failed"
+            return {
+                "status": status,
+                "issues": issues,
+                "return_code": result.returncode,
+            }
+        except Exception as e:
+            return {"status": "error", "error_message": str(e)}
+
+    def run_mypy(self) -> dict[str, Any]:
+        """
+        Run Mypy type checker and return results.
+        """
+        try:
+            result = subprocess.run(
+                ["mypy", str(self.target_path)],
+                capture_output=True,
+                text=True,
+                check=False
+            )
+            status = "success" if result.returncode == 0 else "failed"
+            return {
+                "status": status,
+                "output": result.stdout,
+                "return_code": result.returncode,
+            }
+        except Exception as e:
+            return {"status": "error", "error_message": str(e)}
+
+    def run_ty(self) -> dict[str, Any]:
+        """
+        Run ty language server checks.
+        """
+        try:
+            result = subprocess.run(
+                ["ty", "check", str(self.target_path)],
+                capture_output=True,
+                text=True,
+                check=False
+            )
+            status = "success" if result.returncode == 0 else "failed"
+            return {
+                "status": status,
+                "output": result.stdout,
+                "return_code": result.returncode,
+            }
+        except Exception as e:
+            return {"status": "error", "error_message": str(e)}
+
+    def run_radon_cc(self) -> dict[str, Any]:
+        """
+        Run Radon cyclomatic complexity check.
+        Flag any function/method with CC > 15 as a failure.
+        """
+        try:
+            result = subprocess.run(
+                ["radon", "cc", "-j", "-a", str(self.target_path)],
+                capture_output=True,
+                text=True,
+                check=False
+            )
+            
+            issues = []
+            status = "success"
+            
+            if result.stdout:
+                data = json.loads(result.stdout)
+                for file_path, blocks in data.items():
+                    if isinstance(blocks, list):
+                        for block in blocks:
+                            if block.get('complexity', 0) > 15:
+                                issues.append({
+                                    "file": file_path,
+                                    "name": block.get('name'),
+                                    "type": block.get('type'),
+                                    "complexity": block.get('complexity')
+                                })
+            
+            if issues:
+                status = "failed"
+                
+            return {
+                "status": status,
+                "issues": issues,
+                "return_code": result.returncode,
+                "output": result.stdout
+            }
+        except Exception as e:
+            return {"status": "error", "error_message": str(e)}
+
+    def run_radon_mi(self) -> dict[str, Any]:
+        """
+        Run Radon Maintainability Index (MI) check.
+        This is a diagnostic metric, so it won't fail the build,
+        but it contributes to the scorecard.
+        """
+        try:
+            result = subprocess.run(
+                ["radon", "mi", "-j", str(self.target_path)],
+                capture_output=True,
+                text=True,
+                check=False
+            )
+            
+            mi_scores = {}
+            if result.stdout:
+                data = json.loads(result.stdout)
+                for file_path, info in data.items():
+                    mi_scores[file_path] = info.get('mi', 100.0)
+                    
+            return {
+                "status": "success",
+                "mi_scores": mi_scores,
+                "return_code": result.returncode,
+            }
+        except Exception as e:
+            return {"status": "error", "error_message": str(e)}
+
+    def run_pytest(self) -> dict[str, Any]:
+        """
+        Run Pytest test suite and return coverage results.
+        """
+        try:
+            # When pytest is run within pytest, it can cause issues or hang.
+            # Here we just run it as a subprocess to gather results.
+            result = subprocess.run(
+                ["pytest", str(self.target_path), "--cov", "--cov-report=json"],
+                capture_output=True,
+                text=True,
+                check=False
+            )
+            status = "success" if result.returncode == 0 else "failed"
+            return {
+                "status": status,
+                "output": result.stdout,
+                "return_code": result.returncode,
+            }
+        except Exception as e:
+             return {"status": "error", "error_message": str(e)}
+
+    def evaluate(self) -> dict[str, Any]:
+        """
+        Execute all hard evaluation tools.
+        Returns a dictionary with results and an overall success boolean.
+        """
+        ruff_res = self.run_ruff()
+        mypy_res = self.run_mypy()
+        ty_res = self.run_ty()
+        radon_cc_res = self.run_radon_cc()
+        radon_mi_res = self.run_radon_mi()
+        # pytest_res = self.run_pytest() # Better handled as a separate stage
+        
+        all_passed = (
+            ruff_res.get("status") == "success" and 
+            mypy_res.get("status") == "success" and
+            ty_res.get("status") == "success" and
+            radon_cc_res.get("status") == "success"
+        )
+
+        return {
+            "all_passed": all_passed,
+            "ruff": ruff_res,
+            "mypy": mypy_res,
+            "ty": ty_res,
+            "radon_cc": radon_cc_res,
+            "radon_mi": radon_mi_res
+        }