devagent-ai 0.4.0__tar.gz → 0.5.0__tar.gz

{devagent_ai-0.4.0/devagent_ai.egg-info → devagent_ai-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: devagent-ai
-Version: 0.4.0
+Version: 0.5.0
 Summary: Evidence-driven local autonomous software engineering agent
 Author: Tom Ha
 Maintainer: Tom Ha
@@ -224,8 +224,21 @@ devagent setup --help
 devagent doctor
 devagent models
 devagent status
+devagent benchmark --help
 ```
 
+### Pinned real-world benchmark
+
+DevAgent 0.5 adds an opt-in benchmark runner for pinned GitHub repositories. A benchmark case injects a deterministic defect into an exact commit and uses an **external oracle** before and after DevAgent. This avoids treating DevAgent's own report as the benchmark oracle.
+
+```bash
+devagent benchmark \
+  --catalog /path/to/realworld-cases.json \
+  --report .devagent/realworld-benchmark.json
+```
+
+A `VERIFIED` result with a failing external oracle is explicitly counted as a **false VERIFIED**. See [docs/realworld-benchmark.md](docs/realworld-benchmark.md) for the catalog contract and safety boundary.
+
 ## Outcome contract
 
 ### `VERIFIED`
@@ -284,7 +297,7 @@ Verification can include baseline tests, targeted tests, component/broad checks,
 
 ## Production qualification
 
-DevAgent 0.4.0 adds **production qualification v3**. The release catalog contains 52 required cases and preserves the primary invariant:
+DevAgent 0.5.0 uses **production qualification v4**. It extends the v3 release gate with large-repository bounded-retrieval and real-world benchmark truthfulness contracts while preserving the primary invariant:
 
 ```text
 false_verified == 0
@@ -304,8 +317,8 @@ Run the release qualification locally on a machine with those toolchains:
 
 ```bash
 python -m devagent.qualification \
-  --catalog evaluation/benchmark_v3.json \
-  --report .devagent/production-qualification-v3.json
+  --catalog evaluation/benchmark_v4.json \
+  --report .devagent/production-qualification-v4.json
 ```
 
 Production CI runs Python 3.10/3.11/3.12, a clean wheel install, and this production qualification gate. The qualification JSON is retained as CI evidence.
@@ -366,9 +379,9 @@ Automated provider tests normally use deterministic or mocked clients and do not
 
 ## Project status
 
-DevAgent 0.4 is **beta software** with a production-readiness target of approximately **9/10 for the documented local engineering workflow**. That assessment is based on explicit qualification evidence, not on a claim of universal correctness or parity with every capability of a hosted coding platform.
+DevAgent 0.5 is **beta software**. Its qualification and benchmark results are bounded claims tied to explicit cases, pinned revisions, and external oracles; they are not a claim of universal correctness or parity with every hosted coding platform.
 
-Remaining gaps include browser/UI runtime qualification, a broader Java/.NET/database-migration matrix, very large monorepo benchmarks, parallel multi-agent orchestration, operating-system sandboxing, and continuous paid real-provider testing across every model/provider combination.
+Remaining gaps include a larger published corpus of pinned upstream benchmark cases, browser/UI runtime qualification, a broader Java/.NET/database-migration matrix, very large monorepo stress runs above the current bounded inventory, parallel multi-agent orchestration, operating-system sandboxing, and continuous paid real-provider testing across every model/provider combination.
 
 The project intentionally prioritizes trustworthy outcomes over feature count.
 

{devagent_ai-0.4.0 → devagent_ai-0.5.0}/README.md

@@ -193,8 +193,21 @@ devagent setup --help
 devagent doctor
 devagent models
 devagent status
+devagent benchmark --help
 ```
 
+### Pinned real-world benchmark
+
+DevAgent 0.5 adds an opt-in benchmark runner for pinned GitHub repositories. A benchmark case injects a deterministic defect into an exact commit and uses an **external oracle** before and after DevAgent. This avoids treating DevAgent's own report as the benchmark oracle.
+
+```bash
+devagent benchmark \
+  --catalog /path/to/realworld-cases.json \
+  --report .devagent/realworld-benchmark.json
+```
+
+A `VERIFIED` result with a failing external oracle is explicitly counted as a **false VERIFIED**. See [docs/realworld-benchmark.md](docs/realworld-benchmark.md) for the catalog contract and safety boundary.
+
 ## Outcome contract
 
 ### `VERIFIED`
@@ -253,7 +266,7 @@ Verification can include baseline tests, targeted tests, component/broad checks,
 
 ## Production qualification
 
-DevAgent 0.4.0 adds **production qualification v3**. The release catalog contains 52 required cases and preserves the primary invariant:
+DevAgent 0.5.0 uses **production qualification v4**. It extends the v3 release gate with large-repository bounded-retrieval and real-world benchmark truthfulness contracts while preserving the primary invariant:
 
 ```text
 false_verified == 0
@@ -273,8 +286,8 @@ Run the release qualification locally on a machine with those toolchains:
 
 ```bash
 python -m devagent.qualification \
-  --catalog evaluation/benchmark_v3.json \
-  --report .devagent/production-qualification-v3.json
+  --catalog evaluation/benchmark_v4.json \
+  --report .devagent/production-qualification-v4.json
 ```
 
 Production CI runs Python 3.10/3.11/3.12, a clean wheel install, and this production qualification gate. The qualification JSON is retained as CI evidence.
@@ -335,9 +348,9 @@ Automated provider tests normally use deterministic or mocked clients and do not
 
 ## Project status
 
-DevAgent 0.4 is **beta software** with a production-readiness target of approximately **9/10 for the documented local engineering workflow**. That assessment is based on explicit qualification evidence, not on a claim of universal correctness or parity with every capability of a hosted coding platform.
+DevAgent 0.5 is **beta software**. Its qualification and benchmark results are bounded claims tied to explicit cases, pinned revisions, and external oracles; they are not a claim of universal correctness or parity with every hosted coding platform.
 
-Remaining gaps include browser/UI runtime qualification, a broader Java/.NET/database-migration matrix, very large monorepo benchmarks, parallel multi-agent orchestration, operating-system sandboxing, and continuous paid real-provider testing across every model/provider combination.
+Remaining gaps include a larger published corpus of pinned upstream benchmark cases, browser/UI runtime qualification, a broader Java/.NET/database-migration matrix, very large monorepo stress runs above the current bounded inventory, parallel multi-agent orchestration, operating-system sandboxing, and continuous paid real-provider testing across every model/provider combination.
 
 The project intentionally prioritizes trustworthy outcomes over feature count.
 

{devagent_ai-0.4.0 → devagent_ai-0.5.0}/devagent/__init__.py

@@ -1,3 +1,3 @@
 """DevAgent: evidence-driven local software engineering automation."""
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"

{devagent_ai-0.4.0 → devagent_ai-0.5.0}/devagent/cli.py

@@ -353,6 +353,10 @@ def main(argv: Sequence[str] | None = None) -> int:
         parser = argparse.ArgumentParser(prog="devagent status", description="Show the latest local run")
         parser.add_argument("--repo", "-r", type=Path, default=Path.cwd())
         return _status(parser.parse_args(arguments[1:]).repo)
+    if arguments and arguments[0] == "benchmark":
+        from devagent.realworld import main as benchmark_main
+
+        return benchmark_main(arguments[1:])
     return _run(arguments)
 
 

{devagent_ai-0.4.0 → devagent_ai-0.5.0}/devagent/discovery.py

@@ -67,24 +67,36 @@ def _fingerprint(path: Path) -> str:
 
 
 def _walk(root: Path, limit: int = 12_000) -> list[Path]:
+    """Walk deterministically while pruning generated/vendor trees before descent."""
     files: list[Path] = []
-    for path in root.rglob("*"):
-        relative = path.relative_to(root)
-        if any(part in SKIP_DIRECTORIES for part in relative.parts):
-            continue
-        try:
-            resolved = path.resolve()
-            resolved_relative = resolved.relative_to(root)
-        except (OSError, ValueError):
-            continue
-        if (
-            resolved.is_file()
-            and not is_secret_path(relative)
-            and not is_secret_path(resolved_relative)
-        ):
-            files.append(path)
-            if len(files) >= limit:
-                break
+    maximum = min(max(0, int(limit)), 12_000)
+    if maximum == 0:
+        return files
+    for current, directories, filenames in os.walk(root, topdown=True, followlinks=False):
+        current_path = Path(current)
+        relative_current = current_path.relative_to(root)
+        directories[:] = sorted(
+            directory
+            for directory in directories
+            if directory not in SKIP_DIRECTORIES
+            and not is_secret_path(relative_current / directory)
+        )
+        for filename in sorted(filenames):
+            path = current_path / filename
+            relative = path.relative_to(root)
+            try:
+                resolved = path.resolve()
+                resolved_relative = resolved.relative_to(root)
+            except (OSError, ValueError):
+                continue
+            if (
+                resolved.is_file()
+                and not is_secret_path(relative)
+                and not is_secret_path(resolved_relative)
+            ):
+                files.append(path)
+                if len(files) >= maximum:
+                    return files
     return files
 
 

{devagent_ai-0.4.0 → devagent_ai-0.5.0}/devagent/qualification.py

@@ -157,7 +157,7 @@ def main(argv: list[str] | None = None) -> int:
     parser.add_argument(
         "--catalog",
         type=Path,
-        default=Path("evaluation/benchmark_v3.json"),
+        default=Path("evaluation/benchmark_v4.json"),
         help="Qualification catalog path",
     )
     parser.add_argument(