neosloc 0.3.0__tar.gz

neosloc-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Marco Montanari
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

neosloc-0.3.0/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.4
+Name: neosloc
+Version: 0.3.0
+Summary: Integrability evaluation for software repositories: what sloccount measured, re-asked for the age of LLMs.
+Author: Marco Montanari
+License: MIT
+Project-URL: Homepage, https://github.com/sirmmo/neosloc
+Project-URL: Documentation, https://ingmmo.com/neosloc/
+Project-URL: Repository, https://github.com/sirmmo/neosloc
+Project-URL: Issues, https://github.com/sirmmo/neosloc/issues
+Project-URL: Changelog, https://github.com/sirmmo/neosloc/blob/main/CHANGELOG.md
+Keywords: sloccount,cocomo,integrability,software metrics,llm,agents,api
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: agentic
+Requires-Dist: anthropic>=1.11; python_version >= "3.10" and extra == "agentic"
+Dynamic: license-file
+
+# neosloc
+
+[![ci](https://github.com/sirmmo/neosloc/actions/workflows/ci.yml/badge.svg)](https://github.com/sirmmo/neosloc/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/neosloc)](https://pypi.org/project/neosloc/)
+[![docs](https://img.shields.io/badge/docs-ingmmo.com%2Fneosloc-blue)](https://ingmmo.com/neosloc/)
+
+`sloccount` counted lines of code and fed them to COCOMO to estimate the effort to *build*
+software. With LLMs writing code, lines are cheap and that estimate has lost its meaning.
+What is still expensive is everything *around* the code: how easily other systems and
+agents can call it, run it, rely on it and change it safely, and the knowledge that exists
+only in the code and its history.
+
+neosloc measures that:
+
+- **ten integrability dimensions**, each scored 0–4 with the evidence behind the level and
+  the gaps phrased as actions;
+- the **retrofit effort** to bring every dimension up to "solid", and whether the system
+  is cheap to wrap;
+- **neoCOCOMO**: what the codebase is worth when an agent can re-type it, next to classic
+  COCOMO;
+- optional **LLM evaluators** on Anthropic or [OpenRouter](https://openrouter.ai): a
+  *probe* attempts real integration tasks using only the docs, a *judge* checks the
+  answers against the source, and a *review* audits every static level.
+
+```bash
+pip install neosloc
+neosloc path/to/repo
+```
+
+```text
+Interface surface   ■■■□  3 solid      The framework generates a contract from code, so it tracks the implementation.
+                      + routes:fastapi/flask: 15 route declarations in 3 files (app/routes/admin.py)
+                      + generated-spec: FastAPI (auto OpenAPI) (app/main.py)
+                      - Check the generated spec into the repo so changes are reviewable and diffable.
+...
+Integrability index (0-4, assessed dimensions):  1.80
+Retrofit effort to level 3 (agent-assisted):     7.5 person-days
+
+Value (neoCOCOMO, cost approach; see neosloc/value.py)
+  Classic COCOMO (sloccount):    4.6 KSLOC -> 11.8 person-months, 6.4 months, $133k
+  Behaviour captured:            38% (tests 0.14, contract 0.25, docs 1.00)
+  Reproduce with agents:         x0.38 of classic -> 4.5 PM
+  Rediscover uncaptured history: 102 fix / 474 commits, 19 authors -> 4.4 PM knowledge, 2.7 PM uncaptured
+  Value:                         5.2 PM ~ $59k  (at $11k per PM)
+  Knowledge at risk:             38% of replacement cost lives only in code and history
+```
+
+**Documentation: <https://ingmmo.com/neosloc/>**
+
+## Install
+
+| | |
+|---|---|
+| `pip install neosloc` (or `pipx install neosloc`, `uv tool install neosloc`) | the `neosloc` command; the static analysis has **no dependencies** and runs on Python ≥ 3.8 |
+| `pip install 'neosloc[agentic]'` | adds the anthropic SDK, needed for `anthropic:` models (Python ≥ 3.10) |
+| `pip install git+https://github.com/sirmmo/neosloc` | the development version |
+
+OpenRouter models need no extra package, only `OPENROUTER_API_KEY`.
+
+## Dimensions
+
+| Key | Asks |
+|---|---|
+| `interface` | Is there a machine-readable contract covering the implemented surface, and more than one way in (MCP, CLI `--json`, SDK, typed library)? |
+| `stability` | Semver, changelog, API versioning, deprecations, and the real git history of the OpenAPI files. |
+| `events` | Outbound webhooks (signed, self-service), streams, brokers, CDC, AsyncAPI/CloudEvents. |
+| `identity` | Token auth, OAuth2/OIDC, scopes, managed API keys or service accounts, SCIM. |
+| `portability` | Bulk export/import, open formats, schema in the repo. |
+| `ergonomics` | RFC 9457 errors, validation, idempotency keys, pagination, rate-limit headers, dry-run, `llms.txt`. |
+| `embeddability` | Container, compose/Helm/IaC, documented env config, headless entry point, library packaging. |
+| `extensibility` | Plugin discovery, hooks, scripting, extension docs. |
+| `observability` | Health/readiness, metrics, tracing, structured logs, error tracking. |
+| `legibility` | The heir to SLOC: tokens per module, import cycles, tests, types, CI, lockfiles, agent docs. |
+
+Details: [dimensions](https://ingmmo.com/neosloc/dimensions/),
+[retrofit effort](https://ingmmo.com/neosloc/estimate/),
+[neoCOCOMO](https://ingmmo.com/neosloc/value/).
+
+## LLM evaluators
+
+```bash
+# Probe: a model attempts 8 integration tasks from the docs; answers are graded against the code
+neosloc --agentic --scope both --transcripts runs/ .
+
+# Probe with Claude, judge with another vendor through OpenRouter
+neosloc --judge --judge-model openrouter:openai/gpt-5.6-luna .
+
+# Review: a panel audits every static level against the code
+neosloc --review --review-model openrouter:google/gemini-3.8-flash,openrouter:deepseek/deepseek-v4-pro-0813 .
+```
+
+Models are written `provider:model`. The default is `anthropic:claude-opus-5-5` at effort
+`medium`. A comma-separated list runs a panel and reports agreement. Every model call
+shares one `--max-cost` budget (default $5), and the report states the total. See
+[LLM evaluators](https://ingmmo.com/neosloc/evaluators/) and
+[Using OpenRouter](https://ingmmo.com/neosloc/openrouter/).
+
+## Status
+
+All thresholds and coefficients are uncalibrated, so treat the numbers as rankings.
+[Calibration and limits](https://ingmmo.com/neosloc/calibration/) describes how to fix
+that; `--review` and `--agentic` exist partly to do it.
+
+## Development
+
+```bash
+python -m unittest discover -s tests -t .     # no network, no API spend
+python -m neosloc .
+```
+
+See [Writing detectors](https://ingmmo.com/neosloc/extending/) and `AGENTS.md`.
+Releases: bump `neosloc/__init__.py`, add a `CHANGELOG.md` section, push a `vX.Y.Z` tag.
+
+MIT licensed.

neosloc-0.3.0/README.md

@@ -0,0 +1,114 @@
+# neosloc
+
+[![ci](https://github.com/sirmmo/neosloc/actions/workflows/ci.yml/badge.svg)](https://github.com/sirmmo/neosloc/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/neosloc)](https://pypi.org/project/neosloc/)
+[![docs](https://img.shields.io/badge/docs-ingmmo.com%2Fneosloc-blue)](https://ingmmo.com/neosloc/)
+
+`sloccount` counted lines of code and fed them to COCOMO to estimate the effort to *build*
+software. With LLMs writing code, lines are cheap and that estimate has lost its meaning.
+What is still expensive is everything *around* the code: how easily other systems and
+agents can call it, run it, rely on it and change it safely, and the knowledge that exists
+only in the code and its history.
+
+neosloc measures that:
+
+- **ten integrability dimensions**, each scored 0–4 with the evidence behind the level and
+  the gaps phrased as actions;
+- the **retrofit effort** to bring every dimension up to "solid", and whether the system
+  is cheap to wrap;
+- **neoCOCOMO**: what the codebase is worth when an agent can re-type it, next to classic
+  COCOMO;
+- optional **LLM evaluators** on Anthropic or [OpenRouter](https://openrouter.ai): a
+  *probe* attempts real integration tasks using only the docs, a *judge* checks the
+  answers against the source, and a *review* audits every static level.
+
+```bash
+pip install neosloc
+neosloc path/to/repo
+```
+
+```text
+Interface surface   ■■■□  3 solid      The framework generates a contract from code, so it tracks the implementation.
+                      + routes:fastapi/flask: 15 route declarations in 3 files (app/routes/admin.py)
+                      + generated-spec: FastAPI (auto OpenAPI) (app/main.py)
+                      - Check the generated spec into the repo so changes are reviewable and diffable.
+...
+Integrability index (0-4, assessed dimensions):  1.80
+Retrofit effort to level 3 (agent-assisted):     7.5 person-days
+
+Value (neoCOCOMO, cost approach; see neosloc/value.py)
+  Classic COCOMO (sloccount):    4.6 KSLOC -> 11.8 person-months, 6.4 months, $133k
+  Behaviour captured:            38% (tests 0.14, contract 0.25, docs 1.00)
+  Reproduce with agents:         x0.38 of classic -> 4.5 PM
+  Rediscover uncaptured history: 102 fix / 474 commits, 19 authors -> 4.4 PM knowledge, 2.7 PM uncaptured
+  Value:                         5.2 PM ~ $59k  (at $11k per PM)
+  Knowledge at risk:             38% of replacement cost lives only in code and history
+```
+
+**Documentation: <https://ingmmo.com/neosloc/>**
+
+## Install
+
+| | |
+|---|---|
+| `pip install neosloc` (or `pipx install neosloc`, `uv tool install neosloc`) | the `neosloc` command; the static analysis has **no dependencies** and runs on Python ≥ 3.8 |
+| `pip install 'neosloc[agentic]'` | adds the anthropic SDK, needed for `anthropic:` models (Python ≥ 3.10) |
+| `pip install git+https://github.com/sirmmo/neosloc` | the development version |
+
+OpenRouter models need no extra package, only `OPENROUTER_API_KEY`.
+
+## Dimensions
+
+| Key | Asks |
+|---|---|
+| `interface` | Is there a machine-readable contract covering the implemented surface, and more than one way in (MCP, CLI `--json`, SDK, typed library)? |
+| `stability` | Semver, changelog, API versioning, deprecations, and the real git history of the OpenAPI files. |
+| `events` | Outbound webhooks (signed, self-service), streams, brokers, CDC, AsyncAPI/CloudEvents. |
+| `identity` | Token auth, OAuth2/OIDC, scopes, managed API keys or service accounts, SCIM. |
+| `portability` | Bulk export/import, open formats, schema in the repo. |
+| `ergonomics` | RFC 9457 errors, validation, idempotency keys, pagination, rate-limit headers, dry-run, `llms.txt`. |
+| `embeddability` | Container, compose/Helm/IaC, documented env config, headless entry point, library packaging. |
+| `extensibility` | Plugin discovery, hooks, scripting, extension docs. |
+| `observability` | Health/readiness, metrics, tracing, structured logs, error tracking. |
+| `legibility` | The heir to SLOC: tokens per module, import cycles, tests, types, CI, lockfiles, agent docs. |
+
+Details: [dimensions](https://ingmmo.com/neosloc/dimensions/),
+[retrofit effort](https://ingmmo.com/neosloc/estimate/),
+[neoCOCOMO](https://ingmmo.com/neosloc/value/).
+
+## LLM evaluators
+
+```bash
+# Probe: a model attempts 8 integration tasks from the docs; answers are graded against the code
+neosloc --agentic --scope both --transcripts runs/ .
+
+# Probe with Claude, judge with another vendor through OpenRouter
+neosloc --judge --judge-model openrouter:openai/gpt-5.6-luna .
+
+# Review: a panel audits every static level against the code
+neosloc --review --review-model openrouter:google/gemini-3.8-flash,openrouter:deepseek/deepseek-v4-pro-0813 .
+```
+
+Models are written `provider:model`. The default is `anthropic:claude-opus-5-5` at effort
+`medium`. A comma-separated list runs a panel and reports agreement. Every model call
+shares one `--max-cost` budget (default $5), and the report states the total. See
+[LLM evaluators](https://ingmmo.com/neosloc/evaluators/) and
+[Using OpenRouter](https://ingmmo.com/neosloc/openrouter/).
+
+## Status
+
+All thresholds and coefficients are uncalibrated, so treat the numbers as rankings.
+[Calibration and limits](https://ingmmo.com/neosloc/calibration/) describes how to fix
+that; `--review` and `--agentic` exist partly to do it.
+
+## Development
+
+```bash
+python -m unittest discover -s tests -t .     # no network, no API spend
+python -m neosloc .
+```
+
+See [Writing detectors](https://ingmmo.com/neosloc/extending/) and `AGENTS.md`.
+Releases: bump `neosloc/__init__.py`, add a `CHANGELOG.md` section, push a `vX.Y.Z` tag.
+
+MIT licensed.

neosloc-0.3.0/neosloc/__init__.py

@@ -0,0 +1,2 @@
+"""neosloc - integrability evaluation for software, the successor question to SLOC."""
+__version__ = "0.3.0"

neosloc-0.3.0/neosloc/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from .cli import main
+
+sys.exit(main())

neosloc-0.3.0/neosloc/agentic/__init__.py

@@ -0,0 +1,10 @@
+"""LLM evaluators: probe (attempt integration tasks), judge (check answers
+against the source) and review (audit static levels), on Anthropic or OpenRouter."""
+from .judge import Judge
+from .llm import Budget, make_backend, parse_spec
+from .review import Reviewer, review_all
+from .runner import DEFAULT_EFFORT, DEFAULT_MODEL, Probe, panel
+from .tasks import TASKS, select
+
+__all__ = ["Budget", "DEFAULT_EFFORT", "DEFAULT_MODEL", "Judge", "Probe", "Reviewer", "TASKS",
+           "make_backend", "panel", "parse_spec", "review_all", "select"]

neosloc-0.3.0/neosloc/agentic/grader.py

@@ -0,0 +1,186 @@
+"""Grounding: check an agent's answer against the implementation.
+
+The agent may only have seen the docs; the grader always sees everything. A
+step is grounded when the thing it names exists: the HTTP route is declared
+(or in the contract), the CLI program and its flags exist, the env vars are
+read somewhere, the symbol is defined. With a live instance, HTTP steps must
+also have been exercised successfully.
+"""
+from __future__ import annotations
+
+import posixpath
+import re
+import shlex
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional, Set
+
+from ..repo import Repo
+from ..specs import find_specs
+from .workspace import HttpCall
+
+PARAM_RE = re.compile(r"\{[^}/]+\}|<[^>/]+>|:[A-Za-z_]\w*|\[[^\]/]+\]|\$\{[^}]+\}")
+DYNAMIC_SEG = re.compile(r"^(\d+|[0-9a-f]{8}-[0-9a-f-]{27,}|[0-9a-f]{24,})$", re.I)
+ROUTE_STRING = re.compile(r"""['"`](\^?/?[A-Za-z0-9_{}<>:\[\]$./-]*)['"`]""")
+FLAG_RE = re.compile(r"(?<![\w-])(--[A-Za-z][\w-]*)")
+ENV_NAME = re.compile(r"^[A-Z][A-Z0-9_]{2,}$")
+LAUNCHERS = {"python", "python3", "-m", "npx", "pnpm", "yarn", "npm", "bunx", "uv", "poetry", "pipx",
+             "run", "exec", "sudo", "env", "bundle", "go", "cargo", "dotnet", "java", "-jar", "php"}
+
+
+def normalize_path(path: str) -> str:
+    path = re.sub(r"^\w+://[^/]+", "", path.strip()).split("?", 1)[0].split("#", 1)[0]
+    path = path.lstrip("^").rstrip("$")
+    path = PARAM_RE.sub("{}", path)
+    segs = [("{}" if DYNAMIC_SEG.match(s) else s) for s in path.strip("/").split("/") if s]
+    return "/" + "/".join(segs)
+
+
+@dataclass
+class Grade:
+    grounded: int = 0
+    checked: int = 0
+    problems: List[str] = field(default_factory=list)
+
+
+class Grader:
+    def __init__(self, repo: Repo, route_files: Optional[List[str]] = None):
+        self.repo = repo
+        self.src = repo.source_files(include_tests=False)
+        self.spec_ops: Set[tuple] = set()
+        for s in find_specs(repo):
+            if s.kind == "openapi":
+                for op in s.operations:
+                    m, p = op.split(" ", 1)
+                    self.spec_ops.add((m, normalize_path(p)))
+        self.route_templates: Set[str] = set()
+        for f in route_files or []:
+            for line in repo.read(f).splitlines():
+                for lit in ROUTE_STRING.findall(line):
+                    if lit.strip("/^$") and not lit.startswith("."):
+                        self.route_templates.add(normalize_path(lit))
+        self._all_text = None
+
+    # ---- helpers -----------------------------------------------------------
+
+    def _text(self) -> str:
+        if self._all_text is None:
+            files = self.src + self.repo.manifests() + self.repo.config_files() + self.repo.doc_files()
+            self._all_text = "\n".join(self.repo.read(f) for f in dict.fromkeys(files))
+        return self._all_text
+
+    def _code_text(self) -> str:
+        return "\n".join(self.repo.read(f) for f in self.src + self.repo.config_files())
+
+    @staticmethod
+    def _suffix_match(candidate: str, template: str) -> bool:
+        c = candidate.strip("/").split("/")
+        t = template.strip("/").split("/")
+        if not t or t == [""] or len(t) > len(c):
+            return False
+        return all(a == b or b == "{}" or a == "{}" for a, b in zip(c[-len(t):], t))
+
+    # ---- step checks -------------------------------------------------------
+
+    def http(self, method: Optional[str], path: Optional[str], live: Optional[List[HttpCall]]) -> Optional[str]:
+        if not path:
+            return "http step without a path"
+        cand = normalize_path(path)
+        method = (method or "").upper()
+        in_spec = any((not method or m == method) and self._suffix_match(cand, p) for m, p in self.spec_ops)
+        in_code = any(self._suffix_match(cand, t) for t in self.route_templates)
+        if not (in_spec or in_code):
+            return "%s %s is not a declared route" % (method or "?", path)
+        if live is not None:
+            ok = [c for c in live if self._suffix_match(normalize_path(c.path), cand)
+                  and (not method or c.method == method)]
+            if not ok:
+                return "%s %s was never exercised against the live instance" % (method or "?", path)
+        return None
+
+    def cli(self, command: Optional[str]) -> Optional[str]:
+        if not command:
+            return "cli step without a command"
+        try:
+            words = shlex.split(command.splitlines()[0])
+        except ValueError:
+            words = command.split()
+        if not words:
+            return "empty command"
+        if words[0] == "docker" or words[:2] == ["docker-compose"]:
+            ok = self.repo.glob(r"(^|/)(Dockerfile|Containerfile|(docker-)?compose[\w.-]*\.ya?ml)$")
+            return None if ok else "docker command but no Dockerfile/compose file"
+        if words[0] in ("make", "just"):
+            target = words[1] if len(words) > 1 else None
+            mk = self.repo.glob(r"(^|/)(Makefile|justfile)$")
+            if not mk:
+                return "%s command but no %s" % (words[0], "Makefile" if words[0] == "make" else "justfile")
+            if target and not re.search(r"^%s\s*:" % re.escape(target), self.repo.read(mk[0]), re.M):
+                return "no %s target %r" % (words[0], target)
+            return None
+        program = next((w for w in words if w not in LAUNCHERS and not w.startswith("-")
+                        and "=" not in w), None)
+        if program is None:
+            return "could not identify the program in %r" % command
+        text = self._text()
+        base = posixpath.basename(program.replace("\\", "/"))
+        stem = posixpath.splitext(base)[0]
+        known = (self.repo.exists(program.lstrip("./")) or any(posixpath.basename(f) == base for f in self.repo.files)
+                 or re.search(r"""(^|[\s"'\[])%s\s*=|['"]%s['"]\s*:""" % (re.escape(stem), re.escape(stem)),
+                              "\n".join(self.repo.read(m) for m in self.repo.manifests()), re.M)
+                 or any(f.split("/")[0] == stem or ("/%s/" % stem) in ("/" + f) for f in self.src))
+        if not known:
+            return "program %r is not provided by this repository" % program
+        missing = [fl for fl in FLAG_RE.findall(command) if fl not in text]
+        if missing:
+            return "flags not found in the code: %s" % ", ".join(missing)
+        return None
+
+    def library(self, symbol: Optional[str]) -> Optional[str]:
+        if not symbol:
+            return "library step without a symbol"
+        name = re.split(r"[.:/#]", symbol.strip().rstrip("()"))[-1]
+        if not name:
+            return "unparseable symbol %r" % symbol
+        rx = re.compile(r"\b(def|class|function|func|fn|const|let|var|interface|type|struct|trait|export\s+\w+)\s+"
+                        r"(\([^)]*\)\s*)?%s\b|\b%s\s*[:=]\s*(function|\(|async)" % (re.escape(name), re.escape(name)))
+        if not any(rx.search(self.repo.read(f)) for f in self.src):
+            return "symbol %r is not defined in the code" % symbol
+        return None
+
+    def env(self, names: List[str]) -> Optional[str]:
+        names = [n for n in names if ENV_NAME.match(n)]
+        if not names:
+            return None
+        code = self._code_text()
+        missing = [n for n in names if n not in code]
+        return "not read by the code: %s" % ", ".join(missing) if missing else None
+
+    # ---- whole answer ------------------------------------------------------
+
+    def grade(self, answer: Dict, live: Optional[List[HttpCall]] = None) -> Grade:
+        g = Grade()
+        for step in answer.get("steps", []):
+            kind = step.get("kind")
+            checks = []
+            if kind == "http":
+                checks.append(self.http(step.get("method"), step.get("path"), live))
+            elif kind == "cli":
+                checks.append(self.cli(step.get("command")))
+            elif kind == "library":
+                checks.append(self.library(step.get("symbol")))
+            elif kind == "config" and not step.get("env_vars"):
+                checks.append("config step names no settings")
+            if step.get("env_vars"):
+                checks.append(self.env(step["env_vars"]))
+            if not checks:
+                continue  # "other" steps can't be verified; they neither help nor hurt
+            g.checked += 1
+            errors = [c for c in checks if c]
+            if errors:
+                g.problems.extend(errors)
+            else:
+                g.grounded += 1
+        missing = [p for p in answer.get("evidence", []) if not self.repo.exists(p.lstrip("./").split(":")[0])]
+        if missing:
+            g.problems.append("cited files do not exist: %s" % ", ".join(missing[:3]))
+        return g

neosloc-0.3.0/neosloc/agentic/judge.py

@@ -0,0 +1,59 @@
+"""The judge role: would a grounded answer actually accomplish the task?
+
+Grounding proves that what the probe named exists. The judge, which may be a
+different model and provider, reads the full source and decides whether the
+steps would work as described (right method, required fields, correct order,
+auth). A probe answer only counts as a success when both agree.
+"""
+from __future__ import annotations
+
+import json
+from typing import Any, Dict
+
+from ..repo import Repo
+from .llm import Budget
+from .loop import run_loop
+from .tasks import Task
+from .workspace import Workspace
+
+VERDICT_TOOL = {
+    "name": "submit_verdict",
+    "description": "Submit your verdict on the proposed integration once you have checked it against the code.",
+    "strict": True,
+    "input_schema": {
+        "type": "object", "additionalProperties": False,
+        "required": ["works", "reason", "issues"],
+        "properties": {
+            "works": {"type": "boolean",
+                      "description": "True if following the steps as written would accomplish the task."},
+            "reason": {"type": "string"},
+            "issues": {"type": "array", "items": {"type": "string"},
+                       "description": "Concrete problems: wrong paths, missing fields, missing auth, wrong order."},
+        },
+    },
+}
+
+SYSTEM = """You review integration instructions written by another engineer for the software \
+in this repository. You can read the whole repository, implementation included.
+
+Check the proposed steps against the code: do the routes, commands, settings and symbols \
+behave the way the steps assume, are required inputs and authentication covered, and would \
+following the steps in order accomplish the task? Minor omissions a competent engineer \
+would fill in without reading the code don't make an answer wrong. Then call submit_verdict."""
+
+
+class Judge:
+    def __init__(self, repo: Repo, backend: Any, budget: Budget, max_turns: int = 20):
+        self.repo, self.backend, self.budget, self.max_turns = repo, backend, budget, max_turns
+
+    def judge(self, task: Task, answer: Dict) -> Dict:
+        ws = Workspace(self.repo, "source")
+        tools = [t for t in ws.tools() if t["name"] != "submit_result"] + [VERDICT_TOOL]
+        conv = self.backend.conversation(SYSTEM, tools)
+        prompt = "Task given to the engineer:\n%s\n\nTheir answer:\n%s" % (
+            task.prompt, json.dumps(answer, indent=2, sort_keys=True))
+        res = run_loop(conv, prompt, ws.run, VERDICT_TOOL, self.budget, self.max_turns)
+        verdict = dict(res.answer) if res.answer else {"works": None, "reason": "judge did not decide (%s)"
+                                                                         % res.outcome, "issues": []}
+        verdict.update({"model": self.backend.label, "turns": res.turns, "cost_usd": res.cost_usd})
+        return verdict