nasde-toolkit 0.3.2__py3-none-any.whl

nasde_toolkit/__init__.py

@@ -0,0 +1,5 @@
+"""Noesis Agentic Software Development Evals Toolkit."""
+
+from nasde_toolkit._version import __version__
+
+__all__ = ["__version__"]

nasde_toolkit/_bundled_skills/nasde-benchmark-creator/SKILL.md

@@ -0,0 +1,321 @@
+---
+name: nasde-benchmark-creator
+description: |
+  Create coding agent benchmarks for evaluation with nasde. Use this skill when the user wants to:
+  - Create a new benchmark project (set of tasks for evaluating coding agents)
+  - Add tasks to an existing benchmark
+  - Create or modify agent variants (configurations that control agent behavior)
+  - Set up assessment dimensions and scoring criteria
+  - Verify that a new benchmark's Docker environment and tests work
+  Even if the user doesn't say "benchmark" — if they're talking about creating coding challenges for AI agents or setting up evaluation criteria, this skill applies.
+---
+
+# NASDE Benchmark Creator
+
+Create and configure coding agent benchmarks for evaluation with `nasde`. A benchmark is a set of coding tasks that AI agents solve inside isolated Docker containers, scored both by functional tests (pass/fail) and by an LLM-as-a-Judge architecture assessment.
+
+## Step 1: Understand what to evaluate
+
+Before creating files, clarify with the user:
+
+- What programming language/framework? (determines Dockerfile base image)
+- What kind of coding challenges? (feature implementation, refactoring, bug fixing, etc.)
+- What source repository should the agent work on? (git URL cloned in Dockerfile)
+- What quality dimensions should be assessed? (these are benchmark-specific, not hardcoded)
+
+## Step 2: Scaffold or create the project
+
+For a new benchmark, run:
+
+```bash
+nasde init my-benchmark --name my-benchmark
+```
+
+This creates the base structure. Then customize the generated files.
+
+For adding tasks to an existing benchmark, skip to Step 4.
+
+## Step 3: Define assessment dimensions
+
+Edit `assessment_dimensions.json`. Each benchmark has its OWN dimensions — design them for what matters in this benchmark's domain.
+
+Examples by domain:
+- **Refactoring**: `code_clarity`, `test_preservation`, `api_compatibility`, `performance_impact`
+- **API integration**: `error_handling`, `api_usage_correctness`, `test_coverage`, `documentation`
+- **Security**: `vulnerability_detection`, `fix_correctness`, `regression_safety`, `explanation_quality`
+- **DDD**: `domain_modeling`, `architecture_compliance`, `extensibility`, `test_quality`
+
+Rules:
+- Pick whatever number of dimensions actually captures the quality you care about — there is no required minimum or maximum.
+- Each dimension declares its own `max_score` (any positive integer). Scales are independent — a coarse pass/fail-ish dimension can be 0–3 while a richly graded one can be 0–50 in the same rubric. There is no requirement for the total to sum to 100. `normalized_score` is computed automatically from the actual sum of `max_score` values. See ADR-008.
+- Names in snake_case
+- Each dimension has: `name`, `title`, `max_score`, `description`
+
+## Step 4: Create task files
+
+Each task lives in `tasks/<task-name>/` and needs these files:
+
+### task.toml (required — single task config)
+
+Single config file per task, shared with Harbor. nasde-specific fields live under `[nasde.*]`.
+
+```toml
+version = "1.0"
+
+[task]
+name = "<benchmark-name>/<task-name>"   # Harbor requires org/name format
+description = "Brief description"
+
+[metadata]
+difficulty = "intermediate"
+language = "C#"
+framework = ".NET 8"
+domain = "E-Commerce"
+
+[agent]
+timeout_sec = 1800          # Primary agent timeout. Rule of thumb: estimated_time_minutes × 60.
+
+[environment]
+memory_mb = 4096            # Container memory limit. Claude Code needs 4096+, default 2048 is too low.
+
+[verifier]
+timeout_sec = 300           # Timeout for tests/test.sh.
+
+[nasde.source]              # Only needed when task has no environment/Dockerfile (nasde auto-generates one).
+git = "https://github.com/org/repo.git"
+ref = "main"
+```
+
+**Timeout priority**: `--timeout` CLI flag > task.toml `[agent] timeout_sec` > Harbor default. Timeouts are per-task — there is no project-wide default in nasde.toml.
+
+### instruction.md (required)
+
+Agent-facing task description. Structure it as:
+
+```markdown
+# Task: <Name>
+
+## Context
+Working environment, codebase location (/app), technology stack.
+
+## Requirement
+What the agent must implement/fix/change. Concrete examples with inputs and expected outputs.
+
+## Scope
+What's in scope, what's not.
+
+## Quality Expectations
+Architecture and code quality expectations.
+
+## Success Criteria
+Numbered list matching what test.sh verifies.
+
+## Constraints
+What the agent must NOT do (e.g., don't modify existing tests).
+```
+
+### environment/Dockerfile (required)
+
+```dockerfile
+FROM <base-image>
+
+RUN apt-get update && apt-get install -y git curl wget ca-certificates && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+RUN git clone <repository-url> .
+
+# Pre-install dependencies so the agent doesn't waste time
+RUN <dependency-install-command>
+
+# Verify the environment works
+RUN <build-or-compile-command>
+
+CMD ["/bin/bash"]
+```
+
+The Dockerfile MUST be self-contained — the agent starts working immediately.
+
+### tests/test.sh (required — Harbor verifier)
+
+```bash
+#!/bin/bash
+cd /app
+
+echo "Step 1: Verifying build..."
+if <build-command>; then
+    echo "✓ Build succeeded"
+else
+    echo "✗ Build failed"
+    echo 0 > /logs/verifier/reward.txt
+    exit 1
+fi
+
+echo "Step 2: Running tests..."
+if <test-command>; then
+    echo "✓ Tests pass"
+else
+    echo "✗ Tests failed"
+    echo 0 > /logs/verifier/reward.txt
+    exit 1
+fi
+
+echo "EVALUATION PASSED ✓"
+echo 1 > /logs/verifier/reward.txt
+exit 0
+```
+
+Rules:
+- Every failure: `echo 0 > /logs/verifier/reward.txt` + `exit 1`
+- Final success: `echo 1 > /logs/verifier/reward.txt` + `exit 0`
+- Order steps from fundamental (build) to specific (implementation checks)
+
+### assessment_criteria.md (required for LLM-as-a-Judge evaluation)
+
+Per-task rubric. Structure:
+
+```markdown
+# Assessment Criteria: <Task Name>
+
+Evaluate across N dimensions. Each dimension uses its own scale (0–`max_score`),
+defined in `assessment_dimensions.json`. The ladder below shows what each score
+means for one specific dimension — repeat for each dimension.
+
+## 1. <Dimension Name> (0–<max_score>)
+
+| Score | Criteria |
+|-------|----------|
+| 0           | <worst case> |
+| <middle>    | <middle case> |
+| <max_score> | <best case> |
+
+**Key checks:**
+- Specific things to look for
+```
+
+Pick a `max_score` that matches the granularity you can actually distinguish.
+A coarse pass/fail-ish dimension might be 0–3; a richly graded one might be 0–50.
+Choose the resolution per dimension, independently.
+
+### solution/solve.sh (optional)
+
+Reference solution for verifying test.sh works. Not executed by Harbor.
+
+## Step 5: Create variants
+
+Each variant is a directory under `variants/<variant-name>/` with a required `variant.toml` declaring the agent type.
+
+### variant.toml (required)
+
+```toml
+agent = "claude"   # or "codex" or "gemini"
+```
+
+For Codex variants, **always** set the model explicitly to avoid inheriting the Claude model from `nasde.toml`:
+
+```toml
+agent = "codex"
+model = "gpt-5.3-codex"   # Required for Codex — use an OpenAI model ID
+```
+
+**Codex models** (recommended first, as of 2026-03):
+- `gpt-5.4` — flagship frontier model, best overall for professional work
+- `gpt-5.4-mini` — fast, efficient mini model for responsive coding and subagents
+- `gpt-5.3-codex` — industry-leading coding model for complex software engineering
+- `gpt-5.3-codex-spark` — near-instant real-time coding iteration (ChatGPT Pro only)
+- Older: `gpt-5.2-codex`, `gpt-5.1-codex`, `gpt-5-codex`, `gpt-5-codex-mini`
+
+Without `model` in variant.toml, Codex inherits `nasde.toml`'s default (e.g. `claude-sonnet-4-6`), which silently produces garbage results.
+
+For Gemini CLI variants, **always** set the model with the `google/` prefix:
+
+```toml
+agent = "gemini"
+model = "google/gemini-3-flash-preview"   # Required format: google/<model-name>
+```
+
+**Gemini models** (recommended first, as of 2026-03):
+- `google/gemini-3.1-pro-preview` — advanced thinking model, best for deep reasoning
+- `google/gemini-3-flash-preview` — best quality/speed ratio, daily coding tasks
+- `google/gemini-3.1-flash-lite-preview` — fastest, simple and repetitive tasks
+
+### Claude Code variant
+
+```
+variants/vanilla/
+  variant.toml       # agent = "claude"
+  CLAUDE.md          # Instructions (injected to /app/CLAUDE.md)
+  skills/            # Optional: skill snapshots (injected to /app/.claude/skills/)
+```
+
+### Codex variant
+
+```
+variants/codex-baseline/
+  variant.toml       # agent = "codex"
+  AGENTS.md          # Instructions (injected to /app/AGENTS.md)
+  agents_skills/     # Optional: skill snapshots (injected to /app/.agents/skills/)
+```
+
+### Gemini CLI variant
+
+```
+variants/gemini-baseline/
+  variant.toml       # agent = "gemini"
+  GEMINI.md          # Instructions (injected to /app/GEMINI.md)
+  gemini_skills/     # Optional: skill snapshots (injected to /app/.gemini/skills/)
+```
+
+If no `harbor_config.json` exists, `nasde` auto-generates one from `variant.toml`. To customize (e.g., add MCP servers), create it explicitly:
+
+```json
+{
+  "agents": [
+    {
+      "import_path": "nasde_toolkit.agents.configurable_claude:ConfigurableClaude",
+      "name": "<variant-name>",
+      "kwargs": {
+        "sandbox_files": {
+          "/app/CLAUDE.md": "/absolute/path/to/variants/<variant>/CLAUDE.md",
+          "/logs/agent/sessions/.claude.json": "/absolute/path/to/variants/<variant>/claude_config.json"
+        }
+      }
+    }
+  ]
+}
+```
+
+Critical: `"name"` field is REQUIRED — without it, Opik tagging breaks.
+
+### Variant design patterns
+
+Design variants to test specific hypotheses:
+- **Minimal** (baseline) — bare instructions, no extra guidance
+- **Guided** — detailed domain-specific guidance, patterns to follow
+- **Skill-augmented** — skills injected for domain expertise (e.g., tactical DDD)
+- **Tool-augmented** — MCP server access (e.g., codebase search)
+- **Cross-agent** — same instructions for Claude, Codex, and Gemini to compare agent performance
+
+Every benchmark needs at least one variant (typically `vanilla` or `baseline`).
+
+## Step 6: Verify the benchmark works
+
+Before running with a real agent:
+
+1. **Build the Docker image:**
+   ```bash
+   docker build -t benchmark-test -f tasks/<task>/environment/Dockerfile .
+   ```
+
+2. **Test the verifier** with reference solution (if available):
+   ```bash
+   docker run --rm -it benchmark-test bash
+   # Inside container:
+   bash /path/to/solution/solve.sh
+   bash /path/to/tests/test.sh
+   cat /logs/verifier/reward.txt  # Should be 1
+   ```
+
+3. **Dry run** on a single task:
+   ```bash
+   nasde run --variant vanilla --tasks <task-name> --without-eval -C .
+   ```

nasde_toolkit/_bundled_skills/nasde-benchmark-from-history/SKILL.md

@@ -0,0 +1,298 @@
+---
+name: nasde-benchmark-from-history
+description: |
+  Generate benchmark tasks from git history of the current or specified repository. Use this skill when the user wants to:
+  - Create benchmark tasks based on real problems their team already solved (closed PRs, past commits, resolved issues)
+  - Mine git history for good evaluation candidates
+  - Turn a commit range or set of PRs into a NASDE benchmark
+  - Build a regression test suite from their team's actual work
+  Even if the user doesn't say "benchmark" — if they're talking about turning past work into evaluation tasks, or want to test AI agents against problems they've already solved, this skill applies.
+---
+
+# NASDE Benchmark from Git History
+
+Generate NASDE benchmark tasks by mining git history. You analyze commits, diffs, and PR descriptions to identify self-contained changes that make good evaluation candidates, then generate task files with user approval.
+
+## Prerequisites
+
+- A git repository with meaningful commit history (the repo you're currently in, or a path to another local repo)
+- An existing NASDE benchmark project (run `nasde init` first, or use the `nasde-benchmark-creator` skill)
+- If the benchmark project doesn't exist yet, create it first — this skill generates tasks, not the project scaffold
+
+## Step 1: Identify the source repository and commit range
+
+Ask the user:
+
+- **Which repository?** Default: the current working directory. Can also be a path to another local repo.
+- **What commit range?** Options:
+  - A branch name (analyze all commits on that branch)
+  - A commit range (`abc123..def456`)
+  - Last N commits (`HEAD~20..HEAD`)
+  - Specific PR numbers (if the repo has a GitHub remote, use `gh pr view`)
+  - "Just show me good candidates" — scan the last 50 commits and filter
+
+If the user says "just find good candidates," proceed to Step 2 with the last 50 commits.
+
+## Step 2: Scan commits and identify candidates
+
+For each commit in the range, read the diff and evaluate whether it's a good benchmark candidate.
+
+**Good candidates have:**
+- A self-contained change (clear before/after state — one commit or a squashed PR)
+- A well-defined problem statement (readable from commit message, PR title, or linked issue)
+- Existing tests that can serve as a verifier, OR a change that's testable by inspection
+- Reasonable scope — not too trivial (typo fix) and not too large (multi-week refactor)
+- A clean "before" state — the parent commit should build and run successfully
+
+**Bad candidates (skip these):**
+- Merge commits with no meaningful diff
+- Dependency updates, lockfile changes, CI config tweaks
+- Changes that span too many unrelated files (shotgun surgery)
+- Changes that require external systems not reproducible in Docker (third-party API keys, specific databases with production data)
+
+For each candidate, extract:
+- `before_ref`: the parent commit hash (the state the agent will start from)
+- `after_ref`: the commit hash (the reference solution)
+- `description`: what the change does (from commit message / PR description)
+- `files_changed`: list of modified files
+- `has_tests`: whether the commit includes test changes
+- `estimated_difficulty`: easy / intermediate / hard (based on diff size and complexity)
+
+## Step 3: Present candidates to the user
+
+Present a numbered list of candidates. For each one, show:
+
+```
+[1] abc1234 — "Add discount calculation for threshold-based pricing"
+    Files: src/Pricing/ThresholdDiscount.cs, tests/Pricing/ThresholdDiscountTests.cs
+    Difficulty: intermediate | Has tests: yes
+    Before: abc1233 (parent commit)
+
+[2] def5678 — "Fix race condition in order processing pipeline"
+    Files: src/Orders/OrderProcessor.cs, src/Orders/OrderLock.cs
+    Difficulty: hard | Has tests: yes
+    Before: def5677 (parent commit)
+
+[3] ...
+```
+
+Ask the user to select which candidates to turn into tasks (comma-separated numbers, or "all").
+
+For each selected candidate, proceed to Step 4.
+
+## Step 4: Generate task files for each selected candidate
+
+For each approved candidate, generate the full task directory structure. Work through each file with the user — present it, get approval or edits, then write it.
+
+### 4a: task.toml (single task config, shared with Harbor)
+
+Generate from commit metadata. nasde-specific fields go under `[nasde.*]`.
+
+```toml
+version = "1.0"
+
+[task]
+name = "<benchmark-name>/<slugified-commit-description>"    # Harbor requires org/name format
+description = "<commit message, cleaned up>"
+
+[metadata]
+difficulty = "<estimated_difficulty>"
+language = "<detected-language>"
+framework = "<detected-framework>"
+source_commit = "<after_ref>"
+
+[agent]
+timeout_sec = 1800          # Rule of thumb: estimated_time_minutes × 60
+
+[environment]
+memory_mb = 4096            # Claude Code needs 4096+, default 2048 is too low.
+
+[verifier]
+timeout_sec = 300           # Timeout for tests/test.sh
+
+[nasde.source]              # Only needed when task has no environment/Dockerfile (auto-generation).
+git = "<repo-url-or-local-path>"
+ref = "<before_ref>"
+```
+
+For `[nasde.source] git`:
+- If the repo has a public remote: use the HTTPS clone URL
+- If the repo is local-only (no public remote): use the absolute local path
+- Ask the user if unsure
+
+### 4b: instruction.md
+
+Generate from the commit message, PR description (if available via `gh`), and the diff:
+
+```markdown
+# Task: <Human-readable task name>
+
+## Context
+You are working in a <language/framework> codebase located at `/app`.
+<Brief description of the project and the area of code being modified.>
+
+## Requirement
+<What the agent must implement/fix/change. Derived from the commit message and diff.
+Be specific — describe the expected behavior, not the implementation approach.
+Include concrete examples where possible.>
+
+## Scope
+- Files likely to be modified: <list based on the actual commit diff>
+- Do NOT modify: <files outside the commit's scope, especially tests if they exist>
+
+## Quality Expectations
+<Inferred from the codebase style — mention patterns visible in surrounding code.>
+
+## Success Criteria
+<Numbered list derived from what the commit actually changed and what tests verify.>
+```
+
+**Important:** The instruction must describe the *problem to solve*, not the *solution*. Don't leak implementation details from the actual commit diff into the instruction. The agent should arrive at a solution independently.
+
+Present the generated instruction to the user for review. They may want to:
+- Remove implementation hints that leak from the diff
+- Add context only they know (business rules, team conventions)
+- Adjust scope (widen or narrow what the agent should touch)
+
+### 4c: environment/Dockerfile
+
+Generate based on the repo's tech stack (detected from files like `package.json`, `*.csproj`, `Cargo.toml`, `requirements.txt`, `go.mod`):
+
+```dockerfile
+FROM <base-image-for-detected-stack>
+
+RUN apt-get update && apt-get install -y git curl wget ca-certificates && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+# Clone at the "before" state — the commit BEFORE the fix
+RUN git clone <repo-url> . && git checkout <before_ref>
+
+# Install dependencies
+RUN <dependency-install-command>
+
+# Verify the environment builds
+RUN <build-command>
+
+CMD ["/bin/bash"]
+```
+
+Base image selection:
+- `.csproj` / `.sln` → `mcr.microsoft.com/dotnet/sdk:8.0`
+- `package.json` → `node:20`
+- `requirements.txt` / `pyproject.toml` → `python:3.12`
+- `Cargo.toml` → `rust:1.78`
+- `go.mod` → `golang:1.22`
+- Other → ask the user
+
+### 4d: tests/test.sh
+
+If the commit includes test files, generate a verifier that runs those tests:
+
+```bash
+#!/bin/bash
+cd /app
+
+echo "Step 1: Verifying build..."
+if <build-command>; then
+    echo "✓ Build succeeded"
+else
+    echo "✗ Build failed"
+    echo 0 > /logs/verifier/reward.txt
+    exit 1
+fi
+
+echo "Step 2: Running tests..."
+if <test-command-targeting-relevant-tests>; then
+    echo "✓ Tests pass"
+else
+    echo "✗ Tests failed"
+    echo 0 > /logs/verifier/reward.txt
+    exit 1
+fi
+
+echo "EVALUATION PASSED ✓"
+echo 1 > /logs/verifier/reward.txt
+exit 0
+```
+
+If the commit does NOT include tests, inform the user and offer options:
+1. Write a test script that checks for the expected changes (file existence, specific patterns in code, API behavior)
+2. Skip the verifier and rely only on assessment evaluation (not recommended)
+3. Write the tests together with the user
+
+### 4e: assessment_criteria.md
+
+Generate a per-task rubric using the benchmark project's `assessment_dimensions.json`. For each dimension, create scoring criteria specific to this task:
+
+```markdown
+# Assessment Criteria: <Task Name>
+
+Evaluate the agent's solution across the following dimensions.
+
+## 1. <Dimension from assessment_dimensions.json> (0–<max_score>)
+
+| Score | Criteria |
+|-------|----------|
+| 0     | <worst case — specific to this task> |
+| ...   | <intermediate levels> |
+| <max> | <best case — specific to this task> |
+
+**Key checks:**
+- <What to look for in the code, specific to this task and dimension>
+```
+
+Present to the user for review — they know the codebase best and can add nuance.
+
+### 4f: solution/solve.sh (optional)
+
+Offer to generate a reference solution script that applies the actual commit diff:
+
+```bash
+#!/bin/bash
+cd /app
+git cherry-pick <after_ref> --no-commit
+```
+
+Or, if cherry-pick won't apply cleanly, generate a patch-based approach:
+
+```bash
+#!/bin/bash
+cd /app
+git diff <before_ref> <after_ref> | git apply
+```
+
+This is useful for verifying that `test.sh` passes on a known-good solution.
+
+## Step 5: Verify generated tasks
+
+After all selected tasks are generated:
+
+1. **Confirm the benchmark project has assessment dimensions** — if `assessment_dimensions.json` is missing or empty, prompt the user to define dimensions (delegate to `nasde-benchmark-creator` Step 3).
+
+2. **Build and test each Docker image:**
+   ```bash
+   docker build -t benchmark-test-<task-name> -f tasks/<task-name>/environment/Dockerfile .
+   ```
+
+3. **If solution/solve.sh exists, validate the verifier:**
+   ```bash
+   docker run --rm -v $(pwd)/tasks/<task-name>/solution:/solution \
+     -v $(pwd)/tasks/<task-name>/tests:/tests \
+     benchmark-test-<task-name> bash -c "
+       bash /solution/solve.sh && mkdir -p /logs/verifier && bash /tests/test.sh
+     "
+   ```
+   Expected: exit 0 and `reward.txt` contains 1.
+
+4. **Dry run** on a single task:
+   ```bash
+   nasde run --variant <any-variant> --tasks <task-name> --without-eval -C <benchmark-dir>
+   ```
+
+## Tips
+
+- **Start small.** Pick 3–5 candidates for the first pass. You can always add more later.
+- **Prefer commits with tests.** Tasks with existing tests are much faster to set up — the verifier almost writes itself.
+- **Don't leak the solution.** The biggest risk in generating instructions from diffs is accidentally describing HOW the problem was solved. Describe the WHAT and WHY, not the HOW.
+- **Local repos work fine.** NASDE supports local git paths in `source.git`. No need to push to a public remote for company repos.
+- **Combine with nasde-benchmark-creator.** This skill generates tasks; `nasde-benchmark-creator` handles the project scaffold, dimensions, and variants. Use them together.