nighthawk-python 0.4.0__tar.gz → 0.6.0__tar.gz

nighthawk_python-0.6.0/.claude/rules/docs.md

@@ -0,0 +1,232 @@
+---
+paths:
+  - "docs/**/*.md"
+---
+
+# Documentation rules
+
+## Canonical ownership
+
+Each topic must have exactly one canonical owner file. Other files may restate the topic only as a deliberate derivative summary, quickstart, or routing pointer.
+
+General rules:
+
+- Prefer cross-references over duplication.
+- If a topic appears in multiple files, one file must be the declared source of truth.
+- Derivative documents may compress or subset canonical content when that serves a distinct audience.
+- `for-coding-agents.md` is the main exception: it is a derivative operational guide for coding agents and may restate material from human-oriented docs.
+
+Public API documentation layering:
+
+- `api.md` owns the exhaustive inventory of the supported public API surface (existence, types, signatures, exceptions, docstrings). Low-level utilities and extension hooks belong here.
+- `specification.md` owns API semantics, contracts, boundaries, and runtime behavior. It is the canonical source for *what a symbol means and how it behaves*.
+- `quickstart.md`, `natural-blocks.md`, `patterns.md`, `runtime-configuration.md`, and `verification.md` are task-oriented. They cover *when to use* and *how to use* selected APIs -- not every public symbol. A public API having no coverage in these pages is expected when the symbol serves a narrow or advanced use case already documented by `api.md` and `specification.md`.
+
+`docs/AGENTS.md` is governance-only content. It is a symlink to `.claude/rules/docs.md` and must not appear as an accidental published governance page. It is excluded from the published site via `exclude_docs` in `mkdocs.yml`.
+
+## File roles and boundaries
+
+| File | Audience | Role | Scope |
+|---|---|---|---|
+| `index.md` | First-time visitors | Landing page | Value proposition, one runnable example, entry path routing. No API, no how-to. |
+| `quickstart.md` | New users | First success | Minimal setup, minimal example, minimal troubleshooting, explicit next-step link. |
+| `natural-blocks.md` | Learners | What Natural blocks are | Natural block anatomy, prompt structure, read/write bindings, Pydantic model bindings, f-string injection, functions and discoverability, binding function design (principles and basic examples), responsibility split, structured output design. |
+| `executors.md` | Learners / evaluators | Choose an execution backend | Capability / cost / latency matrix, decision tree, `StepExecutorConfiguration` basics, and routing to side references and `runtime-configuration.md`. |
+| `runtime-configuration.md` | Learners | Configure execution | `nh.run()`, `nh.scope()`, configuration patching, prompt suffix fragments, context limits, JSON rendering style, and runtime execution identity. |
+| `patterns.md` | Practitioners | Apply Natural blocks in workflows | Outcomes, deny frontmatter, error handling, custom exception types, async, carry pattern, cross-block composition, resilience patterns, and common mistakes. |
+| `verification.md` | Practitioners | Verify and debug | Mock tests, integration tests, prompt inspection, diagnosing snipped markers, OpenTelemetry span hierarchy, step events, local trace inspection. |
+| `pydantic-ai-providers.md` | Model configurers | Pydantic AI provider reference | Provider list, installation, model identifiers, credentials, model settings, provider-specific troubleshooting. No chooser. No custom backends. |
+| `coding-agent-backends.md` | Backend users | Backend reference | Backend-specific settings, shared capabilities, skills, MCP tool exposure, working directory, troubleshooting. |
+| `for-coding-agents.md` | Coding agents (LLMs) | Operational guide | Condensed, decision-oriented rules derived from human-oriented docs. Self-contained with absolute URLs. |
+| `specification.md` | Implementors | Canonical spec | Syntax, state layers, tools, outcomes, frontmatter, runtime semantics, observability contract, and custom backend capability/protocol semantics. Numbered section headings. |
+| `philosophy.md` | Evaluators | Design rationale | Execution model, harness evidence, design consequences (resilience, scoped execution contexts, tool exposure, multi-agent coordination, tradeoffs), runtime evaluation rationale, design landscape. |
+| `api.md` | Developers | API reference | Auto-generated from docstrings, including protocol and extension-hook symbols. |
+| `roadmap.md` | Contributors | Future directions | Ideas only. Remove when implemented. |
+| `docs/AGENTS.md` | Coding agents editing docs | Documentation governance | Canonical ownership, page roles, routing rules, and docs test invariants. Symlink to `.claude/rules/docs.md`. |
+
+## Content routing
+
+List only topics that commonly drift across multiple files or are easy to misplace.
+
+- **Credentials** -> `quickstart.md` (minimal first run), `pydantic-ai-providers.md` (Pydantic AI providers), `coding-agent-backends.md` (backend prerequisites)
+- **Executor selection** -> `executors.md` (capability and cost tradeoffs), `coding-agent-backends.md` (backend behavior and constraints), `for-coding-agents.md` (block-level operational guidance), `quickstart.md` (minimal entry example only)
+- **Runtime setup and scoping** (`nh.run()`, `nh.scope()`, configuration patching) -> `runtime-configuration.md` (canonical), `patterns.md` (usage-only references), `for-coding-agents.md` (condensed)
+- **Context limits / JSON rendering / execution identity** -> `runtime-configuration.md` (canonical), `verification.md` (`<snipped>` diagnosis only), `specification.md` (formal semantics), `for-coding-agents.md` (condensed)
+- **Bindings** -> `natural-blocks.md` (canonical), `specification.md` (formal definition), `for-coding-agents.md` (condensed)
+- **Binding function design** -> `natural-blocks.md` (principles and basic examples), `patterns.md` (only when binding functions participate in multi-block patterns), `for-coding-agents.md` (condensed)
+- **Resilience** -> `patterns.md` (canonical patterns), `for-coding-agents.md` (condensed operational rules), `philosophy.md` (positioning only)
+- **Testing** -> `verification.md` (canonical patterns and examples), `for-coding-agents.md` (condensed operational rules), `specification.md` (testing is out of scope except for boundary statements)
+- **Observability** -> `verification.md` (usage and debugging workflow), `specification.md` (specification and runtime semantics), `for-coding-agents.md` (normally omit; mention only when needed to explain execution constraints)
+- **Deny frontmatter** -> `patterns.md` (standard patterns), `specification.md` (canonical specification), `for-coding-agents.md` (condensed operational rules)
+- **Coding agent control** -> `philosophy.md` (execution model and design consequences), `coding-agent-backends.md` (configuration and constraints)
+- **Coding agent backend impact** -> `philosophy.md` (execution model and design consequences), `index.md` (brief summary), `coding-agent-backends.md` (details)
+- **Async** -> `patterns.md` (patterns), `specification.md` (specification), `for-coding-agents.md` (condensed rules)
+- **Structured output / Pydantic models** -> `natural-blocks.md` (design guidelines), `specification.md` (type validation specification)
+- **Custom backends** -> `specification.md` (semantics), `api.md` (protocol symbols), `executors.md` (chooser-level mention only)
+- **Workflow engine comparison** -> `philosophy.md` (canonical, design landscape section), `index.md` (link), `executors.md` (link)
+- **Tool exposure tradeoffs** -> `philosophy.md` (canonical, design consequences section), `index.md` (link), `executors.md` (link)
+- **Docs governance** -> `docs/AGENTS.md` and `.claude/rules/docs.md`. No derivative restatement elsewhere.
+
+## Shared writing guidelines
+
+### General
+
+- Headings: sentence case. Capitalize first word, proper nouns (Nighthawk, Natural, Pydantic), acronyms (LLM, JSON, MCP).
+- Anchors: name-based (`#writing-guidelines`), not number-based (`#1-writing-guidelines`). Exception: `specification.md` is a specification document and may use numbered section headings as its stable citation hierarchy.
+- Cross-references: relative links. Exception: `for-coding-agents.md` uses absolute URLs from `site_url`.
+- Terminology: "task" = structural unit (contract), "judgment" = cognitive act. Use "one task per block".
+- Code examples: self-contained and understandable without surrounding prose.
+- Built-in tools (`nh_eval`, `nh_assign`): implementation details. Only `specification.md` may expose them.
+- `@nh.tool`: `specification.md` documents as spec, `natural-blocks.md` may mention it with a "prefer binding functions" note, all others must not reference it.
+- Package name: always `nighthawk-python` in `pip install` commands.
+- When renaming a document or changing its role, update all inbound references, routing rules here, relevant `tests/docs`, and navigation metadata if applicable.
+- When a governance file under `docs/` is not meant for publication, its MkDocs handling must be explicit.
+- `natural-blocks.md` and `patterns.md` together replace the former `guide.md`.
+- References to `design.md` should use `specification.md`.
+- References to `providers.md` should use `pydantic-ai-providers.md`.
+
+### Prerequisite notes
+
+Pages in the Getting started, Patterns & verification, and Configuration nav groups must open with a short prerequisite note. This supports non-linear readers who jump directly to a topic. The note should be one sentence naming the assumed prior reading. Pages in Reference, Background, and Project groups (`specification.md`, `philosophy.md`, `roadmap.md`, `api.md`) are exempt -- they serve independent audiences that do not follow the learning path.
+
+## Per-file rules
+
+**`index.md`**
+
+- Links list must stay in sync with `nav` in `mkdocs.yml`.
+- One representative code example (Python + Natural block + binding function). The example must be self-contained and runnable (include executor setup and `nh.run()` context).
+- Brief positioning summaries linking to `philosophy.md`. No comparisons or benchmarks.
+
+**`quickstart.md`**
+
+- Optimize for copy-paste.
+- Include only the minimum needed for a first success.
+- Retain one explicit sentence stating the trust model / hard constraint for Natural blocks and imported markdown.
+- End with a next-page link to `natural-blocks.md`.
+- No backend alternatives beyond a one-line link to `executors.md`.
+
+**`natural-blocks.md`**
+
+- Prerequisite note: "This page assumes you have completed [Quickstart](quickstart.md)."
+- Owns Natural block anatomy, prompt structure, binding semantics, discoverability, binding function design (principles and basic examples), responsibility split, and structured output design guidelines.
+- Binding function design includes principles and basic examples that are complete within a single block. Advanced multi-block patterns (carry, branching, resilience) belong in `patterns.md`.
+- Owns migrated `prompt-example` test anchors (`basic-binding`, `fstring-injection`, `local-function-signature`, `global-function-reference`). Exception: `carry-pattern` belongs in `patterns.md`.
+- Backend-agnostic: examples use the Quickstart default executor.
+- Cross-reference `specification.md` for formal definitions.
+- Ends with a routing sentence: "Choosing an executor is in [Executors](executors.md). Runtime configuration (`nh.run()`, `nh.scope()`, limits) is in [Runtime configuration](runtime-configuration.md)."
+
+**`executors.md`**
+
+- Prerequisite note: "This page assumes you have completed [Quickstart](quickstart.md) and [Natural blocks](natural-blocks.md)."
+- Owns executor selection: capability matrix, decision tree, and `StepExecutorConfiguration` basics.
+- Links to `philosophy.md` for positioning instead of duplicating it.
+- Capability matrix must include relative cost and latency columns.
+- Must include an explicit custom-backend routing subsection with a minimal `AgentStepExecutor.from_agent(agent=agent)` runnable example (3-5 lines). Direct `AsyncStepExecutor` implementation belongs in `specification.md`.
+- Ends with routing: side trips to `pydantic-ai-providers.md` and `coding-agent-backends.md`, then next-step link to `runtime-configuration.md`.
+- Runtime configuration topics (`nh.run()`, `nh.scope()`, configuration patching, prompt suffix, context limits, JSON rendering, execution identity) belong in `runtime-configuration.md`, not here.
+
+**`runtime-configuration.md`**
+
+- Prerequisite note: "This page assumes you have completed [Executors](executors.md)."
+- Owns all runtime configuration: `nh.run()`, `nh.scope()`, configuration patching, prompt suffix fragments, context limits, JSON rendering style, and runtime execution identity.
+- These topics are independent of executor choice. The page applies equally to Pydantic AI providers and coding agent backends.
+- Cross-reference `specification.md` for formal semantics.
+- Ends with next-step link to `patterns.md`.
+
+**`patterns.md`**
+
+- Prerequisite note: "This page assumes you have completed [Natural blocks](natural-blocks.md) and [Runtime configuration](runtime-configuration.md)."
+- Owns outcomes, deny, async, carry, composition, resilience, and common mistakes.
+- Scope: multi-block coordination and operational patterns. Single-block-complete topics belong in `natural-blocks.md`.
+- Backend-agnostic: no backend-specific file layouts or credentials.
+- Cross-reference `specification.md` for formal definitions.
+
+**`verification.md`**
+
+- Prerequisite note: "Mock testing is readable after [Natural blocks](natural-blocks.md); later sections assume [Patterns](patterns.md)."
+- Owns mock tests, integration tests, prompt inspection, debugging workflow, and OpenTelemetry usage.
+- Normative observability contracts belong in `specification.md`.
+
+**`pydantic-ai-providers.md`**
+
+- Prerequisite note: "See [Executors](executors.md) for choosing between providers, backends, and custom executors."
+- Pure Pydantic AI provider reference: installation, model identifiers, model settings, troubleshooting.
+- No chooser table.
+- No custom backends.
+
+**`coding-agent-backends.md`**
+
+- Prerequisite note: "See [Executors](executors.md) for when to choose a coding agent backend over a provider-backed executor."
+- Reference-first page: minimal orientation only, then backend-specific settings, skills, MCP, working directory, and troubleshooting.
+- Must not become a second chooser page. Capability, latency, cost, and positioning comparisons belong in `executors.md` and `philosophy.md`.
+- Shared capabilities section for common features.
+- Reference `executors.md` for capability and cost comparisons.
+
+**`specification.md`**
+
+- All current specification rules apply, with the new name.
+- Numbered section headings remain stable.
+- Owns custom backend capability/protocol semantics, placed as a subsection under Section 14 (Step executor) to avoid top-level section number disruption.
+- Includes a non-runnable skeletal shape of the `AsyncStepExecutor` protocol surface under Section 14. No runnable implementation example is required (the runnable `from_agent` example lives in `executors.md`).
+
+**`philosophy.md`**
+
+- Owns the cumulative argument: execution model, harness evidence, design consequences (resilience, scoped execution contexts, tool exposure, multi-agent coordination, tradeoffs), runtime evaluation rationale, and design landscape.
+- External references acceptable. Prefer stable URLs with enough inline context to survive link rot.
+- No how-to code examples for patterns in `natural-blocks.md` or `patterns.md`. Exception: positioning examples may reuse function names from those pages.
+
+**`for-coding-agents.md`**
+
+- Reader is a coding agent. Write for immediate applicability with runnable templates and decision rules.
+- Information flows from human-oriented docs only; never introduce new product behavior here first.
+- May derive from `natural-blocks.md`, `patterns.md`, `runtime-configuration.md`, `verification.md`, `specification.md`, `pydantic-ai-providers.md`, and `coding-agent-backends.md`.
+- Self-contained with absolute URLs from `site_url` (`https://kurusugawa-computer.github.io/nighthawk-python/`).
+- Prefer decision rules over encyclopedic coverage.
+- Recommend provider-backed executors by default and coding agent backends only for blocks that need autonomous long-horizon work.
+- Keep trust-model constraints explicit.
+- Condensation policy: compress tables and lists to inline summaries or subsets with links to canonical docs. Verbatim duplication only for compact, self-contained content.
+- Common mistakes: subset of most impactful items with link to fuller guidance.
+- Include resilience and scoped overrides.
+- Omit observability except when needed to explain execution constraints.
+- Omit exception hierarchy beyond `ExecutionError` unless a narrower rule is essential for safe coding.
+- Published as a derivative operational reference under `Reference` in nav, not as a top-level learner-path peer.
+- Absolute URLs use topic-based canonical owner mapping:
+  - Bindings, block anatomy, responsibility split, binding function design -> `/natural-blocks/`
+  - Carry, deny, async, resilience, multi-block composition -> `/patterns/`
+  - Executor selection, `StepExecutorConfiguration` basics -> `/executors/`
+  - `nh.run()`, `nh.scope()`, configuration patching, context limits, JSON rendering, execution identity -> `/runtime-configuration/`
+  - Provider-specific setup -> `/pydantic-ai-providers/`
+  - Coding agent backend config -> `/coding-agent-backends/`
+  - Spec references -> `/specification/`
+
+**`api.md`**
+
+- Exhaustive inventory of the supported public API surface. Every supported public symbol should appear here.
+- Auto-generated from source docstrings. Hand-editing limited to `:::` directive structure.
+- Use `members` filters to avoid duplicate rendering.
+- A symbol appearing only in `api.md` (with no learner-facing page coverage) is acceptable. Task-oriented docs select symbols by pedagogical value, not completeness.
+
+**`roadmap.md`**
+
+- Future-facing only. Remove items once implemented.
+- Each item should reference the relevant `specification.md` section where that helps maintain traceability.
+
+**`docs/AGENTS.md`**
+
+- Is a symlink to `.claude/rules/docs.md`. No separate synchronization step needed.
+- Must not appear as an accidental published governance page. Exclude via `exclude_docs` in `mkdocs.yml`, listing `AGENTS.md` explicitly by filename.
+
+## Documentation test invariants
+
+- Treat executable or doctrinal claims in docs as testable when practical.
+- `prompt-example` anchors live in `natural-blocks.md` by default (`basic-binding`, `fstring-injection`, `local-function-signature`, `global-function-reference`). Exception: `carry-pattern` lives in `patterns.md` (cross-block composition). Test file: `tests/docs/test_prompt_examples.py`.
+- `for-coding-agents.md` operational examples and core doctrine are guarded by `tests/docs/test_coding_agent_examples.py`. Update the tests when changing executable guidance or non-negotiable rules.
+- When a docs change invalidates an existing test, first decide whether the docs or the test is the canonical truth for that claim, then update both sides to match.
+- Docs architecture regression tests (`tests/docs/test_docs_architecture.py`):
+  - Fail on stale references to deleted/renamed docs (`guide.md`, `design.md`, `providers.md`).
+  - Guard the canonical example relationship between `index.md` and `README.md` via fenced-code-block extraction + normalized exact match.
+  - Guard selected canonical-owner expectations where drift is likely.
+  - Automate `mkdocs.yml` nav entries vs `docs/` file existence as a pytest case.
+  - Guard that obsolete canonical pages do not remain published accidentally alongside their replacements.
+- `mkdocs build` must succeed without `--strict`. Warnings are acceptable.
+- All internal relative links must resolve.

nighthawk_python-0.6.0/.claude/rules/promptfoo.md

@@ -0,0 +1,137 @@
+---
+paths:
+  - "evals/promptfoo/**"
+---
+
+# Prompt evaluation (promptfoo)
+
+## Directory roles
+
+| Directory | Purpose | Determinism | Cost |
+|---|---|---|---|
+| `evals/promptfoo/` | Prompt experimentation: system prompt variants, tool descriptions, suffix fragments, backend comparison | Non-deterministic; use `--repeat N` to measure stability. | API calls x N x providers. |
+| `evals/promptfoo/outputs/` | Transient raw output (gitignored). Working files for in-progress analysis. | N/A | N/A |
+| `evals/promptfoo/evidence/` | Committed eval evidence. Decision rationale for adopted/rejected variants. | N/A | N/A |
+
+## Prompt changes workflow
+
+1. Edit eval-layer prompts or provider config in `evals/promptfoo/`.
+2. Run eval: `eval $PFOO eval --filter-providers "<provider>" --no-cache`.
+3. Compare against previous eval in the promptfoo DB or JSON output.
+4. Once validated, port the change to the corresponding production code (see mapping below).
+5. Run `uv run pytest -q` to confirm no regressions.
+
+**`--filter-providers` caveat**: The flag takes a regex pattern, not an exact label. A pattern like `"gpt-5.4-mini"` matches every provider whose label contains that substring (e.g. both `openai-responses` and `codex` labels). Use an anchored or label-specific pattern (e.g. `"^gpt-5.4-mini"`, `"codex:"`) to target a single provider.
+
+## Prompt variant cleanup (deletion timing and criteria)
+
+- Delete rejected prompt variants immediately when any of the following is true:
+  - Deterministic regression (`N/N` failures across repeats, e.g. `--repeat 3`)
+  - Clear degradation versus baseline on primary metrics
+  - Temporary/scratch variants created for quick checks
+- Keep a variant only if it has clear re-test value (e.g. flaky `1/N` behavior or meaningful metric trade-offs).
+- Retention for kept variants is limited to `min(7 days, 2 experiment cycles)`.
+  - If not re-evaluated within this window, delete it.
+- Run cleanup at three checkpoints:
+  1. Right after each experiment run
+  2. Before opening a PR
+  3. Before merge (final sweep)
+- Before deleting, record a short rejection reason in the corresponding `evals/promptfoo/evidence/` file; do not keep dead prompt files just for history.
+
+## Backend prerequisites
+
+| Backend | Requirement |
+|---|---|
+| `openai-responses` | `OPENAI_API_KEY` environment variable. |
+| `codex` | Pre-authenticated `codex` CLI (`codex login`) or `CODEX_API_KEY` environment variable. |
+| `claude-code-cli` | Pre-authenticated `claude` CLI (`claude login`) or `ANTHROPIC_API_KEY` environment variable. |
+| `claude-code-sdk` | `ANTHROPIC_API_KEY` environment variable. |
+
+Evals that include a backend without its prerequisite will run but produce errors for that backend's test cases. Use `--filter-providers` to exclude unavailable backends.
+
+## Eval-to-production mapping
+
+Eval prompts are experimental copies of production code. Keep them in sync; divergence means eval results do not predict production behavior.
+
+| Eval file | Production counterpart |
+|---|---|
+| `evals/promptfoo/prompts/eval_default.txt` | `configuration.py:DEFAULT_STEP_SYSTEM_PROMPT_TEMPLATE` |
+| `evals/promptfoo/prompts/eval_coding_agent.txt` | No single counterpart; coding agent backends receive this prompt via `system_prompt_file` config. |
+| Suffix variants in `evals/promptfoo/provider.py` (`_build_suffix_*`) | `step_contract.py:build_step_system_prompt_suffix_fragment` |
+| Tool presets in `evals/promptfoo/provider.py` (`_build_tool_preset`) | `tools/registry.py` + `tools/assignment.py` |
+
+## Eval evidence
+
+### When to save evidence
+
+| Save | Do not save |
+|---|---|
+| Eval that decided adoption or rejection of a prompt/suffix/tool variant | In-progress trial runs during development |
+| Regression baseline update | Single-test filter runs |
+| Eval backing a change merged via PR | Transient output already in `outputs/` |
+
+### File path convention
+
+```
+evals/promptfoo/evidence/{YYYY-MM-DD}-{experiment-slug}.md
+```
+
+Examples: `2026-03-20-suffix-ab.md`, `2026-03-25-regression-v2.md`.
+
+### File format
+
+```markdown
+---
+eval_id: <promptfoo eval ID>
+config: <YAML config file used>
+date: YYYY-MM-DD
+decision: <one-line summary of what was adopted/rejected>
+---
+
+## Providers tested
+- <provider label> (<variants if A/B>)
+
+## Results summary
+| Variant | Pass | Fail | Error | Score | Latency |
+|---------|------|------|-------|-------|---------|
+| ...     |      |      |       |       |         |
+
+## Decision rationale
+<Why the chosen variant was adopted.>
+
+## Rejected variants
+- <variant>: <short rejection reason>
+```
+
+### Rules
+
+- Only Markdown files (`*.md`) are committed under `evidence/`. Raw JSON stays in `outputs/` (gitignored).
+- One file per experiment decision. If a follow-up eval revises a prior decision, create a new file; do not overwrite.
+- Reference the `eval_id` so the full raw result can be retrieved from the promptfoo local DB (`promptfoo view`) or `-o` export if needed.
+
+## Eval interpretation
+
+- **Exit code 100**: promptfoo returns exit code 100 when any test case fails. This is not a system error; it signals "some assertions did not pass". CI scripts and background runners should treat exit 100 as "check results" rather than "eval crashed".
+- **OpenAI 500 errors**: Transient; ignore unless persistent across runs.
+- **Codex CLI errors**: Codex backend is unstable; isolate with `--filter-providers`.
+- **Codex binding-not-returned**: Codex may return `None` for write bindings that `openai-responses` handles correctly. This is a distinct failure mode from flaky LLM non-determinism; it typically indicates the coding-agent backend did not invoke the assignment tool.
+- **Mutation vs filter ambiguity**: Natural language instructions like "remove negative numbers" can be interpreted as either in-place mutation or filter-to-new-list. LLMs inconsistently choose between these, producing flaky failures where the binding value is the original unmodified collection. This pattern recurs across providers and suffix variants.
+- **1/N failures (flaky)**: Inherent LLM non-determinism. Use `--repeat 3` to distinguish from deterministic regressions.
+- **Deterministic failures** (N/N across repeats): Require prompt or code fix before merging.
+
+## Baseline workflow
+
+Run a full baseline when: (a) setting up the eval environment for the first time, (b) after a major production code change, or (c) when prior baselines are stale (> 2 weeks or across model version changes).
+
+1. Verify prerequisites for each backend (see Backend prerequisites above).
+2. Run all applicable configs in parallel with `--no-cache` and `-o outputs/baseline-{slug}.json`:
+   - `promptfooconfig.yaml` — regression across all available backends.
+   - `promptfooconfig-prompt-ab.yaml` — prompt/tool variant comparison.
+   - `promptfooconfig-suffix-ab.yaml` — suffix fragment comparison.
+   - `promptfooconfig-agents.yaml` — coding-agent backends (requires `codex` and `claude` CLIs).
+3. For backends without prerequisites, either skip the config or use `--filter-providers` to exclude unavailable providers.
+4. Create one evidence file per config under `evals/promptfoo/evidence/` with the `baseline-` slug prefix.
+
+## Commands
+
+See `CONTRIBUTING.md` "Prompt evaluation with promptfoo" for eval commands, config files, directory layout, and flags.

nighthawk_python-0.6.0/.claude/rules/tests.md

@@ -0,0 +1,27 @@
+---
+paths:
+  - "tests/**"
+---
+
+# Testing (pytest)
+
+## Directory roles
+
+| Directory | Purpose | Determinism | Cost |
+|---|---|---|---|
+| `tests/` | Pytest suite: unit tests (ScriptedExecutor) and integration tests (real LLM) | Unit: deterministic. Integration: non-deterministic but single-run. | Unit: free. Integration: API calls. |
+| `src/nighthawk/testing.py` | Test utility API for deterministic Natural-function tests (`ScriptedExecutor`, `CallbackExecutor`, and response factories). | Deterministic. | Free. |
+
+## Workflow
+
+### Scope boundary (pytest vs promptfoo)
+
+- Do not force prompt behavior validation into pytest-only checks.
+- When prompt rendering, system prompt text, suffix generation, or tool-exposure behavior changes, follow `.claude/rules/promptfoo.md`.
+
+### Python code changes (tools, executor, contracts)
+
+1. Write or update unit tests in `tests/` first.
+2. Prefer helpers from `nighthawk.testing` (for example `ScriptedExecutor`, `CallbackExecutor`, `pass_response`, `return_response`) when avoiding live LLM calls.
+3. Run `uv run pytest -q`.
+4. If the change affects prompt rendering or tool behavior, follow `.claude/rules/promptfoo.md` and run the relevant eval subset.

{nighthawk_python-0.4.0 → nighthawk_python-0.6.0}/.devcontainer/litellm-config.yaml

@@ -30,7 +30,7 @@ model_list:
       additional_drop_params: ["context_management", "output_config"]
   - model_name: claude-haiku-*
     litellm_params:
-      model: openai/gpt-5-mini
+      model: openai/gpt-5.4-mini
       api_key: os.environ/OPENAI_API_KEY
       additional_drop_params: ["context_management", "output_config"]
 litellm_settings:

{nighthawk_python-0.4.0 → nighthawk_python-0.6.0}/.github/workflows/docs.yml

@@ -32,7 +32,7 @@ jobs:
         run: uv sync --group docs
 
       - name: Build docs
-        run: uv run mkdocs build --strict
+        run: uv run mkdocs build
 
       - uses: actions/upload-pages-artifact@v4
         with:

{nighthawk_python-0.4.0 → nighthawk_python-0.6.0}/.gitignore

@@ -3,6 +3,12 @@
 *.local.json
 .vscode/
 
+# promptfoo evaluation outputs
+evals/promptfoo/outputs/
+evals/promptfoo/*.json
+evals/promptfoo/*.html
+evals/promptfoo/*.csv
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[codz]

{nighthawk_python-0.4.0 → nighthawk_python-0.6.0}/AGENTS.md

@@ -13,7 +13,7 @@
 
 ### Allowed abbreviations (Glossary)
 
-`Id` (Identifier), `DSL` (Domain Specific Language), `LLM` (Large Language Model), `NH`/`nh` (Nighthawk), `max` (maximum), `min` (minimum), loop indices `i`/`j`/`k`.
+`Id` (Identifier), `DSL` (Domain Specific Language), `LLM` (Large Language Model), `NH`/`nh` (Nighthawk), `max` (maximum), `min` (minimum), loop indices `i`/`j`/`k`, type parameters `P` (ParamSpec), `R` (Return type), `T` (Type variable).
 
 ### Rules
 
@@ -37,7 +37,7 @@ Natural DSL sources and included markdown are trusted, repository-managed assets
 
 - `src/nighthawk/`: Library package.
 - `tests/`: Pytest suite.
-- `docs/`: `quickstart.md` (first steps), `tutorial.md` (first principles), `design.md` (specification), `roadmap.md` (future items).
+- `docs/`: User-facing documentation (MkDocs).
 - `.agents/`: `execplans/` (on request only), `PLANS.md` (format spec).
 - `.devcontainer/`: Devcontainer definition.
 - `pyproject.toml`, `uv.lock`: Metadata and locked dependencies.
@@ -51,13 +51,16 @@ Python 3.13+, `uv` for dependencies, `pytest` for tests. Prefer LSP-based toolin
 | `uv run python` | Investigate interactively |
 | `uv sync --all-extras --all-groups` | Install/sync dependencies |
 | `uv run ruff format .` | Format |
-| `uv run ruff check .` | Lint |
 | `uv run ruff check --fix .` | Auto-fix lint |
 | `uv run pyright` | Type check |
-| `uv run pytest` | Full test suite |
 | `uv run pytest -q` | Tests (quiet) |
-| `set -a; source .env; set +a; uv run pytest -q` | Integration tests |
+| `NIGHTHAWK_OPENAI_INTEGRATION_TESTS=1 uv run pytest -q` | Integration tests (OpenAI) |
+| `NIGHTHAWK_CODEX_INTEGRATION_TESTS=1 uv run pytest -q` | Integration tests (Codex) |
+| `NIGHTHAWK_CLAUDE_SDK_INTEGRATION_TESTS=1 uv run pytest -q` | Integration tests (Claude Code SDK) |
+| `NIGHTHAWK_CLAUDE_CLI_INTEGRATION_TESTS=1 uv run pytest -q` | Integration tests (Claude Code CLI) |
 
 `uv` hardlinking warnings do not indicate failure. Suppress: `export UV_LINK_MODE=copy`.
 
 Environment: `OPENAI_API_KEY` (OpenAI), `CODEX_API_KEY` (Codex).
+
+Promptfoo evaluation details (commands, configs, directory layout, flags): see `CONTRIBUTING.md` "Prompt evaluation with promptfoo".

nighthawk_python-0.6.0/CHANGELOG.md

@@ -0,0 +1,101 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- `nighthawk.resilience` module with composable function transformers for production resilience: `retrying` (tenacity-based), `fallback`, `vote`/`plurality`, `timeout`, `circuit_breaker`/`CircuitState`/`CircuitOpenError`.
+  - `tenacity>=9` as a core dependency.
+- `BackendModelSettings` base class and `ClaudeCodeModelSettings` intermediate class extracting shared settings across coding agent backends.
+
+### Changed
+- Refactored backend settings hierarchy: extracted shared fields (`allowed_tool_names`, `working_directory`) into `BackendModelSettings` and Claude Code fields (`max_turns`, `permission_mode`, `setting_sources`) into `ClaudeCodeModelSettings`; renamed `claude_executable`/`codex_executable` to `executable`, `claude_max_turns` to `max_turns`.
+- `nh_assign` now resolves type annotations via `get_type_hints` for plain classes and dataclasses, enabling type-mismatch retry beyond Pydantic models.
+- Simplified intent hint formatting: dropped `intent: ` prefix from callable metadata comments in prompt context.
+- Renamed `NIGHTHAWK_RUN_INTEGRATION_TESTS` to `NIGHTHAWK_OPENAI_INTEGRATION_TESTS` for consistency with other per-backend environment variables.
+- Restructured documentation into sectioned navigation: split monolithic tutorial into focused guides (`natural-blocks`, `executors`, `runtime-configuration`, `patterns`, `verification`, `pydantic-ai-providers`); renamed `design.md` to `specification.md`; removed `practices.md`, `providers.md`, `tutorial.md`.
+
+## [0.5.0]
+
+### Added
+- `evals/promptfoo/` evaluation harness for system prompt optimization using [promptfoo](https://www.promptfoo.dev/): custom Python provider, reusable assertions, and prompt variant A/B comparison support. See `CONTRIBUTING.md` for usage.
+- `docs/philosophy.md`: design philosophy and motivation behind Nighthawk.
+- `docs/practices.md`: practical patterns and binding function design guidance (extracted from tutorial).
+
+### Changed
+- Replaced `return_reference_path` with `return_expression` in step execution contract: return values are now specified as Python expressions evaluated against step locals/globals, consistent with `nh_eval`/`nh_assign` expression evaluation. This unblocks coding-agent backends (e.g. Claude Code CLI) that compute results via native tools without bridging values through `nh_assign`.
+- `nh_assign` now infers binding types from initial values when no explicit annotation is provided, enabling type-mismatch retry for unannotated write bindings.
+- Default `json_renderer_style` changed from `"strict"` to `"default"`, making truncation visible via `…` omission markers in prompt context and tool results.
+- Merged `nh_exec` into `nh_eval`: `nh_eval` now handles expression evaluation, function calls, and in-place mutation. `nh_exec` is removed.
+- Condensed system prompt: simplified tool selection guidance (single `nh_eval` tool), added execution order section, clarified tool result format.
+- Condensed step execution contract (outcome prompt suffix) for reduced token usage.
+- Improved `nh_assign` and `nh_eval` tool descriptions for LLM clarity.
+- Restructured documentation: rewrote `index.md`, `tutorial.md`, `for-coding-agents.md`; cross-referenced specification and practice guides.
+- Integration tests: replaced single `NIGHTHAWK_RUN_INTEGRATION_TESTS` gate with per-backend environment variables (`NIGHTHAWK_CODEX_INTEGRATION_TESTS`, `NIGHTHAWK_CLAUDE_SDK_INTEGRATION_TESTS`, `NIGHTHAWK_CLAUDE_CLI_INTEGRATION_TESTS`).
+
+### Removed
+- `nh_exec` tool (functionality absorbed by `nh_eval`).
+- Three redundant OpenAI integration tests from `test_llm_integration.py` (covered by promptfoo evaluation harness).
+- `pytest_sessionstart` credential-check hook (replaced by per-backend skip helpers).
+
+## [0.4.0] - 2026-03-20
+
+### Added
+- `nighthawk.testing` module with test executors and convenience factories for deterministic Natural function testing without LLM API calls.
+
+### Changed
+- Rewrote testing documentation in `tutorial.md` (Section 8) and `for-coding-agents.md` (Section 8): replaced incorrect `TestModel` usage with `nighthawk.testing` utilities, added testing strategy guidance distinguishing mock tests (Python logic) from integration tests (Natural block judgment).
+
+## [0.3.1] - 2026-03-19
+
+### Changed
+- Internal ID generation now uses `ulid.generate_ulid()` (ULID,
+  26-character Crockford Base32, timestamp-sortable) in a dedicated
+  module, replacing the former `generate_id` embedded in
+  `runtime.scoping`.
+
+## [0.3.0] - 2026-03-18
+
+### Added
+- `system_prompt_suffix_fragment_scope` and `user_prompt_suffix_fragment_scope` context managers for lightweight prompt fragment management without full scope overhead.
+- OpenTelemetry tracer now reports `instrumenting_library_version`.
+
+### Changed
+- Simplified OpenTelemetry span hierarchy: removed implicit `nighthawk.scope` spans and `nighthawk.step_executor` spans. `nighthawk.scope` spans are now emitted only for explicit `nh.scope()` calls.
+- `nighthawk.run` span no longer includes `scope.id` attribute; only `run.id` is emitted.
+- Trimmed `for-coding-agents.md` for coding-agent relevance: removed deprecated `@nh.tool` references, condensed exception hierarchy, scoped overrides, and added debugging context to `StepContextLimits`.
+
+## [0.2.0] - 2026-03-16
+
+### Added
+- Added compact step trace support for Natural block execution attempts:
+  - `StepTrace`
+  - `StepTraceError`
+  - `nighthawk.get_step_traces()`
+
+### Changed
+- Updated CI workflow setup (`setup-uv`) in project automation.
+
+### Fixed
+- License badge reference in README.
+- Documentation formatting inconsistencies.
+
+## [0.1.0] - 2026-03-13
+
+### Added
+- Initial public release of `nighthawk-python`.
+- Natural DSL execution runtime with run/scope execution context model.
+- Step executor abstraction and provider integration foundation.
+- Core documentation and project scaffolding.
+
+[Unreleased]: https://github.com/kurusugawa-computer/nighthawk-python/compare/v0.5.0...HEAD
+[0.5.0]: https://github.com/kurusugawa-computer/nighthawk-python/compare/v0.4.0...v0.5.0
+[0.4.0]: https://github.com/kurusugawa-computer/nighthawk-python/compare/v0.3.1...v0.4.0
+[0.3.1]: https://github.com/kurusugawa-computer/nighthawk-python/compare/v0.3.0...v0.3.1
+[0.3.0]: https://github.com/kurusugawa-computer/nighthawk-python/compare/v0.2.0...v0.3.0
+[0.2.0]: https://github.com/kurusugawa-computer/nighthawk-python/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/kurusugawa-computer/nighthawk-python/tree/v0.1.0

{nighthawk_python-0.4.0 → nighthawk_python-0.6.0}/CONTRIBUTING.md

@@ -70,6 +70,48 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 uv run pytest -q tests/integra
 
 Traces appear in the otel-tui terminal UI in real time.
 
+### Prompt evaluation with promptfoo
+
+System prompt, tool descriptions, and backend behavior are evaluated with [promptfoo](https://www.promptfoo.dev/). Requires Node.js (for `npx`). API keys must be loaded from `.env` first.
+
+```bash
+set -a; source .env; set +a
+PFOO="cd evals/promptfoo && PROMPTFOO_PYTHON=\"$(uv python find)\" npx promptfoo@latest"
+```
+
+| Command | Purpose |
+|---|---|
+| `eval $PFOO eval` | Full regression (all backends, all tests) |
+| `eval $PFOO eval -c promptfooconfig-prompt-ab.yaml` | Prompt A/B test (gpt-5.4-mini only) |
+| `eval $PFOO eval -c promptfooconfig-agents.yaml` | Coding agent backends (reduced test set) |
+| `eval $PFOO eval -c promptfooconfig.yaml --filter-pattern "P-BIND-001"` | Single test |
+| `eval $PFOO eval -c promptfooconfig.yaml --filter-providers "claude-code-cli"` | Single backend |
+| `eval $PFOO view` | Open results in browser |
+
+#### Config files (`evals/promptfoo/`)
+
+- `promptfooconfig.yaml` — Regression: winner prompt combo across openai-responses, claude-code-cli, and codex backends. All test cases.
+- `promptfooconfig-prompt-ab.yaml` — A/B testing: 4 prompt/tool variants on gpt-5.4-mini. All test cases.
+- `promptfooconfig-agents.yaml` — Coding agent only: claude-code-cli and codex with reduced test set.
+
+#### Directory layout
+
+- `provider.py` — Custom provider wrapping `AgentStepExecutor`. Handles tool preset installation, backend-specific model settings, and callable fixture resolution.
+- `prompts/` — System prompt variants (`eval_default.txt`, `eval_sequenced.txt`, `eval_mutation_aware.txt`, `eval_coding_agent.txt`, etc.).
+- `test_cases/` — YAML test suites: `binding_operations`, `tool_selection`, `outcome_kinds`, `edge_cases`, `loop_outcomes`, `multi_step`, `null_handling`, `tool_selection_core`.
+- `assertions/` — Custom Python assertions: `binding_value.py`, `outcome_kind.py`, `raise_message.py`.
+
+#### Adding tests
+
+Each test case YAML entry needs: `description` (with `P-SLUG-NNN` Id), `vars` (natural_program, input_bindings, output_binding_names), and `assert` list. Callable bindings use `"__callable:<key>"` resolved from `_CALLABLE_FIXTURE_REGISTRY` in `provider.py`.
+
+#### Useful flags
+
+- `--no-cache` — Skip cache (required after changing provider.py or prompts).
+- `--filter-pattern "<regex>"` — Run only matching test descriptions.
+- `--filter-providers "<regex>"` — Run only matching provider labels.
+- `-o <path>.json` — Write structured results to file for analysis.
+
 ### Environment variables
 
 - `OPENAI_API_KEY`: Required for OpenAI integration tests (also requires `pydantic-ai-slim[openai]`).
@@ -160,7 +202,7 @@ def run(
     Example:
         ```python
         executor = AgentStepExecutor.from_configuration(
-            configuration=StepExecutorConfiguration(model="openai-responses:gpt-5-mini"),
+            configuration=StepExecutorConfiguration(model="openai-responses:gpt-5.4-mini"),
         )
         with nighthawk.run(executor):
             result = my_natural_function()