pixie-qa 0.1.2__tar.gz → 0.1.8__tar.gz

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/.github/copilot-instructions.md

@@ -144,6 +144,103 @@ uv run pytest -k "test_function_name"   # Run specific test
 uv run pytest --cov=pixie               # Run with coverage report
 ```
 
+### 4a. End-to-End Tests for `pixie test`
+
+The `pixie test` CLI command has a dedicated e2e test suite that verifies the full
+command lifecycle — test discovery, execution, console output, exit codes, and
+HTML scorecard generation. The suite uses **realistic fixtures** that mirror how
+a real user would configure datasets, evaluators, and test files.
+
+**Fixture layout:**
+
+```
+tests/pixie/cli/
+  e2e_fixtures/
+    datasets/
+      customer-faq.json          # 5-item golden dataset (Evaluable items)
+    mock_evaluators.py           # Deterministic mock evaluators (no LLM calls)
+    test_customer_faq.py         # Realistic test file using assert_dataset_pass
+  e2e_cases.json                 # Edge-case scenario definitions
+  test_e2e_pixie_test.py         # Automated pytest e2e tests
+```
+
+The automated pytest file (`test_e2e_pixie_test.py`) contains two test classes:
+
+1. **`TestPixieTestRealisticE2E`** (10 tests) — Runs `pixie test` on the
+   realistic fixture (`test_customer_faq.py`) that uses 4 evaluator/criteria
+   combinations against the customer-FAQ dataset. Verifies exit code, console
+   summary, test names, check/cross marks, scorecard HTML generation, evaluator
+   names, PASS/FAIL badges, per-input scores, summary counts, and scoring
+   strategy descriptions.
+
+2. **`TestPixieTestEdgeCases`** (32 tests) — Parametrised from `e2e_cases.json`
+   covering empty dirs, filters, verbose mode, single file targeting, etc.
+
+**Mock evaluators** (`e2e_fixtures/mock_evaluators.py`) are deterministic
+replacements for LLM-as-judge evaluators. They use string similarity, keyword
+overlap, or fixed scores to produce realistic but reproducible results:
+- `MockFactualityEval` — SequenceMatcher string similarity (most items pass)
+- `MockClosedQAEval` — keyword overlap ratio (strict; some items fail)
+- `MockHallucinationEval` — always returns score 0.95
+- `MockFailingEval` (name="MockStrictTone") — always returns score 0.2
+
+**Expected realistic fixture results:**
+- `test_faq_factuality` → PASS (MockFactuality, threshold=0.6, pct=0.8)
+- `test_faq_multi_evaluator` → FAIL (MockFactuality+MockClosedQA, threshold=0.5, pct=1.0)
+- `test_faq_no_hallucinations` → PASS (MockHallucination, threshold=0.5, pct=1.0)
+- `test_faq_tone_check` → FAIL (MockStrictTone, threshold=0.5, pct=1.0)
+- Console: "2 passed, 2 failed", exit code 1
+- Scorecard: HTML with evaluator names, scores, PASS/FAIL badges
+
+**When to run e2e tests:**
+
+Run the e2e suite whenever you change anything in:
+- `pixie/cli/test_command.py` — the `pixie test` entry point
+- `pixie/evals/runner.py` — test discovery, execution, formatting
+- `pixie/evals/scorecard.py` — scorecard models, HTML generation
+- `pixie/evals/eval_utils.py` — `assert_pass` / `assert_dataset_pass`
+- `pixie/evals/criteria.py` — pass criteria
+
+```bash
+uv run pytest tests/pixie/cli/test_e2e_pixie_test.py -v  # Run all 42 e2e tests
+```
+
+**Agent verification protocol (manual inspection):**
+
+In addition to the automated pytest tests, the coding agent should manually
+verify the `pixie test` output after making changes to CLI/eval/scorecard code:
+
+1. **Run the realistic fixture directly:**
+   ```bash
+   PIXIE_ROOT=/tmp/pixie_e2e_verify uv run pixie test tests/pixie/cli/e2e_fixtures/test_customer_faq.py
+   ```
+
+2. **Inspect the console output** — verify that:
+   - All 4 test names appear with correct ✓/✗ marks
+   - Summary shows "2 passed, 2 failed"
+   - No unexpected errors or tracebacks
+
+3. **Inspect the HTML scorecard** — open the generated file and verify:
+   - All 4 evaluator names appear (MockFactuality, MockClosedQA, etc.)
+   - Per-input score cells show reasonable numeric values
+   - PASS/FAIL badges match expectations (2 PASS, 2 FAIL)
+   - Scoring strategy descriptions are human-readable
+   - The scorecard is well-formatted and renders correctly
+
+4. **Evaluate holistically** — given the dataset contents and evaluator
+   definitions, do the scores and pass/fail outcomes make sense? For example,
+   MockFactuality should score high on items where `eval_output` is similar to
+   `expected_output`, and MockStrictTone should always fail.
+
+This manual step catches rendering issues, layout regressions, and semantic
+correctness problems that simple string assertions can miss.
+
+**Adding new edge-case scenarios:**
+
+1. Add a new object to `tests/pixie/cli/e2e_cases.json`.
+2. Run `uv run pytest tests/pixie/cli/test_e2e_pixie_test.py -v` to verify.
+3. No code changes needed in the test file — it auto-discovers all cases.
+
 ### 5. Test Quality Guidelines
 
 **Good tests are:**
@@ -344,11 +441,17 @@ uv run ruff format .             # Format code
 Before committing, run:
 
 ```bash
-uv run pytest                    # All tests must pass
+uv run pytest                    # All tests must pass (includes e2e)
 uv run mypy pixie/               # Zero type errors
 uv run ruff check .              # No linting errors
 ```
 
+When changing `pixie test` or scorecard-related code, also run e2e explicitly:
+
+```bash
+uv run pytest tests/pixie/cli/test_e2e_pixie_test.py -v  # Verify pixie test e2e
+```
+
 Also verify **zero Pylance errors** in VS Code Problems panel (Pylance can catch type mismatches that mypy misses for untyped third-party packages).
 
 ---
@@ -509,6 +612,8 @@ This project has strict error-handling conventions due to operating inside OTel
 6. ✅ Update docstrings / `README.md` / relevant `specs/` docs
 7. ✅ Add/update `changelogs/<feature>.md` for non-trivial changes
 8. ✅ Verify functionality works as expected
+9. ✅ If touching `pixie test` / scorecard / runner / eval code, run `uv run pytest tests/pixie/cli/test_e2e_pixie_test.py -v` — all 42 e2e tests must pass (10 realistic + 32 edge-case)
+10. ✅ If touching `pixie test` / scorecard code, also run the **agent verification protocol** (section 4a) — manually run `pixie test` on the realistic fixture and inspect console + scorecard output
 
 **Development cycle:**
 
@@ -519,8 +624,9 @@ This project has strict error-handling conventions due to operating inside OTel
 5. Implement feature (reuse existing code when possible)
 6. After each task: run tests and type check
 7. Run linting (`uv run ruff check .`)
-8. Update docs and changelog for the task
-9. Fix any issues
-10. Commit
+8. Run `pixie test` e2e suite if CLI/eval/scorecard code changed
+9. Update docs and changelog for the task
+10. Fix any issues
+11. Commit
 
 Following these practices ensures high code quality, type safety, maintainability, and reliability.

pixie_qa-0.1.8/.gitignore

@@ -0,0 +1,3 @@
+.claude
+__pycache__
+pixie_qa/

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pixie-qa
-Version: 0.1.2
+Version: 0.1.8
 Summary: Automated quality assurance for AI applications
 Project-URL: Homepage, https://github.com/yiouli/pixie-qa
 Project-URL: Repository, https://github.com/yiouli/pixie-qa
@@ -45,6 +45,7 @@ Requires-Dist: opentelemetry-api>=1.27.0
 Requires-Dist: opentelemetry-sdk>=1.27.0
 Requires-Dist: piccolo[sqlite]>=1.33.0
 Requires-Dist: pydantic>=2.0
+Requires-Dist: python-dotenv>=1.2.2
 Provides-Extra: all
 Requires-Dist: openinference-instrumentation-anthropic; extra == 'all'
 Requires-Dist: openinference-instrumentation-dspy; extra == 'all'
@@ -67,13 +68,11 @@ Description-Content-Type: text/markdown
 
 An agent skill for **eval-driven development** of LLM-powered applications.
 
-Use this skill to instrument your app, build golden datasets from real runs, write eval-based tests, and catch regressions before they ship — all from a single conversation with Claude.
-
 ## What the Skill Does
 
 The `eval-driven-dev` skill guides your coding agent through the full QA loop for LLM applications:
 
-1. **Understand the app** — read the codebase, trace the data flow, learn what the app is supposed to do
+1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
 2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
 3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
 4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
@@ -85,7 +84,7 @@ The `eval-driven-dev` skill guides your coding agent through the full QA loop fo
 ### 1. Add the skill to your coding agent
 
 ```bash
-npx openskills install yiouli/pixie-qa
+npx skills add yiouli/pixie-qa
 ```
 
 The accompanying python package would be installed by the skill automatically when it's used.

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/README.md

@@ -2,13 +2,11 @@
 
 An agent skill for **eval-driven development** of LLM-powered applications.
 
-Use this skill to instrument your app, build golden datasets from real runs, write eval-based tests, and catch regressions before they ship — all from a single conversation with Claude.
-
 ## What the Skill Does
 
 The `eval-driven-dev` skill guides your coding agent through the full QA loop for LLM applications:
 
-1. **Understand the app** — read the codebase, trace the data flow, learn what the app is supposed to do
+1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
 2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
 3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
 4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
@@ -20,7 +18,7 @@ The `eval-driven-dev` skill guides your coding agent through the full QA loop fo
 ### 1. Add the skill to your coding agent
 
 ```bash
-npx openskills install yiouli/pixie-qa
+npx skills add yiouli/pixie-qa
 ```
 
 The accompanying python package would be installed by the skill automatically when it's used.

pixie_qa-0.1.8/changelogs/deep-research-demo.md

@@ -0,0 +1,43 @@
+# Deep Research Demo Project
+
+## What Changed
+
+Added a simplified demo project under `demo/deep_research/` based on the
+[GPT Researcher](https://github.com/assafelovic/gpt-researcher) project
+(commit `7c32174`, Apache 2.0 license).
+
+The demo serves as a real-world AI application that the **pixie-qa** skill
+can be tested against.
+
+### Simplifications from the original
+
+| Removed | Reason |
+|---------|--------|
+| UI (frontend + backend server) | Not needed for programmatic evaluation |
+| Deep Research mode | Complex multi-agent workflow, out of scope |
+| Image generation | Not needed for text-based evaluation |
+| MCP integrations | External tool integrations, not needed |
+| All retrievers except DuckDuckGo | Simplifies dependencies, avoids paid APIs |
+| Tavily / Firecrawl scrapers | Paid service dependencies |
+| PDF / DOCX export | Not needed for evaluation |
+| Docker / Terraform / multi-agent | Infrastructure, not needed |
+
+### What remains
+
+- Programmatic entry point (`run.py`) to run research with a string query
+- Full agent workflow: query → sub-queries → web search → scrape → summarize → report
+- DuckDuckGo as the sole search retriever (free, no API key needed for search)
+- OpenAI as the LLM provider (requires `OPENAI_API_KEY`)
+
+## Files Affected
+
+- `demo/deep_research/` — entire new directory
+- `demo/deep_research/gpt_researcher/` — simplified core library
+- `demo/deep_research/run.py` — entry point
+- `demo/deep_research/pyproject.toml` — dependencies
+- `demo/deep_research/LICENSE` — Apache 2.0 (from upstream)
+- `demo/deep_research/NOTICE` — attribution notice
+
+## Migration Notes
+
+This is a new addition — no migration required.

pixie_qa-0.1.8/changelogs/pixie-test-e2e-suite.md

@@ -0,0 +1,69 @@
+# pixie test — e2e test suite
+
+## What changed
+
+Added a comprehensive end-to-end test suite for the `pixie test` CLI command
+with two complementary layers:
+
+1. **Realistic fixture tests** (10 tests) — run `pixie test` on a realistic
+   test file with a 5-item customer-FAQ golden dataset and 4 deterministic
+   mock evaluators. Verifies exit code, console summary, test names,
+   check/cross marks, scorecard HTML generation, evaluator names, PASS/FAIL
+   badges, per-input scores, summary counts, and scoring strategy descriptions.
+
+2. **Edge-case tests** (32 tests) — parametrised from `e2e_cases.json`
+   covering empty dirs, filters, verbose mode, single file targeting, etc.
+
+The copilot instructions now include an **agent verification protocol** that
+tells the coding agent to manually run `pixie test` on the realistic fixtures
+and holistically evaluate the console output and HTML scorecard after making
+changes to CLI/eval/scorecard code.
+
+### New files
+
+- **`tests/pixie/cli/e2e_fixtures/datasets/customer-faq.json`** — 5-item
+  golden dataset with FAQ questions, chatbot answers, and reference answers.
+
+- **`tests/pixie/cli/e2e_fixtures/mock_evaluators.py`** — 4 deterministic
+  mock evaluators: MockFactualityEval (SequenceMatcher string similarity),
+  MockClosedQAEval (keyword overlap), MockHallucinationEval (always 0.95),
+  MockFailingEval/MockStrictTone (always 0.2). No LLM calls.
+
+- **`tests/pixie/cli/e2e_fixtures/test_customer_faq.py`** — Realistic test
+  file using `assert_dataset_pass` with different scoring strategies.
+  Expected: 2 PASS (`test_faq_factuality`, `test_faq_no_hallucinations`),
+  2 FAIL (`test_faq_multi_evaluator`, `test_faq_tone_check`).
+
+- **`tests/pixie/cli/e2e_cases.json`** — 8 edge-case scenarios as JSON data.
+
+- **`tests/pixie/cli/test_e2e_pixie_test.py`** — Two test classes:
+  `TestPixieTestRealisticE2E` (10 tests) and `TestPixieTestEdgeCases`
+  (32 tests). Total: 42 test cases.
+
+### Modified files
+
+- **`.github/copilot-instructions.md`** — Rewrote section 4a with realistic
+  fixture layout, mock evaluator descriptions, expected results, and a full
+  agent verification protocol. Updated summary checklist to require both
+  automated e2e tests (42) and manual agent inspection.
+
+- **`specs/evals-harness.md`** — Updated E2E Test Suite section to describe
+  both realistic fixtures and edge-case scenarios.
+
+## Files affected
+
+- `tests/pixie/cli/e2e_fixtures/datasets/customer-faq.json`
+- `tests/pixie/cli/e2e_fixtures/mock_evaluators.py`
+- `tests/pixie/cli/e2e_fixtures/test_customer_faq.py`
+- `tests/pixie/cli/e2e_cases.json`
+- `tests/pixie/cli/test_e2e_pixie_test.py`
+- `.github/copilot-instructions.md`
+- `specs/evals-harness.md`
+
+## Migration notes
+
+No API changes. The e2e test suite is purely additive.
+
+- To add new edge-case scenarios: edit `tests/pixie/cli/e2e_cases.json`.
+- To modify realistic fixture behavior: edit mock evaluators or the test file
+  in `tests/pixie/cli/e2e_fixtures/`.

pixie_qa-0.1.8/changelogs/scorecard-branding-and-skill-version-check.md

@@ -0,0 +1,41 @@
+# Scorecard branding and skill version check
+
+## What changed and why
+
+Two user-facing upgrades were added:
+
+1. The HTML scorecard generated by `pixie test` now has a branded Pixie header
+   with a repo CTA and a feedback modal so users can quickly star the project
+   and send feedback without leaving the report context.
+2. The `eval-driven-dev` skill now ships with version metadata and a bundled
+   version-check script that compares the local skill resource with the latest
+   `main` branch metadata and updates both the skill and Python package when the
+   local version is behind.
+
+## Files affected
+
+- `pixie/evals/scorecard.py` — adds the branded header, repo CTA, feedback
+  modal HTML, styling, and modal toggle script.
+- `tests/pixie/evals/test_scorecard.py` — verifies the branded header and
+  feedback form wiring in generated scorecard HTML.
+- `tests/pixie/cli/test_e2e_pixie_test.py` — verifies the generated scorecard
+  contains the branding and feedback actions end-to-end.
+- `tests/pixie/test_skill_resources.py` — verifies the skill version metadata
+  and update command behavior.
+- `.claude/skills/eval-driven-dev/resources/version.json` — source of truth for
+  the local skill version metadata.
+- `.claude/skills/eval-driven-dev/resources/check_version.py` — checks the
+  remote version file and triggers updates when needed.
+- `.claude/skills/eval-driven-dev/SKILL.md` — instructs the skill to run the
+  version check before any other skill steps.
+- `README.md`, `docs/package.md`, and `specs/evals-harness.md` — document the
+  new header and skill update flow.
+
+## Migration notes
+
+- No API changes are required for existing `pixie test` users. The scorecard
+  remains a self-contained HTML file, now with extra header UI and a feedback
+  form.
+- The version-check script is additive. If network access or package-manager
+  commands are unavailable, it exits cleanly after printing what could not be
+  checked or run.

pixie_qa-0.1.8/changelogs/scorecard-eval-detail-dialog.md

@@ -0,0 +1,28 @@
+# Scorecard: Evaluator cell "details" dialog
+
+## What changed
+
+Each evaluator score cell in the scorecard detail table now has a **"details"** hyperlink.
+Clicking it opens a modal dialog showing:
+
+- **Score** — numeric value with green ✓ or red ✗ indicator
+- **Reasoning** — the full `Evaluation.reasoning` string (previously only shown as a tooltip)
+- **Details** — the `Evaluation.details` dict rendered as pretty-printed JSON (hidden when empty)
+
+The modal is dismiss-able via the **Close** button, the **Esc** key, or a click on the backdrop — consistent with the existing feedback modal.
+
+## Files affected
+
+- `pixie/evals/scorecard.py`
+  - Added `import json`
+  - New `_render_eval_detail_modal()` helper — renders the reusable hidden modal
+  - `generate_scorecard_html()` — calls `_render_eval_detail_modal()` after the brand header
+  - `_render_pass_table()` — each evaluator cell now embeds a `data-eval` JSON attribute and a `details` link
+  - `_HTML_HEAD` — added CSS for `.details-link`, `.eval-detail-body`, `.eval-detail-row`, `.eval-detail-label`, `.eval-detail-value`, `.eval-detail-score-pass/fail`, `.eval-detail-json`
+  - `_HTML_FOOT` — added `showEvalDetail(link)` and `closeEvalDetailModal()` JS functions; updated Esc and backdrop-click handlers to also close the eval-detail modal
+
+## Migration notes
+
+No API changes. The `AssertRecord`, `Evaluation`, and `ScorecardReport` models are unchanged.
+Existing scorecards already stored `Evaluation.reasoning` as a cell `title` attribute (tooltip);
+that attribute has been replaced by the clickable details link — tooltip-only access is no longer available.

pixie_qa-0.1.8/changelogs/skill-v2-and-rootdir-discovery.md

@@ -0,0 +1,76 @@
+# Skill v2: setup-vs-iterate, eval boundary, rootdir discovery
+
+## What changed and why
+
+### 1. Renamed default root directory from `.pixie` to `pixie_qa`
+
+The dot-prefix `.pixie` caused Python import resolution issues (treated as hidden
+directory, confused with relative imports). Renamed to `pixie_qa` — a plain,
+importable name that avoids these problems.
+
+- `pixie/config.py`: `DEFAULT_ROOT` changed from `".pixie"` to `"pixie_qa"`
+- All documentation updated: SKILL.md, pixie-api.md, specs/agent-skill.md
+
+### 2. Test runner rootdir discovery (pytest-style)
+
+The old `_load_module()` in `pixie/evals/runner.py` added the test file's parent
+and grandparent to `sys.path`. This broke for test files nested deeper than two
+levels from the project root (e.g. `pixie_qa/tests/test_foo.py`).
+
+Rewrote `_load_module()` to use rootdir discovery: `_find_rootdir()` walks up from
+the test file directory looking for `pyproject.toml`, `setup.py`, or `setup.cfg` —
+the same strategy pytest uses. The discovered rootdir is added to `sys.path`,
+making project-root imports work regardless of test file depth.
+
+### 3. SKILL.md: setup vs. iteration checkpoint
+
+Added a "Setup vs. Iteration" section at the top of the skill. When the user says
+"setup QA" / "set up evals" / "add tests", the agent now stops after Stage 6
+(first test run) and reports results without fixing anything. It only proceeds
+to Stage 7 (investigate and fix) if the user explicitly confirms.
+
+Previously, the skill had no checkpoint — the agent would eagerly iterate on
+failures, modifying application code without being asked.
+
+### 4. SKILL.md: eval boundary guidance
+
+Added "The eval boundary: what to evaluate" section. Evals focus on LLM-dependent
+behaviour only (response quality, routing decisions, prompt effectiveness). Tool
+implementations, database queries, keyword matching, and other deterministic logic
+are explicitly out of scope — they should be tested with traditional unit tests.
+
+The investigation section (Stage 7) now classifies failures into "LLM-related"
+and "non-LLM" categories with guidance on how to handle each.
+
+### 5. SKILL.md: instrument production code only
+
+Strengthened Stage 3 with explicit rules against creating wrapper functions or
+alternate code paths for eval purposes. Added a ❌ WRONG example showing the
+anti-pattern (creating `run_for_eval()` that duplicates `main()` logic) and
+✅ CORRECT examples showing `@observe` on existing functions and
+`start_observation` context manager inside existing functions.
+
+## Files affected
+
+| File                                                     | Change                                            |
+| -------------------------------------------------------- | ------------------------------------------------- |
+| `pixie/config.py`                                        | `DEFAULT_ROOT = "pixie_qa"`                       |
+| `pixie/instrumentation/handlers.py`                      | Docstring updated                                 |
+| `pixie/evals/runner.py`                                  | New `_find_rootdir()`, rewritten `_load_module()` |
+| `tests/pixie/test_config.py`                             | Updated assertions for `"pixie_qa"` default       |
+| `tests/pixie/evals/test_runner.py`                       | 8 new tests (rootdir + import resolution)         |
+| `.claude/skills/eval-driven-dev/SKILL.md`                | Major rewrite (issues 3, 4, 5 + rename)           |
+| `.claude/skills/eval-driven-dev/references/pixie-api.md` | Config table updated                              |
+| `specs/agent-skill.md`                                   | `.pixie` → `pixie_qa` throughout                  |
+
+## Migration notes
+
+- **Breaking default change**: The default root directory changed from `.pixie` to
+  `pixie_qa`. Existing projects using the old default should either:
+  - Set `PIXIE_ROOT=.pixie` to preserve the old location, or
+  - Rename the directory: `mv .pixie pixie_qa`
+- **Test runner**: `_load_module()` now uses rootdir discovery instead of
+  parent/grandparent. No action needed — this is backwards compatible and more
+  reliable.
+- **Skill behaviour**: Agents following the updated SKILL.md will stop after
+  initial test setup and ask before iterating on failures.

pixie_qa-0.1.8/changelogs/test-scorecard.md

@@ -0,0 +1,54 @@
+# Test Scorecard Feature
+
+## What Changed
+
+Added an HTML scorecard report that is automatically generated and saved to disk
+for every `pixie test` command run. The scorecard provides a detailed,
+human-readable breakdown of eval-based test results beyond the terminal summary.
+
+### Scorecard Contents
+
+- **Test run overview** — command args, timestamp, pass/fail summary, and a
+  table of all discovered tests with their status badges.
+- **Per-test-function detail** — for each test that calls `assert_pass` or
+  `assert_dataset_pass`:
+  - Human-readable scoring strategy description.
+  - Per-evaluator pass rate summary table.
+  - Input × evaluator score grid with hover tooltips showing reasoning.
+  - Tabbed view for multi-pass runs (one tab per pass).
+
+### Scorecard Storage
+
+HTML files are saved to `{config.root}/scorecards/<YYYYMMDD-HHMMSS-normalized-args>.html`.
+The CLI prints the full path after each run so users can click to open it.
+
+## Files Affected
+
+### New Files
+
+- `pixie/evals/scorecard.py` — data models (`AssertRecord`, `TestRecord`,
+  `ScorecardReport`), `ScorecardCollector` (context-var-based accumulator),
+  HTML generation, and `save_scorecard()`.
+- `tests/pixie/evals/test_scorecard.py` — 28 tests covering models, collector,
+  HTML generation, file saving, and integration with `assert_pass` / runner.
+
+### Modified Files
+
+- `pixie/evals/eval_utils.py` — `assert_pass` now publishes an `AssertRecord`
+  to the active `ScorecardCollector` (no-op when no collector is active).
+- `pixie/evals/runner.py` — `_run_single()` activates a `ScorecardCollector`
+  per test; `EvalTestResult` gains an `assert_records` field.
+- `pixie/cli/test_command.py` — builds a `ScorecardReport`, calls
+  `save_scorecard()`, and prints the path.
+- `pixie/evals/__init__.py` — re-exports `ScorecardCollector`,
+  `ScorecardReport`, `generate_scorecard_html`, `save_scorecard`.
+- `docs/package.md` — documents the HTML scorecard section under "Running Tests".
+
+## Migration Notes
+
+- No breaking API changes. Existing `pixie test` invocations behave identically
+  to before, with the addition of an HTML file being written and a path printed
+  at the end.
+- `EvalTestResult.assert_records` defaults to an empty tuple, so any code
+  that accesses `EvalTestResult` is unaffected.
+- The scorecard directory (`{config.root}/scorecards/`) is created on demand.

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/docs/package.md

@@ -188,13 +188,32 @@ All evaluators are importable from `pixie` (e.g. `from pixie import FactualityEv
 
 ## Running Tests
 
-Use `pixie-test` (not bare `pytest`) to run eval tests. It sets up the async environment and provides eval-specific output formatting:
+Use `pixie test` (or the equivalent `pixie-test` entry point, not bare `pytest`)
+to run eval tests. It sets up the async environment and provides eval-specific
+output formatting:
 
 ```bash
-pixie-test                 # run all test_*.py in the current directory
-pixie-test tests/          # specify a path
-pixie-test -k factuality   # filter by name substring
-pixie-test -v              # verbose: shows per-case scores and reasoning
+pixie test                 # run all test_*.py in the current directory
+pixie test tests/          # specify a path
+pixie test -k factuality   # filter by name substring
+pixie test -v              # verbose: shows per-case scores and reasoning
+```
+
+### HTML Scorecard
+
+Every `pixie test` run generates an **HTML scorecard** saved to `{PIXIE_ROOT}/scorecards/<timestamp>.html`. The scorecard contains:
+
+- **Test run overview** — command args, pass/fail summary, and a table of all tests with their status.
+- **Per-test detail** — for each test function that calls `assert_pass` / `assert_dataset_pass`:
+  - Scoring strategy description (human-readable).
+  - Per-evaluator pass rate table.
+  - Per-input × per-evaluator score grid with tooltips showing reasoning.
+  - **Tabbed view** for multi-pass runs (one tab per pass).
+
+After the test run, the CLI prints the scorecard path:
+
+```text
+See /path/to/pixie_qa/scorecards/20250615-120000-pixie-test.html for test details
 ```
 
 ---

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/pixie/cli/main.py

@@ -20,6 +20,7 @@ import json
 import sys
 from typing import TextIO
 
+from dotenv import load_dotenv
 from piccolo.engine.sqlite import SQLiteEngine
 from pydantic import JsonValue
 
@@ -183,6 +184,8 @@ def main(argv: list[str] | None = None) -> int:
         parser.print_help()
         return 1
 
+    load_dotenv()
+
     if args.command == "dataset":
         if args.dataset_action is None:
             parser.parse_args(["dataset", "--help"])

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/pixie/cli/test_command.py

@@ -5,15 +5,48 @@ Usage::
     pixie test [path] [--filter PATTERN] [--verbose]
 
 Discovers and runs eval test functions, reporting pass/fail results.
+Generates an HTML scorecard report saved to
+``{config.root}/scorecards/<timestamp>.html``.
 """
 
 from __future__ import annotations
 
 import argparse
 import sys
+from collections.abc import Sequence
 
 import pixie.instrumentation as px
 from pixie.evals.runner import discover_tests, format_results, run_tests
+from pixie.evals.scorecard import ScorecardReport, TestRecord, save_scorecard
+
+
+def _build_report(
+    results: Sequence[object],
+    command_args: str,
+) -> ScorecardReport:
+    """Build a :class:`ScorecardReport` from runner results.
+
+    Args:
+        results: List of ``EvalTestResult`` objects.
+        command_args: The command-line arguments string.
+
+    Returns:
+        A fully populated ``ScorecardReport``.
+    """
+    from pixie.evals.runner import EvalTestResult
+
+    test_records: list[TestRecord] = []
+    for r in results:
+        assert isinstance(r, EvalTestResult)
+        test_records.append(
+            TestRecord(
+                name=r.name,
+                status=r.status,
+                message=r.message,
+                asserts=list(r.assert_records),
+            )
+        )
+    return ScorecardReport(command_args=command_args, test_records=test_records)
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -60,6 +93,13 @@ def main(argv: list[str] | None = None) -> int:
     output = format_results(results, verbose=args.verbose)
     print(output)  # noqa: T201
 
+    # ── Generate and save scorecard ───────────────────────────────
+    raw_argv = argv if argv is not None else sys.argv[1:]
+    command_str = "pixie test " + " ".join(raw_argv)
+    report = _build_report(results, command_args=command_str)
+    scorecard_path = save_scorecard(report)
+    print(f"\nSee {scorecard_path} for test details")  # noqa: T201
+
     all_passed = all(r.status == "passed" for r in results)
     return 0 if all_passed else 1
 

{pixie_qa-0.1.2 → pixie_qa-0.1.8}/pixie/config.py

@@ -11,14 +11,14 @@ import os
 from dataclasses import dataclass
 
 #: Default root directory for all pixie-generated artefacts.
-DEFAULT_ROOT = ".pixie"
+DEFAULT_ROOT = "pixie_qa"
 
 
 @dataclass(frozen=True)
 class PixieConfig:
     """Immutable configuration snapshot.
 
-    All paths default to subdirectories / files within a single ``.pixie``
+    All paths default to subdirectories / files within a single ``pixie_qa``
     project folder so that observations, datasets, tests, scripts and notes
     live in one predictable location.