pixie-qa 0.2.2__tar.gz → 0.5.0__tar.gz

pixie_qa-0.5.0/.gitignore

@@ -0,0 +1,8 @@
+.claude
+.agents
+__pycache__
+pixie_qa/
+uv.lock
+pixie/assets/index.html
+frontend/node_modules/
+frontend/dist/

{pixie_qa-0.2.2 → pixie_qa-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pixie-qa
-Version: 0.2.2
+Version: 0.5.0
 Summary: Automated quality assurance for AI applications
 Project-URL: Homepage, https://github.com/yiouli/pixie-qa
 Project-URL: Repository, https://github.com/yiouli/pixie-qa
@@ -44,9 +44,11 @@ Requires-Dist: openai>=2.29.0
 Requires-Dist: openinference-instrumentation>=0.1.44
 Requires-Dist: opentelemetry-api>=1.27.0
 Requires-Dist: opentelemetry-sdk>=1.27.0
-Requires-Dist: piccolo[sqlite]>=1.33.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: starlette>=1.0.0
+Requires-Dist: uvicorn>=0.42.0
+Requires-Dist: watchfiles>=1.1.1
 Provides-Extra: all
 Requires-Dist: openinference-instrumentation-anthropic; extra == 'all'
 Requires-Dist: openinference-instrumentation-dspy; extra == 'all'
@@ -67,18 +69,19 @@ Description-Content-Type: text/markdown
 
 # pixie-qa
 
-An agent skill that make coding agent the QA engineer for LLM applications.
+An agent skill that makes coding agents the QA engineer for LLM applications.
 
 ## What the Skill Does
 
 The `qa-eval` skill guides your coding agent through the full eval-based QA loop for LLM applications:
 
 1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
-2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
-3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
-4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
+2. **Instrument it** — use `wrap()` for data-object tracing and OpenInference auto-instrumentation for LLM span capture
+3. **Build a dataset** — create JSON datasets of representative inputs and expected outputs
+4. **Write eval tests** — generate `test_*.py` files with appropriate evaluators
 5. **Run the tests** — `pixie test` to run all evals and report per-case scores
-6. **Investigate failures** — look up the stored trace for each failure, diagnose, fix, repeat
+6. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
+7. **Investigate failures** — diagnose failures, fix, repeat
 
 ## Getting Started
 
@@ -100,4 +103,34 @@ Your coding agent will read your code, instrument it, build a dataset from a few
 
 ## Python Package
 
-The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. For the package API and CLI reference, see [docs/package.md](docs/package.md).
+The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. API docs are auto-generated by `pdoc3` into [docs/pixie/index.md](docs/pixie/index.md) via pre-commit. The markdown renderer uses [scripts/pdoc_templates/text.mako](scripts/pdoc_templates/text.mako) so async functions and methods are explicitly shown as `async def` in signatures.
+
+Install hooks once per clone:
+
+```bash
+uv run pre-commit install
+```
+
+## Web UI
+
+View all eval artifacts (results, markdown docs, datasets, and legacy scorecards) in a live-updating local web UI:
+
+```bash
+pixie start              # initializes pixie_qa/ (if needed) and opens http://localhost:7118
+pixie start my_dir       # use a custom artifact root
+pixie init               # scaffolds pixie_qa/ without starting the server
+```
+
+The web UI provides tabbed navigation for results, scorecards (legacy), datasets, and markdown files. Changes to artifacts are pushed to the browser in real time via SSE.
+
+The server writes a `server.lock` file to the artifact root directory on startup (containing the port number) and removes it on shutdown, allowing other processes to discover whether the server is already running.
+
+## Configuration
+
+Pixie reads configuration from environment variables and a local `.env` file through a single central config layer. Existing process env vars win over `.env` values.
+
+Useful settings include:
+
+- `PIXIE_ROOT` to move all generated artefacts under a different root directory
+- `PIXIE_RATE_LIMIT_ENABLED=true` to enable evaluator throttling for `pixie test`
+- `PIXIE_RATE_LIMIT_RPS`, `PIXIE_RATE_LIMIT_RPM`, `PIXIE_RATE_LIMIT_TPS`, and `PIXIE_RATE_LIMIT_TPM` to tune request and token throughput for LLM-as-judge evaluators

pixie_qa-0.5.0/README.md

@@ -0,0 +1,67 @@
+# pixie-qa
+
+An agent skill that makes coding agents the QA engineer for LLM applications.
+
+## What the Skill Does
+
+The `qa-eval` skill guides your coding agent through the full eval-based QA loop for LLM applications:
+
+1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
+2. **Instrument it** — use `wrap()` for data-object tracing and OpenInference auto-instrumentation for LLM span capture
+3. **Build a dataset** — create JSON datasets of representative inputs and expected outputs
+4. **Write eval tests** — generate `test_*.py` files with appropriate evaluators
+5. **Run the tests** — `pixie test` to run all evals and report per-case scores
+6. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
+7. **Investigate failures** — diagnose failures, fix, repeat
+
+## Getting Started
+
+### 1. Add the skill to your coding agent
+
+```bash
+npx skills add yiouli/pixie-qa
+```
+
+The accompanying python package would be installed by the skill automatically when it's used.
+
+### 2. Ask coding agent to set up evals
+
+Open a conversation and say something like when developing a python based AI project:
+
+> "setup QA for my agent"
+
+Your coding agent will read your code, instrument it, build a dataset from a few real runs, write and run eval-based tests, investigate failures and fix.
+
+## Python Package
+
+The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. API docs are auto-generated by `pdoc3` into [docs/pixie/index.md](docs/pixie/index.md) via pre-commit. The markdown renderer uses [scripts/pdoc_templates/text.mako](scripts/pdoc_templates/text.mako) so async functions and methods are explicitly shown as `async def` in signatures.
+
+Install hooks once per clone:
+
+```bash
+uv run pre-commit install
+```
+
+## Web UI
+
+View all eval artifacts (results, markdown docs, datasets, and legacy scorecards) in a live-updating local web UI:
+
+```bash
+pixie start              # initializes pixie_qa/ (if needed) and opens http://localhost:7118
+pixie start my_dir       # use a custom artifact root
+pixie init               # scaffolds pixie_qa/ without starting the server
+```
+
+The web UI provides tabbed navigation for results, scorecards (legacy), datasets, and markdown files. Changes to artifacts are pushed to the browser in real time via SSE.
+
+The server writes a `server.lock` file to the artifact root directory on startup (containing the port number) and removes it on shutdown, allowing other processes to discover whether the server is already running.
+
+## Configuration
+
+Pixie reads configuration from environment variables and a local `.env` file through a single central config layer. Existing process env vars win over `.env` values.
+
+Useful settings include:
+
+- `PIXIE_ROOT` to move all generated artefacts under a different root directory
+- `PIXIE_RATE_LIMIT_ENABLED=true` to enable evaluator throttling for `pixie test`
+- `PIXIE_RATE_LIMIT_RPS`, `PIXIE_RATE_LIMIT_RPM`, `PIXIE_RATE_LIMIT_TPS`, and `PIXIE_RATE_LIMIT_TPM` to tune request and token throughput for LLM-as-judge evaluators

pixie_qa-0.5.0/pixie/__init__.py

@@ -0,0 +1,90 @@
+"""pixie — automated quality assurance for AI applications.
+
+Re-exports the full public API so users can ``from pixie import ...``
+for every commonly used symbol without needing submodule paths.
+"""
+
+from pixie.eval.evaluable import Evaluable, TestCase
+from pixie.eval.evaluation import Evaluation, Evaluator, evaluate
+from pixie.eval.llm_evaluator import create_llm_evaluator
+from pixie.eval.scorers import (
+    AnswerCorrectness,
+    AnswerRelevancy,
+    AutoevalsAdapter,
+    Battle,
+    ClosedQA,
+    ContextRelevancy,
+    EmbeddingSimilarity,
+    ExactMatch,
+    Factuality,
+    Faithfulness,
+    Humor,
+    JSONDiff,
+    LevenshteinMatch,
+    ListContains,
+    Moderation,
+    NumericDiff,
+    Possible,
+    Security,
+    Sql,
+    Summary,
+    Translation,
+    ValidJSON,
+)
+
+# -- Harness ------------------------------------------------------------------
+from pixie.harness.runnable import Runnable
+
+# -- Instrumentation ----------------------------------------------------------
+from pixie.instrumentation.llm_tracing import (
+    add_handler,
+    enable_llm_tracing,
+    flush,
+    remove_handler,
+)
+from pixie.instrumentation.wrap import (
+    WrappedData,
+    wrap,
+)
+
+__all__ = [
+    # Instrumentation
+    "WrappedData",
+    "flush",
+    "enable_llm_tracing",
+    "add_handler",
+    "remove_handler",
+    "wrap",
+    # Harness
+    "Runnable",
+    # Eval data models
+    "Evaluable",
+    "TestCase",
+    "Evaluation",
+    "Evaluator",
+    "evaluate",
+    "create_llm_evaluator",
+    # Pre-made evaluators (autoevals adapters)
+    "AnswerCorrectness",
+    "AnswerRelevancy",
+    "AutoevalsAdapter",
+    "Battle",
+    "ClosedQA",
+    "ContextRelevancy",
+    "EmbeddingSimilarity",
+    "ExactMatch",
+    "Factuality",
+    "Faithfulness",
+    "Humor",
+    "JSONDiff",
+    "LevenshteinMatch",
+    "ListContains",
+    "Moderation",
+    "NumericDiff",
+    "Possible",
+    "Security",
+    "Sql",
+    "Summary",
+    "Translation",
+    "ValidJSON",
+]

pixie_qa-0.5.0/pixie/assets/mock-data.json

@@ -0,0 +1,407 @@
+{
+    "command_args": "pixie test tests/",
+    "timestamp": "2025-06-15 12:00:00 UTC",
+    "pixie_repo_url": "https://github.com/yiouli/pixie-qa",
+    "feedback_url": "https://feedback.gopixie.ai/feedback",
+    "brand_icon_url": "https://github.com/user-attachments/assets/76c18199-f00a-4fb3-a12f-ce6c173727af",
+    "test_records": [
+        {
+            "name": "test_customer_faq.py::test_faq_factuality",
+            "status": "passed",
+            "message": null,
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockFactuality"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.85,
+                                    "reasoning": "High string similarity between expected and actual output.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.72,
+                                    "reasoning": "Moderate string similarity.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.91,
+                                    "reasoning": "Very high similarity match.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.68,
+                                    "reasoning": "Reasonable similarity.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.77,
+                                    "reasoning": "Good similarity match.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": true,
+                    "criteria_message": "Pass: 4/5 inputs passed (threshold 0.6, required 80%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.6. At least 80% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "name": "test_customer_faq.py::test_faq_multi_evaluator",
+            "status": "failed",
+            "message": "AssertionError: 3/5 inputs failed on at least one evaluator",
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockFactuality",
+                        "MockClosedQA"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.85,
+                                    "reasoning": "High similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.3,
+                                    "reasoning": "Low keyword overlap.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.72,
+                                    "reasoning": "Moderate similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.45,
+                                    "reasoning": "Below threshold keyword overlap.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.91,
+                                    "reasoning": "Very high similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.6,
+                                    "reasoning": "Acceptable keyword overlap.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.68,
+                                    "reasoning": "Reasonable similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.25,
+                                    "reasoning": "Poor keyword match.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.77,
+                                    "reasoning": "Good similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.55,
+                                    "reasoning": "Marginal keyword overlap.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": false,
+                    "criteria_message": "Fail: only 2/5 inputs passed on all evaluators (required 100%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.5. At least 100% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "name": "test_customer_faq.py::test_faq_no_hallucinations",
+            "status": "passed",
+            "message": null,
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockHallucination"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": true,
+                    "criteria_message": "Pass: 5/5 inputs passed (threshold 0.5, required 100%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.5. At least 100% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "name": "test_customer_faq.py::test_faq_tone_check",
+            "status": "failed",
+            "message": "AssertionError: all 5 inputs failed",
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockStrictTone"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": false,
+                    "criteria_message": "Fail: 0/5 inputs passed (threshold 0.5, required 100%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.5. At least 100% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        }
+    ]
+}