pixie-qa 0.2.2__tar.gz → 0.4.0__tar.gz

pixie_qa-0.4.0/.gitignore

@@ -0,0 +1,8 @@
+.claude
+.agents
+__pycache__
+pixie_qa/
+uv.lock
+pixie/assets/index.html
+frontend/node_modules/
+frontend/dist/

{pixie_qa-0.2.2 → pixie_qa-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pixie-qa
-Version: 0.2.2
+Version: 0.4.0
 Summary: Automated quality assurance for AI applications
 Project-URL: Homepage, https://github.com/yiouli/pixie-qa
 Project-URL: Repository, https://github.com/yiouli/pixie-qa
@@ -47,6 +47,9 @@ Requires-Dist: opentelemetry-sdk>=1.27.0
 Requires-Dist: piccolo[sqlite]>=1.33.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: starlette>=1.0.0
+Requires-Dist: uvicorn>=0.42.0
+Requires-Dist: watchfiles>=1.1.1
 Provides-Extra: all
 Requires-Dist: openinference-instrumentation-anthropic; extra == 'all'
 Requires-Dist: openinference-instrumentation-dspy; extra == 'all'
@@ -77,8 +80,10 @@ The `qa-eval` skill guides your coding agent through the full eval-based QA loop
 2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
 3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
 4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
-5. **Run the tests** — `pixie test` to run all evals and report per-case scores
-6. **Investigate failures** — look up the stored trace for each failure, diagnose, fix, repeat
+5. **Validate datasets** — `pixie dataset validate [dir_or_dataset_path]` to catch schema/config errors early
+6. **Run the tests** — `pixie test` to run all evals and report per-case scores
+7. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
+8. **Investigate failures** — look up the stored trace for each failure, diagnose, fix, repeat
 
 ## Getting Started
 
@@ -101,3 +106,27 @@ Your coding agent will read your code, instrument it, build a dataset from a few
 ## Python Package
 
 The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. For the package API and CLI reference, see [docs/package.md](docs/package.md).
+
+## Web UI
+
+View all eval artifacts (results, markdown docs, datasets, and legacy scorecards) in a live-updating local web UI:
+
+```bash
+pixie start              # initializes pixie_qa/ (if needed) and opens http://localhost:7118
+pixie start my_dir       # use a custom artifact root
+pixie init               # scaffolds pixie_qa/ without starting the server
+```
+
+The web UI provides tabbed navigation for results, scorecards (legacy), datasets, and markdown files. Changes to artifacts are pushed to the browser in real time via SSE.
+
+The server writes a `server.lock` file to the artifact root directory on startup (containing the port number) and removes it on shutdown, allowing other processes to discover whether the server is already running.
+
+## Configuration
+
+Pixie reads configuration from environment variables and a local `.env` file through a single central config layer. Existing process env vars win over `.env` values.
+
+Useful settings include:
+
+- `PIXIE_ROOT` to move all generated artefacts under a different root directory
+- `PIXIE_RATE_LIMIT_ENABLED=true` to enable evaluator throttling for `pixie test`
+- `PIXIE_RATE_LIMIT_RPS`, `PIXIE_RATE_LIMIT_RPM`, `PIXIE_RATE_LIMIT_TPS`, and `PIXIE_RATE_LIMIT_TPM` to tune request and token throughput for LLM-as-judge evaluators

pixie_qa-0.4.0/README.md

@@ -0,0 +1,62 @@
+# pixie-qa
+
+An agent skill that make coding agent the QA engineer for LLM applications.
+
+## What the Skill Does
+
+The `qa-eval` skill guides your coding agent through the full eval-based QA loop for LLM applications:
+
+1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
+2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
+3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
+4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
+5. **Validate datasets** — `pixie dataset validate [dir_or_dataset_path]` to catch schema/config errors early
+6. **Run the tests** — `pixie test` to run all evals and report per-case scores
+7. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
+8. **Investigate failures** — look up the stored trace for each failure, diagnose, fix, repeat
+
+## Getting Started
+
+### 1. Add the skill to your coding agent
+
+```bash
+npx skills add yiouli/pixie-qa
+```
+
+The accompanying python package would be installed by the skill automatically when it's used.
+
+### 2. Ask coding agent to set up evals
+
+Open a conversation and say something like when developing a python based AI project:
+
+> "setup QA for my agent"
+
+Your coding agent will read your code, instrument it, build a dataset from a few real runs, write and run eval-based tests, investigate failures and fix.
+
+## Python Package
+
+The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. For the package API and CLI reference, see [docs/package.md](docs/package.md).
+
+## Web UI
+
+View all eval artifacts (results, markdown docs, datasets, and legacy scorecards) in a live-updating local web UI:
+
+```bash
+pixie start              # initializes pixie_qa/ (if needed) and opens http://localhost:7118
+pixie start my_dir       # use a custom artifact root
+pixie init               # scaffolds pixie_qa/ without starting the server
+```
+
+The web UI provides tabbed navigation for results, scorecards (legacy), datasets, and markdown files. Changes to artifacts are pushed to the browser in real time via SSE.
+
+The server writes a `server.lock` file to the artifact root directory on startup (containing the port number) and removes it on shutdown, allowing other processes to discover whether the server is already running.
+
+## Configuration
+
+Pixie reads configuration from environment variables and a local `.env` file through a single central config layer. Existing process env vars win over `.env` values.
+
+Useful settings include:
+
+- `PIXIE_ROOT` to move all generated artefacts under a different root directory
+- `PIXIE_RATE_LIMIT_ENABLED=true` to enable evaluator throttling for `pixie test`
+- `PIXIE_RATE_LIMIT_RPS`, `PIXIE_RATE_LIMIT_RPM`, `PIXIE_RATE_LIMIT_TPS`, and `PIXIE_RATE_LIMIT_TPM` to tune request and token throughput for LLM-as-judge evaluators

{pixie_qa-0.2.2 → pixie_qa-0.4.0}/pixie/__init__.py

@@ -18,29 +18,30 @@ from pixie.evals.eval_utils import (
 )
 from pixie.evals.evaluation import Evaluation, Evaluator, evaluate
 from pixie.evals.llm_evaluator import create_llm_evaluator
+from pixie.evals.rate_limiter import RateLimitConfig, configure_rate_limits
 from pixie.evals.scorers import (
-    AnswerCorrectnessEval,
-    AnswerRelevancyEval,
+    AnswerCorrectness,
+    AnswerRelevancy,
     AutoevalsAdapter,
-    BattleEval,
-    ClosedQAEval,
-    ContextRelevancyEval,
-    EmbeddingSimilarityEval,
-    ExactMatchEval,
-    FactualityEval,
-    FaithfulnessEval,
-    HumorEval,
-    JSONDiffEval,
+    Battle,
+    ClosedQA,
+    ContextRelevancy,
+    EmbeddingSimilarity,
+    ExactMatch,
+    Factuality,
+    Faithfulness,
+    Humor,
+    JSONDiff,
     LevenshteinMatch,
-    ListContainsEval,
-    ModerationEval,
-    NumericDiffEval,
-    PossibleEval,
-    SecurityEval,
-    SqlEval,
-    SummaryEval,
-    TranslationEval,
-    ValidJSONEval,
+    ListContains,
+    Moderation,
+    NumericDiff,
+    Possible,
+    Security,
+    Sql,
+    Summary,
+    Translation,
+    ValidJSON,
 )
 from pixie.evals.trace_capture import MemoryTraceHandler, capture_traces
 from pixie.evals.trace_helpers import last_llm_call, root
@@ -67,36 +68,38 @@ __all__ = [
     "remove_handler",
     "start_observation",
     # Evals
-    "AnswerCorrectnessEval",
-    "AnswerRelevancyEval",
+    "AnswerCorrectness",
+    "AnswerRelevancy",
     "AutoevalsAdapter",
-    "BattleEval",
-    "ClosedQAEval",
-    "ContextRelevancyEval",
-    "EmbeddingSimilarityEval",
+    "Battle",
+    "ClosedQA",
+    "ContextRelevancy",
+    "EmbeddingSimilarity",
     "EvalAssertionError",
     "Evaluation",
     "Evaluator",
-    "ExactMatchEval",
-    "FactualityEval",
-    "FaithfulnessEval",
-    "HumorEval",
-    "JSONDiffEval",
+    "ExactMatch",
+    "Factuality",
+    "Faithfulness",
+    "Humor",
+    "JSONDiff",
     "LevenshteinMatch",
-    "ListContainsEval",
+    "ListContains",
     "MemoryTraceHandler",
-    "ModerationEval",
-    "NumericDiffEval",
-    "PossibleEval",
+    "Moderation",
+    "NumericDiff",
+    "Possible",
+    "RateLimitConfig",
     "ScoreThreshold",
-    "SecurityEval",
-    "SqlEval",
-    "SummaryEval",
-    "TranslationEval",
-    "ValidJSONEval",
+    "Security",
+    "Sql",
+    "Summary",
+    "Translation",
+    "ValidJSON",
     "assert_dataset_pass",
     "assert_pass",
     "capture_traces",
+    "configure_rate_limits",
     "create_llm_evaluator",
     "evaluate",
     "last_llm_call",

pixie_qa-0.4.0/pixie/assets/mock-data.json

@@ -0,0 +1,407 @@
+{
+    "command_args": "pixie test tests/",
+    "timestamp": "2025-06-15 12:00:00 UTC",
+    "pixie_repo_url": "https://github.com/yiouli/pixie-qa",
+    "feedback_url": "https://feedback.gopixie.ai/feedback",
+    "brand_icon_url": "https://github.com/user-attachments/assets/76c18199-f00a-4fb3-a12f-ce6c173727af",
+    "test_records": [
+        {
+            "name": "test_customer_faq.py::test_faq_factuality",
+            "status": "passed",
+            "message": null,
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockFactuality"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.85,
+                                    "reasoning": "High string similarity between expected and actual output.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.72,
+                                    "reasoning": "Moderate string similarity.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.91,
+                                    "reasoning": "Very high similarity match.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.68,
+                                    "reasoning": "Reasonable similarity.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.77,
+                                    "reasoning": "Good similarity match.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": true,
+                    "criteria_message": "Pass: 4/5 inputs passed (threshold 0.6, required 80%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.6. At least 80% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "name": "test_customer_faq.py::test_faq_multi_evaluator",
+            "status": "failed",
+            "message": "AssertionError: 3/5 inputs failed on at least one evaluator",
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockFactuality",
+                        "MockClosedQA"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.85,
+                                    "reasoning": "High similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.3,
+                                    "reasoning": "Low keyword overlap.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.72,
+                                    "reasoning": "Moderate similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.45,
+                                    "reasoning": "Below threshold keyword overlap.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.91,
+                                    "reasoning": "Very high similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.6,
+                                    "reasoning": "Acceptable keyword overlap.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.68,
+                                    "reasoning": "Reasonable similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.25,
+                                    "reasoning": "Poor keyword match.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.77,
+                                    "reasoning": "Good similarity.",
+                                    "details": {}
+                                },
+                                {
+                                    "score": 0.55,
+                                    "reasoning": "Marginal keyword overlap.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": false,
+                    "criteria_message": "Fail: only 2/5 inputs passed on all evaluators (required 100%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.5. At least 100% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "name": "test_customer_faq.py::test_faq_no_hallucinations",
+            "status": "passed",
+            "message": null,
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockHallucination"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.95,
+                                    "reasoning": "No hallucination detected.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": true,
+                    "criteria_message": "Pass: 5/5 inputs passed (threshold 0.5, required 100%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.5. At least 100% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "name": "test_customer_faq.py::test_faq_tone_check",
+            "status": "failed",
+            "message": "AssertionError: all 5 inputs failed",
+            "asserts": [
+                {
+                    "evaluator_names": [
+                        "MockStrictTone"
+                    ],
+                    "input_labels": [
+                        "What is your return policy?",
+                        "How do I track my order?",
+                        "Do you offer international shipping?",
+                        "What payment methods do you accept?",
+                        "How can I contact support?"
+                    ],
+                    "results": [
+                        [
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ],
+                            [
+                                {
+                                    "score": 0.2,
+                                    "reasoning": "Tone does not meet strict requirements.",
+                                    "details": {}
+                                }
+                            ]
+                        ]
+                    ],
+                    "passed": false,
+                    "criteria_message": "Fail: 0/5 inputs passed (threshold 0.5, required 100%)",
+                    "scoring_strategy": "Each evaluator score must be ≥ 0.5. At least 100% of test-case inputs must pass on all evaluators. Uses best-of-N-passes semantics (any single pass meeting criteria is sufficient).",
+                    "evaluable_dicts": [
+                        {
+                            "input": "What is your return policy?",
+                            "expected_output": "You can return items within 30 days of purchase for a full refund.",
+                            "actual_output": "Items can be returned within 30 days for a full refund.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How do I track my order?",
+                            "expected_output": "You can track your order using the tracking link sent to your email.",
+                            "actual_output": "Check the tracking link in your confirmation email to track your order.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "Do you offer international shipping?",
+                            "expected_output": "Yes, we ship to over 50 countries worldwide.",
+                            "actual_output": "Yes, we offer international shipping to over 50 countries.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "What payment methods do you accept?",
+                            "expected_output": "We accept Visa, Mastercard, PayPal, and Apple Pay.",
+                            "actual_output": "We accept major credit cards including Visa and Mastercard, plus PayPal and Apple Pay.",
+                            "metadata": {}
+                        },
+                        {
+                            "input": "How can I contact support?",
+                            "expected_output": "Email us at support@example.com or call 1-800-EXAMPLE.",
+                            "actual_output": "You can reach support at support@example.com or by calling 1-800-EXAMPLE.",
+                            "metadata": {}
+                        }
+                    ]
+                }
+            ]
+        }
+    ]
+}