pixie-qa 0.4.0__tar.gz → 0.5.0__tar.gz

{pixie_qa-0.4.0 → pixie_qa-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pixie-qa
-Version: 0.4.0
+Version: 0.5.0
 Summary: Automated quality assurance for AI applications
 Project-URL: Homepage, https://github.com/yiouli/pixie-qa
 Project-URL: Repository, https://github.com/yiouli/pixie-qa
@@ -44,7 +44,6 @@ Requires-Dist: openai>=2.29.0
 Requires-Dist: openinference-instrumentation>=0.1.44
 Requires-Dist: opentelemetry-api>=1.27.0
 Requires-Dist: opentelemetry-sdk>=1.27.0
-Requires-Dist: piccolo[sqlite]>=1.33.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: python-dotenv>=1.2.2
 Requires-Dist: starlette>=1.0.0
@@ -70,20 +69,19 @@ Description-Content-Type: text/markdown
 
 # pixie-qa
 
-An agent skill that make coding agent the QA engineer for LLM applications.
+An agent skill that makes coding agents the QA engineer for LLM applications.
 
 ## What the Skill Does
 
 The `qa-eval` skill guides your coding agent through the full eval-based QA loop for LLM applications:
 
 1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
-2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
-3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
-4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
-5. **Validate datasets** — `pixie dataset validate [dir_or_dataset_path]` to catch schema/config errors early
-6. **Run the tests** — `pixie test` to run all evals and report per-case scores
-7. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
-8. **Investigate failures** — look up the stored trace for each failure, diagnose, fix, repeat
+2. **Instrument it** — use `wrap()` for data-object tracing and OpenInference auto-instrumentation for LLM span capture
+3. **Build a dataset** — create JSON datasets of representative inputs and expected outputs
+4. **Write eval tests** — generate `test_*.py` files with appropriate evaluators
+5. **Run the tests** — `pixie test` to run all evals and report per-case scores
+6. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
+7. **Investigate failures** — diagnose failures, fix, repeat
 
 ## Getting Started
 
@@ -105,7 +103,13 @@ Your coding agent will read your code, instrument it, build a dataset from a few
 
 ## Python Package
 
-The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. For the package API and CLI reference, see [docs/package.md](docs/package.md).
+The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. API docs are auto-generated by `pdoc3` into [docs/pixie/index.md](docs/pixie/index.md) via pre-commit. The markdown renderer uses [scripts/pdoc_templates/text.mako](scripts/pdoc_templates/text.mako) so async functions and methods are explicitly shown as `async def` in signatures.
+
+Install hooks once per clone:
+
+```bash
+uv run pre-commit install
+```
 
 ## Web UI
 

{pixie_qa-0.4.0 → pixie_qa-0.5.0}/README.md

@@ -1,19 +1,18 @@
 # pixie-qa
 
-An agent skill that make coding agent the QA engineer for LLM applications.
+An agent skill that makes coding agents the QA engineer for LLM applications.
 
 ## What the Skill Does
 
 The `qa-eval` skill guides your coding agent through the full eval-based QA loop for LLM applications:
 
 1. **Understand the code** — read the codebase, trace the data flow, learn what the code is supposed to do
-2. **Instrument it** — add `enable_storage()` and `@observe` so every run is captured to a local SQLite database
-3. **Build a dataset** — save representative traces as test cases with `pixie dataset save`
-4. **Write eval tests** — generate `test_*.py` files with `assert_dataset_pass` and appropriate evaluators
-5. **Validate datasets** — `pixie dataset validate [dir_or_dataset_path]` to catch schema/config errors early
-6. **Run the tests** — `pixie test` to run all evals and report per-case scores
-7. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
-8. **Investigate failures** — look up the stored trace for each failure, diagnose, fix, repeat
+2. **Instrument it** — use `wrap()` for data-object tracing and OpenInference auto-instrumentation for LLM span capture
+3. **Build a dataset** — create JSON datasets of representative inputs and expected outputs
+4. **Write eval tests** — generate `test_*.py` files with appropriate evaluators
+5. **Run the tests** — `pixie test` to run all evals and report per-case scores
+6. **Analyse results** — `pixie analyze <test_id>` to get LLM-generated analysis of test results
+7. **Investigate failures** — diagnose failures, fix, repeat
 
 ## Getting Started
 
@@ -35,7 +34,13 @@ Your coding agent will read your code, instrument it, build a dataset from a few
 
 ## Python Package
 
-The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. For the package API and CLI reference, see [docs/package.md](docs/package.md).
+The `pixie-qa` Python package (imported as `pixie`) is what Claude installs and uses inside your project. API docs are auto-generated by `pdoc3` into [docs/pixie/index.md](docs/pixie/index.md) via pre-commit. The markdown renderer uses [scripts/pdoc_templates/text.mako](scripts/pdoc_templates/text.mako) so async functions and methods are explicitly shown as `async def` in signatures.
+
+Install hooks once per clone:
+
+```bash
+uv run pre-commit install
+```
 
 ## Web UI
 

pixie_qa-0.5.0/pixie/__init__.py

@@ -0,0 +1,90 @@
+"""pixie — automated quality assurance for AI applications.
+
+Re-exports the full public API so users can ``from pixie import ...``
+for every commonly used symbol without needing submodule paths.
+"""
+
+from pixie.eval.evaluable import Evaluable, TestCase
+from pixie.eval.evaluation import Evaluation, Evaluator, evaluate
+from pixie.eval.llm_evaluator import create_llm_evaluator
+from pixie.eval.scorers import (
+    AnswerCorrectness,
+    AnswerRelevancy,
+    AutoevalsAdapter,
+    Battle,
+    ClosedQA,
+    ContextRelevancy,
+    EmbeddingSimilarity,
+    ExactMatch,
+    Factuality,
+    Faithfulness,
+    Humor,
+    JSONDiff,
+    LevenshteinMatch,
+    ListContains,
+    Moderation,
+    NumericDiff,
+    Possible,
+    Security,
+    Sql,
+    Summary,
+    Translation,
+    ValidJSON,
+)
+
+# -- Harness ------------------------------------------------------------------
+from pixie.harness.runnable import Runnable
+
+# -- Instrumentation ----------------------------------------------------------
+from pixie.instrumentation.llm_tracing import (
+    add_handler,
+    enable_llm_tracing,
+    flush,
+    remove_handler,
+)
+from pixie.instrumentation.wrap import (
+    WrappedData,
+    wrap,
+)
+
+__all__ = [
+    # Instrumentation
+    "WrappedData",
+    "flush",
+    "enable_llm_tracing",
+    "add_handler",
+    "remove_handler",
+    "wrap",
+    # Harness
+    "Runnable",
+    # Eval data models
+    "Evaluable",
+    "TestCase",
+    "Evaluation",
+    "Evaluator",
+    "evaluate",
+    "create_llm_evaluator",
+    # Pre-made evaluators (autoevals adapters)
+    "AnswerCorrectness",
+    "AnswerRelevancy",
+    "AutoevalsAdapter",
+    "Battle",
+    "ClosedQA",
+    "ContextRelevancy",
+    "EmbeddingSimilarity",
+    "ExactMatch",
+    "Factuality",
+    "Faithfulness",
+    "Humor",
+    "JSONDiff",
+    "LevenshteinMatch",
+    "ListContains",
+    "Moderation",
+    "NumericDiff",
+    "Possible",
+    "Security",
+    "Sql",
+    "Summary",
+    "Translation",
+    "ValidJSON",
+]