@sanity/ailf-studio 0.1.0

package/README.md

@@ -0,0 +1,255 @@
+# @sanity/ailf-studio
+
+Sanity Studio dashboard plugin for the **AI Literacy Framework**. Visualizes
+evaluation reports, score trends, comparisons, and content impact — directly
+inside Sanity Studio with no external backend.
+
+All data is read from the Sanity Content Lake via GROQ.
+
+## Installation
+
+Install the plugin into any Sanity Studio that has access to the dataset where
+AILF reports are stored.
+
+### 1. Add the dependency
+
+#### Continuous releases (recommended for external projects)
+
+Every merge to `main` that touches `packages/studio/` automatically publishes
+via [pkg.pr.new](https://pkg.pr.new). Install the latest main build:
+
+```bash
+pnpm add https://pkg.pr.new/sanity-labs/ai-literacy-framework/@sanity/ailf-studio@main
+```
+
+Or pin to a specific commit:
+
+```bash
+pnpm add https://pkg.pr.new/sanity-labs/ai-literacy-framework/@sanity/ailf-studio@<commit-sha>
+```
+
+To update to the latest build, re-run the install command — the `@main` URL
+always resolves to the most recent build.
+
+#### PR preview packages
+
+PRs labeled `trigger: preview` also publish preview packages. Install URLs are
+posted as PR comments automatically.
+
+#### Within the monorepo
+
+```bash
+pnpm add @sanity/ailf-studio@workspace:*
+```
+
+### 2. Register the plugin
+
+The recommended approach registers both the document schemas and the dashboard
+tool in one call:
+
+```ts
+// sanity.config.ts
+import { defineConfig } from "sanity"
+import { ailfPlugin } from "@sanity/ailf-studio"
+
+export default defineConfig({
+  // ... your existing config
+  plugins: [
+    ailfPlugin(),
+    // ... other plugins
+  ],
+})
+```
+
+This registers:
+
+- The `ailf.report` document type (read-only evaluation reports)
+- The `ailf.webhookConfig` document type (webhook-triggered evaluation settings)
+- The **AI Literacy** dashboard tool in the Studio sidebar
+
+### 3. Alternative: tool-only installation
+
+If you only want the dashboard tool without the document schemas (e.g., the
+schemas are already registered elsewhere):
+
+```ts
+// sanity.config.ts
+import { defineConfig } from "sanity"
+import { ailfTool } from "@sanity/ailf-studio"
+
+export default defineConfig({
+  // ... your existing config
+  tools: [ailfTool()],
+})
+```
+
+### 4. Alternative: schema-only installation
+
+If you want the document schemas without the dashboard (e.g., to query reports
+programmatically):
+
+```ts
+// sanity.config.ts
+import { defineConfig } from "sanity"
+import { reportSchema, webhookConfigSchema } from "@sanity/ailf-studio"
+
+export default defineConfig({
+  // ... your existing config
+  schema: {
+    types: [reportSchema, webhookConfigSchema],
+  },
+})
+```
+
+## Dashboard Views
+
+The plugin provides five views accessible from tabs in the dashboard:
+
+### Latest Reports
+
+A card list of the most recent evaluation reports. Each card shows:
+
+- Overall score, doc lift, and lowest-scoring area
+- Evaluation mode, source, and trigger type
+- Git metadata (branch, PR number) when available
+- Auto-comparison delta against the previous run
+
+Click any card to navigate to the Report Detail view.
+
+### Score Timeline
+
+A line chart of overall and per-area scores over time. Filterable by:
+
+- **Source** — which documentation source was evaluated (e.g., production,
+  branch deploy)
+- **Mode** — evaluation mode (baseline, observed, agentic)
+
+Data points are interactive — click to jump to the full report.
+
+### Compare
+
+Side-by-side comparison of any two reports. Select a baseline and experiment
+report from dropdowns, then view:
+
+- Overall score and doc-lift deltas
+- Per-area deltas (improved / regressed / unchanged)
+- Per-model deltas (when both reports include per-model breakdowns)
+- Noise threshold classification
+
+### Content Impact
+
+Find all evaluation reports related to a specific Sanity document. Enter a
+document ID to see:
+
+- Which evaluations included that document in their target set
+- Score trends for that document's feature area over time
+- Whether edits to the document improved or regressed scores
+
+### Report Detail
+
+Full drill-down into a single report:
+
+- Per-area score table with all dimensions (task completion, code correctness,
+  doc coverage, lift from docs)
+- Per-model breakdowns with cost-per-quality-point
+- Provenance metadata (trigger, git info, grader model, context hash)
+- Auto-comparison summary against the previous comparable run
+- Link to the Promptfoo web viewer for raw evaluation output
+
+## Filtering
+
+The Dashboard and Score Timeline views share global filters:
+
+- **Source filter** — values are auto-populated from distinct
+  `provenance.source.name` values across all reports
+- **Mode filter** — values are auto-populated from distinct `provenance.mode`
+  values
+
+Filters are applied via GROQ query parameters, so only matching reports are
+fetched.
+
+## Dataset Configuration
+
+The plugin reads reports from whatever dataset the Studio is configured to use.
+To point it at a dedicated report dataset, configure the Studio's dataset:
+
+```ts
+export default defineConfig({
+  projectId: "3do82whm",
+  dataset: "my-report-dataset", // or use AILF_REPORT_DATASET
+  plugins: [ailfPlugin()],
+})
+```
+
+Reports are written by the evaluation pipeline (`turbo pipeline -- --publish`).
+See the [report store design docs](../../docs/design-docs/report-store/index.md)
+for the full architecture.
+
+## Exported API
+
+The plugin exports building blocks for custom views or extensions:
+
+### Plugin & Tool
+
+| Export       | Description                  |
+| ------------ | ---------------------------- |
+| `ailfPlugin` | Full plugin (schemas + tool) |
+| `ailfTool`   | Dashboard tool only          |
+
+### Schemas
+
+| Export                | Description                            |
+| --------------------- | -------------------------------------- |
+| `reportSchema`        | `ailf.report` document type definition |
+| `webhookConfigSchema` | `ailf.webhookConfig` document type     |
+
+### GROQ Queries
+
+| Export                 | Description                             |
+| ---------------------- | --------------------------------------- |
+| `latestReportsQuery`   | N most recent reports (filterable)      |
+| `scoreTimelineQuery`   | Score data points over time             |
+| `reportDetailQuery`    | Full report with all fields             |
+| `comparisonPairQuery`  | Two reports for side-by-side comparison |
+| `contentImpactQuery`   | Reports related to a document ID        |
+| `distinctSourcesQuery` | All unique source names                 |
+| `distinctModesQuery`   | All unique evaluation modes             |
+| `distinctAreasQuery`   | All unique feature areas                |
+
+### Types
+
+| Export              | Description                                    |
+| ------------------- | ---------------------------------------------- |
+| `ReportListItem`    | Shape returned by `latestReportsQuery`         |
+| `ReportDetail`      | Shape returned by `reportDetailQuery`          |
+| `TimelineDataPoint` | Shape returned by `scoreTimelineQuery`         |
+| `ComparisonData`    | Auto-comparison data embedded in reports       |
+| `ContentImpactItem` | Shape returned by `contentImpactQuery`         |
+| `ProvenanceData`    | Report provenance metadata                     |
+| `SummaryData`       | Score summary (overall + per-area + per-model) |
+| `ScoreItem`         | Individual area score entry                    |
+
+## Development
+
+```bash
+# Build the plugin
+pnpm --filter @sanity/ailf-studio build
+
+# Watch mode (rebuilds on file changes)
+pnpm --filter @sanity/ailf-studio dev
+
+# Build everything (from repo root)
+turbo build
+```
+
+The plugin is pure TypeScript (TSC compilation, no bundler). The consuming
+Studio's bundler (Vite) handles the final bundle.
+
+## Related Documentation
+
+- [Report Store Design](../../docs/design-docs/report-store/index.md) — full
+  architecture and implementation plan
+- [Visibility & Workflows](../../docs/design-docs/report-store/visibility-workflows.md)
+  — design rationale for the dashboard views
+- [Report Store Architecture](../../docs/design-docs/report-store/architecture.md)
+  — Sanity Content Lake as the system of record