@sanity/ailf-studio 0.1.6 → 0.1.8

package/README.md

@@ -13,28 +13,13 @@ AILF reports are stored.
 
 ### 1. Add the dependency
 
-#### Continuous releases (recommended for external projects)
-
-Every merge to `main` that touches `packages/studio/` automatically publishes
-via [pkg.pr.new](https://pkg.pr.new). Install the latest main build:
-
-```bash
-pnpm add https://pkg.pr.new/sanity-labs/ai-literacy-framework/@sanity/ailf-studio@main
-```
-
-Or pin to a specific commit:
-
 ```bash
-pnpm add https://pkg.pr.new/sanity-labs/ai-literacy-framework/@sanity/ailf-studio@<commit-sha>
+pnpm add @sanity/ailf-studio
 ```
 
-To update to the latest build, re-run the install command — the `@main` URL
-always resolves to the most recent build.
-
-#### PR preview packages
-
-PRs labeled `trigger: preview` also publish preview packages. Install URLs are
-posted as PR comments automatically.
+> **Note:** The package is published with `restricted` access to the `@sanity`
+> npm scope. You need an npm token with read access — see the root
+> [README](../../README.md#obtain-secrets) for how to obtain one.
 
 #### Within the monorepo
 
@@ -120,7 +105,8 @@ export default defineConfig({
 
 ## Dashboard Views
 
-The plugin provides five views accessible from tabs in the dashboard:
+The plugin provides three tab views plus a detail drill-down, accessible from
+the **AI Literacy Framework** tool in the Studio sidebar.
 
 ### Latest Reports
 
@@ -128,11 +114,14 @@ A card list of the most recent evaluation reports. Each card shows:
 
 - Overall score, doc lift, and lowest-scoring area
 - Evaluation mode, source, and trigger type
-- Git metadata (branch, PR number) when available
+- Git metadata (branch, PR number, origin repo) when available
 - Auto-comparison delta against the previous run
 
 Click any card to navigate to the Report Detail view.
 
+The view includes a **search bar** for filtering reports by document slug, area,
+or content release perspective.
+
 ### Score Timeline
 
 A line chart of overall and per-area scores over time. Filterable by:
@@ -153,25 +142,22 @@ report from dropdowns, then view:
 - Per-model deltas (when both reports include per-model breakdowns)
 - Noise threshold classification
 
-### Content Impact
-
-Find all evaluation reports related to a specific Sanity document. Enter a
-document ID to see:
-
-- Which evaluations included that document in their target set
-- Score trends for that document's feature area over time
-- Whether edits to the document improved or regressed scores
-
 ### Report Detail
 
-Full drill-down into a single report:
-
-- Per-area score table with all dimensions (task completion, code correctness,
-  doc coverage, lift from docs)
-- Per-model breakdowns with cost-per-quality-point
-- Provenance metadata (trigger, git info, grader model, context hash)
-- Auto-comparison summary against the previous comparable run
-- Link to the Promptfoo web viewer for raw evaluation output
+Full drill-down into a single report (navigated from Latest Reports or a direct
+URL):
+
+- **Overview stats** — composite score, doc lift, cost, duration
+- **Per-area score table** with all dimensions (task completion, code
+  correctness, doc coverage, lift from docs)
+- **Three-layer table** — floor / ceiling / actual decomposition (when
+  available)
+- **Per-model breakdowns** with cost-per-quality-point
+- **Judgment list** — individual grader verdicts with reasoning
+- **Recommendations** — gap analysis remediation suggestions (when available)
+- **Provenance card** — trigger, git info (branch, PR, origin repo), grader
+  model, context hash, eval fingerprint
+- **Auto-comparison summary** against the previous comparable run
 
 ## Filtering
 
@@ -198,13 +184,13 @@ export default defineConfig({
 })
 ```
 
-Reports are written by the evaluation pipeline (`turbo pipeline -- --publish`).
-See the [report store design docs](../../docs/design-docs/report-store/index.md)
-for the full architecture.
+Reports are written by the evaluation pipeline (`ailf pipeline --publish`). See
+the [report store design docs](../../docs/design-docs/report-store/index.md) for
+the full architecture.
 
 ## Exported API
 
-The plugin exports building blocks for custom views or extensions:
+The plugin exports building blocks for custom views or extensions.
 
 ### Plugin & Tool
 
@@ -230,6 +216,7 @@ The plugin exports building blocks for custom views or extensions:
 | ------------------- | --------------------------------------------------------------------------------------------- |
 | `AssertionInput`    | Custom input for task assertions with contextual type descriptions and monospace code styling |
 | `CanonicalDocInput` | Custom input for canonical doc references with polymorphic resolution type help               |
+| `ReleasePicker`     | Content release perspective picker for evaluation scoping                                     |
 | `MirrorBanner`      | Banner showing repo source, sync status, and provenance for mirrored tasks                    |
 | `SyncStatusBadge`   | Colored badge (green/yellow/red) showing sync freshness of mirrored tasks                     |
 
@@ -240,31 +227,58 @@ The plugin exports building blocks for custom views or extensions:
 | `GraduateToNativeAction`    | Converts a mirrored (read-only) task to a native (editable) task by removing origin |
 | `createRunEvaluationAction` | Factory for creating a Studio action that triggers evaluations                      |
 
+### Glossary
+
+| Export     | Description                                                              |
+| ---------- | ------------------------------------------------------------------------ |
+| `GLOSSARY` | Centralized tooltip descriptions for all evaluation metrics and concepts |
+
 ### GROQ Queries
 
-| Export                 | Description                             |
-| ---------------------- | --------------------------------------- |
-| `latestReportsQuery`   | N most recent reports (filterable)      |
-| `scoreTimelineQuery`   | Score data points over time             |
-| `reportDetailQuery`    | Full report with all fields             |
-| `comparisonPairQuery`  | Two reports for side-by-side comparison |
-| `contentImpactQuery`   | Reports related to a document ID        |
-| `distinctSourcesQuery` | All unique source names                 |
-| `distinctModesQuery`   | All unique evaluation modes             |
-| `distinctAreasQuery`   | All unique feature areas                |
+| Export                         | Description                                |
+| ------------------------------ | ------------------------------------------ |
+| `latestReportsQuery`           | N most recent reports (filterable)         |
+| `scoreTimelineQuery`           | Score data points over time                |
+| `reportDetailQuery`            | Full report with all fields                |
+| `comparisonPairQuery`          | Two reports for side-by-side comparison    |
+| `contentImpactQuery`           | Reports related to a document ID           |
+| `recentDocumentEvalsQuery`     | Recent evaluations for a specific document |
+| `articleSearchQuery`           | Full-text search across article documents  |
+| `distinctSourcesQuery`         | All unique source names                    |
+| `distinctModesQuery`           | All unique evaluation modes                |
+| `distinctAreasQuery`           | All unique feature areas                   |
+| `distinctModelsQuery`          | All unique model identifiers               |
+| `distinctPerspectivesQuery`    | All unique content release perspectives    |
+| `distinctTargetDocumentsQuery` | All unique target document slugs           |
 
 ### Types
 
-| Export              | Description                                    |
-| ------------------- | ---------------------------------------------- |
-| `ReportListItem`    | Shape returned by `latestReportsQuery`         |
-| `ReportDetail`      | Shape returned by `reportDetailQuery`          |
-| `TimelineDataPoint` | Shape returned by `scoreTimelineQuery`         |
-| `ComparisonData`    | Auto-comparison data embedded in reports       |
-| `ContentImpactItem` | Shape returned by `contentImpactQuery`         |
-| `ProvenanceData`    | Report provenance metadata                     |
-| `SummaryData`       | Score summary (overall + per-area + per-model) |
-| `ScoreItem`         | Individual area score entry                    |
+| Export                       | Description                                                           |
+| ---------------------------- | --------------------------------------------------------------------- |
+| `ReportListItem`             | Shape returned by `latestReportsQuery`                                |
+| `ReportDetail`               | Shape returned by `reportDetailQuery`                                 |
+| `TimelineDataPoint`          | Shape returned by `scoreTimelineQuery`                                |
+| `ComparisonData`             | Auto-comparison data embedded in reports                              |
+| `ContentImpactItem`          | Shape returned by `contentImpactQuery`                                |
+| `ProvenanceData`             | Report provenance metadata                                            |
+| `SummaryData`                | Score summary (overall + per-area + per-model)                        |
+| `ScoreItem`                  | Individual area score entry                                           |
+| `RecommendationGap`          | Single gap analysis recommendation                                    |
+| `RecommendationsData`        | Full recommendations payload                                          |
+| `JudgmentData`               | Individual grader judgment with reasoning                             |
+| `DocumentRef`                | Canonical document reference (re-exported from `@sanity/ailf-shared`) |
+| `ScoreGrade`                 | Letter grade type (re-exported from `@sanity/ailf-shared`)            |
+| `scoreGrade`                 | Function to compute letter grade from numeric score                   |
+| `RunEvaluationActionOptions` | Options for `createRunEvaluationAction` factory                       |
+
+### Utility Functions
+
+| Export               | Description                                               |
+| -------------------- | --------------------------------------------------------- |
+| `formatPercent`      | Format a number as a percentage string                    |
+| `formatRelativeTime` | Format an ISO timestamp as relative time (e.g., "2h ago") |
+| `formatDelta`        | Format a score delta with +/− sign                        |
+| `formatDuration`     | Format milliseconds as human-readable duration            |
 
 ## Development
 
@@ -279,8 +293,8 @@ pnpm --filter @sanity/ailf-studio dev
 turbo build
 ```
 
-The plugin is pure TypeScript (TSC compilation, no bundler). The consuming
-Studio's bundler (Vite) handles the final bundle.
+The plugin uses [tsup](https://github.com/egoist/tsup) for bundling. The
+consuming Studio's bundler (Vite) handles the final bundle.
 
 ## Related Documentation
 

package/dist/index.d.ts

@@ -193,6 +193,15 @@ declare const GLOSSARY: {
     readonly failureMode: "The type of documentation problem: missing-docs (functionality not covered), incorrect-docs (factual errors), outdated-docs (stale API/patterns), or poor-structure (hard to find/understand).";
     readonly estimatedLift: "Estimated composite score improvement if this gap is fully fixed. Based on raising bottleneck dimensions to the median of non-bottlenecked dimensions.";
     readonly confidence: "How confident the classifier is in this diagnosis. High = strong keyword + structural signal agreement. Medium = partial agreement. Low = weak signals only.";
+    readonly agentBehaviorOverview: "How AI agents interacted with your documentation during evaluation: what they searched for, which pages they visited, and how much time they spent on network requests.";
+    readonly searchQueries: "The exact search queries agents used to find documentation. Helps you understand how agents discover your content and whether your docs appear for relevant queries.";
+    readonly docSlugsVisited: "Documentation page slugs that agents actually visited during evaluation. Compare against canonical docs to see if agents found the right pages.";
+    readonly externalDomains: "Non-Sanity domains that agents contacted during evaluation. High external domain counts may indicate agents couldn't find what they needed in your docs.";
+    readonly avgDocPagesVisited: "Average number of documentation pages visited per test. Higher counts can mean agents need to consult many pages (complex task) or can't find the right one quickly.";
+    readonly avgSearchesPerformed: "Average number of web searches performed per test. High search counts can indicate docs are hard to discover through search engines.";
+    readonly avgNetworkTimeMs: "Average time spent on network requests per test. Includes page fetches, search queries, and API calls.";
+    readonly totalRequests: "Total number of HTTP requests the agent made during the test, including searches, page visits, and API calls.";
+    readonly totalBytesDownloaded: "Total bytes downloaded by the agent. Large downloads may indicate the agent is fetching many pages or very large documents.";
     readonly dimTaskCompletion: "Change in task completion between runs. Positive means implementations are more complete.";
     readonly dimCodeCorrectness: "Change in code correctness between runs. Positive means better code quality.";
     readonly dimDocCoverage: "Change in doc coverage between runs. Positive means the docs are providing more useful information.";
@@ -656,8 +665,30 @@ interface JudgmentData {
     score: number;
     taskId: string;
 }
+/** Per-feature agent behavior data — how agents interacted with docs */
+interface FeatureAgentBehaviorData {
+    avgDocPagesVisited: number;
+    avgNetworkTimeMs: number;
+    avgSearchesPerformed: number;
+    docSlugsVisited: string[];
+    externalDomains: string[];
+    feature: string;
+    searchQueries: string[];
+    tasksWithBehaviorData: number;
+}
+/** Overall agent behavior stats (aggregated across all features) */
+interface OverallAgentBehaviorData {
+    avgDocPagesVisited: number;
+    avgNetworkTimeMs: number;
+    avgSearchesPerformed: number;
+    testsWithBehaviorData: number;
+    totalUniqueDocSlugs: number;
+    totalUniqueSearchQueries: number;
+}
 /** Summary data as stored in Sanity */
 interface SummaryData {
+    /** Per-feature agent behavior data (only present when agentic mode ran) */
+    agentBehavior?: FeatureAgentBehaviorData[] | null;
     belowCritical: string[];
     /** All Sanity documents used across the entire evaluation */
     documentManifest?: DocumentRef[];
@@ -665,6 +696,8 @@ interface SummaryData {
     lowestArea: string;
     lowestScore: number;
     overall: {
+        /** Aggregate agent behavior stats (only present when agentic mode ran) */
+        agentBehavior?: OverallAgentBehaviorData;
         avgDocLift: number;
         avgScore: number;
         avgCeilingScore?: number;