@sanity/ailf 0.1.0

package/README.md

@@ -0,0 +1,89 @@
+# @sanity/ailf
+
+CLI and evaluation engine for the **AI Literacy Framework** — measures how
+effectively documentation enables AI coding tools to implement features
+correctly.
+
+## Installation
+
+```bash
+# Run without installing (recommended for quick start)
+npx @sanity/ailf --help
+
+# Or install globally
+pnpm add -g @sanity/ailf
+
+# Or as a project dependency
+pnpm add @sanity/ailf
+```
+
+## Quick start
+
+### 1. Initialize a project
+
+```bash
+npx @sanity/ailf init
+```
+
+This creates a `.ailf/` directory with example configuration and task files:
+
+```
+.ailf/
+├── config.yaml              # Project configuration
+├── .gitignore               # Keeps generated files out of VCS
+└── tasks/
+    ├── example-groq-blog-listing.yaml
+    └── example-studio-custom-input.yaml
+```
+
+### 2. Set up environment
+
+Create a `.env` file in your project root:
+
+```bash
+# Required — LLM provider for evaluation and grading
+OPENAI_API_KEY=sk-...
+
+# Required — read access to Sanity documentation content
+SANITY_API_TOKEN=sk...
+
+# Optional — publish reports to your Sanity Studio
+AILF_REPORT_SANITY_API_TOKEN=sk...
+AILF_REPORT_PROJECT_ID=your-project-id
+AILF_REPORT_DATASET=production
+```
+
+### 3. Edit tasks and run
+
+```bash
+# Edit .ailf/config.yaml with your Sanity project settings
+# Customize or replace the example tasks in .ailf/tasks/
+
+# Validate task definitions
+npx @sanity/ailf validate-tasks .ailf/tasks/
+
+# Run evaluation in debug mode (fast feedback)
+npx @sanity/ailf pipeline --repo-tasks-path .ailf/tasks/ --debug
+
+# Full evaluation
+npx @sanity/ailf pipeline --repo-tasks-path .ailf/tasks/
+```
+
+## Documentation
+
+- **[API Reference](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/API.md)**
+  — all commands, flags, and environment variables
+- **[Contributing Tasks](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/CONTRIBUTING_TASKS.md)**
+  — task authoring guide
+- **[Architecture](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/ARCHITECTURE.md)**
+  — domain model and data flow
+- **[Root README](https://github.com/sanity-labs/ai-literacy-framework)** —
+  project overview
+
+## Related packages
+
+| Package                                                                    | Description                                        |
+| -------------------------------------------------------------------------- | -------------------------------------------------- |
+| [`@sanity/ailf-studio`](https://www.npmjs.com/package/@sanity/ailf-studio) | Sanity Studio dashboard plugin for viewing reports |
+| `@sanity/ailf-core`                                                        | Domain kernel (types, schemas, ports)              |
+| `@sanity/ailf-shared`                                                      | Cross-package contract types                       |

package/bin/ailf.js

@@ -0,0 +1,64 @@
+#!/usr/bin/env node
+
+/**
+ * ailf — CLI launcher for the AI Literacy Framework.
+ *
+ * Runs in two modes:
+ * 1. **Development** (monorepo) — delegates to TypeScript source via tsx.
+ *    Detected by the presence of src/cli.ts.
+ * 2. **Production** (npm install) — imports the compiled dist/cli.js directly.
+ *    This is the path when installed via `npx @sanity/ailf-eval` or
+ *    `pnpm add @sanity/ailf-eval`.
+ */
+
+import { existsSync } from "fs"
+import { execFileSync } from "child_process"
+import { dirname, resolve } from "path"
+import { fileURLToPath } from "url"
+
+const __dirname = dirname(fileURLToPath(import.meta.url))
+const ROOT = resolve(__dirname, "..")
+const tsSrc = resolve(ROOT, "src", "cli.ts")
+const jsDist = resolve(ROOT, "dist", "cli.js")
+
+// Forward all arguments after the script name
+const args = process.argv.slice(2)
+
+// Preserve the caller's working directory so commands like `ailf init`
+// operate on the user's cwd, not the eval package root.
+const callerCwd = process.cwd()
+
+// ---------------------------------------------------------------------------
+// Mode 1: Development — tsx + TypeScript source (no build step needed)
+// ---------------------------------------------------------------------------
+if (existsSync(tsSrc)) {
+  try {
+    execFileSync("npx", ["tsx", tsSrc, ...args], {
+      cwd: ROOT,
+      env: { ...process.env, AILF_CALLER_CWD: callerCwd },
+      stdio: "inherit",
+    })
+    process.exit(0)
+  } catch (err) {
+    // execFileSync throws on non-zero exit codes — propagate them
+    const code =
+      err !== null && typeof err === "object" && "status" in err
+        ? err.status
+        : 1
+    process.exit(code)
+  }
+}
+
+// ---------------------------------------------------------------------------
+// Mode 2: Production — compiled JS (npm install / npx)
+// ---------------------------------------------------------------------------
+if (existsSync(jsDist)) {
+  // Set AILF_CALLER_CWD so cli.ts resolves paths relative to the caller
+  process.env.AILF_CALLER_CWD = callerCwd
+  await import(jsDist)
+} else {
+  console.error(
+    "ailf: Cannot find CLI entry point. Run 'pnpm build' or ensure tsx is available."
+  )
+  process.exit(1)
+}

package/canonical/grader-references/README.md

@@ -0,0 +1,88 @@
+# Grader References — Human-Graded Gold Standard
+
+> Human expert grades for validating LLM grader accuracy.
+
+## Purpose
+
+These YAML files contain **human-graded reference samples** — real LLM responses
+paired with expert human scores across all three dimensions (Task Completion,
+Code Correctness, Doc Coverage). The grader validation tool
+(`pnpm grader-validate`) runs the current grader on these responses and compares
+its scores against the human grades.
+
+## File format
+
+Each YAML file contains an array of reference-graded samples:
+
+```yaml
+- taskId: groq-blog-queries
+  area: groq
+  response: |
+    <the LLM's actual response text from an eval run>
+  rubrics:
+    - dimension: task-completion
+      rubricText: |
+        <the full rubric text used for grading>
+      humanScore: 75
+      notes: "Correct GROQ syntax but missed ordering clause"
+    - dimension: code-correctness
+      rubricText: |
+        <the full rubric text>
+      humanScore: 60
+    - dimension: doc-coverage
+      rubricText: |
+        <the full rubric text>
+      humanScore: 80
+  gradedBy: "engineer-name"
+  gradedAt: "2026-03-10"
+```
+
+## Grading protocol
+
+To minimize subjectivity when creating reference grades:
+
+1. **Use the same rubric text** the LLM grader uses (copy from the expanded task
+   file or the eval results JSON)
+2. **Two independent graders per sample**, reconcile disagreements
+3. **Focus on edge cases** — samples where the grader is likely to be wrong
+   (subtle bugs, plausible but incorrect code, partially correct responses)
+4. **Score on the 0–100 scale** matching the rubric bands:
+   - 0–19: Failed / missing critical info
+   - 20–49: Attempted but fundamentally wrong
+   - 50–79: Partial / mostly correct
+   - 80–100: Complete / fully correct
+5. **Include notes** explaining your reasoning, especially for non-obvious
+   scores. These help diagnose grader failures.
+
+## How to extract responses from eval results
+
+```bash
+# View a specific gold test response from eval results:
+python3 -c "
+import json
+with open('results/latest/eval-results.json') as f:
+    data = json.load(f)
+for r in data['results']['results']:
+    desc = r.get('testCase',{}).get('description','')
+    if 'groq-blog-queries' in desc.lower() and '(gold)' in desc:
+        print(r['response']['output'][:2000])
+        break
+"
+```
+
+## Coverage goals
+
+Start with 2–3 reference-graded samples per feature area, covering:
+
+- A high-quality response (scores mostly 80+)
+- A medium-quality response (scores 40–70)
+- A low-quality or incorrect response (scores 0–30)
+
+This gives the validation tool a range of quality levels to check the grader's
+discrimination power across the full scale.
+
+## Related
+
+- [Grader Reliability Plan](../../../docs/exec-plans/completed/grader-reliability.md)
+  — Phase 2
+- [Rubric Templates](../../config/rubrics.yaml) — the rubrics used for grading

package/canonical/grader-references/groq.yaml

@@ -0,0 +1,234 @@
+# groq.yaml — Human-graded reference samples for GROQ tasks
+#
+# These are real LLM responses from eval runs, graded by a human expert.
+# Used by `pnpm grader-validate` to measure grader accuracy.
+
+- taskId: groq-blog-queries
+  area: groq
+  response: |
+    To write GROQ queries for a Sanity blog application using `@sanity/client`, here is a complete TypeScript implementation:
+
+    ```typescript
+    import { createClient } from "@sanity/client";
+
+    const client = createClient({
+      projectId: "your-project-id",
+      dataset: "production",
+      apiVersion: "2024-01-01",
+      useCdn: true,
+    });
+
+    // Types
+    interface PostSummary {
+      _id: string;
+      title: string;
+      slug: string;
+      publishedAt: string;
+      excerpt: string;
+      authorName: string;
+    }
+
+    // 1. Fetch all published blog posts
+    const allPostsQuery = `*[_type == "post"] | order(publishedAt desc) [0...10] {
+      _id,
+      title,
+      "slug": slug.current,
+      publishedAt,
+      excerpt,
+      "authorName": author->name
+    }`;
+
+    export async function getAllPosts(): Promise<PostSummary[]> {
+      return client.fetch(allPostsQuery);
+    }
+
+    // 2. Fetch a single post by slug
+    const postBySlugQuery = `*[_type == "post" && slug.current == $slug][0] {
+      _id,
+      title,
+      "slug": slug.current,
+      publishedAt,
+      excerpt,
+      "authorName": author->name,
+      body,
+      "categories": categories[]->{ title, "slug": slug.current }
+    }`;
+
+    export async function getPostBySlug(slug: string) {
+      return client.fetch(postBySlugQuery, { slug });
+    }
+
+    // 3. Fetch posts after a date
+    const recentPostsQuery = `*[_type == "post" && dateTime(publishedAt) > dateTime($since)] | order(publishedAt desc) {
+      _id,
+      title,
+      "slug": slug.current,
+      publishedAt
+    }`;
+
+    export async function getRecentPosts(since: string) {
+      return client.fetch(recentPostsQuery, { since });
+    }
+
+    // 4. Fetch posts by category
+    const postsByCategoryQuery = `*[_type == "post" && $categoryId in categories[]._ref] | order(publishedAt desc) {
+      _id,
+      title,
+      "slug": slug.current,
+      publishedAt
+    }`;
+
+    export async function getPostsByCategory(categoryId: string) {
+      return client.fetch(postsByCategoryQuery, { categoryId });
+    }
+    ```
+  rubrics:
+    - dimension: task-completion
+      rubricText: |
+        Score task completion from 0 to 100:
+        - 0: Couldn't attempt — missing critical information
+        - 20: Attempted but fundamentally wrong approach
+        - 50: Partial implementation — major functional gaps
+        - 80: Mostly complete — minor issues or missing edge cases
+        - 100: Fully functional code — works as expected
+
+        Must demonstrate:
+        - GROQ filter with _type == "post"
+        - Projection with aliased slug field ("slug": slug.current)
+        - Reference resolution with -> for author
+        - Ordering with | order(publishedAt desc)
+        - Slice/pagination syntax [0...10] or [0..9]
+        - Parameterized query with $slug for single post fetch
+        - Date filtering with dateTime() or string comparison
+        - Category filtering using references or array contains
+
+        Return ONLY a JSON object: {"score": <number>, "reason": "<explanation>"}
+      humanScore: 95
+      notes:
+        "All 8 criteria met. Clean implementation with types, correct GROQ
+        syntax throughout."
+    - dimension: code-correctness
+      rubricText: |
+        Score code correctness from 0 to 100:
+        - 0: Broken code, syntax errors, or deprecated APIs
+        - 30: Works but uses anti-patterns or inefficient approaches
+        - 50: Works but not idiomatic
+        - 80: Follows most best practices
+        - 100: Follows all best practices, idiomatic implementation
+
+        Check for:
+        - Valid GROQ syntax (proper filter brackets, projection braces)
+        - Uses @sanity/client createClient + client.fetch()
+        - Correct parameter passing syntax ($param)
+        - Proper reference dereference with ->
+        - No deprecated patterns
+
+        Return ONLY a JSON object: {"score": <number>, "reason": "<explanation>"}
+      humanScore: 90
+      notes:
+        "All GROQ syntax correct. Proper use of createClient, client.fetch,
+        params, and ->. Minor: could use const assertions for query strings."
+    - dimension: doc-coverage
+      rubricText: |
+        Score documentation coverage from 0 to 100:
+        - 0: Had to hallucinate/guess most implementation details
+        - 30: Significant gaps — filled with assumptions
+        - 50: Some gaps — inferred from partial information
+        - 80: Minor gaps — almost everything was documented
+        - 100: Complete coverage — all necessary info was in docs
+
+        Return ONLY a JSON object: {"score": <number>, "reason": "<explanation>"}
+      humanScore: 85
+      notes:
+        "All patterns match documented APIs. dateTime() usage, reference syntax,
+        slice pagination — all from docs. Minor: apiVersion string is a
+        reasonable guess."
+  gradedBy: "senior-engineer"
+  gradedAt: "2026-03-10"
+
+- taskId: groq-blog-queries
+  area: groq
+  response: |
+    Here are GROQ queries for a blog:
+
+    ```javascript
+    import sanityClient from '@sanity/client'
+
+    const client = sanityClient({
+      projectId: 'abc123',
+      dataset: 'production'
+    })
+
+    // Get posts
+    const posts = await client.fetch('*[_type == "post"]{title, body}')
+
+    // Get post by slug
+    const post = await client.fetch('*[_type == "post" && slug == $slug]{...}', {slug: 'my-post'})
+
+    // Get recent posts
+    const recent = await client.fetch('*[_type == "post" && publishedAt > "2024-01-01"]')
+    ```
+  rubrics:
+    - dimension: task-completion
+      rubricText: |
+        Score task completion from 0 to 100:
+        - 0: Couldn't attempt — missing critical information
+        - 20: Attempted but fundamentally wrong approach
+        - 50: Partial implementation — major functional gaps
+        - 80: Mostly complete — minor issues or missing edge cases
+        - 100: Fully functional code — works as expected
+
+        Must demonstrate:
+        - GROQ filter with _type == "post"
+        - Projection with aliased slug field ("slug": slug.current)
+        - Reference resolution with -> for author
+        - Ordering with | order(publishedAt desc)
+        - Slice/pagination syntax [0...10] or [0..9]
+        - Parameterized query with $slug for single post fetch
+        - Date filtering with dateTime() or string comparison
+        - Category filtering using references or array contains
+
+        Return ONLY a JSON object: {"score": <number>, "reason": "<explanation>"}
+      humanScore: 30
+      notes:
+        "Only 3 of 8 criteria met (filter, parameterized slug, date filtering).
+        Missing: aliased slug, -> for author, ordering, pagination, category
+        filtering. Uses deprecated sanityClient import."
+    - dimension: code-correctness
+      rubricText: |
+        Score code correctness from 0 to 100:
+        - 0: Broken code, syntax errors, or deprecated APIs
+        - 30: Works but uses anti-patterns or inefficient approaches
+        - 50: Works but not idiomatic
+        - 80: Follows most best practices
+        - 100: Follows all best practices, idiomatic implementation
+
+        Check for:
+        - Valid GROQ syntax (proper filter brackets, projection braces)
+        - Uses @sanity/client createClient + client.fetch()
+        - Correct parameter passing syntax ($param)
+        - Proper reference dereference with ->
+        - No deprecated patterns
+
+        Return ONLY a JSON object: {"score": <number>, "reason": "<explanation>"}
+      humanScore: 20
+      notes:
+        "Uses deprecated sanityClient() instead of createClient(). slug == $slug
+        is wrong (should be slug.current). Missing apiVersion. Spread operator
+        {...} is not standard GROQ."
+    - dimension: doc-coverage
+      rubricText: |
+        Score documentation coverage from 0 to 100:
+        - 0: Had to hallucinate/guess most implementation details
+        - 30: Significant gaps — filled with assumptions
+        - 50: Some gaps — inferred from partial information
+        - 80: Minor gaps — almost everything was documented
+        - 100: Complete coverage — all necessary info was in docs
+
+        Return ONLY a JSON object: {"score": <number>, "reason": "<explanation>"}
+      humanScore: 25
+      notes:
+        "Appears to rely on outdated training data. Deprecated import, wrong
+        slug access pattern, no documented pagination or ordering patterns used."
+  gradedBy: "senior-engineer"
+  gradedAt: "2026-03-10"