@huydao/karrot 0.1.1

package/GUIDE.md

@@ -0,0 +1,484 @@
+# Karrot Guide
+
+This file is the operational reference for using `karrot`.
+
+It is written so:
+- a human engineer can onboard quickly
+- an AI agent can read the file and infer the expected integration pattern
+
+## 1. Purpose
+
+Use `karrot` when you need a reusable AI-level test framework that can:
+- send one or more user turns to an assistant
+- collect assistant output
+- assert response quality
+- evaluate responses with named metrics
+- write stable artifacts and reports
+
+Use cases:
+- product AI chat testing
+- multi-turn assistant regression testing
+- eval-based quality monitoring
+- prompt or orchestration regression checks
+
+## 2. Boundary
+
+`karrot` owns:
+- scenario modeling
+- scenario execution
+- transport execution
+- assertion and eval execution
+- report generation
+
+The consumer project owns:
+- auth/login
+- runtime discovery
+- transport secrets and IDs
+- project-specific scenario context
+- transport variable preparation
+
+Do not put product-specific auth logic into `karrot`.
+
+## 3. Primary Public API
+
+Use these exports first:
+- `execute`
+- `AiScenarioSet`
+- `aiGen`
+
+Use lower-level APIs only when needed:
+- `runScenario`
+- `loadConfig`
+- `resolveVariables`
+- `writeScenarioRunReport`
+
+Preferred pattern:
+
+```ts
+import { execute } from '@huydao/karrot';
+
+await execute('./karrot.config.yml', {
+  variables,
+  scenario: {
+    file: './src/scenarios/my-scenarios.ts',
+    ids: ['S1'],
+  },
+});
+```
+
+## 4. Required Scenario Module Shape
+
+A scenario module must export:
+- `scenarioSet`
+- `buildScenarioContext(projectId)`
+
+Example:
+
+```ts
+import { AiScenarioSet, type AiScenario, type BaseAiScenarioContext } from '@huydao/karrot';
+
+type DemoContext = BaseAiScenarioContext & {
+  projectLabel: string;
+};
+
+const scenarios: AiScenario<DemoContext>[] = [
+  {
+    id: 'S1',
+    name: 'Greeting flow',
+    turns: [
+      {
+        label: 'Turn 1',
+        message: (context) => `Tell me about ${context.projectLabel}.`,
+      },
+    ],
+  },
+];
+
+export const scenarioSet = new AiScenarioSet(scenarios);
+
+export function buildScenarioContext(projectId: string): DemoContext {
+  return {
+    projectId,
+    projectLabel: 'RA Sample Project',
+  };
+}
+```
+
+## 5. Turn Model
+
+Each turn supports:
+- `label`
+- `message`
+- `idleTimeoutMs`
+- `processTimeoutMs`
+- `assertions`
+- `eval`
+- `onComplete`
+
+### Message forms
+
+`message` can be:
+- a function `(context) => string`
+- `aiGen.fromPreviousContext()`
+- `aiGen.fromGuidance(guidance)`
+- `aiGen.fromContent(content)`
+
+### Example
+
+```ts
+{
+  label: 'Turn 2',
+  message: aiGen.fromGuidance(
+    'Ask for 3 follow-up prompts the user can send next.',
+  ),
+  assertions: [
+    { assert: { hasText: 'prompt' } },
+  ],
+  eval: ['correctness', 'helpfulness'],
+}
+```
+
+## 6. Assertions
+
+Karrot supports two assertion families.
+
+### Direct assertions
+
+Use direct assertions when the expected outcome is deterministic enough.
+
+Supported forms:
+- `assert.hasText`
+- `assert.toolcall`
+
+Examples:
+
+```ts
+assertions: [
+  { assert: { hasText: 'Katalon AI' } },
+  { assert: { toolcall: ['search_document'] } },
+  { assert: { toolcall: [] } },
+]
+```
+
+### AI assertions
+
+Use AI assertions when the check is semantic.
+
+Supported forms:
+- `aiAssert.hasContent`
+- `aiAssert.notHasContent`
+
+Examples:
+
+```ts
+assertions: [
+  {
+    aiAssert: {
+      hasContent: 'The answer should describe concrete next steps for the user.',
+    },
+  },
+  {
+    aiAssert: {
+      notHasContent: 'The answer invents unsupported product capabilities.',
+    },
+  },
+]
+```
+
+## 7. Evaluations
+
+Eval is separate from assertion.
+
+Use assertion to decide pass or fail against a requirement.
+Use eval to score response quality along named dimensions.
+
+### Built-in dimensions
+
+Common built-in dimensions:
+- `correctness`
+- `coverage`
+- `helpfulness`
+- `clarity`
+- `completeness`
+- `conciseness`
+- `relevance`
+- `actionability`
+- `structure`
+- `consistency`
+- `safety`
+
+### Custom dimensions
+
+Use either:
+- inline guidance
+- project-level prompt files
+
+Inline guidance example:
+
+```ts
+eval: [
+  'correctness',
+  {
+    dimension: 'productFit',
+    guidance: 'Judge whether the answer is specifically useful for this product domain.',
+  },
+]
+```
+
+Project-level prompt example:
+
+```yml
+evaluation:
+  promptDirectory: ./prompts/eval
+```
+
+Then add files such as:
+- `./prompts/eval/product-fit.md`
+- `./prompts/eval/next-step-quality.md`
+
+Scenario then only needs:
+
+```ts
+eval: ['correctness', 'productFit']
+```
+
+### Full prompt override
+
+If the whole eval rubric changes, use:
+
+```yml
+evaluation:
+  systemPromptPath: ./prompts/turn-eval-testcase-generation-system-prompt.md
+```
+
+Use `systemPromptPath` for a full override.
+Use `promptDirectory` for additive project-specific guidance.
+
+## 8. Config Shape
+
+Karrot config is versioned.
+
+Current shape:
+
+```yml
+version: 1
+transport: ...
+artifacts:
+  directory: ./artifacts
+execution:
+  stopOnFailure: false
+evaluation:
+  systemPromptPath: ./prompts/...
+  promptDirectory: ./prompts/eval
+context:
+  projectId: ${PROJECT_ID}
+report:
+  enabled: true
+  environment: prod
+  projectName: Demo Project
+  runtime: ...
+  scenarioContext: ...
+```
+
+Important design choice:
+- config and scenario are separate
+- `execute()` receives the scenario file independently
+
+That allows one transport config to run many scenario files.
+
+## 9. Variable Resolution
+
+Karrot resolves `${VARIABLE}` from:
+- `options.variables`
+- then `process.env`
+
+Example:
+
+```yml
+context:
+  projectId: ${PROJECT_ID}
+```
+
+```ts
+await execute('./karrot.config.yml', {
+  variables: {
+    PROJECT_ID: '3422056',
+  },
+  scenario: {
+    file: './src/scenarios/basic-two-turn-demo.ts',
+  },
+});
+```
+
+## 10. Transport Patterns
+
+### AG-UI-WSS
+
+Use when the target assistant follows the AG-UI WSS contract.
+
+Config shape:
+
+```yml
+transport:
+  type: ag-ui-wss
+  env:
+    JWT: ${JWT}
+    ACCOUNT_ID: ${ACCOUNT_ID}
+    PROJECT_ID: ${PROJECT_ID}
+    AGENT_URL: ${AGENT_URL}
+    AGENT_ID: ${AGENT_ID}
+    WS_URL: ${WS_URL}
+    WS_TOPIC: ${WS_TOPIC}
+  processTimeoutMs: 120000
+```
+
+Notes:
+- `karrot` uses `ag-ui-wss` as the primary transport
+- raw AG-UI events are written to `.jsonl`
+- assistant output is parsed from `TEXT_MESSAGE_CONTENT`
+
+### AG-UI-POST
+
+Use when the target assistant exposes AG-UI-style HTTP endpoints.
+
+Karrot supports:
+- submit-only
+- submit + connect
+- submit + observe poll
+
+Use this only when the project runtime requires HTTP-based AG-UI, not WSS.
+
+## 11. Artifacts and Reports
+
+Each run creates:
+- `artifacts/<timestamp>/`
+- raw transport logs
+- one JSON report
+- one HTML report
+
+The report includes:
+- environment
+- runtime snapshot
+- scenario context
+- scenario status
+- turn outputs
+- assertion results
+- eval results
+- timing metrics
+
+## 12. OpenAI Requirements
+
+OpenAI features are used for:
+- AI assertions
+- turn evaluation
+- generated user messages
+
+Required:
+- `OPENAI_API_KEY`
+
+Optional:
+- `OPENAI_BASE_URL`
+- `OPENAI_EVAL_MODEL`
+- `OPENAI_MESSAGE_GEN_MODEL`
+
+Defaults:
+- eval model: `gpt-5.4`
+- message generation model: `gpt-5.4-mini`
+
+## 13. Recommended Integration Pattern
+
+For a new consumer project:
+
+1. prepare auth/runtime outside `karrot`
+2. map runtime values to variables
+3. create a `karrot.config.yml`
+4. create one scenario module
+5. call `execute()`
+
+Example structure:
+
+```text
+consumer-project/
+├── karrot.config.yml
+├── prompts/
+│   └── eval/
+├── src/
+│   ├── runtime/
+│   ├── run-demo.ts
+│   └── scenarios/
+└── artifacts/
+```
+
+## 14. Debugging Checklist
+
+If a run fails:
+
+1. inspect the JSON report
+2. inspect the HTML report
+3. inspect raw transport files in the artifact directory
+4. verify the scenario message that was actually sent
+5. verify transport variables after `${...}` resolution
+6. verify `OPENAI_API_KEY` if AI assertion, eval, or message generation is used
+
+For AG-UI-WSS specifically:
+- confirm JSONL contains `TEXT_MESSAGE_CONTENT`
+- confirm `WS_TOPIC` and auth headers are correct
+- do not treat transport progress logs as assistant output
+
+## 15. Rules for AI Agents
+
+If you are an AI agent editing or using `karrot`, follow these rules:
+
+1. Keep product-specific login/runtime discovery outside `karrot`.
+2. Prefer `execute()` unless there is a concrete reason to use lower-level APIs.
+3. Keep transport config in YAML and scenario selection separate from config.
+4. Use scenario context for product data, not hard-coded strings scattered across turns.
+5. Use `promptDirectory` for project-specific eval rubric extensions.
+6. Use `systemPromptPath` only when replacing the entire eval rubric.
+7. Prefer direct assertions when exact checks are stable; use AI assertions only for semantic checks.
+8. Preserve raw artifacts. Do not hide transport logs needed for debugging.
+
+## 16. Quick Reference
+
+### Minimal execution
+
+```ts
+await execute('./karrot.config.yml', {
+  variables,
+  scenario: {
+    file: './src/scenarios/basic-two-turn-demo.ts',
+  },
+});
+```
+
+### Select specific scenarios
+
+```ts
+await execute('./karrot.config.yml', {
+  variables,
+  scenario: {
+    file: './src/scenarios/my-scenarios.ts',
+    ids: ['S1', 'S3'],
+  },
+});
+```
+
+### Generate a user message
+
+```ts
+message: aiGen.fromGuidance('Ask for a shorter follow-up question.')
+```
+
+### Use project prompt files
+
+```yml
+evaluation:
+  promptDirectory: ./prompts/eval
+```
+
+### Override full eval system prompt
+
+```yml
+evaluation:
+  systemPromptPath: ./prompts/turn-eval-testcase-generation-system-prompt.md
+```

package/README.md

@@ -0,0 +1,253 @@
+# karrot
+
+`karrot` is a reusable AI test runner for multi-turn assistant scenarios.
+
+It gives you:
+- scenario execution
+- AG-UI transport integration
+- string and AI-based assertions
+- turn evaluation with OpenAI
+- JSON and HTML reports
+
+This package is designed to be published independently and reused across projects.
+
+## What Karrot Owns
+
+`karrot` is responsible for the AI-test layer:
+- load config
+- resolve `${VARIABLE}` templates
+- load scenario modules
+- execute turns
+- run assertions and evals
+- write artifacts and reports
+
+`karrot` does not own product-specific runtime discovery. The consumer project should prepare data such as:
+- `PROJECT_ID`
+- `JWT`
+- `ACCOUNT_ID`
+- `WS_URL`
+- `WS_TOPIC`
+- any transport-specific headers or IDs
+
+## Core Entry Point
+
+The main high-level API is `execute()`.
+
+```ts
+import { execute } from '@huydao/karrot';
+
+await execute('./karrot.config.yml', {
+  variables: {
+    PROJECT_ID: '3422056',
+    JWT: process.env.JWT,
+    ACCOUNT_ID: process.env.ACCOUNT_ID,
+    WS_URL: process.env.WS_URL,
+    WS_TOPIC: process.env.WS_TOPIC,
+  },
+  scenario: {
+    file: './src/scenarios/basic-two-turn-demo.ts',
+  },
+});
+```
+
+`execute()` will:
+1. load YAML or JSON config
+2. resolve `${...}` variables
+3. create `artifacts/<timestamp>`
+4. load the scenario module
+5. run selected scenarios
+6. write JSON and HTML reports
+
+## Scenario Authoring
+
+A scenario module exports:
+- `scenarioSet`
+- `buildScenarioContext(projectId)`
+
+Minimal example:
+
+```ts
+import { AiScenarioSet, type AiScenario, type BaseAiScenarioContext } from '@huydao/karrot';
+
+const scenarios: AiScenario<BaseAiScenarioContext>[] = [
+  {
+    id: 'BASIC-2T',
+    name: 'Basic Two-Turn Demo',
+    turns: [
+      {
+        label: 'Turn 1',
+        message: () => 'Hello. What can you help me with in Katalon AI?',
+      },
+      {
+        label: 'Turn 2',
+        message: () => 'Give me 3 short example prompts I can ask next.',
+      },
+    ],
+  },
+];
+
+export const scenarioSet = new AiScenarioSet(scenarios);
+
+export function buildScenarioContext(projectId: string): BaseAiScenarioContext {
+  return { projectId };
+}
+```
+
+## Assertions
+
+Karrot supports two assertion styles.
+
+Direct assertions:
+
+```ts
+assertions: [
+  { assert: { hasText: 'Katalon AI' } },
+  { assert: { toolcall: [] } },
+]
+```
+
+AI assertions:
+
+```ts
+assertions: [
+  { aiAssert: { hasContent: 'The answer explains what Katalon AI can do.' } },
+  { aiAssert: { notHasContent: 'The answer invents unsupported product features.' } },
+]
+```
+
+## Evaluations
+
+Turn evals score the assistant response for named dimensions.
+
+```ts
+eval: ['correctness', 'coverage', 'helpfulness']
+```
+
+Custom dimensions are also supported:
+
+```ts
+eval: [
+  'correctness',
+  {
+    dimension: 'productFit',
+    guidance: 'Judge whether the answer is specifically useful for a Katalon AI user.',
+  },
+]
+```
+
+Project-level eval prompts can be configured through:
+- `evaluation.systemPromptPath`
+- `evaluation.promptDirectory`
+
+That lets the project define rubric files without repeating inline guidance in every scenario.
+
+## AI-Generated User Messages
+
+Karrot can generate a user turn message before sending it to the target assistant.
+
+Available helpers:
+- `aiGen.fromPreviousContext()`
+- `aiGen.fromGuidance(guidance)`
+- `aiGen.fromContent(content)`
+
+Example:
+
+```ts
+import { aiGen } from '@huydao/karrot';
+
+message: aiGen.fromGuidance(
+  'Ask for 3 concise follow-up prompts the user can send next based on the previous answer.',
+)
+```
+
+This requires `OPENAI_API_KEY`.
+
+## Config Overview
+
+Karrot config currently supports:
+- `transport`
+- `artifacts.directory`
+- `execution.stopOnFailure`
+- `evaluation.systemPromptPath`
+- `evaluation.promptDirectory`
+- `context`
+- `report`
+
+Example `ag-ui-wss` config:
+
+```yml
+version: 1
+
+transport:
+  type: ag-ui-wss
+  env:
+    JWT: ${JWT}
+    ACCOUNT_ID: ${ACCOUNT_ID}
+    PROJECT_ID: ${PROJECT_ID}
+    AGENT_URL: ${AGENT_URL}
+    AGENT_ID: ${AGENT_ID}
+    WS_URL: ${WS_URL}
+    WS_TOPIC: ${WS_TOPIC}
+    WS_STOMP_HEADERS: Authorization:${JWT}
+    WS_HEADERS: Origin:${WS_ORIGIN},User-Agent:Mozilla/5.0
+
+context:
+  projectId: ${PROJECT_ID}
+
+report:
+  environment: prod
+  projectName: Demo Project
+  runtime:
+    agentUrl: ${AGENT_URL}
+    agentId: ${AGENT_ID}
+    wsUrl: ${WS_URL}
+    wsTopic: ${WS_TOPIC}
+    accountId: ${ACCOUNT_ID}
+    projectId: ${PROJECT_ID}
+    appBaseUrl: ${APP_BASE_URL}
+```
+
+## Reports and Artifacts
+
+Each `execute()` run creates:
+- a run artifact directory under `artifacts/<timestamp>`
+- raw transport logs such as `.jsonl` or `.sse`
+- a JSON run report
+- an HTML run report
+
+## Environment Variables
+
+Common variables:
+- `OPENAI_API_KEY`
+- `OPENAI_BASE_URL`
+- `OPENAI_EVAL_MODEL`
+- `OPENAI_MESSAGE_GEN_MODEL`
+
+Transport-specific variables depend on the integration project.
+
+## Package Structure
+
+- `assertions/`: direct assertions and turn evaluation
+- `executors/`: transport runners and scenario execution
+- `reports/`: JSON and HTML reporting
+- `scenarios/`: scenario types, loaders, generated-message helpers
+- `utils/`: config loading, artifacts, OpenAI helpers
+- `prompts/`: built-in prompt files used by the package
+
+## AI-Friendly Guide
+
+For a fuller operational guide intended for both humans and AI agents, read [GUIDE.md](./GUIDE.md).
+
+## Build
+
+```bash
+cd karrot
+npx tsc -p tsconfig.json
+```
+
+## Publish
+
+```bash
+cd karrot
+npm publish
+```