@indemn/cli 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +28 -0
- package/README.md +29 -23
- package/dist/cli/agents.d.ts.map +1 -1
- package/dist/cli/agents.js +20 -0
- package/dist/cli/agents.js.map +1 -1
- package/dist/cli/eval.d.ts.map +1 -1
- package/dist/cli/eval.js +38 -0
- package/dist/cli/eval.js.map +1 -1
- package/dist/mcp/tools.d.ts.map +1 -1
- package/dist/mcp/tools.js +40 -0
- package/dist/mcp/tools.js.map +1 -1
- package/dist/report/agent-card/cover.d.ts +20 -0
- package/dist/report/agent-card/cover.d.ts.map +1 -0
- package/dist/report/agent-card/cover.js +58 -0
- package/dist/report/agent-card/cover.js.map +1 -0
- package/dist/report/agent-card/document.d.ts +7 -0
- package/dist/report/agent-card/document.d.ts.map +1 -0
- package/dist/report/agent-card/document.js +144 -0
- package/dist/report/agent-card/document.js.map +1 -0
- package/dist/report/agent-card/types.d.ts +77 -0
- package/dist/report/agent-card/types.d.ts.map +1 -0
- package/dist/report/agent-card/types.js +3 -0
- package/dist/report/agent-card/types.js.map +1 -0
- package/dist/report/assets/Indemn_PrimaryLogo_Iris.png +0 -0
- package/dist/report/assets/assets/Indemn_PrimaryLogo_Iris.png +0 -0
- package/dist/report/assets/assets/fonts/Barlow-Bold.ttf +0 -0
- package/dist/report/assets/assets/fonts/Barlow-Medium.ttf +0 -0
- package/dist/report/assets/assets/fonts/Barlow-Regular.ttf +0 -0
- package/dist/report/assets/assets/fonts/Barlow-SemiBold.ttf +0 -0
- package/dist/report/assets/fonts/Barlow-Bold.ttf +0 -0
- package/dist/report/assets/fonts/Barlow-Medium.ttf +0 -0
- package/dist/report/assets/fonts/Barlow-Regular.ttf +0 -0
- package/dist/report/assets/fonts/Barlow-SemiBold.ttf +0 -0
- package/dist/report/eval-report/cover.d.ts +16 -0
- package/dist/report/eval-report/cover.d.ts.map +1 -0
- package/dist/report/eval-report/cover.js +8 -0
- package/dist/report/eval-report/cover.js.map +1 -0
- package/dist/report/eval-report/document.d.ts +19 -0
- package/dist/report/eval-report/document.d.ts.map +1 -0
- package/dist/report/eval-report/document.js +32 -0
- package/dist/report/eval-report/document.js.map +1 -0
- package/dist/report/eval-report/example-page.d.ts +11 -0
- package/dist/report/eval-report/example-page.d.ts.map +1 -0
- package/dist/report/eval-report/example-page.js +18 -0
- package/dist/report/eval-report/example-page.js.map +1 -0
- package/dist/report/eval-report/item-page.d.ts +9 -0
- package/dist/report/eval-report/item-page.d.ts.map +1 -0
- package/dist/report/eval-report/item-page.js +72 -0
- package/dist/report/eval-report/item-page.js.map +1 -0
- package/dist/report/eval-report/matrix-page.d.ts +8 -0
- package/dist/report/eval-report/matrix-page.d.ts.map +1 -0
- package/dist/report/eval-report/matrix-page.js +31 -0
- package/dist/report/eval-report/matrix-page.js.map +1 -0
- package/dist/report/eval-report/types.d.ts +69 -0
- package/dist/report/eval-report/types.d.ts.map +1 -0
- package/dist/report/eval-report/types.js +7 -0
- package/dist/report/eval-report/types.js.map +1 -0
- package/dist/report/eval-report/v2-cover.d.ts +9 -0
- package/dist/report/eval-report/v2-cover.d.ts.map +1 -0
- package/dist/report/eval-report/v2-cover.js +32 -0
- package/dist/report/eval-report/v2-cover.js.map +1 -0
- package/dist/report/export.d.ts +17 -0
- package/dist/report/export.d.ts.map +1 -0
- package/dist/report/export.js +327 -0
- package/dist/report/export.js.map +1 -0
- package/dist/report/generate.d.ts +7 -0
- package/dist/report/generate.d.ts.map +1 -0
- package/dist/report/generate.js +30 -0
- package/dist/report/generate.js.map +1 -0
- package/dist/report/scoring.d.ts +90 -0
- package/dist/report/scoring.d.ts.map +1 -0
- package/dist/report/scoring.js +174 -0
- package/dist/report/scoring.js.map +1 -0
- package/dist/report/styles.d.ts +111 -0
- package/dist/report/styles.d.ts.map +1 -0
- package/dist/report/styles.js +134 -0
- package/dist/report/styles.js.map +1 -0
- package/docs/plans/2026-03-13-cli-exports.md +819 -0
- package/package.json +14 -4
- package/skills/eval-analysis/SKILL.md +178 -0
- package/skills/eval-analysis/references/data-shapes.md +279 -0
- package/src/cli/agents.ts +22 -0
- package/src/cli/eval.ts +42 -0
- package/src/mcp/tools.ts +50 -0
- package/src/report/agent-card/cover.tsx +232 -0
- package/src/report/agent-card/document.tsx +582 -0
- package/src/report/agent-card/types.ts +86 -0
- package/src/report/assets/Indemn_PrimaryLogo_Iris.png +0 -0
- package/src/report/assets/fonts/Barlow-Bold.ttf +0 -0
- package/src/report/assets/fonts/Barlow-Medium.ttf +0 -0
- package/src/report/assets/fonts/Barlow-Regular.ttf +0 -0
- package/src/report/assets/fonts/Barlow-SemiBold.ttf +0 -0
- package/src/report/eval-report/cover.tsx +69 -0
- package/src/report/eval-report/document.tsx +105 -0
- package/src/report/eval-report/example-page.tsx +121 -0
- package/src/report/eval-report/item-page.tsx +211 -0
- package/src/report/eval-report/matrix-page.tsx +124 -0
- package/src/report/eval-report/types.ts +77 -0
- package/src/report/eval-report/v2-cover.tsx +102 -0
- package/src/report/export.ts +394 -0
- package/src/report/generate.tsx +37 -0
- package/src/report/scoring.ts +281 -0
- package/src/report/styles.ts +138 -0
- package/tsconfig.json +2 -1
|
@@ -0,0 +1,819 @@
|
|
|
1
|
+
# CLI Exports Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
4
|
+
|
|
5
|
+
**Goal:** Add three export commands (`eval export`, `eval report`, `agents card`) and an `eval-analysis` skill to `@indemn/cli`, producing markdown dumps and branded PDFs identical to the Copilot Dashboard.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Port React-PDF templates from `indemn-platform-v2/ui/src/components/report/` to run headlessly in Node.js. The CLI fetches data from existing APIs (Copilot Server + Evaluations Service) via the SDK, then renders locally. A new `src/report/` module contains the ported templates, brand assets, and a `generate.ts` driver that produces PDF buffers.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** TypeScript ESM, `@react-pdf/renderer` (headless Node.js PDF generation), `react` (JSX runtime — no DOM), Barlow font family, existing `IndemnClient` SDK.
|
|
10
|
+
|
|
11
|
+
**Design doc:** `operating-system/projects/jarvis-cli-mcp/artifacts/2026-03-13-cli-exports-design.md`
|
|
12
|
+
|
|
13
|
+
**Source templates:** `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/` (1,779 lines across 10 files)
|
|
14
|
+
|
|
15
|
+
**Target repo:** `/Users/home/Repositories/indemn-cli/` (main branch)
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
### Task 1: Add dependencies and configure JSX
|
|
20
|
+
|
|
21
|
+
**Files:**
|
|
22
|
+
- Modify: `/Users/home/Repositories/indemn-cli/package.json`
|
|
23
|
+
- Modify: `/Users/home/Repositories/indemn-cli/tsconfig.json`
|
|
24
|
+
|
|
25
|
+
**Step 1: Install react-pdf and react**
|
|
26
|
+
|
|
27
|
+
Run:
|
|
28
|
+
```bash
|
|
29
|
+
cd /Users/home/Repositories/indemn-cli && npm install @react-pdf/renderer react && npm install -D @types/react
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Note: `react-dom` is NOT needed — react-pdf renders to buffers directly.
|
|
33
|
+
|
|
34
|
+
**Step 2: Add JSX config to tsconfig.json**
|
|
35
|
+
|
|
36
|
+
In `tsconfig.json`, add `"jsx": "react-jsx"` to `compilerOptions`:
|
|
37
|
+
|
|
38
|
+
```json
|
|
39
|
+
{
|
|
40
|
+
"compilerOptions": {
|
|
41
|
+
"target": "ES2022",
|
|
42
|
+
"module": "NodeNext",
|
|
43
|
+
"moduleResolution": "NodeNext",
|
|
44
|
+
"outDir": "dist",
|
|
45
|
+
"rootDir": "src",
|
|
46
|
+
"strict": true,
|
|
47
|
+
"esModuleInterop": true,
|
|
48
|
+
"skipLibCheck": true,
|
|
49
|
+
"forceConsistentCasingInFileNames": true,
|
|
50
|
+
"resolveJsonModule": true,
|
|
51
|
+
"declaration": true,
|
|
52
|
+
"declarationMap": true,
|
|
53
|
+
"sourceMap": true,
|
|
54
|
+
"jsx": "react-jsx"
|
|
55
|
+
},
|
|
56
|
+
"include": ["src/**/*"],
|
|
57
|
+
"exclude": ["node_modules", "dist", "test"]
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
**Step 3: Add postbuild script for asset copying**
|
|
62
|
+
|
|
63
|
+
In `package.json`, add a `postbuild` script:
|
|
64
|
+
|
|
65
|
+
```json
|
|
66
|
+
"scripts": {
|
|
67
|
+
"build": "tsc",
|
|
68
|
+
"postbuild": "cp -r src/report/assets dist/report/assets",
|
|
69
|
+
"dev": "tsx --watch src/cli/index.ts",
|
|
70
|
+
"test": "vitest run",
|
|
71
|
+
"test:watch": "vitest"
|
|
72
|
+
}
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
**Step 4: Verify build still works**
|
|
76
|
+
|
|
77
|
+
Run: `npm run build`
|
|
78
|
+
Expected: Clean compilation (no errors)
|
|
79
|
+
|
|
80
|
+
**Step 5: Commit**
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
git add package.json package-lock.json tsconfig.json
|
|
84
|
+
git commit -m "feat: add react-pdf dependencies and JSX config for PDF exports"
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
### Task 2: Bundle brand assets (fonts + logo)
|
|
90
|
+
|
|
91
|
+
**Files:**
|
|
92
|
+
- Create: `src/report/assets/fonts/Barlow-Regular.ttf`
|
|
93
|
+
- Create: `src/report/assets/fonts/Barlow-Medium.ttf`
|
|
94
|
+
- Create: `src/report/assets/fonts/Barlow-SemiBold.ttf`
|
|
95
|
+
- Create: `src/report/assets/fonts/Barlow-Bold.ttf`
|
|
96
|
+
- Create: `src/report/assets/Indemn_PrimaryLogo_Iris.png`
|
|
97
|
+
|
|
98
|
+
**Step 1: Copy font files from dashboard**
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
cd /Users/home/Repositories/indemn-cli
|
|
102
|
+
mkdir -p src/report/assets/fonts
|
|
103
|
+
cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-Regular.ttf src/report/assets/fonts/
|
|
104
|
+
cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-Medium.ttf src/report/assets/fonts/
|
|
105
|
+
cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-SemiBold.ttf src/report/assets/fonts/
|
|
106
|
+
cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-Bold.ttf src/report/assets/fonts/
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**Step 2: Copy logo**
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
cp /Users/home/Repositories/indemn-platform-v2/ui/public/Indemn_PrimaryLogo_Iris.png src/report/assets/
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
**Step 3: Verify build copies assets**
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
npm run build
|
|
119
|
+
ls dist/report/assets/fonts/
|
|
120
|
+
ls dist/report/assets/Indemn_PrimaryLogo_Iris.png
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Expected: All 4 font files and logo present in `dist/report/assets/`
|
|
124
|
+
|
|
125
|
+
**Step 4: Commit**
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
git add src/report/assets/
|
|
129
|
+
git commit -m "feat: bundle Barlow fonts and Indemn logo for PDF reports"
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
### Task 3: Port styles and scoring utilities
|
|
135
|
+
|
|
136
|
+
**Files:**
|
|
137
|
+
- Create: `src/report/styles.ts`
|
|
138
|
+
- Create: `src/report/scoring.ts`
|
|
139
|
+
|
|
140
|
+
**Step 1: Create styles.ts**
|
|
141
|
+
|
|
142
|
+
Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/styles.ts`. Key changes:
|
|
143
|
+
- Replace `import.meta.env` / `window.location.origin` asset URL logic with local file path resolution using ESM `import.meta.url`
|
|
144
|
+
- Font registration uses absolute file paths to bundled `.ttf` files
|
|
145
|
+
|
|
146
|
+
```typescript
|
|
147
|
+
// src/report/styles.ts
|
|
148
|
+
import { Font, StyleSheet } from '@react-pdf/renderer';
|
|
149
|
+
import { fileURLToPath } from 'url';
|
|
150
|
+
import { dirname, join } from 'path';
|
|
151
|
+
|
|
152
|
+
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
153
|
+
const assetsDir = join(__dirname, 'assets');
|
|
154
|
+
|
|
155
|
+
// Register Barlow font family from bundled files
|
|
156
|
+
Font.register({
|
|
157
|
+
family: 'Barlow',
|
|
158
|
+
fonts: [
|
|
159
|
+
{ src: join(assetsDir, 'fonts', 'Barlow-Regular.ttf'), fontWeight: 400 },
|
|
160
|
+
{ src: join(assetsDir, 'fonts', 'Barlow-Medium.ttf'), fontWeight: 500 },
|
|
161
|
+
{ src: join(assetsDir, 'fonts', 'Barlow-SemiBold.ttf'), fontWeight: 600 },
|
|
162
|
+
{ src: join(assetsDir, 'fonts', 'Barlow-Bold.ttf'), fontWeight: 700 },
|
|
163
|
+
],
|
|
164
|
+
});
|
|
165
|
+
|
|
166
|
+
export const logoPath = join(assetsDir, 'Indemn_PrimaryLogo_Iris.png');
|
|
167
|
+
|
|
168
|
+
// Copy the entire `colors` object and `styles` StyleSheet from the dashboard file verbatim.
|
|
169
|
+
// Source: /Users/home/Repositories/indemn-platform-v2/ui/src/components/report/styles.ts lines 31-147
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
**Step 2: Create scoring.ts**
|
|
173
|
+
|
|
174
|
+
Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/lib/scoring.ts`. Copy these functions verbatim — they are pure arithmetic with no browser dependencies:
|
|
175
|
+
|
|
176
|
+
- `SCORE_THRESHOLDS` constant
|
|
177
|
+
- `ScoreMetrics` interface
|
|
178
|
+
- `ScoreColor` type
|
|
179
|
+
- `getScoreColor()` (lines 192-197)
|
|
180
|
+
- `getScoreColorClasses()` (lines 202-212) — keep hex values, remove Tailwind classes
|
|
181
|
+
- `getScoreHexColor()` (lines 217-219)
|
|
182
|
+
- `getRunScoreMetrics()` (lines 69-107)
|
|
183
|
+
- `getRunCriteriaScore()` (lines 112-115)
|
|
184
|
+
- `getRunRubricScore()` (lines 120-123)
|
|
185
|
+
- `getResultsScoreMetrics()` (lines 139-157)
|
|
186
|
+
- `getAggregatePassRate()` (lines 166-183)
|
|
187
|
+
|
|
188
|
+
Remove all imports from `@/api/hooks/useEvaluations` and `@/components/evaluation/types`. Define the minimal types inline (the function signatures only need `EvaluationRun` and `EvaluationResult` shapes — use the existing types from `src/sdk/types.ts` or define compatible interfaces).
|
|
189
|
+
|
|
190
|
+
**Step 3: Verify build**
|
|
191
|
+
|
|
192
|
+
Run: `npm run build`
|
|
193
|
+
Expected: Clean compilation
|
|
194
|
+
|
|
195
|
+
**Step 4: Commit**
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
git add src/report/styles.ts src/report/scoring.ts
|
|
199
|
+
git commit -m "feat: port brand styles and scoring utilities for PDF reports"
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
---
|
|
203
|
+
|
|
204
|
+
### Task 4: Port agent card templates
|
|
205
|
+
|
|
206
|
+
**Files:**
|
|
207
|
+
- Create: `src/report/agent-card/cover.tsx`
|
|
208
|
+
- Create: `src/report/agent-card/document.tsx`
|
|
209
|
+
- Create: `src/report/agent-card/types.ts`
|
|
210
|
+
|
|
211
|
+
**Step 1: Create types.ts**
|
|
212
|
+
|
|
213
|
+
Define the `AgentCardData`, `AgentCardTool`, `AgentCardKnowledgeBase`, `AgentCardLLMConfig`, `AgentCardEvalRun` interfaces.
|
|
214
|
+
|
|
215
|
+
Copy from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/AgentCardDocument.tsx` lines 9-72. For `LifecycleStage` and `V1Owner`, define inline:
|
|
216
|
+
|
|
217
|
+
```typescript
|
|
218
|
+
export type LifecycleStage = 'prototype' | 'beta' | 'published';
|
|
219
|
+
|
|
220
|
+
export interface V1Owner {
|
|
221
|
+
firstname?: string;
|
|
222
|
+
lastname?: string;
|
|
223
|
+
email?: string;
|
|
224
|
+
}
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
**Step 2: Create cover.tsx**
|
|
228
|
+
|
|
229
|
+
Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/AgentCardCover.tsx` (234 lines). Key changes:
|
|
230
|
+
- Replace `import { assetBaseUrl } from './styles'` → `import { logoPath } from '../styles.js'`
|
|
231
|
+
- Replace `const getLogoUrl = () => \`\${assetBaseUrl}/Indemn_PrimaryLogo_Iris.png\`` → use `logoPath` directly
|
|
232
|
+
- Replace `import { getScoreHexColor } from '@/lib/scoring'` → `import { getScoreHexColor } from '../scoring.js'`
|
|
233
|
+
- Replace `import type { LifecycleStage, V1Owner } from '@/types'` → `import type { LifecycleStage, V1Owner } from './types.js'`
|
|
234
|
+
|
|
235
|
+
**Step 3: Create document.tsx**
|
|
236
|
+
|
|
237
|
+
Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/AgentCardDocument.tsx` (639 lines). This is the largest file — contains 7 page components (SystemPromptPage, ToolsPage, KnowledgeBasesPage, LLMConfigPage, EvaluationHistoryPage, MetadataPage) plus the main `AgentCardDocument`.
|
|
238
|
+
|
|
239
|
+
Key changes:
|
|
240
|
+
- Same import path fixes as cover.tsx
|
|
241
|
+
- Replace `import type { ComponentScores, ComponentScoreDetail } from '@/api/hooks/useEvaluations'` → define inline or import from `./types.js`
|
|
242
|
+
|
|
243
|
+
**Step 4: Verify build**
|
|
244
|
+
|
|
245
|
+
Run: `npm run build`
|
|
246
|
+
Expected: Clean compilation
|
|
247
|
+
|
|
248
|
+
**Step 5: Commit**
|
|
249
|
+
|
|
250
|
+
```bash
|
|
251
|
+
git add src/report/agent-card/
|
|
252
|
+
git commit -m "feat: port agent card PDF templates from dashboard"
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
---
|
|
256
|
+
|
|
257
|
+
### Task 5: Port eval report templates
|
|
258
|
+
|
|
259
|
+
**Files:**
|
|
260
|
+
- Create: `src/report/eval-report/types.ts`
|
|
261
|
+
- Create: `src/report/eval-report/cover.tsx` (V1 cover, 71 lines)
|
|
262
|
+
- Create: `src/report/eval-report/v2-cover.tsx` (V2 cover, 104 lines)
|
|
263
|
+
- Create: `src/report/eval-report/item-page.tsx` (V2 per-item, 212 lines)
|
|
264
|
+
- Create: `src/report/eval-report/matrix-page.tsx` (V1 matrix, 125 lines)
|
|
265
|
+
- Create: `src/report/eval-report/example-page.tsx` (V1 per-question, 122 lines)
|
|
266
|
+
- Create: `src/report/eval-report/document.tsx` (root document, 115 lines)
|
|
267
|
+
|
|
268
|
+
**Step 1: Create types.ts**
|
|
269
|
+
|
|
270
|
+
Define `MatrixData`, `MatrixRow`, `EvaluatorInfo`, `WeightConfig` interfaces needed for V1 reports. Also re-export `EvaluationResult` from SDK types for V2 reports.
|
|
271
|
+
|
|
272
|
+
Source: `/Users/home/Repositories/indemn-platform-v2/ui/src/components/evaluation/types.ts` — extract only the types referenced by the report components.
|
|
273
|
+
|
|
274
|
+
**Step 2: Port V2 components (covers + item page)**
|
|
275
|
+
|
|
276
|
+
These are the primary target — V2 is the current eval system:
|
|
277
|
+
- `v2-cover.tsx` from `EvalReportV2Cover.tsx` (104 lines)
|
|
278
|
+
- `item-page.tsx` from `EvalReportItemPage.tsx` (212 lines)
|
|
279
|
+
|
|
280
|
+
Key import changes:
|
|
281
|
+
- `@/lib/scoring` → `../scoring.js`
|
|
282
|
+
- `@/components/evaluation/types` → `./types.js`
|
|
283
|
+
- `./styles` → `../styles.js`
|
|
284
|
+
|
|
285
|
+
**Step 3: Port V1 components (cover, matrix, example)**
|
|
286
|
+
|
|
287
|
+
- `cover.tsx` from `EvalReportCover.tsx` (71 lines)
|
|
288
|
+
- `matrix-page.tsx` from `EvalReportMatrixPage.tsx` (125 lines)
|
|
289
|
+
- `example-page.tsx` from `EvalReportExamplePage.tsx` (122 lines)
|
|
290
|
+
|
|
291
|
+
Same import path changes.
|
|
292
|
+
|
|
293
|
+
**Step 4: Port document.tsx**
|
|
294
|
+
|
|
295
|
+
From `EvalReportDocument.tsx` (115 lines). This is the root component that selects V1 or V2 format.
|
|
296
|
+
|
|
297
|
+
Import `calculateWeightedScores` from `../scoring.js` (will be ported in Task 3 from `evaluation/utils.ts`).
|
|
298
|
+
|
|
299
|
+
Note: `calculateWeightedScores` lives in `/Users/home/Repositories/indemn-platform-v2/ui/src/components/evaluation/utils.ts`. Port it to `src/report/scoring.ts` alongside the other scoring functions.
|
|
300
|
+
|
|
301
|
+
**Step 5: Verify build**
|
|
302
|
+
|
|
303
|
+
Run: `npm run build`
|
|
304
|
+
Expected: Clean compilation
|
|
305
|
+
|
|
306
|
+
**Step 6: Commit**
|
|
307
|
+
|
|
308
|
+
```bash
|
|
309
|
+
git add src/report/eval-report/
|
|
310
|
+
git commit -m "feat: port eval report PDF templates (V1 matrix + V2 per-item)"
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
---
|
|
314
|
+
|
|
315
|
+
### Task 6: Create generate.ts — PDF rendering driver
|
|
316
|
+
|
|
317
|
+
**Files:**
|
|
318
|
+
- Create: `src/report/generate.ts`
|
|
319
|
+
|
|
320
|
+
**Step 1: Write the rendering driver**
|
|
321
|
+
|
|
322
|
+
This module provides two async functions that take data, render React-PDF components to a buffer, and write to disk:
|
|
323
|
+
|
|
324
|
+
```typescript
|
|
325
|
+
// src/report/generate.ts
|
|
326
|
+
import React from 'react';
|
|
327
|
+
import { renderToBuffer } from '@react-pdf/renderer';
|
|
328
|
+
import * as fs from 'fs';
|
|
329
|
+
import * as path from 'path';
|
|
330
|
+
import { AgentCardDocument } from './agent-card/document.js';
|
|
331
|
+
import { EvalReportDocument } from './eval-report/document.js';
|
|
332
|
+
import type { AgentCardData } from './agent-card/types.js';
|
|
333
|
+
import type { EvalReportData } from './eval-report/document.js';
|
|
334
|
+
|
|
335
|
+
function generateFilename(prefix: string, name: string, date: string, ext: string): string {
|
|
336
|
+
const cleanName = name.replace(/[^a-zA-Z0-9\s]/g, '').replace(/\s+/g, '_');
|
|
337
|
+
const cleanDate = date.replace(/[^a-zA-Z0-9]/g, '_');
|
|
338
|
+
return `${prefix}_${cleanName}_${cleanDate}.${ext}`;
|
|
339
|
+
}
|
|
340
|
+
|
|
341
|
+
export async function generateAgentCardPDF(data: AgentCardData, outputPath?: string): Promise<string> {
|
|
342
|
+
const filename = outputPath || generateFilename('Indemn_Agent_Card', data.name, data.generatedDate, 'pdf');
|
|
343
|
+
const buffer = await renderToBuffer(
|
|
344
|
+
React.createElement(AgentCardDocument, { data })
|
|
345
|
+
);
|
|
346
|
+
const resolvedPath = path.resolve(filename);
|
|
347
|
+
fs.writeFileSync(resolvedPath, buffer);
|
|
348
|
+
return resolvedPath;
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
export async function generateEvalReportPDF(data: EvalReportData, outputPath?: string): Promise<string> {
|
|
352
|
+
const customerName = data.customerName;
|
|
353
|
+
const filename = outputPath || generateFilename('Indemn_Evaluation', customerName, data.reportDate, 'pdf');
|
|
354
|
+
const buffer = await renderToBuffer(
|
|
355
|
+
React.createElement(EvalReportDocument, { data })
|
|
356
|
+
);
|
|
357
|
+
const resolvedPath = path.resolve(filename);
|
|
358
|
+
fs.writeFileSync(resolvedPath, buffer);
|
|
359
|
+
return resolvedPath;
|
|
360
|
+
}
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
**Step 2: Verify build**
|
|
364
|
+
|
|
365
|
+
Run: `npm run build`
|
|
366
|
+
Expected: Clean compilation
|
|
367
|
+
|
|
368
|
+
**Step 3: Smoke test**
|
|
369
|
+
|
|
370
|
+
Write a quick script to verify PDF generation works:
|
|
371
|
+
|
|
372
|
+
```bash
|
|
373
|
+
cd /Users/home/Repositories/indemn-cli
|
|
374
|
+
node -e "
|
|
375
|
+
import('./dist/report/generate.js').then(async ({ generateAgentCardPDF }) => {
|
|
376
|
+
const path = await generateAgentCardPDF({
|
|
377
|
+
id: 'test', name: 'Test Agent', channels: ['WEB'],
|
|
378
|
+
tools: [], knowledge_bases: [], generatedDate: 'Mar 13, 2026',
|
|
379
|
+
});
|
|
380
|
+
console.log('Generated:', path);
|
|
381
|
+
const fs = await import('fs');
|
|
382
|
+
console.log('Size:', fs.statSync(path).size, 'bytes');
|
|
383
|
+
fs.unlinkSync(path);
|
|
384
|
+
});
|
|
385
|
+
"
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
Expected: Generates a PDF file > 0 bytes, no errors.
|
|
389
|
+
|
|
390
|
+
**Step 4: Commit**
|
|
391
|
+
|
|
392
|
+
```bash
|
|
393
|
+
git add src/report/generate.ts
|
|
394
|
+
git commit -m "feat: PDF rendering driver — generates agent card and eval report PDFs"
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
---
|
|
398
|
+
|
|
399
|
+
### Task 7: Implement `indemn eval export` (markdown dump)
|
|
400
|
+
|
|
401
|
+
**Files:**
|
|
402
|
+
- Modify: `src/cli/eval.ts`
|
|
403
|
+
|
|
404
|
+
**Step 1: Add the export subcommand**
|
|
405
|
+
|
|
406
|
+
Add after the existing `eval list` command in `src/cli/eval.ts`:
|
|
407
|
+
|
|
408
|
+
```typescript
|
|
409
|
+
eval_
|
|
410
|
+
.command('export')
|
|
411
|
+
.argument('<run-id>', 'Evaluation run ID')
|
|
412
|
+
.description('Export evaluation results as markdown')
|
|
413
|
+
.option('--output <path>', 'Output file path')
|
|
414
|
+
.action(async (runId: string, options) => {
|
|
415
|
+
try {
|
|
416
|
+
const client = new IndemnClient();
|
|
417
|
+
const evalSdk = new EvalSDK(client);
|
|
418
|
+
const rubricsSdk = new RubricsSDK(client);
|
|
419
|
+
const testSetsSdk = new TestSetsSDK(client);
|
|
420
|
+
const spinner = ora('Fetching evaluation data...').start();
|
|
421
|
+
|
|
422
|
+
// Fetch all data in parallel
|
|
423
|
+
const [run, results] = await Promise.all([
|
|
424
|
+
evalSdk.getRunStatus(runId),
|
|
425
|
+
evalSdk.getRunResults(runId),
|
|
426
|
+
]);
|
|
427
|
+
|
|
428
|
+
// Fetch rubric, test set, bot context based on run data
|
|
429
|
+
const [rubric, testSet, botContext] = await Promise.all([
|
|
430
|
+
run.rubric_id ? rubricsSdk.getRubric(run.rubric_id) : null,
|
|
431
|
+
run.test_set_id ? testSetsSdk.getTestSet(run.test_set_id) : null,
|
|
432
|
+
evalSdk.getBotContext(run.agent_id),
|
|
433
|
+
]);
|
|
434
|
+
|
|
435
|
+
spinner.text = 'Generating markdown...';
|
|
436
|
+
|
|
437
|
+
const markdown = formatEvalMarkdown(run, results, rubric, testSet, botContext);
|
|
438
|
+
const agentName = botContext?.bot_name || run.agent_id;
|
|
439
|
+
const date = new Date().toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' });
|
|
440
|
+
const defaultFilename = `Indemn_Evaluation_${agentName.replace(/[^a-zA-Z0-9\s]/g, '').replace(/\s+/g, '_')}_${date.replace(/[^a-zA-Z0-9]/g, '_')}.md`;
|
|
441
|
+
|
|
442
|
+
const outputPath = path.resolve(options.output || defaultFilename);
|
|
443
|
+
fs.writeFileSync(outputPath, markdown);
|
|
444
|
+
|
|
445
|
+
spinner.stop();
|
|
446
|
+
printSuccess(`Exported to ${outputPath}`);
|
|
447
|
+
} catch (err: any) {
|
|
448
|
+
printError(err.message || String(err));
|
|
449
|
+
process.exit(1);
|
|
450
|
+
}
|
|
451
|
+
});
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
**Step 2: Write the `formatEvalMarkdown` function**
|
|
455
|
+
|
|
456
|
+
This is a pure function that takes the fetched data and returns a markdown string. Follow the format specified in the design doc (Section "Markdown Export Format"):
|
|
457
|
+
|
|
458
|
+
- Header with run ID, date, agent, model, pass rates, component scores
|
|
459
|
+
- Per test case: conversation turns, criteria scores table, rubric scores table
|
|
460
|
+
- Each test case separated by `---`
|
|
461
|
+
|
|
462
|
+
Add `import * as path from 'path'` at the top of `eval.ts` (already has `import * as fs`).
|
|
463
|
+
|
|
464
|
+
**Step 3: Test against real data**
|
|
465
|
+
|
|
466
|
+
```bash
|
|
467
|
+
npm run build
|
|
468
|
+
indemn eval export 2dff7636-296d-4da1-9b5a-145ee610c4b1
|
|
469
|
+
cat Indemn_Evaluation_*.md | head -40
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
Expected: Markdown file with run summary and per-item results.
|
|
473
|
+
|
|
474
|
+
**Step 4: Commit**
|
|
475
|
+
|
|
476
|
+
```bash
|
|
477
|
+
git add src/cli/eval.ts
|
|
478
|
+
git commit -m "feat: indemn eval export — markdown dump of evaluation results"
|
|
479
|
+
```
|
|
480
|
+
|
|
481
|
+
---
|
|
482
|
+
|
|
483
|
+
### Task 8: Implement `indemn eval report` (branded PDF)
|
|
484
|
+
|
|
485
|
+
**Files:**
|
|
486
|
+
- Modify: `src/cli/eval.ts`
|
|
487
|
+
|
|
488
|
+
**Step 1: Add the report subcommand**
|
|
489
|
+
|
|
490
|
+
```typescript
|
|
491
|
+
eval_
|
|
492
|
+
.command('report')
|
|
493
|
+
.argument('<run-id>', 'Evaluation run ID')
|
|
494
|
+
.description('Generate branded evaluation report PDF')
|
|
495
|
+
.option('--output <path>', 'Output file path')
|
|
496
|
+
.action(async (runId: string, options) => {
|
|
497
|
+
try {
|
|
498
|
+
const client = new IndemnClient();
|
|
499
|
+
const evalSdk = new EvalSDK(client);
|
|
500
|
+
const spinner = ora('Fetching evaluation data...').start();
|
|
501
|
+
|
|
502
|
+
const [run, results] = await Promise.all([
|
|
503
|
+
evalSdk.getRunStatus(runId),
|
|
504
|
+
evalSdk.getRunResults(runId),
|
|
505
|
+
]);
|
|
506
|
+
|
|
507
|
+
const botContext = await evalSdk.getBotContext(run.agent_id);
|
|
508
|
+
|
|
509
|
+
spinner.text = 'Generating PDF...';
|
|
510
|
+
|
|
511
|
+
// Detect V1 vs V2 and build report data
|
|
512
|
+
const isV2 = results.length > 0 && results[0].criteria_scores != null;
|
|
513
|
+
const customerName = botContext?.organization_name || botContext?.bot_name || run.agent_id;
|
|
514
|
+
const reportDate = new Date().toLocaleDateString('en-US', { month: 'long', day: 'numeric', year: 'numeric' });
|
|
515
|
+
|
|
516
|
+
let reportData: EvalReportData;
|
|
517
|
+
if (isV2) {
|
|
518
|
+
reportData = { customerName, reportDate, v2Results: results };
|
|
519
|
+
} else {
|
|
520
|
+
// V1: transform results into MatrixData
|
|
521
|
+
// (implement transformToMatrixData in src/report/eval-report/transform.ts)
|
|
522
|
+
reportData = { customerName, reportDate, matrixData: transformToMatrixData(results), weights: defaultWeights };
|
|
523
|
+
}
|
|
524
|
+
|
|
525
|
+
const outputPath = await generateEvalReportPDF(reportData, options.output);
|
|
526
|
+
|
|
527
|
+
spinner.stop();
|
|
528
|
+
printSuccess(`Report saved to ${outputPath}`);
|
|
529
|
+
} catch (err: any) {
|
|
530
|
+
printError(err.message || String(err));
|
|
531
|
+
process.exit(1);
|
|
532
|
+
}
|
|
533
|
+
});
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
Import `generateEvalReportPDF` from `../report/generate.js` and the `EvalReportData` type.
|
|
537
|
+
|
|
538
|
+
**Step 2: If V1 support is needed, create transform.ts**
|
|
539
|
+
|
|
540
|
+
Create `src/report/eval-report/transform.ts` that converts raw V1 API results (with `scores` dict) into `MatrixData` format. If V1 is deferred, throw a clear error: "V1 report format not yet supported. Use eval export for markdown output."
|
|
541
|
+
|
|
542
|
+
**Step 3: Test against real V2 run**
|
|
543
|
+
|
|
544
|
+
```bash
|
|
545
|
+
npm run build
|
|
546
|
+
indemn eval report 2dff7636-296d-4da1-9b5a-145ee610c4b1
|
|
547
|
+
ls -la Indemn_Evaluation_*.pdf
|
|
548
|
+
```
|
|
549
|
+
|
|
550
|
+
Expected: PDF file generated, opens correctly, matches dashboard layout.
|
|
551
|
+
|
|
552
|
+
**Step 4: Commit**
|
|
553
|
+
|
|
554
|
+
```bash
|
|
555
|
+
git add src/cli/eval.ts src/report/
|
|
556
|
+
git commit -m "feat: indemn eval report — branded PDF evaluation report"
|
|
557
|
+
```
|
|
558
|
+
|
|
559
|
+
---
|
|
560
|
+
|
|
561
|
+
### Task 9: Implement `indemn agents card` (agent card PDF)
|
|
562
|
+
|
|
563
|
+
**Files:**
|
|
564
|
+
- Modify: `src/cli/agents.ts`
|
|
565
|
+
|
|
566
|
+
**Step 1: Add the card subcommand**
|
|
567
|
+
|
|
568
|
+
Add to the agents command group in `src/cli/agents.ts`:
|
|
569
|
+
|
|
570
|
+
```typescript
|
|
571
|
+
agents
|
|
572
|
+
.command('card')
|
|
573
|
+
.argument('<agent-id>', 'Agent ID')
|
|
574
|
+
.description('Generate branded agent card PDF')
|
|
575
|
+
.option('--output <path>', 'Output file path')
|
|
576
|
+
.action(async (agentId: string, options) => {
|
|
577
|
+
try {
|
|
578
|
+
const client = new IndemnClient();
|
|
579
|
+
const agentsSdk = new AgentsSDK(client);
|
|
580
|
+
const evalSdk = new EvalSDK(client);
|
|
581
|
+
const spinner = ora('Fetching agent data...').start();
|
|
582
|
+
|
|
583
|
+
// Fetch agent details, config, and functions in parallel
|
|
584
|
+
const [agent, config, functions] = await Promise.all([
|
|
585
|
+
agentsSdk.getAgent(agentId),
|
|
586
|
+
client.get(`/organization/:org_id/ai-studio/bots/${agentId}/configurations`),
|
|
587
|
+
client.get(`/organization/:org_id/ai-studio/bots/${agentId}/functions`),
|
|
588
|
+
]);
|
|
589
|
+
|
|
590
|
+
// Fetch recent eval runs
|
|
591
|
+
let evalRuns: any[] = [];
|
|
592
|
+
try {
|
|
593
|
+
evalRuns = await evalSdk.listRuns(agentId);
|
|
594
|
+
} catch { /* evals may not be available */ }
|
|
595
|
+
|
|
596
|
+
spinner.text = 'Generating PDF...';
|
|
597
|
+
|
|
598
|
+
// Map to AgentCardData
|
|
599
|
+
const cardData = mapToAgentCardData(agent, config, functions, evalRuns);
|
|
600
|
+
const outputPath = await generateAgentCardPDF(cardData, options.output);
|
|
601
|
+
|
|
602
|
+
spinner.stop();
|
|
603
|
+
printSuccess(`Agent card saved to ${outputPath}`);
|
|
604
|
+
} catch (err: any) {
|
|
605
|
+
printError(err.message || String(err));
|
|
606
|
+
process.exit(1);
|
|
607
|
+
}
|
|
608
|
+
});
|
|
609
|
+
```
|
|
610
|
+
|
|
611
|
+
**Step 2: Write `mapToAgentCardData` helper**
|
|
612
|
+
|
|
613
|
+
Map Copilot Server V1 responses to `AgentCardData` interface. Reference the mapping table in the design doc:
|
|
614
|
+
|
|
615
|
+
- `tools`: map function `name` → `tool_name`, function display `name`
|
|
616
|
+
- `knowledge_bases`: from v2 bots response (includes `document_count`)
|
|
617
|
+
- `evaluationSummary`: sum criteria/rubric passed/total across completed runs
|
|
618
|
+
- `componentScores`: from latest completed run's `component_scores`
|
|
619
|
+
- `evaluationRuns`: map runs to `AgentCardEvalRun[]` with computed percentage scores
|
|
620
|
+
- `system_prompt`: from config's `ai_config.system_prompt`
|
|
621
|
+
- `llm_config`: from config's `ai_config` (provider, model, parameters)
|
|
622
|
+
|
|
623
|
+
**Step 3: Test against real agent**
|
|
624
|
+
|
|
625
|
+
```bash
|
|
626
|
+
npm run build
|
|
627
|
+
indemn agents card 69a40cea71552df0c6b680cb
|
|
628
|
+
ls -la Indemn_Agent_Card_*.pdf
|
|
629
|
+
```
|
|
630
|
+
|
|
631
|
+
Expected: Multi-page PDF with cover, system prompt, tools, KBs, LLM config, eval history (if any), metadata.
|
|
632
|
+
|
|
633
|
+
**Step 4: Commit**
|
|
634
|
+
|
|
635
|
+
```bash
|
|
636
|
+
git add src/cli/agents.ts
|
|
637
|
+
git commit -m "feat: indemn agents card — branded agent card PDF"
|
|
638
|
+
```
|
|
639
|
+
|
|
640
|
+
---
|
|
641
|
+
|
|
642
|
+
### Task 10: Add MCP tools for exports
|
|
643
|
+
|
|
644
|
+
**Files:**
|
|
645
|
+
- Modify: `src/mcp/tools.ts`
|
|
646
|
+
|
|
647
|
+
**Step 1: Add three export tools**
|
|
648
|
+
|
|
649
|
+
Add to the `registerTools` function in `src/mcp/tools.ts`:
|
|
650
|
+
|
|
651
|
+
1. `export_eval_markdown` — calls eval export logic, returns `{ file_path, file_name, file_type: 'markdown' }`
|
|
652
|
+
2. `export_eval_report` — calls eval report logic, returns `{ file_path, file_name, file_type: 'pdf' }`
|
|
653
|
+
3. `export_agent_card` — calls agent card logic, returns `{ file_path, file_name, file_type: 'pdf' }`
|
|
654
|
+
|
|
655
|
+
Each tool's input schema needs:
|
|
656
|
+
- `run_id` (string, required) for eval tools
|
|
657
|
+
- `agent_id` (string, required) for agent card
|
|
658
|
+
- `output_path` (string, optional) for all three
|
|
659
|
+
|
|
660
|
+
Extract the rendering logic from the CLI commands into shared functions (in `src/report/generate.ts` or a new `src/report/export.ts`) so both CLI and MCP can call them without duplication.
|
|
661
|
+
|
|
662
|
+
**Step 2: Verify tools register**
|
|
663
|
+
|
|
664
|
+
```bash
|
|
665
|
+
npm run build
|
|
666
|
+
echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | node dist/mcp/server.js 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); tools=[t['name'] for t in d['result']['tools']]; print(f'{len(tools)} tools'); [print(f' {t}') for t in tools if 'export' in t or 'card' in t]"
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
Expected: 58 tools total, including `export_eval_markdown`, `export_eval_report`, `export_agent_card`.
|
|
670
|
+
|
|
671
|
+
**Step 3: Commit**
|
|
672
|
+
|
|
673
|
+
```bash
|
|
674
|
+
git add src/mcp/tools.ts src/report/
|
|
675
|
+
git commit -m "feat: MCP tools for eval export, eval report, and agent card"
|
|
676
|
+
```
|
|
677
|
+
|
|
678
|
+
---
|
|
679
|
+
|
|
680
|
+
### Task 11: Adapt eval-analysis skill for CLI
|
|
681
|
+
|
|
682
|
+
**Files:**
|
|
683
|
+
- Create: `skills/eval-analysis/SKILL.md`
|
|
684
|
+
- Create: `skills/eval-analysis/references/data-shapes.md`
|
|
685
|
+
|
|
686
|
+
**Step 1: Create SKILL.md**
|
|
687
|
+
|
|
688
|
+
Adapt from `/Users/home/Repositories/operating-system/.claude/worktrees/jarvis-cli-mcp/.claude/skills/eval-analysis/SKILL.md`. Key changes:
|
|
689
|
+
|
|
690
|
+
- Frontmatter: `allowed-tools: Bash(indemn *)`
|
|
691
|
+
- Replace all `curl` commands with `indemn` CLI equivalents:
|
|
692
|
+
- `curl .../runs/{run_id}` → `indemn eval status <run-id> --json`
|
|
693
|
+
- `curl .../runs/{run_id}/results` → `indemn eval results <run-id> --json`
|
|
694
|
+
- `curl .../rubrics/{rubric_id}` → `indemn rubric get <rubric-id> --json`
|
|
695
|
+
- `curl .../test-sets/{test_set_id}` → `indemn testset get <testset-id> --json`
|
|
696
|
+
- `curl .../bots/{agent_id}/context` → `indemn eval bot-context <agent-id> --json`
|
|
697
|
+
- `curl -X PUT .../test-sets/{id}` → `indemn testset update <id> --file updated.json`
|
|
698
|
+
- `curl -X PUT .../rubrics/{id}` → `indemn rubric update <id> --file updated.json`
|
|
699
|
+
- Replace `python3 -m json.tool` with `--json` flag
|
|
700
|
+
- Remove OS-specific references (subagents, markdown-pdf skill, project artifacts)
|
|
701
|
+
- Update export section to reference `indemn eval export` and `indemn eval report`
|
|
702
|
+
- Keep the 7-step workflow, failure categories, and divergence analysis exactly as-is
|
|
703
|
+
|
|
704
|
+
**Step 2: Copy data-shapes.md**
|
|
705
|
+
|
|
706
|
+
```bash
|
|
707
|
+
cp /Users/home/Repositories/operating-system/.claude/worktrees/jarvis-cli-mcp/.claude/skills/eval-analysis/references/data-shapes.md skills/eval-analysis/references/data-shapes.md
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
Ship as-is — the API response shapes are identical.
|
|
711
|
+
|
|
712
|
+
**Step 3: Commit**
|
|
713
|
+
|
|
714
|
+
```bash
|
|
715
|
+
git add skills/eval-analysis/
|
|
716
|
+
git commit -m "feat: eval-analysis skill adapted for CLI (replaces curl with indemn commands)"
|
|
717
|
+
```
|
|
718
|
+
|
|
719
|
+
---
|
|
720
|
+
|
|
721
|
+
### Task 12: End-to-end testing and publish
|
|
722
|
+
|
|
723
|
+
**Files:**
|
|
724
|
+
- Modify: `package.json` (version bump)
|
|
725
|
+
|
|
726
|
+
**Step 1: Test all three commands against real dev data**
|
|
727
|
+
|
|
728
|
+
```bash
|
|
729
|
+
npm run build
|
|
730
|
+
|
|
731
|
+
# Eval export (markdown)
|
|
732
|
+
indemn eval export 2dff7636-296d-4da1-9b5a-145ee610c4b1
|
|
733
|
+
cat Indemn_Evaluation_*.md | head -50
|
|
734
|
+
|
|
735
|
+
# Eval report (PDF)
|
|
736
|
+
indemn eval report 2dff7636-296d-4da1-9b5a-145ee610c4b1
|
|
737
|
+
ls -la Indemn_Evaluation_*.pdf
|
|
738
|
+
|
|
739
|
+
# Agent card (PDF)
|
|
740
|
+
indemn agents card 69a40cea71552df0c6b680cb
|
|
741
|
+
ls -la Indemn_Agent_Card_*.pdf
|
|
742
|
+
|
|
743
|
+
# MCP tools
|
|
744
|
+
echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | node dist/mcp/server.js 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['result']['tools']), 'tools')"
|
|
745
|
+
```
|
|
746
|
+
|
|
747
|
+
Expected:
|
|
748
|
+
- Markdown file with full eval data
|
|
749
|
+
- PDF eval report that matches dashboard output
|
|
750
|
+
- PDF agent card with all pages
|
|
751
|
+
- 58 MCP tools registered
|
|
752
|
+
|
|
753
|
+
**Step 2: Open the PDFs and visually verify**
|
|
754
|
+
|
|
755
|
+
```bash
|
|
756
|
+
open Indemn_Evaluation_*.pdf
|
|
757
|
+
open Indemn_Agent_Card_*.pdf
|
|
758
|
+
```
|
|
759
|
+
|
|
760
|
+
Verify: branded cover page, correct data, proper font rendering, all pages present.
|
|
761
|
+
|
|
762
|
+
**Step 3: Bump version**
|
|
763
|
+
|
|
764
|
+
In `package.json`, change `"version": "1.0.0"` to `"version": "1.1.0"`.
|
|
765
|
+
|
|
766
|
+
**Step 4: Clean up test files**
|
|
767
|
+
|
|
768
|
+
```bash
|
|
769
|
+
rm -f Indemn_Evaluation_*.md Indemn_Evaluation_*.pdf Indemn_Agent_Card_*.pdf
|
|
770
|
+
```
|
|
771
|
+
|
|
772
|
+
**Step 5: Commit and push**
|
|
773
|
+
|
|
774
|
+
```bash
|
|
775
|
+
git add -A
|
|
776
|
+
git commit -m "feat: CLI exports — eval markdown, eval report PDF, agent card PDF, eval-analysis skill"
|
|
777
|
+
git push
|
|
778
|
+
```
|
|
779
|
+
|
|
780
|
+
**Step 6: Publish to npm**
|
|
781
|
+
|
|
782
|
+
```bash
|
|
783
|
+
npm publish --access public
|
|
784
|
+
```
|
|
785
|
+
|
|
786
|
+
Note: You'll need npm auth. If using the granular token, set it first:
|
|
787
|
+
```bash
|
|
788
|
+
npm config set //registry.npmjs.org/:_authToken <token>
|
|
789
|
+
```
|
|
790
|
+
|
|
791
|
+
Expected: `@indemn/cli@1.1.0` published.
|
|
792
|
+
|
|
793
|
+
**Step 7: Verify install**
|
|
794
|
+
|
|
795
|
+
```bash
|
|
796
|
+
npm install -g @indemn/cli@1.1.0
|
|
797
|
+
indemn eval export --help
|
|
798
|
+
indemn eval report --help
|
|
799
|
+
indemn agents card --help
|
|
800
|
+
```
|
|
801
|
+
|
|
802
|
+
---
|
|
803
|
+
|
|
804
|
+
## Task Dependency Graph
|
|
805
|
+
|
|
806
|
+
```
|
|
807
|
+
Task 1 (deps + JSX config)
|
|
808
|
+
└→ Task 2 (assets)
|
|
809
|
+
└→ Task 3 (styles + scoring)
|
|
810
|
+
├→ Task 4 (agent card templates)
|
|
811
|
+
└→ Task 5 (eval report templates)
|
|
812
|
+
└→ Task 6 (generate.ts driver)
|
|
813
|
+
├→ Task 7 (eval export CLI)
|
|
814
|
+
├→ Task 8 (eval report CLI)
|
|
815
|
+
├→ Task 9 (agents card CLI)
|
|
816
|
+
└→ Task 10 (MCP tools)
|
|
817
|
+
Task 11 (eval-analysis skill) — independent, can run in parallel with Tasks 4-10
|
|
818
|
+
Task 12 (E2E test + publish) — after all others
|
|
819
|
+
```
|