@indemn/cli 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. package/.claude-plugin/marketplace.json +28 -0
  2. package/README.md +29 -23
  3. package/dist/cli/agents.d.ts.map +1 -1
  4. package/dist/cli/agents.js +20 -0
  5. package/dist/cli/agents.js.map +1 -1
  6. package/dist/cli/eval.d.ts.map +1 -1
  7. package/dist/cli/eval.js +38 -0
  8. package/dist/cli/eval.js.map +1 -1
  9. package/dist/mcp/tools.d.ts.map +1 -1
  10. package/dist/mcp/tools.js +40 -0
  11. package/dist/mcp/tools.js.map +1 -1
  12. package/dist/report/agent-card/cover.d.ts +20 -0
  13. package/dist/report/agent-card/cover.d.ts.map +1 -0
  14. package/dist/report/agent-card/cover.js +58 -0
  15. package/dist/report/agent-card/cover.js.map +1 -0
  16. package/dist/report/agent-card/document.d.ts +7 -0
  17. package/dist/report/agent-card/document.d.ts.map +1 -0
  18. package/dist/report/agent-card/document.js +144 -0
  19. package/dist/report/agent-card/document.js.map +1 -0
  20. package/dist/report/agent-card/types.d.ts +77 -0
  21. package/dist/report/agent-card/types.d.ts.map +1 -0
  22. package/dist/report/agent-card/types.js +3 -0
  23. package/dist/report/agent-card/types.js.map +1 -0
  24. package/dist/report/assets/Indemn_PrimaryLogo_Iris.png +0 -0
  25. package/dist/report/assets/assets/Indemn_PrimaryLogo_Iris.png +0 -0
  26. package/dist/report/assets/assets/fonts/Barlow-Bold.ttf +0 -0
  27. package/dist/report/assets/assets/fonts/Barlow-Medium.ttf +0 -0
  28. package/dist/report/assets/assets/fonts/Barlow-Regular.ttf +0 -0
  29. package/dist/report/assets/assets/fonts/Barlow-SemiBold.ttf +0 -0
  30. package/dist/report/assets/fonts/Barlow-Bold.ttf +0 -0
  31. package/dist/report/assets/fonts/Barlow-Medium.ttf +0 -0
  32. package/dist/report/assets/fonts/Barlow-Regular.ttf +0 -0
  33. package/dist/report/assets/fonts/Barlow-SemiBold.ttf +0 -0
  34. package/dist/report/eval-report/cover.d.ts +16 -0
  35. package/dist/report/eval-report/cover.d.ts.map +1 -0
  36. package/dist/report/eval-report/cover.js +8 -0
  37. package/dist/report/eval-report/cover.js.map +1 -0
  38. package/dist/report/eval-report/document.d.ts +19 -0
  39. package/dist/report/eval-report/document.d.ts.map +1 -0
  40. package/dist/report/eval-report/document.js +32 -0
  41. package/dist/report/eval-report/document.js.map +1 -0
  42. package/dist/report/eval-report/example-page.d.ts +11 -0
  43. package/dist/report/eval-report/example-page.d.ts.map +1 -0
  44. package/dist/report/eval-report/example-page.js +18 -0
  45. package/dist/report/eval-report/example-page.js.map +1 -0
  46. package/dist/report/eval-report/item-page.d.ts +9 -0
  47. package/dist/report/eval-report/item-page.d.ts.map +1 -0
  48. package/dist/report/eval-report/item-page.js +72 -0
  49. package/dist/report/eval-report/item-page.js.map +1 -0
  50. package/dist/report/eval-report/matrix-page.d.ts +8 -0
  51. package/dist/report/eval-report/matrix-page.d.ts.map +1 -0
  52. package/dist/report/eval-report/matrix-page.js +31 -0
  53. package/dist/report/eval-report/matrix-page.js.map +1 -0
  54. package/dist/report/eval-report/types.d.ts +69 -0
  55. package/dist/report/eval-report/types.d.ts.map +1 -0
  56. package/dist/report/eval-report/types.js +7 -0
  57. package/dist/report/eval-report/types.js.map +1 -0
  58. package/dist/report/eval-report/v2-cover.d.ts +9 -0
  59. package/dist/report/eval-report/v2-cover.d.ts.map +1 -0
  60. package/dist/report/eval-report/v2-cover.js +32 -0
  61. package/dist/report/eval-report/v2-cover.js.map +1 -0
  62. package/dist/report/export.d.ts +17 -0
  63. package/dist/report/export.d.ts.map +1 -0
  64. package/dist/report/export.js +327 -0
  65. package/dist/report/export.js.map +1 -0
  66. package/dist/report/generate.d.ts +7 -0
  67. package/dist/report/generate.d.ts.map +1 -0
  68. package/dist/report/generate.js +30 -0
  69. package/dist/report/generate.js.map +1 -0
  70. package/dist/report/scoring.d.ts +90 -0
  71. package/dist/report/scoring.d.ts.map +1 -0
  72. package/dist/report/scoring.js +174 -0
  73. package/dist/report/scoring.js.map +1 -0
  74. package/dist/report/styles.d.ts +111 -0
  75. package/dist/report/styles.d.ts.map +1 -0
  76. package/dist/report/styles.js +134 -0
  77. package/dist/report/styles.js.map +1 -0
  78. package/docs/plans/2026-03-13-cli-exports.md +819 -0
  79. package/package.json +14 -4
  80. package/skills/eval-analysis/SKILL.md +178 -0
  81. package/skills/eval-analysis/references/data-shapes.md +279 -0
  82. package/src/cli/agents.ts +22 -0
  83. package/src/cli/eval.ts +42 -0
  84. package/src/mcp/tools.ts +50 -0
  85. package/src/report/agent-card/cover.tsx +232 -0
  86. package/src/report/agent-card/document.tsx +582 -0
  87. package/src/report/agent-card/types.ts +86 -0
  88. package/src/report/assets/Indemn_PrimaryLogo_Iris.png +0 -0
  89. package/src/report/assets/fonts/Barlow-Bold.ttf +0 -0
  90. package/src/report/assets/fonts/Barlow-Medium.ttf +0 -0
  91. package/src/report/assets/fonts/Barlow-Regular.ttf +0 -0
  92. package/src/report/assets/fonts/Barlow-SemiBold.ttf +0 -0
  93. package/src/report/eval-report/cover.tsx +69 -0
  94. package/src/report/eval-report/document.tsx +105 -0
  95. package/src/report/eval-report/example-page.tsx +121 -0
  96. package/src/report/eval-report/item-page.tsx +211 -0
  97. package/src/report/eval-report/matrix-page.tsx +124 -0
  98. package/src/report/eval-report/types.ts +77 -0
  99. package/src/report/eval-report/v2-cover.tsx +102 -0
  100. package/src/report/export.ts +394 -0
  101. package/src/report/generate.tsx +37 -0
  102. package/src/report/scoring.ts +281 -0
  103. package/src/report/styles.ts +138 -0
  104. package/tsconfig.json +2 -1
@@ -0,0 +1,819 @@
1
+ # CLI Exports Implementation Plan
2
+
3
+ > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
4
+
5
+ **Goal:** Add three export commands (`eval export`, `eval report`, `agents card`) and an `eval-analysis` skill to `@indemn/cli`, producing markdown dumps and branded PDFs identical to the Copilot Dashboard.
6
+
7
+ **Architecture:** Port React-PDF templates from `indemn-platform-v2/ui/src/components/report/` to run headlessly in Node.js. The CLI fetches data from existing APIs (Copilot Server + Evaluations Service) via the SDK, then renders locally. A new `src/report/` module contains the ported templates, brand assets, and a `generate.ts` driver that produces PDF buffers.
8
+
9
+ **Tech Stack:** TypeScript ESM, `@react-pdf/renderer` (headless Node.js PDF generation), `react` (JSX runtime — no DOM), Barlow font family, existing `IndemnClient` SDK.
10
+
11
+ **Design doc:** `operating-system/projects/jarvis-cli-mcp/artifacts/2026-03-13-cli-exports-design.md`
12
+
13
+ **Source templates:** `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/` (1,779 lines across 10 files)
14
+
15
+ **Target repo:** `/Users/home/Repositories/indemn-cli/` (main branch)
16
+
17
+ ---
18
+
19
+ ### Task 1: Add dependencies and configure JSX
20
+
21
+ **Files:**
22
+ - Modify: `/Users/home/Repositories/indemn-cli/package.json`
23
+ - Modify: `/Users/home/Repositories/indemn-cli/tsconfig.json`
24
+
25
+ **Step 1: Install react-pdf and react**
26
+
27
+ Run:
28
+ ```bash
29
+ cd /Users/home/Repositories/indemn-cli && npm install @react-pdf/renderer react && npm install -D @types/react
30
+ ```
31
+
32
+ Note: `react-dom` is NOT needed — react-pdf renders to buffers directly.
33
+
34
+ **Step 2: Add JSX config to tsconfig.json**
35
+
36
+ In `tsconfig.json`, add `"jsx": "react-jsx"` to `compilerOptions`:
37
+
38
+ ```json
39
+ {
40
+ "compilerOptions": {
41
+ "target": "ES2022",
42
+ "module": "NodeNext",
43
+ "moduleResolution": "NodeNext",
44
+ "outDir": "dist",
45
+ "rootDir": "src",
46
+ "strict": true,
47
+ "esModuleInterop": true,
48
+ "skipLibCheck": true,
49
+ "forceConsistentCasingInFileNames": true,
50
+ "resolveJsonModule": true,
51
+ "declaration": true,
52
+ "declarationMap": true,
53
+ "sourceMap": true,
54
+ "jsx": "react-jsx"
55
+ },
56
+ "include": ["src/**/*"],
57
+ "exclude": ["node_modules", "dist", "test"]
58
+ }
59
+ ```
60
+
61
+ **Step 3: Add postbuild script for asset copying**
62
+
63
+ In `package.json`, add a `postbuild` script:
64
+
65
+ ```json
66
+ "scripts": {
67
+ "build": "tsc",
68
+ "postbuild": "cp -r src/report/assets dist/report/assets",
69
+ "dev": "tsx --watch src/cli/index.ts",
70
+ "test": "vitest run",
71
+ "test:watch": "vitest"
72
+ }
73
+ ```
74
+
75
+ **Step 4: Verify build still works**
76
+
77
+ Run: `npm run build`
78
+ Expected: Clean compilation (no errors)
79
+
80
+ **Step 5: Commit**
81
+
82
+ ```bash
83
+ git add package.json package-lock.json tsconfig.json
84
+ git commit -m "feat: add react-pdf dependencies and JSX config for PDF exports"
85
+ ```
86
+
87
+ ---
88
+
89
+ ### Task 2: Bundle brand assets (fonts + logo)
90
+
91
+ **Files:**
92
+ - Create: `src/report/assets/fonts/Barlow-Regular.ttf`
93
+ - Create: `src/report/assets/fonts/Barlow-Medium.ttf`
94
+ - Create: `src/report/assets/fonts/Barlow-SemiBold.ttf`
95
+ - Create: `src/report/assets/fonts/Barlow-Bold.ttf`
96
+ - Create: `src/report/assets/Indemn_PrimaryLogo_Iris.png`
97
+
98
+ **Step 1: Copy font files from dashboard**
99
+
100
+ ```bash
101
+ cd /Users/home/Repositories/indemn-cli
102
+ mkdir -p src/report/assets/fonts
103
+ cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-Regular.ttf src/report/assets/fonts/
104
+ cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-Medium.ttf src/report/assets/fonts/
105
+ cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-SemiBold.ttf src/report/assets/fonts/
106
+ cp /Users/home/Repositories/indemn-platform-v2/ui/public/fonts/Barlow-Bold.ttf src/report/assets/fonts/
107
+ ```
108
+
109
+ **Step 2: Copy logo**
110
+
111
+ ```bash
112
+ cp /Users/home/Repositories/indemn-platform-v2/ui/public/Indemn_PrimaryLogo_Iris.png src/report/assets/
113
+ ```
114
+
115
+ **Step 3: Verify build copies assets**
116
+
117
+ ```bash
118
+ npm run build
119
+ ls dist/report/assets/fonts/
120
+ ls dist/report/assets/Indemn_PrimaryLogo_Iris.png
121
+ ```
122
+
123
+ Expected: All 4 font files and logo present in `dist/report/assets/`
124
+
125
+ **Step 4: Commit**
126
+
127
+ ```bash
128
+ git add src/report/assets/
129
+ git commit -m "feat: bundle Barlow fonts and Indemn logo for PDF reports"
130
+ ```
131
+
132
+ ---
133
+
134
+ ### Task 3: Port styles and scoring utilities
135
+
136
+ **Files:**
137
+ - Create: `src/report/styles.ts`
138
+ - Create: `src/report/scoring.ts`
139
+
140
+ **Step 1: Create styles.ts**
141
+
142
+ Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/styles.ts`. Key changes:
143
+ - Replace `import.meta.env` / `window.location.origin` asset URL logic with local file path resolution using ESM `import.meta.url`
144
+ - Font registration uses absolute file paths to bundled `.ttf` files
145
+
146
+ ```typescript
147
+ // src/report/styles.ts
148
+ import { Font, StyleSheet } from '@react-pdf/renderer';
149
+ import { fileURLToPath } from 'url';
150
+ import { dirname, join } from 'path';
151
+
152
+ const __dirname = dirname(fileURLToPath(import.meta.url));
153
+ const assetsDir = join(__dirname, 'assets');
154
+
155
+ // Register Barlow font family from bundled files
156
+ Font.register({
157
+ family: 'Barlow',
158
+ fonts: [
159
+ { src: join(assetsDir, 'fonts', 'Barlow-Regular.ttf'), fontWeight: 400 },
160
+ { src: join(assetsDir, 'fonts', 'Barlow-Medium.ttf'), fontWeight: 500 },
161
+ { src: join(assetsDir, 'fonts', 'Barlow-SemiBold.ttf'), fontWeight: 600 },
162
+ { src: join(assetsDir, 'fonts', 'Barlow-Bold.ttf'), fontWeight: 700 },
163
+ ],
164
+ });
165
+
166
+ export const logoPath = join(assetsDir, 'Indemn_PrimaryLogo_Iris.png');
167
+
168
+ // Copy the entire `colors` object and `styles` StyleSheet from the dashboard file verbatim.
169
+ // Source: /Users/home/Repositories/indemn-platform-v2/ui/src/components/report/styles.ts lines 31-147
170
+ ```
171
+
172
+ **Step 2: Create scoring.ts**
173
+
174
+ Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/lib/scoring.ts`. Copy these functions verbatim — they are pure arithmetic with no browser dependencies:
175
+
176
+ - `SCORE_THRESHOLDS` constant
177
+ - `ScoreMetrics` interface
178
+ - `ScoreColor` type
179
+ - `getScoreColor()` (lines 192-197)
180
+ - `getScoreColorClasses()` (lines 202-212) — keep hex values, remove Tailwind classes
181
+ - `getScoreHexColor()` (lines 217-219)
182
+ - `getRunScoreMetrics()` (lines 69-107)
183
+ - `getRunCriteriaScore()` (lines 112-115)
184
+ - `getRunRubricScore()` (lines 120-123)
185
+ - `getResultsScoreMetrics()` (lines 139-157)
186
+ - `getAggregatePassRate()` (lines 166-183)
187
+
188
+ Remove all imports from `@/api/hooks/useEvaluations` and `@/components/evaluation/types`. Define the minimal types inline (the function signatures only need `EvaluationRun` and `EvaluationResult` shapes — use the existing types from `src/sdk/types.ts` or define compatible interfaces).
189
+
190
+ **Step 3: Verify build**
191
+
192
+ Run: `npm run build`
193
+ Expected: Clean compilation
194
+
195
+ **Step 4: Commit**
196
+
197
+ ```bash
198
+ git add src/report/styles.ts src/report/scoring.ts
199
+ git commit -m "feat: port brand styles and scoring utilities for PDF reports"
200
+ ```
201
+
202
+ ---
203
+
204
+ ### Task 4: Port agent card templates
205
+
206
+ **Files:**
207
+ - Create: `src/report/agent-card/cover.tsx`
208
+ - Create: `src/report/agent-card/document.tsx`
209
+ - Create: `src/report/agent-card/types.ts`
210
+
211
+ **Step 1: Create types.ts**
212
+
213
+ Define the `AgentCardData`, `AgentCardTool`, `AgentCardKnowledgeBase`, `AgentCardLLMConfig`, `AgentCardEvalRun` interfaces.
214
+
215
+ Copy from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/AgentCardDocument.tsx` lines 9-72. For `LifecycleStage` and `V1Owner`, define inline:
216
+
217
+ ```typescript
218
+ export type LifecycleStage = 'prototype' | 'beta' | 'published';
219
+
220
+ export interface V1Owner {
221
+ firstname?: string;
222
+ lastname?: string;
223
+ email?: string;
224
+ }
225
+ ```
226
+
227
+ **Step 2: Create cover.tsx**
228
+
229
+ Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/AgentCardCover.tsx` (234 lines). Key changes:
230
+ - Replace `import { assetBaseUrl } from './styles'` → `import { logoPath } from '../styles.js'`
231
+ - Replace `const getLogoUrl = () => \`\${assetBaseUrl}/Indemn_PrimaryLogo_Iris.png\`` → use `logoPath` directly
232
+ - Replace `import { getScoreHexColor } from '@/lib/scoring'` → `import { getScoreHexColor } from '../scoring.js'`
233
+ - Replace `import type { LifecycleStage, V1Owner } from '@/types'` → `import type { LifecycleStage, V1Owner } from './types.js'`
234
+
235
+ **Step 3: Create document.tsx**
236
+
237
+ Port from `/Users/home/Repositories/indemn-platform-v2/ui/src/components/report/AgentCardDocument.tsx` (639 lines). This is the largest file — contains 7 page components (SystemPromptPage, ToolsPage, KnowledgeBasesPage, LLMConfigPage, EvaluationHistoryPage, MetadataPage) plus the main `AgentCardDocument`.
238
+
239
+ Key changes:
240
+ - Same import path fixes as cover.tsx
241
+ - Replace `import type { ComponentScores, ComponentScoreDetail } from '@/api/hooks/useEvaluations'` → define inline or import from `./types.js`
242
+
243
+ **Step 4: Verify build**
244
+
245
+ Run: `npm run build`
246
+ Expected: Clean compilation
247
+
248
+ **Step 5: Commit**
249
+
250
+ ```bash
251
+ git add src/report/agent-card/
252
+ git commit -m "feat: port agent card PDF templates from dashboard"
253
+ ```
254
+
255
+ ---
256
+
257
+ ### Task 5: Port eval report templates
258
+
259
+ **Files:**
260
+ - Create: `src/report/eval-report/types.ts`
261
+ - Create: `src/report/eval-report/cover.tsx` (V1 cover, 71 lines)
262
+ - Create: `src/report/eval-report/v2-cover.tsx` (V2 cover, 104 lines)
263
+ - Create: `src/report/eval-report/item-page.tsx` (V2 per-item, 212 lines)
264
+ - Create: `src/report/eval-report/matrix-page.tsx` (V1 matrix, 125 lines)
265
+ - Create: `src/report/eval-report/example-page.tsx` (V1 per-question, 122 lines)
266
+ - Create: `src/report/eval-report/document.tsx` (root document, 115 lines)
267
+
268
+ **Step 1: Create types.ts**
269
+
270
+ Define `MatrixData`, `MatrixRow`, `EvaluatorInfo`, `WeightConfig` interfaces needed for V1 reports. Also re-export `EvaluationResult` from SDK types for V2 reports.
271
+
272
+ Source: `/Users/home/Repositories/indemn-platform-v2/ui/src/components/evaluation/types.ts` — extract only the types referenced by the report components.
273
+
274
+ **Step 2: Port V2 components (covers + item page)**
275
+
276
+ These are the primary target — V2 is the current eval system:
277
+ - `v2-cover.tsx` from `EvalReportV2Cover.tsx` (104 lines)
278
+ - `item-page.tsx` from `EvalReportItemPage.tsx` (212 lines)
279
+
280
+ Key import changes:
281
+ - `@/lib/scoring` → `../scoring.js`
282
+ - `@/components/evaluation/types` → `./types.js`
283
+ - `./styles` → `../styles.js`
284
+
285
+ **Step 3: Port V1 components (cover, matrix, example)**
286
+
287
+ - `cover.tsx` from `EvalReportCover.tsx` (71 lines)
288
+ - `matrix-page.tsx` from `EvalReportMatrixPage.tsx` (125 lines)
289
+ - `example-page.tsx` from `EvalReportExamplePage.tsx` (122 lines)
290
+
291
+ Same import path changes.
292
+
293
+ **Step 4: Port document.tsx**
294
+
295
+ From `EvalReportDocument.tsx` (115 lines). This is the root component that selects V1 or V2 format.
296
+
297
+ Import `calculateWeightedScores` from `../scoring.js` (will be ported in Task 3 from `evaluation/utils.ts`).
298
+
299
+ Note: `calculateWeightedScores` lives in `/Users/home/Repositories/indemn-platform-v2/ui/src/components/evaluation/utils.ts`. Port it to `src/report/scoring.ts` alongside the other scoring functions.
300
+
301
+ **Step 5: Verify build**
302
+
303
+ Run: `npm run build`
304
+ Expected: Clean compilation
305
+
306
+ **Step 6: Commit**
307
+
308
+ ```bash
309
+ git add src/report/eval-report/
310
+ git commit -m "feat: port eval report PDF templates (V1 matrix + V2 per-item)"
311
+ ```
312
+
313
+ ---
314
+
315
+ ### Task 6: Create generate.ts — PDF rendering driver
316
+
317
+ **Files:**
318
+ - Create: `src/report/generate.ts`
319
+
320
+ **Step 1: Write the rendering driver**
321
+
322
+ This module provides two async functions that take data, render React-PDF components to a buffer, and write to disk:
323
+
324
+ ```typescript
325
+ // src/report/generate.ts
326
+ import React from 'react';
327
+ import { renderToBuffer } from '@react-pdf/renderer';
328
+ import * as fs from 'fs';
329
+ import * as path from 'path';
330
+ import { AgentCardDocument } from './agent-card/document.js';
331
+ import { EvalReportDocument } from './eval-report/document.js';
332
+ import type { AgentCardData } from './agent-card/types.js';
333
+ import type { EvalReportData } from './eval-report/document.js';
334
+
335
+ function generateFilename(prefix: string, name: string, date: string, ext: string): string {
336
+ const cleanName = name.replace(/[^a-zA-Z0-9\s]/g, '').replace(/\s+/g, '_');
337
+ const cleanDate = date.replace(/[^a-zA-Z0-9]/g, '_');
338
+ return `${prefix}_${cleanName}_${cleanDate}.${ext}`;
339
+ }
340
+
341
+ export async function generateAgentCardPDF(data: AgentCardData, outputPath?: string): Promise<string> {
342
+ const filename = outputPath || generateFilename('Indemn_Agent_Card', data.name, data.generatedDate, 'pdf');
343
+ const buffer = await renderToBuffer(
344
+ React.createElement(AgentCardDocument, { data })
345
+ );
346
+ const resolvedPath = path.resolve(filename);
347
+ fs.writeFileSync(resolvedPath, buffer);
348
+ return resolvedPath;
349
+ }
350
+
351
+ export async function generateEvalReportPDF(data: EvalReportData, outputPath?: string): Promise<string> {
352
+ const customerName = data.customerName;
353
+ const filename = outputPath || generateFilename('Indemn_Evaluation', customerName, data.reportDate, 'pdf');
354
+ const buffer = await renderToBuffer(
355
+ React.createElement(EvalReportDocument, { data })
356
+ );
357
+ const resolvedPath = path.resolve(filename);
358
+ fs.writeFileSync(resolvedPath, buffer);
359
+ return resolvedPath;
360
+ }
361
+ ```
362
+
363
+ **Step 2: Verify build**
364
+
365
+ Run: `npm run build`
366
+ Expected: Clean compilation
367
+
368
+ **Step 3: Smoke test**
369
+
370
+ Write a quick script to verify PDF generation works:
371
+
372
+ ```bash
373
+ cd /Users/home/Repositories/indemn-cli
374
+ node -e "
375
+ import('./dist/report/generate.js').then(async ({ generateAgentCardPDF }) => {
376
+ const path = await generateAgentCardPDF({
377
+ id: 'test', name: 'Test Agent', channels: ['WEB'],
378
+ tools: [], knowledge_bases: [], generatedDate: 'Mar 13, 2026',
379
+ });
380
+ console.log('Generated:', path);
381
+ const fs = await import('fs');
382
+ console.log('Size:', fs.statSync(path).size, 'bytes');
383
+ fs.unlinkSync(path);
384
+ });
385
+ "
386
+ ```
387
+
388
+ Expected: Generates a PDF file > 0 bytes, no errors.
389
+
390
+ **Step 4: Commit**
391
+
392
+ ```bash
393
+ git add src/report/generate.ts
394
+ git commit -m "feat: PDF rendering driver — generates agent card and eval report PDFs"
395
+ ```
396
+
397
+ ---
398
+
399
+ ### Task 7: Implement `indemn eval export` (markdown dump)
400
+
401
+ **Files:**
402
+ - Modify: `src/cli/eval.ts`
403
+
404
+ **Step 1: Add the export subcommand**
405
+
406
+ Add after the existing `eval list` command in `src/cli/eval.ts`:
407
+
408
+ ```typescript
409
+ eval_
410
+ .command('export')
411
+ .argument('<run-id>', 'Evaluation run ID')
412
+ .description('Export evaluation results as markdown')
413
+ .option('--output <path>', 'Output file path')
414
+ .action(async (runId: string, options) => {
415
+ try {
416
+ const client = new IndemnClient();
417
+ const evalSdk = new EvalSDK(client);
418
+ const rubricsSdk = new RubricsSDK(client);
419
+ const testSetsSdk = new TestSetsSDK(client);
420
+ const spinner = ora('Fetching evaluation data...').start();
421
+
422
+ // Fetch all data in parallel
423
+ const [run, results] = await Promise.all([
424
+ evalSdk.getRunStatus(runId),
425
+ evalSdk.getRunResults(runId),
426
+ ]);
427
+
428
+ // Fetch rubric, test set, bot context based on run data
429
+ const [rubric, testSet, botContext] = await Promise.all([
430
+ run.rubric_id ? rubricsSdk.getRubric(run.rubric_id) : null,
431
+ run.test_set_id ? testSetsSdk.getTestSet(run.test_set_id) : null,
432
+ evalSdk.getBotContext(run.agent_id),
433
+ ]);
434
+
435
+ spinner.text = 'Generating markdown...';
436
+
437
+ const markdown = formatEvalMarkdown(run, results, rubric, testSet, botContext);
438
+ const agentName = botContext?.bot_name || run.agent_id;
439
+ const date = new Date().toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' });
440
+ const defaultFilename = `Indemn_Evaluation_${agentName.replace(/[^a-zA-Z0-9\s]/g, '').replace(/\s+/g, '_')}_${date.replace(/[^a-zA-Z0-9]/g, '_')}.md`;
441
+
442
+ const outputPath = path.resolve(options.output || defaultFilename);
443
+ fs.writeFileSync(outputPath, markdown);
444
+
445
+ spinner.stop();
446
+ printSuccess(`Exported to ${outputPath}`);
447
+ } catch (err: any) {
448
+ printError(err.message || String(err));
449
+ process.exit(1);
450
+ }
451
+ });
452
+ ```
453
+
454
+ **Step 2: Write the `formatEvalMarkdown` function**
455
+
456
+ This is a pure function that takes the fetched data and returns a markdown string. Follow the format specified in the design doc (Section "Markdown Export Format"):
457
+
458
+ - Header with run ID, date, agent, model, pass rates, component scores
459
+ - Per test case: conversation turns, criteria scores table, rubric scores table
460
+ - Each test case separated by `---`
461
+
462
+ Add `import * as path from 'path'` at the top of `eval.ts` (already has `import * as fs`).
463
+
464
+ **Step 3: Test against real data**
465
+
466
+ ```bash
467
+ npm run build
468
+ indemn eval export 2dff7636-296d-4da1-9b5a-145ee610c4b1
469
+ cat Indemn_Evaluation_*.md | head -40
470
+ ```
471
+
472
+ Expected: Markdown file with run summary and per-item results.
473
+
474
+ **Step 4: Commit**
475
+
476
+ ```bash
477
+ git add src/cli/eval.ts
478
+ git commit -m "feat: indemn eval export — markdown dump of evaluation results"
479
+ ```
480
+
481
+ ---
482
+
483
+ ### Task 8: Implement `indemn eval report` (branded PDF)
484
+
485
+ **Files:**
486
+ - Modify: `src/cli/eval.ts`
487
+
488
+ **Step 1: Add the report subcommand**
489
+
490
+ ```typescript
491
+ eval_
492
+ .command('report')
493
+ .argument('<run-id>', 'Evaluation run ID')
494
+ .description('Generate branded evaluation report PDF')
495
+ .option('--output <path>', 'Output file path')
496
+ .action(async (runId: string, options) => {
497
+ try {
498
+ const client = new IndemnClient();
499
+ const evalSdk = new EvalSDK(client);
500
+ const spinner = ora('Fetching evaluation data...').start();
501
+
502
+ const [run, results] = await Promise.all([
503
+ evalSdk.getRunStatus(runId),
504
+ evalSdk.getRunResults(runId),
505
+ ]);
506
+
507
+ const botContext = await evalSdk.getBotContext(run.agent_id);
508
+
509
+ spinner.text = 'Generating PDF...';
510
+
511
+ // Detect V1 vs V2 and build report data
512
+ const isV2 = results.length > 0 && results[0].criteria_scores != null;
513
+ const customerName = botContext?.organization_name || botContext?.bot_name || run.agent_id;
514
+ const reportDate = new Date().toLocaleDateString('en-US', { month: 'long', day: 'numeric', year: 'numeric' });
515
+
516
+ let reportData: EvalReportData;
517
+ if (isV2) {
518
+ reportData = { customerName, reportDate, v2Results: results };
519
+ } else {
520
+ // V1: transform results into MatrixData
521
+ // (implement transformToMatrixData in src/report/eval-report/transform.ts)
522
+ reportData = { customerName, reportDate, matrixData: transformToMatrixData(results), weights: defaultWeights };
523
+ }
524
+
525
+ const outputPath = await generateEvalReportPDF(reportData, options.output);
526
+
527
+ spinner.stop();
528
+ printSuccess(`Report saved to ${outputPath}`);
529
+ } catch (err: any) {
530
+ printError(err.message || String(err));
531
+ process.exit(1);
532
+ }
533
+ });
534
+ ```
535
+
536
+ Import `generateEvalReportPDF` from `../report/generate.js` and the `EvalReportData` type.
537
+
538
+ **Step 2: If V1 support is needed, create transform.ts**
539
+
540
+ Create `src/report/eval-report/transform.ts` that converts raw V1 API results (with `scores` dict) into `MatrixData` format. If V1 is deferred, throw a clear error: "V1 report format not yet supported. Use eval export for markdown output."
541
+
542
+ **Step 3: Test against real V2 run**
543
+
544
+ ```bash
545
+ npm run build
546
+ indemn eval report 2dff7636-296d-4da1-9b5a-145ee610c4b1
547
+ ls -la Indemn_Evaluation_*.pdf
548
+ ```
549
+
550
+ Expected: PDF file generated, opens correctly, matches dashboard layout.
551
+
552
+ **Step 4: Commit**
553
+
554
+ ```bash
555
+ git add src/cli/eval.ts src/report/
556
+ git commit -m "feat: indemn eval report — branded PDF evaluation report"
557
+ ```
558
+
559
+ ---
560
+
561
+ ### Task 9: Implement `indemn agents card` (agent card PDF)
562
+
563
+ **Files:**
564
+ - Modify: `src/cli/agents.ts`
565
+
566
+ **Step 1: Add the card subcommand**
567
+
568
+ Add to the agents command group in `src/cli/agents.ts`:
569
+
570
+ ```typescript
571
+ agents
572
+ .command('card')
573
+ .argument('<agent-id>', 'Agent ID')
574
+ .description('Generate branded agent card PDF')
575
+ .option('--output <path>', 'Output file path')
576
+ .action(async (agentId: string, options) => {
577
+ try {
578
+ const client = new IndemnClient();
579
+ const agentsSdk = new AgentsSDK(client);
580
+ const evalSdk = new EvalSDK(client);
581
+ const spinner = ora('Fetching agent data...').start();
582
+
583
+ // Fetch agent details, config, and functions in parallel
584
+ const [agent, config, functions] = await Promise.all([
585
+ agentsSdk.getAgent(agentId),
586
+ client.get(`/organization/:org_id/ai-studio/bots/${agentId}/configurations`),
587
+ client.get(`/organization/:org_id/ai-studio/bots/${agentId}/functions`),
588
+ ]);
589
+
590
+ // Fetch recent eval runs
591
+ let evalRuns: any[] = [];
592
+ try {
593
+ evalRuns = await evalSdk.listRuns(agentId);
594
+ } catch { /* evals may not be available */ }
595
+
596
+ spinner.text = 'Generating PDF...';
597
+
598
+ // Map to AgentCardData
599
+ const cardData = mapToAgentCardData(agent, config, functions, evalRuns);
600
+ const outputPath = await generateAgentCardPDF(cardData, options.output);
601
+
602
+ spinner.stop();
603
+ printSuccess(`Agent card saved to ${outputPath}`);
604
+ } catch (err: any) {
605
+ printError(err.message || String(err));
606
+ process.exit(1);
607
+ }
608
+ });
609
+ ```
610
+
611
+ **Step 2: Write `mapToAgentCardData` helper**
612
+
613
+ Map Copilot Server V1 responses to `AgentCardData` interface. Reference the mapping table in the design doc:
614
+
615
+ - `tools`: map function `name` → `tool_name`, function display `name`
616
+ - `knowledge_bases`: from v2 bots response (includes `document_count`)
617
+ - `evaluationSummary`: sum criteria/rubric passed/total across completed runs
618
+ - `componentScores`: from latest completed run's `component_scores`
619
+ - `evaluationRuns`: map runs to `AgentCardEvalRun[]` with computed percentage scores
620
+ - `system_prompt`: from config's `ai_config.system_prompt`
621
+ - `llm_config`: from config's `ai_config` (provider, model, parameters)
622
+
623
+ **Step 3: Test against real agent**
624
+
625
+ ```bash
626
+ npm run build
627
+ indemn agents card 69a40cea71552df0c6b680cb
628
+ ls -la Indemn_Agent_Card_*.pdf
629
+ ```
630
+
631
+ Expected: Multi-page PDF with cover, system prompt, tools, KBs, LLM config, eval history (if any), metadata.
632
+
633
+ **Step 4: Commit**
634
+
635
+ ```bash
636
+ git add src/cli/agents.ts
637
+ git commit -m "feat: indemn agents card — branded agent card PDF"
638
+ ```
639
+
640
+ ---
641
+
642
+ ### Task 10: Add MCP tools for exports
643
+
644
+ **Files:**
645
+ - Modify: `src/mcp/tools.ts`
646
+
647
+ **Step 1: Add three export tools**
648
+
649
+ Add to the `registerTools` function in `src/mcp/tools.ts`:
650
+
651
+ 1. `export_eval_markdown` — calls eval export logic, returns `{ file_path, file_name, file_type: 'markdown' }`
652
+ 2. `export_eval_report` — calls eval report logic, returns `{ file_path, file_name, file_type: 'pdf' }`
653
+ 3. `export_agent_card` — calls agent card logic, returns `{ file_path, file_name, file_type: 'pdf' }`
654
+
655
+ Each tool's input schema needs:
656
+ - `run_id` (string, required) for eval tools
657
+ - `agent_id` (string, required) for agent card
658
+ - `output_path` (string, optional) for all three
659
+
660
+ Extract the rendering logic from the CLI commands into shared functions (in `src/report/generate.ts` or a new `src/report/export.ts`) so both CLI and MCP can call them without duplication.
661
+
662
+ **Step 2: Verify tools register**
663
+
664
+ ```bash
665
+ npm run build
666
+ echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | node dist/mcp/server.js 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); tools=[t['name'] for t in d['result']['tools']]; print(f'{len(tools)} tools'); [print(f' {t}') for t in tools if 'export' in t or 'card' in t]"
667
+ ```
668
+
669
+ Expected: 58 tools total, including `export_eval_markdown`, `export_eval_report`, `export_agent_card`.
670
+
671
+ **Step 3: Commit**
672
+
673
+ ```bash
674
+ git add src/mcp/tools.ts src/report/
675
+ git commit -m "feat: MCP tools for eval export, eval report, and agent card"
676
+ ```
677
+
678
+ ---
679
+
680
+ ### Task 11: Adapt eval-analysis skill for CLI
681
+
682
+ **Files:**
683
+ - Create: `skills/eval-analysis/SKILL.md`
684
+ - Create: `skills/eval-analysis/references/data-shapes.md`
685
+
686
+ **Step 1: Create SKILL.md**
687
+
688
+ Adapt from `/Users/home/Repositories/operating-system/.claude/worktrees/jarvis-cli-mcp/.claude/skills/eval-analysis/SKILL.md`. Key changes:
689
+
690
+ - Frontmatter: `allowed-tools: Bash(indemn *)`
691
+ - Replace all `curl` commands with `indemn` CLI equivalents:
692
+ - `curl .../runs/{run_id}` → `indemn eval status <run-id> --json`
693
+ - `curl .../runs/{run_id}/results` → `indemn eval results <run-id> --json`
694
+ - `curl .../rubrics/{rubric_id}` → `indemn rubric get <rubric-id> --json`
695
+ - `curl .../test-sets/{test_set_id}` → `indemn testset get <testset-id> --json`
696
+ - `curl .../bots/{agent_id}/context` → `indemn eval bot-context <agent-id> --json`
697
+ - `curl -X PUT .../test-sets/{id}` → `indemn testset update <id> --file updated.json`
698
+ - `curl -X PUT .../rubrics/{id}` → `indemn rubric update <id> --file updated.json`
699
+ - Replace `python3 -m json.tool` with `--json` flag
700
+ - Remove OS-specific references (subagents, markdown-pdf skill, project artifacts)
701
+ - Update export section to reference `indemn eval export` and `indemn eval report`
702
+ - Keep the 7-step workflow, failure categories, and divergence analysis exactly as-is
703
+
704
+ **Step 2: Copy data-shapes.md**
705
+
706
+ ```bash
707
+ cp /Users/home/Repositories/operating-system/.claude/worktrees/jarvis-cli-mcp/.claude/skills/eval-analysis/references/data-shapes.md skills/eval-analysis/references/data-shapes.md
708
+ ```
709
+
710
+ Ship as-is — the API response shapes are identical.
711
+
712
+ **Step 3: Commit**
713
+
714
+ ```bash
715
+ git add skills/eval-analysis/
716
+ git commit -m "feat: eval-analysis skill adapted for CLI (replaces curl with indemn commands)"
717
+ ```
718
+
719
+ ---
720
+
721
+ ### Task 12: End-to-end testing and publish
722
+
723
+ **Files:**
724
+ - Modify: `package.json` (version bump)
725
+
726
+ **Step 1: Test all three commands against real dev data**
727
+
728
+ ```bash
729
+ npm run build
730
+
731
+ # Eval export (markdown)
732
+ indemn eval export 2dff7636-296d-4da1-9b5a-145ee610c4b1
733
+ cat Indemn_Evaluation_*.md | head -50
734
+
735
+ # Eval report (PDF)
736
+ indemn eval report 2dff7636-296d-4da1-9b5a-145ee610c4b1
737
+ ls -la Indemn_Evaluation_*.pdf
738
+
739
+ # Agent card (PDF)
740
+ indemn agents card 69a40cea71552df0c6b680cb
741
+ ls -la Indemn_Agent_Card_*.pdf
742
+
743
+ # MCP tools
744
+ echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | node dist/mcp/server.js 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['result']['tools']), 'tools')"
745
+ ```
746
+
747
+ Expected:
748
+ - Markdown file with full eval data
749
+ - PDF eval report that matches dashboard output
750
+ - PDF agent card with all pages
751
+ - 58 MCP tools registered
752
+
753
+ **Step 2: Open the PDFs and visually verify**
754
+
755
+ ```bash
756
+ open Indemn_Evaluation_*.pdf
757
+ open Indemn_Agent_Card_*.pdf
758
+ ```
759
+
760
+ Verify: branded cover page, correct data, proper font rendering, all pages present.
761
+
762
+ **Step 3: Bump version**
763
+
764
+ In `package.json`, change `"version": "1.0.0"` to `"version": "1.1.0"`.
765
+
766
+ **Step 4: Clean up test files**
767
+
768
+ ```bash
769
+ rm -f Indemn_Evaluation_*.md Indemn_Evaluation_*.pdf Indemn_Agent_Card_*.pdf
770
+ ```
771
+
772
+ **Step 5: Commit and push**
773
+
774
+ ```bash
775
+ git add -A
776
+ git commit -m "feat: CLI exports — eval markdown, eval report PDF, agent card PDF, eval-analysis skill"
777
+ git push
778
+ ```
779
+
780
+ **Step 6: Publish to npm**
781
+
782
+ ```bash
783
+ npm publish --access public
784
+ ```
785
+
786
+ Note: You'll need npm auth. If using the granular token, set it first:
787
+ ```bash
788
+ npm config set //registry.npmjs.org/:_authToken <token>
789
+ ```
790
+
791
+ Expected: `@indemn/cli@1.1.0` published.
792
+
793
+ **Step 7: Verify install**
794
+
795
+ ```bash
796
+ npm install -g @indemn/cli@1.1.0
797
+ indemn eval export --help
798
+ indemn eval report --help
799
+ indemn agents card --help
800
+ ```
801
+
802
+ ---
803
+
804
+ ## Task Dependency Graph
805
+
806
+ ```
807
+ Task 1 (deps + JSX config)
808
+ └→ Task 2 (assets)
809
+ └→ Task 3 (styles + scoring)
810
+ ├→ Task 4 (agent card templates)
811
+ └→ Task 5 (eval report templates)
812
+ └→ Task 6 (generate.ts driver)
813
+ ├→ Task 7 (eval export CLI)
814
+ ├→ Task 8 (eval report CLI)
815
+ ├→ Task 9 (agents card CLI)
816
+ └→ Task 10 (MCP tools)
817
+ Task 11 (eval-analysis skill) — independent, can run in parallel with Tasks 4-10
818
+ Task 12 (E2E test + publish) — after all others
819
+ ```