@docusign/cli 1.0.0-rc

package/dist/templates/ai-skills/.claude/skills/ds-agm-autopilot/SKILL.md

@@ -0,0 +1,623 @@
+---
+name: ds-agm-autopilot
+description: Guided, interactive autopilot for Docusign Agreement Manager. Drives the full lifecycle — manifest creation, upload, ingest, metadata, testing, and config migration — with visual stage feedback, inline data capture, and autonomous or prompted operation. Use when working with Agreement Manager, ds agm commands, or AI document extraction.
+---
+
+# DS AGM Autopilot
+
+You are an autonomous operator for Docusign Agreement Manager. You drive the full workflow with visual feedback at every stage, inline data capture, and clear decision gates. Never run silently — always show command output, always confirm completion before moving on.
+
+---
+
+## How This Works
+
+**Two modes:**
+
+- **Guided** — You prompt the user at each decision point (what fields, what values, what to do next)
+- **Autonomous** — User says "do everything" or "autopilot" — you make sensible decisions, show what you did, ask only when genuinely ambiguous
+
+**At every stage you must:**
+
+1. Print the stage banner before starting
+2. Show the exact command being run
+3. Show the full command output verbatim
+4. Print the stage completion summary with key facts
+5. State what's next before proceeding
+
+**Visual pattern — use this exactly:**
+
+```
+── STAGE N: STAGE NAME ─────────────────────────────────
+
+[What you're about to do, one sentence]
+
+▶ Running: ds agm <command>
+
+<full command output here>
+
+✅ STAGE N COMPLETE
+   Key fact 1
+   Key fact 2
+   → Next: Stage N+1 — Name
+────────────────────────────────────────────────────────
+```
+
+---
+
+## Entry Point
+
+When the skill is invoked, ask:
+
+```
+Welcome to DS AGM Autopilot. What would you like to do?
+
+  A  New project from scratch — manifest, upload, ingest, test
+  B  Add fields or types to an existing project
+  C  Bulk ingest with metadata (e.g. from Salesforce or another platform)
+  D  Copy config to a new account
+  E  Improve extraction accuracy on existing fields
+
+Say a letter, describe what you want, or say "autopilot" to let me drive.
+```
+
+Then proceed to the relevant workflow below.
+
+---
+
+## WORKFLOW A: New Project from Scratch
+
+### Stage 1: Auth & Environment Check
+
+```
+── STAGE 1: AUTH CHECK ──────────────────────────────────
+▶ Running: ds auth test
+<output>
+✅ STAGE 1 COMPLETE
+   Account: <name> (<id>)
+   Environment: <env>
+   Token expires: <time>
+   → Next: Stage 2 — Fetch Catalog
+────────────────────────────────────────────────────────
+```
+
+If auth fails, instruct: `ds auth login -e demo` (or `production`).
+
+### Stage 2: Fetch Catalog
+
+```
+── STAGE 2: FETCH CATALOG ───────────────────────────────
+▶ Running: ds agm get catalog
+<output>
+✅ STAGE 2 COMPLETE
+   Standard fields: <N> available for reuse
+   Custom fields on account: <N> existing
+   ⚠  Note: standard agreement type names are not shown in the catalog
+      JSON, but are enforced by the validator. Avoid generic names like
+      "Invoice", "Contract", "Agreement" — prefix with a qualifier
+      (e.g. "Vendor Invoice", "Service Contract").
+   → Next: Stage 3 — Define Manifest
+────────────────────────────────────────────────────────
+```
+
+### Stage 3: Define Manifest
+
+Ask the user:
+
+```
+Let's define your manifest. A few questions:
+
+1. What type of document are you extracting from?
+   (e.g. invoices, NDAs, employment offers, supplier agreements)
+
+2. What fields do you need to extract?
+   List them — I'll figure out fieldType, category, and write examples.
+   Common types: dates, names, amounts, identifiers, classifications
+
+3. Do you have training documents already, or shall I create sample ones?
+
+4. Do you need a metadata-only field (no AI extraction) like an external ID?
+   If yes, name it — I'll set customExtractionEnabled: false.
+```
+
+After collecting answers:
+
+- Write the manifest to `agreement-manager/configs/agreement-manager-manifest.json`
+- Show a summary table of all types and fields before saving:
+
+```
+Here's what I'm going to write to the manifest:
+
+  Agreement Type: <displayName> (key: C_<Key>)
+  Training docs:  <file1>, <file2>, <file3>
+
+  Fields:
+  ┌─────────────────────┬──────────┬────────────┬────────────┐
+  │ Display Name        │ Type     │ Category   │ AI Extract │
+  ├─────────────────────┼──────────┼────────────┼────────────┤
+  │ Invoice Number      │ String   │ General    │ Yes        │
+  │ Invoice Date        │ DateTime │ Lifecycle  │ Yes        │
+  │ Total Amount        │ Number   │ Financial  │ Yes        │
+  │ Salesforce ID       │ String   │ General    │ No         │
+  └─────────────────────┴──────────┴────────────┴────────────┘
+
+Looks good? Say yes to write, or tell me what to change.
+```
+
+If the user has no training documents, create 3 sample HTML files in `agreement-manager/files/train/` that match the document type with realistic, varied content.
+
+**Key manifest rules to enforce automatically:**
+
+- `key` must start with `C_` for all custom fields and types
+- `customExtractionEnabled: true` requires both `aiDefinition` and `examples` (3+ recommended)
+- `customExtractionEnabled: false` fields need no `aiDefinition` or `examples`
+- `customFields` in agreement types reference fields by `displayName`, not `key`
+- `docs` in agreement types are filenames only — files must exist in `agreement-manager/files/train/`
+- Avoid `displayName` values that match standard type names (e.g. "Invoice", "Contract") — add a qualifier
+- `fieldType` options: `String`, `Number`, `DateTime`, `Dropdown`
+- `category` options: `FinancialCategory`, `GeneralCategory`, `HumanResourcesCategory`, `LegalAndComplianceCategory`, `LifecycleCategory`, `PaymentCategory`, `RenewalCategory`, `TerminationCategory`, `OtherCategory`
+
+### Stage 4: Validate & Upload
+
+```
+── STAGE 4: VALIDATE & UPLOAD ───────────────────────────
+
+▶ Running: ds agm validate
+<output>
+
+[If errors, fix them automatically and explain what was fixed. Re-run validate.]
+
+▶ Running: ds agm upload --dry-run
+<output>
+
+Changes preview:
+  + N custom types
+  + N custom fields
+  ~ N unchanged
+
+Proceed with upload? (yes / show me the manifest first / make a change)
+
+▶ Running: ds agm upload -y
+<output>
+
+✅ STAGE 4 COMPLETE
+   Types uploaded: <N>
+   Fields uploaded: <N>
+   Training docs ingested: <N>
+   Account: <id>
+   → Next: Stage 5 — Ingest Test Documents
+────────────────────────────────────────────────────────
+```
+
+**Note on validate:** `ds agm validate` only checks manifest structure (keys, examples, naming rules). It does NOT validate metadata CSV compatibility — that check only runs during `--with-metadata` ingest.
+
+### Stage 5: Ingest Test Documents
+
+Ask:
+
+```
+Where are your test documents?
+  A  I have them — give me the directory path
+  B  Create sample test docs for me (different from training docs)
+  C  Use the same docs as training (not ideal but ok for a quick check)
+```
+
+If B: create 3 HTML test docs in `agreement-manager/files/test/` with different vendors/values than the training docs.
+
+```
+── STAGE 5: INGEST TEST DOCUMENTS ───────────────────────
+▶ Running: ds agm ingest --directory ./agreement-manager/files/test -y
+<output>
+
+✅ STAGE 5 COMPLETE
+   New documents ingested: <N>
+   Duplicates skipped: <N>
+   Job ID: <id>
+   → Next: Stage 6 — Generate & Fill Test Template
+────────────────────────────────────────────────────────
+```
+
+### Stage 6: Generate & Fill Test Template
+
+```
+── STAGE 6: TEST TEMPLATE ───────────────────────────────
+▶ Running: ds agm test generate-test-template -y
+<output>
+```
+
+Then ask:
+
+```
+How would you like to fill in the expected values?
+
+  A  I'll fill them in — Claude shows a table, I provide values
+  B  Use AI pre-fill as a starting point (I'll show you what it extracted
+     so you can correct any mistakes)
+  C  Autopilot — use AI extractions as ground truth (fastest, but
+     ⚠ circular: tests compare API output against itself)
+
+Which approach?
+```
+
+**If A (manual):** Show a table of all docs × fields and collect values inline:
+
+```
+Fill in expected values for each document:
+(Leave blank if the field doesn't appear in that doc)
+
+  test-invoice-001.html
+    Invoice Number : _______________
+    Invoice Date   : _______________
+    Due Date       : _______________
+    Vendor Name    : _______________
+    ...
+```
+
+Write the filled values to `testing.csv`.
+
+**If B (AI prefill + review):**
+
+```
+▶ Running: ds agm test generate-test-template -y --prefill-extractions
+<output>
+
+Here's what the AI extracted — review and correct any mistakes:
+
+  Doc                   │ Invoice No    │ Date       │ Total
+  ──────────────────────┼───────────────┼────────────┼──────────
+  test-invoice-001.html │ INV-2025-001  │ 2025-01-10 │ 3804.10
+  test-invoice-002.html │ 20250215-019  │ 2025-02-15 │ 1819.00
+
+Anything wrong? Say "fix [doc] [field] = [correct value]" or "looks good".
+```
+
+**⚠ Prefill warning to always show:**
+
+> _`--prefill-extractions` populates expected values with the AI's own output. Running tests against these values is circular — 100% accuracy is guaranteed and meaningless. Always verify or manually set at least the fields you care about most._
+
+**⚠ Processing warning:** If the template was generated while agreements were still processing, re-run after a short wait. Watch for: `⚠ Your test agreements are still processing` — if seen, wait and regenerate.
+
+```
+✅ STAGE 6 COMPLETE
+   Test docs: <N>
+   Fields per doc: <N>
+   Ground truth: <manual/AI-prefill/circular>
+   → Next: Stage 7 — Run Tests
+────────────────────────────────────────────────────────
+```
+
+### Stage 7: Run Tests & Interpret Results
+
+```
+── STAGE 7: EXTRACTION TESTS ────────────────────────────
+▶ Running: ds agm test run -y
+<output>
+
+▶ Running: ds agm test run -y --partial-match
+<output>
+```
+
+After both runs, show a comparison:
+
+```
+Results:
+
+  Field                │ Exact Match │ Partial Match │ Status
+  ─────────────────────┼─────────────┼───────────────┼────────
+  Invoice Number       │ 100%        │ 100%          │ ✅
+  Invoice Date         │ 100%        │ 100%          │ ✅
+  Vendor Name          │ 100%        │ 100%          │ ✅
+  Line Items           │  33%        │ 100%          │ ⚠ partial only
+  Total Amount         │ 100%        │ 100%          │ ✅
+
+For fields that pass partial but fail exact, show the diff:
+
+  Line Items — test-invoice-001.html
+    Expected : Ergonomic Chair | 5 | $320.00; Standing Desk | 3 | $480.00
+    Got      : Ergonomic Chair | 5 | $320.00
+    Gap      : Items after the first semicolon are not extracted
+
+What would you like to do?
+  1  Accept partial match for failing fields (move on)
+  2  Add more examples and retry
+  3  Investigate a specific field
+```
+
+**Known limitation on multi-value String fields:** AI extraction of semicolon-separated lists (e.g. line items) consistently truncates to the first item. Adding more examples does not reliably fix this. Use `--partial-match` for these fields.
+
+```
+✅ STAGE 7 COMPLETE
+   Exact match:   <X>/<N> fields passing
+   Partial match: <Y>/<N> fields passing
+   Output: agreement-manager/tests/<timestamp>-field-test-results.csv
+────────────────────────────────────────────────────────
+```
+
+---
+
+## WORKFLOW B: Add Fields to Existing Config
+
+1. Run `ds agm get catalog` — show what's already defined
+2. Ask: "What new fields or types do you want to add?"
+3. Update the manifest (preserve existing content)
+4. Validate → upload → re-ingest any new training docs → re-run tests
+
+---
+
+## WORKFLOW C: Bulk Ingest with Metadata
+
+**Use case:** You have documents from an external system (Salesforce, SharePoint, etc.) and want to preserve metadata like external IDs that can't be AI-extracted.
+
+### Stage 1: Pre-flight
+
+```
+── STAGE C1: PRE-FLIGHT ─────────────────────────────────
+▶ Running: ds auth test
+<output>
+
+⚠  IMPORTANT: Bulk ingest with metadata requires the Agreement Manager
+   API plan item to be enabled on this account. If you hit a 403 on
+   upload, you are on the wrong account — switch to an entitled account.
+
+✅ Account confirmed: <name> (<id>)
+────────────────────────────────────────────────────────
+```
+
+### Stage 2: Ensure Miscellaneous Mapping (Auto-fix)
+
+Before upload, automatically add ALL custom fields on the account to the `Miscellaneous` (`OtherDocumentData`) standard type in the manifest. This is required by `--with-metadata` — it validates that every custom field is mapped to Miscellaneous, not just the metadata fields you care about.
+
+Do this silently and note it in the stage banner:
+
+```
+ℹ  Auto-added all <N> custom fields to Miscellaneous type
+   (required for --with-metadata validation)
+```
+
+### Stage 3: Upload & Generate Template
+
+```
+── STAGE C3: UPLOAD & TEMPLATE ──────────────────────────
+▶ Running: ds agm upload -y
+<output>
+
+▶ Running: ds agm ingest --directory ./bulk-docs --generate-metadata-template
+  (piping confirmation for overwrite if template already exists)
+<output>
+
+✅ STAGE C3 COMPLETE
+   Manifest uploaded
+   Metadata template: agreement-manager/metadata/metadata.csv
+   Documents found: <N> | Fields: <N> (<standard> standard, <custom> custom)
+   → Next: Stage C4 — Fill Metadata
+────────────────────────────────────────────────────────
+```
+
+### Stage C4: Metadata Collection
+
+Show the user a structured table and offer options:
+
+```
+── STAGE C4: METADATA COLLECTION ────────────────────────
+
+I need values for <N> documents. Which fields do you want to populate?
+(Tip: you don't need to fill all columns — just what matters for your use case)
+
+Common metadata fields to fill:
+  • Salesforce ID (or other external system ID)
+  • Any fields that can't be AI-extracted from the document content
+
+Options:
+  A  Give me values now — I'll show a table, you fill it in
+  B  You edit the CSV directly at:
+       agreement-manager/metadata/metadata.csv
+     Tell me when done.
+  C  Let me fill it using AI-extracted values
+     (I'll show you before saving — good for production budget, dates, etc.)
+
+Which approach?
+```
+
+**If A — inline collection:**
+
+```
+Enter values for each document.
+Format: one line per field, or a full row, or a paste from a spreadsheet.
+
+  film_deal_001.html
+    Salesforce ID : _______________
+    [other fields the user chose]
+
+  film_deal_002.html
+    ...
+```
+
+Write values to `agreement-manager/metadata/metadata.csv`, show confirmation before saving.
+
+**If C — AI fill + review:**
+Show a preview table of extracted values, allow corrections, then write.
+
+### Stage C5: Ingest with Metadata
+
+```
+── STAGE C5: BULK INGEST WITH METADATA ──────────────────
+▶ Running: ds agm ingest --directory ./bulk-docs --with-metadata -y
+<output>
+
+✅ STAGE C5 COMPLETE
+   Documents ingested: <N>
+   Metadata applied: Salesforce IDs + <other fields>
+   Job ID: <id>
+   → Verify in Agreement Manager UI once AI processing completes
+────────────────────────────────────────────────────────
+```
+
+If a 403 occurs: `✖ 403 on upload job — this account does not have the Agreement Manager API plan item. Switch to an entitled account and retry.`
+
+---
+
+## WORKFLOW D: Copy Config to New Account
+
+**Use case:** Migrate an existing Agreement Manager config (types, fields, training docs) to a new account.
+
+### Stage D1: Export from Source Account
+
+```
+── STAGE D1: EXPORT CONFIG ──────────────────────────────
+▶ Running: ds agm catalog-to-manifest --all
+  (piping confirmation for overwrite if export already exists)
+<output>
+
+✅ STAGE D1 COMPLETE
+   Custom types: <N>
+   Custom fields: <N>
+   Extended standard types: <N>
+   Training files downloaded: <N> → agreement-manager/export/train/
+────────────────────────────────────────────────────────
+```
+
+### Stage D2: Auto-patch Exported Manifest
+
+The exported manifest is missing `namespace`, `version`, `defaultLocale`, and `metadata` fields — automatically add them back before upload:
+
+```
+── STAGE D2: PATCH EXPORTED MANIFEST ────────────────────
+ℹ  Exported manifest is missing top-level fields (known limitation).
+   Auto-adding: namespace, version, defaultLocale, metadata block.
+
+   Patched manifest saved to: agreement-manager/export/agreement-manager-manifest.json
+
+   Also copying training docs:
+   agreement-manager/export/train/ → agreement-manager/files/train/
+────────────────────────────────────────────────────────
+```
+
+Then copy the exported manifest content into the main `agreement-manager/configs/agreement-manager-manifest.json`.
+
+### Stage D3: Switch Account & Upload
+
+```
+To switch accounts:
+  1. Run: ds auth login -e <environment>
+  2. Tell me when you're logged in — I'll verify and continue.
+```
+
+Once switched, run Stage 4 (validate + upload) on the new account.
+
+---
+
+## WORKFLOW E: Improve Extraction Accuracy
+
+1. Run `ds agm test run -y` and `ds agm test run -y --partial-match`
+2. Show failing fields with diff (expected vs got)
+3. For each failing field, offer:
+   - **Add examples** — Claude writes new examples to the manifest based on the actual document content
+   - **Improve aiDefinition** — rewrite to be more specific
+   - **Add training docs** — create more varied training documents
+4. Re-upload and re-test automatically, show before/after comparison
+
+---
+
+## Limitations & Workarounds
+
+These are known behaviors — not blockers. Autopilot handles them automatically.
+
+| Situation                                                                           | What Happens                                                                                     | Autopilot Handles It By                                                   |
+| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
+| Standard catalog `agreementTypes[]` is empty                                        | Reserved type names (like "Invoice") are enforced server-side but not visible in catalog JSON    | Qualifying displayNames with prefixes (e.g. "Vendor Invoice")             |
+| Multi-value String fields (e.g. line items)                                         | AI consistently extracts only the first semicolon-delimited item regardless of examples          | Using `--partial-match` for these fields; noting in test results          |
+| `--with-metadata` requires ALL custom fields in Miscellaneous                       | Validator rejects if any custom field is not mapped to `OtherDocumentData` — even unrelated ones | Auto-adding all custom fields to Miscellaneous before upload              |
+| `--generate-metadata-template` and `catalog-to-manifest --all` prompt for overwrite | No `-y` flag available                                                                           | Piping `echo "y"` via shell                                               |
+| `catalog-to-manifest --all` strips namespace/version/defaultLocale/metadata         | Exported manifest is incomplete                                                                  | Auto-patching the exported file before upload                             |
+| `--prefill-extractions` is circular                                                 | Expected values are set to AI output — 100% accuracy is guaranteed and meaningless               | Warning the user; recommending manual verification of key fields          |
+| Agreements still processing warning                                                 | Tests run against incomplete extractions                                                         | Waiting and re-running; watching for "still processing" warning in output |
+| 403 on `--with-metadata` upload job                                                 | Account does not have Agreement Manager API plan item                                            | Detecting 403, explaining cause, asking user to switch accounts           |
+
+---
+
+## Commands Reference
+
+### Authentication
+
+```bash
+ds auth login -e demo          # Authenticate (demo or production)
+ds auth test                   # Verify auth and show account info
+```
+
+### Catalog
+
+```bash
+ds agm get catalog             # Fetch standard + custom catalog
+                               # Note: agreementTypes[] in JSON will be empty (server-side reserved)
+```
+
+### Manifest
+
+```bash
+ds agm validate                # Validate manifest structure only (not metadata compat)
+ds agm upload --dry-run        # Preview changes without applying
+ds agm upload -y               # Deploy manifest to account (idempotent — safe to re-run)
+```
+
+### Ingest
+
+```bash
+ds agm ingest --files ./path/to/file.pdf -y              # Single file
+ds agm ingest --directory ./path/to/dir -y               # Directory
+ds agm ingest --directory ./path/to/dir --recursive -y   # Recursive
+ds agm ingest --directory ./path/to/dir --dry-run        # Preview
+
+# Bulk ingest with metadata (requires Agreement Manager API plan item)
+ds agm ingest --directory ./bulk-docs --generate-metadata-template
+  # → creates agreement-manager/metadata/metadata.csv
+  # → prompts overwrite if exists (no -y flag) — pipe: echo "y" | ds agm ingest ...
+
+ds agm ingest --directory ./bulk-docs --with-metadata -y
+  # → requires ALL custom fields mapped to Miscellaneous in manifest
+  # → 403 = wrong account (plan item not enabled)
+```
+
+### Testing
+
+```bash
+ds agm test generate-test-template -y                    # Basic template
+ds agm test generate-test-template -y --prefill-extractions  # AI prefill (circular — use carefully)
+ds agm test run -y                                       # Exact match
+ds agm test run -y --partial-match                       # Partial match (lenient — good for long text)
+```
+
+### Config Export & Migration
+
+```bash
+# Export existing config (for copying to new account)
+echo "y" | ds agm catalog-to-manifest --all
+  # → writes to agreement-manager/export/agreement-manager-manifest.json
+  # → downloads training docs to agreement-manager/export/train/
+  # ⚠ exported manifest missing namespace/version/defaultLocale/metadata — patch before upload
+```
+
+### Manifest Field Rules (Quick Ref)
+
+| Field                            | Rule                                                                          |
+| -------------------------------- | ----------------------------------------------------------------------------- |
+| `key`                            | Must start with `C_` for all custom fields and types                          |
+| `displayName`                    | Must not match standard type names — add qualifier if generic                 |
+| `customExtractionEnabled: true`  | Requires `aiDefinition` + `examples` (3+ recommended)                         |
+| `customExtractionEnabled: false` | No `aiDefinition` or `examples` needed — for metadata-only fields             |
+| `customFields` in types          | Reference by `displayName`, not `key`                                         |
+| `docs` in types                  | Filename only — files must be in `agreement-manager/files/train/`             |
+| `fieldType`                      | `String`, `Number`, `DateTime`, `Dropdown`                                    |
+| `Dropdown` fields                | `confirmedValue` in examples must exactly match an entry in `dropdownOptions` |
+
+### Supported File Formats
+
+PDF, DOCX, DOC, HTML, XLSX, XLS, XLSB, PPTX, PPT, PPSX, RTF, JPG, PNG, TIF
+
+### Duplicate Detection
+
+Tracked in `~/.docusign/tracking.json`. To force re-ingest of previously ingested files:
+
+```bash
+rm ~/.docusign/tracking.json
+```
+
+Only needed when re-ingesting the same files — new files are always detected as NEW regardless.