@installagent/sdk 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. package/dist/agent-templates.d.ts +36 -2
  2. package/dist/agent-templates.js +173 -3
  3. package/dist/client-linear-create.d.ts +26 -0
  4. package/dist/client-linear-create.js +46 -0
  5. package/dist/client-linear.d.ts +28 -9
  6. package/dist/client-linear.js +36 -14
  7. package/dist/docs-search.d.ts +19 -6
  8. package/dist/docs-search.js +32 -11
  9. package/dist/doctor.d.ts +9 -0
  10. package/dist/doctor.js +126 -23
  11. package/dist/env-fill.d.ts +13 -0
  12. package/dist/env-fill.js +34 -9
  13. package/dist/env-push.d.ts +3 -0
  14. package/dist/env-push.js +11 -6
  15. package/dist/env-registry.d.ts +25 -0
  16. package/dist/env-registry.js +69 -7
  17. package/dist/exec.js +15 -2
  18. package/dist/index.d.ts +14 -6
  19. package/dist/index.js +14 -6
  20. package/dist/linear-project.d.ts +2 -2
  21. package/dist/linear-project.js +2 -2
  22. package/dist/log.d.ts +28 -0
  23. package/dist/log.js +73 -0
  24. package/dist/n8n-executions.d.ts +85 -0
  25. package/dist/n8n-executions.js +259 -0
  26. package/dist/op-cli.d.ts +29 -0
  27. package/dist/op-cli.js +52 -0
  28. package/dist/platform.d.ts +35 -0
  29. package/dist/platform.js +39 -0
  30. package/dist/portal-admin.d.ts +40 -0
  31. package/dist/portal-admin.js +30 -0
  32. package/dist/standup.d.ts +88 -0
  33. package/dist/standup.js +235 -0
  34. package/dist/sync-repos.d.ts +47 -0
  35. package/dist/sync-repos.js +80 -0
  36. package/dist/template-sync.d.ts +10 -0
  37. package/dist/template-sync.js +11 -0
  38. package/dist/wizard.d.ts +15 -0
  39. package/dist/wizard.js +35 -56
  40. package/dist/workspace-docs.d.ts +44 -0
  41. package/dist/workspace-docs.js +73 -0
  42. package/package.json +2 -2
  43. package/templates/env.example +33 -23
  44. package/templates/workspace-root/AGENT.md +7 -5
  45. package/templates/workspace-root/ONBOARDING.md +65 -8
  46. package/templates/workspace-root/skills/.github/workflows/pr-to-slack.yml +102 -0
  47. package/templates/workspace-root/skills/CLAUDE.md +24 -6
  48. package/templates/workspace-root/skills/README.md +156 -0
  49. package/templates/workspace-root/skills/backtest-acceptance-gates/SKILL.md +293 -0
  50. package/templates/workspace-root/skills/backtest-acceptance-gates/backtest-acceptance-gates-guide.pdf +0 -0
  51. package/templates/workspace-root/skills/backtest-acceptance-gates/references/evidence-record-template.md +128 -0
  52. package/templates/workspace-root/skills/backtest-acceptance-gates/references/harness-checklist.md +90 -0
  53. package/templates/workspace-root/skills/client-latest/SKILL.md +301 -0
  54. package/templates/workspace-root/skills/client-latest/catchup.command.md +17 -0
  55. package/templates/workspace-root/skills/cloud-devops/SKILL.md +235 -0
  56. package/templates/workspace-root/skills/email-triage-rollout/SKILL.md +181 -0
  57. package/templates/workspace-root/skills/gcloud-cli-setup/SKILL.md +100 -0
  58. package/templates/workspace-root/skills/google-chat-app-setup/SKILL.md +136 -0
  59. package/templates/workspace-root/skills/google-chat-app-setup/google-chat-app-setup-guide.pdf +0 -0
  60. package/templates/workspace-root/skills/house-voice/SKILL.md +80 -4
  61. package/templates/workspace-root/skills/ia-cli/SKILL.md +19 -6
  62. package/templates/workspace-root/skills/installagent-brand-kit/SKILL.md +59 -10
  63. package/templates/workspace-root/skills/installagent-brand-kit/assets/installagent-deck-template.pptx +0 -0
  64. package/templates/workspace-root/skills/installagent-brand-kit/references/deck-guide.md +181 -0
  65. package/templates/workspace-root/skills/installagent-brand-kit/scripts/deck.py +652 -0
  66. package/templates/workspace-root/skills/installagent-brand-kit/scripts/deck_lint.py +201 -0
  67. package/templates/workspace-root/skills/installagent-brand-kit/scripts/logo_raster.py +102 -0
  68. package/templates/workspace-root/skills/installagent-brand-kit/scripts/make_deck_template.py +290 -0
  69. package/templates/workspace-root/skills/salesforce-api-snapshot/SKILL.md +379 -0
  70. package/templates/workspace-root/skills/salesforce-api-snapshot/references/schema-discovery-soql.md +153 -0
  71. package/templates/workspace-root/skills/salesforce-api-snapshot/references/snapshot-storage.md +122 -0
  72. package/templates/workspace-root/skills/salesforce-api-snapshot/salesforce-api-snapshot-guide.pdf +0 -0
  73. package/templates/workspace-root/skills/slack-voice/SKILL.md +158 -0
  74. package/templates/workspace-root/skills/sso-gated-client-app/SKILL.md +337 -0
  75. package/templates/workspace-root/skills/sso-gated-client-app/sso-gated-client-app-guide.pdf +0 -0
  76. package/templates/workspace-root/skills/tech-call-prep/SKILL.md +146 -47
  77. package/templates/workspace-root/skills/vendor-feed-ingestion/SKILL.md +323 -0
  78. package/templates/workspace-root/skills/vendor-feed-ingestion/references/delivery-contract-template.md +110 -0
  79. package/templates/workspace-root/skills/vendor-feed-ingestion/references/feed-guards.md +134 -0
  80. package/templates/workspace-root/skills/vendor-feed-ingestion/vendor-feed-ingestion-guide.pdf +0 -0
@@ -0,0 +1,102 @@
1
+ name: PR → Slack (approve / close)
2
+
3
+ # Posts every new PR to Slack with "Merge" and "Close" buttons.
4
+ # This workflow only POSTS (it is read-only). The button clicks are handled by
5
+ # the n8n workflow "GitHub PR Actions (Slack)" at
6
+ # https://n8n.installagent.com/webhook/slack-pr-action , which performs the
7
+ # actual merge/close using the "Github - Install Agent Bot" credential. That
8
+ # receiver derives owner/repo/PR number by regex over the PR URL in the message
9
+ # blocks, so it is repo-agnostic — keep the PR URL in a section block below.
10
+ #
11
+ # Required secret:
12
+ # SLACK_PR_WEBHOOK_URL — a Slack Incoming Webhook URL bound to the review
13
+ # channel (#notify-prs). Set once as an ORGANIZATION
14
+ # secret with visibility "all repositories", so every
15
+ # current and future repo is covered with no per-repo
16
+ # setup. A repo-level secret of the same name also
17
+ # works and takes precedence.
18
+ # The webhook's Slack app must have Interactivity
19
+ # enabled and pointed at the n8n receiver URL above.
20
+ #
21
+ # If the secret is absent the job logs a notice and exits green, so this file is
22
+ # safe to land in a repo before the secret exists.
23
+
24
+ on:
25
+ pull_request:
26
+ types: [opened, reopened, ready_for_review]
27
+
28
+ permissions:
29
+ contents: read
30
+ pull-requests: read
31
+
32
+ jobs:
33
+ notify:
34
+ # Drafts are not ready to review, and skipping them also stops the
35
+ # duplicate post a draft would otherwise make on opened + ready_for_review.
36
+ if: github.event.pull_request.draft == false
37
+ runs-on: ubuntu-latest
38
+ steps:
39
+ - name: Post PR to Slack with action buttons
40
+ env:
41
+ SLACK_WEBHOOK_URL: ${{ secrets.SLACK_PR_WEBHOOK_URL }}
42
+ PR_REPO: ${{ github.event.repository.name }}
43
+ PR_NUMBER: ${{ github.event.pull_request.number }}
44
+ PR_TITLE: ${{ github.event.pull_request.title }}
45
+ PR_URL: ${{ github.event.pull_request.html_url }}
46
+ PR_AUTHOR: ${{ github.event.pull_request.user.login }}
47
+ PR_BASE: ${{ github.event.pull_request.base.ref }}
48
+ PR_HEAD: ${{ github.event.pull_request.head.ref }}
49
+ PR_ADDITIONS: ${{ github.event.pull_request.additions }}
50
+ PR_DELETIONS: ${{ github.event.pull_request.deletions }}
51
+ PR_CHANGED: ${{ github.event.pull_request.changed_files }}
52
+ run: |
53
+ set -euo pipefail
54
+ if [ -z "${SLACK_WEBHOOK_URL:-}" ]; then
55
+ echo "::notice::SLACK_PR_WEBHOOK_URL is not set (org or repo secret) — skipping the Slack post."
56
+ exit 0
57
+ fi
58
+ # Build the Block Kit payload with jq so PR titles/branches that
59
+ # contain quotes or other JSON-breaking characters are escaped safely.
60
+ payload=$(jq -n \
61
+ --arg repo "$PR_REPO" \
62
+ --arg num "$PR_NUMBER" \
63
+ --arg title "$PR_TITLE" \
64
+ --arg url "$PR_URL" \
65
+ --arg author "$PR_AUTHOR" \
66
+ --arg base "$PR_BASE" \
67
+ --arg head "$PR_HEAD" \
68
+ --arg stats "+$PR_ADDITIONS / -$PR_DELETIONS · $PR_CHANGED files" \
69
+ '{
70
+ text: ("New PR in " + $repo + " #" + $num + ": " + $title),
71
+ blocks: [
72
+ { type: "header",
73
+ text: { type: "plain_text", text: ("🔀 " + $repo + " · PR #" + $num) } },
74
+ { type: "section",
75
+ text: { type: "mrkdwn",
76
+ text: ( "*<" + $url + "|" + $title + ">*\n"
77
+ + "by `" + $author + "` • `" + $head + "` → `" + $base + "`\n"
78
+ + $stats ) } },
79
+ { type: "actions",
80
+ block_id: ("pr_" + $num),
81
+ elements: [
82
+ { type: "button", style: "primary", action_id: "pr_merge",
83
+ text: { type: "plain_text", text: "✅ Merge" }, value: $num,
84
+ confirm: {
85
+ title: { type: "plain_text", text: "Merge this PR?" },
86
+ text: { type: "mrkdwn", text: ("Squash-merge *" + $repo + "* PR #" + $num + " into `" + $base + "` and delete `" + $head + "`.") },
87
+ confirm: { type: "plain_text", text: "Merge" },
88
+ deny: { type: "plain_text", text: "Cancel" } } },
89
+ { type: "button", style: "danger", action_id: "pr_close",
90
+ text: { type: "plain_text", text: "🗑️ Close" }, value: $num,
91
+ confirm: {
92
+ title: { type: "plain_text", text: "Close this PR?" },
93
+ text: { type: "mrkdwn", text: ("Close *" + $repo + "* PR #" + $num + " without merging.") },
94
+ confirm: { type: "plain_text", text: "Close" },
95
+ deny: { type: "plain_text", text: "Cancel" } } }
96
+ ] }
97
+ ]
98
+ }')
99
+ # --fail-with-body so a rejected payload or a dead webhook turns the
100
+ # job red instead of passing silently on Slack's 4xx body.
101
+ curl -sS --fail-with-body -X POST -H 'Content-type: application/json' \
102
+ --data "$payload" "$SLACK_WEBHOOK_URL"
@@ -13,18 +13,27 @@ so Claude Code / Cowork auto-register them in the skill menu.
13
13
 
14
14
  ```
15
15
  skills/
16
+ ├── client-latest/ # resume-the-build brief for one client (state + risks/drift, then a next-step CTA)
16
17
  ├── client-deliverables/ # ORCHESTRATOR — the full client engagement package, checklist + sequencing
17
18
  ├── engagement-spec/ # discovery call → overview + tech-spec + onboarding docs
18
- ├── tech-call-prep/ # pre-call prep check + per-client call sheet (docs/05-call-sheet.md)
19
+ ├── tech-call-prep/ # pre-call prep check + per-client call agenda (docs/05-call-agenda.md + Drive copy)
19
20
  ├── client-llm-setup/ # the client's own Claude workspace → branded leave-behind guide
20
21
  ├── client-facing-docs/ # the branded client PDFs (exec overview, proposal/SOW, onboarding)
21
22
  ├── client-workflow-diagram/ # spec → Excalidraw stage-column diagram
22
- ├── installagent-brand-kit/ # our brand for print/PDF/Excalidraw (logo + brand red #9b0506)
23
+ ├── google-chat-app-setup/ # deploy a Google Chat app (n8n-backed) in a client's Workspace, domain-wide
24
+ ├── salesforce-api-snapshot/ # read a client's Salesforce/managed-package org → schema spike + cached JSON snapshot
25
+ ├── sso-gated-client-app/ # a client-facing internal app behind their Google Workspace SSO (Next.js + Vercel)
26
+ ├── vendor-feed-ingestion/ # recurring vendor feeds (SFTP/email → S3 → n8n) + the guards that refuse a bad file
27
+ ├── email-triage-rollout/ # AI inbox triage into a live mailbox: staged soak → label → organize → full + the gate per stage
28
+ ├── backtest-acceptance-gates/ # prove an automation works before the client trusts it (ground truth, evidence, gates)
29
+ ├── installagent-brand-kit/ # our brand for print/PDF/Excalidraw/decks (logo + brand red #d11a1a)
23
30
  ├── design-system/ # the Install Agent Design System (screens/UI)
24
31
  ├── house-voice/ # house voice for client-facing writing, pulled from installagent.com/get-started
25
32
  ├── contract-tech-agnostic/ # contracts/SOWs by FUNCTION and OUTCOME, never tech (+ SOW template in references/)
26
33
  ├── linear/ # Linear issue workflow for client build work (status discipline, sub-issue trees, PR linking)
27
- └── ia-cli/ # route internal ops through the `ia` CLI (team-tools) — decision map + guardrails
34
+ ├── ia-cli/ # route internal ops through the `ia` CLI (team-tools) — decision map + guardrails
35
+ ├── gcloud-cli-setup/ # one named gcloud configuration per account/client + check-before-you-run discipline
36
+ └── slack-voice/ # INTERNAL Slack: terse bullets, hard numbers, no adjectives (client Slack → house-voice)
28
37
  ```
29
38
 
30
39
  Each is `skills/<name>/SKILL.md`, with bulky tables in `references/`.
@@ -36,20 +45,29 @@ Each is `skills/<name>/SKILL.md`, with bulky tables in `references/`.
36
45
  | Kicking off / assembling a full client deliverable package | `client-deliverables` |
37
46
  | Writing up a discovery/scoping call into build docs | `engagement-spec` |
38
47
  | Prepping for an upcoming client call (sheet + readiness check) | `tech-call-prep` |
48
+ | Catching up on a client / resuming a build / checking for drift | `client-latest` |
39
49
  | Setting up a client's own Claude workspace / its leave-behind guide | `client-llm-setup` |
40
50
  | Producing the branded client PDFs | `client-facing-docs` |
41
51
  | Drawing a workflow/architecture diagram for a client | `client-workflow-diagram` |
42
- | Branding a print/PDF/Excalidraw deliverable | `installagent-brand-kit` |
52
+ | Rolling an email-triage agent into a client's live mailbox | `email-triage-rollout` |
53
+ | Deploying a Google Chat app (n8n-backed) in a client's Workspace | `google-chat-app-setup` |
54
+ | Reading a client's Salesforce (or any vendor CRM/EHR) into something we build | `salesforce-api-snapshot` |
55
+ | Building a screen the client's staff log into (dashboard, map, worklist) | `sso-gated-client-app` |
56
+ | Setting up a recurring vendor data feed, or debugging one that went wrong | `vendor-feed-ingestion` |
57
+ | Proving accuracy / hitting an acceptance gate / writing up a validation run | `backtest-acceptance-gates` |
58
+ | Branding a print/PDF/Excalidraw deliverable, or building a client deck | `installagent-brand-kit` |
43
59
  | Building or reviewing any UI / web page / funnel / component | `design-system` |
44
60
  | Writing or reviewing any client-facing copy (proposal, email, guide, web/ad copy) | `house-voice` |
61
+ | Writing any Slack message to a teammate or internal channel | `slack-voice` |
45
62
  | Drafting/reviewing any contract, SOW, MSA, or proposal | `contract-tech-agnostic` |
46
63
  | Working an `INS-xxx` issue / a client's Linear project | `linear` |
47
64
  | Any task an `ia` command covers (client context, intake, agent bank, secrets, setup/doctor) | `ia-cli` |
65
+ | Setting up or switching gcloud CLI accounts/configurations | `gcloud-cli-setup` |
48
66
 
49
67
  `design-system` is the **screen** layer (dark, signal-red, Hanken Grotesk); it
50
68
  mirrors the code source of truth `team-web/packages/ui`.
51
- `installagent-brand-kit` is the **print/deliverable** layer for PDFs and
52
- Excalidraw on white. They share one brand and must agree. `house-voice` is
69
+ `installagent-brand-kit` is the **print/deliverable** layer for PDFs,
70
+ Excalidraw and decks on white. They share one brand and must agree. `house-voice` is
53
71
  the **words** layer that applies across both — what client-facing sentences
54
72
  sound like, regardless of surface.
55
73
 
@@ -0,0 +1,156 @@
1
+ # Install Agent skills
2
+
3
+ Twenty-two reusable, cross-repo how-tos that any coding agent in the workspace
4
+ loads when a task matches the skill's `description`. They encode the parts of
5
+ our delivery process that are easy to get wrong and expensive to rediscover —
6
+ the SOW clause we always forget, the Excalidraw JSON that has to be valid, the
7
+ four ways an SSO-gated app can look fine while being wide open.
8
+
9
+ This directory is its own git repo ([`InstallAgent/skills`](https://github.com/InstallAgent/skills)),
10
+ mounted at the workspace root because a single client engagement touches docs,
11
+ workflows, and diagrams across more than one repo. Repo-local skills live in
12
+ `team-clients/skills/` and `team-infra/skills/` instead.
13
+
14
+ > Working here as an agent? Read [`CLAUDE.md`](./CLAUDE.md) — it's the rule
15
+ > set. This README is the map.
16
+
17
+ ## How these get loaded
18
+
19
+ Every skill is `<name>/SKILL.md` with YAML frontmatter (`name`, `description`).
20
+ 21 of them are symlinked into [`../.claude/skills/`](../.claude/skills/), so Claude
21
+ Code and Cowork register them automatically — they appear in the skill menu and
22
+ fire on their own without being invoked by name. The `description` is the
23
+ trigger; the body is only read once the skill fires.
24
+
25
+ Nothing has to be run or installed. Adding a skill means adding a directory and
26
+ a symlink (see [Adding a skill](#adding-a-skill)).
27
+
28
+ ## The engagement lifecycle
29
+
30
+ Most of these skills are stops along one path — a lead becomes a signed
31
+ engagement becomes a working build. In rough order:
32
+
33
+ | Stage | Skill | What it produces |
34
+ |---|---|---|
35
+ | 0. Landing cold / resuming | [`client-latest`](./client-latest/) | A terminal brief — where the build stands, then risks + surface drift — then a four-option next-step prompt |
36
+ | 1. Before the call | [`tech-call-prep`](./tech-call-prep/) | Readiness check + `docs/05-call-agenda.md` and a Drive copy |
37
+ | 2. After the call | [`engagement-spec`](./engagement-spec/) | Client overview, internal tech spec, onboarding + access checklists, `onboarding.html` |
38
+ | 3. Package it | [`client-deliverables`](./client-deliverables/) | **Orchestrator** — sequences 4, 5, and the diagram into the full engagement set |
39
+ | ↳ | [`client-facing-docs`](./client-facing-docs/) | The three branded PDFs: exec overview, proposal/SOW, onboarding checklist |
40
+ | ↳ | [`contract-tech-agnostic`](./contract-tech-agnostic/) | SOW/MSA language by function and outcome, never by tool |
41
+ | ↳ | [`client-workflow-diagram`](./client-workflow-diagram/) | Stage-column `.excalidraw` + exported image |
42
+ | 4. Week 1 | [`client-llm-setup`](./client-llm-setup/) | The client's own Claude workspace + branded leave-behind guide |
43
+ | 5. Build | the integration skills below | The automation itself |
44
+ | 6. Track | [`linear`](./linear/) | `INS-xxx` status discipline, sub-issue trees, PR linking |
45
+ | 7. Prove it | [`backtest-acceptance-gates`](./backtest-acceptance-gates/) | Ground truth, evidence record, a defensible accuracy number |
46
+
47
+ ### Integration / build skills
48
+
49
+ Reach for these when the build touches a specific system. They're independent
50
+ of each other, except that a Salesforce read usually feeds one of the other two.
51
+
52
+ | Skill | Use when |
53
+ |---|---|
54
+ | [`vendor-feed-ingestion`](./vendor-feed-ingestion/) | A recurring vendor file lands (SFTP/email → S3 → n8n), or one went wrong |
55
+ | [`email-triage-rollout`](./email-triage-rollout/) | An AI triage agent goes into a client's live mailbox — the staged soak → label → organize → full program and its gates |
56
+ | [`salesforce-api-snapshot`](./salesforce-api-snapshot/) | Reading a client's Salesforce or managed-package org (Lumary, Vlocity, nCino) |
57
+ | [`sso-gated-client-app`](./sso-gated-client-app/) | Building a screen the client's staff log into — dashboard, map, worklist |
58
+ | [`google-chat-app-setup`](./google-chat-app-setup/) | Deploying an n8n-backed Chat app across a client's Workspace domain |
59
+
60
+ ### The three house-style layers
61
+
62
+ These are always-on constraints rather than lifecycle stages. They share one
63
+ brand and must agree with each other.
64
+
65
+ | Layer | Skill | Governs |
66
+ |---|---|---|
67
+ | **Screens** | [`design-system`](./design-system/) | Any UI, page, funnel, component. Dark, signal red, Hanken Grotesk. Mirrors `team-web/packages/ui`. |
68
+ | **Print** | [`installagent-brand-kit`](./installagent-brand-kit/) | PDFs, Excalidraw, decks — on white, brand red `#d11a1a`. Ships `scripts/brand.py`, `deck.py`, `deck_lint.py`. |
69
+ | **Words** | [`house-voice`](./house-voice/) | Every client-facing sentence, on any surface. Pulled verbatim from installagent.com. |
70
+
71
+ ### Internal ops
72
+
73
+ [`ia-cli`](./ia-cli/) — the decision map from task to `ia` command (client
74
+ context, intake, agent bank, secrets, `wizard`/`doctor`). Check it before
75
+ writing any custom script against GitHub, Airtable, Linear, Drive, n8n, or the
76
+ portal.
77
+
78
+ ## How they compose
79
+
80
+ Arrows point from a skill to the ones it defers to. `client-deliverables` is
81
+ the only true orchestrator; the rest cross-reference for a specific slice.
82
+
83
+ ```
84
+ client-deliverables ──┬─→ engagement-spec ──┬─→ client-workflow-diagram
85
+ │ ├─→ contract-tech-agnostic
86
+ │ └─→ ia-cli
87
+ ├─→ client-facing-docs
88
+ ├─→ contract-tech-agnostic
89
+ ├─→ client-workflow-diagram
90
+ └─→ installagent-brand-kit
91
+
92
+ tech-call-prep ───────→ engagement-spec
93
+ client-llm-setup ─────→ client-facing-docs, installagent-brand-kit
94
+ salesforce-api-snapshot → sso-gated-client-app, vendor-feed-ingestion
95
+ email-triage-rollout ─→ agents/ bank sop.md (per-provider), linear, house-voice
96
+ house-voice ──────────→ (applies across all client-facing output)
97
+ design-system ────────→ installagent-brand-kit (shared brand source)
98
+ ```
99
+
100
+ Also cross-referenced from here but living elsewhere: `business-research`
101
+ (team-clients) and `n8n-workflow` + `automation-design` (team-infra).
102
+
103
+ ## What's in each directory
104
+
105
+ `SKILL.md` is the whole skill; `references/` holds tables and templates too
106
+ bulky for the body; `scripts/` and `assets/` are only in the brand kit. A few
107
+ skills also ship a rendered PDF guide for handing to a technician.
108
+
109
+ | Skill | SKILL.md | Also ships |
110
+ |---|---:|---|
111
+ | `backtest-acceptance-gates` | 293 | `references/` evidence-record template, harness checklist · PDF guide |
112
+ | `client-deliverables` | 131 | `references/package-layout.md` |
113
+ | `client-latest` | 301 | `catchup.command.md` (the `/catchup` command) |
114
+ | `cloud-devops` | 235 | — · **orphan — adopt or delete (pending decision)**, not symlinked |
115
+ | `client-facing-docs` | 131 | `references/doc-structures.md` |
116
+ | `client-llm-setup` | 97 | `references/guide-template.md` |
117
+ | `client-workflow-diagram` | 141 | `references/excalidraw-format.md` |
118
+ | `contract-tech-agnostic` | 99 | `references/Install_Agent_SOW_Template.docx` |
119
+ | `design-system` | 118 | `references/tokens.md` |
120
+ | `email-triage-rollout` | 181 | — |
121
+ | `engagement-spec` | 272 | `references/spec-templates.md` |
122
+ | `gcloud-cli-setup` | 100 | — |
123
+ | `google-chat-app-setup` | 136 | PDF guide |
124
+ | `house-voice` | 228 | `references/source-copy.md` |
125
+ | `ia-cli` | 87 | — |
126
+ | `installagent-brand-kit` | 161 | 4 `references/`, 5 `scripts/` (brand, deck, deck_lint, logo_raster, make_deck_template), logo SVGs + deck template |
127
+ | `linear` | 186 | — |
128
+ | `salesforce-api-snapshot` | 379 | `references/` schema-discovery SOQL, snapshot storage · PDF guide |
129
+ | `slack-voice` | 158 | — |
130
+ | `sso-gated-client-app` | 337 | PDF guide |
131
+ | `tech-call-prep` | 240 | — |
132
+ | `vendor-feed-ingestion` | 323 | `references/` delivery contract, feed guards · PDF guide |
133
+
134
+ ## Conventions
135
+
136
+ Full rules in [`CLAUDE.md`](./CLAUDE.md). The short version:
137
+
138
+ - One skill per directory, `SKILL.md` under ~500 lines — push long tables into
139
+ `references/`.
140
+ - The `description` carries every "when to use" cue, stated concretely. That's
141
+ what makes the skill fire; the body never gets read otherwise.
142
+ - Nothing sensitive: no secrets, live credentials, internal hostnames, account
143
+ ids, or PII. Use placeholders and point at internal files.
144
+ - If a skill conflicts with [`../AGENT.md`](../AGENT.md), **AGENT.md wins** —
145
+ fix the skill in the same change.
146
+
147
+ ## Adding a skill
148
+
149
+ ```bash
150
+ mkdir -p skills/<name>/references
151
+ $EDITOR skills/<name>/SKILL.md # frontmatter: name, description
152
+ cd .claude/skills && ln -s ../../skills/<name> <name>
153
+ ```
154
+
155
+ Then add a row to the right table above and to the tree in `CLAUDE.md`. Skills
156
+ are how-tos — workspace-wide *rules* belong in `AGENT.md` instead.
@@ -0,0 +1,293 @@
1
+ ---
2
+ name: backtest-acceptance-gates
3
+ description: >-
4
+ Prove an automation actually works before a client is asked to trust it —
5
+ and before we invoice against an acceptance gate. Covers reading the gate
6
+ definition off the signed SOW rather than from memory, pinning down which
7
+ metric the gate means when several plausible ones disagree by 18 points,
8
+ reconstructing ground truth from a client's own historical behaviour
9
+ (including the bimodal-signal problem, no-look-ahead scoring, and
10
+ right-censoring), excluding contaminated events legitimately instead of
11
+ conveniently, segmenting to tell a data gap from a logic gap, discovering
12
+ that ground truth itself is only 79% complete, the evidence-record
13
+ discipline that keeps a months-long build honest about which of its own
14
+ numbers are void, and why a green test suite can mask the exact bug it was
15
+ written to catch. Use this WHENEVER an engagement has an accuracy or
16
+ acceptance criterion — "backtest it", "how accurate is it", "prove it works",
17
+ "we need to hit the gate", "run it against history", "the client wants
18
+ evidence", "is it ready for go-live" — or when an existing accuracy number
19
+ needs re-checking.
20
+ ---
21
+
22
+ # Backtesting & acceptance gates
23
+
24
+ Practice from the CarVision pricing agent
25
+ (`clients/client-carvision/agent-pricing/` + that repo's `evidence/`), where an
26
+ SOW acceptance gate had real money attached and the accuracy number was
27
+ re-derived, contradicted, retracted and rebuilt over two months.
28
+
29
+ **What this skill is really about: staying honest at speed.** Almost every
30
+ mistake below produced a *number*, and the number looked fine. The discipline
31
+ isn't producing measurements — it's knowing which of your measurements are
32
+ still true.
33
+
34
+ ---
35
+
36
+ ## Part 1 — Get the gate from the contract, and pin the metric
37
+
38
+ **Read the acceptance criterion off the signed SOW.** Not the proposal draft,
39
+ not the kickoff deck, not memory. On CarVision the team measured against 85%
40
+ for weeks; the signed SOW said **80%**, on a review sample agreed by both
41
+ parties. Every measured number was unaffected — only the bar they were compared
42
+ against, and therefore every "we're short" judgement, was wrong.
43
+
44
+ **Then pin the metric, because several plausible readings will disagree
45
+ materially.** The same 2,004 decisions scored:
46
+
47
+ | Reading | Result |
48
+ |---|---|
49
+ | Direction match (raise/hold/drop agrees) | 62.2% |
50
+ | Price within tolerance | 56.1% |
51
+ | **Direction OR price** (the lenient reading) | **74.5%** |
52
+
53
+ Eighteen points between the loosest and tightest reading of "accurate." Write
54
+ the chosen definition into the tech spec in the SOW's own words, note which
55
+ readings you're *not* using, and report all of them in the evidence record.
56
+ If you only ever quote the flattering one, someone will eventually ask, and
57
+ that conversation is much worse than the honest table.
58
+
59
+ **Also check the arithmetic isn't being inflated by trivia.** An early run
60
+ showed 82.7% — inflated by 1,046 sub-$1 events that satisfied the price band
61
+ automatically. Excluding them gave 74.5%. Any metric with a tolerance band
62
+ will be gamed by near-zero cases; find them before someone else does.
63
+
64
+ ---
65
+
66
+ ## Part 2 — Reconstruct ground truth
67
+
68
+ Ground truth is what the client's expert actually did, recovered from history.
69
+ Four problems, all of which showed up in one dataset.
70
+
71
+ **The signal is bimodal, and the split is the whole exercise.** The client
72
+ expert's repricing log contained:
73
+
74
+ | Signal | n | What it actually means |
75
+ |---|---|---|
76
+ | ±$1–2 "touch" flips | 1,093 | *"I reviewed this and held."* A **hold** decision. |
77
+ | ≥$100 moves | 2,004 | Genuine repricing decisions. |
78
+ | $3–99 middle | ~16 | Ambiguous — skipped and declared |
79
+
80
+ Reading all 3,113 as "price changes" would have been catastrophic. The $1
81
+ nudges are a review marker in a system with no review button. **Ask the client
82
+ what their idiosyncratic markers mean before interpreting their log** — this
83
+ one only surfaced because someone asked why so many changes were a dollar.
84
+
85
+ **Score at the pre-decision state, with no look-ahead.** For each decision,
86
+ feed the agent the last snapshot *before* the decision — the stale price, that
87
+ row's own last-change date — with every input filtered to `created ≤ asOf`.
88
+ Any input that includes post-decision information turns the backtest into a
89
+ demonstration that the future predicts the past.
90
+
91
+ **Handle censoring explicitly.** Outcome layers ("did the car sell?") are
92
+ right-censored: a unit still listed on the last day has no outcome, not a
93
+ negative one. Say how you treated it.
94
+
95
+ **Absence is not an event.** In a feed where a sold unit simply stops
96
+ appearing, "disappeared" has to be *defined* as an outcome, and the definition
97
+ belongs in the evidence record.
98
+
99
+ ---
100
+
101
+ ## Part 3 — Excluding contaminated events: legitimately, not conveniently
102
+
103
+ Real history contains events that aren't the behaviour you're modelling. On
104
+ CarVision, **246 of 495 raises (49.7%) were a single bulk test** — one
105
+ operator setting a batch of cars to the same price on one day at one location,
106
+ reverted after it failed. Not pricing strategy. Excluding it moved the metric
107
+ from 63.8% to 72.1% and roughly doubled the raise arm's accuracy.
108
+
109
+ That is a big move in our favour, which is exactly why the bar is high:
110
+
111
+ 1. **Name the event.** What happened, when, who did it, and how you know — the
112
+ client confirming it is the strongest form.
113
+ 2. **Encode the exclusion as a dated constant in the harness**, not a hand-
114
+ filtered dataset. `TEST_EVENT_DATE` in the build script, reviewable in the
115
+ diff.
116
+ 3. **Show the control arm is unchanged.** The drop side didn't move at all when
117
+ the raise-side contamination was removed. An exclusion that improves
118
+ everything is suspicious; one that moves only the arm it should is credible.
119
+ 4. **Write it up as its own evidence record**, with the before and after.
120
+ 5. **Report the number both ways** the first time it's used in front of the
121
+ client.
122
+
123
+ If you can't do all five, you're not excluding contamination, you're tuning the
124
+ sample.
125
+
126
+ ---
127
+
128
+ ## Part 4 — Segment, to tell a data gap from a logic gap
129
+
130
+ An aggregate number tells you whether you passed. Segmentation tells you what
131
+ to do next.
132
+
133
+ | Segment | n | GATE |
134
+ |---|---|---|
135
+ | DROP decisions | 1,509 | **94.1%** |
136
+ | RAISE decisions | 495 | **14.7%** |
137
+ | Location A (multi-month lead history) | 825 | **84.7%** |
138
+ | Location B (one month of leads) | 1,179 | **67.3%** |
139
+
140
+ Two readings that changed the plan:
141
+
142
+ - **The raise arm didn't fail on logic.** A raise requires a 7-day lead count
143
+ above a threshold; the lead export was a single late-period snapshot, so
144
+ earlier 7-day windows were structurally empty. The demand signal that
145
+ justifies a raise was *invisible*, not wrong. No amount of tuning fixes that.
146
+ - **Same code, different data density, 17 points apart.** That comparison —
147
+ identical logic across two segments — is the cleanest possible evidence that
148
+ the lever is data completeness, not algorithm quality.
149
+
150
+ **Always segment by the arm/decision type and by data density.** Those two cuts
151
+ have repeatedly turned "the model is bad" into "the feed is thin", which is a
152
+ completely different (and usually cheaper) project.
153
+
154
+ And when you report a count, **report what's inside it**. Roughly half of one
155
+ build's "drops" were units already at a cost floor — no cut was possible. A raw
156
+ drop count quoted without that qualifier overstates the finding every time.
157
+
158
+ ---
159
+
160
+ ## Part 5 — Ground truth can itself be wrong
161
+
162
+ The hardest lesson here, and the one worth carrying into every engagement.
163
+
164
+ An early investigation concluded the ingestion was faithful and the counts were
165
+ correct. It was wrong — **it audited the feed against itself, with no external
166
+ reference.** A later reconciliation against the client's own CRM report found
167
+ the feed carried **79%** of the real records, and that gap had been quietly
168
+ distorting every accuracy figure computed on top of it.
169
+
170
+ Consequences, all of which are the normal cost of finding this late:
171
+
172
+ - Every gate figure computed before the reconciliation became **provisional**,
173
+ and the evidence index had to say so at the top rather than in a footnote.
174
+ - A published headline number was marked **void** — not deleted. Deleting it
175
+ would make the record dishonest; superseding it in place makes it useful.
176
+ - One evidence record's conclusion was struck through and replaced, with the
177
+ part that *was* right (a hypothesis it raised) kept and re-labelled.
178
+
179
+ **So: reconcile your inputs against an independent source before you trust any
180
+ metric built on them.** One export from a second path is enough. Do it early —
181
+ it is much cheaper than retracting a number the client has already seen.
182
+
183
+ ---
184
+
185
+ ## Part 6 — A green test suite can hide the bug it was written to catch
186
+
187
+ A 144-assertion suite ran green while a data-joining layer contributed **zero**
188
+ records to the calculation for three days. The parity test *applied the missing
189
+ conversion itself* before comparing — so it compared two correct things and
190
+ passed, while production compared two incompatible ones.
191
+
192
+ Worse than inert: one field name happened to match across both shapes, so a
193
+ "history exists" flag was set even though the history was empty — which
194
+ **raised the confidence badge** on exactly the recommendations that had no
195
+ supporting evidence behind them.
196
+
197
+ Defences, in order of value:
198
+
199
+ 1. **Test the seam with production's own inputs**, not inputs the test
200
+ pre-normalizes. If the test does work the pipeline doesn't, it isn't testing
201
+ the pipeline.
202
+ 2. **Assert on non-zero contribution, not just on equality.** "The join
203
+ produced ≥ N records" catches silent-empty; "the outputs match" doesn't.
204
+ 3. **Replay a real historical run end to end** and diff against what production
205
+ actually produced. A replay harness that reproduces a known run exactly
206
+ (e.g. 319/319 records) is what lets you attribute a delta to one cause.
207
+ 4. **When a fix changes headline numbers, re-derive every number that depended
208
+ on it** and mark the old ones. Here: drops 111→77 and 178→83, raises 11→36
209
+ and 2→33 — every prior claim about drop volume was void the moment the fix
210
+ landed.
211
+
212
+ ---
213
+
214
+ ## Part 7 — A backtest is a proxy; the gate is a live window
215
+
216
+ **The formal acceptance in the SOW was a ~10-business-day live window with the
217
+ named client reviewer, not a historical backtest.** The backtest is our internal
218
+ de-risking proxy for it. Never let the two be confused — passing a backtest is
219
+ not passing acceptance, and saying so plainly protects everyone.
220
+
221
+ So the go-live sequence is:
222
+
223
+ 1. Backtest to the pinned metric → evidence record.
224
+ 2. **Shadow period**: the automation proposes, a named human decides, for a
225
+ defined window. Every build in these repos does this.
226
+ 3. Only then does the output get treated as decidable truth.
227
+
228
+ Also, take the win when the client changes the frame. When one client declined
229
+ to annotate historical examples and wanted to use the tool live instead, the
230
+ right response was to **retire the annotation ask and the backtest gate for
231
+ that arm**, reframing it as rule-based flags — not to keep chasing a gate the
232
+ client no longer cared about. The gate serves the engagement, not the reverse.
233
+
234
+ ---
235
+
236
+ ## Part 8 — Evidence discipline
237
+
238
+ One dated, self-contained ~1-page record per run, in the build repo's
239
+ `evidence/`. Template and conventions:
240
+ [`references/evidence-record-template.md`](references/evidence-record-template.md).
241
+
242
+ The parts that make it work over months:
243
+
244
+ - **An index table at the top of `evidence/README.md`**, newest first, one line
245
+ each with the headline number.
246
+ - **Supersession is loud and in place.** Strike through the wrong conclusion,
247
+ say what replaced it, keep what was right. Never quietly delete.
248
+ - **A standing caveat block** above the index when something invalidates a
249
+ whole class of prior numbers.
250
+ - **Mark internal-only records**, and keep client-facing ones free of raw
251
+ identifiers and personal data — aggregates only.
252
+ - **Point at the reproduce-it-yourself doc** in the build folder rather than
253
+ duplicating steps.
254
+
255
+ Harness rules: [`references/harness-checklist.md`](references/harness-checklist.md).
256
+ In short — every number comes from a committed script with a copy-pasteable
257
+ command line, thresholds live in one config file rather than scattered through
258
+ code, and the raw inputs stay out of git.
259
+
260
+ ---
261
+
262
+ ## Requirements checklist
263
+
264
+ - [ ] Gate definition read off the **signed** SOW; the number and its wording quoted in the tech spec
265
+ - [ ] Metric pinned; competing readings computed and reported alongside
266
+ - [ ] Trivial/near-zero cases identified and excluded from any tolerance-band metric
267
+ - [ ] Client's idiosyncratic markers understood before interpreting their history
268
+ - [ ] Scoring at pre-decision state; every input filtered to `≤ asOf`; no look-ahead
269
+ - [ ] Censoring and "absence" semantics defined in writing
270
+ - [ ] Contamination exclusions: named, dated, encoded in the harness, control arm shown unchanged, written up
271
+ - [ ] Segmented by decision type **and** by data density
272
+ - [ ] Inputs reconciled against an independent source before trusting any metric
273
+ - [ ] Seam tests use production's own inputs; assertions include non-zero contribution
274
+ - [ ] A real historical run replayed end to end and diffed
275
+ - [ ] Every number traceable to a committed script and a command line
276
+ - [ ] Evidence record written, indexed, and superseded-in-place where relevant
277
+ - [ ] Shadow period run with a named reviewer before go-live
278
+ - [ ] Delivery-log entry linking the evidence record
279
+
280
+ ## Known gaps to flag, not silently skip
281
+
282
+ - **A backtest measures agreement with past behaviour, not correctness.** If
283
+ the client's historical decisions were poor, high agreement means we
284
+ reproduced poor decisions. Say which one the gate is actually testing.
285
+ - **Historical data is rarely balanced across arms.** An arm with few events
286
+ and sparse inputs cannot be validated to the same confidence as the main arm.
287
+ Report per-arm sample sizes; a 14.7% on 495 events and a 94.1% on 1,509 are
288
+ not comparable evidence.
289
+ - **Thresholds tuned on a dataset and validated on the same dataset are
290
+ overfit.** Hold out a validation slice, and say whether you did.
291
+ - **The gate is a point-in-time reading.** Feed quality, client behaviour and
292
+ the market all move. Date every figure, and re-run before quoting an old one
293
+ in a new conversation.