@installagent/sdk 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/agent-templates.d.ts +36 -2
- package/dist/agent-templates.js +173 -3
- package/dist/client-linear-create.d.ts +26 -0
- package/dist/client-linear-create.js +46 -0
- package/dist/client-linear.d.ts +28 -9
- package/dist/client-linear.js +36 -14
- package/dist/docs-search.d.ts +19 -6
- package/dist/docs-search.js +32 -11
- package/dist/doctor.d.ts +9 -0
- package/dist/doctor.js +126 -23
- package/dist/env-fill.d.ts +13 -0
- package/dist/env-fill.js +34 -9
- package/dist/env-push.d.ts +3 -0
- package/dist/env-push.js +11 -6
- package/dist/env-registry.d.ts +25 -0
- package/dist/env-registry.js +69 -7
- package/dist/exec.js +15 -2
- package/dist/index.d.ts +14 -6
- package/dist/index.js +14 -6
- package/dist/linear-project.d.ts +2 -2
- package/dist/linear-project.js +2 -2
- package/dist/log.d.ts +28 -0
- package/dist/log.js +73 -0
- package/dist/n8n-executions.d.ts +85 -0
- package/dist/n8n-executions.js +259 -0
- package/dist/op-cli.d.ts +29 -0
- package/dist/op-cli.js +52 -0
- package/dist/platform.d.ts +35 -0
- package/dist/platform.js +39 -0
- package/dist/portal-admin.d.ts +40 -0
- package/dist/portal-admin.js +30 -0
- package/dist/standup.d.ts +88 -0
- package/dist/standup.js +235 -0
- package/dist/sync-repos.d.ts +47 -0
- package/dist/sync-repos.js +80 -0
- package/dist/template-sync.d.ts +10 -0
- package/dist/template-sync.js +11 -0
- package/dist/wizard.d.ts +15 -0
- package/dist/wizard.js +35 -56
- package/dist/workspace-docs.d.ts +44 -0
- package/dist/workspace-docs.js +73 -0
- package/package.json +2 -2
- package/templates/env.example +33 -23
- package/templates/workspace-root/AGENT.md +7 -5
- package/templates/workspace-root/ONBOARDING.md +65 -8
- package/templates/workspace-root/skills/.github/workflows/pr-to-slack.yml +102 -0
- package/templates/workspace-root/skills/CLAUDE.md +24 -6
- package/templates/workspace-root/skills/README.md +156 -0
- package/templates/workspace-root/skills/backtest-acceptance-gates/SKILL.md +293 -0
- package/templates/workspace-root/skills/backtest-acceptance-gates/backtest-acceptance-gates-guide.pdf +0 -0
- package/templates/workspace-root/skills/backtest-acceptance-gates/references/evidence-record-template.md +128 -0
- package/templates/workspace-root/skills/backtest-acceptance-gates/references/harness-checklist.md +90 -0
- package/templates/workspace-root/skills/client-latest/SKILL.md +301 -0
- package/templates/workspace-root/skills/client-latest/catchup.command.md +17 -0
- package/templates/workspace-root/skills/cloud-devops/SKILL.md +235 -0
- package/templates/workspace-root/skills/email-triage-rollout/SKILL.md +181 -0
- package/templates/workspace-root/skills/gcloud-cli-setup/SKILL.md +100 -0
- package/templates/workspace-root/skills/google-chat-app-setup/SKILL.md +136 -0
- package/templates/workspace-root/skills/google-chat-app-setup/google-chat-app-setup-guide.pdf +0 -0
- package/templates/workspace-root/skills/house-voice/SKILL.md +80 -4
- package/templates/workspace-root/skills/ia-cli/SKILL.md +19 -6
- package/templates/workspace-root/skills/installagent-brand-kit/SKILL.md +59 -10
- package/templates/workspace-root/skills/installagent-brand-kit/assets/installagent-deck-template.pptx +0 -0
- package/templates/workspace-root/skills/installagent-brand-kit/references/deck-guide.md +181 -0
- package/templates/workspace-root/skills/installagent-brand-kit/scripts/deck.py +652 -0
- package/templates/workspace-root/skills/installagent-brand-kit/scripts/deck_lint.py +201 -0
- package/templates/workspace-root/skills/installagent-brand-kit/scripts/logo_raster.py +102 -0
- package/templates/workspace-root/skills/installagent-brand-kit/scripts/make_deck_template.py +290 -0
- package/templates/workspace-root/skills/salesforce-api-snapshot/SKILL.md +379 -0
- package/templates/workspace-root/skills/salesforce-api-snapshot/references/schema-discovery-soql.md +153 -0
- package/templates/workspace-root/skills/salesforce-api-snapshot/references/snapshot-storage.md +122 -0
- package/templates/workspace-root/skills/salesforce-api-snapshot/salesforce-api-snapshot-guide.pdf +0 -0
- package/templates/workspace-root/skills/slack-voice/SKILL.md +158 -0
- package/templates/workspace-root/skills/sso-gated-client-app/SKILL.md +337 -0
- package/templates/workspace-root/skills/sso-gated-client-app/sso-gated-client-app-guide.pdf +0 -0
- package/templates/workspace-root/skills/tech-call-prep/SKILL.md +146 -47
- package/templates/workspace-root/skills/vendor-feed-ingestion/SKILL.md +323 -0
- package/templates/workspace-root/skills/vendor-feed-ingestion/references/delivery-contract-template.md +110 -0
- package/templates/workspace-root/skills/vendor-feed-ingestion/references/feed-guards.md +134 -0
- package/templates/workspace-root/skills/vendor-feed-ingestion/vendor-feed-ingestion-guide.pdf +0 -0
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
name: PR → Slack (approve / close)
|
|
2
|
+
|
|
3
|
+
# Posts every new PR to Slack with "Merge" and "Close" buttons.
|
|
4
|
+
# This workflow only POSTS (it is read-only). The button clicks are handled by
|
|
5
|
+
# the n8n workflow "GitHub PR Actions (Slack)" at
|
|
6
|
+
# https://n8n.installagent.com/webhook/slack-pr-action , which performs the
|
|
7
|
+
# actual merge/close using the "Github - Install Agent Bot" credential. That
|
|
8
|
+
# receiver derives owner/repo/PR number by regex over the PR URL in the message
|
|
9
|
+
# blocks, so it is repo-agnostic — keep the PR URL in a section block below.
|
|
10
|
+
#
|
|
11
|
+
# Required secret:
|
|
12
|
+
# SLACK_PR_WEBHOOK_URL — a Slack Incoming Webhook URL bound to the review
|
|
13
|
+
# channel (#notify-prs). Set once as an ORGANIZATION
|
|
14
|
+
# secret with visibility "all repositories", so every
|
|
15
|
+
# current and future repo is covered with no per-repo
|
|
16
|
+
# setup. A repo-level secret of the same name also
|
|
17
|
+
# works and takes precedence.
|
|
18
|
+
# The webhook's Slack app must have Interactivity
|
|
19
|
+
# enabled and pointed at the n8n receiver URL above.
|
|
20
|
+
#
|
|
21
|
+
# If the secret is absent the job logs a notice and exits green, so this file is
|
|
22
|
+
# safe to land in a repo before the secret exists.
|
|
23
|
+
|
|
24
|
+
on:
|
|
25
|
+
pull_request:
|
|
26
|
+
types: [opened, reopened, ready_for_review]
|
|
27
|
+
|
|
28
|
+
permissions:
|
|
29
|
+
contents: read
|
|
30
|
+
pull-requests: read
|
|
31
|
+
|
|
32
|
+
jobs:
|
|
33
|
+
notify:
|
|
34
|
+
# Drafts are not ready to review, and skipping them also stops the
|
|
35
|
+
# duplicate post a draft would otherwise make on opened + ready_for_review.
|
|
36
|
+
if: github.event.pull_request.draft == false
|
|
37
|
+
runs-on: ubuntu-latest
|
|
38
|
+
steps:
|
|
39
|
+
- name: Post PR to Slack with action buttons
|
|
40
|
+
env:
|
|
41
|
+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_PR_WEBHOOK_URL }}
|
|
42
|
+
PR_REPO: ${{ github.event.repository.name }}
|
|
43
|
+
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
44
|
+
PR_TITLE: ${{ github.event.pull_request.title }}
|
|
45
|
+
PR_URL: ${{ github.event.pull_request.html_url }}
|
|
46
|
+
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
|
|
47
|
+
PR_BASE: ${{ github.event.pull_request.base.ref }}
|
|
48
|
+
PR_HEAD: ${{ github.event.pull_request.head.ref }}
|
|
49
|
+
PR_ADDITIONS: ${{ github.event.pull_request.additions }}
|
|
50
|
+
PR_DELETIONS: ${{ github.event.pull_request.deletions }}
|
|
51
|
+
PR_CHANGED: ${{ github.event.pull_request.changed_files }}
|
|
52
|
+
run: |
|
|
53
|
+
set -euo pipefail
|
|
54
|
+
if [ -z "${SLACK_WEBHOOK_URL:-}" ]; then
|
|
55
|
+
echo "::notice::SLACK_PR_WEBHOOK_URL is not set (org or repo secret) — skipping the Slack post."
|
|
56
|
+
exit 0
|
|
57
|
+
fi
|
|
58
|
+
# Build the Block Kit payload with jq so PR titles/branches that
|
|
59
|
+
# contain quotes or other JSON-breaking characters are escaped safely.
|
|
60
|
+
payload=$(jq -n \
|
|
61
|
+
--arg repo "$PR_REPO" \
|
|
62
|
+
--arg num "$PR_NUMBER" \
|
|
63
|
+
--arg title "$PR_TITLE" \
|
|
64
|
+
--arg url "$PR_URL" \
|
|
65
|
+
--arg author "$PR_AUTHOR" \
|
|
66
|
+
--arg base "$PR_BASE" \
|
|
67
|
+
--arg head "$PR_HEAD" \
|
|
68
|
+
--arg stats "+$PR_ADDITIONS / -$PR_DELETIONS · $PR_CHANGED files" \
|
|
69
|
+
'{
|
|
70
|
+
text: ("New PR in " + $repo + " #" + $num + ": " + $title),
|
|
71
|
+
blocks: [
|
|
72
|
+
{ type: "header",
|
|
73
|
+
text: { type: "plain_text", text: ("🔀 " + $repo + " · PR #" + $num) } },
|
|
74
|
+
{ type: "section",
|
|
75
|
+
text: { type: "mrkdwn",
|
|
76
|
+
text: ( "*<" + $url + "|" + $title + ">*\n"
|
|
77
|
+
+ "by `" + $author + "` • `" + $head + "` → `" + $base + "`\n"
|
|
78
|
+
+ $stats ) } },
|
|
79
|
+
{ type: "actions",
|
|
80
|
+
block_id: ("pr_" + $num),
|
|
81
|
+
elements: [
|
|
82
|
+
{ type: "button", style: "primary", action_id: "pr_merge",
|
|
83
|
+
text: { type: "plain_text", text: "✅ Merge" }, value: $num,
|
|
84
|
+
confirm: {
|
|
85
|
+
title: { type: "plain_text", text: "Merge this PR?" },
|
|
86
|
+
text: { type: "mrkdwn", text: ("Squash-merge *" + $repo + "* PR #" + $num + " into `" + $base + "` and delete `" + $head + "`.") },
|
|
87
|
+
confirm: { type: "plain_text", text: "Merge" },
|
|
88
|
+
deny: { type: "plain_text", text: "Cancel" } } },
|
|
89
|
+
{ type: "button", style: "danger", action_id: "pr_close",
|
|
90
|
+
text: { type: "plain_text", text: "🗑️ Close" }, value: $num,
|
|
91
|
+
confirm: {
|
|
92
|
+
title: { type: "plain_text", text: "Close this PR?" },
|
|
93
|
+
text: { type: "mrkdwn", text: ("Close *" + $repo + "* PR #" + $num + " without merging.") },
|
|
94
|
+
confirm: { type: "plain_text", text: "Close" },
|
|
95
|
+
deny: { type: "plain_text", text: "Cancel" } } }
|
|
96
|
+
] }
|
|
97
|
+
]
|
|
98
|
+
}')
|
|
99
|
+
# --fail-with-body so a rejected payload or a dead webhook turns the
|
|
100
|
+
# job red instead of passing silently on Slack's 4xx body.
|
|
101
|
+
curl -sS --fail-with-body -X POST -H 'Content-type: application/json' \
|
|
102
|
+
--data "$payload" "$SLACK_WEBHOOK_URL"
|
|
@@ -13,18 +13,27 @@ so Claude Code / Cowork auto-register them in the skill menu.
|
|
|
13
13
|
|
|
14
14
|
```
|
|
15
15
|
skills/
|
|
16
|
+
├── client-latest/ # resume-the-build brief for one client (state + risks/drift, then a next-step CTA)
|
|
16
17
|
├── client-deliverables/ # ORCHESTRATOR — the full client engagement package, checklist + sequencing
|
|
17
18
|
├── engagement-spec/ # discovery call → overview + tech-spec + onboarding docs
|
|
18
|
-
├── tech-call-prep/ # pre-call prep check + per-client call
|
|
19
|
+
├── tech-call-prep/ # pre-call prep check + per-client call agenda (docs/05-call-agenda.md + Drive copy)
|
|
19
20
|
├── client-llm-setup/ # the client's own Claude workspace → branded leave-behind guide
|
|
20
21
|
├── client-facing-docs/ # the branded client PDFs (exec overview, proposal/SOW, onboarding)
|
|
21
22
|
├── client-workflow-diagram/ # spec → Excalidraw stage-column diagram
|
|
22
|
-
├──
|
|
23
|
+
├── google-chat-app-setup/ # deploy a Google Chat app (n8n-backed) in a client's Workspace, domain-wide
|
|
24
|
+
├── salesforce-api-snapshot/ # read a client's Salesforce/managed-package org → schema spike + cached JSON snapshot
|
|
25
|
+
├── sso-gated-client-app/ # a client-facing internal app behind their Google Workspace SSO (Next.js + Vercel)
|
|
26
|
+
├── vendor-feed-ingestion/ # recurring vendor feeds (SFTP/email → S3 → n8n) + the guards that refuse a bad file
|
|
27
|
+
├── email-triage-rollout/ # AI inbox triage into a live mailbox: staged soak → label → organize → full + the gate per stage
|
|
28
|
+
├── backtest-acceptance-gates/ # prove an automation works before the client trusts it (ground truth, evidence, gates)
|
|
29
|
+
├── installagent-brand-kit/ # our brand for print/PDF/Excalidraw/decks (logo + brand red #d11a1a)
|
|
23
30
|
├── design-system/ # the Install Agent Design System (screens/UI)
|
|
24
31
|
├── house-voice/ # house voice for client-facing writing, pulled from installagent.com/get-started
|
|
25
32
|
├── contract-tech-agnostic/ # contracts/SOWs by FUNCTION and OUTCOME, never tech (+ SOW template in references/)
|
|
26
33
|
├── linear/ # Linear issue workflow for client build work (status discipline, sub-issue trees, PR linking)
|
|
27
|
-
|
|
34
|
+
├── ia-cli/ # route internal ops through the `ia` CLI (team-tools) — decision map + guardrails
|
|
35
|
+
├── gcloud-cli-setup/ # one named gcloud configuration per account/client + check-before-you-run discipline
|
|
36
|
+
└── slack-voice/ # INTERNAL Slack: terse bullets, hard numbers, no adjectives (client Slack → house-voice)
|
|
28
37
|
```
|
|
29
38
|
|
|
30
39
|
Each is `skills/<name>/SKILL.md`, with bulky tables in `references/`.
|
|
@@ -36,20 +45,29 @@ Each is `skills/<name>/SKILL.md`, with bulky tables in `references/`.
|
|
|
36
45
|
| Kicking off / assembling a full client deliverable package | `client-deliverables` |
|
|
37
46
|
| Writing up a discovery/scoping call into build docs | `engagement-spec` |
|
|
38
47
|
| Prepping for an upcoming client call (sheet + readiness check) | `tech-call-prep` |
|
|
48
|
+
| Catching up on a client / resuming a build / checking for drift | `client-latest` |
|
|
39
49
|
| Setting up a client's own Claude workspace / its leave-behind guide | `client-llm-setup` |
|
|
40
50
|
| Producing the branded client PDFs | `client-facing-docs` |
|
|
41
51
|
| Drawing a workflow/architecture diagram for a client | `client-workflow-diagram` |
|
|
42
|
-
|
|
|
52
|
+
| Rolling an email-triage agent into a client's live mailbox | `email-triage-rollout` |
|
|
53
|
+
| Deploying a Google Chat app (n8n-backed) in a client's Workspace | `google-chat-app-setup` |
|
|
54
|
+
| Reading a client's Salesforce (or any vendor CRM/EHR) into something we build | `salesforce-api-snapshot` |
|
|
55
|
+
| Building a screen the client's staff log into (dashboard, map, worklist) | `sso-gated-client-app` |
|
|
56
|
+
| Setting up a recurring vendor data feed, or debugging one that went wrong | `vendor-feed-ingestion` |
|
|
57
|
+
| Proving accuracy / hitting an acceptance gate / writing up a validation run | `backtest-acceptance-gates` |
|
|
58
|
+
| Branding a print/PDF/Excalidraw deliverable, or building a client deck | `installagent-brand-kit` |
|
|
43
59
|
| Building or reviewing any UI / web page / funnel / component | `design-system` |
|
|
44
60
|
| Writing or reviewing any client-facing copy (proposal, email, guide, web/ad copy) | `house-voice` |
|
|
61
|
+
| Writing any Slack message to a teammate or internal channel | `slack-voice` |
|
|
45
62
|
| Drafting/reviewing any contract, SOW, MSA, or proposal | `contract-tech-agnostic` |
|
|
46
63
|
| Working an `INS-xxx` issue / a client's Linear project | `linear` |
|
|
47
64
|
| Any task an `ia` command covers (client context, intake, agent bank, secrets, setup/doctor) | `ia-cli` |
|
|
65
|
+
| Setting up or switching gcloud CLI accounts/configurations | `gcloud-cli-setup` |
|
|
48
66
|
|
|
49
67
|
`design-system` is the **screen** layer (dark, signal-red, Hanken Grotesk); it
|
|
50
68
|
mirrors the code source of truth `team-web/packages/ui`.
|
|
51
|
-
`installagent-brand-kit` is the **print/deliverable** layer for PDFs
|
|
52
|
-
Excalidraw on white. They share one brand and must agree. `house-voice` is
|
|
69
|
+
`installagent-brand-kit` is the **print/deliverable** layer for PDFs,
|
|
70
|
+
Excalidraw and decks on white. They share one brand and must agree. `house-voice` is
|
|
53
71
|
the **words** layer that applies across both — what client-facing sentences
|
|
54
72
|
sound like, regardless of surface.
|
|
55
73
|
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Install Agent skills
|
|
2
|
+
|
|
3
|
+
Twenty-two reusable, cross-repo how-tos that any coding agent in the workspace
|
|
4
|
+
loads when a task matches the skill's `description`. They encode the parts of
|
|
5
|
+
our delivery process that are easy to get wrong and expensive to rediscover —
|
|
6
|
+
the SOW clause we always forget, the Excalidraw JSON that has to be valid, the
|
|
7
|
+
four ways an SSO-gated app can look fine while being wide open.
|
|
8
|
+
|
|
9
|
+
This directory is its own git repo ([`InstallAgent/skills`](https://github.com/InstallAgent/skills)),
|
|
10
|
+
mounted at the workspace root because a single client engagement touches docs,
|
|
11
|
+
workflows, and diagrams across more than one repo. Repo-local skills live in
|
|
12
|
+
`team-clients/skills/` and `team-infra/skills/` instead.
|
|
13
|
+
|
|
14
|
+
> Working here as an agent? Read [`CLAUDE.md`](./CLAUDE.md) — it's the rule
|
|
15
|
+
> set. This README is the map.
|
|
16
|
+
|
|
17
|
+
## How these get loaded
|
|
18
|
+
|
|
19
|
+
Every skill is `<name>/SKILL.md` with YAML frontmatter (`name`, `description`).
|
|
20
|
+
21 of them are symlinked into [`../.claude/skills/`](../.claude/skills/), so Claude
|
|
21
|
+
Code and Cowork register them automatically — they appear in the skill menu and
|
|
22
|
+
fire on their own without being invoked by name. The `description` is the
|
|
23
|
+
trigger; the body is only read once the skill fires.
|
|
24
|
+
|
|
25
|
+
Nothing has to be run or installed. Adding a skill means adding a directory and
|
|
26
|
+
a symlink (see [Adding a skill](#adding-a-skill)).
|
|
27
|
+
|
|
28
|
+
## The engagement lifecycle
|
|
29
|
+
|
|
30
|
+
Most of these skills are stops along one path — a lead becomes a signed
|
|
31
|
+
engagement becomes a working build. In rough order:
|
|
32
|
+
|
|
33
|
+
| Stage | Skill | What it produces |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| 0. Landing cold / resuming | [`client-latest`](./client-latest/) | A terminal brief — where the build stands, then risks + surface drift — then a four-option next-step prompt |
|
|
36
|
+
| 1. Before the call | [`tech-call-prep`](./tech-call-prep/) | Readiness check + `docs/05-call-agenda.md` and a Drive copy |
|
|
37
|
+
| 2. After the call | [`engagement-spec`](./engagement-spec/) | Client overview, internal tech spec, onboarding + access checklists, `onboarding.html` |
|
|
38
|
+
| 3. Package it | [`client-deliverables`](./client-deliverables/) | **Orchestrator** — sequences 4, 5, and the diagram into the full engagement set |
|
|
39
|
+
| ↳ | [`client-facing-docs`](./client-facing-docs/) | The three branded PDFs: exec overview, proposal/SOW, onboarding checklist |
|
|
40
|
+
| ↳ | [`contract-tech-agnostic`](./contract-tech-agnostic/) | SOW/MSA language by function and outcome, never by tool |
|
|
41
|
+
| ↳ | [`client-workflow-diagram`](./client-workflow-diagram/) | Stage-column `.excalidraw` + exported image |
|
|
42
|
+
| 4. Week 1 | [`client-llm-setup`](./client-llm-setup/) | The client's own Claude workspace + branded leave-behind guide |
|
|
43
|
+
| 5. Build | the integration skills below | The automation itself |
|
|
44
|
+
| 6. Track | [`linear`](./linear/) | `INS-xxx` status discipline, sub-issue trees, PR linking |
|
|
45
|
+
| 7. Prove it | [`backtest-acceptance-gates`](./backtest-acceptance-gates/) | Ground truth, evidence record, a defensible accuracy number |
|
|
46
|
+
|
|
47
|
+
### Integration / build skills
|
|
48
|
+
|
|
49
|
+
Reach for these when the build touches a specific system. They're independent
|
|
50
|
+
of each other, except that a Salesforce read usually feeds one of the other two.
|
|
51
|
+
|
|
52
|
+
| Skill | Use when |
|
|
53
|
+
|---|---|
|
|
54
|
+
| [`vendor-feed-ingestion`](./vendor-feed-ingestion/) | A recurring vendor file lands (SFTP/email → S3 → n8n), or one went wrong |
|
|
55
|
+
| [`email-triage-rollout`](./email-triage-rollout/) | An AI triage agent goes into a client's live mailbox — the staged soak → label → organize → full program and its gates |
|
|
56
|
+
| [`salesforce-api-snapshot`](./salesforce-api-snapshot/) | Reading a client's Salesforce or managed-package org (Lumary, Vlocity, nCino) |
|
|
57
|
+
| [`sso-gated-client-app`](./sso-gated-client-app/) | Building a screen the client's staff log into — dashboard, map, worklist |
|
|
58
|
+
| [`google-chat-app-setup`](./google-chat-app-setup/) | Deploying an n8n-backed Chat app across a client's Workspace domain |
|
|
59
|
+
|
|
60
|
+
### The three house-style layers
|
|
61
|
+
|
|
62
|
+
These are always-on constraints rather than lifecycle stages. They share one
|
|
63
|
+
brand and must agree with each other.
|
|
64
|
+
|
|
65
|
+
| Layer | Skill | Governs |
|
|
66
|
+
|---|---|---|
|
|
67
|
+
| **Screens** | [`design-system`](./design-system/) | Any UI, page, funnel, component. Dark, signal red, Hanken Grotesk. Mirrors `team-web/packages/ui`. |
|
|
68
|
+
| **Print** | [`installagent-brand-kit`](./installagent-brand-kit/) | PDFs, Excalidraw, decks — on white, brand red `#d11a1a`. Ships `scripts/brand.py`, `deck.py`, `deck_lint.py`. |
|
|
69
|
+
| **Words** | [`house-voice`](./house-voice/) | Every client-facing sentence, on any surface. Pulled verbatim from installagent.com. |
|
|
70
|
+
|
|
71
|
+
### Internal ops
|
|
72
|
+
|
|
73
|
+
[`ia-cli`](./ia-cli/) — the decision map from task to `ia` command (client
|
|
74
|
+
context, intake, agent bank, secrets, `wizard`/`doctor`). Check it before
|
|
75
|
+
writing any custom script against GitHub, Airtable, Linear, Drive, n8n, or the
|
|
76
|
+
portal.
|
|
77
|
+
|
|
78
|
+
## How they compose
|
|
79
|
+
|
|
80
|
+
Arrows point from a skill to the ones it defers to. `client-deliverables` is
|
|
81
|
+
the only true orchestrator; the rest cross-reference for a specific slice.
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
client-deliverables ──┬─→ engagement-spec ──┬─→ client-workflow-diagram
|
|
85
|
+
│ ├─→ contract-tech-agnostic
|
|
86
|
+
│ └─→ ia-cli
|
|
87
|
+
├─→ client-facing-docs
|
|
88
|
+
├─→ contract-tech-agnostic
|
|
89
|
+
├─→ client-workflow-diagram
|
|
90
|
+
└─→ installagent-brand-kit
|
|
91
|
+
|
|
92
|
+
tech-call-prep ───────→ engagement-spec
|
|
93
|
+
client-llm-setup ─────→ client-facing-docs, installagent-brand-kit
|
|
94
|
+
salesforce-api-snapshot → sso-gated-client-app, vendor-feed-ingestion
|
|
95
|
+
email-triage-rollout ─→ agents/ bank sop.md (per-provider), linear, house-voice
|
|
96
|
+
house-voice ──────────→ (applies across all client-facing output)
|
|
97
|
+
design-system ────────→ installagent-brand-kit (shared brand source)
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Also cross-referenced from here but living elsewhere: `business-research`
|
|
101
|
+
(team-clients) and `n8n-workflow` + `automation-design` (team-infra).
|
|
102
|
+
|
|
103
|
+
## What's in each directory
|
|
104
|
+
|
|
105
|
+
`SKILL.md` is the whole skill; `references/` holds tables and templates too
|
|
106
|
+
bulky for the body; `scripts/` and `assets/` are only in the brand kit. A few
|
|
107
|
+
skills also ship a rendered PDF guide for handing to a technician.
|
|
108
|
+
|
|
109
|
+
| Skill | SKILL.md | Also ships |
|
|
110
|
+
|---|---:|---|
|
|
111
|
+
| `backtest-acceptance-gates` | 293 | `references/` evidence-record template, harness checklist · PDF guide |
|
|
112
|
+
| `client-deliverables` | 131 | `references/package-layout.md` |
|
|
113
|
+
| `client-latest` | 301 | `catchup.command.md` (the `/catchup` command) |
|
|
114
|
+
| `cloud-devops` | 235 | — · **orphan — adopt or delete (pending decision)**, not symlinked |
|
|
115
|
+
| `client-facing-docs` | 131 | `references/doc-structures.md` |
|
|
116
|
+
| `client-llm-setup` | 97 | `references/guide-template.md` |
|
|
117
|
+
| `client-workflow-diagram` | 141 | `references/excalidraw-format.md` |
|
|
118
|
+
| `contract-tech-agnostic` | 99 | `references/Install_Agent_SOW_Template.docx` |
|
|
119
|
+
| `design-system` | 118 | `references/tokens.md` |
|
|
120
|
+
| `email-triage-rollout` | 181 | — |
|
|
121
|
+
| `engagement-spec` | 272 | `references/spec-templates.md` |
|
|
122
|
+
| `gcloud-cli-setup` | 100 | — |
|
|
123
|
+
| `google-chat-app-setup` | 136 | PDF guide |
|
|
124
|
+
| `house-voice` | 228 | `references/source-copy.md` |
|
|
125
|
+
| `ia-cli` | 87 | — |
|
|
126
|
+
| `installagent-brand-kit` | 161 | 4 `references/`, 5 `scripts/` (brand, deck, deck_lint, logo_raster, make_deck_template), logo SVGs + deck template |
|
|
127
|
+
| `linear` | 186 | — |
|
|
128
|
+
| `salesforce-api-snapshot` | 379 | `references/` schema-discovery SOQL, snapshot storage · PDF guide |
|
|
129
|
+
| `slack-voice` | 158 | — |
|
|
130
|
+
| `sso-gated-client-app` | 337 | PDF guide |
|
|
131
|
+
| `tech-call-prep` | 240 | — |
|
|
132
|
+
| `vendor-feed-ingestion` | 323 | `references/` delivery contract, feed guards · PDF guide |
|
|
133
|
+
|
|
134
|
+
## Conventions
|
|
135
|
+
|
|
136
|
+
Full rules in [`CLAUDE.md`](./CLAUDE.md). The short version:
|
|
137
|
+
|
|
138
|
+
- One skill per directory, `SKILL.md` under ~500 lines — push long tables into
|
|
139
|
+
`references/`.
|
|
140
|
+
- The `description` carries every "when to use" cue, stated concretely. That's
|
|
141
|
+
what makes the skill fire; the body never gets read otherwise.
|
|
142
|
+
- Nothing sensitive: no secrets, live credentials, internal hostnames, account
|
|
143
|
+
ids, or PII. Use placeholders and point at internal files.
|
|
144
|
+
- If a skill conflicts with [`../AGENT.md`](../AGENT.md), **AGENT.md wins** —
|
|
145
|
+
fix the skill in the same change.
|
|
146
|
+
|
|
147
|
+
## Adding a skill
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
mkdir -p skills/<name>/references
|
|
151
|
+
$EDITOR skills/<name>/SKILL.md # frontmatter: name, description
|
|
152
|
+
cd .claude/skills && ln -s ../../skills/<name> <name>
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Then add a row to the right table above and to the tree in `CLAUDE.md`. Skills
|
|
156
|
+
are how-tos — workspace-wide *rules* belong in `AGENT.md` instead.
|
|
@@ -0,0 +1,293 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: backtest-acceptance-gates
|
|
3
|
+
description: >-
|
|
4
|
+
Prove an automation actually works before a client is asked to trust it —
|
|
5
|
+
and before we invoice against an acceptance gate. Covers reading the gate
|
|
6
|
+
definition off the signed SOW rather than from memory, pinning down which
|
|
7
|
+
metric the gate means when several plausible ones disagree by 18 points,
|
|
8
|
+
reconstructing ground truth from a client's own historical behaviour
|
|
9
|
+
(including the bimodal-signal problem, no-look-ahead scoring, and
|
|
10
|
+
right-censoring), excluding contaminated events legitimately instead of
|
|
11
|
+
conveniently, segmenting to tell a data gap from a logic gap, discovering
|
|
12
|
+
that ground truth itself is only 79% complete, the evidence-record
|
|
13
|
+
discipline that keeps a months-long build honest about which of its own
|
|
14
|
+
numbers are void, and why a green test suite can mask the exact bug it was
|
|
15
|
+
written to catch. Use this WHENEVER an engagement has an accuracy or
|
|
16
|
+
acceptance criterion — "backtest it", "how accurate is it", "prove it works",
|
|
17
|
+
"we need to hit the gate", "run it against history", "the client wants
|
|
18
|
+
evidence", "is it ready for go-live" — or when an existing accuracy number
|
|
19
|
+
needs re-checking.
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
# Backtesting & acceptance gates
|
|
23
|
+
|
|
24
|
+
Practice from the CarVision pricing agent
|
|
25
|
+
(`clients/client-carvision/agent-pricing/` + that repo's `evidence/`), where an
|
|
26
|
+
SOW acceptance gate had real money attached and the accuracy number was
|
|
27
|
+
re-derived, contradicted, retracted and rebuilt over two months.
|
|
28
|
+
|
|
29
|
+
**What this skill is really about: staying honest at speed.** Almost every
|
|
30
|
+
mistake below produced a *number*, and the number looked fine. The discipline
|
|
31
|
+
isn't producing measurements — it's knowing which of your measurements are
|
|
32
|
+
still true.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Part 1 — Get the gate from the contract, and pin the metric
|
|
37
|
+
|
|
38
|
+
**Read the acceptance criterion off the signed SOW.** Not the proposal draft,
|
|
39
|
+
not the kickoff deck, not memory. On CarVision the team measured against 85%
|
|
40
|
+
for weeks; the signed SOW said **80%**, on a review sample agreed by both
|
|
41
|
+
parties. Every measured number was unaffected — only the bar they were compared
|
|
42
|
+
against, and therefore every "we're short" judgement, was wrong.
|
|
43
|
+
|
|
44
|
+
**Then pin the metric, because several plausible readings will disagree
|
|
45
|
+
materially.** The same 2,004 decisions scored:
|
|
46
|
+
|
|
47
|
+
| Reading | Result |
|
|
48
|
+
|---|---|
|
|
49
|
+
| Direction match (raise/hold/drop agrees) | 62.2% |
|
|
50
|
+
| Price within tolerance | 56.1% |
|
|
51
|
+
| **Direction OR price** (the lenient reading) | **74.5%** |
|
|
52
|
+
|
|
53
|
+
Eighteen points between the loosest and tightest reading of "accurate." Write
|
|
54
|
+
the chosen definition into the tech spec in the SOW's own words, note which
|
|
55
|
+
readings you're *not* using, and report all of them in the evidence record.
|
|
56
|
+
If you only ever quote the flattering one, someone will eventually ask, and
|
|
57
|
+
that conversation is much worse than the honest table.
|
|
58
|
+
|
|
59
|
+
**Also check the arithmetic isn't being inflated by trivia.** An early run
|
|
60
|
+
showed 82.7% — inflated by 1,046 sub-$1 events that satisfied the price band
|
|
61
|
+
automatically. Excluding them gave 74.5%. Any metric with a tolerance band
|
|
62
|
+
will be gamed by near-zero cases; find them before someone else does.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## Part 2 — Reconstruct ground truth
|
|
67
|
+
|
|
68
|
+
Ground truth is what the client's expert actually did, recovered from history.
|
|
69
|
+
Four problems, all of which showed up in one dataset.
|
|
70
|
+
|
|
71
|
+
**The signal is bimodal, and the split is the whole exercise.** The client
|
|
72
|
+
expert's repricing log contained:
|
|
73
|
+
|
|
74
|
+
| Signal | n | What it actually means |
|
|
75
|
+
|---|---|---|
|
|
76
|
+
| ±$1–2 "touch" flips | 1,093 | *"I reviewed this and held."* A **hold** decision. |
|
|
77
|
+
| ≥$100 moves | 2,004 | Genuine repricing decisions. |
|
|
78
|
+
| $3–99 middle | ~16 | Ambiguous — skipped and declared |
|
|
79
|
+
|
|
80
|
+
Reading all 3,113 as "price changes" would have been catastrophic. The $1
|
|
81
|
+
nudges are a review marker in a system with no review button. **Ask the client
|
|
82
|
+
what their idiosyncratic markers mean before interpreting their log** — this
|
|
83
|
+
one only surfaced because someone asked why so many changes were a dollar.
|
|
84
|
+
|
|
85
|
+
**Score at the pre-decision state, with no look-ahead.** For each decision,
|
|
86
|
+
feed the agent the last snapshot *before* the decision — the stale price, that
|
|
87
|
+
row's own last-change date — with every input filtered to `created ≤ asOf`.
|
|
88
|
+
Any input that includes post-decision information turns the backtest into a
|
|
89
|
+
demonstration that the future predicts the past.
|
|
90
|
+
|
|
91
|
+
**Handle censoring explicitly.** Outcome layers ("did the car sell?") are
|
|
92
|
+
right-censored: a unit still listed on the last day has no outcome, not a
|
|
93
|
+
negative one. Say how you treated it.
|
|
94
|
+
|
|
95
|
+
**Absence is not an event.** In a feed where a sold unit simply stops
|
|
96
|
+
appearing, "disappeared" has to be *defined* as an outcome, and the definition
|
|
97
|
+
belongs in the evidence record.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## Part 3 — Excluding contaminated events: legitimately, not conveniently
|
|
102
|
+
|
|
103
|
+
Real history contains events that aren't the behaviour you're modelling. On
|
|
104
|
+
CarVision, **246 of 495 raises (49.7%) were a single bulk test** — one
|
|
105
|
+
operator setting a batch of cars to the same price on one day at one location,
|
|
106
|
+
reverted after it failed. Not pricing strategy. Excluding it moved the metric
|
|
107
|
+
from 63.8% to 72.1% and roughly doubled the raise arm's accuracy.
|
|
108
|
+
|
|
109
|
+
That is a big move in our favour, which is exactly why the bar is high:
|
|
110
|
+
|
|
111
|
+
1. **Name the event.** What happened, when, who did it, and how you know — the
|
|
112
|
+
client confirming it is the strongest form.
|
|
113
|
+
2. **Encode the exclusion as a dated constant in the harness**, not a hand-
|
|
114
|
+
filtered dataset. `TEST_EVENT_DATE` in the build script, reviewable in the
|
|
115
|
+
diff.
|
|
116
|
+
3. **Show the control arm is unchanged.** The drop side didn't move at all when
|
|
117
|
+
the raise-side contamination was removed. An exclusion that improves
|
|
118
|
+
everything is suspicious; one that moves only the arm it should is credible.
|
|
119
|
+
4. **Write it up as its own evidence record**, with the before and after.
|
|
120
|
+
5. **Report the number both ways** the first time it's used in front of the
|
|
121
|
+
client.
|
|
122
|
+
|
|
123
|
+
If you can't do all five, you're not excluding contamination, you're tuning the
|
|
124
|
+
sample.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Part 4 — Segment, to tell a data gap from a logic gap
|
|
129
|
+
|
|
130
|
+
An aggregate number tells you whether you passed. Segmentation tells you what
|
|
131
|
+
to do next.
|
|
132
|
+
|
|
133
|
+
| Segment | n | GATE |
|
|
134
|
+
|---|---|---|
|
|
135
|
+
| DROP decisions | 1,509 | **94.1%** |
|
|
136
|
+
| RAISE decisions | 495 | **14.7%** |
|
|
137
|
+
| Location A (multi-month lead history) | 825 | **84.7%** |
|
|
138
|
+
| Location B (one month of leads) | 1,179 | **67.3%** |
|
|
139
|
+
|
|
140
|
+
Two readings that changed the plan:
|
|
141
|
+
|
|
142
|
+
- **The raise arm didn't fail on logic.** A raise requires a 7-day lead count
|
|
143
|
+
above a threshold; the lead export was a single late-period snapshot, so
|
|
144
|
+
earlier 7-day windows were structurally empty. The demand signal that
|
|
145
|
+
justifies a raise was *invisible*, not wrong. No amount of tuning fixes that.
|
|
146
|
+
- **Same code, different data density, 17 points apart.** That comparison —
|
|
147
|
+
identical logic across two segments — is the cleanest possible evidence that
|
|
148
|
+
the lever is data completeness, not algorithm quality.
|
|
149
|
+
|
|
150
|
+
**Always segment by the arm/decision type and by data density.** Those two cuts
|
|
151
|
+
have repeatedly turned "the model is bad" into "the feed is thin", which is a
|
|
152
|
+
completely different (and usually cheaper) project.
|
|
153
|
+
|
|
154
|
+
And when you report a count, **report what's inside it**. Roughly half of one
|
|
155
|
+
build's "drops" were units already at a cost floor — no cut was possible. A raw
|
|
156
|
+
drop count quoted without that qualifier overstates the finding every time.
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
160
|
+
## Part 5 — Ground truth can itself be wrong
|
|
161
|
+
|
|
162
|
+
The hardest lesson here, and the one worth carrying into every engagement.
|
|
163
|
+
|
|
164
|
+
An early investigation concluded the ingestion was faithful and the counts were
|
|
165
|
+
correct. It was wrong — **it audited the feed against itself, with no external
|
|
166
|
+
reference.** A later reconciliation against the client's own CRM report found
|
|
167
|
+
the feed carried **79%** of the real records, and that gap had been quietly
|
|
168
|
+
distorting every accuracy figure computed on top of it.
|
|
169
|
+
|
|
170
|
+
Consequences, all of which are the normal cost of finding this late:
|
|
171
|
+
|
|
172
|
+
- Every gate figure computed before the reconciliation became **provisional**,
|
|
173
|
+
and the evidence index had to say so at the top rather than in a footnote.
|
|
174
|
+
- A published headline number was marked **void** — not deleted. Deleting it
|
|
175
|
+
would make the record dishonest; superseding it in place makes it useful.
|
|
176
|
+
- One evidence record's conclusion was struck through and replaced, with the
|
|
177
|
+
part that *was* right (a hypothesis it raised) kept and re-labelled.
|
|
178
|
+
|
|
179
|
+
**So: reconcile your inputs against an independent source before you trust any
|
|
180
|
+
metric built on them.** One export from a second path is enough. Do it early —
|
|
181
|
+
it is much cheaper than retracting a number the client has already seen.
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## Part 6 — A green test suite can hide the bug it was written to catch
|
|
186
|
+
|
|
187
|
+
A 144-assertion suite ran green while a data-joining layer contributed **zero**
|
|
188
|
+
records to the calculation for three days. The parity test *applied the missing
|
|
189
|
+
conversion itself* before comparing — so it compared two correct things and
|
|
190
|
+
passed, while production compared two incompatible ones.
|
|
191
|
+
|
|
192
|
+
Worse than inert: one field name happened to match across both shapes, so a
|
|
193
|
+
"history exists" flag was set even though the history was empty — which
|
|
194
|
+
**raised the confidence badge** on exactly the recommendations that had no
|
|
195
|
+
supporting evidence behind them.
|
|
196
|
+
|
|
197
|
+
Defences, in order of value:
|
|
198
|
+
|
|
199
|
+
1. **Test the seam with production's own inputs**, not inputs the test
|
|
200
|
+
pre-normalizes. If the test does work the pipeline doesn't, it isn't testing
|
|
201
|
+
the pipeline.
|
|
202
|
+
2. **Assert on non-zero contribution, not just on equality.** "The join
|
|
203
|
+
produced ≥ N records" catches silent-empty; "the outputs match" doesn't.
|
|
204
|
+
3. **Replay a real historical run end to end** and diff against what production
|
|
205
|
+
actually produced. A replay harness that reproduces a known run exactly
|
|
206
|
+
(e.g. 319/319 records) is what lets you attribute a delta to one cause.
|
|
207
|
+
4. **When a fix changes headline numbers, re-derive every number that depended
|
|
208
|
+
on it** and mark the old ones. Here: drops 111→77 and 178→83, raises 11→36
|
|
209
|
+
and 2→33 — every prior claim about drop volume was void the moment the fix
|
|
210
|
+
landed.
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
|
|
214
|
+
## Part 7 — A backtest is a proxy; the gate is a live window
|
|
215
|
+
|
|
216
|
+
**The formal acceptance in the SOW was a ~10-business-day live window with the
|
|
217
|
+
named client reviewer, not a historical backtest.** The backtest is our internal
|
|
218
|
+
de-risking proxy for it. Never let the two be confused — passing a backtest is
|
|
219
|
+
not passing acceptance, and saying so plainly protects everyone.
|
|
220
|
+
|
|
221
|
+
So the go-live sequence is:
|
|
222
|
+
|
|
223
|
+
1. Backtest to the pinned metric → evidence record.
|
|
224
|
+
2. **Shadow period**: the automation proposes, a named human decides, for a
|
|
225
|
+
defined window. Every build in these repos does this.
|
|
226
|
+
3. Only then does the output get treated as decidable truth.
|
|
227
|
+
|
|
228
|
+
Also, take the win when the client changes the frame. When one client declined
|
|
229
|
+
to annotate historical examples and wanted to use the tool live instead, the
|
|
230
|
+
right response was to **retire the annotation ask and the backtest gate for
|
|
231
|
+
that arm**, reframing it as rule-based flags — not to keep chasing a gate the
|
|
232
|
+
client no longer cared about. The gate serves the engagement, not the reverse.
|
|
233
|
+
|
|
234
|
+
---
|
|
235
|
+
|
|
236
|
+
## Part 8 — Evidence discipline
|
|
237
|
+
|
|
238
|
+
One dated, self-contained ~1-page record per run, in the build repo's
|
|
239
|
+
`evidence/`. Template and conventions:
|
|
240
|
+
[`references/evidence-record-template.md`](references/evidence-record-template.md).
|
|
241
|
+
|
|
242
|
+
The parts that make it work over months:
|
|
243
|
+
|
|
244
|
+
- **An index table at the top of `evidence/README.md`**, newest first, one line
|
|
245
|
+
each with the headline number.
|
|
246
|
+
- **Supersession is loud and in place.** Strike through the wrong conclusion,
|
|
247
|
+
say what replaced it, keep what was right. Never quietly delete.
|
|
248
|
+
- **A standing caveat block** above the index when something invalidates a
|
|
249
|
+
whole class of prior numbers.
|
|
250
|
+
- **Mark internal-only records**, and keep client-facing ones free of raw
|
|
251
|
+
identifiers and personal data — aggregates only.
|
|
252
|
+
- **Point at the reproduce-it-yourself doc** in the build folder rather than
|
|
253
|
+
duplicating steps.
|
|
254
|
+
|
|
255
|
+
Harness rules: [`references/harness-checklist.md`](references/harness-checklist.md).
|
|
256
|
+
In short — every number comes from a committed script with a copy-pasteable
|
|
257
|
+
command line, thresholds live in one config file rather than scattered through
|
|
258
|
+
code, and the raw inputs stay out of git.
|
|
259
|
+
|
|
260
|
+
---
|
|
261
|
+
|
|
262
|
+
## Requirements checklist
|
|
263
|
+
|
|
264
|
+
- [ ] Gate definition read off the **signed** SOW; the number and its wording quoted in the tech spec
|
|
265
|
+
- [ ] Metric pinned; competing readings computed and reported alongside
|
|
266
|
+
- [ ] Trivial/near-zero cases identified and excluded from any tolerance-band metric
|
|
267
|
+
- [ ] Client's idiosyncratic markers understood before interpreting their history
|
|
268
|
+
- [ ] Scoring at pre-decision state; every input filtered to `≤ asOf`; no look-ahead
|
|
269
|
+
- [ ] Censoring and "absence" semantics defined in writing
|
|
270
|
+
- [ ] Contamination exclusions: named, dated, encoded in the harness, control arm shown unchanged, written up
|
|
271
|
+
- [ ] Segmented by decision type **and** by data density
|
|
272
|
+
- [ ] Inputs reconciled against an independent source before trusting any metric
|
|
273
|
+
- [ ] Seam tests use production's own inputs; assertions include non-zero contribution
|
|
274
|
+
- [ ] A real historical run replayed end to end and diffed
|
|
275
|
+
- [ ] Every number traceable to a committed script and a command line
|
|
276
|
+
- [ ] Evidence record written, indexed, and superseded-in-place where relevant
|
|
277
|
+
- [ ] Shadow period run with a named reviewer before go-live
|
|
278
|
+
- [ ] Delivery-log entry linking the evidence record
|
|
279
|
+
|
|
280
|
+
## Known gaps to flag, not silently skip
|
|
281
|
+
|
|
282
|
+
- **A backtest measures agreement with past behaviour, not correctness.** If
|
|
283
|
+
the client's historical decisions were poor, high agreement means we
|
|
284
|
+
reproduced poor decisions. Say which one the gate is actually testing.
|
|
285
|
+
- **Historical data is rarely balanced across arms.** An arm with few events
|
|
286
|
+
and sparse inputs cannot be validated to the same confidence as the main arm.
|
|
287
|
+
Report per-arm sample sizes; a 14.7% on 495 events and a 94.1% on 1,509 are
|
|
288
|
+
not comparable evidence.
|
|
289
|
+
- **Thresholds tuned on a dataset and validated on the same dataset are
|
|
290
|
+
overfit.** Hold out a validation slice, and say whether you did.
|
|
291
|
+
- **The gate is a point-in-time reading.** Feed quality, client behaviour and
|
|
292
|
+
the market all move. Date every figure, and re-run before quoting an old one
|
|
293
|
+
in a new conversation.
|