@whamp/pi-pstack 0.7.0

package/skills/why/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: why
+description: "Use for 'why does X work this way', 'why we picked Y', design rationale, regressions, postmortems, or data-backed thresholds. Discovers available MCPs and queries each evidence category (source control, issue tracker, long-form docs, real-time chat, infrastructure observability, error tracking, product analytics warehouse) in parallel, then returns a cited read on decisions and tradeoffs. Use how for runtime behavior."
+---
+
+# Why
+
+Investigate the motivation and intent behind code.
+
+Companion to the `how` skill. `how` answers what the code does and how it works. `why` answers what forces led to its shape.
+
+## Operating Posture
+
+Operate as a **careful, cautious, and precise investigator**. Be honest about what you know vs what you're inferring. Read `references/epistemics.md` for the full confidence framework and phrasing guide. The synthesizer must follow it.
+
+## Step 1. Understand the Target and the Question
+
+Parse what the user is asking. The **target** is usually a chunk of code, a pattern, a feature, or a named design decision. The **question** is usually a design rationale, a tradeoff, a motivating edge case, an external constraint, dead code, or a broad history sweep.
+
+If the target is vague ("why do we do it this way?" with no clear referent), make your best guess from conversation context (open files, recent edits, cursor location, what was just discussed). State your interpretation briefly so the user can redirect if you're off, then proceed.
+
+## Step 2. Establish the Code Anchor
+
+Before spawning investigators, anchor the investigation in concrete code. You need:
+
+- The relevant file path(s) and line range(s)
+- The key symbols (function names, class names, constants)
+- An initial commit list. The last few commits touching the target.
+- PR numbers from merge commits (pattern `(#1234)` in the subject line)
+
+Build this inline.
+
+```bash
+# Blame target lines for last-touch commits
+git blame -L <start>,<end> <file>
+
+# Full file history, with patches, through renames
+git log --follow -p -- <file>
+
+# Last N commits touching the file, PR numbers visible
+git log --oneline -20 -- <file>
+
+# Extract PR numbers from a commit message
+git log -1 --format=%B <commit>
+```
+
+Pull PR bodies and discussion via `gh` for any substantive commits:
+
+```bash
+gh pr view <number> --json title,body,author,createdAt,mergedAt,labels,closingIssuesReferences,comments,reviews
+```
+
+Capture this as seed context (file paths, symbols, commits, PR numbers, linked ticket IDs). Pass it to the investigators.
+
+## Step 3. Spawn Parallel Investigators (default posture)
+
+**Default to the full parallel investigation.**
+
+### Discovery
+
+Before spawning investigators, the parent lists its available MCP and extension tools. Map each available provider to one evidence category:
+
+1. Source control history
+2. Issue / ticket tracker
+3. Long-form documents
+4. Real-time team chat
+5. Infrastructure observability
+6. Error / exception tracking
+7. Product analytics warehouse
+
+Source control is always available through git and `gh`. For the other six, classify using the MCP name, server instructions, tool names, and resource descriptors. If an MCP could fit more than one category, choose the one matching its primary evidence. Record ambiguous cases in the coverage map.
+
+Aim for a complete **coverage map**, not a minimal one. Document the null, don't skip the search. The parent queries each available MCP and builds one bounded evidence packet per category before launching children. A child does not inherit ambient MCP or extension tools. Use a custom agent for a child-side lookup only when that agent explicitly lists the tool and loads its provider through `extensions` or `subagentOnlyExtensions`.
+
+Launch all matching investigators and the dependent synthesizer with one `subagent({ action: "execute", input: { async: true, maxSubagentSpawnsPerRun: N + 1, workflowScript } })` call. In `workflowScript`, await the investigators with `runs.all([{ key: "investigate-<category>", agent: "worker", task, model }])`, then return `runs.run("synthesize-why", { agent: "worker", task, model })` with their outputs. `N` is the number of evidence categories launched. Don't ask one agent to cover multiple categories.
+
+Each investigator uses:
+- agent: "worker"
+- `model`: `why investigators` (default inherit-parent)
+- `task`: instruct the investigator to inspect only
+
+Each investigator gets:
+1. The base prompt from `references/investigator-prompt.md`
+2. The category playbook `references/sources/<source>.md` as an analysis rubric for the parent's evidence packet, not as child tool instructions
+3. The parent's evidence packet for that category, including null results and gaps
+4. The cross-cutting `references/sources/incident-postmortem.md` **if the target code looks defensive** (null checks, retry logic, timeout handling, rate limiting, feature flags, egress guards, OOM handlers)
+5. The code anchor from Step 2 (file paths, symbols, commit hashes, PR numbers, ticket IDs)
+6. The user's original question
+
+### Investigator roster. One per available evidence category
+
+Spawn one investigator per category with source-control evidence or a matching parent MCP. Each owns exactly one evidence packet.
+
+Each entry names the category and the kind of "why" it uniquely surfaces. Use it to know what to expect back, how to name a gap when a category returns empty, and (only in the rare provably-irrelevant case) to justify a skip.
+
+1. **Source control investigator**. Git history, `gh` for PRs, code comments, tests. Always spawn. The only guaranteed source. Best at surfacing *implementation-time rationale captured during review*.
+
+2. **Issue / ticket tracker investigator** (e.g. Linear, Jira, GitHub Issues, Plane, Shortcut MCP). Best at surfacing *the product or business forcing function*. Strongest when the why is external to engineering.
+
+3. **Long-form documents investigator** (e.g. Notion, Confluence, Google Docs, Coda MCP). Best at surfacing *long-form design rationale*. Where the why is written out before it becomes code.
+
+4. **Real-time team chat investigator** (e.g. Slack, Discord, Microsoft Teams, Mattermost MCP). Best at surfacing *real-time deliberation that never reached a doc*. Especially important when the source control, ticket, and doc paper trail is thin.
+
+5. **Infrastructure observability investigator** (e.g. Datadog, New Relic, Honeycomb, Grafana, Splunk MCP). Infra/runtime view. Best at surfacing *infrastructure and runtime reality that motivated the code*. Strongest when the target reacts to an infra signal (timeouts, retries, rate limits, circuit breakers).
+
+6. **Error / exception tracking investigator** (e.g. Sentry, Rollbar, Bugsnag, Airbrake MCP). Best at surfacing *the specific exceptions and error trajectories that motivated defensive or corrective code*. Strongest for catch blocks, null guards, type checks, retries, and other defenses.
+
+7. **Product analytics warehouse investigator** (e.g. Databricks, Snowflake, BigQuery, ClickHouse, dbt, Redshift MCP). Product/data view. Best at surfacing *product and data reality that shaped the code*. Strongest for flag-gated code, experiment-driven ships, data migrations, and "where did this number come from" questions.
+
+### When to skip an investigator
+
+Only skip with an **explicit, written justification** that goes in the final "Sources Consulted" section. Two valid reasons:
+
+- **No MCP is available for that category** in this environment. Flag this as a gap, not a choice. Example: "Real-time team chat skipped. No matching MCP available, so the conversational record was not searchable."
+- **The source is provably irrelevant**, not just "probably irrelevant." A high bar. Example: "Error / exception tracking skipped. Target is a build-time script with no runtime code path."
+
+If your scope assessment suggests a single-commit trivial target where the PR description already contains the complete answer, you may answer inline **only after** confirming all seven available category searches would be redundant. Say so explicitly. This should be rare.
+
+## Step 4. Synthesize
+
+The same workflow launches `synthesize-why` after every investigator settles. It uses:
+- agent: "worker"
+- `model`: `why synthesizer` (default inherit-parent)
+
+The synthesizer gets:
+1. The investigator findings, including any null results and any categories skipped with justification
+2. The code anchor from Step 2 (file paths, symbols, commit hashes, PR numbers, ticket IDs)
+3. The user's original question
+4. The epistemics framework from `references/epistemics.md`
+5. The synthesizer prompt template from `references/synthesizer-prompt.md`
+
+After the workflow completes, the parent spot-verifies citations with its own MCP and extension tools before presenting the result.
+
+## Step 5. Present
+
+Take the synthesizer's output and present it to the user. You may lightly edit for clarity or add context from the conversation, but **do not rewrite the confidence language**.
+
+## Output Format
+
+The output structure is the one in `references/synthesizer-prompt.md`: The Question, The Code in Question, What We Found, What We Can Reasonably Infer, Competing Hypotheses, What We Don't Know, Sources Consulted, Confidence Summary. Adapt as needed, but keep the confidence separation intact, and keep Sources Consulted as one line per investigator, including the ones that returned nothing or were skipped, with the reason.
+
+After the Sources Consulted block, if the user's `why` question is a precursor to actually changing this code, convert the lineage findings into a Preserve / Change / Avoid / Risk constraint set suitable for planning the change.
+
+## Common Failure Modes to Avoid
+
+- **Recency bias**. Assuming the most recent commit is authoritative. The current shape is often the accretion of many earlier decisions. Trace back.
+
+## Reference Files
+
+- `references/epistemics.md`. Confidence tiers and phrasing guide. The synthesizer must follow it.
+- `references/investigator-prompt.md`. Base prompt template for investigator subagents.
+- `references/source-playbook.md`. Index pointing at the category playbooks below.
+- `references/sources/*.md`. One self-contained example playbook per category, plus cross-cutting `incident-postmortem.md`. Give an investigator the single file that matches its category and adapt it to the available MCP.
+- `references/synthesizer-prompt.md`. Prompt template for the synthesizer subagent, including the output format.

package/skills/why/references/epistemics.md

@@ -0,0 +1,144 @@
+# Epistemics
+
+How to reason about confidence when evidence is historical, fragmentary, and sometimes contradictory, and how to communicate it without flattening it into false certainty.
+
+Code doesn't carry its own motivation. You can read what code does. You can't read *why it exists*. That lives in commits, PRs, tickets, docs, and conversations, all incomplete, biased, and sometimes missing entirely. Pretending otherwise produces confident-sounding guesses that mislead the user.
+
+## Confidence Tiers
+
+Every claim in the final output must sit in one of these tiers. The tier determines which output section the claim goes in and how it's phrased.
+
+### 1. Direct
+
+An explicit, textual citation that answers the question. Not "the code does X so the author must have wanted X." Something an author actually *wrote* that says why.
+
+Examples:
+- A PR description that says "this fixes the bug where users with >1000 items couldn't paginate"
+- A ticket that says "we're adding this because customer Acme requested it in their security review"
+- A code comment that says "// clamp to 100 because the upstream API rejects larger values"
+- A design doc that says "we chose option A over option B because we need persistence across restarts"
+- A chat message from the author saying "switching to this approach since the old one was flaky in tests"
+
+Phrasing: confident, present tense. "This exists because X." Cite the source.
+
+### 2. Supported
+
+Multiple pieces of indirect evidence converge. No single source states it explicitly, but the pattern across sources makes it likely.
+
+Examples:
+- The PR title says "improve performance," the ticket is labeled "perf," and the surrounding commits all touch the same hot path
+- Multiple tests were added alongside the change, all exercising edge cases with very large inputs
+- The author's other PRs from the same week all mention the same incident in their descriptions
+
+Phrasing: confident but clearly derived. "The evidence points strongly to X: [the specific pieces]." Cite multiple sources.
+
+### 3. Inferred
+
+A reasonable reading of the context, but nothing explicitly supports it. The reader should understand this is *your interpretation*, not a fact from the record.
+
+Examples:
+- The PR doesn't say why, but given the error was happening in production (per the incident channel timing) and the fix was rushed (merged the same day), it was likely a hotfix.
+- The function name suggests retry logic. The retry count is 3. This matches the team's general convention of "3 retries" seen elsewhere in the codebase.
+
+Phrasing: hedged. "It appears", "likely", "suggests", "is consistent with", "one reading is". Make the inference chain explicit: "Given A and B, C seems likely because D."
+
+### 4. Speculative
+
+A plausible hypothesis, but the evidence is thin and other explanations fit equally well. Presenting these is valuable, but mark them clearly as guesses.
+
+Examples:
+- "This might be a workaround for a browser bug that's since been fixed, but we found no contemporary evidence of that."
+- "It's possible this threshold was chosen to match an SLA commitment, but no SLA doc references it."
+
+Phrasing: explicitly speculative. "One possibility is X, but we have no direct evidence." Usually lives in the "Competing Hypotheses" section alongside other possibilities.
+
+### 5. Unknown
+
+You looked and couldn't find out. A valid and important outcome. Document it.
+
+Phrasing: "We searched X, Y, and Z and found no evidence of why." Be specific about *what* you searched. "We couldn't find out" is less useful than "we searched the ticket tracker with keywords A and B, scanned the 6 PRs that touched this file since 2023, and grep'd the repo for string literals matching the threshold. None surfaced a rationale."
+
+## Phrasing Guide
+
+### Words that carry confidence. Use carefully
+
+These imply **Direct** or **Supported** confidence. Don't use them for inferences.
+
+- "because". Implies a causal claim with evidence
+- "the reason is". Same
+- "was designed to". Claims author intent
+- "fixes", "addresses", "solves". Claims the change achieved its goal
+- "the team decided". Claims a group decision happened
+
+If you're using these, you should have a citation immediately adjacent.
+
+### Words that hedge. Use for inferences
+
+- "appears to"
+- "seems to"
+- "likely"
+- "suggests"
+- "is consistent with"
+- "one reading is"
+- "plausibly"
+- "may have been"
+- "the evidence points toward"
+
+These signal that you're interpreting, not reporting. Use them liberally in the "What We Can Reasonably Infer" section.
+
+### Words to avoid
+
+- "obviously". If it were obvious, the user wouldn't be asking
+- "clearly". Almost always precedes a claim that isn't clear
+- "of course". Same
+- "just" (as in "it's just X for performance"). Dismissive and usually hides uncertainty
+- "I think" / "I believe". You're synthesizing evidence, not giving a personal opinion. Use "the evidence suggests" instead.
+
+### Avoid rationalization
+
+Code that "makes sense" today may have been written for reasons that no longer apply, or that were wrong when they were written. Don't retrofit a clean rationale onto messy history.
+
+Resist the urge to:
+- Assume the author did the "right" thing and work backward to justify it
+- Assume a consistent pattern across the codebase was intentional when it might be copy-paste
+- Turn an absence of evidence into evidence of absence ("no one mentioned security concerns, so it must not have been a concern")
+
+## The Sycophancy Trap
+
+Users often phrase `why` questions with an embedded hypothesis: "Why do we do it this way, I assume it's for performance?" Don't simply confirm it. Treat it as one candidate among others and check the evidence independently. If the evidence supports it, say so with citations. If not, say so and present what the evidence *does* support.
+
+The user's guess is a prompt for investigation, not a conclusion to validate.
+
+## When Evidence Contradicts
+
+If two sources disagree (the PR description says one thing, the ticket says another), surface both. Don't pick the one that fits a tidier narrative. A typical pattern:
+
+- **The ticket says** "we need this for customer X's compliance requirement"
+- **The PR says** "cleaning up tech debt in this area"
+
+Both may be true (the ticket motivated the work, the PR is the author's framing of it), or one may be wrong. Present both with their citations and let the user make the call.
+
+## When Evidence Is Missing
+
+An honest "we don't know" is one of the most valuable outputs this skill can produce. The user now knows:
+
+- The answer isn't in the obvious places
+- They'll need to ask a human (the original author, the product owner, the team lead) to find out
+- Or they can decide the question isn't worth pursuing further
+
+Failing to mark a gap and filling it with a confident guess actively harms the user. They'll act on the guess.
+
+When you hit a gap, name it concretely:
+- What question you were trying to answer
+- What sources you searched
+- What you searched for in each
+- What you found (nothing, or only tangentially related material)
+
+## Calibration Check Before Finalizing
+
+Before delivering the output, the synthesizer should review every claim in "What We Found" and "What We Can Reasonably Infer" and ask:
+
+1. Does this claim have a citation? If not, either add one or move it to "Inferred" / "Hypotheses".
+2. Is the phrasing calibrated to the tier? (A Direct claim can use "because". An Inferred claim cannot.)
+3. Am I treating the code itself as evidence for its own intent? If so, that's not evidence. Remove or reclassify.
+4. Does the output include a "What We Don't Know" section? If no gaps are mentioned, that's suspicious. Either the evidence was unusually complete or something is being swept under the rug.

package/skills/why/references/investigator-prompt.md

@@ -0,0 +1,103 @@
+# Investigator Prompt Template
+
+Build each investigator's prompt from this template. Fill in the placeholders. Append the single category playbook `sources/<source>.md` matching this investigator's evidence category (see `source-playbook.md` for the index). If the target code looks defensive (null checks, retry logic, timeout handling, rate limiting, feature flags, egress guards, OOM handlers), also append `sources/incident-postmortem.md` for the incident-flavored queries to run inside its own source.
+
+---
+
+You are investigating the historical context and motivation behind a piece of code. A separate synthesizer combines your findings with other investigators' into a final answer, so gather evidence accurately rather than writing prose.
+
+Other investigators search different sources in parallel. Don't try to cover everything. Focus on your assigned source and go deep.
+
+## Operating Posture
+
+Work like a careful, cautious, precise investigator. Don't produce a narrative. Surface evidence and describe it accurately, including the parts that don't fit a tidy story. The more boring and exact your output, the more useful it is. A single verbatim quote with a precise citation beats a paragraph of plausible-sounding summary.
+
+- **Quote, don't paraphrase** when the exact wording matters. Citations should let the reader jump to the source and confirm the claim in seconds.
+- **Go wide before going deep.** Cast a broad first net so you don't miss related context. Only then narrow in.
+- **Track what you searched, not just what you found.** An absence is only useful if the reader knows what was looked for. Record queries verbatim.
+- **Resist the story.** If three pieces of evidence line up neatly and a fourth contradicts them, the contradiction is the most interesting finding. Don't file it away.
+- **Consider the counterfactual.** Before reporting a finding as strong, ask whether you would expect to find it if your current reading were wrong, and how the evidence would differ.
+- **Never invent.** If you're tempted to round a partial finding up into a confident statement, stop and label it partial. The synthesizer is counting on your output being accurate.
+
+## The Question
+
+> {QUESTION}
+
+## The Code Anchor
+
+**Target files:** {FILES_WITH_LINE_RANGES}
+
+**Key symbols:** {SYMBOLS}
+
+**Initial commits touching this code (most recent first):**
+{COMMIT_LIST}
+
+**PR numbers extracted from commit messages:** {PR_NUMBERS}
+
+**Ticket IDs mentioned in commits or PR bodies (if any):** {TICKET_IDS}
+
+## Your Assigned Source
+
+{SOURCE_NAME}
+
+{SOURCE_PLAYBOOK_SECTION}
+
+## Investigation Instructions
+
+Gather **evidence**. Don't answer the question directly. The synthesizer weighs the evidence and forms conclusions. Follow this loop:
+
+1. **Cast a wide net first.** Start broad so you don't miss related context, then narrow in on specific items.
+2. **Read the whole thing.** Read any PR, ticket, doc, or thread fully, not just the title or summary. The key evidence is often buried in a comment, a subtask, or a follow-up.
+3. **Follow links within your assigned source.** If a PR references another PR or commit, pull it. If a ticket links a parent or sibling, pull it. If a doc links another doc, pull it. Stay inside your assigned source. When you spot a cross-source reference, do NOT chase it yourself. Record it under "Additional Leads" so the investigator assigned to that source can pick it up. The one-investigator-per-category design depends on this. Chasing cross-source links duplicates work and confuses scope.
+4. **Capture quotes verbatim** with their location (PR number, ticket ID, URL, commit hash, file:line). The synthesizer needs to cite this precisely.
+5. **Note absences.** If you searched for something and came up empty, that's also a finding. Record what you searched for and what you didn't find.
+6. **Watch for contradictions.** If two items in your source disagree, record both. Don't suppress the inconvenient one.
+
+Don't synthesize or form a final opinion on "the why." Collect the raw material honestly and completely. The synthesizer does the reasoning.
+
+## Epistemic Discipline
+
+- **Don't confuse mechanics with motivation.** A commit changing `limit = 50` to `limit = 100` shows the change, not necessarily why. Look for the explanation in the commit message, PR description, linked ticket, or review comments.
+- **Don't infer intent from code style.** "The author chose a functional approach" is an observation about code, not evidence of intent. Claim intent only when the author stated it.
+- **Preserve uncertainty.** If the evidence is ambiguous, say so. If one reading is more plausible but not certain, say that. Don't collapse ambiguity to look decisive.
+- **No silent substitutions.** If the question is about feature X and you only find evidence about feature Y, don't present Y's evidence as if it answers X.
+
+## Output Format
+
+Return your findings in this structure. The synthesizer will read it directly.
+
+### Source
+Which source you investigated (source control, issue / ticket tracker, long-form documents, real-time team chat, infrastructure observability, error / exception tracking, product analytics warehouse, code comments, etc.).
+
+### What I Searched
+The queries you ran, the items you opened, the places you looked. Be specific. This tells the synthesizer how thorough the investigation was and what might still be unsearched.
+
+### Direct Evidence Found
+For each piece that explicitly addresses the question:
+- **What it says**: verbatim quote or accurate paraphrase
+- **Where it's from**: PR #123, ticket ID, doc URL, chat permalink, commit hash, or file:line
+- **Author and date** (if available)
+- **Relevance**: one sentence on how it bears on the question
+
+### Indirect / Circumstantial Evidence
+Items that don't explicitly answer the question but bear on it. For each:
+- **What it is**: brief description
+- **Where it's from**: location
+- **What it suggests**: what a careful reader might infer, and why. Name the inference chain.
+- **Alternative readings**: if the same evidence could support a different interpretation, note it
+
+### Contradictions
+Two items that disagree with each other, with both citations.
+
+### Gaps
+What you searched for and didn't find. Be specific: "Searched the issue tracker for [query] across [time range]. No matching issues." These absences are valuable data.
+
+### Additional Leads
+Anything that suggests further investigation in a different source. For example, if a PR references a chat thread that wasn't in your source, note it so the real-time team chat investigator or a follow-up pass can pursue it.
+
+## What You're Not Doing
+
+- Writing the final answer. The synthesizer does that.
+- Picking sides in contradictions. Surface them.
+- Speculating beyond what the evidence supports. A hunch with no evidence isn't evidence.
+- Reading the code itself to figure out intent. You may read the code to understand what the target *is*, but don't confuse "what the code does" with "why."

package/skills/why/references/source-playbook.md

@@ -0,0 +1,17 @@
+# Source playbooks
+
+The why skill spawns one investigator per available evidence category, each reading a single source-specific playbook below. The playbooks are concrete examples for common MCPs. Adapt them for a different MCP in the same category.
+
+| Category | Playbook | Example MCP it documents |
+|---|---|---|
+| Source control history | [`code-archaeology.md`](./sources/code-archaeology.md) | git, `gh` |
+| Issue / ticket tracker | [`linear.md`](./sources/linear.md) | Linear (adapt for Jira, GitHub Issues, Plane, Shortcut) |
+| Long-form documents | [`notion.md`](./sources/notion.md) | Notion (adapt for Confluence, Google Docs, Coda) |
+| Real-time team chat | [`slack.md`](./sources/slack.md) | Slack (adapt for Discord, Microsoft Teams, Mattermost) |
+| Infrastructure observability | [`datadog.md`](./sources/datadog.md) | Datadog (adapt for New Relic, Honeycomb, Grafana, Splunk) |
+| Error / exception tracking | [`sentry.md`](./sources/sentry.md) | Sentry (adapt for Rollbar, Bugsnag, Airbrake) |
+| Product analytics warehouse | [`databricks.md`](./sources/databricks.md) | Databricks SQL (adapt for Snowflake, BigQuery, ClickHouse, dbt) |
+
+Cross-cutting:
+
+- [`incident-postmortem.md`](./sources/incident-postmortem.md). Add this if the target code looks defensive (null checks, retry, timeout, rate limit, feature flag, egress guard, OOM handler).

package/skills/why/references/sources/code-archaeology.md

@@ -0,0 +1,88 @@
+# Code Archaeology (git + in-repo)
+
+## What this source contains
+
+- Commit history (messages, dates, authors, diffs)
+- PR descriptions, review comments, and discussion threads (via `gh`)
+- Inline code comments, TODOs, FIXMEs, deprecation notes
+- ADRs (architectural decision records) if the repo keeps them
+- Tests. Names and assertions often encode the edge cases that motivated a change
+- Related files modified in the same commits (co-change signal)
+- CHANGELOG entries, release notes in the repo
+- Issue/ticket IDs mentioned in commit messages and PR bodies
+
+The most trustworthy source, tied directly to the code, and the most complete. Everything that went through the repo should be here.
+
+## How to search it
+
+Expand the seed commit list:
+
+```bash
+# Full history of the file through renames
+git log --follow --oneline -- <file>
+
+# Pickaxe: commits that added or removed this exact text
+git log -S '<exact_string_from_code>' -- <file>
+
+# Or for patterns:
+git log -G '<regex>' -- <file>
+
+# Who wrote each line and when
+git blame -L <start>,<end> <file>
+
+# The full diff of a specific commit
+git show <hash>
+
+# Commits between two points affecting this file
+git log <old>..<new> -p -- <file>
+```
+
+For each substantive commit, pull the PR context:
+
+```bash
+# Find the PR number from the merge commit or branch
+git log -1 --format=%B <hash>
+
+# Full PR context: body, review comments, linked issues
+gh pr view <number> --json title,body,author,createdAt,mergedAt,labels,closingIssuesReferences,comments,reviews,files
+
+# The --json reviews and comments fields are where the real signal is
+```
+
+Look for out-of-band docs:
+
+```bash
+# ADRs often live in docs/adr/ or similar
+rg -l -i 'architecture.decision' --glob '*.md'
+
+# TODOs and FIXMEs near the target
+rg -n -C2 '(TODO|FIXME|HACK|XXX|NOTE)' <target_file>
+
+# Related tests. Names often encode the "why"
+rg -l '<symbol>' --glob '*test*'
+```
+
+## What good evidence looks like here
+
+- A PR description that explains the problem being solved, not just the change ("This fixes the pagination bug that caused X")
+- A long review thread where alternatives were debated
+- An inline comment near the target line that explains a non-obvious constraint
+- A test named `test_handles_edge_case_when_X` that reveals an edge case motivating the code
+- A commit message that references a ticket or incident ID
+- A CHANGELOG entry that summarizes the user-visible rationale
+
+## Common pitfalls
+
+- **Squash-merge flatlands.** If the repo squashes PRs, individual commits in the branch history are lost. Fall back to PR body and comments.
+- **Misleading commit messages.** "Small refactor" sometimes hides an intentional behavior change. Look at the diff, not the message.
+- **Cargo-culted patterns.** The author may have copied a pattern without understanding why. Check if the pattern originated earlier in the codebase and investigate *that* commit.
+- **Bot commits and auto-merges.** Dependabot, Renovate, and automated backports usually don't carry motivation. Skip them when trying to find intent.
+- **Treating code as evidence of intent.** The code itself isn't evidence for why it exists. Evidence comes from commit messages, PRs, comments, tests, docs. Don't cite "the function is named X" as evidence of intent.
+
+## What to return
+
+Every commit/PR/comment that bears on the question, with:
+- The exact text (quoted)
+- The hash / PR number / file:line
+- Author and date
+- Whether it's direct (explicitly addresses the question) or circumstantial

package/skills/why/references/sources/databricks.md

@@ -0,0 +1,70 @@
+# Databricks Analytics & System Tables
+
+## What this source contains
+
+Databricks is the product-analytics, data-pipeline, and warehouse-telemetry layer. It complements Datadog. Datadog is the *infra/runtime* view, Databricks is the *product/data* view (what users did, which experiments ran, how feature usage evolved, where a threshold constant came from).
+
+- **Product analytics events.** `your_warehouse.events.analytics_track_event` (raw) and typed, deduplicated per-event dbt models in `<your_analytics_db>.<schema>.<table>`. User behavior: feature invocations, clicks, accepts/rejects, submissions, client-reported errors.
+- **Usage & billing events.** `your_warehouse.events.usage_event` / `<your_analytics_db>.<schema>.stg_usage_events`, `your_warehouse.events.raw_model_event` / `<your_analytics_db>.<schema>.stg_raw_model_events`. For cost- or volume-driven decisions.
+- **Experiment / feature-flag data.** Exposure and outcome tables. **Schema is company-specific.** Probe with `SHOW TABLES` before assuming names.
+- **System tables.** `system.query.history`, `system.compute.warehouses`, `system.billing.*`, `system.access.audit`. Answer "was this query expensive?", "how often did anyone run this?", "when did warehouse load spike?"
+- **dbt lineage.** Models in `<your_analytics_db>.<schema>` reveal what pipelines depend on a table/field. Upstream changes frequently motivate consumer-code changes.
+- **Databricks notebooks.** Exploratory analyses engineers wrote before code changes. **Not queryable via the SQL MCP.** If you suspect the rationale lives in a notebook, name it as a gap.
+
+## How to search it
+
+Use the Databricks SQL MCP. Primary tool: `execute_sql_read_only`. If it returns a `statement_id`, poll with `poll_sql_result` rather than re-running.
+
+**Orient before querying.** Schemas are company-specific. Probe before trusting a table name:
+
+```sql
+SHOW TABLES IN <your_analytics_db>.<schema> LIKE '*<keyword>*';
+DESCRIBE TABLE <your_analytics_db>.<schema>.stg_<event>;
+```
+
+**Time-bound every query.** These tables are huge and unconstrained scans time out. Filter on `_timestamp` (events) or `start_time` (`system.query.history`) with a window bracketing the ship date, typically ~30 days before and after, wider only for strong reason.
+
+**Prefer typed dbt models over the raw table.** `<your_analytics_db>.<schema>.<table>` is deduplicated, typed, and liquid-clustered. `your_warehouse.events.analytics_track_event` has duplicates and untyped `properties_json`. Model-name pattern: `stg_<source>_<event_name_with_underscores>`, where `<source>` is `app`, `backend`, `website`, or `cli`. Confirm the exact model name with `SHOW TABLES` when the pattern alone doesn't resolve it. Drop to the raw table only when there's no dbt model yet, or you need events from inside the dbt refresh lag.
+
+**Column conventions on the typed dbt models** (knowing these avoids a `DESCRIBE` round-trip):
+
+- `_timestamp`, `_id`, `_auth_id`, `_request_id`, `event_name`. Standard on every model
+- `properties_<name>`. Typed, underscore-cased event properties (`properties_entrypoint`, `properties_size_bytes`, …)
+- `context_team_id`, `context_client_version`, `context_country`, `context_client_os`. Pre-extracted client context
+
+### Investigation patterns that tend to pay off
+
+Pick the table + column combination that matches the target:
+
+1. **Event usage trajectory.** Daily counts on the relevant `stg_*` model across a ±30d window around the PR merge. A step function from zero to steady volume within a day or two of the merge is strong circumstantial evidence the PR launched the feature. A decay to zero suggests a deprecation or deletion.
+2. **Guard-rail / defensive-check origin.** Distribution (median / p99 / max) of the relevant `properties_<name>` column in the 14 days *before* the PR. A p99 that matches the target's threshold constant suggests the number was chosen from data.
+3. **Experiment / feature-flag lookup.** `SHOW TABLES ... LIKE '*experiment*'` to find the exposure table, then pull exposure counts by variant for the relevant flag key near the PR date.
+4. **Query-history evidence for migrations, backfills, or perf rewrites.** `system.query.history` filtered by `statement_text ILIKE '%<table_or_symbol>%'` with a tight `start_time` window surfaces the expensive queries that likely motivated the change (sort by `total_duration_ms` or aggregate `SUM(read_bytes)`, `COUNT(*)`).
+5. **dbt lineage.** If the target reads from or writes into a `<your_analytics_db>.<schema>` model, the model's own git history (in this repo) often carries the rationale. Hand that lead back to the git investigator rather than chasing it yourself.
+
+## What good evidence looks like here
+
+Beyond the pattern shapes above:
+
+- An error-classifying event's count drops to near zero in the days after a defensive-code PR. Suggests the PR resolved that error class
+- An exposure table row names the target's feature-flag key with a "shipped" / "concluded" decision around the PR ship date
+
+## Common pitfalls
+
+- **Instrumented ≠ caused.** An event's existence means someone cared enough to log it, not that the target code exists *because* of it. Pair with a PR/commit citation from the git investigator before claiming causation.
+- **Silent instrumentation changes.** A step function in event volume may mean a new event started being logged, not that user behavior changed. Check for instrumentation PRs in the same window before reading the ramp as a feature-launch signal.
+- **Schema drift.** Event properties evolve. A column on the typed dbt model today may not have existed when the target was written. Older data may carry the property only inside raw `properties_json`.
+- **dbt refresh lag.** `<your_analytics_db>.<schema>.*` is rebuilt on a schedule (often hourly/daily). For events from the last few hours, fall back to `your_warehouse.events.*` and deduplicate by `_id`.
+- **Company-specific tables.** Experiment, feature-flag, billing, and usage tables vary. Reporting a result from a table whose existence you never confirmed is a classic failure mode. Probe with `SHOW TABLES` / `DESCRIBE TABLE` first.
+- **Retention cliff.** If the relevant window predates the table's retention or the dbt model's creation date, that's a *gap*, not a null result. Name it explicitly so the synthesizer doesn't read "no results" as "no activity."
+- **Notebooks aren't queryable.** The SQL MCP can't see Databricks notebooks. If you suspect the rationale lives in one, return a gap.
+
+## What to return
+
+For each relevant finding:
+- Type (product event / experiment exposure / usage or billing event / system-table row / dbt model)
+- Fully-qualified table name and the exact query you ran
+- Time window queried
+- Compact numeric summary (counts, percentiles, first/last-seen timestamps). **Don't dump raw rows.**
+- Temporal correlation with the target's ship date (e.g., "first row 2024-08-15, PR #49074 merged 2024-08-14")
+- Relevance + strength: direct / circumstantial / weak