kushi-agents 5.0.4 → 5.2.0

package/plugin/instructions/otel.instructions.md

@@ -0,0 +1,75 @@
+# OpenTelemetry export
+
+> Doctrine: `otel.instructions.md` — kushi v5.2.0+
+
+## Purpose
+
+Opt-in observability for kushi pipelines via OpenTelemetry Protocol (OTLP). Enables enterprise teams to trace kushi runs in their existing observability stack (Jaeger, Grafana Tempo, Azure Monitor, Datadog, etc.).
+
+## Opt-in activation
+
+Set the environment variable:
+
+```
+KUSHI_OTEL_ENDPOINT=http://localhost:4318/v1/traces
+```
+
+When unset or empty, all OTel helpers are **no-ops** with zero overhead (no HTTP calls, no object allocation beyond the guard check).
+
+## Events emitted
+
+| Span name | Fires during | Attributes |
+|-----------|-------------|------------|
+| `kushi.pull` | Each `pull-*` source | `project`, `alias`, `source`, `duration_ms`, `success`, `evidence_count` |
+| `kushi.build_state` | `build-state` | `project`, `alias`, `pages_written`, `contradictions_flagged`, `duration_ms`, `success` |
+| `kushi.lint` | `lint-state` | `project`, `alias`, `findings_count`, `duration_ms`, `success` |
+| `kushi.contradiction.flagged` | Inside `build-state` when a new contradiction is detected | `project`, `alias`, `entity`, `field`, `source` |
+| `kushi.hook.invoked` | Each hook invocation | `project`, `alias`, `event`, `hook_type`, `target`, `duration_ms`, `success` |
+
+## Wire format
+
+Uses the [OTLP/HTTP JSON](https://opentelemetry.io/docs/specs/otlp/#otlphttp) format:
+- Endpoint: `$KUSHI_OTEL_ENDPOINT` (must include path, e.g. `/v1/traces`)
+- Method: POST
+- Content-Type: `application/json`
+- No external dependencies — pure PowerShell via `Invoke-RestMethod`
+
+## Privacy contract
+
+**CRITICAL**: OTel spans carry ONLY:
+- Operation metadata (names, counts, durations, success/failure)
+- Project/alias identifiers
+
+OTel spans NEVER carry:
+- Evidence content (email bodies, meeting transcripts, etc.)
+- PII (user emails, file paths with usernames beyond alias)
+- Authentication tokens or secrets
+
+This is enforced structurally: `Emit-OtelSpan.ps1` accepts only a fixed set of attribute keys.
+
+## Implementation
+
+The shared helper `plugin/skills/_shared/Emit-OtelSpan.ps1`:
+1. Checks `$env:KUSHI_OTEL_ENDPOINT` — returns immediately if unset/empty.
+2. Constructs a minimal OTLP JSON payload with a single span.
+3. POSTs via `Invoke-RestMethod` with a 5s timeout.
+4. On failure: logs a warning to stderr, never throws, never blocks the pipeline.
+
+## Service identity
+
+```json
+{
+  "resource": {
+    "attributes": [
+      { "key": "service.name", "value": { "stringValue": "kushi" } },
+      { "key": "service.version", "value": { "stringValue": "<package.json version>" } }
+    ]
+  }
+}
+```
+
+## References
+
+- `Emit-OtelSpan.ps1` — shared helper (`plugin/skills/_shared/Emit-OtelSpan.ps1`)
+- `hooks.instructions.md` — hook invocations emit `kushi.hook.invoked` spans
+- `parallel-execution.instructions.md` — parallel pulls emit one span per worker

package/plugin/instructions/parallel-execution.instructions.md

@@ -0,0 +1,81 @@
+# Parallel execution
+
+> Doctrine: `parallel-execution.instructions.md` — kushi v5.2.0+
+
+## Purpose
+
+Defines when and how kushi parallelizes work. The primary use case is parallel `pull-*` dispatch during `refresh-project`, but the patterns apply to any future parallel workload.
+
+## When to parallelize
+
+| Scenario | Parallel? | Why |
+|----------|-----------|-----|
+| Multiple `pull-*` in a refresh | ✅ Yes | Sources are independent; each writes to its own `Evidence/<alias>/<source>/` subtree |
+| `build-state` page writes | ❌ No | Pages may reference each other; deterministic ordering required |
+| `lint-state` finding checks | ✅ Yes (future) | Read-only checks are embarrassingly parallel |
+| Hook dispatch | ❌ No | Sequential for predictable log ordering |
+
+## Configuration
+
+In `.kushi/config.yml` (project-level) or `~/.kushi/config.yml` (user-level):
+
+```yaml
+parallel:
+  max_workers: 4        # default; 1 = sequential
+  throttle_ms: 200      # minimum delay between worker starts (rate-limit courtesy)
+```
+
+Override at runtime: `kushi refresh <project> --sequential` forces `max_workers: 1`.
+
+## Throttling considerations
+
+M365 Graph (via WorkIQ) enforces per-user rate limits:
+- **429 Too Many Requests** — back off exponentially.
+- Kushi's `throttle_ms` provides a minimum inter-start delay (not a per-request throttle — individual pull-* skills handle their own retries via `auth-and-retry.instructions.md`).
+- Default 200ms is conservative; increase for tenants with aggressive throttling.
+
+## Error aggregation
+
+- Each worker runs independently. A failure in `pull-email` does NOT halt `pull-teams`.
+- Worker results are collected into a `Map<source, {status, items, duration, error?}>`.
+- After all workers complete, results are sorted in **canonical source order** (same order as sequential dispatch per `refresh-project/SKILL.md` Step 2).
+- The run summary and run-log entries are written in canonical order regardless of completion order.
+
+## Deterministic output ordering
+
+Despite parallel execution, all observable outputs (run-log entries, refresh report tables, State/log.md entries) are written in **canonical source order**:
+
+1. pull-email
+2. pull-teams
+3. pull-meetings
+4. pull-onenote
+5. pull-loop
+6. pull-sharepoint
+7. pull-crm
+8. pull-ado
+9. pull-misc
+
+This ensures `git diff` is stable across runs and self-check D38 passes.
+
+## Implementation pattern (PowerShell)
+
+```powershell
+# Worker pool using Start-ThreadJob (PowerShell 7+)
+$maxWorkers = $config.parallel.max_workers ?? 4
+$throttleMs = $config.parallel.throttle_ms ?? 200
+$jobs = @()
+foreach ($source in $enabledSources) {
+    while (($jobs | Where-Object { $_.State -eq 'Running' }).Count -ge $maxWorkers) {
+        Start-Sleep -Milliseconds 100
+    }
+    $jobs += Start-ThreadJob -ScriptBlock { param($s) & pull-$s ... } -ArgumentList $source
+    Start-Sleep -Milliseconds $throttleMs
+}
+$results = $jobs | Receive-Job -Wait -AutoRemoveJob
+```
+
+## References
+
+- `refresh-project/SKILL.md` — orchestrator that dispatches parallel pulls
+- `auth-and-retry.instructions.md` — per-request throttling within each worker
+- `hooks.instructions.md` — hooks fire AFTER parallel aggregation completes

package/plugin/instructions/schema-evolve.instructions.md

@@ -0,0 +1,73 @@
+# Schema evolve
+
+> Doctrine: `schema-evolve.instructions.md` — kushi v5.2.0+
+
+## Purpose
+
+Allows users to teach kushi project-specific conventions that persist across runs. When a user says "from now on always do X for this project," kushi captures the rule and applies it in all future operations.
+
+## Storage location
+
+Rules are stored in `Evidence/<alias>/State/CLAUDE.md` under the project's evidence tree.
+
+**Decision rationale**: `CLAUDE.md` is the Karpathy-pattern file for agent-specific conventions (already present in the State layout since v5.0.0). Placing rules here means:
+- They're versioned alongside evidence (git-tracked engagement roots get history).
+- They're co-located with the State pages that apply them.
+- They're readable by any agent (not kushi-specific format).
+
+## Rule format
+
+Rules are appended to `CLAUDE.md` with structure:
+
+```markdown
+## Rule: <short-title>
+
+- **Added**: 2026-05-29T14:30:00Z
+- **Scope**: project | alias | global
+- **Source**: user (natural language) | schema-evolve skill
+
+<rule text as stated by the user>
+```
+
+## Scope levels
+
+| Scope | Stored at | Applies to |
+|-------|-----------|------------|
+| `project` | `Evidence/<alias>/State/CLAUDE.md` | All operations on this project |
+| `alias` | `Evidence/<alias>/State/CLAUDE.md` | Only this contributor's operations |
+| `global` | `~/.kushi/conventions.md` | All projects (future — v5.3.0) |
+
+v5.2.0 implements `project` scope only. `alias` and `global` are documented for forward compatibility.
+
+## How rules are applied
+
+1. At the start of `build-state`, `ask-project`, and `refresh-project`, the skill reads `Evidence/<alias>/State/CLAUDE.md`.
+2. Rules are parsed into a list and included as context for the operation.
+3. Rules affect preferences (formatting, naming, emphasis) but NEVER override hard doctrine (WorkIQ-only, verbatim-by-default, CSC format).
+
+## Conflict resolution
+
+- Hard doctrine always wins over user rules.
+- Later rules override earlier rules on the same topic.
+- If a rule contradicts doctrine, it's logged as a warning but not applied.
+
+## CLI verb
+
+`kushi remember <rule>` — captures a rule at project scope.
+
+Auto-detection: when the user says "from now on...", "always...", "never...", "for this project..." in `ask-project`, the skill offers to persist it as a convention.
+
+## Examples
+
+```
+> kushi remember "always use 'HCA' not 'Healthcare Accelerator' in summaries"
+> kushi remember "treat John Smith as the primary EM for this project"
+> kushi remember "CRM entity 'opportunity' maps to our internal term 'deal'"
+```
+
+## References
+
+- `karpathy-state-layout.instructions.md` — CLAUDE.md file in State layout
+- `living-wiki.instructions.md` — rules interact with incremental State maintenance
+- `build-state` skill — reads rules at start of run
+- `ask-project` skill — reads rules + offers to capture new ones

package/plugin/instructions/wiki-lint.instructions.md

@@ -0,0 +1,110 @@
+---
+name: "wiki-lint"
+description: "v5.1.0 — Wiki lint finding classes for State/. Detects contradictions, stale claims, orphan pages, missing cross-refs, and data gaps. Each finding includes a ready-to-paste fix snippet. Adapted from sametbrr/llm-wiki-manager lint patterns. Authored to agentskills.io spec; deltas: kushi-specific finding classes (contradiction-flagged, stale-claim, orphan-page, missing-cross-ref, data-gap), fix-snippet convention."
+applies_to: "lint-state skill"
+since: "kushi v5.1.0"
+---
+
+# wiki-lint — doctrine
+
+> **Authored to [agentskills.io](https://agentskills.io/skill-creation/best-practices) spec.**
+> Deltas from source (sametbrr/llm-wiki-manager): kushi-specific finding classes mapped to State/ layout, Obsidian-callout aware, fix-snippet per finding (not just a diagnostic), integration with `_review-queue.md`.
+
+Adapted from [sametbrr/llm-wiki-manager](https://github.com/sametbrr/llm-wiki-manager) — wiki lint patterns.
+
+## Finding classes
+
+### 1. `contradiction-flagged`
+
+**What:** Count unresolved `> [!warning] Contradicted` callouts in State/ pages.
+
+**Detection:** Regex `^> \[!warning\] Contradicted` in any `State/**/*.md`.
+
+**Severity:** warning (informational — contradictions are expected during active projects).
+
+**Fix snippet:**
+```markdown
+<!-- To resolve: review both values, pick the correct one, then either:
+     (a) Delete the callout pair and keep the winning value, OR
+     (b) Add <!-- kushi:human-override --> above the old value to prevent auto-resolve. -->
+```
+
+### 2. `stale-claim`
+
+**What:** A claim in a State page is ≥ 60 days old (based on its `[source: ... · YYYY-MM-DD]` citation date) with no corroborating refresh since.
+
+**Detection:** Parse inline citations `[source: ... · YYYY-MM-DD]`; flag if the newest citation for an entity property is > 60 days old AND no `build-state` or `refresh` log entry touched that entity since.
+
+**Severity:** warning.
+
+**Fix snippet:**
+```markdown
+<!-- Stale claim (>60 days). Run '@Kushi refresh <project>' to pull fresh evidence,
+     then '@Kushi state <project>' to rebuild. If the claim is still valid, add a
+     corroborating citation from a recent source. -->
+```
+
+### 3. `orphan-page`
+
+**What:** A page in `State/` (with `kushi_state_page: true`) that is NOT linked from `index.md` or any other State page.
+
+**Detection:** Enumerate all `State/**/*.md` with `kushi_state_page: true`. For each, check if its path or `[[wikilink]]` slug appears in `index.md` or any sibling page body.
+
+**Severity:** warning.
+
+**Fix snippet:**
+```markdown
+<!-- Orphan page: not linked from index.md or any other State page.
+     Fix: add a [[category/slug]] link to index.md under the correct category heading,
+     OR delete this page if it was generated in error. -->
+```
+
+### 4. `missing-cross-ref`
+
+**What:** An entity mentioned in a page body (matching a known entity name from `Evidence/_graph/project-graph.json` nodes) but NOT listed in the page's `entity_ids` or `related` frontmatter arrays.
+
+**Detection:** Load entity names from graph nodes. For each State page, scan body for entity-name matches not in frontmatter.
+
+**Severity:** info.
+
+**Fix snippet:**
+```markdown
+<!-- Missing cross-ref: entity "<name>" mentioned in body but not in frontmatter.
+     Fix: add to `related:` array in front-matter:
+     related:
+       - category/entity-slug -->
+```
+
+### 5. `data-gap`
+
+**What:** A required section header is present in a State page but its body (content between that header and the next header or EOF) is empty or contains only whitespace/placeholder text.
+
+**Detection:** For each State page, parse `###` and `##` sections. Flag sections whose body is empty, contains only `<!-- TODO -->`, or is fewer than 2 non-blank lines.
+
+**Severity:** info.
+
+**Fix snippet:**
+```markdown
+<!-- Data gap: section "<heading>" has no content.
+     Fix: run '@Kushi refresh <project>' to pull evidence that populates this section,
+     or remove the section header if it's not applicable to this entity. -->
+```
+
+## Output format
+
+Each finding in the lint report:
+
+```markdown
+### <class>: <entity/page> — <short description>
+
+**File:** `State/<path>`
+**Line:** <N>
+**Fix:**
+<fix snippet>
+```
+
+## References
+
+- [sametbrr/llm-wiki-manager](https://github.com/sametbrr/llm-wiki-manager)
+- `living-wiki.instructions.md` (contradiction lifecycle)
+- `karpathy-state-layout.instructions.md` (page convention)

package/plugin/skills/_shared/Append-StateLog.ps1

@@ -0,0 +1,73 @@
+<#
+.SYNOPSIS
+Appends a canonical entry to State/log.md per log-format.instructions.md.
+
+.DESCRIPTION
+Idempotent helper called by every writer skill after completing a State/ write.
+Prepends (reverse-chronological) a new entry after the file header.
+
+.PARAMETER StateDir
+Path to the State/ directory (e.g., Evidence/<alias>/State/).
+
+.PARAMETER Op
+Operation name from the closed taxonomy: bootstrap, refresh, build-state, lint-state,
+ask-fileback, link-entities, dashboard, tour.
+
+.PARAMETER Title
+Short one-line title for the log entry.
+
+.PARAMETER Summary
+1–3 line summary body.
+
+.PARAMETER Sources
+Comma-separated source pointers.
+#>
+[CmdletBinding()]
+param(
+    [Parameter(Mandatory)][string]$StateDir,
+    [Parameter(Mandatory)][ValidateSet('bootstrap','refresh','build-state','lint-state','ask-fileback','link-entities','dashboard','tour')][string]$Op,
+    [Parameter(Mandatory)][string]$Title,
+    [string]$Summary = '',
+    [string]$Sources = ''
+)
+
+$ErrorActionPreference = 'Stop'
+
+$logFile = Join-Path $StateDir 'log.md'
+$timestamp = Get-Date -Format 'yyyy-MM-dd HH:mm'
+$entry = "## [$timestamp] $Op | $Title`n"
+if ($Summary) { $entry += "`n$Summary" }
+if ($Sources) { $entry += "`nSources: $Sources" }
+$entry += "`n"
+
+if (-not (Test-Path $logFile)) {
+    # Create with header
+    $header = @"
+---
+kushi_state_log: true
+---
+
+# State Log
+
+$entry
+"@
+    New-Item -Path $logFile -ItemType File -Force | Out-Null
+    Set-Content -Path $logFile -Value $header -Encoding utf8NoBOM
+} else {
+    $content = Get-Content -Raw $logFile
+    # Insert after the header block (after "# State Log" line or after front-matter)
+    if ($content -match '(?ms)(^---\r?\n.*?\r?\n---\r?\n\r?\n# [^\r\n]+\r?\n)(.*)$') {
+        $headerPart = $Matches[1]
+        $bodyPart = $Matches[2]
+        $newContent = $headerPart + "`n" + $entry + "`n" + $bodyPart
+    } elseif ($content -match '(?ms)(^# [^\r\n]+\r?\n)(.*)$') {
+        $headerPart = $Matches[1]
+        $bodyPart = $Matches[2]
+        $newContent = $headerPart + "`n" + $entry + "`n" + $bodyPart
+    } else {
+        $newContent = $entry + "`n" + $content
+    }
+    Set-Content -Path $logFile -Value $newContent -Encoding utf8NoBOM
+}
+
+Write-Verbose "Appended log entry: [$timestamp] $Op | $Title"

package/plugin/skills/_shared/Emit-OtelSpan.ps1

@@ -0,0 +1,111 @@
+<#
+.SYNOPSIS
+Emits an OpenTelemetry span via OTLP/HTTP JSON.
+
+.DESCRIPTION
+Opt-in OTel export. If KUSHI_OTEL_ENDPOINT is unset or empty, this is a
+complete no-op (no HTTP calls, no allocations). Pure PowerShell — no
+external dependencies.
+
+.PARAMETER SpanName
+The span name (e.g., 'kushi.pull', 'kushi.build_state').
+
+.PARAMETER Attributes
+Hashtable of span attributes. Only metadata — never evidence content.
+
+.PARAMETER DurationMs
+Duration in milliseconds (optional — computed from Start if provided).
+
+.PARAMETER Start
+Optional start time [datetime]. Defaults to (now - DurationMs).
+#>
+[CmdletBinding()]
+param(
+    [Parameter(Mandatory)][string]$SpanName,
+    [hashtable]$Attributes = @{},
+    [int]$DurationMs = 0,
+    [datetime]$Start = [datetime]::MinValue
+)
+
+# No-op guard — zero overhead when endpoint is unset
+$endpoint = $env:KUSHI_OTEL_ENDPOINT
+if (-not $endpoint) { return }
+
+$ErrorActionPreference = 'Continue'
+
+try {
+    # Compute timestamps (nanoseconds since epoch)
+    $now = [DateTimeOffset]::UtcNow
+    if ($Start -eq [datetime]::MinValue) {
+        $startOffset = $now.AddMilliseconds(-$DurationMs)
+    } else {
+        $startOffset = [DateTimeOffset]::new($Start.ToUniversalTime())
+    }
+    $endOffset = $now
+
+    $startNs = ($startOffset.ToUnixTimeMilliseconds() * 1000000).ToString()
+    $endNs = ($endOffset.ToUnixTimeMilliseconds() * 1000000).ToString()
+
+    # Generate trace/span IDs
+    $traceId = [System.Guid]::NewGuid().ToString('N')
+    $spanId = [System.Guid]::NewGuid().ToString('N').Substring(0, 16)
+
+    # Build attributes array
+    $attrArray = @()
+    # Allowed keys only (privacy contract)
+    $allowedKeys = @('project','alias','source','duration_ms','success','evidence_count',
+                     'pages_written','contradictions_flagged','findings_count','entity',
+                     'field','event','hook_type','target','report_path','items_pulled')
+    foreach ($key in $Attributes.Keys) {
+        if ($key -notin $allowedKeys) { continue }
+        $val = $Attributes[$key]
+        $valObj = if ($val -is [bool]) {
+            @{ boolValue = $val }
+        } elseif ($val -is [int] -or $val -is [long] -or $val -is [double]) {
+            @{ intValue = [string]$val }
+        } else {
+            @{ stringValue = [string]$val }
+        }
+        $attrArray += @{ key = $key; value = $valObj }
+    }
+
+    # Get service version
+    $version = '5.2.0'
+    $pkgJson = Join-Path $PSScriptRoot '..\..\..\..\package.json'
+    if (Test-Path $pkgJson) {
+        try { $version = (Get-Content -Raw $pkgJson | ConvertFrom-Json).version } catch {}
+    }
+
+    # Build OTLP payload
+    $payload = @{
+        resourceSpans = @(@{
+            resource = @{
+                attributes = @(
+                    @{ key = 'service.name'; value = @{ stringValue = 'kushi' } }
+                    @{ key = 'service.version'; value = @{ stringValue = $version } }
+                )
+            }
+            scopeSpans = @(@{
+                scope = @{ name = 'kushi'; version = $version }
+                spans = @(@{
+                    traceId = $traceId
+                    spanId = $spanId
+                    name = $SpanName
+                    kind = 1  # SPAN_KIND_INTERNAL
+                    startTimeUnixNano = $startNs
+                    endTimeUnixNano = $endNs
+                    attributes = $attrArray
+                    status = @{
+                        code = if ($Attributes['success'] -eq $false) { 2 } else { 1 }
+                    }
+                })
+            })
+        })
+    }
+
+    $json = $payload | ConvertTo-Json -Depth 20 -Compress
+    Invoke-RestMethod -Uri $endpoint -Method POST -Body $json `
+        -ContentType 'application/json' -TimeoutSec 5 -ErrorAction Stop | Out-Null
+} catch {
+    Write-Warning "OTel export failed (non-blocking): $($_.Exception.Message)"
+}