@deftai/directive-content 0.55.0

package/tools/telemetry.md

@@ -0,0 +1,285 @@
+# Telemetry Standards
+
+Logging, tracing, metrics, and error tracking for production systems.
+
+Legend (from RFC2119): !=MUST, ~=SHOULD, ≉=SHOULD NOT, ⊗=MUST NOT, ?=MAY.
+
+**Scope:** Observability, monitoring, and debugging across all modes.
+
+**⚠️ See also**: [main.md](../../main.md) | [coding.md](../coding/coding.md)
+
+## Mode-Specific Requirements
+
+### fun Mode (Learning/Prototyping)
+
+**Logging:**
+- ? Use print() or basic logging
+- ? Structured logging optional
+
+**Error Tracking:**
+- ? Optional
+
+**Tracing:**
+- ? Optional
+
+**Metrics:**
+- ? Optional
+
+### fast Mode (Development - Default)
+
+**Logging:**
+- ~ Structured logging (structlog, loguru, zerolog)
+- ~ Log levels: DEBUG, INFO, WARN, ERROR
+- ~ Include context: timestamp, level, message, request_id
+
+**Error Tracking:**
+- ~ Use Sentry.io or equivalent
+- ~ Sample rate: 10-50% in dev, 100% errors
+- ~ Capture stack traces
+
+**Tracing:**
+- ? Optional for debugging
+- ~ Use for complex workflows
+
+**Metrics:**
+- ~ Basic metrics (request count, latency)
+- ? Custom business metrics
+
+### pro Mode (Production)
+
+**Logging:**
+- ! Structured logging mandatory
+- ! Log levels: INFO, WARN, ERROR (⊗ DEBUG in production)
+- ! Include: timestamp, level, message, trace_id, span_id, user_id, request_id
+- ! Log rotation and retention policy
+
+**Error Tracking:**
+- ! Sentry.io or equivalent mandatory
+- ! Sample rate: 100% errors, 10-20% transactions
+- ! Release tracking and source maps
+- ! User feedback integration
+
+**Tracing:**
+- ! Distributed tracing (OpenTelemetry, logfire, Sentry)
+- ! Instrument critical paths
+- ! Trace sampling: 1-10% in production
+- ! Include: service name, operation name, duration, tags
+
+**Metrics:**
+- ! RED metrics (Rate, Error, Duration)
+- ! System metrics (CPU, memory, disk)
+- ! Business metrics (orders, users, revenue)
+- ! Alerting on SLOs/SLIs
+
+## Recommended Tools
+
+**Structured Logging:**
+- Python: logfire (pro), structlog (fast/pro), loguru (fast)
+- Go: zerolog, zap
+- TypeScript: pino, winston
+- C++: spdlog
+
+**Error Tracking:**
+- Sentry.io (all languages, all modes)
+- Rollbar, Bugsnag (alternatives)
+
+**Tracing:**
+- OpenTelemetry (all languages, pro mode)
+- logfire (Python, pro mode)
+- Sentry (all languages, fast/pro mode)
+
+**Metrics:**
+- Prometheus (self-hosted)
+- Datadog, New Relic (SaaS)
+- OpenTelemetry metrics
+
+## Logging Best Practices
+
+**Structure:**
+- ! JSON format for machine parsing
+- ! Human-readable format for development
+- ! Consistent field names across services
+
+**Content:**
+- ! Include context: timestamp, level, logger name, message
+- ~ Include identifiers: user_id, request_id, trace_id, span_id
+- ~ Include metadata: environment, version, host
+- ⊗ Log sensitive data (PII, secrets, passwords)
+
+**Levels:**
+- **DEBUG**: Detailed diagnostic information (fun/fast only)
+- **INFO**: General informational messages
+- **WARN**: Warning messages, potentially harmful situations
+- **ERROR**: Error events, but application continues
+- **FATAL**: Severe errors causing shutdown
+
+**Examples:**
+```python
+# Good
+log.info("order_created", order_id=order.id, user_id=user.id, total=order.total)
+
+# Bad
+log.info(f"Order {order.id} created by {user.email} for ${order.total}")  # Not structured
+log.debug(f"Password: {password}")  # Sensitive data
+```
+
+## Tracing Best Practices
+
+**Span Naming:**
+- ! Use verb + noun: `process_order`, `validate_user`, `fetch_data`
+- ! Include operation type: `http.request`, `db.query`, `cache.get`
+- ⊗ Generic names: `process`, `handle`, `do_work`
+
+**Span Attributes:**
+- ~ Add business context: order_id, user_id, amount
+- ~ Add technical context: http.method, http.status_code, db.statement
+- ~ Add errors: error=true, error.message, error.type
+- ⊗ Add sensitive data
+
+**Sampling:**
+- ! Sample strategically to control costs
+- ~ 100% errors, 1-10% success in production
+- ~ Higher rate for critical paths
+- ? Adaptive sampling based on latency
+
+## Metrics Best Practices
+
+**RED Metrics (Mandatory in pro):**
+- **Rate**: Requests per second
+- **Error**: Error rate (%)
+- **Duration**: Latency (p50, p95, p99)
+
+**Naming:**
+- ! Use descriptive names: `http_requests_total`, `order_processing_duration_seconds`
+- ! Include units: `_seconds`, `_bytes`, `_total`
+- ! Use labels for dimensions: method, status, endpoint
+
+**Types:**
+- **Counter**: Monotonically increasing (requests, errors)
+- **Gauge**: Current value (memory, queue depth)
+- **Histogram**: Distribution (latency, size)
+- **Summary**: Similar to histogram (p50, p95, p99)
+
+## Error Tracking
+
+**What to Track:**
+- ! All unhandled exceptions
+- ! Explicit error captures (validation failures, business logic errors)
+- ~ Performance issues (slow queries, timeouts)
+- ~ User feedback (bug reports, feature requests)
+
+**Context:**
+- ! Stack trace
+- ! Request data (sanitized)
+- ! User context (ID, not PII)
+- ! Environment (version, host, OS)
+- ~ Breadcrumbs (previous events)
+
+**Sentry Configuration:**
+```python
+import sentry_sdk
+
+sentry_sdk.init(
+    dsn="https://...@sentry.io/...",
+    environment="production",
+    release="my-app@1.2.3",
+    traces_sample_rate=0.1,
+    profiles_sample_rate=0.1,
+    before_send=sanitize_data,  # ! Remove PII
+)
+```
+
+## Integration Patterns
+
+**Context Propagation:**
+- ! Propagate trace_id and span_id across service boundaries
+- ! Use W3C Trace Context headers
+- ! Include trace_id/span_id in logs for correlation
+
+**Alerting:**
+- ! Alert on error rate spikes
+- ! Alert on latency degradation (p95 > threshold)
+- ! Alert on critical business metrics
+- ~ Use runbooks for common issues
+
+## Anti-Patterns
+
+- ⊗ Logging in tight loops (performance impact)
+- ⊗ Logging sensitive data (PII, secrets, passwords)
+- ⊗ Using string formatting instead of structured logging
+- ⊗ No sampling in high-traffic production (cost explosion)
+- ⊗ Ignoring errors silently
+- ⊗ Missing context (no trace_id, request_id)
+- ⊗ Generic span/metric names
+- ⊗ Over-instrumenting (trace everything)
+
+## Mode Decision Matrix
+
+| Feature | fun | fast | pro |
+|---------|-----|------|-----|
+| Structured Logging | ? | ~ | ! |
+| Error Tracking | ? | ~ | ! |
+| Distributed Tracing | ? | ? | ! |
+| Metrics (RED) | ? | ~ | ! |
+| Log Rotation | ? | ~ | ! |
+| Alerting | ? | ? | ! |
+| PII Sanitization | ? | ~ | ! |
+| Source Maps | ? | ~ | ! |
+| Release Tracking | ? | ~ | ! |
+
+## Testing Telemetry
+
+**Development:**
+- ~ Use local exporters (console, file)
+- ~ Test with sampling=1.0
+- ~ Verify context propagation
+
+**Staging:**
+- ~ Mirror production config
+- ~ Test alerting thresholds
+- ~ Validate dashboards
+
+**Production:**
+- ! Gradual rollout of telemetry changes
+- ! Monitor telemetry overhead (<5% CPU)
+- ! Validate data quality
+
+## LLM-specific observability (#481)
+
+LLM applications carry observability needs that conventional request/response tracing does not address. Poisoning, prompt drift, output schema regressions, and token-budget exhaustion are invisible to standard logging unless the prompt and response are captured verbatim and the model + token telemetry is recorded alongside the call. This section extends the general telemetry guidance for projects that call LLM APIs (see [../patterns/llm-app.md](../patterns/llm-app.md) for the full LLM-application standards; this section is the observability slice).
+
+**Per-call audit log (mandatory in pro mode; recommended in fast mode):**
+- ! Log every LLM call: model identifier, prompt hash, response hash, latency, token count (input + output), tool calls invoked
+- ! Store prompt/response pairs in a queryable audit log, separate from application logs — application logs carry pointers/hashes only; the audit log carries the verbatim content
+- ! Log tool invocations alongside the LLM call that produced them so the audit trail `(prompt -> response -> tool call -> outcome)` is recoverable as a single chain
+- ⊗ MUST NOT log raw secrets or PII that leaked into the prompt; redact at log-write time, not at log-read time (the general `⊗ Log sensitive data` rule applies equally to LLM audit logs)
+
+**Token budget tracking:**
+- ~ Track token budgets per session and per user — budget exhaustion is a denial-of-service vector AND a cost-attack vector
+- ~ Emit a metric on every LLM call: `llm_tokens_total{model, kind="input|output", user_id, session_id}`
+- ~ Alert on per-user token spikes (anomaly detection on the distribution; a sudden 10x in tokens/hour for one user is a probe signal)
+
+**Latency histograms per model/prompt variant:**
+- ~ Record latency as a histogram dimensioned by `{model, prompt_variant}` so a slow-down on one prompt template is visible without polluting the model-wide aggregate
+- ~ Track p50/p95/p99 separately for streaming vs. non-streaming calls (streaming p99 is end-of-first-token; non-streaming p99 is end-of-full-response — conflating them hides regressions)
+
+**Evaluation harness logging:**
+- ~ Record evaluation-harness results in the audit log so output quality can be tracked over time (regression detection on the model output distribution, not just the application's behavior)
+- ~ Tag each eval run with the prompt-template version and the model identifier so a quality regression can be traced to either a prompt edit or a provider model update
+
+**Anomaly detection on output distributions:**
+- ~ Track output-length distribution per prompt variant; a sudden shift (longer outputs, shorter outputs, or higher variance) is a probe signal worth investigating
+- ~ Track refusal-rate per prompt variant; an unexpected refusal-rate change typically points at a provider-side policy update or a prompt drift
+- ~ Alert when a deployed prompt template diverges from the reviewed baseline (the **prompt drift detector**); a code review approved one prompt, a runtime change to that prompt without review is a regression signal
+
+**Cross-references:**
+- [../patterns/llm-app.md](../patterns/llm-app.md) `## LLM-specific observability` -- the full standards body this section anchors to
+- [../coding/coding.md](../coding/coding.md) `## Calling LLM APIs (#481)` -- short addendum cross-linking the patterns file
+
+## References
+
+- OpenTelemetry: https://opentelemetry.io/
+- Sentry.io: https://sentry.io/
+- logfire: https://pydantic.dev/logfire
+- structlog: https://www.structlog.org/
+- zerolog: https://github.com/rs/zerolog

package/vbrief/schemas/cache-meta.schema.json

@@ -0,0 +1,137 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://vbrief.dev/schemas/cache-meta-1.0.schema.json",
+  "title": "Deft Cache Entry meta.json",
+  "description": "Per-entry metadata file for the unified content cache (#883 v1). Validates on read AND write so a corrupt or version-incompatible meta.json fails closed (epic v1_must_include M4). Layout: .deft-cache/<source>/<key>/meta.json -- one file per cache entry, sibling to raw.json (always present) and content.md (only present when scan_result.passed=true OR scanner severity is fence-and-pass / strip-and-pass).",
+  "type": "object",
+  "required": [
+    "source",
+    "key",
+    "fetched_at",
+    "ttl_seconds",
+    "expires_at",
+    "scan_result",
+    "size_bytes",
+    "stale"
+  ],
+  "properties": {
+    "source": {
+      "type": "string",
+      "description": "Cache source type. v1 ships 'github-issue' only; 5 additional types (github-pr, github-review, url, email, file) are deferred to v2 per the epic deferred_to_v2 list. The schema constrains the enum so a typo or premature v2 source fails closed at write time.",
+      "enum": ["github-issue"]
+    },
+    "key": {
+      "type": "string",
+      "minLength": 1,
+      "description": "Source-specific lookup key. For github-issue: '<owner>/<repo>/<N>' (e.g. 'deftai/directive/883'). The cache.py CLI normalises and validates the shape per source before write."
+    },
+    "fetched_at": {
+      "$ref": "#/$defs/dateTime",
+      "description": "ISO-8601 UTC timestamp at which raw.json was retrieved from upstream. Consumed by triage_refresh.py (Story 3) for staleness detection."
+    },
+    "ttl_seconds": {
+      "type": "integer",
+      "minimum": 0,
+      "description": "Source-specific TTL in seconds (github-issue=604800 / 7d in v1). Hard-coded per source by cache.py; persisted here so future TTL bumps do not silently re-classify pre-existing entries."
+    },
+    "expires_at": {
+      "$ref": "#/$defs/dateTime",
+      "description": "ISO-8601 UTC timestamp at which this entry should be considered stale. Computed at write time as fetched_at + ttl_seconds; pinned in meta.json so the staleness check is a simple time comparison without re-derivation."
+    },
+    "etag": {
+      "type": "string",
+      "description": "Reserved for v2 ETag-based conditional refresh. Deferred from v1 (gh CLI does not expose ETag at the JSON layer). Optional; omit on write in v1 callers."
+    },
+    "scan_result": {
+      "$ref": "#/$defs/ScanResult"
+    },
+    "size_bytes": {
+      "type": "integer",
+      "minimum": 0,
+      "description": "Byte length of raw.json on disk at write time. Audit-only -- consumed by cache:stats / cache:doctor (deferred to v2) for sanity checks; v1 callers persist the value but do not act on it."
+    },
+    "stale": {
+      "type": "boolean",
+      "description": "Frozen at write time as 'false'. The runtime staleness check on cache:get re-computes from current time vs expires_at; this field exists so a downstream consumer reading meta.json directly (without invoking cache:get) sees a defined value. v1 substitute for the deferred cache:check command per the epic deferred_to_v2 list."
+    }
+  },
+  "additionalProperties": false,
+  "$defs": {
+    "ScanResult": {
+      "type": "object",
+      "required": [
+        "passed",
+        "scanned_at",
+        "scanner_version",
+        "flags"
+      ],
+      "description": "Quarantine scan outcome (#583 / scanner v2). passed=false means the credentials category fired (hard-fail severity); fence-and-pass / strip-and-pass severities still record flags but passed remains true and content.md is written.",
+      "properties": {
+        "passed": {
+          "type": "boolean",
+          "description": "Whether the entry's content.md was safe to write. False only when at least one flag carries severity hard-fail (credentials category)."
+        },
+        "scanned_at": {
+          "$ref": "#/$defs/dateTime",
+          "description": "ISO-8601 UTC timestamp at which the scan ran. Distinct from fetched_at because a future cache:doctor --rescan can update scan_result without re-fetching."
+        },
+        "scanner_version": {
+          "type": "string",
+          "pattern": "^\\d+\\.\\d+\\.\\d+$",
+          "description": "Scanner SemVer (e.g. '2.0.0') as reported by scripts/cache_scanner.SCANNER_VERSION. Documented bump rule in the scanner module: patch=patterns, minor=categories, major=semantic rewrites."
+        },
+        "flags": {
+          "type": "array",
+          "description": "Per-flag findings. Empty array means a clean pass with no scanner action; non-empty does not by itself imply passed=false (only hard-fail severity does).",
+          "items": {
+            "$ref": "#/$defs/ScanFlag"
+          }
+        }
+      },
+      "additionalProperties": false
+    },
+    "ScanFlag": {
+      "type": "object",
+      "required": [
+        "category",
+        "severity",
+        "detail"
+      ],
+      "properties": {
+        "category": {
+          "type": "string",
+          "enum": [
+            "injection-heading",
+            "credentials",
+            "invisible-unicode"
+          ],
+          "description": "Scanner v2 category. v1 baseline ships these three; shell-cmd-injection and exfiltration-patterns are deferred to v2."
+        },
+        "severity": {
+          "type": "string",
+          "enum": [
+            "fence-and-pass",
+            "hard-fail",
+            "strip-and-pass"
+          ],
+          "description": "Per-category severity. injection-heading=fence-and-pass; credentials=hard-fail; invisible-unicode=strip-and-pass. Per-category severity is a documented epic departure from the design doc's uniform hard-fail (rationale: hard-fail-on-injection-headings would break legitimate '## STEP' template headings)."
+        },
+        "detail": {
+          "type": "string",
+          "description": "Human-readable summary of what was matched (e.g. 'matched credential pattern: ghp_<32 chars>' or 'stripped 3 invisible unicode codepoints'). Detail strings MUST NOT include the matched secret itself for the credentials category -- emit a redacted descriptor only."
+        },
+        "match_count": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "Number of distinct matches for this flag. Useful for invisible-unicode strip-and-pass where the meaningful number is the codepoint count, not a string excerpt."
+        }
+      },
+      "additionalProperties": false
+    },
+    "dateTime": {
+      "type": "string",
+      "format": "date-time",
+      "pattern": "(Z|[+-]\\d{2}:\\d{2})$"
+    }
+  }
+}

package/vbrief/schemas/candidates.schema.json

@@ -0,0 +1,130 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://deft.dev/schemas/candidates-1.0.schema.json",
+  "title": "Triage Candidates Log Entry v1",
+  "description": "Frozen interface contract for one line in vbrief/.eval/candidates.jsonl. Each line is a single JSON object capturing one triage decision. The log is append-only -- existing lines are never edited or deleted. Consumed by Story 3 (triage actions), Story 4 (triage:bulk + triage:refresh), and Story 6 (bootstrap). Field set, types, and enums are FROZEN -- new fields may only be added in additive, optional positions in a follow-up schema version.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "decision_id",
+    "timestamp",
+    "repo",
+    "issue_number",
+    "decision",
+    "actor"
+  ],
+  "properties": {
+    "decision_id": {
+      "type": "string",
+      "format": "uuid",
+      "description": "RFC 4122 UUID (any version) uniquely identifying this decision row.",
+      "pattern": "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$"
+    },
+    "timestamp": {
+      "type": "string",
+      "format": "date-time",
+      "description": "ISO-8601 UTC timestamp with the literal Z suffix (e.g. 2026-05-03T16:32:54Z) when the decision was recorded. Non-UTC offsets are intentionally rejected because latest_decision() sorts by lexicographic timestamp comparison, which is correct only when every entry is anchored to the same zone.",
+      "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(\\.\\d+)?Z$"
+    },
+    "repo": {
+      "type": "string",
+      "description": "GitHub-style 'owner/name' coordinate of the repository the issue belongs to.",
+      "pattern": "^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$"
+    },
+    "issue_number": {
+      "type": "integer",
+      "minimum": 1,
+      "description": "Issue number within `repo`."
+    },
+    "decision": {
+      "type": "string",
+      "enum": [
+        "accept",
+        "reject",
+        "defer",
+        "needs-ac",
+        "mark-duplicate",
+        "reset",
+        "resume-eligible"
+      ],
+      "description": "Decision verb. `mark-duplicate` requires `linked_to`; `reset` and `resume-eligible` require `prior_decision_id`. `resume-eligible` is appended by the resume-condition evaluator (#1123 / D3) when a prior `defer` entry's `resume_on` expression fires; it does NOT auto-un-defer."
+    },
+    "actor": {
+      "type": "string",
+      "minLength": 1,
+      "description": "Decision actor identity -- a human username (e.g. 'msadams') or an agent tag (e.g. 'agent:warp-oz')."
+    },
+    "reason": {
+      "type": "string",
+      "description": "Optional free-form rationale captured at decision time."
+    },
+    "resume_on": {
+      "type": "string",
+      "description": "Optional structured resume condition recorded on `defer` entries (#1123 / D3). Grammar v1: `ref:closed:#N` | `ref:merged:#N` | `date:>=YYYY-MM-DD` | `pending-count:>=N` | `pending-count:<=N`, joined by a single top-level `AND` / `OR`. Evaluated by `scripts/resume_conditions.py`; firings append a sibling `resume-eligible` entry.",
+      "minLength": 1
+    },
+    "linked_to": {
+      "type": "integer",
+      "minimum": 1,
+      "description": "Required ONLY when `decision` is `mark-duplicate`: issue number this duplicate links to."
+    },
+    "prior_decision_id": {
+      "type": "string",
+      "pattern": "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$",
+      "description": "Required ONLY when `decision` is `reset`: the prior decision's `decision_id` being undone."
+    }
+  },
+  "allOf": [
+    {
+      "if": {
+        "properties": {
+          "decision": {
+            "const": "mark-duplicate"
+          }
+        },
+        "required": [
+          "decision"
+        ]
+      },
+      "then": {
+        "required": [
+          "linked_to"
+        ]
+      },
+      "else": {
+        "not": {
+          "required": [
+            "linked_to"
+          ]
+        }
+      }
+    },
+    {
+      "if": {
+        "properties": {
+          "decision": {
+            "enum": [
+              "reset",
+              "resume-eligible"
+            ]
+          }
+        },
+        "required": [
+          "decision"
+        ]
+      },
+      "then": {
+        "required": [
+          "prior_decision_id"
+        ]
+      },
+      "else": {
+        "not": {
+          "required": [
+            "prior_decision_id"
+          ]
+        }
+      }
+    }
+  ]
+}