@simbimbo/brainstem 0.0.1 → 0.0.2

package/docs/api.md

@@ -0,0 +1,380 @@
+# API Spec
+
+## Goal
+
+Provide a small, explainable API for ingesting events, scoring patterns, promoting memory, and retrieving relevant operational history.
+
+The API should support both machine-driven ingestion and operator-driven investigation.
+
+---
+
+## 1. Ingest Event
+
+### `POST /events/ingest`
+
+Ingest one normalized or semi-normalized event.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "asset_id": "fw-01",
+  "source_type": "syslog",
+  "source_path": "/var/log/syslog",
+  "host": "fw-01",
+  "service": "charon",
+  "timestamp": "2026-03-22T03:30:00Z",
+  "severity": "warning",
+  "facility": "daemon",
+  "message_raw": "IPsec SA rekey failed; retrying",
+  "structured_fields": {
+    "peer": "203.0.113.10"
+  }
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "event_id": "evt_123",
+  "signature_id": "sig_456",
+  "candidate_ids": ["cand_789"]
+}
+```
+
+---
+
+## 2. Bulk Ingest
+
+### `POST /events/ingest_bulk`
+
+For log file batches, webhook batches, or replay.
+
+Request:
+```json
+{
+  "events": [ ... ],
+  "source_label": "syslog-replay"
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "ingested": 500,
+  "signatures_created": 12,
+  "candidates_created": 6
+}
+```
+
+---
+
+## 3. Search Events
+
+### `POST /events/search`
+
+Search raw/normalized events.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "query": "rekey failed",
+  "host": "fw-01",
+  "service": "charon",
+  "since": "2026-03-22T00:00:00Z",
+  "limit": 50
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "results": [ ... ]
+}
+```
+
+---
+
+## 4. Search Candidates
+
+### `POST /candidates/search`
+
+Search meaningful derived patterns.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "decision_band": ["review", "urgent_human_review"],
+  "candidate_type": ["recurrence", "self_heal"],
+  "limit": 20
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "results": [ ... ]
+}
+```
+
+---
+
+## 5. Get Candidate Explanation
+
+### `POST /candidates/explain`
+
+Request:
+```json
+{
+  "candidate_id": "cand_789"
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "candidate": { ... },
+  "score_breakdown": {
+    "recurrence": 0.8,
+    "recovery": 0.6,
+    "precursor": 0.7
+  },
+  "evidence": {
+    "event_count": 12,
+    "signature_ids": ["sig_456"],
+    "related_incidents": ["inc_100"]
+  },
+  "explanation": "This recurring self-healing VPN issue has increased in frequency and matched a prior promoted incident."
+}
+```
+
+---
+
+## 6. Promote Candidate
+
+### `POST /candidates/promote`
+
+Promote a candidate into durable incident memory.
+
+Request:
+```json
+{
+  "candidate_id": "cand_789",
+  "title": "Recurring VPN rekey instability for client-a",
+  "summary": "Observed 12 self-resolving rekey failures over 7 days.",
+  "incident_type": "vpn_instability"
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "incident_memory_id": "inc_100"
+}
+```
+
+---
+
+## 7. Search Incident Memory
+
+### `POST /incidents/search`
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "query": "vpn flaps",
+  "limit": 10
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "results": [ ... ]
+}
+```
+
+---
+
+## 8. Related History
+
+### `POST /history/related`
+
+Given an event, signature, or candidate, return related past operational memory.
+
+Request:
+```json
+{
+  "subject_type": "signature",
+  "subject_id": "sig_456",
+  "limit": 10
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "related_candidates": [ ... ],
+  "related_incidents": [ ... ],
+  "related_lessons": [ ... ]
+}
+```
+
+---
+
+## 9. Daily Digest
+
+### `POST /digest/daily`
+
+Generate the operator digest for a tenant or environment.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "since": "2026-03-21T00:00:00Z",
+  "limit": 10
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "items": [
+    {
+      "candidate_id": "cand_789",
+      "title": "Recurring VPN rekey instability",
+      "decision_band": "review",
+      "why_it_matters": "Repeated 12 times this week, self-resolved each time, and matches a prior incident.",
+      "score_total": 0.84
+    }
+  ]
+}
+```
+
+---
+
+## 10. Review Decision
+
+### `POST /review/record`
+
+Capture operator judgment.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "subject_type": "candidate",
+  "subject_id": "cand_789",
+  "decision": "useful",
+  "notes": "This pattern caused tickets last month.",
+  "reviewer": "steven"
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "review_id": "rev_101"
+}
+```
+
+---
+
+## 11. LogicMonitor ingest
+
+### `POST /connectors/logicmonitor/events`
+
+Accept LogicMonitor-originated alert/event payloads.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "resource_id": 12345,
+  "host": "edge-fw-01",
+  "service": "vpn",
+  "severity": "warning",
+  "alert_id": 998877,
+  "message_raw": "VPN tunnel dropped and recovered",
+  "timestamp": "2026-03-22T00:00:00Z",
+  "metadata": {
+    "datasource": "IPSec Tunnel",
+    "instance_name": "site-b",
+    "acknowledged": false,
+    "cleared_at": "2026-03-22T00:00:32Z"
+  }
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "event_id": "evt_123",
+  "signature_id": "sig_456",
+  "candidate_ids": ["cand_789"]
+}
+```
+
+### `POST /connectors/logicmonitor/sync`
+
+Poll/sync LogicMonitor data in batches.
+
+Request:
+```json
+{
+  "tenant_id": "client-a",
+  "since": "2026-03-21T00:00:00Z",
+  "mode": "alerts"
+}
+```
+
+Response:
+```json
+{
+  "ok": true,
+  "ingested": 250,
+  "signatures_created": 9,
+  "candidates_created": 4
+}
+```
+
+## 12. Health / Readiness
+
+### `GET /healthz`
+Simple liveness/readiness.
+
+### `GET /status`
+Compact runtime and ingest status.
+
+### `GET /metrics`
+Bounded metrics and counts. Must not block on deep correlation or full integrity scans.
+
+---
+
+## MVP API subset
+
+For MVP, implement first:
+- `/events/ingest`
+- `/events/ingest_bulk`
+- `/candidates/search`
+- `/candidates/explain`
+- `/candidates/promote`
+- `/digest/daily`
+- `/history/related`
+- `/healthz`
+- `/metrics`

package/docs/architecture.md

@@ -0,0 +1,333 @@
+# Architecture
+
+> This document should be read together with `design-governance.md`, which is the canonical governor for early product scope, attention-model decisions, and intake/discovery/output boundaries.
+
+## Overview
+
+brAInstem converts raw operational inputs into a canonical event stream, assigns and evolves operator attention, and promotes only the most meaningful weak signals into higher-order operational memory.
+
+Canonical early pipeline:
+1. receive raw inputs from source-specific adapters
+2. wrap them in provenance-preserving raw input envelopes
+3. parse/canonicalize into a shared internal event model
+4. normalize and fingerprint
+5. assign initial attention
+6. evolve attention over time based on recurrence/spread/history/context
+7. route events/signatures/candidates according to attention band
+8. surface meaningful operator-facing outputs and promote durable patterns when justified
+
+The core architectural idea is not merely "store logs and analyze them later."
+It is to maintain a continuously updated stream of operational attention.
+
+## Core layers
+
+### 1. Input apparatus
+
+The input apparatus is the trust boundary of the system.
+Its job is not just to parse data, but to receive inputs robustly, preserve provenance, and decouple ingestion from deeper analysis.
+
+Target source classes over time:
+- syslog
+- app logs
+- auth logs
+- firewall / VPN logs
+- file tailing
+- webhook event streams
+- LogicMonitor alerts / events / resource metadata
+- later: journald, Windows events, richer vendor/event adapters
+
+The input apparatus should eventually support broad source coverage, but that breadth should be implemented through a stable adapter model rather than ad hoc source-specific hacks.
+
+### 2. Raw input envelope
+
+All source-specific inputs should first become a provenance-preserving raw envelope.
+
+Suggested raw-envelope fields:
+- `envelope_id`
+- `source_id`
+- `source_type`
+- `tenant_id`
+- `received_at`
+- `observed_at`
+- `transport`
+- `raw_payload`
+- `source_metadata`
+- `parse_status`
+
+This layer exists so the system can:
+- preserve ugly real-world input faithfully
+- survive malformed records
+- replay and audit parser behavior
+- keep source/protocol complexity at the edge
+
+### 3. Canonical event
+
+After parsing/canonicalization, all valid inputs should enter one shared internal stream as canonical events.
+
+Canonical-event fields should be stable enough that downstream discovery does not care whether an event came from syslog, file tailing, webhook JSON, or a monitoring connector.
+
+Suggested canonical-event fields:
+- `event_id`
+- `tenant_id`
+- `source_type`
+- `source_name`
+- `asset_id`
+- `host`
+- `service`
+- `timestamp`
+- `severity`
+- `kind`
+- `message_raw`
+- `message_normalized`
+- `structured_fields`
+- `correlation_keys`
+- `labels`
+- `raw_ref`
+- `ingest_metadata`
+
+### 4. Normalization
+
+Normalization responsibilities:
+- parse timestamps
+- standardize field names and types
+- extract host / service / actor / IP / user where possible
+- normalize obvious message variants
+- remove or suppress volatile values where appropriate
+- produce a stable signature-friendly representation
+
+Normalization is the step that turns heterogeneous inputs into a true shared stream of consciousness.
+
+### 5. Signatures and fingerprints
+
+Each event should map to a reusable signature:
+- same issue family
+- same subsystem
+- same rough operational meaning
+
+Examples:
+- repeated SSH auth failures from one source
+- VPN tunnel flap for site X
+- service Y exited unexpectedly
+- DNS timeout for resolver Z
+
+This enables:
+- recurrence counting
+- cross-host spread detection
+- historical comparison
+
+### 6. Attention model
+
+Attention is the central primitive of brAInstem.
+
+The system should not think in a naive binary of keep vs drop.
+Instead, it should assign and evolve **attention** over time.
+
+This matters because many important weak signals begin as tiny, individually inconsequential events.
+Those events should not necessarily be promoted immediately, but they should be able to earn more attention later.
+
+Suggested early attention bands:
+- `ignore_fast`
+- `background`
+- `watch`
+- `investigate`
+- `promote`
+
+Attention should influence:
+- retention depth
+- compute budget
+- eligibility for candidate generation
+- operator visibility
+- promotion into durable memory
+
+### 7. Candidate generation / discovery apparatus
+
+Candidates are higher-level interpretations of raw events and signatures, such as:
+- recurrence candidate
+- burst candidate
+- spread candidate
+- self-heal candidate
+- precursor candidate
+- anomaly candidate
+
+Examples:
+- "VPN tunnel at client A flapped 7 times this week but always recovered within 30 seconds"
+- "same auth anomaly appeared across 4 devices in 2 hours"
+- "service restart storm follows backup completion on host B"
+
+The discovery apparatus should review the canonical stream in near-real-time but spend the most effort where attention has already been earned.
+
+### 8. Scoring
+
+Candidates should receive an operator-attention score, not just a severity score.
+
+The score should combine:
+- recurrence
+- recovery behavior
+- spread
+- novelty
+- temporal correlation
+- human impact likelihood
+- precursor likelihood
+- prior memory weight
+- optionally source trust / source criticality
+
+Scoring should remain inspectable and explainable.
+Opaque scoring should be avoided early.
+
+### 9. Routing
+
+After attention/scoring, events and candidates should be routed according to what kind of handling they deserve.
+
+Examples:
+- ignore quickly and cheaply
+- retain in background state
+- maintain on a watchlist
+- investigate now
+- promote into operator-facing output
+- promote into durable incident memory / lessons later
+
+### 10. Promotion
+
+Important candidates can be promoted into durable memory forms:
+- `incident_memory`
+- `lesson`
+- `runbook_hint`
+- `watch_pattern`
+- `tenant_risk_marker`
+
+### 7. Retrieval
+
+Two main retrieval modes:
+
+#### Investigative retrieval
+For questions like:
+- have we seen this before?
+- what happened before the outage?
+- what similar incidents exist?
+
+#### Proactive retrieval
+For new events:
+- show prior related incidents
+- show matching lessons / runbook hints
+- explain why this signal matters now
+
+## LogicMonitor connector
+
+LogicMonitor should be treated as a high-value connector, not a hard dependency.
+
+### Why it fits
+LogicMonitor already knows about:
+- devices and resources
+- alerts and alert history
+- collectors and monitored services
+- severity and topology-adjacent metadata
+
+brAInstem should add:
+- recurrence memory
+- weak-signal detection
+- cross-time pattern recall
+- human-threshold significance scoring
+
+### Integration modes
+1. **Webhook ingest**
+   - preferred for fast MVP integration
+   - LogicMonitor pushes alert/event payloads into brAInstem
+
+2. **Polling connector**
+   - periodic fetch of alert/event history and resource metadata
+   - useful for replay, backfill, and correlation
+
+3. **Context enrichment**
+   - use LogicMonitor device/resource metadata to enrich events with:
+     - resource id
+     - datasource
+     - instance
+     - host group / site
+     - collector context
+
+### Mapping into brAInstem
+LogicMonitor payloads should map into native objects like:
+- `tenant`
+- `asset`
+- `event`
+- `signature`
+- `candidate`
+
+Recommended event metadata fields:
+- `source_type = logicmonitor`
+- `alert_id`
+- `resource_id`
+- `datasource`
+- `instance_name`
+- `severity`
+- `acknowledged`
+- `cleared_at`
+
+### Product role
+LogicMonitor should supply operational signal and context.
+brAInstem should decide whether a pattern is recurring, meaningful, and worth a human's attention even when the original LM alert auto-cleared or stayed below escalation importance.
+
+## Suggested storage model
+
+### Main DB
+- events
+- signatures
+- candidates
+- incident_memory
+- lessons
+- runbooks
+- links
+- provenance
+- review decisions
+
+### File / object store
+- raw log bundles
+- retained evidence files
+- archived slices for drill-down
+
+### Vector / semantic layer
+Use selectively for:
+- fuzzy similarity
+- natural-language search over incidents and lessons
+
+But keep the main architecture centered on:
+- deterministic fingerprints
+- structured recurrence
+- temporal correlation
+- explainable scoring
+
+## Operator-facing outputs
+
+The product value appears here, not merely in the internal pipeline.
+
+Early operator-facing outputs should aim to answer:
+- what is happening?
+- why does it matter?
+- where / how often is it happening?
+- have we seen this before?
+- what should a human check next?
+
+Suggested early outputs:
+- interesting items list
+- daily weak-signal digest
+- recurring issue feed
+- incident precursor feed
+- prior-incident recall on demand
+- explainability payload for every surfaced item
+
+The output contract is as important as the event/attention contract.
+If the surfaced artifacts are noisy, bloated, or unactionable, the architecture has failed even if the internals are elegant.
+
+## Relationship to ocmemog
+
+Shared mechanisms:
+- ingest -> candidate -> promotion
+- provenance
+- explainability
+- retrieval
+- confidence / scoring concepts
+
+Different data model:
+- raw events instead of chat turns
+- signatures instead of prompt-centric queries
+- incidents and lessons instead of primarily continuity memories