@simbimbo/brainstem 0.0.3 → 0.0.5

package/docs/api.md

@@ -1,380 +1,360 @@
-# API Spec
+# Runtime API (Current Implementation)
+
+This document reflects the runtime and API surfaces that are implemented today.
+
+## What is exposed today
+
+- ingest endpoints
+  - `POST /ingest/event`
+  - `POST /ingest/batch`
+  - `POST /ingest/logicmonitor`
+  - `POST /replay/raw`
+- inspection endpoints
+  - `GET /interesting`
+  - `GET /candidates`
+  - `GET /signatures`
+  - `GET /canonical_events`
+  - `GET /stats`
+  - `GET /failures`
+  - `GET /failures/{raw_envelope_id}`
+  - `GET /raw_envelopes`
+- `GET /ingest/recent`
+- `GET /sources`
+- `GET /sources/status`
+- runtime state endpoints
+  - `GET /runtime`
+  - `GET /status`
+  - `GET /healthz`
+
+## Runtime auth
+
+`BRAINSTEM_API_TOKEN` controls API auth.
+- if unset: write and read endpoints are open
+- if set: endpoints above require one of:
+  - `X-API-Token: <token>`
+  - `Authorization: Bearer <token>`
+
+`/healthz`, `/runtime`, and `/status` are currently unauthenticated.
+
+## Source surface used by the API
+
+Registered source types in this milestone are:
+- `syslog`
+- `file`
+- `logicmonitor`
+
+Use `source_type` to select `syslog` or `file` payload drivers for raw envelope ingestion.
+`logicmonitor` payloads are expected on `/ingest/logicmonitor`.
+
+## Shared ingest request fields
+
+`POST /ingest/event` payload:
 
-## Goal
-
-Provide a small, explainable API for ingesting events, scoring patterns, promoting memory, and retrieving relevant operational history.
-
-The API should support both machine-driven ingestion and operator-driven investigation.
-
----
-
-## 1. Ingest Event
-
-### `POST /events/ingest`
-
-Ingest one normalized or semi-normalized event.
-
-Request:
 ```json
 {
-  "tenant_id": "client-a",
-  "asset_id": "fw-01",
+  "tenant_id": "demo-tenant",
   "source_type": "syslog",
+  "source_id": "fw-01",
+  "source_name": "edge-fw-01",
   "source_path": "/var/log/syslog",
+  "timestamp": "2026-03-22T03:00:00Z",
   "host": "fw-01",
   "service": "charon",
-  "timestamp": "2026-03-22T03:30:00Z",
-  "severity": "warning",
+  "severity": "info",
+  "asset_id": "fw-01",
   "facility": "daemon",
   "message_raw": "IPsec SA rekey failed; retrying",
-  "structured_fields": {
-    "peer": "203.0.113.10"
-  }
-}
-```
-
-Response:
-```json
-{
-  "ok": true,
-  "event_id": "evt_123",
-  "signature_id": "sig_456",
-  "candidate_ids": ["cand_789"]
+  "structured_fields": { "peer": "203.0.113.10" },
+  "correlation_keys": {},
+  "metadata": {}
 }
 ```
 
----
+Optional query params:
+- `threshold` (default: `2`)
+- `db_path` (default: `BRAINSTEM_DB_PATH` or `.brainstem-state/brainstem.sqlite3`)
 
-## 2. Bulk Ingest
+The response includes ingest accounting for this request:
 
-### `POST /events/ingest_bulk`
-
-For log file batches, webhook batches, or replay.
-
-Request:
-```json
-{
-  "events": [ ... ],
-  "source_label": "syslog-replay"
-}
-```
-
-Response:
 ```json
 {
   "ok": true,
-  "ingested": 500,
-  "signatures_created": 12,
-  "candidates_created": 6
+  "tenant_id": "demo-tenant",
+  "event_count": 1,
+  "signature_count": 1,
+  "candidate_count": 0,
+  "parse_failed": 0,
+  "interesting_items": []
 }
 ```
 
----
-
-## 3. Search Events
+`POST /ingest/batch` accepts:
 
-### `POST /events/search`
-
-Search raw/normalized events.
-
-Request:
 ```json
 {
-  "tenant_id": "client-a",
-  "query": "rekey failed",
-  "host": "fw-01",
-  "service": "charon",
-  "since": "2026-03-22T00:00:00Z",
-  "limit": 50
-}
-```
-
-Response:
-```json
-{
-  "ok": true,
-  "results": [ ... ]
+  "threshold": 2,
+  "db_path": "/tmp/brainstem.sqlite3",
+  "events": [
+    {
+      "tenant_id": "demo-tenant",
+      "source_type": "syslog",
+      "message_raw": "Mar 22 03:00:00 fw-01 charon: VPN tunnel recovered",
+      "host": "fw-01",
+      "service": "charon"
+    },
+    {
+      "tenant_id": "demo-tenant",
+      "source_type": "syslog",
+      "message_raw": "Mar 22 03:00:10 fw-01 charon: VPN tunnel recovered",
+      "host": "fw-01",
+      "service": "charon"
+    }
+  ]
 }
 ```
 
----
-
-## 4. Search Candidates
+Response shape is the same as single-event ingest, with counts based on all batch rows.
 
-### `POST /candidates/search`
+`POST /ingest/logicmonitor` accepts:
 
-Search meaningful derived patterns.
-
-Request:
 ```json
 {
-  "tenant_id": "client-a",
-  "decision_band": ["review", "urgent_human_review"],
-  "candidate_type": ["recurrence", "self_heal"],
-  "limit": 20
-}
-```
-
-Response:
-```json
-{
-  "ok": true,
-  "results": [ ... ]
+  "tenant_id": "demo-tenant",
+  "source_path": "/logicmonitor/ingest",
+  "threshold": 2,
+  "db_path": "/tmp/brainstem.sqlite3",
+  "events": [
+    {
+      "resource_id": 123,
+      "resource_name": "edge-fw-01",
+      "message_raw": "VPN tunnel dropped and recovered",
+      "severity": "warning",
+      "metadata": {
+        "datasource": "IPSec Tunnel"
+      }
+    }
+  ]
 }
 ```
 
----
+Response shape is the same as other ingest endpoints.
 
-## 5. Get Candidate Explanation
+## Replay raw envelopes
 
-### `POST /candidates/explain`
+`POST /replay/raw` requires a DB-backed set of raw envelope IDs:
 
-Request:
 ```json
 {
-  "candidate_id": "cand_789"
+  "db_path": "/tmp/brainstem.sqlite3",
+  "raw_envelope_ids": [1, 2, 5],
+  "threshold": 2,
+  "force": false,
+  "allowed_statuses": ["received", "parse_failed"]
 }
 ```
 
-Response:
+Valid status values are:
+- `received`
+- `canonicalized`
+- `parse_failed`
+- `unsupported`
+
+Response example:
+
 ```json
 {
   "ok": true,
-  "candidate": { ... },
-  "score_breakdown": {
-    "recurrence": 0.8,
-    "recovery": 0.6,
-    "precursor": 0.7
-  },
-  "evidence": {
-    "event_count": 12,
-    "signature_ids": ["sig_456"],
-    "related_incidents": ["inc_100"]
-  },
-  "explanation": "This recurring self-healing VPN issue has increased in frequency and matched a prior promoted incident."
+  "db_path": "/tmp/brainstem.sqlite3",
+  "requested_raw_envelope_ids": [1,2,5],
+  "attempted_raw_envelope_ids": [1,2],
+  "skipped": [
+    {"raw_envelope_id": 5, "reason": "not_replayable", "status": "canonicalized"}
+  ],
+  "event_count": 2,
+  "signature_count": 1,
+  "candidate_count": 1,
+  "parse_failed": 0
 }
 ```
 
----
-
-## 6. Promote Candidate
+## Inspection endpoints
 
-### `POST /candidates/promote`
+### `GET /interesting`
 
-Promote a candidate into durable incident memory.
+Query:
+- `limit` (default: `5`)
+- `db_path` (required for non-empty output)
 
-Request:
+Returns:
 ```json
-{
-  "candidate_id": "cand_789",
-  "title": "Recurring VPN rekey instability for client-a",
-  "summary": "Observed 12 self-resolving rekey failures over 7 days.",
-  "incident_type": "vpn_instability"
-}
+{"ok": true, "items": []}
 ```
+If `db_path` is omitted, items are empty by design.
 
-Response:
-```json
-{
-  "ok": true,
-  "incident_memory_id": "inc_100"
-}
-```
-
----
-
-## 7. Search Incident Memory
+### `GET /candidates`
 
-### `POST /incidents/search`
+Query:
+- `limit` (default: `5`)
+- `candidate_type` (optional)
+- `decision_band` (optional)
+- `min_score_total` (optional)
+- `db_path` (optional)
 
-Request:
+Returns:
 ```json
-{
-  "tenant_id": "client-a",
-  "query": "vpn flaps",
-  "limit": 10
-}
+{"ok": true, "count": 0, "items": []}
 ```
 
-Response:
-```json
-{
-  "ok": true,
-  "results": [ ... ]
-}
-```
+### `GET /signatures`
 
----
+Query:
+- `limit` (default: `5`)
+- `event_family` (optional)
+- `service` (optional)
+- `min_occurrence_count` (optional)
+- `db_path` (optional)
 
-## 8. Related History
+Returns:
+```json
+{"ok": true, "count": 0, "items": []}
+```
 
-### `POST /history/related`
+### `GET /canonical_events`
 
-Given an event, signature, or candidate, return related past operational memory.
+Query:
+- `limit` (default: `20`)
+- `tenant_id` (optional)
+- `source` (optional)
+- `host` (optional)
+- `service` (optional)
+- `severity` (optional)
+- `db_path` (optional)
 
-Request:
+Returns:
 ```json
-{
-  "subject_type": "signature",
-  "subject_id": "sig_456",
-  "limit": 10
-}
+{"ok": true, "count": 0, "items": []}
 ```
 
-Response:
+### `GET /stats`
+
+Query:
+- `db_path` (optional)
+
+Returns:
 ```json
 {
   "ok": true,
-  "related_candidates": [ ... ],
-  "related_incidents": [ ... ],
-  "related_lessons": [ ... ]
+  "received": 10,
+  "canonicalized": 9,
+  "parse_failed": 1,
+  "unsupported": 0,
+  "candidates_generated": 4,
+  "source_summaries": {
+    "source_type": [{"value":"syslog","count":9}],
+    "...": []
+  }
 }
 ```
 
----
-
-## 9. Daily Digest
+### `GET /failures`
 
-### `POST /digest/daily`
+Query:
+- `limit` (default: `20`)
+- `status` (optional; one of received/canonicalized/parse_failed/unsupported)
+- `db_path` (optional)
 
-Generate the operator digest for a tenant or environment.
-
-Request:
+Returns:
 ```json
-{
-  "tenant_id": "client-a",
-  "since": "2026-03-21T00:00:00Z",
-  "limit": 10
-}
+{"ok": true, "items": [], "count": 0, "status": null}
 ```
 
-Response:
-```json
-{
-  "ok": true,
-  "items": [
-    {
-      "candidate_id": "cand_789",
-      "title": "Recurring VPN rekey instability",
-      "decision_band": "review",
-      "why_it_matters": "Repeated 12 times this week, self-resolved each time, and matches a prior incident.",
-      "score_total": 0.84
-    }
-  ]
-}
-```
+### `GET /ingest/recent`
 
----
+Query:
+- `limit` (default: `20`)
+- `status` (optional filter)
+- `db_path` (optional)
 
-## 10. Review Decision
+Returns latest raw envelopes ordered newest-first.
 
-### `POST /review/record`
+### `GET /failures/{raw_envelope_id}`
 
-Capture operator judgment.
+Returns one raw envelope row or 404 if missing.
 
-Request:
-```json
-{
-  "tenant_id": "client-a",
-  "subject_type": "candidate",
-  "subject_id": "cand_789",
-  "decision": "useful",
-  "notes": "This pattern caused tickets last month.",
-  "reviewer": "steven"
-}
-```
+### `GET /raw_envelopes`
 
-Response:
-```json
-{
-  "ok": true,
-  "review_id": "rev_101"
-}
-```
+Query:
+- `limit` (default: `20`)
+- `status` (optional; one of received/canonicalized/parse_failed/unsupported)
+- `tenant_id` (optional)
+- `source_type` (optional)
+- `source_id` (optional)
+- `source_path` (optional)
+- `db_path` (optional)
 
----
+Returns matching raw envelopes ordered newest-first with compact inspection fields (tenant/source metadata, message, status, parsing metadata).
 
-## 11. LogicMonitor ingest
+### `GET /sources`
 
-### `POST /connectors/logicmonitor/events`
+Query:
+- `limit` (default: `10`)
+- `db_path` (optional)
 
-Accept LogicMonitor-originated alert/event payloads.
+Returns per-dimension counts for source metadata.
 
-Request:
-```json
-{
-  "tenant_id": "client-a",
-  "resource_id": 12345,
-  "host": "edge-fw-01",
-  "service": "vpn",
-  "severity": "warning",
-  "alert_id": 998877,
-  "message_raw": "VPN tunnel dropped and recovered",
-  "timestamp": "2026-03-22T00:00:00Z",
-  "metadata": {
-    "datasource": "IPSec Tunnel",
-    "instance_name": "site-b",
-    "acknowledged": false,
-    "cleared_at": "2026-03-22T00:00:32Z"
-  }
-}
-```
+### `GET /sources/status`
 
-Response:
-```json
-{
-  "ok": true,
-  "event_id": "evt_123",
-  "signature_id": "sig_456",
-  "candidate_ids": ["cand_789"]
-}
-```
+Query:
+- `limit` (default: `20`)
+- `tenant_id` (optional filter)
+- `source_type` (optional filter)
+- `source_id` (optional filter)
+- `source_path` (optional filter)
+- `db_path` (optional)
 
-### `POST /connectors/logicmonitor/sync`
+Returns one row per source dimension with raw/failed/canonicalized counts.
 
-Poll/sync LogicMonitor data in batches.
+### `GET /healthz`
 
-Request:
-```json
-{
-  "tenant_id": "client-a",
-  "since": "2026-03-21T00:00:00Z",
-  "mode": "alerts"
-}
-```
+Always available; useful for liveness.
 
-Response:
-```json
-{
-  "ok": true,
-  "ingested": 250,
-  "signatures_created": 9,
-  "candidates_created": 4
-}
-```
+Response is identical to `GET /status` and includes:
+- `ok`
+- `status`
+- runtime summary including auth/capability/config state
 
-## 12. Health / Readiness
+### `GET /status`
 
-### `GET /healthz`
-Simple liveness/readiness.
+Operator-oriented runtime summary with the same payload as `/healthz`.
+- `ok`
+- `status`
+- `runtime`: version, auth_state, defaults, limits, capabilities
+- `api_token_enabled`
 
-### `GET /status`
-Compact runtime and ingest status.
+### `GET /runtime`
+
+Returns canonical runtime snapshot:
+- `version`
+- `api_token_env`
+- `capability_flags.source_capabilities` (`source_types` + per-source ingest-mode matrix)
+- runtime defaults and limits
+- listener defaults
+- endpoint capability flags
+
+Current `capability_flags.source_capabilities` includes:
+- `source_types`: `file`, `logicmonitor`, `syslog`
+- `ingest_modes_by_source_type`:
+  - `syslog`: `single_event_api`, `batch_api`, `udp_listener`
+  - `file`: `single_event_api`, `batch_api`
+  - `logicmonitor`: `logicmonitor_webhook`
 
-### `GET /metrics`
-Bounded metrics and counts. Must not block on deep correlation or full integrity scans.
+`GET /runtime` is a fuller diagnostic snapshot with the same runtime summary object used by `/status`/`/healthz`.
 
----
+## Listener + file/syslog foundation alignment
 
-## MVP API subset
+The runtime path is deliberately split:
+- API path (`/ingest/*`) for HTTP ingestion, replay, and inspection
+- UDP syslog path (`brainstem.listener`) for direct syslog intake and optional stdout event emission
+- file intake for helper ingestion flow through `source_type: file` payloads
 
-For MVP, implement first:
-- `/events/ingest`
-- `/events/ingest_bulk`
-- `/candidates/search`
-- `/candidates/explain`
-- `/candidates/promote`
-- `/digest/daily`
-- `/history/related`
-- `/healthz`
-- `/metrics`
+For listener operator notes and full step-by-step startup, see [docs/README.md](README.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simbimbo/brainstem",
-  "version": "0.0.3",
+  "version": "0.0.5",
   "description": "brAInstem — operational memory for weak signals.",
   "license": "MIT",
   "type": "module",

package/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "brainstem"
-version = "0.0.3"
+version = "0.0.5"
 description = "brAInstem — operational memory for weak signals."
 readme = "README.md"
 requires-python = ">=3.9"