@simbimbo/brainstem 0.0.4 → 0.0.5

package/docs/adapters.md

@@ -1,131 +1,435 @@
 # Adapters and Canonical Event Contract
 
-_Status: aligned contract for implemented intake runtime_
+_Status: design contract for intake breadth without connector chaos_
 
-This document defines the current adapter and driver surface and preserves the same intent as design governance without promising unshipped connectors.
+## Purpose
 
-Read with:
+This document defines how brAInstem should accept many different source types without turning into an ungoverned pile of bespoke connector logic.
 
+Its job is to answer:
+- what an adapter is
+- what an adapter is allowed to do
+- what an adapter is not allowed to do
+- what the raw input contract is
+- what the canonical event contract is
+- how failures should be handled
+- what "universal input" means in practice
+
+This document should be read together with:
 - `design-governance.md`
 - `architecture.md`
 - `v0.0.1.md`
 
-## 1. Current runtime intake scope
+---
+
+## 1. Why adapters exist
+
+brAInstem should eventually ingest many classes of operational input:
+- syslog
+- local log files
+- JSON log streams
+- webhook payloads
+- monitoring/alert APIs
+- vendor-specific event formats
+- later: journald, Windows events, queue/stream sources, cloud audit feeds
+
+Those sources all have different:
+- transport behavior
+- payload shapes
+- timestamp formats
+- metadata conventions
+- source identity hints
+- failure modes
+
+The adapter layer exists so that source-specific ugliness stays at the edge.
+
+The rest of the product should primarily deal with:
+- raw input envelopes
+- canonical events
+- attention and routing
+- discovery and memory
+
+### Design rule
+Adapter complexity belongs at the edges.
+The discovery apparatus should not need to know what transport/protocol originally delivered an event.
+
+---
+
+## 2. What "universal input" means
+
+"Universal" does **not** mean:
+- native first-class support for every source in the first release
+- a giant vendor integration matrix before the core event model is stable
+- a custom parser for every odd format on day one
+
+"Universal" **does** mean:
+- every source can be represented by a raw input envelope
+- every successful parse can become a canonical event
+- every canonical event can enter the same attention/discovery pipeline
+- new sources can be added by implementing a constrained adapter contract instead of creating system-wide exceptions
+
+Universal input is a property of the architecture, not a promise of immediate breadth.
+
+---
+
+## 3. Adapter responsibilities
+
+An adapter is responsible for:
+1. receiving source data from a specific source class
+2. preserving enough provenance to audit where the input came from
+3. emitting a valid `RawInputEnvelope`
+4. optionally performing source-local pre-parse validation
+5. handing the envelope into the parser/canonicalizer stage
+
+An adapter is **not** responsible for:
+- long-term memory decisions
+- discovery logic
+- attention scoring policy
+- promotion policy
+- operator-facing explanation generation
+
+Adapters should stay narrow.
+
+---
+
+## 4. Adapter categories
+
+Early useful categories:
+
+### 4.1 File adapter
+For:
+- local file tails
+- rotated logs
+- directory watch patterns
+- line-oriented service/application logs
+
+### 4.2 Syslog adapter
+For:
+- UDP syslog
+- TCP syslog
+- later TLS syslog if needed
+
+### 4.3 HTTP/webhook adapter
+For:
+- generic JSON event POST
+- vendor webhooks
+- batched events
+
+### 4.4 API pull adapter
+For:
+- periodic polling of event history
+- alert/event backfill
+- vendor APIs like LogicMonitor where polling is useful
+
+### 4.5 Stream adapter
+For:
+- stdin or pipeline-fed events
+- queue/stream integrations later
+- replay tooling
+
+These are categories, not promises that all must ship in `v0.0.1`.
+
+---
+
+## 5. Raw input envelope contract
+
+Every adapter must emit a raw input envelope before deeper normalization.
+
+This contract preserves:
+- provenance
+- transport identity
+- original payload fidelity
+- parser/debug visibility
+
+## Required fields
+
+### `envelope_id`
+- unique id for the raw envelope
+- generated at receipt if source does not provide one
+
+### `source_id`
+- stable identifier for the source instance
+- examples:
+  - `syslog:edge-fw-01`
+  - `file:/var/log/auth.log`
+  - `http:logicmonitor-prod-webhook`
+
+### `source_type`
+- broad source class
+- examples:
+  - `syslog`
+  - `file`
+  - `http`
+  - `logicmonitor`
+  - `stream`
+
+### `tenant_id`
+- logical tenant/environment owner
+- may be defaulted in early local mode
+- should still exist conceptually even if the first release uses a single tenant
+
+### `received_at`
+- timestamp when brAInstem received the input
+
+### `raw_payload`
+- original raw line/body/payload in preserved form
+- may be string, bytes, or structured object depending on implementation, but must remain recoverable
+
+## Strongly recommended fields
+
+### `observed_at`
+- source-reported timestamp if available
+- may differ from `received_at`
+
+### `transport`
+- example values:
+  - `syslog-udp`
+  - `syslog-tcp`
+  - `http-post`
+  - `file-tail`
+  - `api-poll`
+
+### `source_metadata`
+- adapter/source-specific metadata
+- examples:
+  - file path
+  - listener port
+  - remote ip
+  - vendor alert id
+  - request headers subset
+  - offset/sequence info
+
+### `parse_status`
+- initial parse state marker
+- example values:
+  - `pending`
+  - `parsed`
+  - `parse_error`
+  - `unsupported`
+
+### `sequence_hint`
+- optional source ordering hint when available
+
+## Design rule
+The raw envelope is not the product output.
+It is the preserved intake truth that allows everything else to be audited.
+
+---
+
+## 6. Canonical event contract
+
+After parsing/canonicalization, a successful input should become a canonical event.
+
+This is the shared internal stream of consciousness.
+
+Once an event becomes canonical, the discovery apparatus should not care whether it came from:
+- syslog
+- file tail
+- webhook
+- LogicMonitor
+- future sources
+
+## Required fields
+
+### `event_id`
+- stable unique id for the canonical event
+
+### `tenant_id`
+- the tenant/environment the event belongs to
+
+### `source_type`
+- normalized source family
+
+### `timestamp`
+- best normalized event timestamp
+- prefers true observed time when trustworthy
+- may fall back to receipt time
+
+### `kind`
+- normalized event class
+- examples:
+  - `auth_failure`
+  - `service_restart`
+  - `vpn_flap`
+  - `generic_warning`
+
+### `message_raw`
+- original message body after basic extraction
+
+### `message_normalized`
+- normalized message used for fingerprinting/grouping
+
+### `raw_ref`
+- reference back to the raw input envelope or raw store
+
+## Strongly recommended fields
+
+### `source_name`
+- human-meaningful source instance name
+
+### `host`
+- normalized host/device identity where possible
+
+### `asset_id`
+- stable asset identifier if known
+
+### `service`
+- normalized service/subsystem name
+
+### `severity`
+- normalized severity value or band
+
+### `labels`
+- tag-like annotations for routing/discovery
+
+### `structured_fields`
+- extracted structured values
+
+### `correlation_keys`
+- fields likely useful for grouping/spread/recurrence logic
+
+### `ingest_metadata`
+- useful canonicalization metadata that should travel downstream
+
+---
+
+## 7. Parse failure handling
+
+brAInstem must never quietly erase parse failures.
+
+If an adapter can receive input but canonicalization fails, the system should:
+- preserve the raw envelope
+- emit or record a parse-failure state
+- increment parse/decode error counters
+- allow operators/builders to inspect representative failures
+
+### Why this matters
+A malformed payload can still be operationally meaningful.
+Also, if adapters or parsers silently drop bad inputs, trust dies.
+
+### Design rule
+Bad parse is a first-class ingest outcome, not an invisible discard path.
+
+---
+
+## 8. Normalization responsibilities
+
+The parser/canonicalizer layer, not the adapter, should own the canonical transformation rules where possible.
+
+Normalization responsibilities include:
+- timestamp parsing
+- host/service extraction
+- volatility stripping
+- field normalization
+- message cleanup
+- kind classification
+- preparation for fingerprinting
+
+Adapters may do source-local preprocessing when unavoidable, but the canonicalization logic should remain centralized enough that the system has one real opinion about event shape.
+
+---
+
+## 9. Adapter boundaries
+
+To prevent connector chaos, adapters should obey these rules.
+
+### Adapter may:
+- receive source input
+- preserve provenance
+- perform source-local validation
+- map obvious source metadata into envelope fields
+- pass through source-specific metadata needed later
+
+### Adapter should avoid:
+- inventing bespoke downstream fields that only one adapter knows about
+- performing discovery logic
+- performing long-term suppression policy
+- making promotion decisions
+- reshaping canonical semantics without going through the canonicalization contract
+
+### Strong anti-pattern
+"This source is special, so we built a one-off downstream path just for it."
+
+That is how architecture rots.
 
-The implemented foundation includes:
-- `syslog` adapter + source driver
-- `file` adapter + source driver
+---
 
-Both are line-oriented, envelope-first sources. Everything else remains in the roadmap.
+## 10. Early recommended source support strategy
 
-The UDP listener in `brainstem.listener` is built on the same `syslog` source driver used by API ingestion.
+To keep the product honest and focused, new adapters should be added in this order:
 
-## 2. Why this adapter layer exists
+### First
+- file/log ingestion
+- syslog-like ingestion
+- generic HTTP/webhook ingestion
 
-The adapter layer contains source-specific parsing and provenance capture so downstream discovery stays source-agnostic.
+### Then
+- LogicMonitor
+- other monitoring/alert sources with strong MSP relevance
 
-It should remain narrow:
-- parse raw input into a `RawInputEnvelope`
-- preserve enough source context for replay and forensics
-- avoid discovery/scoring/promotion policy in adapter code
+### Later
+- richer vendor connectors
+- queue/stream integrations
+- platform-specific event systems
 
-## 3. Source-driver contract (runtime code)
+This order keeps the architecture universal without pretending infinite source breadth on day one.
 
-`source_drivers.py` registers drivers by `source_type`.
+---
 
-Current contract:
-- `source_type` string
-- `parse_payload(payload, tenant_id, source_path="", on_parse_error=None) -> list[RawInputEnvelope]`
+## 11. Attention and adapters
 
-Implemented drivers:
-- `file`
-- `syslog`
+Adapters do not assign final operator attention.
 
-Runtime behavior:
-- returns zero or more envelopes
-- may call `on_parse_error` callback on parse failure
-- should not swallow parse exceptions silently
+However, adapters may contribute source metadata that attention scoring later uses, such as:
+- source reliability/trust
+- source criticality
+- source class
+- environment/tenant tags
+- transport characteristics
 
-## 4. Raw input envelope contract
+This distinction matters:
+- adapters provide evidence and provenance
+- the scoring/discovery apparatus decides attention
 
-Adapters are required to return:
-- `tenant_id` (required by ingestion API)
-- `source_type` (`file` or `syslog`)
-- `timestamp` (ISO-ish string; defaults to UTC now when unknown)
-- `message_raw` (non-empty for successful canonicalization)
+---
 
-Adapters may populate:
-- `source_id`
-- `source_name`
-- `source_path`
-- `host`
-- `service`
-- `severity`
-- `asset_id`
-- `facility`
-- `structured_fields`
-- `correlation_keys`
-- `metadata` (adapter-local audit data; e.g. `raw_line`)
+## 12. Audit and replay expectations
 
-Canonicalization outcomes are not part of this contract; they are recorded by the storage layer.
+The adapter + raw envelope system should eventually support:
+- replay of raw inputs into canonicalization/discovery
+- inspection of parse failures
+- verification of source attribution
+- sampling of suppressed/ignored inputs for trust calibration
 
-## 5. Canonical event contract
+Even if replay tooling is not fully mature in `v0.0.1`, the architecture should preserve the possibility.
 
-`canonicalize_raw_input_envelope` currently emits:
-- `tenant_id`
-- `source_type`
-- `timestamp`
-- `message_raw`
-- optional `raw_envelope_id`
-- `host`
-- `service`
-- `severity`
-- `asset_id`
-- `source_path`
-- `facility`
-- `structured_fields`
-- `correlation_keys`
-- `message_normalized`
-- `signature_input`
-- `ingest_metadata` (including `canonicalization_source`, `canonicalized_at`, and raw envelope linkage)
+---
 
-There is no separate explicit `source_id`/`source_name` field on canonical events in this milestone.
+## 13. What a good adapter contract enables
 
-## 6. Parse failure handling
+If this contract is followed, brAInstem can:
+- expand source breadth over time without discovery-layer chaos
+- remain source-agnostic in the core pipeline
+- preserve trust via provenance and replayability
+- maintain one real stream of operational consciousness
 
-When canonicalization fails:
-- intake row is still tracked as `parse_failed`
-- the raw envelope remains queryable in storage
-- parsing failure reason is captured
-- later replay is possible via `/replay/raw` (DB-backed replay path)
+If this contract is ignored, brAInstem becomes:
+- connector soup
+- parsing exceptions everywhere
+- brittle discovery logic
+- untrustworthy ingestion
 
-This is an explicit trust boundary requirement, not a silent discard path.
+That must be avoided.
 
-## 7. Adapter boundaries
+---
 
-Adapters may:
-- adapt transport/source quirks
-- extract obvious source metadata
+## 14. v0.0.1 implication
 
-Adapters should not:
-- assign attention
-- alter candidate generation policy
-- run promotion logic
-- persist raw envelopes or candidates directly
-
-## 8. Planned intake categories
+For `v0.0.1`, the key requirement is not broad adapter count.
+It is that the repo clearly defines:
+- the adapter model
+- the raw envelope concept
+- the canonical event concept
+- the relationship between adapters and the attention/discovery pipeline
 
-These remain design targets beyond the current milestone:
-- TCP/TLS syslog transport
-- webhook/API pull sources
-- queue/stream drivers
-- richer vendor-native adapters
-
-## 9. Why this is still the right architecture now
-
-Even with two drivers, the architecture can stay universal:
-- all registered drivers produce one `RawInputEnvelope` shape
-- all successful envelopes become canonical events in one stream
-- parse failures remain inspectable and replayable
-
-This is the smallest practical intake foundation that matches current runtime implementation.
+That is enough for a truthful first release.

package/docs/api.md

@@ -7,6 +7,7 @@ This document reflects the runtime and API surfaces that are implemented today.
 - ingest endpoints
   - `POST /ingest/event`
   - `POST /ingest/batch`
+  - `POST /ingest/logicmonitor`
   - `POST /replay/raw`
 - inspection endpoints
   - `GET /interesting`
@@ -40,8 +41,10 @@ This document reflects the runtime and API surfaces that are implemented today.
 Registered source types in this milestone are:
 - `syslog`
 - `file`
+- `logicmonitor`
 
-Use `source_type` to select the source driver for raw ingestion events.
+Use `source_type` to select `syslog` or `file` payload drivers for raw envelope ingestion.
+`logicmonitor` payloads are expected on `/ingest/logicmonitor`.
 
 ## Shared ingest request fields
 
@@ -112,6 +115,30 @@ The response includes ingest accounting for this request:
 
 Response shape is the same as single-event ingest, with counts based on all batch rows.
 
+`POST /ingest/logicmonitor` accepts:
+
+```json
+{
+  "tenant_id": "demo-tenant",
+  "source_path": "/logicmonitor/ingest",
+  "threshold": 2,
+  "db_path": "/tmp/brainstem.sqlite3",
+  "events": [
+    {
+      "resource_id": 123,
+      "resource_name": "edge-fw-01",
+      "message_raw": "VPN tunnel dropped and recovered",
+      "severity": "warning",
+      "metadata": {
+        "datasource": "IPSec Tunnel"
+      }
+    }
+  ]
+}
+```
+
+Response shape is the same as other ingest endpoints.
+
 ## Replay raw envelopes
 
 `POST /replay/raw` requires a DB-backed set of raw envelope IDs:
@@ -309,10 +336,18 @@ Operator-oriented runtime summary with the same payload as `/healthz`.
 Returns canonical runtime snapshot:
 - `version`
 - `api_token_env`
+- `capability_flags.source_capabilities` (`source_types` + per-source ingest-mode matrix)
 - runtime defaults and limits
 - listener defaults
 - endpoint capability flags
 
+Current `capability_flags.source_capabilities` includes:
+- `source_types`: `file`, `logicmonitor`, `syslog`
+- `ingest_modes_by_source_type`:
+  - `syslog`: `single_event_api`, `batch_api`, `udp_listener`
+  - `file`: `single_event_api`, `batch_api`
+  - `logicmonitor`: `logicmonitor_webhook`
+
 `GET /runtime` is a fuller diagnostic snapshot with the same runtime summary object used by `/status`/`/healthz`.
 
 ## Listener + file/syslog foundation alignment

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simbimbo/brainstem",
-  "version": "0.0.4",
+  "version": "0.0.5",
   "description": "brAInstem — operational memory for weak signals.",
   "license": "MIT",
   "type": "module",

package/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "brainstem"
-version = "0.0.4"
+version = "0.0.5"
 description = "brAInstem — operational memory for weak signals."
 readme = "README.md"
 requires-python = ">=3.9"

package/tests/test_adapters.py

@@ -16,6 +16,7 @@ def test_syslog_adapter_registry_is_populated() -> None:
     sources = list_raw_input_source_types()
     assert "syslog" in sources
     assert "file" in sources
+    assert "logicmonitor" in sources
     assert isinstance(get_raw_input_adapter("syslog"), RawInputAdapter)