@simbimbo/brainstem 0.0.3 → 0.0.4

package/docs/adapters.md

@@ -1,435 +1,131 @@
 # Adapters and Canonical Event Contract
 
-_Status: design contract for intake breadth without connector chaos_
+_Status: aligned contract for implemented intake runtime_
 
-## Purpose
+This document defines the current adapter and driver surface and preserves the same intent as design governance without promising unshipped connectors.
 
-This document defines how brAInstem should accept many different source types without turning into an ungoverned pile of bespoke connector logic.
+Read with:
 
-Its job is to answer:
-- what an adapter is
-- what an adapter is allowed to do
-- what an adapter is not allowed to do
-- what the raw input contract is
-- what the canonical event contract is
-- how failures should be handled
-- what "universal input" means in practice
-
-This document should be read together with:
 - `design-governance.md`
 - `architecture.md`
 - `v0.0.1.md`
 
----
-
-## 1. Why adapters exist
-
-brAInstem should eventually ingest many classes of operational input:
-- syslog
-- local log files
-- JSON log streams
-- webhook payloads
-- monitoring/alert APIs
-- vendor-specific event formats
-- later: journald, Windows events, queue/stream sources, cloud audit feeds
-
-Those sources all have different:
-- transport behavior
-- payload shapes
-- timestamp formats
-- metadata conventions
-- source identity hints
-- failure modes
-
-The adapter layer exists so that source-specific ugliness stays at the edge.
-
-The rest of the product should primarily deal with:
-- raw input envelopes
-- canonical events
-- attention and routing
-- discovery and memory
-
-### Design rule
-Adapter complexity belongs at the edges.
-The discovery apparatus should not need to know what transport/protocol originally delivered an event.
-
----
-
-## 2. What "universal input" means
-
-"Universal" does **not** mean:
-- native first-class support for every source in the first release
-- a giant vendor integration matrix before the core event model is stable
-- a custom parser for every odd format on day one
-
-"Universal" **does** mean:
-- every source can be represented by a raw input envelope
-- every successful parse can become a canonical event
-- every canonical event can enter the same attention/discovery pipeline
-- new sources can be added by implementing a constrained adapter contract instead of creating system-wide exceptions
-
-Universal input is a property of the architecture, not a promise of immediate breadth.
-
----
-
-## 3. Adapter responsibilities
-
-An adapter is responsible for:
-1. receiving source data from a specific source class
-2. preserving enough provenance to audit where the input came from
-3. emitting a valid `RawInputEnvelope`
-4. optionally performing source-local pre-parse validation
-5. handing the envelope into the parser/canonicalizer stage
-
-An adapter is **not** responsible for:
-- long-term memory decisions
-- discovery logic
-- attention scoring policy
-- promotion policy
-- operator-facing explanation generation
-
-Adapters should stay narrow.
-
----
-
-## 4. Adapter categories
-
-Early useful categories:
-
-### 4.1 File adapter
-For:
-- local file tails
-- rotated logs
-- directory watch patterns
-- line-oriented service/application logs
-
-### 4.2 Syslog adapter
-For:
-- UDP syslog
-- TCP syslog
-- later TLS syslog if needed
-
-### 4.3 HTTP/webhook adapter
-For:
-- generic JSON event POST
-- vendor webhooks
-- batched events
-
-### 4.4 API pull adapter
-For:
-- periodic polling of event history
-- alert/event backfill
-- vendor APIs like LogicMonitor where polling is useful
-
-### 4.5 Stream adapter
-For:
-- stdin or pipeline-fed events
-- queue/stream integrations later
-- replay tooling
-
-These are categories, not promises that all must ship in `v0.0.1`.
-
----
-
-## 5. Raw input envelope contract
-
-Every adapter must emit a raw input envelope before deeper normalization.
-
-This contract preserves:
-- provenance
-- transport identity
-- original payload fidelity
-- parser/debug visibility
-
-## Required fields
-
-### `envelope_id`
-- unique id for the raw envelope
-- generated at receipt if source does not provide one
-
-### `source_id`
-- stable identifier for the source instance
-- examples:
-  - `syslog:edge-fw-01`
-  - `file:/var/log/auth.log`
-  - `http:logicmonitor-prod-webhook`
-
-### `source_type`
-- broad source class
-- examples:
-  - `syslog`
-  - `file`
-  - `http`
-  - `logicmonitor`
-  - `stream`
-
-### `tenant_id`
-- logical tenant/environment owner
-- may be defaulted in early local mode
-- should still exist conceptually even if the first release uses a single tenant
-
-### `received_at`
-- timestamp when brAInstem received the input
-
-### `raw_payload`
-- original raw line/body/payload in preserved form
-- may be string, bytes, or structured object depending on implementation, but must remain recoverable
-
-## Strongly recommended fields
-
-### `observed_at`
-- source-reported timestamp if available
-- may differ from `received_at`
-
-### `transport`
-- example values:
-  - `syslog-udp`
-  - `syslog-tcp`
-  - `http-post`
-  - `file-tail`
-  - `api-poll`
-
-### `source_metadata`
-- adapter/source-specific metadata
-- examples:
-  - file path
-  - listener port
-  - remote ip
-  - vendor alert id
-  - request headers subset
-  - offset/sequence info
-
-### `parse_status`
-- initial parse state marker
-- example values:
-  - `pending`
-  - `parsed`
-  - `parse_error`
-  - `unsupported`
-
-### `sequence_hint`
-- optional source ordering hint when available
-
-## Design rule
-The raw envelope is not the product output.
-It is the preserved intake truth that allows everything else to be audited.
-
----
-
-## 6. Canonical event contract
-
-After parsing/canonicalization, a successful input should become a canonical event.
-
-This is the shared internal stream of consciousness.
-
-Once an event becomes canonical, the discovery apparatus should not care whether it came from:
-- syslog
-- file tail
-- webhook
-- LogicMonitor
-- future sources
-
-## Required fields
-
-### `event_id`
-- stable unique id for the canonical event
-
-### `tenant_id`
-- the tenant/environment the event belongs to
-
-### `source_type`
-- normalized source family
-
-### `timestamp`
-- best normalized event timestamp
-- prefers true observed time when trustworthy
-- may fall back to receipt time
-
-### `kind`
-- normalized event class
-- examples:
-  - `auth_failure`
-  - `service_restart`
-  - `vpn_flap`
-  - `generic_warning`
-
-### `message_raw`
-- original message body after basic extraction
-
-### `message_normalized`
-- normalized message used for fingerprinting/grouping
-
-### `raw_ref`
-- reference back to the raw input envelope or raw store
-
-## Strongly recommended fields
-
-### `source_name`
-- human-meaningful source instance name
-
-### `host`
-- normalized host/device identity where possible
-
-### `asset_id`
-- stable asset identifier if known
-
-### `service`
-- normalized service/subsystem name
-
-### `severity`
-- normalized severity value or band
-
-### `labels`
-- tag-like annotations for routing/discovery
-
-### `structured_fields`
-- extracted structured values
-
-### `correlation_keys`
-- fields likely useful for grouping/spread/recurrence logic
-
-### `ingest_metadata`
-- useful canonicalization metadata that should travel downstream
-
----
-
-## 7. Parse failure handling
-
-brAInstem must never quietly erase parse failures.
-
-If an adapter can receive input but canonicalization fails, the system should:
-- preserve the raw envelope
-- emit or record a parse-failure state
-- increment parse/decode error counters
-- allow operators/builders to inspect representative failures
-
-### Why this matters
-A malformed payload can still be operationally meaningful.
-Also, if adapters or parsers silently drop bad inputs, trust dies.
-
-### Design rule
-Bad parse is a first-class ingest outcome, not an invisible discard path.
-
----
-
-## 8. Normalization responsibilities
-
-The parser/canonicalizer layer, not the adapter, should own the canonical transformation rules where possible.
-
-Normalization responsibilities include:
-- timestamp parsing
-- host/service extraction
-- volatility stripping
-- field normalization
-- message cleanup
-- kind classification
-- preparation for fingerprinting
-
-Adapters may do source-local preprocessing when unavoidable, but the canonicalization logic should remain centralized enough that the system has one real opinion about event shape.
-
----
-
-## 9. Adapter boundaries
-
-To prevent connector chaos, adapters should obey these rules.
-
-### Adapter may:
-- receive source input
-- preserve provenance
-- perform source-local validation
-- map obvious source metadata into envelope fields
-- pass through source-specific metadata needed later
-
-### Adapter should avoid:
-- inventing bespoke downstream fields that only one adapter knows about
-- performing discovery logic
-- performing long-term suppression policy
-- making promotion decisions
-- reshaping canonical semantics without going through the canonicalization contract
-
-### Strong anti-pattern
-"This source is special, so we built a one-off downstream path just for it."
-
-That is how architecture rots.
+## 1. Current runtime intake scope
 
----
+The implemented foundation includes:
+- `syslog` adapter + source driver
+- `file` adapter + source driver
 
-## 10. Early recommended source support strategy
+Both are line-oriented, envelope-first sources. Everything else remains in the roadmap.
 
-To keep the product honest and focused, new adapters should be added in this order:
+The UDP listener in `brainstem.listener` is built on the same `syslog` source driver used by API ingestion.
 
-### First
-- file/log ingestion
-- syslog-like ingestion
-- generic HTTP/webhook ingestion
+## 2. Why this adapter layer exists
 
-### Then
-- LogicMonitor
-- other monitoring/alert sources with strong MSP relevance
+The adapter layer contains source-specific parsing and provenance capture so downstream discovery stays source-agnostic.
 
-### Later
-- richer vendor connectors
-- queue/stream integrations
-- platform-specific event systems
+It should remain narrow:
+- parse raw input into a `RawInputEnvelope`
+- preserve enough source context for replay and forensics
+- avoid discovery/scoring/promotion policy in adapter code
 
-This order keeps the architecture universal without pretending infinite source breadth on day one.
+## 3. Source-driver contract (runtime code)
 
----
+`source_drivers.py` registers drivers by `source_type`.
 
-## 11. Attention and adapters
+Current contract:
+- `source_type` string
+- `parse_payload(payload, tenant_id, source_path="", on_parse_error=None) -> list[RawInputEnvelope]`
 
-Adapters do not assign final operator attention.
+Implemented drivers:
+- `file`
+- `syslog`
 
-However, adapters may contribute source metadata that attention scoring later uses, such as:
-- source reliability/trust
-- source criticality
-- source class
-- environment/tenant tags
-- transport characteristics
+Runtime behavior:
+- returns zero or more envelopes
+- may call `on_parse_error` callback on parse failure
+- should not swallow parse exceptions silently
 
-This distinction matters:
-- adapters provide evidence and provenance
-- the scoring/discovery apparatus decides attention
+## 4. Raw input envelope contract
 
----
+Adapters are required to return:
+- `tenant_id` (required by ingestion API)
+- `source_type` (`file` or `syslog`)
+- `timestamp` (ISO-ish string; defaults to UTC now when unknown)
+- `message_raw` (non-empty for successful canonicalization)
 
-## 12. Audit and replay expectations
+Adapters may populate:
+- `source_id`
+- `source_name`
+- `source_path`
+- `host`
+- `service`
+- `severity`
+- `asset_id`
+- `facility`
+- `structured_fields`
+- `correlation_keys`
+- `metadata` (adapter-local audit data; e.g. `raw_line`)
 
-The adapter + raw envelope system should eventually support:
-- replay of raw inputs into canonicalization/discovery
-- inspection of parse failures
-- verification of source attribution
-- sampling of suppressed/ignored inputs for trust calibration
+Canonicalization outcomes are not part of this contract; they are recorded by the storage layer.
 
-Even if replay tooling is not fully mature in `v0.0.1`, the architecture should preserve the possibility.
+## 5. Canonical event contract
 
----
+`canonicalize_raw_input_envelope` currently emits:
+- `tenant_id`
+- `source_type`
+- `timestamp`
+- `message_raw`
+- optional `raw_envelope_id`
+- `host`
+- `service`
+- `severity`
+- `asset_id`
+- `source_path`
+- `facility`
+- `structured_fields`
+- `correlation_keys`
+- `message_normalized`
+- `signature_input`
+- `ingest_metadata` (including `canonicalization_source`, `canonicalized_at`, and raw envelope linkage)
 
-## 13. What a good adapter contract enables
+There is no separate explicit `source_id`/`source_name` field on canonical events in this milestone.
 
-If this contract is followed, brAInstem can:
-- expand source breadth over time without discovery-layer chaos
-- remain source-agnostic in the core pipeline
-- preserve trust via provenance and replayability
-- maintain one real stream of operational consciousness
+## 6. Parse failure handling
 
-If this contract is ignored, brAInstem becomes:
-- connector soup
-- parsing exceptions everywhere
-- brittle discovery logic
-- untrustworthy ingestion
+When canonicalization fails:
+- intake row is still tracked as `parse_failed`
+- the raw envelope remains queryable in storage
+- parsing failure reason is captured
+- later replay is possible via `/replay/raw` (DB-backed replay path)
 
-That must be avoided.
+This is an explicit trust boundary requirement, not a silent discard path.
 
----
+## 7. Adapter boundaries
 
-## 14. v0.0.1 implication
+Adapters may:
+- adapt transport/source quirks
+- extract obvious source metadata
 
-For `v0.0.1`, the key requirement is not broad adapter count.
-It is that the repo clearly defines:
-- the adapter model
-- the raw envelope concept
-- the canonical event concept
-- the relationship between adapters and the attention/discovery pipeline
+Adapters should not:
+- assign attention
+- alter candidate generation policy
+- run promotion logic
+- persist raw envelopes or candidates directly
+
+## 8. Planned intake categories
 
-That is enough for a truthful first release.
+These remain design targets beyond the current milestone:
+- TCP/TLS syslog transport
+- webhook/API pull sources
+- queue/stream drivers
+- richer vendor-native adapters
+
+## 9. Why this is still the right architecture now
+
+Even with two drivers, the architecture can stay universal:
+- all registered drivers produce one `RawInputEnvelope` shape
+- all successful envelopes become canonical events in one stream
+- parse failures remain inspectable and replayable
+
+This is the smallest practical intake foundation that matches current runtime implementation.