@simbimbo/brainstem 0.0.1 → 0.0.3

package/docs/adapters.md

@@ -0,0 +1,435 @@
+# Adapters and Canonical Event Contract
+
+_Status: design contract for intake breadth without connector chaos_
+
+## Purpose
+
+This document defines how brAInstem should accept many different source types without turning into an ungoverned pile of bespoke connector logic.
+
+Its job is to answer:
+- what an adapter is
+- what an adapter is allowed to do
+- what an adapter is not allowed to do
+- what the raw input contract is
+- what the canonical event contract is
+- how failures should be handled
+- what "universal input" means in practice
+
+This document should be read together with:
+- `design-governance.md`
+- `architecture.md`
+- `v0.0.1.md`
+
+---
+
+## 1. Why adapters exist
+
+brAInstem should eventually ingest many classes of operational input:
+- syslog
+- local log files
+- JSON log streams
+- webhook payloads
+- monitoring/alert APIs
+- vendor-specific event formats
+- later: journald, Windows events, queue/stream sources, cloud audit feeds
+
+Those sources all have different:
+- transport behavior
+- payload shapes
+- timestamp formats
+- metadata conventions
+- source identity hints
+- failure modes
+
+The adapter layer exists so that source-specific ugliness stays at the edge.
+
+The rest of the product should primarily deal with:
+- raw input envelopes
+- canonical events
+- attention and routing
+- discovery and memory
+
+### Design rule
+Adapter complexity belongs at the edges.
+The discovery apparatus should not need to know what transport/protocol originally delivered an event.
+
+---
+
+## 2. What "universal input" means
+
+"Universal" does **not** mean:
+- native first-class support for every source in the first release
+- a giant vendor integration matrix before the core event model is stable
+- a custom parser for every odd format on day one
+
+"Universal" **does** mean:
+- every source can be represented by a raw input envelope
+- every successful parse can become a canonical event
+- every canonical event can enter the same attention/discovery pipeline
+- new sources can be added by implementing a constrained adapter contract instead of creating system-wide exceptions
+
+Universal input is a property of the architecture, not a promise of immediate breadth.
+
+---
+
+## 3. Adapter responsibilities
+
+An adapter is responsible for:
+1. receiving source data from a specific source class
+2. preserving enough provenance to audit where the input came from
+3. emitting a valid `RawInputEnvelope`
+4. optionally performing source-local pre-parse validation
+5. handing the envelope into the parser/canonicalizer stage
+
+An adapter is **not** responsible for:
+- long-term memory decisions
+- discovery logic
+- attention scoring policy
+- promotion policy
+- operator-facing explanation generation
+
+Adapters should stay narrow.
+
+---
+
+## 4. Adapter categories
+
+Early useful categories:
+
+### 4.1 File adapter
+For:
+- local file tails
+- rotated logs
+- directory watch patterns
+- line-oriented service/application logs
+
+### 4.2 Syslog adapter
+For:
+- UDP syslog
+- TCP syslog
+- later TLS syslog if needed
+
+### 4.3 HTTP/webhook adapter
+For:
+- generic JSON event POST
+- vendor webhooks
+- batched events
+
+### 4.4 API pull adapter
+For:
+- periodic polling of event history
+- alert/event backfill
+- vendor APIs like LogicMonitor where polling is useful
+
+### 4.5 Stream adapter
+For:
+- stdin or pipeline-fed events
+- queue/stream integrations later
+- replay tooling
+
+These are categories, not promises that all must ship in `v0.0.1`.
+
+---
+
+## 5. Raw input envelope contract
+
+Every adapter must emit a raw input envelope before deeper normalization.
+
+This contract preserves:
+- provenance
+- transport identity
+- original payload fidelity
+- parser/debug visibility
+
+## Required fields
+
+### `envelope_id`
+- unique id for the raw envelope
+- generated at receipt if source does not provide one
+
+### `source_id`
+- stable identifier for the source instance
+- examples:
+  - `syslog:edge-fw-01`
+  - `file:/var/log/auth.log`
+  - `http:logicmonitor-prod-webhook`
+
+### `source_type`
+- broad source class
+- examples:
+  - `syslog`
+  - `file`
+  - `http`
+  - `logicmonitor`
+  - `stream`
+
+### `tenant_id`
+- logical tenant/environment owner
+- may be defaulted in early local mode
+- should still exist conceptually even if the first release uses a single tenant
+
+### `received_at`
+- timestamp when brAInstem received the input
+
+### `raw_payload`
+- original raw line/body/payload in preserved form
+- may be string, bytes, or structured object depending on implementation, but must remain recoverable
+
+## Strongly recommended fields
+
+### `observed_at`
+- source-reported timestamp if available
+- may differ from `received_at`
+
+### `transport`
+- example values:
+  - `syslog-udp`
+  - `syslog-tcp`
+  - `http-post`
+  - `file-tail`
+  - `api-poll`
+
+### `source_metadata`
+- adapter/source-specific metadata
+- examples:
+  - file path
+  - listener port
+  - remote ip
+  - vendor alert id
+  - request headers subset
+  - offset/sequence info
+
+### `parse_status`
+- initial parse state marker
+- example values:
+  - `pending`
+  - `parsed`
+  - `parse_error`
+  - `unsupported`
+
+### `sequence_hint`
+- optional source ordering hint when available
+
+## Design rule
+The raw envelope is not the product output.
+It is the preserved intake truth that allows everything else to be audited.
+
+---
+
+## 6. Canonical event contract
+
+After parsing/canonicalization, a successful input should become a canonical event.
+
+This is the shared internal stream of consciousness.
+
+Once an event becomes canonical, the discovery apparatus should not care whether it came from:
+- syslog
+- file tail
+- webhook
+- LogicMonitor
+- future sources
+
+## Required fields
+
+### `event_id`
+- stable unique id for the canonical event
+
+### `tenant_id`
+- the tenant/environment the event belongs to
+
+### `source_type`
+- normalized source family
+
+### `timestamp`
+- best normalized event timestamp
+- prefers true observed time when trustworthy
+- may fall back to receipt time
+
+### `kind`
+- normalized event class
+- examples:
+  - `auth_failure`
+  - `service_restart`
+  - `vpn_flap`
+  - `generic_warning`
+
+### `message_raw`
+- original message body after basic extraction
+
+### `message_normalized`
+- normalized message used for fingerprinting/grouping
+
+### `raw_ref`
+- reference back to the raw input envelope or raw store
+
+## Strongly recommended fields
+
+### `source_name`
+- human-meaningful source instance name
+
+### `host`
+- normalized host/device identity where possible
+
+### `asset_id`
+- stable asset identifier if known
+
+### `service`
+- normalized service/subsystem name
+
+### `severity`
+- normalized severity value or band
+
+### `labels`
+- tag-like annotations for routing/discovery
+
+### `structured_fields`
+- extracted structured values
+
+### `correlation_keys`
+- fields likely useful for grouping/spread/recurrence logic
+
+### `ingest_metadata`
+- useful canonicalization metadata that should travel downstream
+
+---
+
+## 7. Parse failure handling
+
+brAInstem must never quietly erase parse failures.
+
+If an adapter can receive input but canonicalization fails, the system should:
+- preserve the raw envelope
+- emit or record a parse-failure state
+- increment parse/decode error counters
+- allow operators/builders to inspect representative failures
+
+### Why this matters
+A malformed payload can still be operationally meaningful.
+Also, if adapters or parsers silently drop bad inputs, trust dies.
+
+### Design rule
+Bad parse is a first-class ingest outcome, not an invisible discard path.
+
+---
+
+## 8. Normalization responsibilities
+
+The parser/canonicalizer layer, not the adapter, should own the canonical transformation rules where possible.
+
+Normalization responsibilities include:
+- timestamp parsing
+- host/service extraction
+- volatility stripping
+- field normalization
+- message cleanup
+- kind classification
+- preparation for fingerprinting
+
+Adapters may do source-local preprocessing when unavoidable, but the canonicalization logic should remain centralized enough that the system has one real opinion about event shape.
+
+---
+
+## 9. Adapter boundaries
+
+To prevent connector chaos, adapters should obey these rules.
+
+### Adapter may:
+- receive source input
+- preserve provenance
+- perform source-local validation
+- map obvious source metadata into envelope fields
+- pass through source-specific metadata needed later
+
+### Adapter should avoid:
+- inventing bespoke downstream fields that only one adapter knows about
+- performing discovery logic
+- performing long-term suppression policy
+- making promotion decisions
+- reshaping canonical semantics without going through the canonicalization contract
+
+### Strong anti-pattern
+"This source is special, so we built a one-off downstream path just for it."
+
+That is how architecture rots.
+
+---
+
+## 10. Early recommended source support strategy
+
+To keep the product honest and focused, new adapters should be added in this order:
+
+### First
+- file/log ingestion
+- syslog-like ingestion
+- generic HTTP/webhook ingestion
+
+### Then
+- LogicMonitor
+- other monitoring/alert sources with strong MSP relevance
+
+### Later
+- richer vendor connectors
+- queue/stream integrations
+- platform-specific event systems
+
+This order keeps the architecture universal without pretending infinite source breadth on day one.
+
+---
+
+## 11. Attention and adapters
+
+Adapters do not assign final operator attention.
+
+However, adapters may contribute source metadata that attention scoring later uses, such as:
+- source reliability/trust
+- source criticality
+- source class
+- environment/tenant tags
+- transport characteristics
+
+This distinction matters:
+- adapters provide evidence and provenance
+- the scoring/discovery apparatus decides attention
+
+---
+
+## 12. Audit and replay expectations
+
+The adapter + raw envelope system should eventually support:
+- replay of raw inputs into canonicalization/discovery
+- inspection of parse failures
+- verification of source attribution
+- sampling of suppressed/ignored inputs for trust calibration
+
+Even if replay tooling is not fully mature in `v0.0.1`, the architecture should preserve the possibility.
+
+---
+
+## 13. What a good adapter contract enables
+
+If this contract is followed, brAInstem can:
+- expand source breadth over time without discovery-layer chaos
+- remain source-agnostic in the core pipeline
+- preserve trust via provenance and replayability
+- maintain one real stream of operational consciousness
+
+If this contract is ignored, brAInstem becomes:
+- connector soup
+- parsing exceptions everywhere
+- brittle discovery logic
+- untrustworthy ingestion
+
+That must be avoided.
+
+---
+
+## 14. v0.0.1 implication
+
+For `v0.0.1`, the key requirement is not broad adapter count.
+It is that the repo clearly defines:
+- the adapter model
+- the raw envelope concept
+- the canonical event concept
+- the relationship between adapters and the attention/discovery pipeline
+
+That is enough for a truthful first release.