@simbimbo/brainstem 0.0.1 → 0.0.3

package/docs/design-governance.md

@@ -0,0 +1,595 @@
+# brAInstem Design Governance
+
+_Status: canonical working governance doc for early product design and v0.0.1 execution_
+
+## Purpose
+
+This document exists to keep brAInstem from becoming a vague, overbuilt "AIOps platform" before it becomes a useful product.
+
+It is the design governor for:
+- what brAInstem is
+- what it is not
+- how we decide what belongs in early versions
+- how ingestion, attention, discovery, memory, and operator output should fit together
+- how to keep the product honest while still moving fast
+
+If a future feature, architecture idea, connector, or AI capability conflicts with this document, the burden of proof is on the new idea.
+
+---
+
+## 1. Core product thesis
+
+brAInstem is an **always-on operational memory runtime for weak signals**.
+
+Its job is to:
+1. continuously listen to operational event streams
+2. normalize noisy raw inputs into a canonical event stream
+3. assign and update **attention** over time
+4. suppress most inconsequential activity quickly and cheaply
+5. preserve enough evidence so small signals can earn more attention later
+6. surface only the meaningful "meat" of the stream to humans
+7. remember historically meaningful patterns so future weak signals can be compared against the past
+
+### Short version
+
+brAInstem remembers the weird operational stuff that monitoring systems do not escalate and humans do not have time to remember.
+
+### Even shorter version
+
+It tells operators what is quietly becoming important before it becomes an incident.
+
+---
+
+## 2. The problem brAInstem is solving
+
+Modern operational environments produce enormous amounts of low-grade noise:
+- recurring warnings
+- self-healing service failures
+- auth oddities
+- VPN/firewall flap patterns
+- restart storms
+- low-severity events that never page anyone
+- event clusters that matter only when viewed across time, hosts, or sources
+
+Most existing tools are optimized for one or more of the following:
+- hard failures
+- alerting thresholds
+- active incident response
+- event correlation around incidents
+- generic observability queries
+
+But there is still a gap around the **forgotten operational middle**:
+- patterns too weak to alert on today
+- patterns too repetitive for humans to remember accurately
+- patterns too individually small to justify immediate investigation
+- patterns that later become tickets or outages
+
+brAInstem is designed to fill that gap.
+
+---
+
+## 3. Category position
+
+brAInstem is **not** best described as a generic AIOps platform.
+
+That category is crowded, vague, and easy to disappear inside.
+
+### Preferred category language
+- operational memory engine
+- weak-signal operational memory
+- pre-incident pattern memory
+- always-on operational attention engine
+
+### Anti-positioning
+Do **not** lead with:
+- "AIOps platform"
+- "AI observability"
+- "chat with all your logs"
+- "generic event correlation"
+- "autonomous operations"
+
+### Why
+Those descriptions flatten the most differentiated part of brAInstem:
+its ability to remember and score weak, recurring, self-healing, pre-threshold operational patterns across time.
+
+---
+
+## 4. Primary buyer and beachhead
+
+### First target buyer
+- MSPs
+
+### Secondary buyers
+- lean NOC teams
+- infrastructure/SRE teams with persistent alert fatigue
+- operations-heavy consultants who repeatedly see the same patterns across environments
+
+### Why MSPs first
+MSPs are a strong beachhead because they:
+- manage many customer environments with recurring pattern overlap
+- suffer from alert fatigue and technician memory loss
+- need to reduce surprise tickets
+- benefit directly from surfacing weak recurring issues before customers feel them
+- can monetize proactive identification of trouble patterns
+
+### Buyer-value language
+brAInstem should help buyers:
+- reduce surprise tickets
+- identify recurring self-healing issues earlier
+- preserve senior technician intuition as reusable memory
+- make junior technicians more effective
+- surface what to check before users notice symptoms
+
+---
+
+## 5. Product form
+
+brAInstem should be designed as a **self-contained runtime**, not just an analytics script.
+
+### Required properties
+- always listening
+- self-contained
+- source-installable
+- OS-agnostic as far as practical
+- dependency-self-sufficient where possible
+- suitable for local, edge, or on-prem deployment
+
+### What this means in practice
+The product should eventually include:
+- input apparatus
+- normalization apparatus
+- discovery apparatus
+- memory/store apparatus
+- operator-facing output apparatus
+
+The product is not just a batch analyzer. It is a continuously operating system for operational attention.
+
+---
+
+## 6. Non-negotiable day-1 principle: robust input apparatus
+
+The input apparatus must be robust from the first serious release.
+
+If ingestion is flaky, everything above it is untrustworthy.
+
+### Day-1 robustness requirements
+The intake layer must:
+- never lose data silently
+- survive malformed input
+- preserve provenance
+- support bursty input without collapsing downstream logic
+- expose queue depth / health / parse failure visibility
+- preserve enough raw evidence for audit and replay
+- decouple intake from heavy processing using durable buffering/spooling
+
+### Robust does not mean universal on day one
+We should distinguish between:
+- **robustness**: strong ingestion behavior under ugly real conditions
+- **breadth**: number of supported source types
+
+Robustness is non-negotiable on day one.
+Breadth expands over time through adapter contracts.
+
+---
+
+## 7. Universal input philosophy
+
+brAInstem should be architected so that many source types can eventually feed one canonical internal stream.
+
+### Examples of desired source classes
+- syslog
+- file tails
+- JSON logs
+- structured webhooks
+- monitoring/alert APIs
+- vendor event payloads
+- streaming inputs
+- audit/event exports
+
+### Universal does **not** mean
+- bespoke first-class support for every source in v0.0.1
+- a giant connector zoo before the core event model is stable
+- turning brAInstem into a generic log shipper
+
+### Universal **does** mean
+- a stable raw envelope contract
+- a stable canonical event contract
+- adapters that map input into those contracts cleanly
+- downstream discovery logic that is source-agnostic once normalization is complete
+
+---
+
+## 8. Canonical stream of consciousness
+
+The system should internally treat all normalized events as part of one continuous operational stream of consciousness.
+
+That does **not** mean one giant undifferentiated blob.
+It means all events, regardless of source, should become compatible units inside one shared attention/discovery pipeline.
+
+### Canonical pipeline concept
+1. raw input arrives
+2. raw input is wrapped in a provenance-preserving envelope
+3. parser/canonicalizer emits canonical events
+4. canonical events are normalized / deduped / baseline-adjusted
+5. attention is assigned
+6. attention evolves over time based on recurrence/spread/context/history
+7. only the meaningful material is surfaced to human review or promoted into memory
+
+This canonical stream is the substrate of the product.
+
+---
+
+## 9. Attention is the central primitive
+
+This is one of the most important design rules in the entire product.
+
+brAInstem should not operate on a naive binary:
+- keep
+n- drop
+
+Instead, it should operate on **attention**.
+
+### Why
+A lot of operationally meaningful patterns begin as tiny, inconsequential-looking events.
+A single event often means nothing.
+A recurring, spreading, or historically familiar version of that same event may matter a lot.
+
+### Therefore
+Most events should not be hard-dropped immediately unless they are clearly worthless.
+Instead, they should be assigned an initial attention score and routed according to that score.
+
+### Attention should be:
+- cheap to assign initially
+- dynamic over time
+- evidence-based
+- inspectable
+- auditable
+- promotable
+
+### Attention is not just a score
+It is a product behavior model.
+Attention determines:
+- retention depth
+- compute budget
+- routing behavior
+- whether humans see something
+- whether something enters long-term memory
+
+---
+
+## 10. Attention bands
+
+brAInstem should use explicit attention bands rather than one magical opaque score.
+
+A suggested early model:
+
+### 0. Ignore-fast
+Characteristics:
+- trivial or clearly low-value noise
+- low novelty
+- low recurrence significance
+- low spread
+- low historical relevance
+
+Behavior:
+- aggregate counters only or very short retention
+- do not send to deep discovery
+- do not surface to humans
+
+### 1. Background
+Characteristics:
+- probably not meaningful alone
+- worth retaining at low cost
+- may matter if repeated later
+
+Behavior:
+- keep counts/fingerprint statistics
+- short evidence retention
+- eligible for score growth through recurrence/time
+
+### 2. Watch
+Characteristics:
+- weak but plausibly meaningful
+- repeated enough or novel enough to monitor
+
+Behavior:
+- maintain context window
+- keep more evidence
+- eligible for candidate formation
+
+### 3. Investigate
+Characteristics:
+- likely meaningful weak signal
+- strong enough to justify active discovery logic now
+
+Behavior:
+- full candidate generation
+- historical comparison
+- richer explanation generation
+
+### 4. Promote
+Characteristics:
+- meaningfully important
+- digest-worthy
+- review-worthy
+- incident-memory-worthy
+
+Behavior:
+- surface to humans
+- optionally persist as durable memory / incident pattern
+
+### Design rule
+A low-attention event must be able to **earn more attention later**.
+
+---
+
+## 11. Discovery apparatus
+
+The discovery apparatus is the part of brAInstem that turns canonical events plus accumulated attention into operationally meaningful signals.
+
+### The discovery apparatus should:
+- review events in near-real-time
+- consume normalized event/signal state, not just raw payloads
+- operate cheaply on the common path
+- reserve heavier analysis for events/signatures/candidates that have earned attention
+
+### Candidate classes that matter early
+- recurrence candidate
+- burst candidate
+- self-heal candidate
+- spread candidate
+- precursor candidate
+
+### Core design rule
+Do not waste expensive reasoning on the whole stream.
+Let the stream earn deeper review.
+
+---
+
+## 12. The "meat" principle
+
+Most raw operational data is not the product.
+The product is the meaningful reduction of that data.
+
+brAInstem should be explicitly designed to:
+- suppress the junk quickly
+- preserve enough evidence for audit
+- retain enough weak material for attention growth
+- promote only what actually matters
+
+### The "meat" is:
+- recurring weak signals
+- self-healing instability
+- growing or spreading oddities
+- patterns with historical precedent
+- patterns that deserve a technician's eyes
+
+If the system simply captures everything and summarizes indiscriminately, it has failed.
+
+---
+
+## 13. Output is the product
+
+The internal pipeline matters, but operator-facing output is what creates value.
+
+Every surfaced item should aim to answer:
+- what is happening?
+- why does it matter?
+- how often / where is it happening?
+- have we seen this before?
+- what should a human check next?
+
+### Early output surfaces
+- interesting items list
+- attention queue
+- weak-signal digest
+- history recall for similar patterns
+
+### Anti-pattern
+Do not surface giant blobs of internal state, giant raw logs, or inscrutable score dumps and call that product.
+
+The output must feel like technician-grade operational guidance.
+
+---
+
+## 14. What brAInstem is not
+
+brAInstem is not, at least in its early phases:
+- a full SIEM
+- a generic observability backend
+- a generic log shipper
+- a full root-cause engine
+- a complete event correlation platform for all use cases
+- a chatbot over arbitrary logs
+- a replacement for monitoring, alerting, and dashboards
+
+It can integrate with or complement those things.
+It should not try to become all of them at once.
+
+---
+
+## 15. v0.0.1 release philosophy
+
+A real v0.0.1 should be **honest, narrow, and usable**.
+
+It should not pretend the universal always-on intake appliance is fully complete if it is not.
+
+### v0.0.1 should prove:
+- the category framing is coherent
+- the canonical event model exists
+- the attention model exists
+- a small set of input types can feed one normalized stream
+- recurrence/interesting-item output is real
+- the product can emit something operator-meaningful
+
+### v0.0.1 should not claim:
+- full universal source coverage
+- fully mature always-on distributed ingestion
+- enterprise-grade multi-tenancy
+- full discovery sophistication
+- full memory retrieval maturity
+
+The point of v0.0.1 is to put a truthful first stake in the ground.
+
+---
+
+## 16. Recommended exact v0.0.1 shape
+
+### In scope
+- self-contained local runtime / prototype framing
+- canonical event schema
+- narrow but real ingestion path(s)
+- recurrence/interesting-item generation
+- early attention scoring language and behavior
+- SQLite persistence
+- demo or CLI path that proves end-to-end flow
+- documentation that explains the architecture honestly
+
+### Out of scope
+- broad connector ecosystem
+- complete syslog appliance behavior
+- production-grade multi-tenant orchestration
+- heavy UI work
+- cloud dependency sprawl
+- black-box AI-heavy ranking
+
+### Honest release description
+brAInstem 0.0.1 is an experimental operational memory prototype that ingests early source types, normalizes them into a canonical stream, assigns attention, detects recurring weak signals, and emits concise operator-facing interesting items from a self-contained local runtime.
+
+---
+
+## 17. Day-1 architecture priority order
+
+The right implementation spine is:
+
+1. **Canonical event model**
+2. **Adapter/raw envelope contract**
+3. **Input apparatus**
+4. **Normalization + dedupe**
+5. **Attention scoring**
+6. **Discovery/candidate generation**
+7. **Output/interesting items/digest**
+8. **Broader connectors**
+
+### Why this order matters
+If we build connectors before the event model and attention model are stable, we will create a swamp of special cases.
+
+---
+
+## 18. Dependency strategy
+
+The dependency strategy should follow these rules:
+
+### Prefer
+- widely used Python runtime/web/schema packages
+- small, understandable libraries
+- dependencies that can be installed from source
+- offline-friendly/self-contained deployment assumptions
+
+### Avoid early
+- giant vendor SDK dependency sprawl
+- dependencies that imply cloud lock-in
+- systems that require distributed infra for basic operation
+- magic ML dependencies before the baseline rules are proven useful
+
+### General principle
+Use packages to accelerate protocol handling and schema validation, not to outsource the core product thesis.
+
+---
+
+## 19. Auditability and trust
+
+Every significant system behavior must be explainable enough for operators and builders to trust it.
+
+This applies especially to:
+- parse failures
+- dropped/suppressed events
+- attention changes
+- candidate promotion
+- digest inclusion
+
+### Early trust requirements
+The system should be able to explain:
+- why something was ignored
+- why something stayed background
+- why something was promoted
+- why something surfaced now instead of earlier
+
+If the system cannot explain its own behavior, it is not ready for operational trust.
+
+---
+
+## 20. Biggest design mistakes to avoid
+
+### Mistake 1
+Building a connector zoo before freezing the canonical event model.
+
+### Mistake 2
+Treating adapters as arbitrary plugin emitters rather than strict envelope emitters.
+
+### Mistake 3
+Conflating robustness with breadth.
+
+### Mistake 4
+Using opaque AI scoring before transparent attention logic exists.
+
+### Mistake 5
+Dropping tiny signals too aggressively and losing the very patterns the product exists to catch.
+
+### Mistake 6
+Treating dashboards as the product before the outputs are actually useful.
+
+### Mistake 7
+Shipping category language that overclaims beyond what the system truly does.
+
+---
+
+## 21. Working decision framework
+
+When deciding whether a feature belongs now, ask:
+
+1. Does this improve the canonical stream of consciousness?
+2. Does this improve attention assignment or attention evolution?
+3. Does this make the input apparatus more robust?
+4. Does this make operator output more useful?
+5. Does this preserve self-contained, source-installable deployment?
+6. Does this tighten the product wedge for MSP/lean ops weak-signal detection?
+
+If the answer is mostly no, it probably does not belong yet.
+
+---
+
+## 22. Recommended near-term sequence
+
+### Immediate design work
+1. write explicit `v0.0.1` scope doc
+2. revise architecture doc around raw envelope → canonical event → attention → discovery → output
+3. define adapter contract
+4. define attention bands and routing behavior
+5. define operator output contract
+
+### Immediate implementation work after that
+1. make the current prototype language align with attention
+2. make the demo output read like product
+3. ensure tests/run-path are reliable in a real env
+4. cut an honest `0.0.1`
+
+### Next milestone after `0.0.1`
+Build the first truly robust always-on intake runtime:
+- syslog UDP/TCP
+- HTTP ingest/batch
+- durable spool
+- parse/decode observability
+- source health/status surfaces
+
+---
+
+## 23. Final governing statement
+
+brAInstem wins if it becomes the system that:
+- listens continuously
+- forgets the right things cheaply
+- remembers the right weird things accurately
+- and tells a human what is quietly becoming important before it becomes painful
+
+Everything else is secondary.

package/docs/mvp-flow.md

@@ -0,0 +1,109 @@
+# MVP Flow
+
+> This document is subordinate to `design-governance.md`. If the MVP flow here drifts beyond the governance boundaries, the governance doc wins.
+
+## Goal
+
+Build the smallest possible brAInstem flow that proves the product thesis:
+- logs can be remembered
+- weak signals can be scored
+- recurring self-resolving issues can be surfaced before traditional alerts
+
+## MVP source types
+
+Start with:
+- syslog files
+- line-oriented service logs
+- auth logs
+- firewall / VPN logs
+- LogicMonitor alert/event payloads via a simple connector contract
+
+## MVP processing flow
+
+### 1. Ingest
+Read raw lines from a source file or stream.
+
+Example input:
+- VPN rekey failure
+- SSH auth failure burst
+- repeated service restart line
+
+### 2. Normalize
+Convert each line into a standard event:
+- tenant
+- asset
+- host
+- service
+- timestamp
+- severity
+- message_raw
+- message_normalized
+
+### 3. Fingerprint
+Map normalized messages into signatures.
+
+Example:
+- many slightly different VPN errors -> one signature family
+
+### 4. Candidate generation
+Create candidates when patterns emerge:
+- recurrence candidate
+- self-heal candidate
+- spread candidate
+- burst candidate
+
+### 5. Score
+Compute human-significance score using:
+- recurrence
+- recovery
+- spread
+- novelty
+- temporal correlation
+- human impact likelihood
+- precursor likelihood
+- memory weight
+
+### 6. Surface
+Produce:
+- daily digest item
+- candidate explanation
+- related prior history
+
+### 7. Promote
+If a candidate repeatedly matters, promote it into incident memory.
+
+## First killer demo
+
+### Scenario
+Recurring self-healing VPN instability.
+
+Input pattern:
+- tunnel drops briefly
+- recovers before alert threshold
+- happens 10+ times over several days
+
+Desired output:
+- a digest item surfaces it
+- score explains why it matters
+- prior similar incident memory is linked if available
+- raw log evidence is still drillable
+
+## MVP success criteria
+
+A successful MVP should:
+- ingest raw syslog-like events
+- cluster similar weak signals
+- produce an explainable candidate score
+- surface at least one meaningful item that classic threshold alerting would miss
+- support a concise operator digest
+
+## Non-goals for MVP
+
+Do not try to solve:
+- full SIEM workflows
+- complete root-cause analysis
+- all log formats
+- all tenants and data sources at once
+- perfect semantic understanding
+
+The MVP should prove the weak-signal memory model works.