smart_brain 0.1.2 → 0.3.0

data/README.en.md

@@ -1,173 +1,206 @@
-# SmartBrain
-
-SmartBrain is an Agent Memory Runtime and Context Composer.
-
-Core responsibilities:
-- `commit_turn`: persist event truth and structured memory
-- `compose_context`: build minimal, sufficient context per turn
-- integrate with SmartRAG: SmartBrain handles conversation memory, SmartRAG handles resource retrieval
-
-## Current Progress
-
-The repository now includes a runnable v0.1 flow with:
-- `commit_turn` / `compose_context` end-to-end pipeline
-- retention / consolidation / retrieval / composition policies
-- retrievers: exact + relational
-- fusion: dedupe, rule-based rerank, diversity, budget truncation
-- SmartRAG adapters: `NullClient`, `HttpClient`, `DirectClient`
-- traceability via `request_id` / `plan_id` / `context_id`
-- RSpec coverage (unit + integration + regression)
-
-## Project Layout
-
-- `lib/smart_brain.rb`: public API
-- `lib/smart_brain/runtime.rb`: runtime orchestration
-- `lib/smart_brain/contracts/`: RetrievalPlan / EvidencePack / ContextPackage validation
-- `lib/smart_brain/observability/`: logs and metrics
-- `lib/smart_brain/event_store/`: event storage (in-memory currently)
-- `lib/smart_brain/memory_store/`: memory storage (in-memory currently)
-- `lib/smart_brain/retrievers/`: exact/relational retrievers
-- `lib/smart_brain/fusion/`: multi-source fusion
-- `lib/smart_brain/context_composer/`: context assembly
-- `lib/smart_brain/adapters/smart_rag/`: SmartRAG adapters
-- `config/brain.yml`: policy config
-- `example.rb`: SmartBrain + SmartAgent + SmartPrompt + SmartRAG demo
-- `docs/`: design and protocol documents
-
-## Installation
-
-```bash
-bundle install
-```
-
-If you hit permission/shared-gem issues:
-
-```bash
-bundle config set --local path 'vendor/bundle'
-bundle config set --local disable_shared_gems 'true'
-```
-
-## Quick Start (SmartBrain Only)
-
-```ruby
-require_relative 'lib/smart_brain'
-
-SmartBrain.configure
-
-SmartBrain.commit_turn(
-  session_id: 'demo',
-  turn_events: {
-    messages: [
-      { role: 'user', content: 'Remember this: default DB is Postgres.' },
-      { role: 'assistant', content: 'Saved.' }
-    ],
-    decisions: [
-      { key: 'decision:smartbrain:storage', decision: 'Use Postgres by default' }
-    ]
-  }
-)
-
-context = SmartBrain.compose_context(
-  session_id: 'demo',
-  user_message: 'Continue and summarize key points'
-)
-
-puts context[:context_id]
-puts context.dig(:debug, :trace, :request_id)
-puts context.dig(:debug, :trace, :plan_id)
-```
-
-## SmartRAG Integration Options
-
-### 1) NullClient (default)
-
-If no SmartRAG client is injected, resource evidence is empty.
-
-### 2) HttpClient
-
-```ruby
-transport = lambda do |plan, timeout_seconds:|
-  {
-    plan_id: 'p1',
-    supports_language_filter: true,
-    evidences: []
-  }
-end
-
-client = SmartBrain::Adapters::SmartRag::HttpClient.new(transport: transport, timeout_seconds: 2)
-SmartBrain.configure(smart_rag_client: client)
-```
-
-### 3) DirectClient (used in `example.rb`)
-
-```ruby
-require 'smart_rag'
-require_relative 'lib/smart_brain/adapters/smart_rag/direct_client'
-
-rag_config = SmartRAG::Config.load(ENV.fetch('SMARTRAG_CONFIG_PATH'))
-rag = SmartRAG::SmartRAG.new(rag_config)
-client = SmartBrain::Adapters::SmartRag::DirectClient.new(rag: rag)
-
-SmartBrain.configure(smart_rag_client: client)
-```
-
-## `example.rb` (Updated)
-
-The example demonstrates the real loop:
-1. SmartBrain `compose_context`
-2. SmartAgent calls SmartPrompt worker via `call_worker`
-3. SmartBrain `commit_turn`
-4. prints `evidence(memory/resource)` so you can verify SmartRAG participation
-
-Files used by the demo:
-- `config/example_agent.yml`
-- `config/example_llm.yml`
-- `agents/brain_assistant.rb`
-- `workers/brain_assistant.rb`
-- `templates/brain_assistant.erb`
-
-Run:
-
-```bash
-bundle exec ruby example.rb
-```
-
-## Core API
-
-### `SmartBrain.configure(config_path: nil, smart_rag_client: nil, clock: -> { Time.now.utc })`
-Initialize runtime and optionally inject a SmartRAG client.
-
-### `SmartBrain.commit_turn(session_id:, turn_events:)`
-Persist events, extract memory, resolve conflicts, update summary.
-
-### `SmartBrain.compose_context(session_id:, user_message:, agent_state: {})`
-Build a `ContextPackage` with planning and fused evidence.
-
-### `SmartBrain.diagnostics`
-Return observability snapshot for compose/commit logs and metrics.
-
-## Test
-
-```bash
-rspec
-```
-
-## Troubleshooting
-
-### 1) `cannot load such file -- sequel/extensions/pgvector`
-`example.rb` already applies compatibility handling (`Sequel.extension 'pgvector'` and strips `database.extensions` from DB connect config).
-
-### 2) `Config file not found: config/llm_config.yml`
-`example.rb` injects `config_path: ./config/example_llm.yml` for SmartRAG EmbeddingService startup.
-
-### 3) `ruby-lsp: not found`
-```bash
-gem install --user-install ruby-lsp debug
-```
-and ensure user gem `bin` is in `PATH`.
-
-## Roadmap
-
-- migrate EventStore/MemoryStore from in-memory to Postgres-backed implementations
-- integrate real reranker/embedding models
-- improve SmartRAG ingest pipeline and cross-session evaluation tooling
+# SmartBrain
+
+SmartBrain is an Agent Memory Runtime and Context Composer.
+
+Core responsibilities:
+- `commit_turn`: persist event truth and structured memory
+- `compose_context`: build minimal, sufficient context per turn
+- integrate with SmartRAG: SmartBrain handles conversation memory, SmartRAG handles resource retrieval
+
+## Current Progress
+
+The repository now includes a runnable v0.1 flow with:
+- `commit_turn` / `compose_context` end-to-end pipeline
+- retention / consolidation / retrieval / composition policies
+- retrievers: exact + relational
+- fusion: dedupe, rule-based rerank, diversity, budget truncation
+- SmartRAG adapters: `NullClient`, `HttpClient`, `DirectClient`
+- traceability via `request_id` / `plan_id` / `context_id`
+- RSpec coverage (unit + integration + regression)
+
+## Project Layout
+
+- `lib/smart_brain.rb`: public API
+- `lib/smart_brain/runtime.rb`: runtime orchestration
+- `lib/smart_brain/contracts/`: RetrievalPlan / EvidencePack / ContextPackage validation
+- `lib/smart_brain/observability/`: logs and metrics
+- `lib/smart_brain/event_store/`: event storage (in-memory currently)
+- `lib/smart_brain/memory_store/`: memory storage (in-memory currently)
+- `lib/smart_brain/retrievers/`: exact/relational retrievers
+- `lib/smart_brain/fusion/`: multi-source fusion
+- `lib/smart_brain/context_composer/`: context assembly
+- `lib/smart_brain/adapters/smart_rag/`: SmartRAG adapters
+- `config/brain.yml`: policy config
+- `example.rb`: SmartBrain + SmartAgent + SmartPrompt + SmartRAG demo
+- `docs/`: design and protocol documents
+- `docs/media_memory_schema.md`: media-memory contract, production controls, and SmartRAG migration requirements (Chinese)
+
+## Installation
+
+```bash
+bundle install
+```
+
+If you hit permission/shared-gem issues:
+
+```bash
+bundle config set --local path 'vendor/bundle'
+bundle config set --local disable_shared_gems 'true'
+```
+
+## Quick Start (SmartBrain Only)
+
+```ruby
+require_relative 'lib/smart_brain'
+
+SmartBrain.configure
+
+SmartBrain.commit_turn(
+  session_id: 'demo',
+  turn_events: {
+    messages: [
+      { role: 'user', content: 'Remember this: default DB is Postgres.' },
+      { role: 'assistant', content: 'Saved.' }
+    ],
+    decisions: [
+      { key: 'decision:smartbrain:storage', decision: 'Use Postgres by default' }
+    ]
+  }
+)
+
+context = SmartBrain.compose_context(
+  session_id: 'demo',
+  user_message: 'Continue and summarize key points'
+)
+
+puts context[:context_id]
+puts context.dig(:debug, :trace, :request_id)
+puts context.dig(:debug, :trace, :plan_id)
+```
+
+## SmartRAG Integration Options
+
+### 1) NullClient (default)
+
+If no SmartRAG client is injected, resource evidence is empty.
+
+### 2) HttpClient
+
+```ruby
+transport = lambda do |plan, timeout_seconds:|
+  {
+    plan_id: 'p1',
+    supports_language_filter: true,
+    evidences: []
+  }
+end
+
+client = SmartBrain::Adapters::SmartRag::HttpClient.new(transport: transport, timeout_seconds: 2)
+SmartBrain.configure(smart_rag_client: client)
+```
+
+For the built-in SmartRAG HTTP API, configure the remote endpoint and Bearer token directly:
+
+```ruby
+client = SmartBrain::Adapters::SmartRag::HttpClient.for_url(
+  base_url: 'http://127.0.0.1:9393',
+  timeout_seconds: 30,
+  headers: { 'Authorization' => "Bearer #{ENV.fetch('SMARTRAG_TOKEN')}" }
+)
+SmartBrain.configure(smart_rag_client: client)
+```
+
+### 3) DirectClient (used in `example.rb`)
+
+```ruby
+require 'smart_rag'
+require_relative 'lib/smart_brain/adapters/smart_rag/direct_client'
+
+rag_config = SmartRAG::Config.load(ENV.fetch('SMARTRAG_CONFIG_PATH'))
+rag = SmartRAG::SmartRAG.new(rag_config)
+client = SmartBrain::Adapters::SmartRag::DirectClient.new(rag: rag)
+
+SmartBrain.configure(smart_rag_client: client)
+```
+
+## Media Queue Production Contract
+
+SmartBrain delegates image, audio, and video persistence to SmartRAG. Before deploying the current queue code, migrate the **SmartRAG database** through `017_add_media_job_request_fingerprint`; running `smart_brain migrate` does not apply SmartRAG migrations. Migrations 015 and 016 add leases, shared S3/MinIO object lifecycle, staging protection, and principal-owned documents. Migration 017 backfills and requires a canonical SHA-256 request fingerprint.
+
+Use a stable idempotency key for asynchronous retries:
+
+```ruby
+job = SmartBrain.enqueue_media(
+  source: '/data/product-demo.mp4',
+  options: { media_type: 'video', idempotency_key: 'product-demo-v1' }
+)
+```
+
+For the same authenticated principal, repeating the same operation, source, and canonicalized options returns the original job with `deduplicated: true`. Reusing the key with a different payload returns SmartRAG HTTP `409` and `code: "idempotency_conflict"`. Keep both the original key and payload for network retries; generate a new key when the business payload changes.
+
+`HttpClient` converts a non-2xx response into a SmartBrain result with `status: "failed"`; an idempotency conflict is visible in `warnings` as `SmartRAG HTTP 409`. `DirectClient` propagates `SmartRAG::Core::MediaJobQueue::IdempotencyConflict` to the caller.
+
+Authenticated SmartRAG retrieval uses defense in depth: it pushes the principal's PostgreSQL-owned document IDs into search and rechecks returned candidates before emitting evidence. Invalid Bearer tokens return 401. For multi-instance workers, configure a shared S3 or MinIO endpoint, bucket, and prefix; retained failed jobs protect their staging objects until pruning.
+
+SmartRAG's opt-in real integration suites verify MinIO cross-instance storage and deletion, asynchronous worker materialization, failed-job GC protection, Bearer-token tenant isolation, HTTP 409 behavior, and concurrent idempotency races. See the SmartRAG README section "Real storage and isolation verification" for environment variables and commands.
+
+## `example.rb` (Updated)
+
+The example demonstrates the real loop:
+1. SmartBrain `compose_context`
+2. SmartAgent calls SmartPrompt worker via `call_worker`
+3. SmartBrain `commit_turn`
+4. prints `evidence(memory/resource)` so you can verify SmartRAG participation
+
+Files used by the demo:
+- `config/example_agent.yml`
+- `config/example_llm.yml`
+- `agents/brain_assistant.rb`
+- `workers/brain_assistant.rb`
+- `templates/brain_assistant.erb`
+
+Run:
+
+```bash
+bundle exec ruby example.rb
+```
+
+## Core API
+
+### `SmartBrain.configure(config_path: nil, smart_rag_client: nil, clock: -> { Time.now.utc })`
+Initialize runtime and optionally inject a SmartRAG client.
+
+### `SmartBrain.commit_turn(session_id:, turn_events:)`
+Persist events, extract memory, resolve conflicts, update summary.
+
+### `SmartBrain.compose_context(session_id:, user_message:, agent_state: {})`
+Build a `ContextPackage` with planning and fused evidence.
+
+### `SmartBrain.diagnostics`
+Return observability snapshot for compose/commit logs and metrics.
+
+## Test
+
+```bash
+rspec
+```
+
+## Troubleshooting
+
+### 1) `cannot load such file -- sequel/extensions/pgvector`
+`example.rb` already applies compatibility handling (`Sequel.extension 'pgvector'` and strips `database.extensions` from DB connect config).
+
+### 2) `Config file not found: config/llm_config.yml`
+`example.rb` injects `config_path: ./config/example_llm.yml` for SmartRAG EmbeddingService startup.
+
+### 3) `ruby-lsp: not found`
+```bash
+gem install --user-install ruby-lsp debug
+```
+and ensure user gem `bin` is in `PATH`.
+
+## Roadmap
+
+- migrate EventStore/MemoryStore from in-memory to Postgres-backed implementations
+- integrate real reranker/embedding models
+- improve SmartRAG ingest pipeline and cross-session evaluation tooling