crow-cli 0.1.37__tar.gz → 0.1.38__tar.gz

{crow_cli-0.1.37 → crow_cli-0.1.38}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: crow-cli
-Version: 0.1.37
+Version: 0.1.38
 Summary: Add your description here
 Project-URL: Homepage, https://github.com/crow-cli/crow-cli
 Project-URL: Repository, https://github.com/crow-cli/crow-cli
@@ -9,6 +9,7 @@ Project-URL: Documentation, https://github.com/crow-cli/crow-cli#readme
 License-File: LICENSE.md
 Requires-Python: >=3.14
 Requires-Dist: agent-client-protocol[http]<0.13.0,>=0.12.0
+Requires-Dist: boto3>=1.40.0
 Requires-Dist: coolname>=2.2.0
 Requires-Dist: directory-tree>=1.0.0
 Requires-Dist: docker>=7.2.0
@@ -19,6 +20,7 @@ Requires-Dist: jinja2>=3.1.6
 Requires-Dist: markdownify>=0.14.0
 Requires-Dist: openai>=2.21.0
 Requires-Dist: opencv-python>=4.13.0.92
+Requires-Dist: psycopg[binary]>=3.1
 Requires-Dist: pydantic>=2.12.5
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: readabilipy>=0.3.0
@@ -122,15 +124,15 @@ The agent detects client capabilities (terminals, file read/write) and uses the
 
 An ACP-native agent: a streaming ReAct loop with tool calling, cancellation, conversation compaction, and multimodal input. Provider and model configuration lives in `~/.agents/crow/config.yaml`.
 
-### Persistence — sqlite memory
+### Persistence — SQL memory (sqlite or PostgreSQL)
 
-Sessions persist to a single sqlite database (`~/.agents/crow/crow.db`, schema v5, WAL mode) with an FTS5 index for BM25 keyword search. Images in messages are written to `~/.agents/crow/images/` and referenced by path; they are hydrated to base64 data URLs only when the conversation is sent to the LLM. The same database backs the memory API, exposed to agents as three tools:
+Sessions persist to a SQL database reached via the `db_uri` in `config.yaml`. The default is a single sqlite file (`~/.agents/crow/crow.db`, schema v5, WAL mode) with an FTS5 index for BM25 keyword search — zero setup, one file. Point `db_uri` at PostgreSQL instead (`postgresql+psycopg://…`; a `postgres` service ships in `compose.yaml`) and the same schema runs there for memory shared across machines — agents on any box see the same sessions, messages and task mailboxes. The dialect seams are isolated: keyword search is FTS5/BM25 on sqlite and tsvector/GIN on postgres behind the same `search_messages` contract, and the MCP read-only path is enforced per dialect (sqlite `mode=ro` file URI, postgres session `READ ONLY`). Images never live in the database: they are content-addressed (`<sha256hex><ext>`, so duplicates dedupe for free) in an image store and referenced by key; they are hydrated to base64 data URLs only when the conversation is sent to the LLM. The default store is the filesystem (`~/.agents/crow/images/`). Optionally, point `image_store.s3` in `config.yaml` at an S3 endpoint (a RustFS service ships in `compose.yaml`) and images go there instead — crow probes the endpoint once at startup and falls back to the filesystem when it is down, and reads always fall back to the filesystem too, so images stored before the switch keep working. The same database backs the memory API, exposed to agents as three tools:
 
 - `list_sessions()` — sessions ordered by recent activity (who's working on what)
 - `query_memory(query)` — find which session discussed something, across all sessions
 - `query_session(session_id)` — read or search within one session (spans all of that session's agents)
 
-This is what makes multi-agent delegation work: launch a worker, then read its thoughts from any other agent. No service to run — the sqlite file is the integration point.
+This is what makes multi-agent delegation work: launch a worker, then read its thoughts from any other agent. On sqlite there is no service to run — the file is the integration point. On PostgreSQL the server is the integration point, and every agent on the network shares one authoritative memory.
 
 ### Built-in MCP tool server
 
@@ -199,7 +201,7 @@ uv run pytest tests/e2e           # live LLM calls (costs $)
 src/crow_cli/           the agent — ACP server, ReAct loop, CLI
 src/crow_cli/config/    config loading, defaults, overrides (shared by every layer)
 src/crow_cli/mcp/       built-in MCP tool server (`crow-cli mcp`)
-src/crow_cli/memory/    shared SQL persistence (sqlite default, postgres-ready)
+src/crow_cli/memory/    shared SQL persistence (sqlite default, PostgreSQL supported)
 ```
 
 ## License

crow_cli-0.1.38/PLAN.md

@@ -0,0 +1,93 @@
+# PLAN — PostgreSQL memory backend (same treatment as RustFS) — COMPLETE 2026-08-26
+
+## **DO NOT ASK USER FOR FEEDBACK — THIS IS THE USER FEEDBACK.**
+## **DO NOT ASK USER FOR NEXT STEPS — THESE ARE THE NEXT STEPS.**
+
+Build/test gate (every phase boundary):
+`./run_tests.sh tests/unit tests/memory tests/mcp tests/integration -q`
+(rustfs sprint ended at 446 green on this gate; live-LLM e2e excluded.)
+
+Trajectory: 1 → 2 → 3 → 4 → 5. Commit after each phase, Session-Id trailer.
+
+## Phase 1 — mechanical portability — DONE 2026-08-26
+
+1. models.py: `from sqlalchemy import JSON` (drop the sqlite dialect import).
+2. db.py `_require_v5`: `sqlalchemy.inspect(engine).get_columns("agents")`
+   instead of `PRAGMA table_info` — portable, same fail-fast on v4 DBs.
+3. pyproject: `psycopg[binary]` dep; `uv --project . sync`.
+- Verify: gate green (sqlite behavior untouched). Commit.
+
+## Phase 2 — the FTS seam — DONE 2026-08-26
+
+1. New `src/crow_cli/memory/fts.py`: `create_fts(conn, engine)`,
+   `insert_fts(conn, engine, row_id, agent_id, role, fork_idx, text)`,
+   `search_fts(conn, engine, query, limit) -> list[(rowid, rank)]`
+   (best-first, lower=better on BOTH backends — postgres negates ts_rank).
+   sqlite branch = exact current SQL; postgres branch =
+   `CREATE TABLE IF NOT EXISTS messages_fts (rowid BIGINT PRIMARY KEY,
+   tsv tsvector)` + GIN index; `to_tsvector('simple', :t)` insert;
+   `plainto_tsquery('simple', :q)` + `ts_rank` search.
+2. db.py create_database → fts.create_fts (drop FTS_DDL constant).
+3. writes.py add_message → fts.insert_fts.
+4. reads.py search_messages → fts.search_fts (drop inline MATCH SQL).
+- Verify: gate green — sqlite search tests pass unchanged. Commit.
+
+## Phase 3 — MCP dialect awareness — DONE 2026-08-26
+
+1. db.py: `get_ro_engine(db_uri)` — sqlite → `?mode=ro&uri=true` rewrite;
+   postgres → connect listener `SET SESSION CHARACTERISTICS AS TRANSACTION
+   READ ONLY`. Export from memory/__init__.
+2. mcp/memory/store.py `_ro_engine`: keep sqlite file-existence check,
+   delegate engine creation to get_ro_engine.
+- Verify: gate green + new unit tests (sqlite rewrite unchanged; postgres
+  branch registers the listener — engine creation needs no live server).
+  Commit.
+
+## Phase 4 — init/compose/config (the rustfs treatment) — DONE 2026-08-26
+
+1. defaults.py COMPOSE_YAML: postgres service — postgres:17-alpine,
+   `${POSTGRES_PORT}:5432`, POSTGRES_USER/PASSWORD/DB env refs,
+   pg_isready healthcheck, postgres_data volume.
+2. init_cmd.py Step 2c (mirror 2b): --yes → install; YES_INSTALL_POSTGRES;
+   else Confirm.ask. env_vars: POSTGRES_PORT=5432, POSTGRES_USER=crow,
+   POSTGRES_DB=crow, POSTGRES_PASSWORD=<env or secrets.token_hex(16)>.
+   setup_postgres → db_uri = postgresql+psycopg://${POSTGRES_USER}:
+   ${POSTGRES_PASSWORD}@localhost:${POSTGRES_PORT}/${POSTGRES_DB};
+   compose writer includes postgres; review + done panels reflect backend.
+3. config.py apply_config_overrides: resolve_env_vars on db_uri.
+4. CONFIG_YAML template: commented postgres db_uri example.
+- Verify: tests/unit/test_init_postgres.py (mirror test_init_rustfs.py) +
+  gate green. Commit.
+
+## Phase 5 — live e2e + docs — DONE 2026-08-26
+
+1. scripts/e2e_postgres_live.py: ephemeral postgres:17-alpine container
+   (port 15432, wait pg_isready); create_database on the postgres URI;
+   add_message → search_messages round trip (FTS works, lower=better);
+   JSON data round trip byte-identical; claim_deliveries with 2 concurrent
+   claimers → exactly one wins; get_ro_engine SELECT ok / INSERT raises;
+   kill container in finally; print E2E-POSTGRES-OK.
+2. README persistence section: postgres option + when to use it
+   (multi-machine state). memory/__init__.py docstring: both backends now.
+- Verify: script passes against live container; gate green; commit.
+
+## Evidence log
+
+- Phase 1 (58061203, 2026-08-26): generic JSON + inspector v5 check +
+  psycopg 3.3.4 installed; gate 446 passed.
+- Phase 2 (73823339): memory/fts.py seam; sqlite search behavior unchanged —
+  gate 446 passed.
+- Phase 3 (80e72807): get_ro_engine dialect-aware; tests/memory/test_db.py
+  (sqlite write actually refused by OS); gate 450 passed.
+- Phase 4 (0de9c278): compose service + wizard Step 2c + db_uri env
+  resolution; tests/unit/test_init_postgres.py (5 tests); rendered compose
+  passes `docker compose config` with live .env interpolation; gate 455.
+- Phase 5 (this commit): scripts/e2e_postgres_live.py → E2E-POSTGRES-OK
+  against a REAL postgres:17-alpine container: schema+GIN index created,
+  tsvector search round trip (lower=better), JSON round trip identical,
+  2 concurrent claimers → exactly one delivery claimed, RO engine SELECT ok
+  / INSERT refused by the server. README persistence section rewritten.
+  Gate 455 passed.
+
+Sprint complete: db_uri picks the backend — postgres config → shared
+postgres; absent → sqlite + local FS, exactly as before.

{crow_cli-0.1.37 → crow_cli-0.1.38}/README.md

@@ -93,15 +93,15 @@ The agent detects client capabilities (terminals, file read/write) and uses the
 
 An ACP-native agent: a streaming ReAct loop with tool calling, cancellation, conversation compaction, and multimodal input. Provider and model configuration lives in `~/.agents/crow/config.yaml`.
 
-### Persistence — sqlite memory
+### Persistence — SQL memory (sqlite or PostgreSQL)
 
-Sessions persist to a single sqlite database (`~/.agents/crow/crow.db`, schema v5, WAL mode) with an FTS5 index for BM25 keyword search. Images in messages are written to `~/.agents/crow/images/` and referenced by path; they are hydrated to base64 data URLs only when the conversation is sent to the LLM. The same database backs the memory API, exposed to agents as three tools:
+Sessions persist to a SQL database reached via the `db_uri` in `config.yaml`. The default is a single sqlite file (`~/.agents/crow/crow.db`, schema v5, WAL mode) with an FTS5 index for BM25 keyword search — zero setup, one file. Point `db_uri` at PostgreSQL instead (`postgresql+psycopg://…`; a `postgres` service ships in `compose.yaml`) and the same schema runs there for memory shared across machines — agents on any box see the same sessions, messages and task mailboxes. The dialect seams are isolated: keyword search is FTS5/BM25 on sqlite and tsvector/GIN on postgres behind the same `search_messages` contract, and the MCP read-only path is enforced per dialect (sqlite `mode=ro` file URI, postgres session `READ ONLY`). Images never live in the database: they are content-addressed (`<sha256hex><ext>`, so duplicates dedupe for free) in an image store and referenced by key; they are hydrated to base64 data URLs only when the conversation is sent to the LLM. The default store is the filesystem (`~/.agents/crow/images/`). Optionally, point `image_store.s3` in `config.yaml` at an S3 endpoint (a RustFS service ships in `compose.yaml`) and images go there instead — crow probes the endpoint once at startup and falls back to the filesystem when it is down, and reads always fall back to the filesystem too, so images stored before the switch keep working. The same database backs the memory API, exposed to agents as three tools:
 
 - `list_sessions()` — sessions ordered by recent activity (who's working on what)
 - `query_memory(query)` — find which session discussed something, across all sessions
 - `query_session(session_id)` — read or search within one session (spans all of that session's agents)
 
-This is what makes multi-agent delegation work: launch a worker, then read its thoughts from any other agent. No service to run — the sqlite file is the integration point.
+This is what makes multi-agent delegation work: launch a worker, then read its thoughts from any other agent. On sqlite there is no service to run — the file is the integration point. On PostgreSQL the server is the integration point, and every agent on the network shares one authoritative memory.
 
 ### Built-in MCP tool server
 
@@ -170,7 +170,7 @@ uv run pytest tests/e2e           # live LLM calls (costs $)
 src/crow_cli/           the agent — ACP server, ReAct loop, CLI
 src/crow_cli/config/    config loading, defaults, overrides (shared by every layer)
 src/crow_cli/mcp/       built-in MCP tool server (`crow-cli mcp`)
-src/crow_cli/memory/    shared SQL persistence (sqlite default, postgres-ready)
+src/crow_cli/memory/    shared SQL persistence (sqlite default, PostgreSQL supported)
 ```
 
 ## License

crow_cli-0.1.38/TODO.md

@@ -0,0 +1,73 @@
+# TODO — PostgreSQL memory backend (same treatment as RustFS)
+
+## **DO NOT ASK USER FOR FEEDBACK — THIS IS THE USER FEEDBACK.**
+## **DO NOT ASK USER FOR NEXT STEPS — THESE ARE THE NEXT STEPS.**
+
+Goal: `db_uri` decides the backend. A postgres `db_uri` in config → postgres
+(compose service offered by init, FTS via tsvector/GIN, MCP read-only path
+enforced per-dialect). Absent that → sqlite + local FS, exactly as today.
+Motivation: central agent state shared across machines (coast-after-3 + this
+box) — sqlite has no cross-host story; the task/delivery mailbox design is
+already row-level-safe under postgres.
+
+Prior sprint (image object store / RustFS) is COMPLETE — see git history
+86c1c474..f114d966. This file replaces it.
+
+## Items (unordered)
+
+- [x] models.py: swap `sqlalchemy.dialects.sqlite.JSON` → generic
+      `sqlalchemy.JSON` (no JSON-operator queries anywhere, so generic is
+      correct; keep schema dialect-free).
+- [x] db.py `_require_v5`: PRAGMA table_info is sqlite-only → use SQLAlchemy
+      inspector (portable).
+- [x] pyproject: add `psycopg[binary]` dependency (sync driver; engine is
+      sync SQLAlchemy throughout). URI form `postgresql+psycopg://`.
+- [x] FTS seam: new `memory/fts.py` owning ALL dialect-specific full-text
+      code — create_fts / insert_fts / search_fts. sqlite = FTS5+bm25
+      (unchanged behavior); postgres = messages_fts(rowid BIGINT PK, tsv
+      tsvector) + GIN index, to_tsvector('simple'), plainto_tsquery,
+      ts_rank NEGATED so the contract stays "lower = better, best first"
+      (mcp/memory/main.py:359 negates again for display — keep it working).
+      Wire into db.create_database, writes.add_message, reads.search_messages.
+- [x] MCP dialect awareness: `get_ro_engine` in memory/db.py — sqlite keeps
+      `?mode=ro&uri=true`; postgres gets a connect listener
+      `SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY`. store.py
+      `_ro_engine` delegates; sqlite file-existence check stays.
+- [x] defaults.py COMPOSE_YAML: postgres service (postgres:17-alpine,
+      POSTGRES_USER/PASSWORD/DB env refs, pg_isready healthcheck,
+      postgres_data volume) — same env-ref-credentials pattern as rustfs.
+- [x] init_cmd.py: Step 2c PostgreSQL (mirror Step 2b rustfs: --yes default,
+      YES_INSTALL_POSTGRES, Confirm.ask; POSTGRES_PORT/USER/DB/PASSWORD into
+      env_vars, password random-default via secrets; db_uri becomes
+      postgresql+psycopg://${...}@localhost:${POSTGRES_PORT}/${POSTGRES_DB};
+      compose writer includes postgres; review/done panels reflect it).
+- [x] config.py apply_config_overrides: resolve_env_vars on db_uri (load()
+      already resolves whole file at line 335).
+- [x] CONFIG_YAML template: db_uri comment shows the postgres option.
+- [x] Tests: tests/unit/test_init_postgres.py (mirror test_init_rustfs.py);
+      RO-engine dialect branch tests; fts seam tests (sqlite path = existing
+      suite; postgres path = live e2e tier).
+- [x] Live e2e: scripts/e2e_postgres_live.py — ephemeral postgres:17-alpine
+      container; create_database; add_message→search_messages round trip;
+      JSON data round trip; claim_deliveries exactly-once with 2 concurrent
+      claimers; RO engine refuses writes; print E2E-POSTGRES-OK.
+- [x] Docs: README persistence section (postgres option), memory/__init__.py
+      docstring ("except FTS5" is now false — both backends supported).
+- [x] Full suite green at every phase boundary; commit per phase with
+      Session-Id trailer.
+
+## Decisions (locked)
+
+- Generic `sqlalchemy.JSON`, not JSONB variant: no JSON-operator queries
+  exist; keep the schema dialect-free.
+- Postgres FTS mirrors the sqlite architecture (side table maintained from
+  Python in the same transaction) — NO trigger on messages.data, because
+  message_text() extraction logic lives in Python.
+- 'simple' ts config for keyword parity with FTS5 (no stemming surprises).
+- Score contract preserved: search_messages returns lower=better on both
+  backends (ts_rank negated).
+- No sqlite→postgres history migration tool this sprint (fresh postgres DB;
+  migration is a future task if wanted).
+- postgres compose service has NO custom network (default compose network is
+  fine; rustfs-network stays rustfs-only).
+## Status: COMPLETE 2026-08-26 — all items done; evidence in PLAN.md + commits 58061203..HEAD.

{crow_cli-0.1.37 → crow_cli-0.1.38}/pyproject.toml

@@ -1,11 +1,12 @@
 [project]
 name = "crow-cli"
-version = "0.1.37"
+version = "0.1.38"
 description = "Add your description here"
 readme = "README.md"
 requires-python = ">=3.14"
 dependencies = [
     "agent-client-protocol[http]>=0.12.0,<0.13.0",
+    "boto3>=1.40.0",
     "hypercorn>=0.17",
     "coolname>=2.2.0",
     "directory-tree>=1.0.0",
@@ -16,6 +17,7 @@ dependencies = [
     "markdownify>=0.14.0",
     "openai>=2.21.0",
     "opencv-python>=4.13.0.92",
+    "psycopg[binary]>=3.1",
     "pydantic>=2.12.5",
     "pyyaml>=6.0.3",
     "readabilipy>=0.3.0",
@@ -43,6 +45,7 @@ packages = ["src/crow_cli"]
 [dependency-groups]
 dev = [
     "ipython>=9.10.0",
+    "moto[s3,server]>=5.0.0",
     "pytest>=9.0.2",
     "pytest-asyncio>=0.23.0",
     "pyinstaller>=6.0.0",

crow_cli-0.1.38/scripts/e2e_image_store_live.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python
+"""Live e2e: image store against a REAL RustFS container.
+
+Boots an ephemeral rustfs/rustfs container (SNSD), then drives the real
+MemoryClient through the full seam:
+  1. S3 reachable  -> resolve_image_store picks HybridReadStore(S3, FS)
+  2. save a message with an inline base64 image -> object lands in the bucket
+  3. load + hydrate -> identical base64 comes back
+  4. legacy FS image (pre-placed) still hydrates through the hybrid read
+  5. container killed -> next init falls back to FsImageStore
+
+Usage:
+  uv --project . run python scripts/e2e_image_store_live.py
+"""
+
+import asyncio
+import base64
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+import boto3
+import httpx
+
+from crow_cli.agent.memory import MemoryClient
+from crow_cli.memory.image_store import FsImageStore, HybridReadStore
+
+PORT = 19000
+ENDPOINT = f"http://127.0.0.1:{PORT}"
+CONTAINER = "crow-rustfs-e2e"
+BUCKET = "crow-images"
+ACCESS_KEY = "crowe2e"
+SECRET_KEY = "crowe2e"
+
+PNG_BYTES = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 1, 2, 3, 4])
+PNG_B64 = base64.b64encode(PNG_BYTES).decode()
+
+
+def docker(*args: str) -> str:
+    return subprocess.run(
+        ["docker", *args], capture_output=True, text=True, check=True
+    ).stdout.strip()
+
+
+def start_rustfs() -> None:
+    subprocess.run(["docker", "rm", "-f", CONTAINER], capture_output=True)
+    docker(
+        "run", "-d", "--name", CONTAINER,
+        "-p", f"{PORT}:9000",
+        "-e", f"RUSTFS_ACCESS_KEY={ACCESS_KEY}",
+        "-e", f"RUSTFS_SECRET_KEY={SECRET_KEY}",
+        "-e", "RUSTFS_ADDRESS=0.0.0.0:9000",
+        "-e", "RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true",
+        "rustfs/rustfs:latest", "/data",
+    )
+    for _ in range(60):
+        try:
+            r = httpx.get(f"{ENDPOINT}/health", timeout=1.0)
+            if r.status_code < 500:
+                print(f"[ok] rustfs healthy at {ENDPOINT}")
+                return
+        except httpx.HTTPError:
+            pass
+        time.sleep(1)
+    raise SystemExit("[FAIL] rustfs never became healthy")
+
+
+def stop_rustfs() -> None:
+    subprocess.run(["docker", "rm", "-f", CONTAINER], capture_output=True)
+    print("[ok] container removed")
+
+
+def write_config(config_dir: Path) -> None:
+    (config_dir / "config.yaml").write_text(
+        f"db_uri: sqlite:///{config_dir / 'crow.db'}\n"
+        "image_store:\n"
+        "  s3:\n"
+        f"    endpoint: {ENDPOINT}\n"
+        f"    bucket: {BUCKET}\n"
+        f"    access_key: {ACCESS_KEY}\n"
+        f"    secret_key: {SECRET_KEY}\n"
+    )
+
+
+def image_msg() -> dict:
+    return {
+        "role": "tool",
+        "tool_call_id": "e2e-c1",
+        "content": [
+            {"type": "text", "text": "live rustfs drill"},
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/png;base64,{PNG_B64}"},
+            },
+        ],
+    }
+
+
+async def run() -> None:
+    tmp = Path(tempfile.mkdtemp(prefix="crow-rustfs-e2e-"))
+    write_config(tmp)
+
+    # ---- 1-3: S3 up — store choice, write into bucket, hydrate back ----
+    async with MemoryClient(config_dir=tmp) as client:
+        assert isinstance(client.image_store, HybridReadStore), (
+            f"expected HybridReadStore, got {type(client.image_store).__name__}"
+        )
+        print("[ok] resolve_image_store picked S3 (HybridReadStore)")
+
+        agent_id = "e2e-rustfs-1-1"
+        await client.create_agent(
+            agent_id=agent_id, session_id="e2e-rustfs", agent_idx=1,
+            cwd="/tmp", system_prompt="sp", tool_definitions=[],
+            request_params={}, model_identifier="m",
+        )
+        await client.add_message(agent_id, image_msg())
+
+        # object actually landed in the bucket (checked via independent client)
+        s3 = boto3.client(
+            "s3", endpoint_url=ENDPOINT,
+            aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY,
+            region_name="us-east-1",
+        )
+        keys = [o["Key"] for o in s3.list_objects_v2(Bucket=BUCKET).get("Contents", [])]
+        assert len(keys) == 1 and keys[0].endswith(".png"), f"bucket keys: {keys}"
+        print(f"[ok] object in bucket: {keys[0]}")
+
+        # stored row carries an image_ref, NOT base64
+        _, stored = await client.load(agent_id)
+        ref = stored[0]["content"][1]
+        assert ref["type"] == "image_ref" and ref["path"] == keys[0]
+        print("[ok] sqlite row carries image_ref only")
+
+        # hydrated load round-trips the exact bytes
+        _, hydrated = await client.load(agent_id, hydrate=True)
+        url = hydrated[0]["content"][1]["image_url"]["url"]
+        assert url == f"data:image/png;base64,{PNG_B64}"
+        print("[ok] hydrate round-trip: identical base64")
+
+        # ---- 4: legacy FS image still hydrates through the hybrid read ----
+        images_dir = client.images_dir
+        fs = FsImageStore(images_dir)
+        fs.put("legacydeadbeef.png", PNG_BYTES)
+        await client.add_message(
+            agent_id,
+            {
+                "role": "tool",
+                "tool_call_id": "e2e-c2",
+                "content": [{"type": "image_ref", "path": "legacydeadbeef.png", "mime": "image/png"}],
+            },
+        )
+        _, hydrated2 = await client.load(agent_id, hydrate=True)
+        url2 = hydrated2[1]["content"][0]["image_url"]["url"]
+        assert url2 == f"data:image/png;base64,{PNG_B64}"
+        print("[ok] legacy FS image hydrates via hybrid read-fallback")
+
+    # ---- 5: kill the container -> next init falls back to FS ----
+    stop_rustfs()
+    async with MemoryClient(config_dir=tmp) as client:
+        assert isinstance(client.image_store, FsImageStore), (
+            f"expected FsImageStore fallback, got {type(client.image_store).__name__}"
+        )
+        print("[ok] endpoint down -> FsImageStore fallback")
+
+    print("\nE2E-IMAGE-STORE-OK")
+
+
+if __name__ == "__main__":
+    if subprocess.run(["docker", "pull", "rustfs/rustfs:latest"], capture_output=True).returncode != 0:
+        print("[..] pull failed — trying cached image")
+    start_rustfs()
+    try:
+        asyncio.run(run())
+    finally:
+        stop_rustfs()

crow_cli-0.1.38/scripts/e2e_postgres_live.py

@@ -0,0 +1,204 @@
+#!/usr/bin/env python
+"""Live e2e: memory layer against a REAL PostgreSQL container.
+
+Boots an ephemeral postgres:17-alpine container, then drives the real
+memory package through the postgres path of every dialect seam:
+  1. create_database   -> tables + messages_fts tsvector table + GIN index
+  2. add_message/search_messages -> tsvector FTS round trip, lower=better
+  3. JSON column round trip (Message.data) byte-identical
+  4. claim_deliveries with 2 concurrent claimers -> exactly one wins
+     (row-level atomicity on postgres — the multi-machine mailbox story)
+  5. get_ro_engine     -> SELECT ok, INSERT refused by the server
+
+Usage:
+  uv --project . run python scripts/e2e_postgres_live.py
+"""
+
+import subprocess
+import threading
+import time
+
+from sqlalchemy import text
+from sqlalchemy.exc import SQLAlchemyError
+
+from crow_cli.memory import (
+    add_message,
+    claim_deliveries,
+    create_agent,
+    create_database,
+    finish_task,
+    get_engine,
+    get_ro_engine,
+    launch_task,
+    load_messages,
+    search_messages,
+)
+
+PORT = 15432
+CONTAINER = "crow-pg-e2e"
+USER = PASSWORD = DB = "crowe2e"
+URI = f"postgresql+psycopg://{USER}:{PASSWORD}@127.0.0.1:{PORT}/{DB}"
+
+
+def docker(*args: str) -> str:
+    return subprocess.run(
+        ["docker", *args], capture_output=True, text=True, check=True
+    ).stdout.strip()
+
+
+def start_postgres() -> None:
+    subprocess.run(["docker", "rm", "-f", CONTAINER], capture_output=True)
+    docker(
+        "run", "-d", "--name", CONTAINER,
+        "-p", f"{PORT}:5432",
+        "-e", f"POSTGRES_USER={USER}",
+        "-e", f"POSTGRES_PASSWORD={PASSWORD}",
+        "-e", f"POSTGRES_DB={DB}",
+        "postgres:17-alpine",
+    )
+    # Probe with a REAL connection: pg_isready can succeed against the
+    # throwaway server the image runs during initdb, before the restart.
+    for _ in range(60):
+        try:
+            probe = get_engine(URI)
+            with probe.connect() as conn:
+                conn.execute(text("SELECT 1"))
+            probe.dispose()
+            print(f"[ok] postgres accepting connections at 127.0.0.1:{PORT}")
+            return
+        except SQLAlchemyError:
+            time.sleep(1)
+    raise SystemExit("[FAIL] postgres never became ready")
+
+
+def stop_postgres() -> None:
+    subprocess.run(["docker", "rm", "-f", CONTAINER], capture_output=True)
+    print("[ok] container removed")
+
+
+def run() -> None:
+    # ---- 1: schema + FTS objects on postgres ----
+    create_database(URI)
+    engine = get_engine(URI)
+    with engine.connect() as conn:
+        tables = {
+            r[0]
+            for r in conn.execute(
+                text(
+                    "SELECT table_name FROM information_schema.tables "
+                    "WHERE table_schema = 'public'"
+                )
+            )
+        }
+        indexes = {
+            r[0]
+            for r in conn.execute(
+                text("SELECT indexname FROM pg_indexes WHERE schemaname = 'public'")
+            )
+        }
+    for t in ("agents", "messages", "tasks", "task_deliveries", "prompts", "messages_fts"):
+        assert t in tables, f"missing table {t}: {tables}"
+    assert "idx_messages_fts_tsv" in indexes, f"missing GIN index: {indexes}"
+    print("[ok] create_database: tables + messages_fts + GIN index")
+
+    # ---- 2: FTS write/search round trip, lower=better contract ----
+    create_agent(
+        engine,
+        agent_id="e2e-pg-1-1",
+        session_id="e2e-pg",
+        agent_idx=1,
+        cwd="/tmp",
+        system_prompt="sp",
+        tool_definitions=[],
+        request_params={},
+        model_identifier="m",
+    )
+    add_message(
+        engine, "e2e-pg-1-1",
+        {"role": "user", "content": "the quick brown fox jumps over the lazy dog"},
+    )
+    add_message(
+        engine, "e2e-pg-1-1",
+        {"role": "user", "content": "postgres full text search drill"},
+    )
+    hits = search_messages(engine, "postgres search", limit=10)
+    assert len(hits) == 1, f"expected 1 hit, got {len(hits)}"
+    assert hits[0]["session_id"] == "e2e-pg"
+    assert hits[0]["score"] < 0, f"ts_rank must be negated (lower=better): {hits[0]['score']}"
+    hits2 = search_messages(engine, "fox dog", limit=10)
+    assert len(hits2) == 1 and "fox" in hits2[0]["data"]["content"]
+    assert search_messages(engine, "   ", limit=10) == []
+    print("[ok] search_messages: tsvector round trip, lower=better, empty-safe")
+
+    # ---- 3: JSON column round trip ----
+    rich_msg = {
+        "role": "assistant",
+        "content": [
+            {"type": "text", "text": "json round trip"},
+            {"type": "tool_call", "name": "terminal", "arguments": {"command": "ls -la"}},
+        ],
+        "reasoning_content": "nested dicts survive",
+    }
+    add_message(engine, "e2e-pg-1-1", rich_msg)
+    rows = load_messages(engine, "e2e-pg-1-1")
+    stored = [r for r in rows if r.get("reasoning_content")]
+    assert len(stored) == 1 and stored[0] == rich_msg, f"round trip mismatch: {stored}"
+    print("[ok] Message.data JSON round trip: identical dict")
+
+    # ---- 4: mailbox exactly-once under concurrent claimers ----
+    launch_task(
+        engine,
+        task_id="e2e-t1",
+        owner_session="e2e-pg",
+        tool_call_id="tc1",
+        prompt="drill",
+    )
+    assert finish_task(engine, "e2e-t1", result="done", content="completion payload")
+
+    barrier = threading.Barrier(2)
+    results: list[list] = [[], []]
+
+    def claim(i: int) -> None:
+        barrier.wait()
+        results[i] = claim_deliveries(engine, "e2e-pg")
+
+    threads = [threading.Thread(target=claim, args=(i,)) for i in range(2)]
+    for t in threads:
+        t.start()
+    for t in threads:
+        t.join()
+    total = [d for r in results for d in r]
+    assert len(total) == 1, f"delivery claimed {len(total)} times: {results}"
+    assert total[0]["content"] == "completion payload"
+    print("[ok] claim_deliveries: 2 concurrent claimers, exactly one won")
+
+    # ---- 5: read-only engine enforced by the server ----
+    ro = get_ro_engine(URI)
+    with ro.connect() as conn:
+        assert conn.execute(text("SELECT COUNT(*) FROM messages")).scalar() == 3
+    try:
+        with ro.begin() as conn:
+            conn.execute(
+                text(
+                    "INSERT INTO prompts(id, name, template, created_at) "
+                    "VALUES ('x', 'x', 'x', 'x')"
+                )
+            )
+        raise AssertionError("read-only engine accepted a write")
+    except SQLAlchemyError as e:
+        assert "read-only" in str(e).lower(), f"unexpected error: {e}"
+    print("[ok] get_ro_engine: SELECT ok, INSERT refused by postgres")
+
+    engine.dispose()
+    ro.dispose()
+    print("\nE2E-POSTGRES-OK")
+
+
+if __name__ == "__main__":
+    if subprocess.run(["docker", "pull", "postgres:17-alpine"], capture_output=True).returncode != 0:
+        print("[..] pull failed — trying cached image")
+    start_postgres()
+    try:
+        run()
+    finally:
+        stop_postgres()