astrocyte-postgres 0.13.1__tar.gz → 0.14.2__tar.gz

{astrocyte_postgres-0.13.1 → astrocyte_postgres-0.14.2}/.gitignore

@@ -52,3 +52,5 @@ target/
 
 # IDE (optional)
 .idea/
+astrocyte-py/logs/
+astrocyte-py/logs/ # if not already there

{astrocyte_postgres-0.13.1 → astrocyte_postgres-0.14.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: astrocyte-postgres
-Version: 0.13.1
+Version: 0.14.2
 Summary: PostgreSQL adapter for Astrocyte (vector + document + wiki stores backed by pgvector and tsvector)
 License-Expression: Apache-2.0
 Requires-Python: >=3.11
@@ -16,7 +16,7 @@ Description-Content-Type: text/markdown
 
 # astrocyte-postgres
 
-**PostgreSQL + [pgvector](https://github.com/pgvector/pgvector)** implementation of the Astrocyte **`VectorStore`** and **`WikiStore`** SPIs ([`provider-spi.md`](../../docs/_plugins/provider-spi.md)).
+**PostgreSQL** implementation of the Astrocyte **`VectorStore`** and **`WikiStore`** SPIs ([`provider-spi.md`](../../docs/_plugins/provider-spi.md)). Uses **[pgvectorscale](https://github.com/timescale/pgvectorscale)** (DiskANN) for ANN indexing — pgvectorscale CASCADE-installs pgvector, so pgvector's `vector` type and `<=>` operator are available too. The shipped Docker image bakes pgvectorscale in; for self-host, use that image, Supabase, Neon, or Timescale Cloud.
 
 ## Install
 
@@ -72,6 +72,16 @@ After migrations are applied, set **`bootstrap_schema: false`** in `vector_store
 
 **Embedding width:** [`migrations/002_astrocyte_vectors.sql`](migrations/002_astrocyte_vectors.sql) creates `vector(${ASTROCYTE_EMBEDDING_DIMENSIONS:-128})`. That must match **`embedding_dimensions`** in config. For OpenAI `text-embedding-3-small`, run migrations with `ASTROCYTE_EMBEDDING_DIMENSIONS=1536`.
 
+**ANN backend (DiskANN via pgvectorscale):** vector indexes are created with `USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)`. pgvectorscale CASCADE-installs pgvector, so application code using pgvector's `<=>` operator works unchanged. Astrocyte standardized on pgvectorscale in 2026-05 — the LongMemEval bench surfaced HNSW per-page write-lock drift (~1.0s → 2.0s/session as the index grew under concurrent retain), and DiskANN's graph layout serializes less aggressively. pgvectorscale is OSS under the PostgreSQL License with no feature gates.
+
+```bash
+./scripts/migrate.sh
+```
+
+**Supported deployment targets:** the shipped `ghcr.io/astrocyteai/astrocyte-postgres` image (bakes pgvectorscale in), Supabase, Neon, Timescale Cloud, or any Postgres with `postgresql-16-pgvectorscale` from Timescale's apt repo. AWS RDS, GCP Cloud SQL, and Azure Database for PostgreSQL do **not** ship pgvectorscale and are not supported targets — run the shipped image on ECS / GCE / a VM instead.
+
+The DiskANN index DDL is in [`migrations/003_indexes.sql`](migrations/003_indexes.sql) and [`migrations/009_entities_trigram_embedding.sql`](migrations/009_entities_trigram_embedding.sql); the `CREATE EXTENSION vectorscale CASCADE` lives in [`migrations/001_extension.sql`](migrations/001_extension.sql).
+
 **Custom `table_name`:** The shipped SQL targets **`astrocyte_vectors`**. If you use another table name, copy and adjust the migration files accordingly.
 
 The later migrations add the Hindsight-comparable Postgres substrate around vectors: bank metadata and access grants, lifecycle columns (`retained_at`, `forgotten_at`), durable wiki pages/revisions/provenance, canonical entity/link tables, and normalized temporal facts.
@@ -83,7 +93,7 @@ The later migrations add the Hindsight-comparable Postgres substrate around vect
 | `dsn` | PostgreSQL connection URI (or set `DATABASE_URL` / `ASTROCYTE_PG_DSN`) |
 | `table_name` | Table name (default `astrocyte_vectors`; alphanumeric + underscore only) |
 | `embedding_dimensions` | Fixed `vector(N)` width; must match your embedding model and the **`vector(N)`** in SQL migrations (default **128**) |
-| `bootstrap_schema` | If **`true`** (default), create extension / table / btree index on first use (dev-friendly; no HNSW). If **`false`**, assume **`migrate.sh`** already applied [`migrations/`](migrations/) (production). |
+| `bootstrap_schema` | If **`true`** (default), create extension / table / btree index on first use (dev-friendly; no DiskANN). If **`false`**, assume **`migrate.sh`** already applied [`migrations/`](migrations/) (production). |
 
 ## How this fits `astrocyte_gateway`
 
@@ -128,6 +138,6 @@ cd astrocyte-services-py/astrocyte-gateway-py && uv sync --extra postgres
 
 ## Production notes
 
-- **HNSW** parameters (`m`, `ef_construction`) live in [`migrations/003_indexes.sql`](migrations/003_indexes.sql); tune with DBA guidance as load grows.
+- **DiskANN** parameters (`num_neighbors`) live in [`migrations/003_indexes.sql`](migrations/003_indexes.sql); tune with DBA guidance as load grows.
 - **Embedding dimension** must match the **`LLMProvider.embed()`** output used by the pipeline.
 - Use **secrets** for `dsn`, not committed YAML.

{astrocyte_postgres-0.13.1 → astrocyte_postgres-0.14.2}/README.md

@@ -1,6 +1,6 @@
 # astrocyte-postgres
 
-**PostgreSQL + [pgvector](https://github.com/pgvector/pgvector)** implementation of the Astrocyte **`VectorStore`** and **`WikiStore`** SPIs ([`provider-spi.md`](../../docs/_plugins/provider-spi.md)).
+**PostgreSQL** implementation of the Astrocyte **`VectorStore`** and **`WikiStore`** SPIs ([`provider-spi.md`](../../docs/_plugins/provider-spi.md)). Uses **[pgvectorscale](https://github.com/timescale/pgvectorscale)** (DiskANN) for ANN indexing — pgvectorscale CASCADE-installs pgvector, so pgvector's `vector` type and `<=>` operator are available too. The shipped Docker image bakes pgvectorscale in; for self-host, use that image, Supabase, Neon, or Timescale Cloud.
 
 ## Install
 
@@ -56,6 +56,16 @@ After migrations are applied, set **`bootstrap_schema: false`** in `vector_store
 
 **Embedding width:** [`migrations/002_astrocyte_vectors.sql`](migrations/002_astrocyte_vectors.sql) creates `vector(${ASTROCYTE_EMBEDDING_DIMENSIONS:-128})`. That must match **`embedding_dimensions`** in config. For OpenAI `text-embedding-3-small`, run migrations with `ASTROCYTE_EMBEDDING_DIMENSIONS=1536`.
 
+**ANN backend (DiskANN via pgvectorscale):** vector indexes are created with `USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)`. pgvectorscale CASCADE-installs pgvector, so application code using pgvector's `<=>` operator works unchanged. Astrocyte standardized on pgvectorscale in 2026-05 — the LongMemEval bench surfaced HNSW per-page write-lock drift (~1.0s → 2.0s/session as the index grew under concurrent retain), and DiskANN's graph layout serializes less aggressively. pgvectorscale is OSS under the PostgreSQL License with no feature gates.
+
+```bash
+./scripts/migrate.sh
+```
+
+**Supported deployment targets:** the shipped `ghcr.io/astrocyteai/astrocyte-postgres` image (bakes pgvectorscale in), Supabase, Neon, Timescale Cloud, or any Postgres with `postgresql-16-pgvectorscale` from Timescale's apt repo. AWS RDS, GCP Cloud SQL, and Azure Database for PostgreSQL do **not** ship pgvectorscale and are not supported targets — run the shipped image on ECS / GCE / a VM instead.
+
+The DiskANN index DDL is in [`migrations/003_indexes.sql`](migrations/003_indexes.sql) and [`migrations/009_entities_trigram_embedding.sql`](migrations/009_entities_trigram_embedding.sql); the `CREATE EXTENSION vectorscale CASCADE` lives in [`migrations/001_extension.sql`](migrations/001_extension.sql).
+
 **Custom `table_name`:** The shipped SQL targets **`astrocyte_vectors`**. If you use another table name, copy and adjust the migration files accordingly.
 
 The later migrations add the Hindsight-comparable Postgres substrate around vectors: bank metadata and access grants, lifecycle columns (`retained_at`, `forgotten_at`), durable wiki pages/revisions/provenance, canonical entity/link tables, and normalized temporal facts.
@@ -67,7 +77,7 @@ The later migrations add the Hindsight-comparable Postgres substrate around vect
 | `dsn` | PostgreSQL connection URI (or set `DATABASE_URL` / `ASTROCYTE_PG_DSN`) |
 | `table_name` | Table name (default `astrocyte_vectors`; alphanumeric + underscore only) |
 | `embedding_dimensions` | Fixed `vector(N)` width; must match your embedding model and the **`vector(N)`** in SQL migrations (default **128**) |
-| `bootstrap_schema` | If **`true`** (default), create extension / table / btree index on first use (dev-friendly; no HNSW). If **`false`**, assume **`migrate.sh`** already applied [`migrations/`](migrations/) (production). |
+| `bootstrap_schema` | If **`true`** (default), create extension / table / btree index on first use (dev-friendly; no DiskANN). If **`false`**, assume **`migrate.sh`** already applied [`migrations/`](migrations/) (production). |
 
 ## How this fits `astrocyte_gateway`
 
@@ -112,6 +122,6 @@ cd astrocyte-services-py/astrocyte-gateway-py && uv sync --extra postgres
 
 ## Production notes
 
-- **HNSW** parameters (`m`, `ef_construction`) live in [`migrations/003_indexes.sql`](migrations/003_indexes.sql); tune with DBA guidance as load grows.
+- **DiskANN** parameters (`num_neighbors`) live in [`migrations/003_indexes.sql`](migrations/003_indexes.sql); tune with DBA guidance as load grows.
 - **Embedding dimension** must match the **`LLMProvider.embed()`** output used by the pipeline.
 - Use **secrets** for `dsn`, not committed YAML.

{astrocyte_postgres-0.13.1 → astrocyte_postgres-0.14.2}/astrocyte_postgres/__init__.py

@@ -1,17 +1,20 @@
-"""PostgreSQL adapter for Astrocyte Tier 1.
+"""PostgreSQL adapter for Astrocyte storage and section recall.
 
 Backed by pgvector (HNSW vector index) and tsvector (BM25 keyword search).
-Provides VectorStore, DocumentStore, WikiStore, MentalModelStore, and
-SourceStore implementations against a single Postgres database.
+Provides VectorStore, DocumentStore, WikiStore, MentalModelStore,
+SourceStore, and (M9) PageIndexStore implementations against a single
+Postgres database.
 """
 
 from astrocyte_postgres.mental_model_store import PostgresMentalModelStore
+from astrocyte_postgres.pageindex_store import PostgresPageIndexStore
 from astrocyte_postgres.source_store import PostgresSourceStore
 from astrocyte_postgres.store import PostgresStore
 from astrocyte_postgres.wiki_store import PostgresWikiStore
 
 __all__ = [
     "PostgresMentalModelStore",
+    "PostgresPageIndexStore",
     "PostgresSourceStore",
     "PostgresStore",
     "PostgresWikiStore",

astrocyte_postgres-0.14.2/astrocyte_postgres/conversation_store.py

@@ -0,0 +1,269 @@
+"""PostgresConversationStore — Postgres impl of ConversationStore.
+
+Mirrors PostgresDocumentStore's pattern: psycopg async pool, optional
+schema bootstrap, narrow upsert / read / delete surface. Schema lives
+in migrations 027 (conversations) + 028 (conversation_turns).
+
+Composability invariant: this store does NOT reference any Memory
+Engine table. Conversation→Memory references go through opaque
+metadata strings, not typed FKs.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import os
+from typing import Any
+
+import psycopg
+from astrocyte.conversations.storage import ConversationStore
+from astrocyte.conversations.types import Conversation, ConversationTurn
+from psycopg_pool import AsyncConnectionPool
+
+logger = logging.getLogger(__name__)
+
+
+class PostgresConversationStore(ConversationStore):
+    """Durable ConversationStore on Postgres (migrations 027 + 028)."""
+
+    def __init__(
+        self,
+        dsn: str | None = None,
+        *,
+        bootstrap_schema: bool = True,
+        **kwargs: Any,
+    ) -> None:
+        self._dsn = dsn or os.environ.get("DATABASE_URL") or os.environ.get("ASTROCYTE_PG_DSN")
+        if not self._dsn:
+            raise ValueError(
+                "PostgresConversationStore requires `dsn` or DATABASE_URL / ASTROCYTE_PG_DSN",
+            )
+        self._bootstrap_schema = bool(bootstrap_schema)
+        self._pool: AsyncConnectionPool | None = None
+        self._pool_lock = asyncio.Lock()
+        self._schema_ready = False
+        self._schema_lock = asyncio.Lock()
+
+    # ── lifecycle ─────────────────────────────────────────────────────
+
+    async def _ensure_pool(self) -> AsyncConnectionPool:
+        async with self._pool_lock:
+            if self._pool is None:
+
+                async def configure(conn: psycopg.AsyncConnection) -> None:
+                    await conn.execute('SET search_path = public, "$user"')
+                    await conn.commit()
+
+                self._pool = AsyncConnectionPool(
+                    conninfo=self._dsn,
+                    configure=configure,
+                    open=False,
+                    min_size=1,
+                    max_size=10,
+                    kwargs={"connect_timeout": 10},
+                )
+                await self._pool.open()
+            return self._pool
+
+    async def close(self) -> None:
+        async with self._pool_lock:
+            if self._pool is not None:
+                await self._pool.close()
+                self._pool = None
+
+    async def _ensure_schema(self, pool: AsyncConnectionPool) -> None:
+        if not self._bootstrap_schema or self._schema_ready:
+            return
+        async with self._schema_lock:
+            if self._schema_ready:
+                return
+            async with pool.connection() as conn:
+                async with conn.cursor() as cur:
+                    await cur.execute(_DDL_CONVERSATIONS)
+                    await cur.execute(_DDL_CONVERSATION_TURNS)
+                    await cur.execute(_INDEXES)
+                await conn.commit()
+            self._schema_ready = True
+
+    # ── save ──────────────────────────────────────────────────────────
+
+    async def save_conversation(self, conversation: Conversation) -> None:
+        pool = await self._ensure_pool()
+        await self._ensure_schema(pool)
+        async with pool.connection() as conn:
+            async with conn.cursor() as cur:
+                # Upsert the conversation row
+                await cur.execute(
+                    """
+                    INSERT INTO astrocyte_conversations
+                        (id, source_uri, title, metadata, created_at)
+                    VALUES (%s, %s, %s, %s::jsonb, %s)
+                    ON CONFLICT (id) DO UPDATE SET
+                        source_uri = EXCLUDED.source_uri,
+                        title      = EXCLUDED.title,
+                        metadata   = EXCLUDED.metadata
+                    """,
+                    (
+                        conversation.id,
+                        conversation.source_uri,
+                        conversation.title,
+                        json.dumps(conversation.metadata),
+                        conversation.created_at,
+                    ),
+                )
+                # Replace any prior turns
+                await cur.execute(
+                    "DELETE FROM astrocyte_conversation_turns WHERE conversation_id = %s",
+                    (conversation.id,),
+                )
+                if conversation.turns:
+                    rows = [
+                        (
+                            t.id,
+                            conversation.id,
+                            i,
+                            t.role,
+                            t.content,
+                            t.timestamp,
+                            json.dumps(t.metadata),
+                        )
+                        for i, t in enumerate(conversation.turns)
+                    ]
+                    await cur.executemany(
+                        """
+                        INSERT INTO astrocyte_conversation_turns
+                            (id, conversation_id, turn_index, role, content, "timestamp", metadata)
+                        VALUES (%s, %s, %s, %s, %s, %s, %s::jsonb)
+                        """,
+                        rows,
+                    )
+            await conn.commit()
+
+    # ── reads ─────────────────────────────────────────────────────────
+
+    async def get_conversation(self, conversation_id: str) -> Conversation | None:
+        pool = await self._ensure_pool()
+        await self._ensure_schema(pool)
+        async with pool.connection() as conn:
+            async with conn.cursor() as cur:
+                await cur.execute(
+                    """
+                    SELECT id, source_uri, title, metadata, created_at
+                    FROM astrocyte_conversations
+                    WHERE id = %s
+                    """,
+                    (conversation_id,),
+                )
+                row = await cur.fetchone()
+                if row is None:
+                    return None
+                conv = Conversation(
+                    id=str(row[0]),
+                    turns=[],
+                    source_uri=row[1] or "",
+                    title=row[2] or "",
+                    metadata=row[3] or {},
+                    created_at=row[4],
+                )
+                await cur.execute(
+                    """
+                    SELECT id, turn_index, role, content, "timestamp", metadata
+                    FROM astrocyte_conversation_turns
+                    WHERE conversation_id = %s
+                    ORDER BY turn_index
+                    """,
+                    (conversation_id,),
+                )
+                turn_rows = await cur.fetchall()
+        conv.turns = [
+            ConversationTurn(
+                id=str(r[0]),
+                role=r[2] or "user",
+                content=r[3] or "",
+                timestamp=r[4],
+                metadata=r[5] or {},
+            )
+            for r in turn_rows
+        ]
+        return conv
+
+    async def list_conversations(self, *, limit: int = 100) -> list[Conversation]:
+        pool = await self._ensure_pool()
+        await self._ensure_schema(pool)
+        async with pool.connection() as conn:
+            async with conn.cursor() as cur:
+                await cur.execute(
+                    """
+                    SELECT id FROM astrocyte_conversations
+                    ORDER BY created_at DESC LIMIT %s
+                    """,
+                    (limit,),
+                )
+                ids = [str(r[0]) for r in await cur.fetchall()]
+        # Eager-load turns. For huge conversations this is expensive;
+        # callers should pass small `limit` or add an `include_turns=False`
+        # variant (Phase 3+ if needed).
+        result: list[Conversation] = []
+        for cid in ids:
+            conv = await self.get_conversation(cid)
+            if conv is not None:
+                result.append(conv)
+        return result
+
+    # ── delete ────────────────────────────────────────────────────────
+
+    async def delete_conversation(self, conversation_id: str) -> None:
+        pool = await self._ensure_pool()
+        await self._ensure_schema(pool)
+        async with pool.connection() as conn:
+            async with conn.cursor() as cur:
+                # FK CASCADE handles the turns
+                await cur.execute(
+                    "DELETE FROM astrocyte_conversations WHERE id = %s",
+                    (conversation_id,),
+                )
+            await conn.commit()
+
+
+# ─── inline DDL (mirrors migrations 027+028) ──────────────────────────
+
+_DDL_CONVERSATIONS = """
+CREATE TABLE IF NOT EXISTS astrocyte_conversations (
+    id              UUID PRIMARY KEY,
+    source_uri      TEXT NOT NULL DEFAULT '',
+    title           TEXT NOT NULL DEFAULT '',
+    metadata        JSONB NOT NULL DEFAULT '{}'::jsonb,
+    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+"""
+
+_DDL_CONVERSATION_TURNS = """
+CREATE TABLE IF NOT EXISTS astrocyte_conversation_turns (
+    id               UUID PRIMARY KEY,
+    conversation_id  UUID NOT NULL REFERENCES astrocyte_conversations(id) ON DELETE CASCADE,
+    turn_index       INTEGER NOT NULL,
+    role             TEXT NOT NULL,
+    content          TEXT NOT NULL,
+    "timestamp"      TIMESTAMPTZ,
+    metadata         JSONB NOT NULL DEFAULT '{}'::jsonb,
+    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+    UNIQUE (conversation_id, turn_index)
+);
+"""
+
+_INDEXES = """
+CREATE INDEX IF NOT EXISTS astrocyte_conversations_source_idx
+    ON astrocyte_conversations (source_uri)
+    WHERE source_uri <> '';
+CREATE INDEX IF NOT EXISTS astrocyte_conversations_created_at_idx
+    ON astrocyte_conversations (created_at DESC);
+CREATE INDEX IF NOT EXISTS astrocyte_conversation_turns_conv_idx
+    ON astrocyte_conversation_turns (conversation_id, turn_index);
+CREATE INDEX IF NOT EXISTS astrocyte_conversation_turns_role_idx
+    ON astrocyte_conversation_turns (conversation_id, role);
+CREATE INDEX IF NOT EXISTS astrocyte_conversation_turns_ts_idx
+    ON astrocyte_conversation_turns (conversation_id, "timestamp")
+    WHERE "timestamp" IS NOT NULL;
+"""