jumpstart-mode 1.1.13 → 1.1.14

package/.jumpstart/skills/postgresql-pgvector/references/retrieval-patterns.md

@@ -0,0 +1,235 @@
+# Retrieval Patterns Reference
+
+This reference covers semantic search queries, operator selection, metadata-filtered
+retrieval, reranking-ready output, and tenant-aware retrieval patterns.
+
+---
+
+## Operator selection
+
+| Operator | Distance type | Use case |
+|---|---|---|
+| `<=>` | Cosine | Default for normalized text embeddings (OpenAI, Azure OpenAI, Cohere) |
+| `<->` | L2 / Euclidean | Geometric or numeric feature vectors |
+| `<#>` | Negative inner product | Models trained with dot product objective (some HuggingFace models) |
+
+**Default to `<=>`** unless the embedding model documentation explicitly recommends another
+distance function.
+
+---
+
+## Basic semantic search
+
+```sql
+SELECT
+    dc.id,
+    dc.document_id,
+    dc.chunk_text,
+    dc.metadata,
+    d.title                             AS document_title,
+    d.source_uri,
+    1 - (dc.embedding <=> :query_embedding) AS similarity
+FROM document_chunks dc
+JOIN documents d ON d.id = dc.document_id
+WHERE dc.is_deleted = FALSE
+ORDER BY dc.embedding <=> :query_embedding
+LIMIT :top_k;
+```
+
+---
+
+## Metadata-filtered semantic search
+
+```sql
+SELECT
+    dc.id,
+    dc.chunk_text,
+    dc.metadata,
+    1 - (dc.embedding <=> :query_embedding) AS similarity
+FROM document_chunks dc
+WHERE dc.is_deleted = FALSE
+  AND dc.tenant_id = :tenant_id
+  AND dc.metadata->>'document_type' = :document_type
+  AND dc.metadata->>'security_label' IN ('public', 'internal')
+ORDER BY dc.embedding <=> :query_embedding
+LIMIT :top_k;
+```
+
+---
+
+## Threshold-based search
+
+Return only chunks above a minimum similarity score:
+
+```sql
+SELECT
+    id,
+    chunk_text,
+    1 - (embedding <=> :query_embedding) AS similarity
+FROM document_chunks
+WHERE is_deleted = FALSE
+  AND 1 - (embedding <=> :query_embedding) >= :min_similarity  -- e.g. 0.75
+ORDER BY embedding <=> :query_embedding
+LIMIT :top_k;
+```
+
+Note: Computing `1 - (embedding <=> :query_embedding)` in the WHERE clause forces a
+sequential scan. For large tables, oversample from the ANN scan and filter in a subquery.
+
+```sql
+SELECT * FROM (
+    SELECT id, chunk_text, 1 - (embedding <=> :query_embedding) AS similarity
+    FROM document_chunks
+    WHERE is_deleted = FALSE
+    ORDER BY embedding <=> :query_embedding
+    LIMIT 200
+) sub
+WHERE similarity >= :min_similarity
+ORDER BY similarity DESC
+LIMIT :top_k;
+```
+
+---
+
+## Reranking-ready retrieval
+
+Retrieve more candidates than needed, include metadata for a reranking step:
+
+```sql
+SELECT
+    dc.id,
+    dc.chunk_text,
+    dc.metadata,
+    dc.token_count,
+    d.source_uri,
+    d.title,
+    1 - (dc.embedding <=> :query_embedding) AS semantic_score
+FROM document_chunks dc
+JOIN documents d ON d.id = dc.document_id
+WHERE dc.is_deleted = FALSE
+  AND dc.tenant_id = :tenant_id
+ORDER BY dc.embedding <=> :query_embedding
+LIMIT 50;  -- reranker will reduce this to top_k
+```
+
+Pass `chunk_text` and `semantic_score` to a cross-encoder reranker (Cohere Rerank,
+Azure AI Search semantic reranker, or a local model).
+
+---
+
+## Recency-weighted retrieval
+
+Blend semantic similarity with document recency:
+
+```sql
+SELECT
+    dc.id,
+    dc.chunk_text,
+    d.source_uri,
+    (1 - (dc.embedding <=> :query_embedding)) * 0.8
+    + (EXTRACT(EPOCH FROM (now() - d.created_at)) / 86400.0 * -0.001 + 1.0) * 0.2
+    AS blended_score
+FROM document_chunks dc
+JOIN documents d ON d.id = dc.document_id
+WHERE dc.is_deleted = FALSE
+  AND dc.tenant_id = :tenant_id
+  AND d.created_at >= now() - INTERVAL ':recency_days days'
+ORDER BY blended_score DESC
+LIMIT :top_k;
+```
+
+Adjust the recency decay factor for the use case. RFP answer retrieval should heavily
+penalize answers older than 18 months.
+
+---
+
+## Citation output retrieval
+
+Return chunks with full citation grounding for answer display:
+
+```sql
+SELECT
+    dc.id         AS chunk_id,
+    dc.chunk_text,
+    1 - (dc.embedding <=> :query_embedding) AS similarity,
+    cc.source_uri,
+    cc.page_number,
+    cc.section_heading,
+    cc.citation_text
+FROM document_chunks dc
+LEFT JOIN chunk_citations cc ON cc.chunk_id = dc.id
+WHERE dc.is_deleted = FALSE
+  AND dc.tenant_id = :tenant_id
+ORDER BY dc.embedding <=> :query_embedding
+LIMIT :top_k;
+```
+
+---
+
+## Semantic deduplication
+
+Find near-duplicate chunks within the same document or across documents:
+
+```sql
+SELECT
+    a.id    AS chunk_a,
+    b.id    AS chunk_b,
+    1 - (a.embedding <=> b.embedding) AS similarity
+FROM document_chunks a
+CROSS JOIN LATERAL (
+    SELECT id, embedding
+    FROM document_chunks
+    WHERE id != a.id
+      AND is_deleted = FALSE
+    ORDER BY embedding <=> a.embedding
+    LIMIT 5
+) b
+WHERE a.is_deleted = FALSE
+  AND 1 - (a.embedding <=> b.embedding) >= 0.97  -- near-duplicate threshold
+ORDER BY similarity DESC;
+```
+
+Use this to detect duplicate ingestion, near-identical boilerplate, or redundant chunks
+before building a knowledge base.
+
+---
+
+## No-answer detection
+
+When all retrieved chunks fall below a quality threshold, surface a no-answer response
+rather than generating a low-confidence answer:
+
+```python
+MINIMUM_SIMILARITY = 0.65
+NO_ANSWER_THRESHOLD = 0  # chunks above minimum
+
+results = retrieve(query, top_k=10)
+high_quality = [r for r in results if r["similarity"] >= MINIMUM_SIMILARITY]
+
+if len(high_quality) < NO_ANSWER_THRESHOLD + 1:
+    return {"answer": None, "reason": "no_relevant_context", "chunks": []}
+```
+
+Log the no-answer event to `retrieval_events` with `result_used = FALSE`.
+
+---
+
+## Conflict detection across sources
+
+When retrieved chunks contradict each other, surface the conflict rather than choosing:
+
+```python
+def detect_conflicts(chunks: list[dict]) -> list[tuple]:
+    """Return pairs of chunks that appear to contradict each other."""
+    conflicts = []
+    for i, a in enumerate(chunks):
+        for b in chunks[i+1:]:
+            # Simple heuristic: same topic but opposite sentiment or different numbers
+            # Replace with a cross-encoder or an LLM conflict check
+            if chunks_conflict(a["chunk_text"], b["chunk_text"]):
+                conflicts.append((a, b))
+    return conflicts
+```
+
+Present conflicting sources to the user with both source citations rather than silently
+choosing one.

package/.jumpstart/skills/postgresql-pgvector/references/schema-patterns.md

@@ -0,0 +1,217 @@
+# Schema Patterns Reference
+
+This reference defines table structures, design rules, and schema tiers for PostgreSQL
+pgvector applications built by AI Innovation Pod and Jump Start teams.
+
+---
+
+## Schema tier selection
+
+| Tier | When to use |
+|---|---|
+| **Basic RAG** | Single tenant, internal tooling, prototype, < 500K chunks |
+| **Multi-tenant RAG** | Multiple clients or teams, shared infrastructure, Pod-managed SaaS |
+| **RFP Retrieval** | Prior answer reuse, approval-gated knowledge library |
+| **Knowledge base** | Organizational memory, Jump Start asset retrieval |
+
+---
+
+## Core tables — all tiers
+
+### `embedding_models` — always create first
+
+Prevents dimension mismatch. Register the model before inserting any embeddings.
+
+```sql
+CREATE TABLE embedding_models (
+    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    provider    TEXT NOT NULL,         -- 'openai', 'azure_openai', 'cohere', etc.
+    model_name  TEXT NOT NULL,         -- 'text-embedding-3-small'
+    dimensions  INTEGER NOT NULL,      -- 1536, 3072, 768, etc.
+    distance_metric TEXT NOT NULL DEFAULT 'cosine', -- 'cosine', 'l2', 'inner_product'
+    is_active   BOOLEAN NOT NULL DEFAULT TRUE,
+    created_at  TIMESTAMPTZ DEFAULT now(),
+    UNIQUE (provider, model_name, dimensions)
+);
+```
+
+### `documents`
+
+Source document registry. One row per unique source file or record.
+
+```sql
+CREATE TABLE documents (
+    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    source_system   TEXT NOT NULL,         -- 'sharepoint', 'confluence', 'blob_storage'
+    source_uri      TEXT NOT NULL,         -- canonical path or URL
+    title           TEXT,
+    content_hash    TEXT,                  -- SHA-256 of raw content; use for change detection
+    file_type       TEXT,                  -- 'pdf', 'docx', 'md', 'txt'
+    page_count      INTEGER,
+    language        TEXT DEFAULT 'en',
+    security_label  TEXT DEFAULT 'internal', -- 'public', 'internal', 'confidential', 'restricted'
+    metadata        JSONB DEFAULT '{}'::jsonb,
+    ingested_at     TIMESTAMPTZ,
+    created_at      TIMESTAMPTZ DEFAULT now(),
+    updated_at      TIMESTAMPTZ DEFAULT now()
+);
+```
+
+### `document_chunks`
+
+Primary vector storage table. One row per chunk.
+
+```sql
+CREATE TABLE document_chunks (
+    id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    document_id         UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
+    embedding_model_id  UUID NOT NULL REFERENCES embedding_models(id),
+    chunk_index         INTEGER NOT NULL,        -- 0-based position within document
+    chunk_text          TEXT NOT NULL,
+    token_count         INTEGER,
+    embedding           vector(1536),            -- dimension must match embedding_model
+    chunk_strategy      TEXT DEFAULT 'fixed',    -- 'fixed', 'sentence', 'semantic'
+    chunk_version       INTEGER NOT NULL DEFAULT 1,
+    metadata            JSONB DEFAULT '{}'::jsonb,
+    is_deleted          BOOLEAN NOT NULL DEFAULT FALSE,
+    created_at          TIMESTAMPTZ DEFAULT now()
+);
+```
+
+**Important**: The `vector(1536)` declaration must exactly match the model dimension in
+`embedding_models`. Create a separate column or table if you need to store multiple model
+dimensions simultaneously.
+
+### `chunk_citations`
+
+Source grounding for answers. Enables verifiable citations.
+
+```sql
+CREATE TABLE chunk_citations (
+    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    chunk_id        UUID NOT NULL REFERENCES document_chunks(id) ON DELETE CASCADE,
+    source_uri      TEXT NOT NULL,
+    page_number     INTEGER,
+    section_heading TEXT,
+    start_char      INTEGER,
+    end_char        INTEGER,
+    citation_text   TEXT             -- verbatim excerpt for citation display
+);
+```
+
+### `ingestion_batches`
+
+Tracks ingestion runs. Enables idempotent re-ingestion.
+
+```sql
+CREATE TABLE ingestion_batches (
+    id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    source_system       TEXT NOT NULL,
+    batch_label         TEXT,
+    started_at          TIMESTAMPTZ DEFAULT now(),
+    completed_at        TIMESTAMPTZ,
+    status              TEXT NOT NULL DEFAULT 'running',  -- 'running', 'completed', 'failed'
+    records_processed   INTEGER DEFAULT 0,
+    records_failed      INTEGER DEFAULT 0,
+    error_summary       TEXT,
+    metadata            JSONB DEFAULT '{}'::jsonb
+);
+```
+
+### `retrieval_events`
+
+Logs every retrieval operation. Required for observability and offline evaluation.
+
+```sql
+CREATE TABLE retrieval_events (
+    id                      UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    trace_id                TEXT,                          -- from application tracing (e.g. OTEL)
+    session_id              TEXT,
+    user_id                 TEXT,
+    query_text              TEXT NOT NULL,
+    query_embedding_model   TEXT,
+    filters                 JSONB DEFAULT '{}'::jsonb,
+    retrieved_chunk_ids     UUID[],
+    scores                  JSONB,                         -- {"chunk_id": score, ...}
+    top_k                   INTEGER,
+    latency_ms              INTEGER,
+    result_used             BOOLEAN,                       -- did the result make it into the answer?
+    created_at              TIMESTAMPTZ DEFAULT now()
+);
+```
+
+---
+
+## Multi-tenant additions
+
+Add `tenant_id` as a dedicated column — not JSONB — on all tables with tenant scope.
+
+```sql
+ALTER TABLE documents        ADD COLUMN tenant_id UUID NOT NULL;
+ALTER TABLE document_chunks  ADD COLUMN tenant_id UUID NOT NULL;
+ALTER TABLE retrieval_events ADD COLUMN tenant_id UUID;
+```
+
+Never encode `tenant_id` only in JSONB metadata. JSONB predicates cannot use standard B-tree
+indexes and will lead to sequential scans at scale.
+
+---
+
+## Ingestion pipeline stages
+
+Design ingestion to advance through these stages with status tracking:
+
+| Stage | Status value |
+|---|---|
+| Source discovery | `discovered` |
+| File extraction | `extracted` |
+| Cleaning | `cleaned` |
+| Chunking | `chunked` |
+| Embedding generation | `embedded` |
+| Upsert to PostgreSQL | `stored` |
+| Index refresh | `indexed` |
+| Retrieval validation | `validated` |
+| Evaluation logging | `evaluated` |
+
+Track stage status in `ingestion_batches.status` or a separate `document_ingestion_status`
+table for fine-grained per-document tracking.
+
+---
+
+## Idempotent upsert pattern
+
+Use `content_hash` to detect unchanged documents and skip re-embedding.
+
+```sql
+INSERT INTO documents (source_system, source_uri, content_hash, ...)
+VALUES (:source_system, :source_uri, :content_hash, ...)
+ON CONFLICT (source_uri)
+DO UPDATE SET
+    content_hash = EXCLUDED.content_hash,
+    updated_at   = now()
+WHERE documents.content_hash IS DISTINCT FROM EXCLUDED.content_hash;
+```
+
+For chunks, soft-delete before re-inserting when content changes:
+
+```sql
+UPDATE document_chunks
+SET    is_deleted = TRUE
+WHERE  document_id = :document_id
+AND    chunk_version < :new_version;
+```
+
+---
+
+## Dedicated column vs. JSONB guidance
+
+| Field | Recommendation |
+|---|---|
+| `tenant_id` | Dedicated `UUID` column |
+| `source_system` | Dedicated `TEXT` column |
+| `document_type` | Dedicated `TEXT` column |
+| `security_label` | Dedicated `TEXT` column |
+| `created_at`, `updated_at` | Dedicated `TIMESTAMPTZ` columns |
+| Client-specific tags | JSONB |
+| Arbitrary metadata from source | JSONB |
+| Feature flags or dynamic properties | JSONB |

package/.jumpstart/skills/postgresql-pgvector/scripts/benchmark_retrieval.py

@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+"""
+benchmark_retrieval.py
+
+Measures p50 / p95 / p99 retrieval latency for semantic and hybrid search
+queries against a running PostgreSQL pgvector database.
+
+Usage:
+    DATABASE_URL=postgresql://app:app@localhost:5432/appdb \
+    OPENAI_API_KEY=sk-... \
+    python benchmark_retrieval.py --queries 50 --top-k 10
+
+Requires:
+    pip install psycopg2-binary openai numpy
+"""
+
+import argparse
+import os
+import statistics
+import sys
+import time
+
+import numpy as np
+import psycopg2
+
+SAMPLE_QUERIES = [
+    "What is the rate card for senior architect consulting engagements?",
+    "Describe the AI Innovation Pod delivery model.",
+    "What are the RLS policies for multi-tenant document isolation?",
+    "How does HNSW index affect retrieval latency at scale?",
+    "What are the acceptance criteria for a Jump Start workshop?",
+    "Explain hybrid search weight tuning for RFP answer retrieval.",
+    "What is the recommended chunk size for PDF ingestion?",
+    "How do we handle embedding model upgrades without downtime?",
+    "What observability metrics should be captured per retrieval call?",
+    "Describe the golden dataset construction process for evaluation.",
+]
+
+
+def get_synthetic_embedding(dim: int = 1536) -> list[float]:
+    """Return a normalized random embedding vector."""
+    vec = np.random.randn(dim).astype(np.float32)
+    vec = vec / np.linalg.norm(vec)
+    return vec.tolist()
+
+
+def run_semantic_query(cur, embedding: list[float], top_k: int) -> float:
+    start = time.perf_counter()
+    cur.execute(
+        """
+        SELECT id, 1 - (embedding <=> %s::vector) AS similarity
+        FROM document_chunks
+        WHERE is_deleted = FALSE
+        ORDER BY embedding <=> %s::vector
+        LIMIT %s
+        """,
+        (str(embedding), str(embedding), top_k),
+    )
+    cur.fetchall()
+    return (time.perf_counter() - start) * 1000
+
+
+def run_hybrid_query(cur, query_text: str, embedding: list[float], top_k: int) -> float:
+    start = time.perf_counter()
+    cur.execute(
+        """
+        WITH semantic AS (
+            SELECT id, 1 - (embedding <=> %s::vector) AS sem_score
+            FROM document_chunks
+            WHERE is_deleted = FALSE
+            ORDER BY embedding <=> %s::vector
+            LIMIT %s
+        ),
+        keyword AS (
+            SELECT id,
+                   ts_rank_cd(to_tsvector('english', chunk_text),
+                              plainto_tsquery('english', %s)) AS kw_score
+            FROM document_chunks
+            WHERE is_deleted = FALSE
+              AND to_tsvector('english', chunk_text) @@ plainto_tsquery('english', %s)
+            LIMIT %s
+        ),
+        combined AS (
+            SELECT COALESCE(s.id, k.id) AS id,
+                   COALESCE(s.sem_score, 0) * 0.7 + COALESCE(k.kw_score, 0) * 0.3 AS score
+            FROM semantic s FULL OUTER JOIN keyword k ON k.id = s.id
+        )
+        SELECT id, score FROM combined ORDER BY score DESC LIMIT %s
+        """,
+        (str(embedding), str(embedding), top_k * 5,
+         query_text, query_text, top_k * 5,
+         top_k),
+    )
+    cur.fetchall()
+    return (time.perf_counter() - start) * 1000
+
+
+def percentile(data: list[float], pct: float) -> float:
+    return statistics.quantiles(data, n=100)[int(pct) - 1]
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Benchmark pgvector retrieval latency")
+    parser.add_argument("--queries", type=int, default=50, help="Number of benchmark iterations")
+    parser.add_argument("--top-k", type=int, default=10, help="Results per query")
+    args = parser.parse_args()
+
+    database_url = os.environ.get("DATABASE_URL")
+    if not database_url:
+        print("ERROR: DATABASE_URL is not set.", file=sys.stderr)
+        sys.exit(1)
+
+    conn_url = database_url.replace("postgresql+asyncpg://", "postgresql://")
+    conn = psycopg2.connect(conn_url)
+    cur = conn.cursor()
+
+    print(f"Running {args.queries} iterations, top_k={args.top_k}...\n")
+
+    semantic_latencies: list[float] = []
+    hybrid_latencies: list[float] = []
+
+    for i in range(args.queries):
+        query_text = SAMPLE_QUERIES[i % len(SAMPLE_QUERIES)]
+        embedding = get_synthetic_embedding()
+
+        semantic_latencies.append(run_semantic_query(cur, embedding, args.top_k))
+        hybrid_latencies.append(run_hybrid_query(cur, query_text, embedding, args.top_k))
+
+        if (i + 1) % 10 == 0:
+            print(f"  Progress: {i + 1}/{args.queries}")
+
+    cur.close()
+    conn.close()
+
+    print("\n=== Semantic Search Latency (ms) ===")
+    print(f"  p50 : {percentile(semantic_latencies, 50):.1f} ms")
+    print(f"  p95 : {percentile(semantic_latencies, 95):.1f} ms")
+    print(f"  p99 : {percentile(semantic_latencies, 99):.1f} ms")
+    print(f"  mean: {statistics.mean(semantic_latencies):.1f} ms")
+
+    print("\n=== Hybrid Search Latency (ms) ===")
+    print(f"  p50 : {percentile(hybrid_latencies, 50):.1f} ms")
+    print(f"  p95 : {percentile(hybrid_latencies, 95):.1f} ms")
+    print(f"  p99 : {percentile(hybrid_latencies, 99):.1f} ms")
+    print(f"  mean: {statistics.mean(hybrid_latencies):.1f} ms")
+
+
+if __name__ == "__main__":
+    main()