nlp2sql 0.2.0rc5__tar.gz → 0.2.0rc7__tar.gz

{nlp2sql-0.2.0rc5 → nlp2sql-0.2.0rc7}/.gitignore

@@ -180,6 +180,7 @@ query_cache.db
 !.github/**/*.md
 !.claude/**/*.md
 !CLAUDE.md
+!examples/**/*.md
 
 # Claude Code local settings
 .claude/settings.local.json

{nlp2sql-0.2.0rc5 → nlp2sql-0.2.0rc7}/CLAUDE.md

@@ -51,6 +51,8 @@ The codebase follows Clean Architecture (Hexagonal/Ports & Adapters):
 ```
 src/nlp2sql/
 ├── core/           # Business entities (pure Python, no external dependencies)
+│   ├── entities.py            # Query, SQLQuery, DatabaseType
+│   └── database_prompts.py    # SQL dialect hints for AI providers
 ├── ports/          # Interfaces/abstractions (contracts)
 │   ├── ai_provider.py          # AIProviderPort - interface for AI providers
 │   ├── embedding_provider.py   # EmbeddingProviderPort - interface for embeddings
@@ -69,7 +71,8 @@ src/nlp2sql/
 ├── schema/         # Schema management
 │   ├── schema_manager.py       # Coordinates filtering and strategies
 │   ├── schema_analyzer.py      # Scores schema relevance
-│   └── schema_embedding_manager.py  # FAISS vector embeddings
+│   ├── schema_embedding_manager.py  # FAISS vector embeddings
+│   └── example_store.py        # ExampleStore - FAISS-indexed few-shot examples
 ├── config/         # Configuration (Pydantic Settings)
 ├── exceptions/     # Custom exceptions hierarchy
 └── cli.py          # Click-based CLI

{nlp2sql-0.2.0rc5 → nlp2sql-0.2.0rc7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nlp2sql
-Version: 0.2.0rc5
+Version: 0.2.0rc7
 Summary: Enterprise-ready Natural Language to SQL converter with multi-provider support. Built for production scale (1000+ tables) with Clean Architecture.
 Project-URL: Homepage, https://github.com/luiscarbonel1991/nlp2sql
 Project-URL: Bug Tracker, https://github.com/luiscarbonel1991/nlp2sql/issues
@@ -77,11 +77,17 @@ Provides-Extra: oracle
 Requires-Dist: oracledb>=2.0.0; extra == 'oracle'
 Description-Content-Type: text/markdown
 
-# nlp2sql
+<p align="center">
+  <img src="assets/nlp2sql-logo.png" alt="nlp2sql logo" width="400">
+</p>
+
+<p align="center">
+  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
+  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.9+-blue.svg" alt="Python 3.9+"></a>
+  <a href="https://github.com/psf/black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Code style: black"></a>
+</p>
 
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+# nlp2sql
 
 **Enterprise-ready Natural Language to SQL converter with multi-provider support**
 
@@ -174,7 +180,33 @@ async def main():
     result2 = await service.generate_sql("Show recent orders")
 ```
 
-### 4. Large Database with Schema Filtering
+### 4. Few-Shot Examples (Improved Accuracy)
+
+```python
+from nlp2sql import ExampleStore, create_embedding_provider, create_and_initialize_service
+
+# Create example store with domain-specific examples
+embedding_provider = create_embedding_provider("openai", api_key=os.getenv("OPENAI_API_KEY"))
+example_store = ExampleStore(embedding_provider=embedding_provider)
+await example_store.add_examples([
+    {
+        "question": "What was the total revenue last month?",
+        "sql": "SELECT SUM(gross_revenue) FROM sales WHERE date >= DATE_TRUNC('month', DATEADD(month, -1, CURRENT_DATE))",
+        "database_type": "redshift",
+        "metadata": {"category": "aggregation"}
+    }
+])
+
+service = await create_and_initialize_service(
+    database_url="redshift://user:pass@host:5439/db",
+    ai_provider="openai",
+    api_key=os.getenv("OPENAI_API_KEY"),
+    example_store=example_store
+)
+result = await service.generate_sql("Show me total sales this quarter")
+```
+
+### 5. Large Database with Schema Filtering
 
 ```python
 from nlp2sql import create_and_initialize_service
@@ -208,6 +240,23 @@ nlp2sql inspect --database-url postgresql://localhost/mydb
 nlp2sql benchmark --database-url postgresql://localhost/mydb
 ```
 
+## How It Works
+
+```
+Question ──► Cache check ──► Schema retrieval ──► Relevance filtering ──► Context building ──► AI generation ──► Validation
+                                    │                     │                      │
+                              SchemaRepository    FAISS + TF-IDF hybrid   Reuses precomputed
+                              (+ disk cache)      + batch scoring          relevance scores
+```
+
+1. **Schema retrieval** -- Fetches tables from database via `SchemaRepository` (with disk cache for Redshift)
+2. **Relevance filtering** -- FAISS dense search + TF-IDF sparse search (50/50 hybrid) finds candidate tables; batch scoring refines with precomputed embeddings
+3. **Context building** -- Builds optimized schema context within token limits, reusing scores from step 2 (zero additional embedding calls)
+4. **SQL generation** -- AI provider (OpenAI, Anthropic, or Gemini) generates SQL from question + schema context
+5. **Validation** -- SQL syntax and safety checks before returning results
+
+See [Architecture](docs/ARCHITECTURE.md) for the detailed flow with method references and design decisions.
+
 ## Provider Comparison
 
 | Provider | Context Size | Best For |
@@ -220,17 +269,21 @@ See [Configuration](docs/CONFIGURATION.md) for detailed provider setup.
 
 ## Architecture
 
+Clean Architecture (Ports & Adapters) with three layers: core entities, port interfaces, and adapter implementations. The schema management layer uses FAISS + TF-IDF hybrid search for relevance filtering at scale.
+
 ```
 nlp2sql/
-├── core/           # Business entities
-├── ports/          # Interfaces/abstractions
-├── adapters/       # External implementations (AI providers, databases)
-├── services/       # Application services
-├── schema/         # Schema management and embeddings
-├── config/         # Configuration
-└── exceptions/     # Custom exceptions
+├── core/           # Business entities (pure Python, no dependencies)
+├── ports/          # Interfaces (AIProviderPort, SchemaRepositoryPort, EmbeddingProviderPort)
+├── adapters/       # Implementations (OpenAI, Anthropic, Gemini, PostgreSQL, Redshift)
+├── services/       # Orchestration (QueryGenerationService)
+├── schema/         # Schema management (SchemaManager, SchemaAnalyzer, SchemaEmbeddingManager)
+├── config/         # Pydantic Settings configuration
+└── exceptions/     # Custom exception hierarchy
 ```
 
+See [Architecture](docs/ARCHITECTURE.md) for the full component diagram, data flow, and design decisions.
+
 ## Development
 
 ```bash

{nlp2sql-0.2.0rc5 → nlp2sql-0.2.0rc7}/README.md

@@ -1,8 +1,14 @@
-# nlp2sql
+<p align="center">
+  <img src="assets/nlp2sql-logo.png" alt="nlp2sql logo" width="400">
+</p>
+
+<p align="center">
+  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
+  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.9+-blue.svg" alt="Python 3.9+"></a>
+  <a href="https://github.com/psf/black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Code style: black"></a>
+</p>
 
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+# nlp2sql
 
 **Enterprise-ready Natural Language to SQL converter with multi-provider support**
 
@@ -95,7 +101,33 @@ async def main():
     result2 = await service.generate_sql("Show recent orders")
 ```
 
-### 4. Large Database with Schema Filtering
+### 4. Few-Shot Examples (Improved Accuracy)
+
+```python
+from nlp2sql import ExampleStore, create_embedding_provider, create_and_initialize_service
+
+# Create example store with domain-specific examples
+embedding_provider = create_embedding_provider("openai", api_key=os.getenv("OPENAI_API_KEY"))
+example_store = ExampleStore(embedding_provider=embedding_provider)
+await example_store.add_examples([
+    {
+        "question": "What was the total revenue last month?",
+        "sql": "SELECT SUM(gross_revenue) FROM sales WHERE date >= DATE_TRUNC('month', DATEADD(month, -1, CURRENT_DATE))",
+        "database_type": "redshift",
+        "metadata": {"category": "aggregation"}
+    }
+])
+
+service = await create_and_initialize_service(
+    database_url="redshift://user:pass@host:5439/db",
+    ai_provider="openai",
+    api_key=os.getenv("OPENAI_API_KEY"),
+    example_store=example_store
+)
+result = await service.generate_sql("Show me total sales this quarter")
+```
+
+### 5. Large Database with Schema Filtering
 
 ```python
 from nlp2sql import create_and_initialize_service
@@ -129,6 +161,23 @@ nlp2sql inspect --database-url postgresql://localhost/mydb
 nlp2sql benchmark --database-url postgresql://localhost/mydb
 ```
 
+## How It Works
+
+```
+Question ──► Cache check ──► Schema retrieval ──► Relevance filtering ──► Context building ──► AI generation ──► Validation
+                                    │                     │                      │
+                              SchemaRepository    FAISS + TF-IDF hybrid   Reuses precomputed
+                              (+ disk cache)      + batch scoring          relevance scores
+```
+
+1. **Schema retrieval** -- Fetches tables from database via `SchemaRepository` (with disk cache for Redshift)
+2. **Relevance filtering** -- FAISS dense search + TF-IDF sparse search (50/50 hybrid) finds candidate tables; batch scoring refines with precomputed embeddings
+3. **Context building** -- Builds optimized schema context within token limits, reusing scores from step 2 (zero additional embedding calls)
+4. **SQL generation** -- AI provider (OpenAI, Anthropic, or Gemini) generates SQL from question + schema context
+5. **Validation** -- SQL syntax and safety checks before returning results
+
+See [Architecture](docs/ARCHITECTURE.md) for the detailed flow with method references and design decisions.
+
 ## Provider Comparison
 
 | Provider | Context Size | Best For |
@@ -141,17 +190,21 @@ See [Configuration](docs/CONFIGURATION.md) for detailed provider setup.
 
 ## Architecture
 
+Clean Architecture (Ports & Adapters) with three layers: core entities, port interfaces, and adapter implementations. The schema management layer uses FAISS + TF-IDF hybrid search for relevance filtering at scale.
+
 ```
 nlp2sql/
-├── core/           # Business entities
-├── ports/          # Interfaces/abstractions
-├── adapters/       # External implementations (AI providers, databases)
-├── services/       # Application services
-├── schema/         # Schema management and embeddings
-├── config/         # Configuration
-└── exceptions/     # Custom exceptions
+├── core/           # Business entities (pure Python, no dependencies)
+├── ports/          # Interfaces (AIProviderPort, SchemaRepositoryPort, EmbeddingProviderPort)
+├── adapters/       # Implementations (OpenAI, Anthropic, Gemini, PostgreSQL, Redshift)
+├── services/       # Orchestration (QueryGenerationService)
+├── schema/         # Schema management (SchemaManager, SchemaAnalyzer, SchemaEmbeddingManager)
+├── config/         # Pydantic Settings configuration
+└── exceptions/     # Custom exception hierarchy
 ```
 
+See [Architecture](docs/ARCHITECTURE.md) for the full component diagram, data flow, and design decisions.
+
 ## Development
 
 ```bash

{nlp2sql-0.2.0rc5 → nlp2sql-0.2.0rc7}/docs/API.md

@@ -4,6 +4,50 @@ Complete reference for the nlp2sql Python API and CLI.
 
 ## Python API
 
+### Request Flow
+
+The three entry points differ in lifecycle and performance characteristics:
+
+```
+generate_sql_from_db(question)
+  └─ create_and_initialize_service()     # creates service + loads schema every call
+       └─ create_query_service()          # wires components (AI adapter, repository, embeddings)
+            └─ service.initialize()       # fetches schema, builds FAISS index (skips if cached on disk)
+                 └─ service.generate_sql()
+
+create_and_initialize_service()          # init once, reuse for many queries (recommended)
+  └─ create_query_service()
+       └─ service.initialize()
+            └─ service.generate_sql()    # schema + FAISS index already loaded
+            └─ service.generate_sql()    # in-memory caches warm
+
+create_query_service()                   # full manual control
+  └─ caller must call service.initialize() before first query
+```
+
+**When to use which:**
+
+| Function | Best for | Schema loading |
+|----------|----------|----------------|
+| `generate_sql_from_db` | One-off scripts, notebooks | Every call (init + query) |
+| `create_and_initialize_service` | APIs, multi-query sessions | Once at startup |
+| `create_query_service` | Custom initialization logic | Manual (`initialize()`) |
+
+For multiple queries, always prefer `create_and_initialize_service` -- it loads the schema and FAISS index once, and subsequent `generate_sql()` calls benefit from in-memory caches:
+
+```python
+# Init once, query many times
+service = await create_and_initialize_service(
+    database_url="postgresql://user:pass@localhost:5432/db",
+    ai_provider="openai",
+    api_key=os.getenv("OPENAI_API_KEY"),
+)
+
+for question in ["Count users", "Show recent orders", "Revenue by month"]:
+    result = await service.generate_sql(question)
+    print(result["sql"])
+```
+
 ### Quick Start Functions
 
 #### `generate_sql_from_db`
@@ -36,6 +80,7 @@ print(result['validation'])   # Validation results
 | `api_key` | `str` | Yes | Provider API key |
 | `embedding_provider_type` | `str` | No | Embedding provider: `local` (default), `openai` |
 | `schema_filters` | `dict` | No | Schema filtering options |
+| `example_store` | `ExampleStore` | No | Pre-loaded few-shot examples for better SQL generation |
 
 **Returns:** `dict` with keys `sql`, `confidence`, `explanation`, `validation`
 
@@ -65,8 +110,11 @@ result2 = await service.generate_sql("Show recent orders")
 | `ai_provider` | `str` | Yes | AI provider name |
 | `api_key` | `str` | Yes | Provider API key |
 | `schema_filters` | `dict` | No | Schema filtering options |
-| `embedding_provider_type` | `str` | No | Embedding provider type |
+| `embedding_provider_type` | `str` | No | Embedding provider type (see note below) |
 | `database_type` | `DatabaseType` | No | Database type (auto-detected) |
+| `example_store` | `ExampleStore` | No | Pre-loaded few-shot examples for better SQL generation |
+
+**Note on `embedding_provider_type`:** This controls how schema relevance filtering works. With `"local"` or `"openai"`, the system builds a FAISS vector index for semantic search (recommended for databases with 50+ tables). With `None` (default), the system attempts to load local embeddings silently and falls back to text-only matching if `sentence-transformers` is not installed. Text-only matching uses exact name, normalized singular/plural, and token overlap -- effective for small schemas but less accurate for large ones.
 
 #### `create_query_service`
 
@@ -95,6 +143,42 @@ result = await service.generate_sql(
 )
 ```
 
+### Few-Shot Examples
+
+Improve SQL accuracy by providing domain-specific examples via `ExampleStore`. Examples are indexed with FAISS for semantic retrieval -- the most relevant examples are automatically included in the LLM prompt.
+
+```python
+from nlp2sql import ExampleStore, create_embedding_provider, create_and_initialize_service
+
+# Create example store
+embedding_provider = create_embedding_provider("openai", api_key="your-key")
+example_store = ExampleStore(embedding_provider=embedding_provider)
+
+# Add domain-specific examples
+await example_store.add_examples([
+    {
+        "question": "What was the total revenue last month?",
+        "sql": "SELECT SUM(gross_revenue) FROM sales WHERE date >= DATE_TRUNC('month', DATEADD(month, -1, CURRENT_DATE))",
+        "database_type": "redshift",
+        "metadata": {"category": "aggregation"}
+    },
+    {
+        "question": "Show orders by country",
+        "sql": "SELECT country, COUNT(*) AS total_orders FROM orders GROUP BY country ORDER BY total_orders DESC",
+        "database_type": "redshift",
+        "metadata": {"category": "aggregation"}
+    }
+])
+
+# Pass to service
+service = await create_and_initialize_service(
+    database_url="redshift://user:pass@host:5439/db",
+    ai_provider="openai",
+    api_key="your-key",
+    example_store=example_store
+)
+```
+
 ### Embedding Providers
 
 ```python

nlp2sql-0.2.0rc7/docs/ARCHITECTURE.md

@@ -0,0 +1,267 @@
+# Architecture
+
+nlp2sql follows Clean Architecture (Hexagonal/Ports & Adapters) principles, enabling multi-provider support and maintainability at enterprise scale.
+
+## Component Diagram
+
+```mermaid
+graph TB
+    subgraph "Public API"
+        API[create_query_service<br>generate_sql_from_db]
+    end
+
+    subgraph "Service Layer"
+        QGS[QueryGenerationService]
+    end
+
+    subgraph "Schema Management"
+        SM[SchemaManager]
+        SA[SchemaAnalyzer<br>- relevance scoring<br>- compression]
+        SEM[SchemaEmbeddingManager<br>- FAISS index<br>- TF-IDF search]
+    end
+
+    subgraph "Ports (Interfaces)"
+        AIP[AIProviderPort]
+        SRP[SchemaRepositoryPort]
+        EP[EmbeddingProviderPort]
+    end
+
+    subgraph "AI Adapters"
+        OA[OpenAI Adapter]
+        AA[Anthropic Adapter]
+        GA[Gemini Adapter]
+    end
+
+    subgraph "Database Adapters"
+        PG[PostgreSQL Repository]
+        RS[Redshift Repository]
+    end
+
+    subgraph "Embedding Adapters"
+        LE[Local Embeddings<br>sentence-transformers]
+        OE[OpenAI Embeddings]
+    end
+
+    API --> QGS
+    QGS --> SM
+    QGS --> AIP
+    SM --> SA
+    SM --> SEM
+    SM --> SRP
+    SEM --> EP
+    AIP --> OA
+    AIP --> AA
+    AIP --> GA
+    SRP --> PG
+    SRP --> RS
+    EP --> LE
+    EP --> OE
+```
+
+## Component Descriptions
+
+### Public API
+
+Entry points for the library (`__init__.py`):
+
+| Function | Purpose |
+|----------|---------|
+| `generate_sql_from_db` | One-line convenience: creates service, loads schema, generates SQL, returns result |
+| `create_and_initialize_service` | Creates and initializes service once; reuse for multiple queries (recommended) |
+| `create_query_service` | Creates service with full control; caller must call `initialize()` manually |
+
+### Service Layer
+
+**QueryGenerationService** (`services/query_service.py`) orchestrates the full pipeline:
+
+1. Check query cache
+2. Get optimal schema context via `SchemaManager`
+3. Find relevant examples (from `ExampleStore` or hardcoded fallback)
+4. Build `QueryContext` and call `AIProvider.generate_query()`
+5. Optionally optimize via `QueryOptimizerPort`
+6. Validate SQL via `AIProvider.validate_query()`
+7. Cache valid results
+
+### Schema Management
+
+| Component | File | Responsibility |
+|-----------|------|----------------|
+| **SchemaManager** | `schema/manager.py` | Coordinates filtering, relevance search, and context building |
+| **SchemaAnalyzer** | `schema/analyzer.py` | Scores table/column relevance (text + semantic), compresses schema within token limits |
+| **SchemaEmbeddingManager** | `schema/embedding_manager.py` | Maintains FAISS vector index + TF-IDF sparse index for hybrid search |
+
+### Ports (Interfaces)
+
+Abstract contracts in `ports/` that define layer boundaries:
+
+- **AIProviderPort** (`ports/ai_provider.py`): `generate_query()`, `validate_query()`
+- **SchemaRepositoryPort** (`ports/schema_repository.py`): `get_tables()`, `get_table_info()`, `get_related_tables()`
+- **EmbeddingProviderPort** (`ports/embedding_provider.py`): `encode()`, `get_embedding_dimension()`
+- **SchemaStrategyPort** (`ports/schema_strategy.py`): `build_context()`, `score_relevance()`, `chunk_schema()`
+- **CachePort** (`ports/cache.py`): `get()`, `set()`, `clear()`
+
+### Adapters
+
+Concrete implementations of ports:
+
+| Layer | Adapters | Files |
+|-------|----------|-------|
+| AI Providers | OpenAI GPT, Anthropic Claude, Google Gemini | `adapters/openai_adapter.py`, `anthropic_adapter.py`, `gemini_adapter.py` |
+| Databases | PostgreSQL, Amazon Redshift | `adapters/postgres_repository.py`, `redshift_adapter.py` |
+| Embeddings | Local (sentence-transformers), OpenAI | `adapters/local_embedding_adapter.py`, `openai_embedding_adapter.py` |
+
+## Detailed Query Flow
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant QGS as QueryGenerationService
+    participant SM as SchemaManager
+    participant SEM as SchemaEmbeddingManager
+    participant SA as SchemaAnalyzer
+    participant Repo as SchemaRepository
+    participant AI as AIProvider
+
+    User->>QGS: generate_sql(question)
+    QGS->>QGS: Check query cache
+    QGS->>SM: get_optimal_schema_context(question)
+
+    SM->>SM: _find_relevant_tables(question)
+    SM->>SEM: search_similar_with_embedding(query)
+    SEM->>SEM: FAISS dense + TF-IDF sparse (50/50 hybrid)
+    SEM-->>SM: (results, query_embedding)
+
+    SM->>SEM: get_table_embeddings(table_names)
+    SEM->>SEM: Reconstruct from FAISS index (zero API calls)
+    SEM-->>SM: precomputed_element_embeddings
+
+    SM->>SA: score_relevance_batch(precomputed embeddings)
+    SA->>SA: Text scoring + semantic scoring (zero API calls)
+    SA-->>SM: scored tables
+
+    SM->>Repo: get_table_info(per table)
+    Repo-->>SM: TableInfo with columns, keys, indexes
+
+    Note over SM: Attaches relevance_score to each table dict
+
+    SM->>SA: build_context(tables with relevance_score)
+    SA->>SA: Reuses existing scores (zero API calls)
+    SA-->>SM: schema context string
+
+    SM-->>QGS: schema_context
+
+    QGS->>AI: generate_query(QueryContext)
+    AI-->>QGS: SQL + confidence + explanation
+    QGS->>AI: validate_query(sql, schema_context)
+    AI-->>QGS: validation result
+    QGS-->>User: {sql, confidence, explanation, validation}
+```
+
+### Step-by-step with file references
+
+1. **`generate_sql()`** (`services/query_service.py`) -- checks query cache, then calls `SchemaManager.get_optimal_schema_context()`
+2. **`get_optimal_schema_context()`** (`schema/manager.py`) -- checks schema cache, then calls `_find_relevant_tables()`
+3. **`_find_relevant_tables()`** (`schema/manager.py`) -- two strategies:
+   - **Strategy 1**: `SchemaEmbeddingManager.search_similar_with_embedding()` -- FAISS + TF-IDF hybrid search returns candidate tables + the query embedding
+   - **Strategy 2**: `SchemaAnalyzer.score_relevance_batch()` -- batch scores token-matched tables using precomputed query + element embeddings (reconstructed from FAISS via `get_table_embeddings()`)
+4. **`get_table_info()`** (`ports/schema_repository.py`) -- fetches full table metadata; manager attaches `relevance_score` to each table dict
+5. **`build_context()`** (`schema/analyzer.py`) -- builds schema context string within token limits; reuses pre-attached `relevance_score` (no re-scoring, zero additional embedding calls)
+6. **`generate_query()`** (`ports/ai_provider.py`) -- AI provider generates SQL from question + schema context
+7. **`validate_query()`** (`ports/ai_provider.py`) -- validates SQL syntax and safety
+
+## Schema Relevance Pipeline
+
+The pipeline in `_find_relevant_tables()` (`schema/manager.py`) uses two complementary strategies to find the most relevant tables for a query:
+
+### Strategy 1: Hybrid FAISS + TF-IDF Search
+
+`SchemaEmbeddingManager._search_similar_core()` (`schema/embedding_manager.py`) combines:
+
+- **Dense search (FAISS)**: Semantic similarity via vector embeddings (`IndexFlatIP` inner product). Captures meaning (e.g., "revenue" matches "sales_amount").
+- **Sparse search (TF-IDF)**: Exact keyword matching via `TfidfVectorizer` with unigram+bigram features. Captures exact names (e.g., "organizations" matches `organization` table).
+- **Hybrid score**: `0.5 * dense_score + 0.5 * sparse_score`. Equal weighting ensures both semantic meaning and exact keyword matches contribute.
+
+This returns candidate tables + the query embedding (for reuse downstream).
+
+### Strategy 2: Batch Scoring with Precomputed Embeddings
+
+`SchemaAnalyzer.score_relevance_batch()` (`schema/analyzer.py`) refines candidates:
+
+1. **Text-based scoring** (fast, no API calls): exact name match, normalized singular/plural matching, token overlap, column-name matching
+2. **Semantic scoring** (only for elements with text score < 0.8): uses precomputed embeddings passed from the manager, avoiding new API calls
+3. The manager passes both the `query_embedding` from Strategy 1 and `element_embeddings` reconstructed from the FAISS index via `get_table_embeddings()`
+
+### Score Reuse in build_context
+
+`SchemaManager.get_optimal_schema_context()` attaches the final `relevance_score` to each table dict before calling `SchemaAnalyzer.build_context()`. The analyzer checks for this key and skips re-scoring when present. This eliminates ~10-15 embedding API calls per query that would otherwise happen in the per-table `score_relevance()` path.
+
+**Net result**: ~1 embedding API call per query (for the initial FAISS search), down from ~15-18 without these optimizations.
+
+## Caching Layers
+
+The system has multiple caching layers:
+
+| Layer | Scope | Storage | TTL |
+|-------|-------|---------|-----|
+| **Query result cache** | Full SQL response | `CachePort` (external) | Indefinite (valid results) |
+| **Schema context cache** | Built schema string | `CachePort` (external) | Per query+db+tokens key |
+| **Table relevance cache** | Relevant tables list | In-memory dict | Per `SchemaManager` instance |
+| **All-tables cache** | Filtered table list | In-memory dict (`_schema_cache`) | Until `refresh_schema()` |
+| **FAISS + TF-IDF index** | Schema embeddings | Disk (pickle + FAISS) | Until `NLP2SQL_SCHEMA_CACHE_TTL_HOURS` expires |
+| **Query embedding cache** | Per-query embeddings | In-memory dict | Cleared per `_find_relevant_tables()` call |
+| **Repository disk cache** | Schema metadata (Redshift) | Pickle file | `NLP2SQL_SCHEMA_CACHE_TTL_HOURS` |
+
+## Design Decisions
+
+| Decision | Rationale |
+|----------|-----------|
+| **Ports & Adapters** | Swap AI/DB/embedding providers without touching core logic. Adding a new provider means implementing one port interface. |
+| **FAISS + TF-IDF hybrid (50/50)** | Semantic embeddings alone miss exact keyword matches (e.g., "count organizations" should find `organization` table). TF-IDF catches these. Equal weighting balances both signals. |
+| **Disk-cached FAISS index** | Avoids re-embedding 600+ schema elements on every run. Isolated per database+schema via MD5 hash of `{database_url}:{schema_name}`. |
+| **Precomputed embeddings in batch scoring** | `_find_relevant_tables` reuses the query embedding from FAISS search and reconstructs table embeddings from the index. Result: ~1 embedding API call per query instead of ~15. |
+| **relevance_score passthrough to build_context** | Manager already scored tables during `_find_relevant_tables`; analyzer reuses those scores in `build_context` instead of re-computing them (zero redundant API calls). |
+| **Repository disk cache (Redshift)** | Redshift's `SVV_TABLES`/`SVV_COLUMNS` queries are expensive. Bulk query + pickle cache avoids repeated system catalog scans. |
+| **Graceful embedding degradation** | System works without embeddings (text-based matching only). If `sentence-transformers` is not installed, logs a warning and continues. |
+| **Schema filters applied before indexing** | Filters reduce the number of elements indexed in FAISS, making search faster and more focused. |
+
+## Directory Structure
+
+```
+src/nlp2sql/
+├── core/           # Business entities (pure Python, no external dependencies)
+│   ├── entities.py         # Query, SQLQuery, DatabaseType
+│   └── database_prompts.py # SQL dialect hints for AI providers (centralized)
+├── ports/          # Interfaces/abstractions
+│   ├── ai_provider.py      # AIProviderPort, QueryContext, QueryResponse
+│   ├── schema_repository.py # SchemaRepositoryPort, TableInfo
+│   ├── embedding_provider.py # EmbeddingProviderPort
+│   ├── schema_strategy.py   # SchemaStrategyPort, SchemaContext, SchemaChunk
+│   ├── cache.py             # CachePort
+│   └── query_optimizer.py   # QueryOptimizerPort
+├── adapters/       # External implementations
+│   ├── openai_adapter.py           # OpenAI GPT
+│   ├── anthropic_adapter.py        # Anthropic Claude
+│   ├── gemini_adapter.py           # Google Gemini
+│   ├── postgres_repository.py      # PostgreSQL
+│   ├── redshift_adapter.py         # Amazon Redshift (with disk cache)
+│   ├── local_embedding_adapter.py  # sentence-transformers
+│   └── openai_embedding_adapter.py # OpenAI embeddings
+├── services/       # Application services
+│   └── query_service.py    # QueryGenerationService (orchestrator)
+├── schema/         # Schema management
+│   ├── manager.py           # SchemaManager (coordinates strategies)
+│   ├── analyzer.py          # SchemaAnalyzer (scoring, compression)
+│   ├── embedding_manager.py # SchemaEmbeddingManager (FAISS + TF-IDF)
+│   └── example_store.py     # ExampleStore (FAISS-indexed few-shot examples)
+├── config/         # Configuration
+│   └── settings.py          # Pydantic Settings (env vars)
+├── exceptions/     # Custom exception hierarchy
+└── factories.py    # RepositoryFactory
+```
+
+## Related Documentation
+
+- [API Reference](API.md) - Python API and CLI
+- [Configuration](CONFIGURATION.md) - Environment variables and design rationale
+- [Enterprise Guide](ENTERPRISE.md) - Large-scale deployment
+- [Redshift Support](Redshift.md) - Amazon Redshift setup

{nlp2sql-0.2.0rc5 → nlp2sql-0.2.0rc7}/docs/CONFIGURATION.md

@@ -55,6 +55,13 @@ export NLP2SQL_EMBEDDING_MODEL=all-MiniLM-L6-v2
 export NLP2SQL_EMBEDDING_PROVIDER=local
 ```
 
+**Design rationale for key variables:**
+
+- **`NLP2SQL_MAX_SCHEMA_TOKENS`** -- Limits the schema context sent to the AI provider. Higher values give the AI more schema information (better accuracy for complex joins) but cost more tokens. The default 8000 covers ~20-30 tables with full column definitions. For large schemas with many relevant tables, increase to 12000-16000.
+- **`NLP2SQL_SCHEMA_CACHE_TTL_HOURS`** -- Controls when the FAISS index and schema metadata are refreshed from the database. The 24h default balances freshness against the cost of re-querying system catalogs (especially expensive for Redshift's `SVV_TABLES`/`SVV_COLUMNS`). Set lower in development or when schema changes frequently.
+- **`NLP2SQL_EMBEDDINGS_DIR`** -- Where FAISS index files and TF-IDF metadata are stored on disk. Each database+schema combination gets its own subdirectory (MD5 hash of `{database_url}:{schema_name}`). Must be writable. In containerized environments (e.g., Claude Desktop MCP), the system falls back to `/tmp/nlp2sql_embeddings` if the default `./embeddings` is read-only.
+- **`NLP2SQL_EMBEDDING_MODEL`** -- The `sentence-transformers` model for local embeddings. `all-MiniLM-L6-v2` (384 dimensions) is a good balance of speed and quality. Changing this after an index is built requires clearing the cache (`nlp2sql cache clear --embeddings`) since dimensions must match.
+
 ### General Settings
 
 | Variable | Required | Default | Description |
@@ -76,6 +83,8 @@ export TOKENIZERS_PARALLELISM=false
 
 Schema filters reduce the database schema to relevant tables for better performance and accuracy.
 
+**Important:** Filters are applied *before* FAISS indexing (`SchemaManager._apply_schema_filters()` in `schema/manager.py`). Fewer elements indexed means faster semantic search and a more focused index. For enterprise databases (1000+ tables), well-chosen filters can reduce indexing time from minutes to seconds.
+
 ### Filter Options
 
 | Filter | Type | Description |