aetherdialect 0.1.0__tar.gz → 0.1.2__tar.gz

{aetherdialect-0.1.0/src/aetherdialect.egg-info → aetherdialect-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aetherdialect
-Version: 0.1.0
+Version: 0.1.2
 Summary: Deterministic, validation-first Text-to-SQL system for business databases
 Author-email: Akul Ameya <akul.ameya@gmail.com>
 License: MIT
@@ -10,14 +10,15 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: jsonschema<5,>=4.0
 Requires-Dist: openai<3,>=2.0.0
-Requires-Dist: sqlglot<30,>=29.0
 Requires-Dist: platformdirs<5,>=2.0.0
 Requires-Dist: python-dotenv<2,>=1.0.0
+Requires-Dist: SQLAlchemy<3,>=2.0
 Provides-Extra: databricks
+Requires-Dist: sqlglot<30,>=29.0; extra == "databricks"
 Requires-Dist: pyspark<4,>=3.3; extra == "databricks"
 Requires-Dist: databricks-sql-connector<4,>=3.0; extra == "databricks"
+Requires-Dist: databricks-sqlalchemy<3,>=2.0; extra == "databricks"
 Provides-Extra: postgresql
-Requires-Dist: SQLAlchemy<3,>=2.0; extra == "postgresql"
 Requires-Dist: psycopg2-binary<3,>=2.9; extra == "postgresql"
 Requires-Dist: pglast<8,>=5.0; extra == "postgresql"
 Provides-Extra: dev
@@ -35,6 +36,8 @@ Dynamic: license-file
 
 # Deterministic, validation-first Text-to-SQL for business databases
 
+Questions resolve more reliably when you state analytical intent explicitly—entities, grain, filters, time scope, and ordering—instead of leaving those details implied.
+
 ## Installation
 
 ```bash
@@ -44,14 +47,15 @@ pip install "text2sql[databricks]"
 pip install "text2sql[postgresql,databricks]"
 ```
 
-Requires Python ≥ 3.10 and an [OpenAI API key](https://platform.openai.com/api-keys).
+Requires Python ≥ 3.10 and either an [OpenAI API key](https://platform.openai.com/api-keys) or Azure OpenAI credentials.
 
-| Extra        | Brings in                                      | Use when                    |
-| ------------ | ---------------------------------------------- | --------------------------- |
-| `postgresql` | SQLAlchemy, PostgreSQL driver, `pglast`        | `engine="postgresql"`       |
-| `databricks` | PySpark, Databricks SQL connector              | `engine="databricks"`       |
+| Extra        | Brings in                                                                                        | Use when              |
+| ------------ | ------------------------------------------------------------------------------------------------ | --------------------- |
+| (base)       | **SQLAlchemy** (shared introspection / execution interface)                                      | Always installed      |
+| `postgresql` | PostgreSQL driver (`psycopg2-binary`), **`pglast`**                                              | `engine="postgresql"` |
+| `databricks` | Databricks SQL connector (preferred), PySpark (fallback), **`databricks-sqlalchemy`**, `sqlglot` | `engine="databricks"` |
 
-**SQL parsing for validation:** PostgreSQL uses **`pglast`** for structural AST checks (join pairs, CTE bodies, `ast_validate`). Databricks / Spark SQL uses **`sqlglot`** with the **Spark** dialect. The base package already depends on `sqlglot`; `pglast` is installed with the `postgresql` extra.
+**SQL parsing for validation:** PostgreSQL uses **`pglast`** for structural AST checks (join pairs, CTE bodies, `ast_validate`). Databricks / Spark SQL uses **`sqlglot`** with the **Spark** dialect.
 
 ---
 
@@ -71,7 +75,9 @@ t2s = Text2SQL(
 t2s.run_interactive()
 ```
 
-Constructor options, modes, optional files, and the full method list are in **[USAGE.md](USAGE.md)**.
+Constructor options, credentials (`set_openai_api_key`, `set_azure_openai_api_key`, `set_env`), modes, and the full API are in **[USAGE.md](USAGE.md)**. Pass **`artifacts_dir=`** to put the per-connection cache under a root you choose; otherwise it lives under the platform user-data directory (see USAGE.md).
+
+**Interactive two ways:** **`run_interactive()`** is a stdin loop. For your own UI or protocol, use **`Text2SQL.pipeline_session()`** and drive **`PipelineSession`** step by step: one natural-language question is a **turn** that may return several **`SessionEvent`** objects (prompt / answer / …) until **`done`** is true. Types and methods are documented in **[USAGE.md](USAGE.md)**.
 
 ---
 
@@ -91,7 +97,7 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 - **Determinism over creativity** — prefer a correct, boring plan to a novel one.
 - **Correct joins over clever SQL** — join paths come from **foreign keys** and precomputed paths, not free-form guessing.
 - **Validate before execute** — schema checks, intent consistency, join shape, and dialect-safe **read-only** `SELECT` rules.
-- **Minimal LLM surface** — parse intent, resolve ambiguous joins when needed, generate or repair SQL; everything else is rules and stores.
+- **Minimal LLM surface** — parse intent, resolve ambiguous joins when needed, repair SQL; everything else is rules and stores.
 - **Safe defaults for non-analysts** — narrow allowed SQL; you still choose **database credentials** (read-only vs write-capable is outside this library).
 
 ---
@@ -100,35 +106,35 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 
 **Backends**
 
-- PostgreSQL via SQLAlchemy.
-- Databricks via Unity Catalog introspection (with optional DDL file fallback when the catalog is empty).
+- PostgreSQL
+- Databricks
 
 **Schema**
 
 - Load from **live introspection** (primary).
 - Optional **`CREATE TABLE` file** as extra or fallback (especially when you cannot reach all metadata from the driver).
 - Cached schema snapshot per connection fingerprint so restarts avoid re-reflecting unchanged databases.
-- **Table roles** (e.g. fact vs dimension), **column roles** (measure, categorical, temporal, identifier, etc.), **filter / aggregation / HAVING** allowances per column, **value domains** from profiling.
-- Optional **human notes** (plain text) fed once when the schema graph is built: richer **descriptions**, **roles**, and optional **sensitivity** labels — without renaming tables or inventing columns.
+- **Table roles** (e.g. fact vs dimension), **column roles** (measure, categorical, temporal, identifier, etc.), **filter / aggregation / HAVING** allowances per column, **value domains** from profiling — all assigned when the graph is built (reflection, DDL, profiling, and optional notes).
+- Optional **human notes** (plain text), via `Text2SQL(..., notes_file=...)` (see **[USAGE.md](USAGE.md)**): merged when the graph is built or when notes change; if the cache already contains notes and you omit `notes_file` on a later run, cached roles and hints are kept.
 - Optional **deny lists** for tables or columns so they stay out of prompts and can be stripped from intents.
 
 **Intent / SQL shape (analytical subset)**
 
 - **Queries:** `SELECT` only (enforced with pattern checks and dialect parsing). **CTEs** reuse the same intent model as the outer query.
-- **Joins:** only along **FK-backed paths**; join type for injected joins is chosen **deterministically** from table roles (e.g. dimension side as `LEFT` where applicable).
+- **Joins:** related tables are wired using the schema’s relationships; when more than one valid path could link the same tables, one coherent path is chosen for the whole query, and join style follows table roles (e.g. `LEFT` toward dimensions where that fits). Self-joins use **CTEs** instead of repeating the same base table in one `FROM` chain.
 - **Select list:** bare columns, **aggregates** (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, etc.), **arithmetic and string expressions** where the schema allows them, **`DISTINCT`**, and **scalar functions** subject to column metadata.
 - **Filters / boolean logic:** comparisons, **`AND` / `OR`**, **`IN`**, **`LIKE`**; **`ILIKE` / `NOT ILIKE` on PostgreSQL only** (intent stays dialect-agnostic; SQL rendering differs). Null / boolean value normalization in the repair chain.
 - **`BETWEEN`** in intent is **decomposed** into a pair of comparable predicates.
 - **Grouping / ordering:** **`GROUP BY`**, **`HAVING`** (aggregate-aware), **`ORDER BY`**, **`LIMIT`**; rules tie **grain** (row-level vs grouped) to aggregates and grouped columns.
 - **Dates:** structured **`date_window`** (anchor unit + offset) and **date-difference** filters between columns where supported.
-- **Windows:** `ROW_NUMBER`, `RANK`, `DENSE_RANK`, and windowed `SUM` / `AVG` on select columns (main query and CTEs).
+- **Windows:** `ROW_NUMBER`, `RANK`, `DENSE_RANK`, and windowed `SUM`/`AVG`, `LAG`, `LEAD`, `FIRST_VALUE`, and `LAST_VALUE` on select columns (main query and CTEs).
 - **`CASE` / `WHEN`:** only in the **select list** in the intent model (not in `WHERE` / `HAVING`).
 - **Arrays / lists:** membership-style filters; SQL uses dialect-appropriate forms; optional **UNNEST / EXPLODE-style** expansion in CTE select lists for typed array columns.
 - **Metadata:** **UNIQUE** (and related) when reflection or DDL exposes it, for ranking “human readable” identifiers.
 
 **Operational modes**
 
-- **Interactive** — ask questions, accept/reject, results export.
+- **Interactive** — ask questions, accept/reject, results export; via **`run_interactive()`** or a programmatic **`PipelineSession`** (see Quickstart above and **[USAGE.md](USAGE.md)**).
 - **Coverage simulator** — seed questions → gold intents → **deterministic expansion** (many operators, deduplicated) → validate/execute → NL question generation for new templates.
 - **QSim** — reproducible synthetic questions from schema and profiles (seeded randomness).
 
@@ -137,7 +143,7 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 ## What it is not
 
 - Not a **full SQL** or **stored-procedure** generator: no `UNION`/`INTERSECT`/`EXCEPT`, no correlated subqueries, no `EXISTS`, no `LATERAL`, no **DML/DDL**, no arbitrary **RIGHT/FULL OUTER** join policy in the constrained path.
-- Not a substitute for **database security**: use credentials with **least privilege** (`SELECT` / `EXPLAIN` as you prefer).
+- Not a substitute for **database security**: use credentials with **least privilege** (`SELECT` and `EXPLAIN`).
 - Not **schema-agnostic**: quality depends on **FKs**, sensible types, and optional notes for domain language.
 
 ---
@@ -145,31 +151,27 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 ## How a question becomes SQL
 
 1. **Template match** — if a trusted pattern fits, reuse parameterized SQL (often **no** SQL LLM call).
-2. **Intent parse** — structured intent from the question + schema summary, then a long **deterministic repair chain** (see below).
+2. **Intent parse** — structured intent from the question + schema summary, then a long **deterministic repair chain**.
 3. **Join resolution** — choose among valid **FK paths** for the intent’s tables; disambiguation may use the LLM when multiple paths tie.
-4. **SQL generation & validation** — deterministic skeleton, injected joins, LLM fill/repair **under constraints**, then **semantic validation**, **`SELECT`-only / forbidden-pattern checks**, **dialect AST** validation (**`pglast`** or **`sqlglot`**), and optionally **`EXPLAIN`** when an engine is available.
+4. **SQL generation & validation** — deterministic skeleton, injected joins, LLM fill/repair **under constraints**, then **semantic validation**, **`SELECT`-only / forbidden-pattern checks**, **dialect AST** validation (**`pglast`** or **`sqlglot`**), and **`EXPLAIN`**.
 5. **Execute** (where the mode allows) and **learn** — accept → promote template trust; reject → record negative pattern.
 
 ---
 
 ## Validation (layers)
 
-- **Safety / shape** — `SELECT`-only enforcement, configurable **forbidden SQL** substrings, then **dialect `ast_validate`** (`pglast` on PostgreSQL, **`sqlglot` (Spark)** on Databricks). When a live **engine** is passed in, **`EXPLAIN`** can be used as an extra executability check.
+- **Safety / shape** — `SELECT`-only enforcement, configurable **forbidden SQL** substrings, then **dialect `ast_validate`** (**`pglast`** or **`sqlglot`**). **`EXPLAIN`** is used as an extra executability check.
 - **Schema vs intent** — tables/columns/CTEs, **selectability**, access and sensitivity policy, window / CASE / array shapes, filter and HAVING ops per column, aggregate roles in select/HAVING/ORDER BY, scalar function typing, filter value types vs column types, null/date-window/date-diff rules.
-- **Semantic consistency** — grouped queries require proper aggregation; contradictions and impossible HAVING; **grain** alignment (one of many cross-checks, not the only story).
+- **Semantic consistency** — grouped queries require proper aggregation; contradictions and impossible HAVING; **grain** alignment.
 - **Joins** — paths must match the FK graph; guarded path avoids ad-hoc join guessing.
 
-## Deterministic repairs (after intent parse)
-
-Applied in order (high level): `COUNT(*)` normalization; CTE naming and output aliases; qualify CTE outputs; sanitize table names; grain rules for grouped CTE usage and **grain consistency**; strip redundant `GROUP BY`; normalize filters/HAVING; null-equality fixes; strip join-condition leakage into filters; per-CTE sort order; simplify expressions; `IN` value normalization; date-diff classification and raw-value fixes; **`BETWEEN` → paired predicates**; auto-repair filters/HAVING; strip impossible HAVING; FK filter type repair; filter value case / enum alignment; boolean and null filter values; expand FK selects to descriptive columns; deduplicate contradictory filters; redundant PK re-qualification; **window**, **CASE**, and **array** intent repairs; sensitivity and access policy enforcement on the intent.
-
 ---
 
 ## Learning and reuse
 
 - **Accepted templates** — intent fingerprint, parameterized SQL, optional example question, **trust** that rises with validation and falls with rejection.
 - **Rejected templates** — categorized failures so similar bad intents are discouraged.
-- Persistence is under a **per-connection artifact directory** (see USAGE); you can back it up or reset it by removing that directory.
+- Persistence is under a **per-connection artifact directory** (see **[USAGE.md](USAGE.md)**); you can back it up or reset it by removing that directory.
 
 ---
 
@@ -180,7 +182,7 @@ Applied in order (high level): `COUNT(*)` normalization; CTE naming and output a
 3. Resolve joins once per table set where possible; validate and **execute** as a gate.
 4. One LLM step to produce a **natural language question** from the SQL, with a **realism** filter.
 
-The simulator loads at most **500** seed lines per run (fixed internal cap; larger files are truncated). **`estimate_simulator_costs`** uses the same loader, so its seed count reflects that cap. It returns **rough** LLM-call and execution estimates from a seed file and schema stats.
+The simulator loads at most **500** seed lines per run (fixed internal cap; larger files are truncated). **`estimate_simulator_costs`** uses the same loader, prints **rough** per-phase upper-bound lines to stdout, and returns **`None`**.
 
 ---
 

{aetherdialect-0.1.0 → aetherdialect-0.1.2}/README.md

@@ -1,5 +1,7 @@
 # Deterministic, validation-first Text-to-SQL for business databases
 
+Questions resolve more reliably when you state analytical intent explicitly—entities, grain, filters, time scope, and ordering—instead of leaving those details implied.
+
 ## Installation
 
 ```bash
@@ -9,14 +11,15 @@ pip install "text2sql[databricks]"
 pip install "text2sql[postgresql,databricks]"
 ```
 
-Requires Python ≥ 3.10 and an [OpenAI API key](https://platform.openai.com/api-keys).
+Requires Python ≥ 3.10 and either an [OpenAI API key](https://platform.openai.com/api-keys) or Azure OpenAI credentials.
 
-| Extra        | Brings in                                      | Use when                    |
-| ------------ | ---------------------------------------------- | --------------------------- |
-| `postgresql` | SQLAlchemy, PostgreSQL driver, `pglast`        | `engine="postgresql"`       |
-| `databricks` | PySpark, Databricks SQL connector              | `engine="databricks"`       |
+| Extra        | Brings in                                                                                        | Use when              |
+| ------------ | ------------------------------------------------------------------------------------------------ | --------------------- |
+| (base)       | **SQLAlchemy** (shared introspection / execution interface)                                      | Always installed      |
+| `postgresql` | PostgreSQL driver (`psycopg2-binary`), **`pglast`**                                              | `engine="postgresql"` |
+| `databricks` | Databricks SQL connector (preferred), PySpark (fallback), **`databricks-sqlalchemy`**, `sqlglot` | `engine="databricks"` |
 
-**SQL parsing for validation:** PostgreSQL uses **`pglast`** for structural AST checks (join pairs, CTE bodies, `ast_validate`). Databricks / Spark SQL uses **`sqlglot`** with the **Spark** dialect. The base package already depends on `sqlglot`; `pglast` is installed with the `postgresql` extra.
+**SQL parsing for validation:** PostgreSQL uses **`pglast`** for structural AST checks (join pairs, CTE bodies, `ast_validate`). Databricks / Spark SQL uses **`sqlglot`** with the **Spark** dialect.
 
 ---
 
@@ -36,7 +39,9 @@ t2s = Text2SQL(
 t2s.run_interactive()
 ```
 
-Constructor options, modes, optional files, and the full method list are in **[USAGE.md](USAGE.md)**.
+Constructor options, credentials (`set_openai_api_key`, `set_azure_openai_api_key`, `set_env`), modes, and the full API are in **[USAGE.md](USAGE.md)**. Pass **`artifacts_dir=`** to put the per-connection cache under a root you choose; otherwise it lives under the platform user-data directory (see USAGE.md).
+
+**Interactive two ways:** **`run_interactive()`** is a stdin loop. For your own UI or protocol, use **`Text2SQL.pipeline_session()`** and drive **`PipelineSession`** step by step: one natural-language question is a **turn** that may return several **`SessionEvent`** objects (prompt / answer / …) until **`done`** is true. Types and methods are documented in **[USAGE.md](USAGE.md)**.
 
 ---
 
@@ -56,7 +61,7 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 - **Determinism over creativity** — prefer a correct, boring plan to a novel one.
 - **Correct joins over clever SQL** — join paths come from **foreign keys** and precomputed paths, not free-form guessing.
 - **Validate before execute** — schema checks, intent consistency, join shape, and dialect-safe **read-only** `SELECT` rules.
-- **Minimal LLM surface** — parse intent, resolve ambiguous joins when needed, generate or repair SQL; everything else is rules and stores.
+- **Minimal LLM surface** — parse intent, resolve ambiguous joins when needed, repair SQL; everything else is rules and stores.
 - **Safe defaults for non-analysts** — narrow allowed SQL; you still choose **database credentials** (read-only vs write-capable is outside this library).
 
 ---
@@ -65,35 +70,35 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 
 **Backends**
 
-- PostgreSQL via SQLAlchemy.
-- Databricks via Unity Catalog introspection (with optional DDL file fallback when the catalog is empty).
+- PostgreSQL
+- Databricks
 
 **Schema**
 
 - Load from **live introspection** (primary).
 - Optional **`CREATE TABLE` file** as extra or fallback (especially when you cannot reach all metadata from the driver).
 - Cached schema snapshot per connection fingerprint so restarts avoid re-reflecting unchanged databases.
-- **Table roles** (e.g. fact vs dimension), **column roles** (measure, categorical, temporal, identifier, etc.), **filter / aggregation / HAVING** allowances per column, **value domains** from profiling.
-- Optional **human notes** (plain text) fed once when the schema graph is built: richer **descriptions**, **roles**, and optional **sensitivity** labels — without renaming tables or inventing columns.
+- **Table roles** (e.g. fact vs dimension), **column roles** (measure, categorical, temporal, identifier, etc.), **filter / aggregation / HAVING** allowances per column, **value domains** from profiling — all assigned when the graph is built (reflection, DDL, profiling, and optional notes).
+- Optional **human notes** (plain text), via `Text2SQL(..., notes_file=...)` (see **[USAGE.md](USAGE.md)**): merged when the graph is built or when notes change; if the cache already contains notes and you omit `notes_file` on a later run, cached roles and hints are kept.
 - Optional **deny lists** for tables or columns so they stay out of prompts and can be stripped from intents.
 
 **Intent / SQL shape (analytical subset)**
 
 - **Queries:** `SELECT` only (enforced with pattern checks and dialect parsing). **CTEs** reuse the same intent model as the outer query.
-- **Joins:** only along **FK-backed paths**; join type for injected joins is chosen **deterministically** from table roles (e.g. dimension side as `LEFT` where applicable).
+- **Joins:** related tables are wired using the schema’s relationships; when more than one valid path could link the same tables, one coherent path is chosen for the whole query, and join style follows table roles (e.g. `LEFT` toward dimensions where that fits). Self-joins use **CTEs** instead of repeating the same base table in one `FROM` chain.
 - **Select list:** bare columns, **aggregates** (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, etc.), **arithmetic and string expressions** where the schema allows them, **`DISTINCT`**, and **scalar functions** subject to column metadata.
 - **Filters / boolean logic:** comparisons, **`AND` / `OR`**, **`IN`**, **`LIKE`**; **`ILIKE` / `NOT ILIKE` on PostgreSQL only** (intent stays dialect-agnostic; SQL rendering differs). Null / boolean value normalization in the repair chain.
 - **`BETWEEN`** in intent is **decomposed** into a pair of comparable predicates.
 - **Grouping / ordering:** **`GROUP BY`**, **`HAVING`** (aggregate-aware), **`ORDER BY`**, **`LIMIT`**; rules tie **grain** (row-level vs grouped) to aggregates and grouped columns.
 - **Dates:** structured **`date_window`** (anchor unit + offset) and **date-difference** filters between columns where supported.
-- **Windows:** `ROW_NUMBER`, `RANK`, `DENSE_RANK`, and windowed `SUM` / `AVG` on select columns (main query and CTEs).
+- **Windows:** `ROW_NUMBER`, `RANK`, `DENSE_RANK`, and windowed `SUM`/`AVG`, `LAG`, `LEAD`, `FIRST_VALUE`, and `LAST_VALUE` on select columns (main query and CTEs).
 - **`CASE` / `WHEN`:** only in the **select list** in the intent model (not in `WHERE` / `HAVING`).
 - **Arrays / lists:** membership-style filters; SQL uses dialect-appropriate forms; optional **UNNEST / EXPLODE-style** expansion in CTE select lists for typed array columns.
 - **Metadata:** **UNIQUE** (and related) when reflection or DDL exposes it, for ranking “human readable” identifiers.
 
 **Operational modes**
 
-- **Interactive** — ask questions, accept/reject, results export.
+- **Interactive** — ask questions, accept/reject, results export; via **`run_interactive()`** or a programmatic **`PipelineSession`** (see Quickstart above and **[USAGE.md](USAGE.md)**).
 - **Coverage simulator** — seed questions → gold intents → **deterministic expansion** (many operators, deduplicated) → validate/execute → NL question generation for new templates.
 - **QSim** — reproducible synthetic questions from schema and profiles (seeded randomness).
 
@@ -102,7 +107,7 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 ## What it is not
 
 - Not a **full SQL** or **stored-procedure** generator: no `UNION`/`INTERSECT`/`EXCEPT`, no correlated subqueries, no `EXISTS`, no `LATERAL`, no **DML/DDL**, no arbitrary **RIGHT/FULL OUTER** join policy in the constrained path.
-- Not a substitute for **database security**: use credentials with **least privilege** (`SELECT` / `EXPLAIN` as you prefer).
+- Not a substitute for **database security**: use credentials with **least privilege** (`SELECT` and `EXPLAIN`).
 - Not **schema-agnostic**: quality depends on **FKs**, sensible types, and optional notes for domain language.
 
 ---
@@ -110,31 +115,27 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 ## How a question becomes SQL
 
 1. **Template match** — if a trusted pattern fits, reuse parameterized SQL (often **no** SQL LLM call).
-2. **Intent parse** — structured intent from the question + schema summary, then a long **deterministic repair chain** (see below).
+2. **Intent parse** — structured intent from the question + schema summary, then a long **deterministic repair chain**.
 3. **Join resolution** — choose among valid **FK paths** for the intent’s tables; disambiguation may use the LLM when multiple paths tie.
-4. **SQL generation & validation** — deterministic skeleton, injected joins, LLM fill/repair **under constraints**, then **semantic validation**, **`SELECT`-only / forbidden-pattern checks**, **dialect AST** validation (**`pglast`** or **`sqlglot`**), and optionally **`EXPLAIN`** when an engine is available.
+4. **SQL generation & validation** — deterministic skeleton, injected joins, LLM fill/repair **under constraints**, then **semantic validation**, **`SELECT`-only / forbidden-pattern checks**, **dialect AST** validation (**`pglast`** or **`sqlglot`**), and **`EXPLAIN`**.
 5. **Execute** (where the mode allows) and **learn** — accept → promote template trust; reject → record negative pattern.
 
 ---
 
 ## Validation (layers)
 
-- **Safety / shape** — `SELECT`-only enforcement, configurable **forbidden SQL** substrings, then **dialect `ast_validate`** (`pglast` on PostgreSQL, **`sqlglot` (Spark)** on Databricks). When a live **engine** is passed in, **`EXPLAIN`** can be used as an extra executability check.
+- **Safety / shape** — `SELECT`-only enforcement, configurable **forbidden SQL** substrings, then **dialect `ast_validate`** (**`pglast`** or **`sqlglot`**). **`EXPLAIN`** is used as an extra executability check.
 - **Schema vs intent** — tables/columns/CTEs, **selectability**, access and sensitivity policy, window / CASE / array shapes, filter and HAVING ops per column, aggregate roles in select/HAVING/ORDER BY, scalar function typing, filter value types vs column types, null/date-window/date-diff rules.
-- **Semantic consistency** — grouped queries require proper aggregation; contradictions and impossible HAVING; **grain** alignment (one of many cross-checks, not the only story).
+- **Semantic consistency** — grouped queries require proper aggregation; contradictions and impossible HAVING; **grain** alignment.
 - **Joins** — paths must match the FK graph; guarded path avoids ad-hoc join guessing.
 
-## Deterministic repairs (after intent parse)
-
-Applied in order (high level): `COUNT(*)` normalization; CTE naming and output aliases; qualify CTE outputs; sanitize table names; grain rules for grouped CTE usage and **grain consistency**; strip redundant `GROUP BY`; normalize filters/HAVING; null-equality fixes; strip join-condition leakage into filters; per-CTE sort order; simplify expressions; `IN` value normalization; date-diff classification and raw-value fixes; **`BETWEEN` → paired predicates**; auto-repair filters/HAVING; strip impossible HAVING; FK filter type repair; filter value case / enum alignment; boolean and null filter values; expand FK selects to descriptive columns; deduplicate contradictory filters; redundant PK re-qualification; **window**, **CASE**, and **array** intent repairs; sensitivity and access policy enforcement on the intent.
-
 ---
 
 ## Learning and reuse
 
 - **Accepted templates** — intent fingerprint, parameterized SQL, optional example question, **trust** that rises with validation and falls with rejection.
 - **Rejected templates** — categorized failures so similar bad intents are discouraged.
-- Persistence is under a **per-connection artifact directory** (see USAGE); you can back it up or reset it by removing that directory.
+- Persistence is under a **per-connection artifact directory** (see **[USAGE.md](USAGE.md)**); you can back it up or reset it by removing that directory.
 
 ---
 
@@ -145,7 +146,7 @@ Applied in order (high level): `COUNT(*)` normalization; CTE naming and output a
 3. Resolve joins once per table set where possible; validate and **execute** as a gate.
 4. One LLM step to produce a **natural language question** from the SQL, with a **realism** filter.
 
-The simulator loads at most **500** seed lines per run (fixed internal cap; larger files are truncated). **`estimate_simulator_costs`** uses the same loader, so its seed count reflects that cap. It returns **rough** LLM-call and execution estimates from a seed file and schema stats.
+The simulator loads at most **500** seed lines per run (fixed internal cap; larger files are truncated). **`estimate_simulator_costs`** uses the same loader, prints **rough** per-phase upper-bound lines to stdout, and returns **`None`**.
 
 ---
 

{aetherdialect-0.1.0 → aetherdialect-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aetherdialect"
-version = "0.1.0"
+version = "0.1.2"
 description = "Deterministic, validation-first Text-to-SQL system for business databases"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -13,15 +13,19 @@ authors = [{name = "Akul Ameya", email = "akul.ameya@gmail.com"}]
 dependencies = [
     "jsonschema>=4.0,<5",
     "openai>=2.0.0,<3",
-    "sqlglot>=29.0,<30",
     "platformdirs>=2.0.0,<5",
     "python-dotenv>=1.0.0,<2",
+    "SQLAlchemy>=2.0,<3",
 ]
 
 [project.optional-dependencies]
-databricks = ["pyspark>=3.3,<4", "databricks-sql-connector>=3.0,<4"]
+databricks = [
+    "sqlglot>=29.0,<30",
+    "pyspark>=3.3,<4",
+    "databricks-sql-connector>=3.0,<4",
+    "databricks-sqlalchemy>=2.0,<3",
+]
 postgresql = [
-    "SQLAlchemy>=2.0,<3",
     "psycopg2-binary>=2.9,<3",
     "pglast>=5.0,<8",
 ]
@@ -62,12 +66,19 @@ quote-style = "double"
 indent-style = "space"
 line-ending = "auto"
 
+[tool.docformatter]
+wrap-summaries = 72
+wrap-descriptions = 72
+style = "google"
+force-wrap = true
+
 [tool.mypy]
 python_version = "3.10"
 strict = true
 
 [tool.pytest.ini_options]
 testpaths = ["tests", "live_tests"]
+pythonpath = ["src"]
 markers = [
     "live: marks tests that require a live LLM and database connection (deselect with '-m \"not live\"')",
 ]

{aetherdialect-0.1.0 → aetherdialect-0.1.2/src/aetherdialect.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aetherdialect
-Version: 0.1.0
+Version: 0.1.2
 Summary: Deterministic, validation-first Text-to-SQL system for business databases
 Author-email: Akul Ameya <akul.ameya@gmail.com>
 License: MIT
@@ -10,14 +10,15 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: jsonschema<5,>=4.0
 Requires-Dist: openai<3,>=2.0.0
-Requires-Dist: sqlglot<30,>=29.0
 Requires-Dist: platformdirs<5,>=2.0.0
 Requires-Dist: python-dotenv<2,>=1.0.0
+Requires-Dist: SQLAlchemy<3,>=2.0
 Provides-Extra: databricks
+Requires-Dist: sqlglot<30,>=29.0; extra == "databricks"
 Requires-Dist: pyspark<4,>=3.3; extra == "databricks"
 Requires-Dist: databricks-sql-connector<4,>=3.0; extra == "databricks"
+Requires-Dist: databricks-sqlalchemy<3,>=2.0; extra == "databricks"
 Provides-Extra: postgresql
-Requires-Dist: SQLAlchemy<3,>=2.0; extra == "postgresql"
 Requires-Dist: psycopg2-binary<3,>=2.9; extra == "postgresql"
 Requires-Dist: pglast<8,>=5.0; extra == "postgresql"
 Provides-Extra: dev
@@ -35,6 +36,8 @@ Dynamic: license-file
 
 # Deterministic, validation-first Text-to-SQL for business databases
 
+Questions resolve more reliably when you state analytical intent explicitly—entities, grain, filters, time scope, and ordering—instead of leaving those details implied.
+
 ## Installation
 
 ```bash
@@ -44,14 +47,15 @@ pip install "text2sql[databricks]"
 pip install "text2sql[postgresql,databricks]"
 ```
 
-Requires Python ≥ 3.10 and an [OpenAI API key](https://platform.openai.com/api-keys).
+Requires Python ≥ 3.10 and either an [OpenAI API key](https://platform.openai.com/api-keys) or Azure OpenAI credentials.
 
-| Extra        | Brings in                                      | Use when                    |
-| ------------ | ---------------------------------------------- | --------------------------- |
-| `postgresql` | SQLAlchemy, PostgreSQL driver, `pglast`        | `engine="postgresql"`       |
-| `databricks` | PySpark, Databricks SQL connector              | `engine="databricks"`       |
+| Extra        | Brings in                                                                                        | Use when              |
+| ------------ | ------------------------------------------------------------------------------------------------ | --------------------- |
+| (base)       | **SQLAlchemy** (shared introspection / execution interface)                                      | Always installed      |
+| `postgresql` | PostgreSQL driver (`psycopg2-binary`), **`pglast`**                                              | `engine="postgresql"` |
+| `databricks` | Databricks SQL connector (preferred), PySpark (fallback), **`databricks-sqlalchemy`**, `sqlglot` | `engine="databricks"` |
 
-**SQL parsing for validation:** PostgreSQL uses **`pglast`** for structural AST checks (join pairs, CTE bodies, `ast_validate`). Databricks / Spark SQL uses **`sqlglot`** with the **Spark** dialect. The base package already depends on `sqlglot`; `pglast` is installed with the `postgresql` extra.
+**SQL parsing for validation:** PostgreSQL uses **`pglast`** for structural AST checks (join pairs, CTE bodies, `ast_validate`). Databricks / Spark SQL uses **`sqlglot`** with the **Spark** dialect.
 
 ---
 
@@ -71,7 +75,9 @@ t2s = Text2SQL(
 t2s.run_interactive()
 ```
 
-Constructor options, modes, optional files, and the full method list are in **[USAGE.md](USAGE.md)**.
+Constructor options, credentials (`set_openai_api_key`, `set_azure_openai_api_key`, `set_env`), modes, and the full API are in **[USAGE.md](USAGE.md)**. Pass **`artifacts_dir=`** to put the per-connection cache under a root you choose; otherwise it lives under the platform user-data directory (see USAGE.md).
+
+**Interactive two ways:** **`run_interactive()`** is a stdin loop. For your own UI or protocol, use **`Text2SQL.pipeline_session()`** and drive **`PipelineSession`** step by step: one natural-language question is a **turn** that may return several **`SessionEvent`** objects (prompt / answer / …) until **`done`** is true. Types and methods are documented in **[USAGE.md](USAGE.md)**.
 
 ---
 
@@ -91,7 +97,7 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 - **Determinism over creativity** — prefer a correct, boring plan to a novel one.
 - **Correct joins over clever SQL** — join paths come from **foreign keys** and precomputed paths, not free-form guessing.
 - **Validate before execute** — schema checks, intent consistency, join shape, and dialect-safe **read-only** `SELECT` rules.
-- **Minimal LLM surface** — parse intent, resolve ambiguous joins when needed, generate or repair SQL; everything else is rules and stores.
+- **Minimal LLM surface** — parse intent, resolve ambiguous joins when needed, repair SQL; everything else is rules and stores.
 - **Safe defaults for non-analysts** — narrow allowed SQL; you still choose **database credentials** (read-only vs write-capable is outside this library).
 
 ---
@@ -100,35 +106,35 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 
 **Backends**
 
-- PostgreSQL via SQLAlchemy.
-- Databricks via Unity Catalog introspection (with optional DDL file fallback when the catalog is empty).
+- PostgreSQL
+- Databricks
 
 **Schema**
 
 - Load from **live introspection** (primary).
 - Optional **`CREATE TABLE` file** as extra or fallback (especially when you cannot reach all metadata from the driver).
 - Cached schema snapshot per connection fingerprint so restarts avoid re-reflecting unchanged databases.
-- **Table roles** (e.g. fact vs dimension), **column roles** (measure, categorical, temporal, identifier, etc.), **filter / aggregation / HAVING** allowances per column, **value domains** from profiling.
-- Optional **human notes** (plain text) fed once when the schema graph is built: richer **descriptions**, **roles**, and optional **sensitivity** labels — without renaming tables or inventing columns.
+- **Table roles** (e.g. fact vs dimension), **column roles** (measure, categorical, temporal, identifier, etc.), **filter / aggregation / HAVING** allowances per column, **value domains** from profiling — all assigned when the graph is built (reflection, DDL, profiling, and optional notes).
+- Optional **human notes** (plain text), via `Text2SQL(..., notes_file=...)` (see **[USAGE.md](USAGE.md)**): merged when the graph is built or when notes change; if the cache already contains notes and you omit `notes_file` on a later run, cached roles and hints are kept.
 - Optional **deny lists** for tables or columns so they stay out of prompts and can be stripped from intents.
 
 **Intent / SQL shape (analytical subset)**
 
 - **Queries:** `SELECT` only (enforced with pattern checks and dialect parsing). **CTEs** reuse the same intent model as the outer query.
-- **Joins:** only along **FK-backed paths**; join type for injected joins is chosen **deterministically** from table roles (e.g. dimension side as `LEFT` where applicable).
+- **Joins:** related tables are wired using the schema’s relationships; when more than one valid path could link the same tables, one coherent path is chosen for the whole query, and join style follows table roles (e.g. `LEFT` toward dimensions where that fits). Self-joins use **CTEs** instead of repeating the same base table in one `FROM` chain.
 - **Select list:** bare columns, **aggregates** (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, etc.), **arithmetic and string expressions** where the schema allows them, **`DISTINCT`**, and **scalar functions** subject to column metadata.
 - **Filters / boolean logic:** comparisons, **`AND` / `OR`**, **`IN`**, **`LIKE`**; **`ILIKE` / `NOT ILIKE` on PostgreSQL only** (intent stays dialect-agnostic; SQL rendering differs). Null / boolean value normalization in the repair chain.
 - **`BETWEEN`** in intent is **decomposed** into a pair of comparable predicates.
 - **Grouping / ordering:** **`GROUP BY`**, **`HAVING`** (aggregate-aware), **`ORDER BY`**, **`LIMIT`**; rules tie **grain** (row-level vs grouped) to aggregates and grouped columns.
 - **Dates:** structured **`date_window`** (anchor unit + offset) and **date-difference** filters between columns where supported.
-- **Windows:** `ROW_NUMBER`, `RANK`, `DENSE_RANK`, and windowed `SUM` / `AVG` on select columns (main query and CTEs).
+- **Windows:** `ROW_NUMBER`, `RANK`, `DENSE_RANK`, and windowed `SUM`/`AVG`, `LAG`, `LEAD`, `FIRST_VALUE`, and `LAST_VALUE` on select columns (main query and CTEs).
 - **`CASE` / `WHEN`:** only in the **select list** in the intent model (not in `WHERE` / `HAVING`).
 - **Arrays / lists:** membership-style filters; SQL uses dialect-appropriate forms; optional **UNNEST / EXPLODE-style** expansion in CTE select lists for typed array columns.
 - **Metadata:** **UNIQUE** (and related) when reflection or DDL exposes it, for ranking “human readable” identifiers.
 
 **Operational modes**
 
-- **Interactive** — ask questions, accept/reject, results export.
+- **Interactive** — ask questions, accept/reject, results export; via **`run_interactive()`** or a programmatic **`PipelineSession`** (see Quickstart above and **[USAGE.md](USAGE.md)**).
 - **Coverage simulator** — seed questions → gold intents → **deterministic expansion** (many operators, deduplicated) → validate/execute → NL question generation for new templates.
 - **QSim** — reproducible synthetic questions from schema and profiles (seeded randomness).
 
@@ -137,7 +143,7 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 ## What it is not
 
 - Not a **full SQL** or **stored-procedure** generator: no `UNION`/`INTERSECT`/`EXCEPT`, no correlated subqueries, no `EXISTS`, no `LATERAL`, no **DML/DDL**, no arbitrary **RIGHT/FULL OUTER** join policy in the constrained path.
-- Not a substitute for **database security**: use credentials with **least privilege** (`SELECT` / `EXPLAIN` as you prefer).
+- Not a substitute for **database security**: use credentials with **least privilege** (`SELECT` and `EXPLAIN`).
 - Not **schema-agnostic**: quality depends on **FKs**, sensible types, and optional notes for domain language.
 
 ---
@@ -145,31 +151,27 @@ A **validation-first** text-to-SQL layer for **PostgreSQL** and **Databricks**.
 ## How a question becomes SQL
 
 1. **Template match** — if a trusted pattern fits, reuse parameterized SQL (often **no** SQL LLM call).
-2. **Intent parse** — structured intent from the question + schema summary, then a long **deterministic repair chain** (see below).
+2. **Intent parse** — structured intent from the question + schema summary, then a long **deterministic repair chain**.
 3. **Join resolution** — choose among valid **FK paths** for the intent’s tables; disambiguation may use the LLM when multiple paths tie.
-4. **SQL generation & validation** — deterministic skeleton, injected joins, LLM fill/repair **under constraints**, then **semantic validation**, **`SELECT`-only / forbidden-pattern checks**, **dialect AST** validation (**`pglast`** or **`sqlglot`**), and optionally **`EXPLAIN`** when an engine is available.
+4. **SQL generation & validation** — deterministic skeleton, injected joins, LLM fill/repair **under constraints**, then **semantic validation**, **`SELECT`-only / forbidden-pattern checks**, **dialect AST** validation (**`pglast`** or **`sqlglot`**), and **`EXPLAIN`**.
 5. **Execute** (where the mode allows) and **learn** — accept → promote template trust; reject → record negative pattern.
 
 ---
 
 ## Validation (layers)
 
-- **Safety / shape** — `SELECT`-only enforcement, configurable **forbidden SQL** substrings, then **dialect `ast_validate`** (`pglast` on PostgreSQL, **`sqlglot` (Spark)** on Databricks). When a live **engine** is passed in, **`EXPLAIN`** can be used as an extra executability check.
+- **Safety / shape** — `SELECT`-only enforcement, configurable **forbidden SQL** substrings, then **dialect `ast_validate`** (**`pglast`** or **`sqlglot`**). **`EXPLAIN`** is used as an extra executability check.
 - **Schema vs intent** — tables/columns/CTEs, **selectability**, access and sensitivity policy, window / CASE / array shapes, filter and HAVING ops per column, aggregate roles in select/HAVING/ORDER BY, scalar function typing, filter value types vs column types, null/date-window/date-diff rules.
-- **Semantic consistency** — grouped queries require proper aggregation; contradictions and impossible HAVING; **grain** alignment (one of many cross-checks, not the only story).
+- **Semantic consistency** — grouped queries require proper aggregation; contradictions and impossible HAVING; **grain** alignment.
 - **Joins** — paths must match the FK graph; guarded path avoids ad-hoc join guessing.
 
-## Deterministic repairs (after intent parse)
-
-Applied in order (high level): `COUNT(*)` normalization; CTE naming and output aliases; qualify CTE outputs; sanitize table names; grain rules for grouped CTE usage and **grain consistency**; strip redundant `GROUP BY`; normalize filters/HAVING; null-equality fixes; strip join-condition leakage into filters; per-CTE sort order; simplify expressions; `IN` value normalization; date-diff classification and raw-value fixes; **`BETWEEN` → paired predicates**; auto-repair filters/HAVING; strip impossible HAVING; FK filter type repair; filter value case / enum alignment; boolean and null filter values; expand FK selects to descriptive columns; deduplicate contradictory filters; redundant PK re-qualification; **window**, **CASE**, and **array** intent repairs; sensitivity and access policy enforcement on the intent.
-
 ---
 
 ## Learning and reuse
 
 - **Accepted templates** — intent fingerprint, parameterized SQL, optional example question, **trust** that rises with validation and falls with rejection.
 - **Rejected templates** — categorized failures so similar bad intents are discouraged.
-- Persistence is under a **per-connection artifact directory** (see USAGE); you can back it up or reset it by removing that directory.
+- Persistence is under a **per-connection artifact directory** (see **[USAGE.md](USAGE.md)**); you can back it up or reset it by removing that directory.
 
 ---
 
@@ -180,7 +182,7 @@ Applied in order (high level): `COUNT(*)` normalization; CTE naming and output a
 3. Resolve joins once per table set where possible; validate and **execute** as a gate.
 4. One LLM step to produce a **natural language question** from the SQL, with a **realism** filter.
 
-The simulator loads at most **500** seed lines per run (fixed internal cap; larger files are truncated). **`estimate_simulator_costs`** uses the same loader, so its seed count reflects that cap. It returns **rough** LLM-call and execution estimates from a seed file and schema stats.
+The simulator loads at most **500** seed lines per run (fixed internal cap; larger files are truncated). **`estimate_simulator_costs`** uses the same loader, prints **rough** per-phase upper-bound lines to stdout, and returns **`None`**.
 
 ---
 

{aetherdialect-0.1.0 → aetherdialect-0.1.2}/src/aetherdialect.egg-info/SOURCES.txt

@@ -13,7 +13,6 @@ src/text2sql/contracts_core.py
 src/text2sql/core_utils.py
 src/text2sql/dialect.py
 src/text2sql/expansion_ops.py
-src/text2sql/expansion_rules.py
 src/text2sql/intent_expr.py
 src/text2sql/intent_process.py
 src/text2sql/intent_repair.py
@@ -40,13 +39,15 @@ tests/test_contracts.py
 tests/test_core_utils.py
 tests/test_dialect.py
 tests/test_expansion_ops.py
-tests/test_expansion_rules.py
 tests/test_intent_expr.py
 tests/test_intent_process.py
 tests/test_intent_repair.py
 tests/test_intent_resolve.py
+tests/test_join_bool_cte_matrix.py
 tests/test_live_testing.py
 tests/test_main_execution.py
+tests/test_pipeline_session.py
+tests/test_pipeline_targeted.py
 tests/test_pipeline_units.py
 tests/test_qsim_ops.py
 tests/test_qsim_sample.py

{aetherdialect-0.1.0 → aetherdialect-0.1.2}/src/aetherdialect.egg-info/requires.txt

@@ -1,12 +1,14 @@
 jsonschema<5,>=4.0
 openai<3,>=2.0.0
-sqlglot<30,>=29.0
 platformdirs<5,>=2.0.0
 python-dotenv<2,>=1.0.0
+SQLAlchemy<3,>=2.0
 
 [databricks]
+sqlglot<30,>=29.0
 pyspark<4,>=3.3
 databricks-sql-connector<4,>=3.0
+databricks-sqlalchemy<3,>=2.0
 
 [dev]
 pytest>=8.0
@@ -21,6 +23,5 @@ black<25,>=24
 docformatter<2,>=1.7
 
 [postgresql]
-SQLAlchemy<3,>=2.0
 psycopg2-binary<3,>=2.9
 pglast<8,>=5.0

aetherdialect-0.1.2/src/text2sql/__init__.py

@@ -0,0 +1,10 @@
+from importlib.metadata import PackageNotFoundError, version
+
+from .text2sql import Text2SQL
+
+try:
+    __version__ = version("aetherdialect")
+except PackageNotFoundError:
+    __version__ = "0.0.0+dev"
+
+__all__ = ["Text2SQL", "__version__"]