sql-dialect-master 1.1.1__tar.gz

sql_dialect_master-1.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sql_dialect_master-1.1.1/PKG-INFO

@@ -0,0 +1,360 @@
+Metadata-Version: 2.4
+Name: sql-dialect-master
+Version: 1.1.1
+Summary: SQL Dialect Master - Multi-database SQL conversion engine
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqlglot>=25.0.0
+Requires-Dist: streamlit>=1.38.0
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: uvicorn>=0.30.0
+Requires-Dist: pydantic>=2.9.0
+Requires-Dist: pydantic-settings>=2.0.0
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6.112.0; extra == "dev"
+Requires-Dist: pytest>=8.3.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
+Requires-Dist: httpx>=0.27.0; extra == "dev"
+Provides-Extra: qa
+Requires-Dist: ruff>=0.6; extra == "qa"
+Requires-Dist: bandit>=1.7; extra == "qa"
+Requires-Dist: pip-audit>=2.7; extra == "qa"
+Provides-Extra: semantic
+Requires-Dist: duckdb>=1.5.5; extra == "semantic"
+Requires-Dist: psycopg[binary]>=3.3.5; extra == "semantic"
+Provides-Extra: redis
+Requires-Dist: redis>=5.0.0; extra == "redis"
+Dynamic: license-file
+
+# 🔄 SQL Dialect Master
+
+Enterprise-grade multi-database SQL conversion engine supporting 12 database dialects.
+
+Current release: 1.1.1
+
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![FastAPI](https://img.shields.io/badge/FastAPI-0.115+-green.svg)](https://fastapi.tiangolo.com/)
+[![Streamlit](https://img.shields.io/badge/Streamlit-1.38+-red.svg)](https://streamlit.io/)
+
+## ✨ Features
+
+| Feature | Description |
+|---------|-------------|
+| 🔄 **SQL Conversion** | Convert SQL between 12 database dialects with a complete 12 × 12 conversion matrix |
+| ✅ **Output Validation** | Validate converted SQL with target-dialect parsing plus safe generic-parser fallback |
+| 🧠 **Semantic Regression Coverage** | Preserve predicates, joins, grouping, ordering, limits, windows, and round-trip validity |
+| 🔍 **AST Semantic Diff** | Compare dialect-aware SQL ASTs to detect structural changes introduced by conversion |
+| 🗄️ **Runtime Semantic Tests** | Execute representative conversions against PostgreSQL and DuckDB and compare result sets |
+| 📚 **Function Encyclopedia** | 298 SQL functions with cross-database comparison |
+| 🗂️ **Type Mapping** | 39 data types × 12 databases matrix |
+| 💬 **NL2SQL** | Natural language to SQL (Chinese/English) |
+| 🔧 **Rule Engine** | Deterministic priority ordering, compiled patterns, dialect-aware conflict detection |
+| 🛡️ **Security Validation** | Statement-level stacked-query detection with configurable blocking policy |
+| ⚡ **Performance** | Rule-aware cache versioning, bounded async batch conversion, and reproducible benchmark |
+| 🎨 **Modern UI** | Streamlit developer workspace: dark / light theme system, sidebar navigation, command palette |
+
+## 💾 Supported Databases
+
+| Category | Databases |
+|----------|-----------|
+| **RDBMS** | MySQL 🐬, PostgreSQL 🐘, Oracle 🔴, SQL Server 🟦 |
+| **Big Data** | Hive 🐝, Spark ⚡, Trino 🔷, Databricks 🧱 |
+| **Cloud DW** | Snowflake ❄️, Redshift 🔶 |
+| **OLAP** | ClickHouse 🏠, DuckDB 🦆 |
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/zengxiangdi/sql-dialect-master.git
+cd sql-dialect-master
+
+# Install the application package
+python -m pip install -e .
+```
+
+For development and testing, install the optional development dependencies:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+For database-backed semantic tests:
+
+```bash
+python -m pip install -e ".[dev,semantic]"
+```
+
+### Run Streamlit UI (Recommended)
+
+```bash
+streamlit run sdm_local_v2.py
+```
+
+Open http://localhost:8501 in your browser.
+
+### Run FastAPI Backend
+
+```bash
+uvicorn backend.api.main:app --reload
+```
+
+API docs available at http://localhost:8000/docs
+
+## 📖 Usage Examples
+
+### Python SDK
+
+```python
+from backend.core import SQLTranspiler
+
+transpiler = SQLTranspiler()
+result = transpiler.transpile(
+    sql="SELECT DATE_FORMAT(created_at, '%Y-%m-%d') FROM orders LIMIT 10",
+    source="mysql",
+    target="postgres"
+)
+print(result.target_sql)
+# Output: SELECT TO_CHAR(created_at, 'YYYY-MM-DD') FROM orders LIMIT 10
+```
+
+### REST API
+
+```bash
+curl -X POST http://localhost:8000/api/convert \
+  -H "Content-Type: application/json" \
+  -d '{
+    "sql": "SELECT * FROM users LIMIT 10",
+    "source_dialect": "mysql",
+    "target_dialect": "hive"
+  }'
+```
+
+Successful and failed conversions return structured metadata. Failed conversions expose a stable machine-readable `error_code` alongside the human-readable `error` message.
+
+### NL2SQL (Natural Language)
+
+```python
+from backend.core import NL2SQLGenerator
+
+generator = NL2SQLGenerator()
+result = generator.generate("查询最近7天的订单", dialect="mysql")
+print(result.sql)
+# Output: SELECT * FROM orders WHERE created_at >= DATE_SUB(CURRENT_DATE, 7)
+```
+
+#### Relational Query Support (D2)
+
+NL2SQL supports existential relationship queries using `EXISTS` / `NOT EXISTS` subqueries instead of physical `JOIN`s, preserving row count and avoiding duplicate expansion from 1:N relationships.
+
+**Known relationships** (explicit mapping):
+- `users ↔ orders`: `users.id = orders.user_id`
+- `customers ↔ orders`: `customers.id = orders.customer_id`
+- `orders ↔ products`: `orders.product_id = products.id`
+- `employees ↔ departments`: `employees.dept_id = departments.id`
+
+**Examples:**
+```python
+# Positive relational query → EXISTS
+result = generator.generate("find users who have orders", "postgres")
+# Output: SELECT * FROM users WHERE EXISTS (SELECT 1 FROM orders WHERE users.id = orders.user_id)
+
+# Negative relational query → NOT EXISTS
+result = generator.generate("find users without orders", "postgres")
+# Output: SELECT * FROM users WHERE NOT EXISTS (SELECT 1 FROM orders WHERE users.id = orders.user_id)
+```
+
+**Unknown relationships fail safely:**
+```python
+result = generator.generate("find users who have invoices", "postgres")
+# Output: success=False, explanation="Unable to safely infer the relationship..."
+```
+
+Relationship inference uses explicit mapping only — no fabricated foreign keys. Use `table_hint` to specify schema-qualified tables when needed.
+
+#### Date Arithmetic Semantics (D3)
+
+Date expressions follow a **rolling vs calendar-relative** distinction:
+
+| Expression | Semantic | DSL (postgres) |
+|------------|----------|----------------|
+| `today` | current calendar date | `= CURRENT_DATE` |
+| `yesterday` | rolling -1 day | `= DATE_SUB(CURRENT_DATE, 1)` |
+| `tomorrow` | rolling +1 day | `= DATE_ADD(CURRENT_DATE, 1)` |
+| `last N days` | rolling N days | `>= DATE_SUB(CURRENT_DATE, N)` |
+| `last N weeks` | rolling N×7 days | `>= DATE_SUB(CURRENT_DATE, N×7)` |
+| `last week` | rolling 7 days | `>= DATE_SUB(CURRENT_DATE, 7)` |
+| `last N months` | calendar -N months | `>= ADD_MONTHS(CURRENT_DATE, -N)` |
+| `last month` | calendar -1 month | `>= ADD_MONTHS(CURRENT_DATE, -1)` |
+| `last N years` | calendar -N years | `>= ADD_MONTHS(CURRENT_DATE, -N×12)` |
+| `last year` | calendar -1 year | `>= ADD_MONTHS(CURRENT_DATE, -12)` |
+
+**Key distinction:** `last week` uses a rolling 7-day window, while `last month`/`last year` use calendar arithmetic (`ADD_MONTHS`). This means `last month` from March 31 yields February 28/29 (not 30 days prior), and `last year` preserves month/day within the calendar.
+
+Date conditions in combined queries (e.g., "users who have orders from last 7 days") are scoped to the relation table's date column inside the `EXISTS` subquery.
+
+#### Type Mapping with Precision (D1)
+
+TypeMapper now supports parameterized types and precision-aware warnings:
+
+```python
+from backend.core import TypeMapper
+
+mapper = TypeMapper()
+result = mapper.map_type("VARCHAR2(2000)", "oracle", "postgres")
+# Output: source_type="VARCHAR2(n)", target_type="VARCHAR(n)", precision=2000
+
+result = mapper.map_type("VARCHAR2(8000)", "oracle", "postgres")
+# Output: warnings include "exceeds Oracle's default VARCHAR2 limit of 4000"
+```
+
+Supports `VARCHAR`, `VARCHAR2`, `NVARCHAR`, `CHAR` with numeric precision or `MAX`. Unknown parameterized types resolve to the canonical base type.
+
+## 🏗️ Project Structure
+
+```
+sql-dialect-master/
+├── backend/
+│   ├── api/
+│   │   ├── main.py              # FastAPI application
+│   │   └── middleware.py        # Rate limiting, logging
+│   ├── benchmarks/
+│   │   └── transpiler_benchmark.py # Reproducible conversion benchmark
+│   ├── core/
+│   │   ├── nl2sql_components/   # Modular NL2SQL components
+│   │   │   ├── tokenizer.py     # Language tokenization
+│   │   │   ├── templates.py     # Query templates
+│   │   │   └── mappings.py      # Language mappings
+│   │   ├── config.py            # Configuration management
+│   │   ├── transpiler.py        # SQL conversion engine
+│   │   ├── parser.py            # SQL parser
+│   │   ├── semantic_diff.py     # AST-based semantic difference detector
+│   │   ├── rules.py             # Transformation rules
+│   │   ├── post_processor.py    # Post-processing
+│   │   ├── nl2sql.py            # Natural language to SQL
+│   │   ├── functions_lookup.py  # Function encyclopedia
+│   │   ├── functions_db.json    # Functions database
+│   │   ├── type_mapping.py      # Type mapping
+│   │   ├── type_mapping.json    # Types database
+│   │   ├── cache.py             # TTL cache
+│   │   └── exceptions.py        # Custom exceptions + ErrorCode taxonomy
+│   ├── utils/
+│   │   └── validators.py        # Input validation
+│   │
+│   └── tests/                   # Test suite and regression coverage
+├── frontend/                    # Streamlit UI Components (v2 architecture)
+│   ├── core/                    # Cross-cutting concerns
+│   │   ├── design_tokens.py     # Theme model (dark/light) + semantic status
+│   │   ├── navigation.py        # NavigationIntent contract
+│   │   ├── state.py             # SessionState adapter
+│   │   ├── themes.py            # Theme switching (dark/light only)
+│   │   ├── styles.py            # CSS generation from design tokens
+│   │   ├── escaping.py          # HTML-safe rendering helpers
+│   │   └── viewmodels.py        # Typed view models
+│   ├── pages/                   # One module per workspace page
+│   │   ├── convert.py
+│   │   ├── nl2sql.py
+│   │   ├── diff.py
+│   │   ├── lineage.py
+│   │   ├── runtime.py
+│   │   ├── functions.py
+│   │   ├── types.py
+│   │   ├── templates.py
+│   │   ├── history.py
+│   │   ├── settings.py
+│   │   └── query_analysis.py
+│   ├── ui/                      # Reusable UI widgets
+│   │   ├── command_palette.py
+│   │   ├── diff.py
+│   │   ├── editor.py
+│   │   └── status.py
+│   ├── app_context_v2.py        # Shared application context (v2)
+│   └── templates_v2.py          # SQL templates (v2)
+├── examples/
+│   ├── api_client.py             # API usage examples
+│   └── basic_usage.py            # SDK usage examples
+├── scripts/
+│   └── verify_readme_consistency.py # README source-of-truth checks
+├── sdm_local_v2.py             # Streamlit Application Entry (v2)
+├── pyproject.toml               # Project configuration
+├── requirements.txt              # Legacy/runtime dependency list
+├── .env.example                 # Environment template
+├── LICENSE                      # MIT License
+└── README.md
+```
+
+## 🔧 API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/convert` | POST | Convert SQL between dialects; response includes `error_code` on failure |
+| `/api/parse` | POST | Parse SQL and extract structural elements |
+| `/api/nl2sql` | POST | Generate SQL from natural language |
+| `/api/functions` | GET | Search SQL functions |
+| `/api/functions/{name}` | GET | Get function details |
+| `/api/types` | GET | Get type mapping matrix |
+| `/api/types/map` | POST | Map type between dialects |
+| `/api/dialects` | GET | List supported dialects |
+| `/health` | GET | Health check |
+| `/health/deep` | GET | Deep health check |
+| `/api/stats` | GET | Service and dataset statistics |
+
+## 🧪 Quality, Validation & Performance
+
+The project maintains a 12 × 12 source-to-target dialect regression matrix, semantic regression tests, post-processing safety regressions, security boundary checks, deterministic rule-engine tests, cache-versioning tests, async concurrency tests, and stable error-code tests.
+
+Converted SQL is validated after post-processing. Target-dialect parsing is preferred; when a target parser limitation prevents validation but the generic parser accepts the SQL, the result is retained with a compatibility warning. Only when both validations fail is the conversion reported as unsuccessful.
+
+The AST semantic diff utility canonicalizes source and target SQL through SQLGlot and reports structural changes such as different predicates or AST node shapes. It is a regression detector rather than a proof of runtime equivalence.
+
+The semantic CI gate executes representative conversions against PostgreSQL and DuckDB and compares the source and converted result sets. This provides runtime evidence for supported SQL subsets without claiming universal cross-database semantic equivalence.
+
+Security validation structurally detects stacked SQL statements using the parser when available, with configurable policy controlling whether dangerous input is blocked or only warned about.
+
+The async batch API bounds concurrent work and preserves input order. A reproducible transpiler benchmark is available under `backend/benchmarks/` and is also executed in CI for performance observability. The benchmark intentionally reports measurements without failing the build on a fixed latency threshold because hosted CI timing is variable.
+
+README source-of-truth checks run in CI and verify the release version, Python requirement, supported dialect list, and benchmark references against repository sources.
+
+To run the benchmark locally:
+
+```bash
+python backend/benchmarks/transpiler_benchmark.py --iterations 100 --repeats 3
+```
+
+To validate README consistency locally:
+
+```bash
+python scripts/verify_readme_consistency.py
+```
+
+## ⚠️ Limitations
+
+SQL dialect conversion is not universally semantics-preserving. The engine validates generated SQL against the target parser, but parser acceptance does not prove equivalent runtime behavior. Database-specific differences in NULL handling, implicit casts, date/time semantics, collation, timezone behavior, JSON/array operations, and other vendor features may require manual review or execution against the target database.
+
+NL2SQL is heuristic/template-driven rather than a calibrated probabilistic model. The returned confidence score should be treated as a generation-quality signal, not as a statistical probability that the SQL is correct.
+
+## ⚙️ Configuration
+
+The project uses `pydantic-settings` for robust configuration.
+
+Copy `.env.example` to `.env` and customize:
+
+```bash
+cp .env.example .env
+```
+
+Key settings (prefixed with `SDM_`):
+- `SDM_CACHE_ENABLED` - Enable/disable caching
+
+### Rate limiting deployment
+
+Rate limiting is on by default. The two backends are:
+
+- `SDM_RATE_LIMIT_BACKEND=memory` (default) — per-process, thread-safe. Limits are **per instance**, not shared. Run this only when the API is served by a single process; in a multi-worker or multi-instance deployment each worker enforces its own limit, so the effective ceiling is `workers × SDM_RATE_LIMIT_REQUESTS`.
+- `SDM_RATE_LIMIT_BACKEND=redis` — shared, atomic limits across all workers. Set `SDM_REDIS_URL` when using this backend. The store connects and pings Redis at startup and fails fast if it is unreachable; the `redis` package is required (install with `pip install redis`).
+
+`SDM_RATE_LIMIT_ENABLED=false` disables enforcement entirely. Use the in-memory backend for local development and the Redis backend for any deployment with more than one worker or instance that must enforce a single global limit.