nl2data-postgres 0.1.0__tar.gz

nl2data_postgres-0.1.0/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.4
+Name: nl2data-postgres
+Version: 0.1.0
+Summary: PostgreSQL metadata discovery and governed SQL execution adapter for nl2data-core.
+Author: NL2Data Contributors
+License: Apache-2.0
+Keywords: nl2data,postgresql,metadata,sql,adapter
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: nl2data-core>=0.1.0
+Requires-Dist: psycopg[binary,pool]<4,>=3.1
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: ruff>=0.5; extra == "dev"
+
+# nl2data-postgres
+
+An optional PostgreSQL backend integration for
+[nl2data-core](https://github.com/emmansun/nl2data-core). It combines
+metadata discovery and governed read-only SQL execution behind the core
+provider-neutral contracts.
+
+`psycopg` is an optional dependency and is loaded lazily at first use. The
+core import boundary never loads the PostgreSQL driver.
+
+## Install
+
+```bash
+pip install nl2data-postgres
+```
+
+Requires Python 3.11+, `nl2data-core>=0.1.0`, and
+`psycopg[binary,pool]>=3.1,<4`.
+
+From a source checkout:
+
+```bash
+pip install -e packages/nl2data-postgres
+```
+
+## Public surface
+
+```python
+from nl2data_postgres import (
+    PostgresAdapterConfig,
+    PostgresMetadataDiscoverer,
+    PostgresQueryAdapter,
+)
+```
+
+- `PostgresAdapterConfig` — bounded discovery/execution configuration with
+  host-injected DSN references, object/field allowlists, and result bounds.
+- `PostgresMetadataDiscoverer` — the core `MetadataDiscoverer` port
+  implementation for PostgreSQL.
+- `PostgresQueryAdapter` — the core `QueryAdapter` port implementation
+  for read-only PostgreSQL SQL.
+
+## Configuration
+
+Configuration is credential-free; hosts inject DSNs through a resolver
+or environment at runtime. `PostgresAdapterConfig` carries only
+bounded, fingerprintable settings.
+
+```python
+from nl2data_postgres import PostgresAdapterConfig
+
+config = PostgresAdapterConfig(
+    dsn_reference="env:NL2DATA_POSTGRES_DSN",
+    allowed_objects={"users", "orders"},
+    max_objects=256,
+    max_rows=10_000,
+)
+```
+
+## Discovery
+
+```python
+from nl2data_postgres import PostgresAdapterConfig, PostgresMetadataDiscoverer
+from nl2data_core.metadata.protocol import MetadataDiscoveryConfig
+
+discoverer = PostgresMetadataDiscoverer(
+    config=PostgresAdapterConfig(dsn_reference="env:NL2DATA_POSTGRES_DSN"),
+    allowed_objects={"users", "orders"},
+)
+snapshot = await discoverer.discover(MetadataDiscoveryConfig())
+```
+
+Discovery returns a core `MetadataSnapshot` with normalized tables,
+columns, types, primary/foreign keys, and bounded protected statistics.
+
+## Query execution
+
+```python
+from nl2data_postgres import PostgresAdapterConfig, PostgresQueryAdapter
+from nl2data_core.adapters.models import ValidationContext
+
+adapter = PostgresQueryAdapter(
+    config=PostgresAdapterConfig(dsn_reference="env:NL2DATA_POSTGRES_DSN"),
+    allowed_objects={"users"},
+)
+artifact = adapter.parse("SELECT id, name FROM users LIMIT 10", ValidationContext())
+validated = adapter.validate(artifact, ValidationContext())
+result = await adapter.execute(validated, ValidationContext())
+```
+
+Execution uses a read-only connection pool, enforces statement timeout,
+limits rows/columns/bytes, and maps every result value to the protected
+scalar set.
+
+## Failure classification
+
+| Condition | Normalized result |
+| --- | --- |
+| Missing driver / unreachable service | Retryable `MetadataUnavailableError` |
+| Authorization or permission denial | `MetadataUnauthorizedError` |
+| Lifecycle misuse (unparsed/unvalidated artifacts) | `PostgresAdapterError` |
+| Guard rejection (scope/obligation/limit) | `SQLGuardError` |
+| Statement/result bounds exceeded | `PostgresExecutionError` |
+| Malformed SQL | `SQLParseError` |
+
+DSNs, raw SQL, native exceptions, and credentials never cross the
+boundary.
+
+## Compatibility
+
+The in-core `nl2data_core.adapters.sql.SqlMetadataDiscoverer` still
+accepts `dialect="postgresql"` as a temporary compatibility path, but it
+always uses the legacy in-core implementation and emits a deprecation
+warning; it does not delegate to this package. New code should import
+from `nl2data_postgres` directly.
+
+## More documentation
+
+- [Documentation index](../../docs/README.md)
+- [Adding an adapter or provider](../../docs/development/adding-adapter-or-provider.md)
+- [Secrets and live testing](../../docs/operations/secrets.md)
+- [Capabilities and support](../../docs/reference/capabilities.md)

nl2data_postgres-0.1.0/README.md

@@ -0,0 +1,122 @@
+# nl2data-postgres
+
+An optional PostgreSQL backend integration for
+[nl2data-core](https://github.com/emmansun/nl2data-core). It combines
+metadata discovery and governed read-only SQL execution behind the core
+provider-neutral contracts.
+
+`psycopg` is an optional dependency and is loaded lazily at first use. The
+core import boundary never loads the PostgreSQL driver.
+
+## Install
+
+```bash
+pip install nl2data-postgres
+```
+
+Requires Python 3.11+, `nl2data-core>=0.1.0`, and
+`psycopg[binary,pool]>=3.1,<4`.
+
+From a source checkout:
+
+```bash
+pip install -e packages/nl2data-postgres
+```
+
+## Public surface
+
+```python
+from nl2data_postgres import (
+    PostgresAdapterConfig,
+    PostgresMetadataDiscoverer,
+    PostgresQueryAdapter,
+)
+```
+
+- `PostgresAdapterConfig` — bounded discovery/execution configuration with
+  host-injected DSN references, object/field allowlists, and result bounds.
+- `PostgresMetadataDiscoverer` — the core `MetadataDiscoverer` port
+  implementation for PostgreSQL.
+- `PostgresQueryAdapter` — the core `QueryAdapter` port implementation
+  for read-only PostgreSQL SQL.
+
+## Configuration
+
+Configuration is credential-free; hosts inject DSNs through a resolver
+or environment at runtime. `PostgresAdapterConfig` carries only
+bounded, fingerprintable settings.
+
+```python
+from nl2data_postgres import PostgresAdapterConfig
+
+config = PostgresAdapterConfig(
+    dsn_reference="env:NL2DATA_POSTGRES_DSN",
+    allowed_objects={"users", "orders"},
+    max_objects=256,
+    max_rows=10_000,
+)
+```
+
+## Discovery
+
+```python
+from nl2data_postgres import PostgresAdapterConfig, PostgresMetadataDiscoverer
+from nl2data_core.metadata.protocol import MetadataDiscoveryConfig
+
+discoverer = PostgresMetadataDiscoverer(
+    config=PostgresAdapterConfig(dsn_reference="env:NL2DATA_POSTGRES_DSN"),
+    allowed_objects={"users", "orders"},
+)
+snapshot = await discoverer.discover(MetadataDiscoveryConfig())
+```
+
+Discovery returns a core `MetadataSnapshot` with normalized tables,
+columns, types, primary/foreign keys, and bounded protected statistics.
+
+## Query execution
+
+```python
+from nl2data_postgres import PostgresAdapterConfig, PostgresQueryAdapter
+from nl2data_core.adapters.models import ValidationContext
+
+adapter = PostgresQueryAdapter(
+    config=PostgresAdapterConfig(dsn_reference="env:NL2DATA_POSTGRES_DSN"),
+    allowed_objects={"users"},
+)
+artifact = adapter.parse("SELECT id, name FROM users LIMIT 10", ValidationContext())
+validated = adapter.validate(artifact, ValidationContext())
+result = await adapter.execute(validated, ValidationContext())
+```
+
+Execution uses a read-only connection pool, enforces statement timeout,
+limits rows/columns/bytes, and maps every result value to the protected
+scalar set.
+
+## Failure classification
+
+| Condition | Normalized result |
+| --- | --- |
+| Missing driver / unreachable service | Retryable `MetadataUnavailableError` |
+| Authorization or permission denial | `MetadataUnauthorizedError` |
+| Lifecycle misuse (unparsed/unvalidated artifacts) | `PostgresAdapterError` |
+| Guard rejection (scope/obligation/limit) | `SQLGuardError` |
+| Statement/result bounds exceeded | `PostgresExecutionError` |
+| Malformed SQL | `SQLParseError` |
+
+DSNs, raw SQL, native exceptions, and credentials never cross the
+boundary.
+
+## Compatibility
+
+The in-core `nl2data_core.adapters.sql.SqlMetadataDiscoverer` still
+accepts `dialect="postgresql"` as a temporary compatibility path, but it
+always uses the legacy in-core implementation and emits a deprecation
+warning; it does not delegate to this package. New code should import
+from `nl2data_postgres` directly.
+
+## More documentation
+
+- [Documentation index](../../docs/README.md)
+- [Adding an adapter or provider](../../docs/development/adding-adapter-or-provider.md)
+- [Secrets and live testing](../../docs/operations/secrets.md)
+- [Capabilities and support](../../docs/reference/capabilities.md)

nl2data_postgres-0.1.0/pyproject.toml

@@ -0,0 +1,56 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "nl2data-postgres"
+version = "0.1.0"
+description = "PostgreSQL metadata discovery and governed SQL execution adapter for nl2data-core."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "Apache-2.0" }
+authors = [{ name = "NL2Data Contributors" }]
+keywords = ["nl2data", "postgresql", "metadata", "sql", "adapter"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Libraries",
+]
+dependencies = [
+    "nl2data-core>=0.1.0",
+    "psycopg[binary,pool]>=3.1,<4",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.23",
+    "mypy>=1.10",
+    "ruff>=0.5",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.mypy]
+python_version = "3.11"
+files = ["src"]
+mypy_path = ["../../src"]
+check_untyped_defs = true
+disallow_untyped_defs = true
+no_implicit_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+warn_return_any = true
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+src = ["src"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]

nl2data_postgres-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

nl2data_postgres-0.1.0/src/nl2data_postgres/__init__.py

@@ -0,0 +1,26 @@
+"""PostgreSQL metadata discovery and governed SQL execution for nl2data-core.
+
+The package implements the core ``MetadataDiscoverer`` and ``QueryAdapter``
+ports using ``psycopg``.  The driver is never imported at package import
+time; it is loaded lazily when a discoverer or adapter is first used.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from .config import PostgresAdapterConfig
+
+__all__ = ["PostgresAdapterConfig"]
+
+
+def __getattr__(name: str) -> Any:
+    if name == "PostgresMetadataDiscoverer":
+        from .discovery import PostgresMetadataDiscoverer
+
+        return PostgresMetadataDiscoverer
+    if name == "PostgresQueryAdapter":
+        from .adapter import PostgresQueryAdapter
+
+        return PostgresQueryAdapter
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

nl2data_postgres-0.1.0/src/nl2data_postgres/adapter.py

@@ -0,0 +1,215 @@
+"""PostgreSQL ``QueryAdapter`` port implementation."""
+
+from __future__ import annotations
+
+import asyncio
+from dataclasses import replace
+from typing import Any
+
+from nl2data_core.adapters.models import (
+    AdapterCapabilities,
+    AdapterLimits,
+    AsyncMode,
+    CostEstimate,
+    ExecutionResult,
+    GeneratedArtifact,
+    ParsedArtifact,
+    ValidatedArtifact,
+    ValidationContext,
+)
+from nl2data_core.adapters.sql.guard import SQLGuardPolicy, assert_guarded
+from nl2data_core.adapters.sql.parsing import parse_sql
+from nl2data_core.metadata.protocol import MetadataDiscoveryCapability
+from sqlglot import exp
+
+from .config import PostgresAdapterConfig
+from .errors import PostgresAdapterError
+from .execution import PostgresExecutor
+
+
+class PostgresQueryAdapter:
+    """Read-only, single-statement PostgreSQL adapter over the core contract."""
+
+    def __init__(
+        self,
+        config: PostgresAdapterConfig,
+        *,
+        allowed_objects: frozenset[str] = frozenset(),
+        allowed_columns: frozenset[str] | None = None,
+        require_limit: bool = True,
+        snapshot_fingerprint: str | None = None,
+    ) -> None:
+        self._config = config
+        self._policy = SQLGuardPolicy(
+            allowed_objects=allowed_objects,
+            allowed_columns=allowed_columns,
+            max_rows=config.max_rows,
+            require_limit=require_limit,
+        )
+        self._snapshot_fingerprint = snapshot_fingerprint
+        self._executor = PostgresExecutor(config)
+        self._parsed_by_id: dict[str, Any] = {}
+        self._sql_by_id: dict[str, str] = {}
+
+    def capabilities(self) -> AdapterCapabilities:
+        return AdapterCapabilities(
+            adapter_type="sql",
+            query_language="sql",
+            async_mode=AsyncMode.THREAD_OFFLOAD,
+            features=frozenset(
+                {
+                    "read_only",
+                    "single_statement",
+                    "bounded_results",
+                    "ast_validation",
+                    "aggregation",
+                    "ordering",
+                    "list_ops",
+                    "contains",
+                    "metadata_discovery",
+                    "join",
+                    "multi_entity",
+                    "cte",
+                    "grouping",
+                    "union",
+                    "computed_expressions",
+                    "prior_period",
+                }
+            ),
+            limits=AdapterLimits(
+                max_query_length=self._config.max_query_length,
+                max_result_rows=self._policy.max_rows,
+            ),
+        )
+
+    def metadata_discovery_capability(self) -> MetadataDiscoveryCapability:
+        return MetadataDiscoveryCapability(
+            backend="sql:postgresql",
+            supported=True,
+            max_objects=1_024,
+            max_fields_per_object=16_384,
+            supports_statistics=True,
+            supports_sampling=False,
+            description="bounded postgresql catalog introspection",
+        )
+
+    def parse(self, query: str, context: ValidationContext) -> ParsedArtifact:
+        parsed = parse_sql(
+            query,
+            dialect="postgres",
+            artifact_id=f"pg-{len(self._parsed_by_id) + 1}",
+            max_query_length=self._config.max_query_length,
+        )
+        self._parsed_by_id[parsed.artifact_id] = parsed
+        return ParsedArtifact(
+            artifact_id=parsed.artifact_id,
+            fingerprint=parsed.fingerprint,
+            parse_metadata={
+                "statement_type": parsed.statement_type,
+                "tables": ",".join(parsed.tables),
+                "dialect": parsed.dialect,
+            },
+        )
+
+    def _obligation_policy(self, context: ValidationContext) -> SQLGuardPolicy:
+        if not context.required_obligation_fingerprints:
+            return self._policy
+        return replace(
+            self._policy,
+            required_obligation_fingerprints=(
+                self._policy.required_obligation_fingerprints
+                | context.required_obligation_fingerprints
+            ),
+        )
+
+    def validate(self, artifact: ParsedArtifact, context: ValidationContext) -> ValidatedArtifact:
+        parsed = self._parsed_by_id.get(artifact.artifact_id)
+        if parsed is None:
+            raise PostgresAdapterError(
+                "cannot validate an artifact that was not parsed by this adapter",
+                details={"artifact_id": artifact.artifact_id},
+            )
+        if (
+            self._snapshot_fingerprint is not None
+            and context.snapshot_fingerprint is not None
+            and context.snapshot_fingerprint != self._snapshot_fingerprint
+        ):
+            raise PostgresAdapterError(
+                "the metadata snapshot does not match the adapter's bound snapshot",
+                details={"artifact_id": artifact.artifact_id},
+            )
+        guard = assert_guarded(
+            parsed,
+            self._obligation_policy(context),
+            statement=exp.maybe_parse(parsed.sql_text, dialect="postgres"),
+            field_bindings=context.field_bindings,
+        )
+        self._sql_by_id[artifact.artifact_id] = parsed.sql_text
+        return ValidatedArtifact(
+            artifact_id=artifact.artifact_id,
+            fingerprint=guard.fingerprint,
+            snapshot_fingerprint=context.snapshot_fingerprint,
+            validation_metadata={
+                "tables": ",".join(parsed.tables),
+                "limit": str(parsed.limit_value or 0),
+                "columns": ",".join(parsed.columns),
+            },
+            obligations_verified=guard.obligations_verified,
+            bounded_rows=parsed.limit_value,
+        )
+
+    async def generate(self, query: str, context: ValidationContext) -> GeneratedArtifact:
+        parsed = parse_sql(
+            query,
+            dialect="postgres",
+            artifact_id=f"pg-{len(self._parsed_by_id) + 1}",
+            max_query_length=self._config.max_query_length,
+        )
+        guard = assert_guarded(
+            parsed,
+            self._obligation_policy(context),
+            statement=exp.maybe_parse(parsed.sql_text, dialect="postgres"),
+            field_bindings=context.field_bindings,
+        )
+        self._parsed_by_id[parsed.artifact_id] = parsed
+        self._sql_by_id[parsed.artifact_id] = parsed.sql_text
+        return GeneratedArtifact(
+            artifact_id=parsed.artifact_id,
+            fingerprint=guard.fingerprint,
+            content_type="text/sql",
+            size_bytes=len(parsed.sql_text.encode("utf-8")),
+            metadata={"dialect": parsed.dialect},
+        )
+
+    async def estimate_cost(
+        self, artifact: ValidatedArtifact, context: ValidationContext
+    ) -> CostEstimate:
+        parsed = self._parsed_by_id.get(artifact.artifact_id)
+        if parsed is None:
+            raise PostgresAdapterError(
+                "cannot estimate an artifact that was not parsed by this adapter",
+                details={"artifact_id": artifact.artifact_id},
+            )
+        return CostEstimate(estimated_units=parsed.limit_value or self._policy.max_rows)
+
+    async def execute(
+        self, artifact: ValidatedArtifact, context: ValidationContext
+    ) -> ExecutionResult:
+        sql_text = self._sql_by_id.get(artifact.artifact_id)
+        if sql_text is None:
+            raise PostgresAdapterError(
+                "cannot execute an artifact that was not validated by this adapter",
+                details={"artifact_id": artifact.artifact_id},
+            )
+        return await asyncio.to_thread(
+            self._executor.execute,
+            sql_text,
+            max_rows=context.limits.max_result_rows if context.limits else self._policy.max_rows,
+            max_result_bytes=context.max_result_bytes,
+            timeout_seconds=context.execution_timeout_seconds,
+        )
+
+    async def close(self) -> None:
+        self._parsed_by_id.clear()
+        self._sql_by_id.clear()
+        self._executor.close()

nl2data_postgres-0.1.0/src/nl2data_postgres/client.py

@@ -0,0 +1,121 @@
+"""Lazy psycopg client loading and bounded read-only connection helpers."""
+
+from __future__ import annotations
+
+import contextlib
+import threading
+from collections.abc import Iterator
+from importlib import import_module
+from importlib.util import find_spec
+from typing import Any
+
+from nl2data.errors import NL2DataError
+from nl2data_core.metadata.protocol import (
+    MetadataDiscoveryError,
+    MetadataUnauthorizedError,
+    MetadataUnavailableError,
+)
+
+from .config import PostgresAdapterConfig
+
+_AUTH_SQLSTATES = frozenset({"28P01", "42501", "42502"})
+
+
+def _pool_module() -> Any:
+    """Return the psycopg_pool module, raising a safe error if absent."""
+    if find_spec("psycopg_pool") is None:
+        raise MetadataUnavailableError(
+            "the psycopg pool driver is not installed; install the 'postgres' extra",
+            details={"cause_type": "ImportError"},
+        )
+    return import_module("psycopg_pool")
+
+
+def _raise_normalized_discovery_error(error: BaseException, message: str) -> None:
+    """Normalize a psycopg failure into the safe discovery error family."""
+    cause_type = error.__class__.__name__
+    sqlstate = getattr(error, "sqlstate", None)
+    if sqlstate in _AUTH_SQLSTATES or cause_type in {
+        "InsufficientPrivilege",
+        "InvalidPassword",
+        "PasswordMismatch",
+    }:
+        raise MetadataUnauthorizedError(message, details={"cause_type": cause_type}) from error
+    if cause_type in {
+        "OperationalError",
+        "InterfaceError",
+        "ConnectionError",
+        "PoolTimeout",
+    }:
+        raise MetadataUnavailableError(message, details={"cause_type": cause_type}) from error
+    raise MetadataDiscoveryError(message, details={"cause_type": cause_type}) from error
+
+
+class PostgresPool:
+    """A lazily-created bounded psycopg connection pool.
+
+    The pool is created only on first use and is closed idempotently.
+    DSN resolution and driver loading happen lazily, so importing this
+    module does not require ``psycopg``.
+    """
+
+    def __init__(self, config: PostgresAdapterConfig) -> None:
+        self._config = config
+        self._pool: Any | None = None
+        self._lock = threading.Lock()
+
+    def _connect_timeout(self) -> float:
+        return min(self._config.timeout_seconds, 5.0)
+
+    def _ensure_pool(self) -> Any:
+        if self._pool is not None:
+            return self._pool
+        with self._lock:
+            if self._pool is not None:
+                return self._pool
+            pool_module = _pool_module()
+            try:
+                dsn = self._config.resolve_dsn()
+            except ValueError as error:
+                raise MetadataUnavailableError(
+                    "postgresql dsn reference could not be resolved",
+                    details={"cause_type": type(error).__name__},
+                ) from error
+            try:
+                self._pool = pool_module.ConnectionPool(
+                    dsn,
+                    min_size=self._config.pool_min_size,
+                    max_size=self._config.pool_max_size,
+                    open=True,
+                    kwargs={"connect_timeout": self._connect_timeout()},
+                )
+                self._pool.wait(timeout=self._connect_timeout())
+            except Exception as error:
+                self._pool = None
+                _raise_normalized_discovery_error(error, "could not connect to postgresql")
+            return self._pool
+
+    @contextlib.contextmanager
+    def connection(self) -> Iterator[Any]:
+        """Yield a read-only or standard connection from the pool."""
+        pool = self._ensure_pool()
+        connection: Any | None = None
+        try:
+            with pool.connection() as connection:
+                if self._config.read_only:
+                    connection.execute("SET TRANSACTION READ ONLY")
+                yield connection
+        except NL2DataError:
+            raise
+        except Exception as error:
+            if connection is None:
+                _raise_normalized_discovery_error(error, "could not obtain a postgresql connection")
+            _raise_normalized_discovery_error(error, "postgresql operation failed")
+
+    def close(self) -> None:
+        """Release the pool idempotently."""
+        with self._lock:
+            if self._pool is not None:
+                with contextlib.suppress(Exception):
+                    self._pool.close()
+                self._pool = None