datus-dws 0.1.0__tar.gz

datus_dws-0.1.0/.gitignore

@@ -0,0 +1,143 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+Pipfile.lock
+
+# uv
+uv.lock
+
+# PEP 582
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# IDEs
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+
+.omc
+
+uv.toml
+*/uv.toml

datus_dws-0.1.0/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.5
+Name: datus-dws
+Version: 0.1.0
+Summary: Huawei Cloud GaussDB(DWS) database adapter for Datus
+Project-URL: Homepage, https://github.com/Datus-ai/datus-db-adapters
+Project-URL: Repository, https://github.com/Datus-ai/datus-db-adapters
+Project-URL: Issues, https://github.com/Datus-ai/datus-db-adapters/issues
+Author-email: DatusAI <support@datus.ai>
+License: Apache-2.0
+Keywords: adapter,database,datus,dws,gaussdb,huawei,postgresql
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.12
+Requires-Dist: datus-db-core>=0.1.6
+Requires-Dist: datus-postgresql>=0.1.8
+Requires-Dist: pydantic>=2.0.0
+Description-Content-Type: text/markdown
+
+# datus-dws
+
+Huawei Cloud GaussDB(DWS) adapter for [Datus](https://github.com/Datus-ai/datus-agent).
+
+DWS is a shared-nothing MPP analytical warehouse that speaks the PostgreSQL wire
+protocol. This adapter builds on `datus-postgresql`: it reuses the psycopg2
+transport unchanged and overrides only what DWS does differently — its system
+schema set, its native table-definition function, and the Oracle-compatibility
+semantics of ORA mode.
+
+## Installation
+
+```bash
+pip install datus-dws
+```
+
+## Configuration
+
+```yaml
+datasources:
+  dws_analytics:
+    type: dws
+    host: example.dws.myhuaweicloud.com
+    port: 8000
+    database: gaussdb
+    schema: public
+    username: dbadmin
+    password: ${DWS_PASSWORD}
+    sslmode: verify-ca
+    sslrootcert: /path/to/cacert.pem
+    timeout_seconds: 30
+```
+
+`host` also accepts the console's `host:port` form, in which case `port` may be
+omitted. The cluster's default database is normally `gaussdb`.
+
+## TLS
+
+| `sslmode` | Behaviour |
+|---|---|
+| `prefer` (default) | Encrypts when the server offers it; upgrades automatically if the cluster enforces SSL |
+| `require` | Encrypts without verifying the server certificate |
+| `verify-ca` | Verifies the server certificate against `sslrootcert` |
+| `verify-full` | **Cannot succeed** — see below |
+| `disable` | Fails if the cluster has SSL enforcement switched on |
+
+Two things to know about DWS certificates:
+
+- **`verify-full` is not usable.** The default server certificate has `CN=server`
+  and carries no `subjectAltName`, so hostname verification can never match a
+  real endpoint. This is a property of the certificate, not a misconfiguration.
+- **Use the v2 CA.** The console's `dws_ssl_cert` bundle contains both
+  `v1/sslcert/cacert.pem` and `v2/sslcert/cacert.pem`. Only v2 matches the server
+  certificate issuer; v1 is `CN=Huawei Equipment CA` and fails verification.
+
+`sslrootcert` accepts either a filesystem path or inline PEM content, so a hosted
+deployment can pass an uploaded certificate directly.
+
+## Compatibility modes
+
+DWS databases run in `ORA`, `TD` or `MySQL` compatibility mode, reported by
+`pg_database.datcompatibility` and surfaced through
+`DWSConnector.get_compatibility_mode()`.
+
+ORA mode — the default for new clusters — changes expression semantics in ways
+that can silently produce wrong results:
+
+- `7/2` is `3.5` (double precision), not integer `3`.
+- `''` is stored as NULL and `'' IS NULL` is true; `col = ''` never matches.
+- `'a' || NULL` is `'a'`; concatenation absorbs NULL.
+- `DATE` is `timestamp(0)` and `DATE - DATE` yields an `interval`.
+
+TD and MySQL modes are not verified by this adapter's test suite.
+
+## Table DDL
+
+Table DDL comes from DWS's `pg_get_tabledef()`, which preserves `orientation`,
+`compression`, `DISTRIBUTE BY`, partitioning, `TABLESPACE` and `TO GROUP`.
+
+`TABLESPACE` and `TO GROUP` name objects of the source cluster and will not
+replay elsewhere. Use `DWSConnector.strip_cluster_specific_clauses()` to remove
+them before applying the DDL to a migration target.
+
+## Testing
+
+```bash
+# Unit tests (no cluster required)
+python -m pytest tests/unit/ -v
+
+# Integration tests against a live cluster
+export DWS_HOST=... DWS_PORT=8000 DWS_DATABASE=gaussdb \
+       DWS_USERNAME=dbadmin DWS_PASSWORD=... DWS_SSLROOTCERT=/path/to/cacert.pem
+python -m pytest tests/integration/ -v
+```
+
+Integration tests skip automatically when `DWS_HOST` is unset. They create
+objects in a run-scoped schema and drop it on teardown.
+
+## Verified against
+
+DWS 9.1.0.227, storage-decoupled (compute-group mode), 3 CN + 3 DN, ORA
+compatibility mode.

datus_dws-0.1.0/README.md

@@ -0,0 +1,102 @@
+# datus-dws
+
+Huawei Cloud GaussDB(DWS) adapter for [Datus](https://github.com/Datus-ai/datus-agent).
+
+DWS is a shared-nothing MPP analytical warehouse that speaks the PostgreSQL wire
+protocol. This adapter builds on `datus-postgresql`: it reuses the psycopg2
+transport unchanged and overrides only what DWS does differently — its system
+schema set, its native table-definition function, and the Oracle-compatibility
+semantics of ORA mode.
+
+## Installation
+
+```bash
+pip install datus-dws
+```
+
+## Configuration
+
+```yaml
+datasources:
+  dws_analytics:
+    type: dws
+    host: example.dws.myhuaweicloud.com
+    port: 8000
+    database: gaussdb
+    schema: public
+    username: dbadmin
+    password: ${DWS_PASSWORD}
+    sslmode: verify-ca
+    sslrootcert: /path/to/cacert.pem
+    timeout_seconds: 30
+```
+
+`host` also accepts the console's `host:port` form, in which case `port` may be
+omitted. The cluster's default database is normally `gaussdb`.
+
+## TLS
+
+| `sslmode` | Behaviour |
+|---|---|
+| `prefer` (default) | Encrypts when the server offers it; upgrades automatically if the cluster enforces SSL |
+| `require` | Encrypts without verifying the server certificate |
+| `verify-ca` | Verifies the server certificate against `sslrootcert` |
+| `verify-full` | **Cannot succeed** — see below |
+| `disable` | Fails if the cluster has SSL enforcement switched on |
+
+Two things to know about DWS certificates:
+
+- **`verify-full` is not usable.** The default server certificate has `CN=server`
+  and carries no `subjectAltName`, so hostname verification can never match a
+  real endpoint. This is a property of the certificate, not a misconfiguration.
+- **Use the v2 CA.** The console's `dws_ssl_cert` bundle contains both
+  `v1/sslcert/cacert.pem` and `v2/sslcert/cacert.pem`. Only v2 matches the server
+  certificate issuer; v1 is `CN=Huawei Equipment CA` and fails verification.
+
+`sslrootcert` accepts either a filesystem path or inline PEM content, so a hosted
+deployment can pass an uploaded certificate directly.
+
+## Compatibility modes
+
+DWS databases run in `ORA`, `TD` or `MySQL` compatibility mode, reported by
+`pg_database.datcompatibility` and surfaced through
+`DWSConnector.get_compatibility_mode()`.
+
+ORA mode — the default for new clusters — changes expression semantics in ways
+that can silently produce wrong results:
+
+- `7/2` is `3.5` (double precision), not integer `3`.
+- `''` is stored as NULL and `'' IS NULL` is true; `col = ''` never matches.
+- `'a' || NULL` is `'a'`; concatenation absorbs NULL.
+- `DATE` is `timestamp(0)` and `DATE - DATE` yields an `interval`.
+
+TD and MySQL modes are not verified by this adapter's test suite.
+
+## Table DDL
+
+Table DDL comes from DWS's `pg_get_tabledef()`, which preserves `orientation`,
+`compression`, `DISTRIBUTE BY`, partitioning, `TABLESPACE` and `TO GROUP`.
+
+`TABLESPACE` and `TO GROUP` name objects of the source cluster and will not
+replay elsewhere. Use `DWSConnector.strip_cluster_specific_clauses()` to remove
+them before applying the DDL to a migration target.
+
+## Testing
+
+```bash
+# Unit tests (no cluster required)
+python -m pytest tests/unit/ -v
+
+# Integration tests against a live cluster
+export DWS_HOST=... DWS_PORT=8000 DWS_DATABASE=gaussdb \
+       DWS_USERNAME=dbadmin DWS_PASSWORD=... DWS_SSLROOTCERT=/path/to/cacert.pem
+python -m pytest tests/integration/ -v
+```
+
+Integration tests skip automatically when `DWS_HOST` is unset. They create
+objects in a run-scoped schema and drop it on teardown.
+
+## Verified against
+
+DWS 9.1.0.227, storage-decoupled (compute-group mode), 3 CN + 3 DN, ORA
+compatibility mode.

datus_dws-0.1.0/datus_dws/__init__.py

@@ -0,0 +1,33 @@
+# Copyright 2025-present DatusAI, Inc.
+# Licensed under the Apache License, Version 2.0.
+# See http://www.apache.org/licenses/LICENSE-2.0 for details.
+
+from .config import DWSConfig
+from .connector import DWSConnector
+from .handlers import (
+    build_dws_uri,
+    parse_dws_identifier,
+    resolve_dws_context,
+)
+from .skills import get_dws_sql_generation_notes
+
+__version__ = "0.1.0"
+__all__ = ["DWSConnector", "DWSConfig", "register"]
+
+
+def register():
+    """Register DWS and its generic Agent integration hooks."""
+    from datus_db_core import connector_registry
+
+    connector_registry.register(
+        "dws",
+        DWSConnector,
+        config_class=DWSConfig,
+        display_name="Huawei DWS",
+        capabilities={"database", "schema"},
+        uri_builder=build_dws_uri,
+        context_resolver=resolve_dws_context,
+        parser_dialect="postgres",
+        identifier_parser=parse_dws_identifier,
+        sql_generation_notes=get_dws_sql_generation_notes,
+    )

datus_dws-0.1.0/datus_dws/_ca_cert.py

@@ -0,0 +1,76 @@
+# Copyright 2025-present DatusAI, Inc.
+# Licensed under the Apache License, Version 2.0.
+# See http://www.apache.org/licenses/LICENSE-2.0 for details.
+
+"""Accept a CA certificate as inline PEM, not only as a path on disk.
+
+A hosted caller uploads the certificate through a browser, so what reaches the
+adapter is the file's *content*: there is no operator to drop a file on the
+server first, and on a multi-tenant runtime there is nowhere stable to drop it.
+
+psycopg2 takes a filename rather than bytes, so inline PEM is spilled to a
+private temp file, once per distinct certificate and reused thereafter.
+"""
+
+import atexit
+import hashlib
+import os
+import tempfile
+import threading
+from typing import Dict, Optional
+
+from datus_db_core import get_logger
+
+logger = get_logger(__name__)
+
+_PEM_MARKER = "-----BEGIN CERTIFICATE-----"
+
+_lock = threading.Lock()
+_materialized: Dict[str, str] = {}
+
+
+def is_inline_pem(value: Optional[str]) -> bool:
+    """Whether *value* is certificate content rather than a path to one."""
+    return bool(value) and _PEM_MARKER in value
+
+
+def as_path(value: Optional[str]) -> Optional[str]:
+    """Return a filesystem path for *value*, writing it out if it is inline PEM.
+
+    Paths pass through untouched, so a self-hosted deployment that mounts its
+    CA file keeps working exactly as before.
+    """
+    if not is_inline_pem(value):
+        return value
+
+    digest = hashlib.sha256(value.encode("utf-8")).hexdigest()
+    with _lock:
+        path = _materialized.get(digest)
+        if path and os.path.exists(path):
+            return path
+
+        fd, path = tempfile.mkstemp(prefix=f"datus-dws-ca-{digest[:12]}-", suffix=".pem")
+        try:
+            # 0600 before the bytes land: a world-readable trust anchor is an
+            # invitation to swap it.
+            os.fchmod(fd, 0o600)
+            with os.fdopen(fd, "w", encoding="utf-8") as handle:
+                handle.write(value)
+        except BaseException:
+            os.unlink(path)
+            raise
+
+        _materialized[digest] = path
+        logger.debug("Materialized inline DWS CA certificate to %s", path)
+        return path
+
+
+@atexit.register
+def _cleanup() -> None:
+    with _lock:
+        for path in _materialized.values():
+            try:
+                os.unlink(path)
+            except OSError:
+                pass
+        _materialized.clear()

datus_dws-0.1.0/datus_dws/config.py

@@ -0,0 +1,104 @@
+# Copyright 2025-present DatusAI, Inc.
+# Licensed under the Apache License, Version 2.0.
+# See http://www.apache.org/licenses/LICENSE-2.0 for details.
+
+from typing import Any, Literal, Optional
+from urllib.parse import urlsplit
+
+from pydantic import Field, model_validator
+
+from datus_postgresql import PostgreSQLConfig
+
+DEFAULT_DWS_PORT = 8000
+
+
+def normalize_dws_endpoint(host: str, port: Any = None) -> tuple[str, int]:
+    """Normalize a console endpoint into separate host and port values.
+
+    The DWS console presents the public endpoint as ``host:port``, so accept
+    that form directly rather than making the user split it by hand.
+    """
+    endpoint = str(host or "").strip()
+    if not endpoint:
+        raise ValueError("DWS host is required")
+    if "://" in endpoint:
+        raise ValueError("DWS host must not include a URI scheme")
+
+    # A bare IPv6 literal has to be bracketed before urlsplit will read it as a
+    # host rather than a host:port pair. Config keeps host and port in separate
+    # fields, so users write the address unbracketed.
+    if endpoint.count(":") > 1 and not endpoint.startswith("["):
+        endpoint = f"[{endpoint}]"
+
+    parsed = urlsplit(f"//{endpoint}")
+    if parsed.username is not None or parsed.password is not None:
+        raise ValueError("DWS host must not contain user information")
+    if parsed.path or parsed.query or parsed.fragment:
+        raise ValueError("DWS host must contain only a hostname and optional port")
+    if not parsed.hostname:
+        raise ValueError("DWS host is invalid")
+
+    try:
+        embedded_port = parsed.port
+    except ValueError as exc:
+        raise ValueError(f"Invalid DWS endpoint: {endpoint}") from exc
+
+    explicit_port = None
+    if port is not None and str(port).strip():
+        try:
+            explicit_port = int(port)
+        except (TypeError, ValueError) as exc:
+            raise ValueError(f"Invalid DWS port: {port}") from exc
+
+    if embedded_port is not None and explicit_port is not None and embedded_port != explicit_port:
+        raise ValueError(f"DWS endpoint port {embedded_port} conflicts with explicit port {explicit_port}")
+
+    effective_port = embedded_port if embedded_port is not None else explicit_port
+    effective_port = DEFAULT_DWS_PORT if effective_port is None else effective_port
+    if not 1 <= effective_port <= 65535:
+        raise ValueError(f"DWS port must be between 1 and 65535: {effective_port}")
+    return parsed.hostname, effective_port
+
+
+class DWSConfig(PostgreSQLConfig):
+    """Connection configuration for Huawei Cloud GaussDB(DWS)."""
+
+    @model_validator(mode="before")
+    @classmethod
+    def normalize_endpoint(cls, values):
+        if not isinstance(values, dict) or "host" not in values:
+            return values
+        normalized = dict(values)
+        host, port = normalize_dws_endpoint(normalized.get("host"), normalized.get("port"))
+        normalized["host"] = host
+        normalized["port"] = port
+        return normalized
+
+    host: str = Field(..., min_length=1, description="DWS coordinator endpoint, optionally with a port")
+    port: int = Field(default=DEFAULT_DWS_PORT, ge=1, le=65535, description="DWS coordinator port")
+    username: str = Field(..., min_length=1, description="DWS database user")
+    password: str = Field(
+        default="",
+        repr=False,
+        description="DWS password",
+        json_schema_extra={"input_type": "password"},
+    )
+    database: str = Field(..., min_length=1, description="DWS database name (the cluster default is 'gaussdb')")
+    schema_name: str = Field(default="public", alias="schema", min_length=1, description="Default schema name")
+    sslmode: Literal["disable", "allow", "prefer", "require", "verify-ca", "verify-full"] = Field(
+        default="prefer",
+        description=(
+            "PostgreSQL SSL mode. 'prefer' upgrades automatically when the cluster enforces SSL. "
+            "'verify-full' cannot succeed against the DWS default server certificate, whose CN is "
+            "'server' and which carries no subjectAltName"
+        ),
+    )
+    sslrootcert: Optional[str] = Field(
+        default=None,
+        description=(
+            "CA certificate for sslmode=verify-ca, as a file path or inline PEM content. "
+            "Use v2/sslcert/cacert.pem from the console's dws_ssl_cert bundle; the v1 CA does "
+            "not match the server certificate issuer"
+        ),
+    )
+    timeout_seconds: int = Field(default=30, gt=0, description="Connection and pool timeout in seconds")