dbt-for-apache-doris 1.1.0__tar.gz

dbt_for_apache_doris-1.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

dbt_for_apache_doris-1.1.0/MANIFEST.in

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# encoding: utf-8
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+include LICENSE
+include NOTICE
+include README.md
+include UPSTREAM.md
+include pyproject.toml
+include pytest.ini
+include tox.ini
+include dev-requirements.txt
+prune docs
+recursive-include dbt/include *.sql *.yml
+recursive-include scripts *.py
+recursive-include test *.py
+include test/doris_test.env

dbt_for_apache_doris-1.1.0/NOTICE

@@ -0,0 +1,14 @@
+dbt-for-apache-doris
+
+This product includes software developed at
+The Apache Software Foundation (https://www.apache.org/).
+
+Portions of this product are derived from the dbt-doris component of
+Apache Doris:
+https://github.com/apache/doris/tree/master/extension/dbt-doris
+
+Apache Doris
+Copyright 2018-2026 The Apache Software Foundation
+
+This distribution is maintained by the VeloDB community and is not an official
+Apache Software Foundation release.

dbt_for_apache_doris-1.1.0/PKG-INFO

@@ -0,0 +1,285 @@
+Metadata-Version: 2.4
+Name: dbt-for-apache-doris
+Version: 1.1.0
+Summary: A dbt adapter for VeloDB and Apache Doris
+Home-page: https://github.com/velodb/dbt-for-apache-doris
+Author: Apache Doris contributors
+Maintainer: VeloDB contributors
+License: Apache-2.0
+Project-URL: Documentation, https://github.com/velodb/dbt-for-apache-doris#readme
+Project-URL: Issues, https://github.com/velodb/dbt-for-apache-doris/issues
+Project-URL: Source, https://github.com/velodb/dbt-for-apache-doris
+Project-URL: Upstream, https://github.com/apache/doris/tree/master/extension/dbt-doris
+Classifier: Development Status :: 4 - Beta
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: dbt-core~=1.12.0
+Requires-Dist: mysql-connector-python>=8.0.33
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: maintainer
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# dbt for Apache Doris
+
+[![CI](https://github.com/velodb/dbt-for-apache-doris/actions/workflows/ci.yml/badge.svg)](https://github.com/velodb/dbt-for-apache-doris/actions/workflows/ci.yml)
+![Python](https://img.shields.io/badge/Python-%3E%3D3.10-blue)
+![dbt Core](https://img.shields.io/badge/dbt--core-1.12.x-orange)
+[![License](https://img.shields.io/badge/License-Apache--2.0-blue)](https://github.com/velodb/dbt-for-apache-doris/blob/main/LICENSE)
+
+`dbt-for-apache-doris` enables Python dbt Core projects to transform data in
+Apache Doris through the Doris MySQL protocol. It is maintained by the VeloDB
+community.
+
+> [!IMPORTANT]
+> This adapter is **Beta**. CI covers lint, Unit tests, and package validation,
+> but not a live Doris cluster. Current-release compatibility and VeloDB
+> verification are pending.
+
+**[Installation](#installation)** · **[Quickstart](#quickstart)** ·
+**[Compatibility](#compatibility)** ·
+**[dbt docs](https://docs.getdbt.com/)** ·
+**[Doris docs](https://doris.apache.org/docs/)** ·
+**[Issues](https://github.com/velodb/dbt-for-apache-doris/issues)** ·
+**[Releases](https://github.com/velodb/dbt-for-apache-doris/releases)**
+
+## Supported capabilities
+
+Status: ✅ Supported · ❌ Not supported
+
+Feature status and database-version compatibility are separate contracts.
+`Supported` means the documented scope is implemented and tested; explicit
+platform boundaries are described alongside each capability.
+
+### Materializations
+
+| Capability | Status | Current support and boundaries |
+| --- | --- | --- |
+| Table | ✅ Supported | Duplicate Key CTAS; configurable HASH distribution, integer buckets, RANGE/LIST partitions, properties, contracts, docs, grants, and hooks. Unique Key creation belongs to incremental `merge` |
+| View | ✅ Supported | Standard lifecycle, contracts, docs, grants, and hooks; relation-type switching is not zero-downtime |
+| Incremental | ✅ Supported | Four strategies and every `on_schema_change` mode; boundaries are listed below |
+| Snapshot | ✅ Supported | `check`/`timestamp`, hard-delete modes, schema evolution, atomic replacement, and recovery; same-target runs must be serialized by the scheduler |
+| Materialized view | ✅ Supported | Standard dbt `materialized_view`, implemented with Doris Async MV; build/refresh lifecycle, task waiting, configuration changes, atomic replacement, and recovery. Same-target dbt runs must be serialized by the scheduler |
+| Seed | ✅ Supported | CSV loading, type inference, `column_types`, and `ref` |
+| Ephemeral | ✅ Supported | Compiled and inlined by dbt Core |
+
+### dbt capabilities
+
+| Capability | Status | Current support and boundaries |
+| --- | --- | --- |
+| Sources and freshness | ✅ Supported | `loaded_at_field`, filter, and `loaded_at_query`; cross-database uses database-as-schema, not External Catalog |
+| Data tests | ✅ Supported | Singular, generic, ephemeral, and `store_failures` paths |
+| dbt Unit tests | ✅ Supported | Inline-row and CSV fixtures, case-insensitive columns, invalid-input validation, quoted reserved words, Doris-adapted data-type fixtures, and non-truncating VARCHAR fixtures |
+| Model contracts | ✅ Supported | Column names/types for Table, View, and Incremental; not database PK/NOT NULL constraints |
+| Persisted docs | ✅ Supported | Relation and column comments for Table, View, Incremental, Snapshot, Seed, and Async MV; updating View comments or comment text containing both quote delimiters may require recreation/full refresh |
+| Grants | ✅ Supported | Reconciles supported Doris table privileges for `user` and `user@host` principals on Table, View, Incremental, Seed, Snapshot, and Async MV; role principals are not reconciled |
+| Hooks | ✅ Supported | Pre-hooks and post-hooks across adapter materializations; Doris does not provide transactional rollback for hook side effects |
+| Internal metadata and dbt docs catalog | ✅ Supported | Relation discovery and docs catalog for Doris databases, tables, views, columns, comments, and Async MVs |
+| Cross-database sources | ✅ Supported | Sources in other Doris databases, including database-only source definitions; External Catalog three-part names are not supported |
+| Advanced metadata / External Catalog | ❌ Not supported | Catalogs V2, metadata-by-relation, single-relation catalog, last-modified metadata, and External Catalog three-part namespaces are not declared |
+
+## Compatibility
+
+| Component | Declared or runtime constraint | Current evidence or status |
+| --- | --- | --- |
+| Python | `>=3.10` | Unit CI covers 3.10 and 3.14; the distribution-build job uses 3.12 |
+| dbt Core | `>=1.12,<1.13` | Declared lower bound is 1.12.0; Python dbt Core v1 only. Fusion/v2 compatibility is not claimed |
+| MySQL connector | `>=8.0.33` | Installed automatically with the adapter |
+| Apache Doris | No package-wide minimum has been declared | Historical exact-release evidence exists, but the current release-candidate SHA still needs a complete live matrix |
+| Async MV | Doris 2.x >=2.1.5; Doris 3.x except 3.0.0; Doris 4.x+ | This runtime gate is not a whole-adapter compatibility guarantee. Identifiable source builds are accepted for development testing only |
+| VeloDB | No release range has been declared | Release-specific live-cluster verification is pending |
+
+Before production use, run Functional tests against your exact release and
+topology. Historical results are not a current-release compatibility promise.
+
+## Installation
+
+Install the VeloDB-maintained distribution from PyPI:
+
+```shell
+python -m venv .venv
+source .venv/bin/activate
+python -m pip install "dbt-for-apache-doris==1.1.0"
+dbt --version
+```
+
+> [!WARNING]
+> The separate [`dbt-doris==1.0.0` distribution on
+> PyPI](https://pypi.org/project/dbt-doris/1.0.0/) does not contain the code
+> from this repository. Install `dbt-for-apache-doris` as shown above.
+
+On Windows, create the environment with `py -m venv .venv`, activate it using
+`.venv\Scripts\Activate.ps1`, and run the same `pip install` command.
+
+The adapter declares dbt Core and the MySQL connector as dependencies, so they
+are installed automatically. You do not need to install dbt Core separately or
+download a standalone binary.
+
+## Quickstart
+
+Add a Doris output to `~/.dbt/profiles.yml`. Keep credentials outside version
+control; this example reads the password from an environment variable:
+
+```yaml
+doris_demo:
+  target: dev
+  outputs:
+    dev:
+      type: doris
+      host: 127.0.0.1
+      port: 9030
+      username: root
+      password: "{{ env_var('DORIS_PASSWORD') }}"
+      schema: analytics
+      threads: 4
+```
+
+On Doris, `schema` is a database; an optional `database` must match it.
+
+Create a new `doris-demo` directory with a `models` subdirectory, then add:
+
+```yaml
+# dbt_project.yml
+name: doris_demo
+version: 1.0.0
+config-version: 2
+profile: doris_demo
+model-paths: ["models"]
+```
+
+```sql
+-- models/example.sql
+{{ config(materialized='table', replication_num=1) }}
+select 1 as id, 'hello from dbt-for-apache-doris' as message
+```
+
+```yaml
+# models/schema.yml
+version: 2
+models:
+  - name: example
+    columns:
+      - name: id
+        data_tests: [not_null, unique]
+```
+
+`replication_num=1` is only for a local single-BE Quickstart.
+
+```shell
+export DORIS_PASSWORD='<your-password>'
+dbt debug
+dbt build
+```
+
+On Windows PowerShell, set the password with
+`$env:DORIS_PASSWORD = '<your-password>'`, then run the same dbt commands.
+
+## Doris-specific highlights
+
+### Incremental strategies
+
+| Strategy | Doris target | Behavior and boundaries |
+| --- | --- | --- |
+| `append` | Duplicate Key table | Appends rows with `INSERT INTO` |
+| `merge` | MOW or MOR Unique Key table | Full-row `INSERT INTO` upsert using Doris Unique Key semantics; requires `unique_key` and does not emit SQL `MERGE INTO` |
+| `insert_overwrite` | Writable Doris table | Whole-table, named-partition, or dynamic-partition `INSERT OVERWRITE`; `unique_key` is rejected |
+| `microbatch` | Duplicate Key table with exact RANGE partitions | One named-partition overwrite per dbt Core UTC window; hour/day/month/year windows; static or dynamic partitions; batches run serially |
+
+Without an explicit strategy, `unique_key` selects `merge`; otherwise dbt uses
+`append`. `delete+insert`, partial-column merge, and `incremental_predicates`
+are rejected.
+
+### Materialized views
+
+Use dbt's standard `materialized_view` materialization. The adapter implements
+it with Doris Async MV and exposes Doris-specific refresh configuration:
+
+```sql
+{{ config(
+    materialized='materialized_view',
+    refresh_trigger='manual',
+    wait_for_refresh=true
+) }}
+
+select order_date, sum(amount) as sales
+from {{ ref('orders') }}
+group by order_date
+```
+
+Supported lifecycles include immediate/deferred build, manual/schedule/commit
+refresh, task waiting, configuration changes, docs, grants, and recovery.
+Overlapping dbt runs against the same MV target must be serialized, and a dbt
+wait timeout does not cancel a submitted Doris task.
+
+## Known limitations
+
+- Microbatch execution is serial.
+- Do not run multiple dbt invocations against the same Snapshot target concurrently;
+  serialize overlapping production jobs in the scheduler.
+- Serialize overlapping dbt runs against the same MV target; a dbt wait timeout
+  does not cancel the submitted Doris task.
+- Ordinary `table` models create Duplicate Key tables. Aggregate Key modeling,
+  secondary indexes, and a complete Doris table abstraction are not implemented.
+- `delete+insert`, partial-column merge, and `incremental_predicates` are not
+  supported.
+- A complete External Catalog namespace is unsupported.
+- SSL configuration, timeout/retry, multi-FE failover, server-side cancellation,
+  and complete query telemetry are not implemented.
+- Some Table/View/MV type changes have a short canonical-name availability
+  window rather than a zero-downtime switch.
+
+## Development and testing
+
+Install development dependencies, then run local checks:
+
+```shell
+python -m pip install -r dev-requirements.txt
+python -m pip install -e .
+make lint
+make test-unit
+```
+
+Functional tests need a dedicated non-production cluster. Edit
+`test/doris_test.env`; use an external file for private credentials:
+
+```shell
+make test
+make test DORIS_TEST_CONFIG=/secure/path/doris_test.env
+```
+
+Preflight records live FE/BE versions, checks replication against live BEs, and
+requires `cross_db_test` to be absent. Tests create/drop databases, relations,
+users, and grants, so the account needs those permissions. Never use a shared
+or production cluster or run Functional sessions concurrently.
+
+```shell
+python scripts/run_doris_functional_tests.py --preflight-only
+python scripts/run_doris_functional_tests.py -- -k snapshot -vv
+```
+
+The runner records evidence about the connected cluster but does not certify a
+release compatibility matrix.
+
+## License
+
+The code is licensed under Apache License 2.0. See the
+[license](https://github.com/velodb/dbt-for-apache-doris/blob/main/LICENSE),
+[notice](https://github.com/velodb/dbt-for-apache-doris/blob/main/NOTICE), and
+[migration provenance](https://github.com/velodb/dbt-for-apache-doris/blob/main/UPSTREAM.md).