dagster-dex 0.1.0__tar.gz

dagster_dex-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 David Anaya
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

dagster_dex-0.1.0/PKG-INFO

@@ -0,0 +1,301 @@
+Metadata-Version: 2.4
+Name: dagster-dex
+Version: 0.1.0
+Summary: Let dex read your Dagster asset graph the way it reads a dbt project: keys, joins, sources, semantics, drift
+Author: David Anaya
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/catincloud-labs/dagster-dex
+Project-URL: Repository, https://github.com/catincloud-labs/dagster-dex
+Project-URL: Issues, https://github.com/catincloud-labs/dagster-dex/issues
+Keywords: dagster,dex,dbt,data-quality,drift,project-format
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dagster
+Requires-Dist: dagster>=1.13; extra == "dagster"
+Provides-Extra: dex
+Requires-Dist: exmergo-dex-core~=1.6.4; extra == "dex"
+Provides-Extra: test
+Requires-Dist: pytest>=8.4; extra == "test"
+Dynamic: license-file
+
+# dagster-dex
+
+A Dagster asset graph read as a **project format** for
+[dex](https://github.com/exmergo/dex), and the tiered seam it proposes for
+reading one.
+
+Copyright 2026 David Anaya. Apache-2.0.
+
+```bash
+pip install dagster-dex          # the core: no orchestrator, no engine
+pip install 'dagster-dex[dex]'   # with the dex-core boundary
+```
+
+Or from source, for `main` ahead of the last release:
+
+```bash
+pip install 'git+https://github.com/catincloud-labs/dagster-dex'
+pip install 'dagster-dex[dex] @ git+https://github.com/catincloud-labs/dagster-dex'
+```
+
+Python 3.10+, matching Dagster's own floor. The `[dex]` extra needs 3.11+,
+because the engine does: on 3.10 you can still reduce a graph and write an
+artifact for a machine that has the engine to read it.
+
+Once installed, dex can name this format directly:
+
+```yaml
+# .dex/config.yml
+project:
+  format: dagster
+  options:
+    assets: my_project.definitions:all_assets
+```
+
+**dex** reads a project's declared structure (keys, joins, sources, semantics)
+and reports where the warehouse has drifted from it. It reads dbt projects
+natively. This package makes a Dagster asset graph readable the same way,
+without the graph having to pretend to be a dbt project first.
+
+## What this is
+
+A working sketch of a contract, not a finished integration. It exists to make a
+design argument concrete enough to disagree with: an asset graph is a perfectly
+good source of project truth, and reading one does not require it to pretend to
+be a dbt project first.
+
+## The seam, in three tiers
+
+| Tier | Adds | A format implements it when |
+| --- | --- | --- |
+| `ProjectSource` | `declarations()` | it can say what it declares |
+| `FingerprintedProject` | `fingerprint()` | it can be a drift baseline |
+| `EditableProject` | `propose_edits()` | its source of truth can receive an edit |
+
+`DagsterProject` implements tiers 1 and 2, and **not** tier 3 - not by setting a
+flag, but by not having the method. `isinstance(project, EditableProject)` is
+False, so a caller finds out by asking rather than by receiving an empty result
+that looks like success. A generated project is safe from stray writes today
+only where a naming convention happens not to match; declining a tier makes that
+structural.
+
+## Four model decisions worth arguing about
+
+**A key is always a tuple of columns.** A single-column grain is the `n == 1`
+case, not a different type. Modelling it the other way is what makes a composite
+grain inexpressible until someone widens the type - and after widening, every
+reader handles two shapes forever.
+
+**There is no project directory.** A path is one format's way of locating
+itself, not a property every project has.
+
+**Freshness is a tri-state.** `FRESH` / `STALE` / `NOT_APPLICABLE`. A boolean
+cannot tell "current" from "there is no such artifact here", so a format with
+nothing to compile reports itself identically to one that just compiled.
+
+**A required field is a refusal to invent.** `ExternalSource` requires a source
+system because the consumer requires one; making it optional would only move the
+moment of invention into the boundary code, where a fabricated value cannot be
+told from a declared one. Where the consumer requires something a project may
+genuinely lack, such as a file path, the field is optional here and the loss is
+reported instead of filled in.
+
+## External sources: the edge a graph cannot see
+
+Everything in `models` is something the project builds. `ExternalSource` is
+where it stops and depends on a warehouse object somebody else keeps honoring.
+
+These **cannot be derived from the asset graph**. A model that reads a table in
+its own SQL, because the source system offers no other interface, draws no
+dependency edge, so the read is invisible to any amount of traversal. It is
+genuinely extra information and has to be declared, keyed by the model that
+reads it. A format that inferred sources from dependencies would find none and
+look like it had checked.
+
+They are not decorative on the consumer side either: they drive a
+`dangling_source` finding and the set of tables a project is understood to
+cover. A format that reports none silently narrows its own blast radius.
+
+## Layout, and why the dependencies sit where they do
+
+```text
+model.py         the neutral model          stdlib only
+protocol.py      the tiered seam            stdlib only
+declarations.py  input parsing              + pyyaml
+project.py       the Dagster reduction      + dagster, lazily
+conformance.py   the importable contract    + pytest
+dex.py           the dex-core boundary      + exmergo-dex-core, lazily
+```
+
+The engine coupling is one file. Everything else, including the conformance
+suite, runs with dex-core uninstalled, which is what keeps the design from
+being shaped by whatever the engine happens to look like today. `dex.py` imports
+**public API only**; `tests/test_dex_bridge.py` asserts that with an AST walk
+rather than trusting it.
+
+## Running it
+
+From the repository root:
+
+```bash
+# the core: no orchestrator, no engine
+uv run --no-project --with-editable . \
+    --with pytest==8.4.1 \
+    python -m pytest tests \
+    --ignore=tests/test_dex_bridge.py \
+    --ignore=tests/test_upstream_contract.py
+
+# with the engine, to exercise the boundary
+uv run --no-project --with-editable . \
+    --with pytest==8.4.1 --with exmergo-dex-core==1.6.4 --with sqlglot==30.13.0 \
+    python -m pytest tests \
+    --ignore=tests/test_upstream_contract.py
+```
+
+`--with-editable` is not optional: without it the package is never installed and
+every test errors at collection on `No module named 'dagster_dex'`. These
+are the two commands CI runs, and the first is a **control**: it is what holds
+the engine coupling to one file, so an `exmergo_dex_core` import anywhere else
+turns it red at collection.
+
+The engine is pinned to `==1.6.4` here and in CI, which is **not** what the
+`[dex]` extra publishes. The extra is `~=1.6.4`, so consumers are not forced onto
+one patch release; the exact pin is where the demonstration lives, because a
+claim about what passed should name the version it passed against.
+
+### Against dex-core's own contract
+
+`tests/test_upstream_contract.py` runs this format against the conformance suite
+**dex-core ships**, which is upstream's acceptance criterion for a second project
+format (`exmergo/dex#144`). That suite ships in **v1.5.2**, released 2026-08-05,
+via [`exmergo/dex#192`](https://github.com/exmergo/dex/pull/192) - our own PR, so
+the criterion is now judged by upstream's released code rather than by a branch:
+
+```bash
+DEX_UPSTREAM_CONTRACT_REQUIRED=1 \
+uv run --no-project --with-editable . \
+    --with pytest==8.4.1 --with exmergo-dex-core==1.6.4 --with sqlglot==30.13.0 \
+    python -m pytest tests/test_upstream_contract.py
+```
+
+`DEX_UPSTREAM_CONTRACT_REQUIRED=1` is what makes it a signal, and the release did
+not change that. Without the variable the file skips itself when the contract is
+unimportable, which is right at a developer's desk and wrong in CI, where an unrun
+file and a passing file look the same. With it, a missing contract is a collection
+error - which still matters against a release, because a yank or a bad bump lands
+in exactly the same place a rewritten branch used to.
+
+## Using the contract on another format
+
+```python
+from dagster_dex.conformance import FingerprintedProjectContract
+
+class TestMyFormat(FingerprintedProjectContract):
+    def make_project(self, declarations, semantics, sources):
+        return MyProject(declaration_sources=declarations,
+                         semantic_sources=semantics,
+                         source_declarations=sources)
+```
+
+The assertions that matter most are behavioural: `declarations()` must not raise
+on an absent, empty, or malformed project. That is the property a second
+implementation is most likely to get wrong, and a suite checking only shapes
+will pass it.
+
+`make_project` took two arguments before external sources landed. The third
+is a **breaking change to this contract**, taken deliberately while the suite has
+no outside implementers rather than carried forever as an optional hook that
+could be skipped - and a skipped assertion is not a passing one.
+
+## Status
+
+**Alpha, and the contract may move before 1.0.** The tiered seam is a proposal,
+not a settled interface: it exists to be argued with, and the argument may change
+it. Pin the minor if you depend on it.
+
+**The entry point stopped being inert on 2026-08-08.** This section used to end:
+*"nothing resolves it today, and an entry point nobody looks up is inert."*
+dex-core **1.6.0** added resolution for exactly the `exmergo_dex_core.projects`
+group this package has declared since `0.1.0` - a format can now be named from
+`.dex/config.yml`, from a dotted `mypkg.projects:my_project` path, or from that
+entry-point group, with `--project-format` overriding on the CLI.
+
+That arrived through [`exmergo/dex#171`](https://github.com/exmergo/dex/issues/171),
+which this package's own constraint shaped: a host reaching dex as a subprocess
+cannot hand an object in, so name resolution was the only door that worked.
+
+**And it was registered wrong.** That paragraph originally ended *"what is left
+between here and a resolvable format is packaging, not design"*, written before
+anyone had run it. Resolution found the entry point immediately and then refused:
+it named the **class**, and dex-core's `ProjectFactory` calls what it resolves
+**with a `ProjectContext`**, so the context bound to `models`. A second gap sat
+behind it: a bare `DagsterProject` is refused as *"missing name, definitions"*,
+because this package says `format`/`declarations()` where the seam says
+`name`/`definitions()`.
+
+=> The lesson is worth more than the fix: **a declared-but-unresolved extension
+point is not evidence that registration works.** It was inert from `0.1.0`, so
+there was no moment before 2026-08-08 at which it could have failed.
+
+Both are fixed. The entry point names `dex:project_from_context`, which takes a
+context and returns a `DexProject`, and a regression test asserts the registration
+*and* that it loads to the callable.
+
+**This paragraph used to end with a hand-counted end-to-end result** - a model
+count, a source count, semantic models and metrics, from dex driven as a
+subprocess against a private asset graph. It was true where it was written and it
+is unverifiable here: nobody outside can run it, and it was four numbers in a
+document, which is the thing this project's own rules forbid. What replaces it is
+smaller and checkable by anyone:
+
+| Verified in CI, every commit | Where |
+| --- | --- |
+| The suite passes with the engine uninstalled | control step |
+| The suite passes against the engine | boundary step |
+| The format passes dex-core's own project contract | criterion step |
+| The reduction works on real Dagster objects | reduction step |
+| The annotations type-check at the floor | type check |
+| The built wheel imports, declares its entry point, ships `py.typed` | release workflow |
+
+Everything above runs from a clean checkout with two commands and no access to
+anything private, which is the property the old sentence did not have.
+
+### Two ways to name the project, and the reason there are two
+
+`assets:` reduces a live asset graph in the calling process. That is the honest
+form, and on a real project it costs **~2.6 s** - almost entirely the import of
+the code location, which is not something laziness or caching can reach, because
+a host that builds a project per command never holds one long enough to amortize
+it.
+
+`artifact:` is the answer for a host that cannot pay that: the side that already
+has the graph reduces it once and writes the result down, and the side that
+answers requests reads it back.
+
+```yaml
+project:
+  format: dagster
+  options:
+    artifact: project/my_project.json   # written by dagster_dex.artifact.dumps
+```
+
+Exactly one of the two is required, and naming both is refused rather than
+resolved: a snapshot and a live graph disagree by design. An artifact carries
+its own declarations and its own name, so `declarations:`/`semantics:`/`sources:`
+and `name:` are refused beside it rather than silently ignored.
+
+**A missing artifact is refused, not read as an empty project.** An empty
+project is a valid one, so the tolerant reading would report a broken deploy as a
+warehouse with nothing declared, quietly, and for as long as it lasted.