masterly 0.1.1__tar.gz → 0.2.0__tar.gz

masterly-0.2.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.egg-info/
+.venv/
+dist/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+
+# Claude Code worktrees. Each one is its own git repository, so committing the path adds
+# a gitlink rather than files — and `git add -A` picks it up silently.
+.claude/worktrees/

masterly-0.2.0/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.5
+Name: masterly
+Version: 0.2.0
+Summary: Python client for Masterly — extract governed data products and ingest records from your lakehouse
+Project-URL: Homepage, https://masterlydata.com
+Project-URL: Documentation, https://masterlydata.com/docs/reference/python-sdk/
+Project-URL: Repository, https://github.com/masterly-data/masterly-python-sdk
+Project-URL: Changelog, https://github.com/masterly-data/masterly-python-sdk/releases
+License: Apache-2.0
+License-File: LICENSE
+Keywords: data products,databricks,fabric,lakehouse,master data,mdm,snowflake
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.27
+Provides-Extra: pandas
+Requires-Dist: pandas>=2.0; extra == 'pandas'
+Description-Content-Type: text/markdown
+
+# masterly — Python client
+
+Extract governed data products and ingest records from Python — notebooks on Databricks,
+Microsoft Fabric, or anywhere else your pipelines run.
+
+```python
+from masterly import Client
+
+client = Client(
+    base_url="https://app.example.com",       # your Masterly install
+    token="<session or service-account token>",
+    environment="env_prod_eu",
+)
+
+# Extract: page through a data product (cursor-managed for you)
+for row in client.products.read("Customer 360"):
+    ...
+
+# Or straight into a DataFrame
+df = client.products.read("Customer 360").to_pandas()
+
+# Follow the change feed with a resumable cursor
+feed = client.products.changes("Customer 360", cursor=saved_cursor)
+for change in feed:
+    ...
+save(feed.cursor)  # persist for the next run
+
+# Ingest: deliver records to a source (chunked automatically)
+report = client.sources.ingest("crm", records)
+print(report.records, "records in", report.batches, "batches")
+```
+
+## Tokens: two personas
+
+- **Consumer** (extract in a Databricks/Fabric job): use a **service-account token**
+  (minted under Users → Service accounts). It pins its Environment and consumes published
+  products — reference them **by product id** (`dp_…`), since listing products needs a
+  session. Access policies (row rules, masks) apply per consumer automatically.
+- **Integrator** (ingest + golden reads): use a **session token**. Sources and products can
+  be referenced by name.
+
+## Governed writes: state the revision you are replacing
+
+Extract and ingest are ungoverned — a read is not a write, and an ingest batch is not derived
+from a read of what it lands in. A **governed** write is one where more than one principal may
+write the object and your request was composed from an earlier read of it: a rule set, an access
+policy, a shared view, a record you edited by hand. Those writes state which revision they
+replace, and the server refuses rather than letting the other editor's work disappear.
+
+Read, edit, write — the version comes back with the object:
+
+```python
+from masterly import ApiError
+
+policy = client.request("GET", "/v1/access-policies/pol_1")
+policy["rules"] = edited
+
+try:
+    client.request(
+        "PUT", "/v1/access-policies/pol_1", json=policy, if_match=policy["version"],
+    )
+except ApiError as error:
+    conflict = error.conflict
+    if conflict is None:
+        raise
+    print(f"{conflict.changed_by} changed {', '.join(conflict.changed_fields)}")
+    if conflict.undisclosed_changes:
+        print(f"and {conflict.undisclosed_changes} further changes you cannot see")
+    if not conflict.may_auto_merge:
+        ...  # re-read and decide by hand; the change set cannot be trusted as complete
+```
+
+- `if_match` takes the object's `version` field, or a `Precondition` — `Precondition.from_etag`
+  if you kept the response header instead, `Precondition.unconditional()` for `If-Match: *`
+  ("it must exist; I do not care which revision"), which is recorded in the audit trail as an
+  unconditional write.
+- **A conflict names what moved, never what it moved to.** There is no "theirs" column in the
+  refusal, by design: it would be a read you may not be entitled to, arriving by the error path.
+  Re-read the object to compare — that runs the ordinary authorization and access-policy path.
+- **Gate any automatic merge on `conflict.may_auto_merge`, never on a zero
+  `undisclosed_changes`.** A zero count means nothing was withheld from *you*; it is also what
+  you get when the writer recorded no field list at all — unknown, not empty.
+- Omitting the precondition still works today and answers with a `Deprecation` header. Governed
+  operations move to `428 PRECONDITION_REQUIRED` as their grace ends, and unconditionally at
+  `/v2`.
+
+`client.request(...)` is the escape hatch for endpoints the typed surface has not reached, on
+the same connection: auth, Environment, timeouts, and error mapping. It also takes `headers=`
+for anything per-request, such as an `Idempotency-Key`.
+
+## Errors
+
+The client is a thin, typed wrapper over the stable `/v1` REST contract — the same API
+every other channel uses. Errors raise `masterly.ApiError` carrying the server's error
+code, message, and `details` — the envelope's own payload, passed through as sent, which is
+where a refusal explains itself. Re-delivery is idempotent: records upsert by their source key,
+so running the same notebook twice never duplicates data.
+
+`pandas` is an optional dependency: `pip install masterly[pandas]`.
+
+## Configuration
+
+Extract and ingest run on a configuration someone built. These build it — enough to stand
+an Environment up from a script:
+
+```python
+client.workspaces.create("Demo")
+client.domains.create("Sales", workspace="Demo")
+client.data_models.create("Customer", domain="Sales", definition={...})
+client.data_models.publish("Customer")
+
+client.sources.create(
+    "erp",
+    target_model="Customer",
+    source_key="customer_number",              # required: without it every record quarantines
+    field_map={"KUNNR": "customer_number", "NAME1": "name"},
+)
+```
+
+Everything takes an id or an exact name, so a script reads the way the domain is discussed.
+A model's `definition` carries its attributes, keys and constraints — ingest validates every
+record against them, so widening or tightening one changes what the pipeline accepts.
+
+Editing a model is a **governed write**: more than one person edits models, and the edit is
+derived from a read, so `update` states the revision it replaces (ADR 0070) and a stale one
+is refused rather than silently overwriting:
+
+```python
+model = client.data_models.get("Customer")
+model["definition"]["attributes"].append({"name": "segment", "type": "string"})
+client.data_models.update(
+    "Customer", definition=model["definition"], if_match=model["version"]
+)
+```
+
+A refused write raises `ApiError`; `error.conflict` names the fields that moved. Anything
+without a typed method is reached through `client.request(...)`, which takes the same
+`if_match`.
+
+## Examples
+
+[`examples/demo_data.py`](examples/README.md) generates realistic demo master data —
+customers, suppliers and products as three source systems would actually deliver them,
+duplicates and defects included — and ingests it into an Environment.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).

masterly-0.2.0/README.md

@@ -0,0 +1,148 @@
+# masterly — Python client
+
+Extract governed data products and ingest records from Python — notebooks on Databricks,
+Microsoft Fabric, or anywhere else your pipelines run.
+
+```python
+from masterly import Client
+
+client = Client(
+    base_url="https://app.example.com",       # your Masterly install
+    token="<session or service-account token>",
+    environment="env_prod_eu",
+)
+
+# Extract: page through a data product (cursor-managed for you)
+for row in client.products.read("Customer 360"):
+    ...
+
+# Or straight into a DataFrame
+df = client.products.read("Customer 360").to_pandas()
+
+# Follow the change feed with a resumable cursor
+feed = client.products.changes("Customer 360", cursor=saved_cursor)
+for change in feed:
+    ...
+save(feed.cursor)  # persist for the next run
+
+# Ingest: deliver records to a source (chunked automatically)
+report = client.sources.ingest("crm", records)
+print(report.records, "records in", report.batches, "batches")
+```
+
+## Tokens: two personas
+
+- **Consumer** (extract in a Databricks/Fabric job): use a **service-account token**
+  (minted under Users → Service accounts). It pins its Environment and consumes published
+  products — reference them **by product id** (`dp_…`), since listing products needs a
+  session. Access policies (row rules, masks) apply per consumer automatically.
+- **Integrator** (ingest + golden reads): use a **session token**. Sources and products can
+  be referenced by name.
+
+## Governed writes: state the revision you are replacing
+
+Extract and ingest are ungoverned — a read is not a write, and an ingest batch is not derived
+from a read of what it lands in. A **governed** write is one where more than one principal may
+write the object and your request was composed from an earlier read of it: a rule set, an access
+policy, a shared view, a record you edited by hand. Those writes state which revision they
+replace, and the server refuses rather than letting the other editor's work disappear.
+
+Read, edit, write — the version comes back with the object:
+
+```python
+from masterly import ApiError
+
+policy = client.request("GET", "/v1/access-policies/pol_1")
+policy["rules"] = edited
+
+try:
+    client.request(
+        "PUT", "/v1/access-policies/pol_1", json=policy, if_match=policy["version"],
+    )
+except ApiError as error:
+    conflict = error.conflict
+    if conflict is None:
+        raise
+    print(f"{conflict.changed_by} changed {', '.join(conflict.changed_fields)}")
+    if conflict.undisclosed_changes:
+        print(f"and {conflict.undisclosed_changes} further changes you cannot see")
+    if not conflict.may_auto_merge:
+        ...  # re-read and decide by hand; the change set cannot be trusted as complete
+```
+
+- `if_match` takes the object's `version` field, or a `Precondition` — `Precondition.from_etag`
+  if you kept the response header instead, `Precondition.unconditional()` for `If-Match: *`
+  ("it must exist; I do not care which revision"), which is recorded in the audit trail as an
+  unconditional write.
+- **A conflict names what moved, never what it moved to.** There is no "theirs" column in the
+  refusal, by design: it would be a read you may not be entitled to, arriving by the error path.
+  Re-read the object to compare — that runs the ordinary authorization and access-policy path.
+- **Gate any automatic merge on `conflict.may_auto_merge`, never on a zero
+  `undisclosed_changes`.** A zero count means nothing was withheld from *you*; it is also what
+  you get when the writer recorded no field list at all — unknown, not empty.
+- Omitting the precondition still works today and answers with a `Deprecation` header. Governed
+  operations move to `428 PRECONDITION_REQUIRED` as their grace ends, and unconditionally at
+  `/v2`.
+
+`client.request(...)` is the escape hatch for endpoints the typed surface has not reached, on
+the same connection: auth, Environment, timeouts, and error mapping. It also takes `headers=`
+for anything per-request, such as an `Idempotency-Key`.
+
+## Errors
+
+The client is a thin, typed wrapper over the stable `/v1` REST contract — the same API
+every other channel uses. Errors raise `masterly.ApiError` carrying the server's error
+code, message, and `details` — the envelope's own payload, passed through as sent, which is
+where a refusal explains itself. Re-delivery is idempotent: records upsert by their source key,
+so running the same notebook twice never duplicates data.
+
+`pandas` is an optional dependency: `pip install masterly[pandas]`.
+
+## Configuration
+
+Extract and ingest run on a configuration someone built. These build it — enough to stand
+an Environment up from a script:
+
+```python
+client.workspaces.create("Demo")
+client.domains.create("Sales", workspace="Demo")
+client.data_models.create("Customer", domain="Sales", definition={...})
+client.data_models.publish("Customer")
+
+client.sources.create(
+    "erp",
+    target_model="Customer",
+    source_key="customer_number",              # required: without it every record quarantines
+    field_map={"KUNNR": "customer_number", "NAME1": "name"},
+)
+```
+
+Everything takes an id or an exact name, so a script reads the way the domain is discussed.
+A model's `definition` carries its attributes, keys and constraints — ingest validates every
+record against them, so widening or tightening one changes what the pipeline accepts.
+
+Editing a model is a **governed write**: more than one person edits models, and the edit is
+derived from a read, so `update` states the revision it replaces (ADR 0070) and a stale one
+is refused rather than silently overwriting:
+
+```python
+model = client.data_models.get("Customer")
+model["definition"]["attributes"].append({"name": "segment", "type": "string"})
+client.data_models.update(
+    "Customer", definition=model["definition"], if_match=model["version"]
+)
+```
+
+A refused write raises `ApiError`; `error.conflict` names the fields that moved. Anything
+without a typed method is reached through `client.request(...)`, which takes the same
+`if_match`.
+
+## Examples
+
+[`examples/demo_data.py`](examples/README.md) generates realistic demo master data —
+customers, suppliers and products as three source systems would actually deliver them,
+duplicates and defects included — and ingests it into an Environment.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).

masterly-0.2.0/examples/README.md

@@ -0,0 +1,145 @@
+# Demo data generator
+
+[`demo_data.py`](demo_data.py) fills a Masterly Environment with realistic master data —
+customers, suppliers and products as they would actually arrive from a CRM, an ERP and a
+webshop. Use it to demo the product, to have something to click through while building a
+feature, or to load-test an install.
+
+```bash
+# bootstrap + generate + ingest against a local install on the dev identity binding
+uv run examples/demo_data.py \
+    --base-url http://localhost:8001 \
+    --dev-login you@example.com \
+    --environment env_demo_prd_eu \
+    --wait
+```
+
+From this checkout, `uv run` picks up the project's own `masterly`. Anywhere else, bring the
+client along: `uv run --with 'masterly>=0.2.0' demo_data.py …`, or `pip install masterly`
+and `python demo_data.py …`. It needs **0.2.0 or later** — that is the release that carries
+the configuration surface it bootstraps with.
+
+The script itself is thin: `client.workspaces`, `client.domains`, `client.data_models` and
+`client.sources` build the configuration, `client.sources.ingest` delivers the records. The
+only raw HTTP left in it is the pair of calls that happen before a connection exists —
+minting a token and asking which Environments you may use.
+
+## What it creates
+
+Against an empty Environment the script is self-sufficient: it creates a Workspace, a
+Domain, three published Data Models, and one Source per originating system. Every step is
+find-or-create, so running it twice changes nothing about the configuration.
+
+Find-or-create matches on name, and a name is not proof of identity: an Environment can
+already hold someone else's `Customer`. So before delivering into a model or source it
+found rather than created, the script checks that it is the one it defines — same
+attributes, same target model, same natural key — and stops with an explanation instead of
+ingesting a batch that would quarantine wholesale. Use a clean Environment, remove the
+stale object, or generate for the existing model on its own terms with `--model`.
+
+| Data Model | Sources | Field names on the wire |
+|---|---|---|
+| Customer | `crm`, `erp`, `webshop` | Salesforce-style, SAP-style, and passthrough |
+| Supplier | `erp-suppliers`, `procurement-portal` | SAP-style and portal-style |
+| Product | `pim`, `erp-products` | PIM-style and SAP-style |
+
+Each Source carries the field map that conforms its own vocabulary to the model
+(`Account_Name`, `NAME1` and `name` all land in `name`), and declares its natural key.
+`webshop` is deliberately left as passthrough, so both mapping styles are represented.
+
+## What makes the data worth ingesting
+
+The population is generated once, then delivered *through* each system — so the same
+company arrives from the CRM, the ERP and the webshop under three different customer
+numbers, formatted the way that system formats things:
+
+- **Legal-form drift** — `Nordwind Logistik AB`, `NORDWIND LOGISTIK AB`, `Nordwind
+  Logistik Aktiebolag`, `Nordwind Logistik`
+- **Address drift** — `Storgatan 12` vs `Storg. 12`, `111 22` vs `11122`
+- **Phone formats** — `+46 8 123 45 67`, `08123 45 67`, `4681234567`
+- **Keying slips** — transposed, dropped and doubled characters; trailing whitespace from
+  a spreadsheet paste
+- **Coverage gaps** — the ERP holds no website or employee count, the webshop holds no
+  registration number, and optional fields go missing at random
+- **Intra-source duplicates** — the same company entered twice in one system under two
+  keys, never byte-identical (`--duplicate-rate`)
+- **Structural defects** — malformed e-mail, country spelled out, unknown enum value,
+  empty required field, negative amount, a number delivered as text, and a record with no
+  natural key at all (`--defect-rate`)
+
+The defects are the ones the platform actually rejects, so they land in the source's
+quarantine with a real reason rather than passing through unnoticed. `tests/test_demo_data.py`
+holds the generator to that: clean records must satisfy the model's own constraints, and
+every defect must break one.
+
+Everything is seeded — the same `--seed` produces the same records, and re-delivery
+upserts by source key, so running twice never doubles the mastered data. (Quarantine rows
+do accumulate: each delivery of a bad record is its own quarantine entry.)
+
+## Common runs
+
+```bash
+# look at the records without an install
+uv run examples/demo_data.py --dry-run --customers 5
+
+# write them to a file instead
+uv run examples/demo_data.py --dry-run --out /tmp/records.json
+
+# a bigger population, customers only
+uv run examples/demo_data.py --base-url … --dev-login … --models customer --customers 5000
+
+# messier data, for the stewardship queues
+uv run examples/demo_data.py --base-url … --dev-login … --defect-rate 0.2 --duplicate-rate 0.3
+
+# configuration already exists — just deliver more records
+uv run examples/demo_data.py --base-url … --token "$MASTERLY_TOKEN" --no-bootstrap
+
+# ingest into a model of your own, generated from its live definition
+uv run examples/demo_data.py --base-url … --token "$MASTERLY_TOKEN" \
+    --model "Customer 360" --source crm --records 500
+```
+
+`--model` reads the model's definition from the install and invents values that fit it —
+types, enum values, ranges, and a guess from each attribute's name (`*_email` gets an
+e-mail, `*_city` a city, `*_date` a date). The natural key comes from the Source's own
+`source_key`. Where an attribute has a format constraint the generator cannot satisfy, it
+says so and leaves the attribute empty rather than quarantining every record.
+
+## Authenticating
+
+- `--dev-login EMAIL` mints a session on installs running the **dev identity binding**
+  (local development, the demo install). It sends the credential as `dev:<email>`.
+- `--token` (or `MASTERLY_TOKEN`) takes a session token from any install — use this on
+  Stytch or OIDC bindings. `--base-url` and `--environment` also read
+  `MASTERLY_BASE_URL` / `MASTERLY_ENVIRONMENT`.
+
+Bootstrapping needs `workspace:create`, `domain:create`, `data-model:create`,
+`data-model:update`, `source:create` and `ingest:run` in the target Environment — the
+**Modeler** preset role covers all six. With `--no-bootstrap` or `--model`, an
+**Integrator** is enough.
+
+## After the run
+
+`--wait` polls until the pipeline stops moving and prints where the records landed:
+
+```
+source                  accepted  quarantined
+---------------------------------------------
+crm                           36            4
+erp                           26            8
+webshop                       14            2
+...
+
+model                   source records  golden entities
+-------------------------------------------------------
+Customer                            76               76
+```
+
+One golden entity per source record means **no match rules are configured yet** — the
+duplicates this script generates on purpose are all still sitting there unlinked. Set up
+matching for a model — `PUT /v1/match/config/{model_name}`, or the matching editor in the
+app — and saving the config re-resolves the model, dropping the entity count below the
+source-record count. That is the demo worth showing.
+
+Ingest is asynchronous. Without `--wait` the script returns as soon as the batches are
+accepted, and the records appear a few seconds later.