data360-autodoc 0.2.0__tar.gz

data360_autodoc-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Valentina Tihova
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data360_autodoc-0.2.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: data360-autodoc
+Version: 0.2.0
+Summary: Auto-generate human-readable documentation for Salesforce Data 360 orgs.
+Author: data360-autodoc contributors
+License: MIT
+Project-URL: Homepage, https://github.com/valentinatihova/data360-autodoc
+Project-URL: Repository, https://github.com/valentinatihova/data360-autodoc
+Project-URL: Issues, https://github.com/valentinatihova/data360-autodoc/issues
+Keywords: salesforce,data-cloud,data-360,documentation,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Documentation
+Classifier: Topic :: Software Development :: Documentation
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.1
+Requires-Dist: PyJWT>=2.8
+Requires-Dist: cryptography>=42.0
+Requires-Dist: requests>=2.31
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: responses>=0.25; extra == "dev"
+Requires-Dist: black>=24.0; extra == "dev"
+Dynamic: license-file
+
+# data360-autodoc
+
+[![PyPI version](https://img.shields.io/badge/pypi-coming%20soon-blue)](https://pypi.org/project/data360-autodoc/)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](#)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+**Auto-generate human-readable documentation for Salesforce Data 360 (Data Cloud) orgs — in seconds, not days.**
+
+Point it at an org and it produces a full data dictionary (DMOs, DLOs, fields, keys), the data streams and field-level mappings behind them, the DMO relationship graph, an ERD, and a deterministic JSON snapshot.
+
+- 📓 **Data dictionary** — every DMO and DLO as clean Markdown tables, with field names, types, and keys.
+- 🌊 **Data streams + field mappings** — per-stream source/refresh metadata, the Stream → DLO field map, and the DLO → DMO field map (with real labels, not just API names).
+- 🔗 **Relationships + ERD** — DMO-to-DMO relationships with cardinality and status, plus a Mermaid graph of DLO → DMO mappings and relationship edges.
+- 🧊 **JSON snapshot** — a deterministic, diff-friendly export of your whole org schema (the foundation for drift detection — see below).
+
+## For who
+
+Built for **Salesforce SI consultants and Data Cloud practitioners** who lose days hand-writing org documentation for every engagement. Works against any Data 360 org you can authenticate to with a connected app — including **Developer Edition / Data Cloud Dev orgs**, so you can try it on a sandbox before pointing it at a client.
+
+## Quick start
+
+```bash
+pip install data360-autodoc
+
+data360-autodoc generate \
+  --instance-url https://mydomain.my.salesforce.com \
+  --client-id <connected-app-consumer-key> \
+  --private-key ./server.pem \
+  --username admin@myorg.com \
+  --output ./docs \
+  --format all
+```
+
+```
+Wrote acme-data-cloud.md
+Wrote acme-data-cloud.mmd
+Wrote acme-data-cloud.json
+Generated docs for 24 DMOs, 11 DLOs, 0 Identity Rulesets
+```
+
+Authentication uses the **OAuth 2.0 JWT Bearer flow** (connected app + private key — no passwords stored).
+
+**Options that affect the metadata fetch:**
+
+- `--sandbox` — authenticate against `test.salesforce.com` (sandbox / scratch orgs).
+- `--api-version` — the Salesforce REST API version used for the `/ssot/*` metadata calls (e.g. `v62.0`). **By default the tool auto-detects your org's highest supported version** (from `GET /services/data/`), so you normally don't set this. Force it only if auto-detection picks a version where a Data Cloud endpoint misbehaves, or to pin output to a specific version. It must be a valid Salesforce REST API version your org supports.
+
+(The `Identity Rulesets` count is currently always `0` — see "Not supported yet" below.)
+
+## What you get
+
+`--format` controls the output:
+
+| Format | Files | What it is |
+|--------|-------|------------|
+| `markdown` | `.md` + `.mmd` | Data dictionary + Mermaid ERD |
+| `json` | `.json` | Deterministic org-schema snapshot |
+| `pdf` | — | _Coming soon_ |
+| `all` | all of the above | Everything |
+
+### Example output
+
+The Markdown data dictionary. DMO field types come from the org's relationships metadata, and fall back to the mapped DLO field type when the DMO endpoint returns a generic type (shown as `(via DLO)`); DLO keys come from the data streams:
+
+```markdown
+## Data Model Objects (DMOs)
+
+### Individual (`Individual__dmo`)
+
+| Name | Type | Key |
+| --- | --- | --- |
+| Email__c | EmailAddress |  |
+| Id__c | Text |  |
+
+## Data Lake Objects (DLOs)
+
+### Order (Home) (`Order_Home__dll`)
+
+| Name | Type | Key |
+| --- | --- | --- |
+| Amount | Number |  |
+| OrderId | Text | PrimaryKey |
+```
+
+Beyond the dictionary, the document includes (in this order):
+
+- **Data Streams** — one row per stream: data source, category, primary key, schedule, refresh mode.
+- **Field Mapping (Streams → DLO)** — every Data Lake field with its source field, DLO label, type, and a `KQ_`-prefix foreign-key flag.
+- **DLO → DMO Field Mappings** — field-level source → target mappings, grouped by DLO → DMO pairing, with real labels joined from DLO metadata.
+- **Relationships** — DMO-to-DMO links with cardinality and status (inactive standard relationships stay visible, never dropped):
+
+```markdown
+## Relationships
+
+| Object | Field | Cardinality | Related Object | Related Field | Status |
+| --- | --- | --- | --- | --- | --- |
+| Account | ssot__PrimarySalesContactPointId__c | N:1 | ssot__ContactPointEmail__dlm | ssot__Id__c | INACTIVE |
+```
+
+The ERD (renders natively in GitHub). Solid arrows are DLO → DMO mappings; dashed, cardinality-labeled arrows are active DMO → DMO relationships:
+
+```mermaid
+graph LR
+  Order_Home__dll["Order (Home)"]
+  Individual__dmo["Individual"]
+  Order_Home__dll --> Individual__dmo
+  ContactPointEmail__dlm["Contact Point Email"] -.->|N:1| Individual__dmo
+```
+
+Output is **deterministic** — the same org always produces byte-identical docs (collections are sorted alphabetically). That makes the output safe to commit and easy to diff.
+
+### What it reads — and what it doesn't yet
+
+Under the hood it calls the **Data 360 Connect REST API** (`/services/data/v…/ssot/*`): `data-model-objects` (DMOs), `data-model-object-mappings` (DLO→DMO mappings + field names), `…/{dmo}/relationships` (DMO field types **and** the DMO-to-DMO relationship graph), and `data-streams` (DLOs + their fields + the per-stream and field-level mappings, including primary keys). Full request/response shapes are in [`agent_docs/api_reference.md`](agent_docs/api_reference.md).
+
+**Not supported yet.** Calculated Insights and Identity Resolution rulesets are **not fetched** — those sections render as empty placeholders (e.g. `_No Calculated Insights found._`) and the `Identity Rulesets` count stays `0`. Documenting them is on the roadmap. (Profile and Engagement DMOs *are* covered — those are DMO categories, not separate entities.)
+
+**Resilient by default.** If one DMO's metadata can't be read, that DMO is skipped with a warning and the rest of the document is still produced. If the org has more than 500 DMOs, the list is capped (with a warning). A failure fetching the DMO list or the data streams stops the run with a clean one-line error — never a stack trace.
+
+## Future: drift monitoring (paid tier)
+
+The open-source CLI documents your org once. The thing that actually bites consultants is when an org **changes** after you've documented it — a client admin adds a DLO, a field type changes, an identity rule shifts — and your beautiful docs quietly go stale.
+
+A hosted tier (planned) will turn the deterministic JSON snapshot into **drift monitoring**: re-run on a schedule, diff today's snapshot against the last one, and get a client-ready changelog of exactly what changed — without ever handing over your org credentials (drift runs in your own environment; the hosted service only stores snapshots and sends alerts). The CLI stays free forever; the recurring watching, history, and multi-org dashboard are the paid layer.
+
+## Hosted version
+
+A hosted web UI is in the works at **[data360doc.com](https://data360doc.com)** _(placeholder)_ — same docs, plus scheduled drift alerts and a multi-org dashboard for agencies.
+
+## License
+
+MIT

data360_autodoc-0.2.0/README.md

@@ -0,0 +1,132 @@
+# data360-autodoc
+
+[![PyPI version](https://img.shields.io/badge/pypi-coming%20soon-blue)](https://pypi.org/project/data360-autodoc/)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](#)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+**Auto-generate human-readable documentation for Salesforce Data 360 (Data Cloud) orgs — in seconds, not days.**
+
+Point it at an org and it produces a full data dictionary (DMOs, DLOs, fields, keys), the data streams and field-level mappings behind them, the DMO relationship graph, an ERD, and a deterministic JSON snapshot.
+
+- 📓 **Data dictionary** — every DMO and DLO as clean Markdown tables, with field names, types, and keys.
+- 🌊 **Data streams + field mappings** — per-stream source/refresh metadata, the Stream → DLO field map, and the DLO → DMO field map (with real labels, not just API names).
+- 🔗 **Relationships + ERD** — DMO-to-DMO relationships with cardinality and status, plus a Mermaid graph of DLO → DMO mappings and relationship edges.
+- 🧊 **JSON snapshot** — a deterministic, diff-friendly export of your whole org schema (the foundation for drift detection — see below).
+
+## For who
+
+Built for **Salesforce SI consultants and Data Cloud practitioners** who lose days hand-writing org documentation for every engagement. Works against any Data 360 org you can authenticate to with a connected app — including **Developer Edition / Data Cloud Dev orgs**, so you can try it on a sandbox before pointing it at a client.
+
+## Quick start
+
+```bash
+pip install data360-autodoc
+
+data360-autodoc generate \
+  --instance-url https://mydomain.my.salesforce.com \
+  --client-id <connected-app-consumer-key> \
+  --private-key ./server.pem \
+  --username admin@myorg.com \
+  --output ./docs \
+  --format all
+```
+
+```
+Wrote acme-data-cloud.md
+Wrote acme-data-cloud.mmd
+Wrote acme-data-cloud.json
+Generated docs for 24 DMOs, 11 DLOs, 0 Identity Rulesets
+```
+
+Authentication uses the **OAuth 2.0 JWT Bearer flow** (connected app + private key — no passwords stored).
+
+**Options that affect the metadata fetch:**
+
+- `--sandbox` — authenticate against `test.salesforce.com` (sandbox / scratch orgs).
+- `--api-version` — the Salesforce REST API version used for the `/ssot/*` metadata calls (e.g. `v62.0`). **By default the tool auto-detects your org's highest supported version** (from `GET /services/data/`), so you normally don't set this. Force it only if auto-detection picks a version where a Data Cloud endpoint misbehaves, or to pin output to a specific version. It must be a valid Salesforce REST API version your org supports.
+
+(The `Identity Rulesets` count is currently always `0` — see "Not supported yet" below.)
+
+## What you get
+
+`--format` controls the output:
+
+| Format | Files | What it is |
+|--------|-------|------------|
+| `markdown` | `.md` + `.mmd` | Data dictionary + Mermaid ERD |
+| `json` | `.json` | Deterministic org-schema snapshot |
+| `pdf` | — | _Coming soon_ |
+| `all` | all of the above | Everything |
+
+### Example output
+
+The Markdown data dictionary. DMO field types come from the org's relationships metadata, and fall back to the mapped DLO field type when the DMO endpoint returns a generic type (shown as `(via DLO)`); DLO keys come from the data streams:
+
+```markdown
+## Data Model Objects (DMOs)
+
+### Individual (`Individual__dmo`)
+
+| Name | Type | Key |
+| --- | --- | --- |
+| Email__c | EmailAddress |  |
+| Id__c | Text |  |
+
+## Data Lake Objects (DLOs)
+
+### Order (Home) (`Order_Home__dll`)
+
+| Name | Type | Key |
+| --- | --- | --- |
+| Amount | Number |  |
+| OrderId | Text | PrimaryKey |
+```
+
+Beyond the dictionary, the document includes (in this order):
+
+- **Data Streams** — one row per stream: data source, category, primary key, schedule, refresh mode.
+- **Field Mapping (Streams → DLO)** — every Data Lake field with its source field, DLO label, type, and a `KQ_`-prefix foreign-key flag.
+- **DLO → DMO Field Mappings** — field-level source → target mappings, grouped by DLO → DMO pairing, with real labels joined from DLO metadata.
+- **Relationships** — DMO-to-DMO links with cardinality and status (inactive standard relationships stay visible, never dropped):
+
+```markdown
+## Relationships
+
+| Object | Field | Cardinality | Related Object | Related Field | Status |
+| --- | --- | --- | --- | --- | --- |
+| Account | ssot__PrimarySalesContactPointId__c | N:1 | ssot__ContactPointEmail__dlm | ssot__Id__c | INACTIVE |
+```
+
+The ERD (renders natively in GitHub). Solid arrows are DLO → DMO mappings; dashed, cardinality-labeled arrows are active DMO → DMO relationships:
+
+```mermaid
+graph LR
+  Order_Home__dll["Order (Home)"]
+  Individual__dmo["Individual"]
+  Order_Home__dll --> Individual__dmo
+  ContactPointEmail__dlm["Contact Point Email"] -.->|N:1| Individual__dmo
+```
+
+Output is **deterministic** — the same org always produces byte-identical docs (collections are sorted alphabetically). That makes the output safe to commit and easy to diff.
+
+### What it reads — and what it doesn't yet
+
+Under the hood it calls the **Data 360 Connect REST API** (`/services/data/v…/ssot/*`): `data-model-objects` (DMOs), `data-model-object-mappings` (DLO→DMO mappings + field names), `…/{dmo}/relationships` (DMO field types **and** the DMO-to-DMO relationship graph), and `data-streams` (DLOs + their fields + the per-stream and field-level mappings, including primary keys). Full request/response shapes are in [`agent_docs/api_reference.md`](agent_docs/api_reference.md).
+
+**Not supported yet.** Calculated Insights and Identity Resolution rulesets are **not fetched** — those sections render as empty placeholders (e.g. `_No Calculated Insights found._`) and the `Identity Rulesets` count stays `0`. Documenting them is on the roadmap. (Profile and Engagement DMOs *are* covered — those are DMO categories, not separate entities.)
+
+**Resilient by default.** If one DMO's metadata can't be read, that DMO is skipped with a warning and the rest of the document is still produced. If the org has more than 500 DMOs, the list is capped (with a warning). A failure fetching the DMO list or the data streams stops the run with a clean one-line error — never a stack trace.
+
+## Future: drift monitoring (paid tier)
+
+The open-source CLI documents your org once. The thing that actually bites consultants is when an org **changes** after you've documented it — a client admin adds a DLO, a field type changes, an identity rule shifts — and your beautiful docs quietly go stale.
+
+A hosted tier (planned) will turn the deterministic JSON snapshot into **drift monitoring**: re-run on a schedule, diff today's snapshot against the last one, and get a client-ready changelog of exactly what changed — without ever handing over your org credentials (drift runs in your own environment; the hosted service only stores snapshots and sends alerts). The CLI stays free forever; the recurring watching, history, and multi-org dashboard are the paid layer.
+
+## Hosted version
+
+A hosted web UI is in the works at **[data360doc.com](https://data360doc.com)** _(placeholder)_ — same docs, plus scheduled drift alerts and a multi-org dashboard for agencies.
+
+## License
+
+MIT

data360_autodoc-0.2.0/data360_autodoc/__init__.py

@@ -0,0 +1 @@
+"""data360-autodoc: auto-generate documentation for Salesforce Data 360 orgs."""

data360_autodoc-0.2.0/data360_autodoc/cli/__init__.py

@@ -0,0 +1 @@
+"""Command-line interface for data360-autodoc."""

data360_autodoc-0.2.0/data360_autodoc/cli/main.py

@@ -0,0 +1,209 @@
+"""``data360-autodoc`` command-line entry point.
+
+Orchestrates the one-shot pipeline::
+
+    auth (JWT bearer) -> fetch metadata -> render outputs -> write files
+
+Outputs are selected with ``--format``:
+
+- ``markdown`` — human-readable ``.md`` plus a ``.mmd`` Mermaid diagram
+- ``json``     — deterministic ``.json`` snapshot (the drift-detection seam)
+- ``pdf``      — not yet implemented (stub; warns and skips)
+- ``all``      — markdown + json (+ pdf stub warning)
+
+The ``.json`` snapshot is a first-class output, not a debug artifact: the paid
+drift tier loads a prior snapshot and diffs it against a fresh fetch.
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import click
+
+from data360_autodoc.fetcher.auth import (
+    AUD_PRODUCTION,
+    AUD_SANDBOX,
+    DEFAULT_TOKEN_URL,
+    SANDBOX_TOKEN_URL,
+    AuthError,
+    get_access_token,
+)
+from data360_autodoc.fetcher.metadata import MetadataError, fetch_metadata
+from data360_autodoc.generator.markdown import render_markdown
+from data360_autodoc.generator.mermaid import render_mermaid
+from data360_autodoc.generator.snapshot import render_json
+
+#: Valid values for the ``--format`` option.
+FORMATS = ["markdown", "json", "pdf", "all"]
+
+
+@click.group()
+def cli() -> None:
+    """Auto-generate documentation for Salesforce Data 360 orgs."""
+
+
+@cli.command()
+@click.option("--instance-url", required=True, help="Org base URL.")
+@click.option(
+    "--access-token",
+    default=None,
+    help="Use a pre-obtained OAuth access token and skip JWT auth. When set, "
+    "--client-id / --private-key / --username are not needed.",
+)
+@click.option(
+    "--client-id", default=None, help="Connected app consumer key (JWT auth)."
+)
+@click.option(
+    "--private-key",
+    "private_key_path",
+    default=None,
+    type=click.Path(exists=True, dir_okay=False),
+    help="Path to the connected app's PEM private key (JWT auth).",
+)
+@click.option(
+    "--username", default=None, help="Salesforce username to impersonate (JWT auth)."
+)
+@click.option(
+    "--output",
+    "output_dir",
+    default=".",
+    type=click.Path(file_okay=False),
+    help="Directory to write output files into (created if missing).",
+)
+@click.option(
+    "--format",
+    "output_format",
+    type=click.Choice(FORMATS),
+    default="all",
+    show_default=True,
+    help="Which artifacts to generate.",
+)
+@click.option(
+    "--sandbox",
+    is_flag=True,
+    default=False,
+    help="Authenticate against test.salesforce.com (sandbox/scratch orgs).",
+)
+@click.option(
+    "--api-version",
+    "api_version",
+    default=None,
+    help="Data API version (e.g. v62.0). Default: auto-detect the org's highest.",
+)
+@click.option(
+    "--timeout",
+    default=120.0,
+    show_default=True,
+    type=float,
+    help="Per-request timeout (seconds) for metadata calls. Data Cloud is slow.",
+)
+def generate(
+    instance_url: str,
+    access_token: str | None,
+    client_id: str | None,
+    private_key_path: str | None,
+    username: str | None,
+    output_dir: str,
+    output_format: str,
+    sandbox: bool,
+    api_version: str | None,
+    timeout: float,
+) -> None:
+    """Fetch an org's metadata and write documentation artifacts."""
+    if access_token:
+        token = access_token
+        org_url = instance_url
+    else:
+        missing = [
+            flag
+            for flag, value in (
+                ("--client-id", client_id),
+                ("--private-key", private_key_path),
+                ("--username", username),
+            )
+            if not value
+        ]
+        if missing:
+            raise click.UsageError(
+                "Provide --access-token, or all of --client-id, --private-key, "
+                f"--username for JWT auth. Missing: {', '.join(missing)}."
+            )
+        token_url = SANDBOX_TOKEN_URL if sandbox else DEFAULT_TOKEN_URL
+        audience = AUD_SANDBOX if sandbox else AUD_PRODUCTION
+        try:
+            auth = get_access_token(
+                instance_url=instance_url,
+                client_id=client_id,
+                private_key_path=private_key_path,
+                username=username,
+                token_url=token_url,
+                audience=audience,
+            )
+        except AuthError as exc:
+            raise click.ClickException(str(exc)) from exc
+        token = auth["access_token"]
+        org_url = auth["instance_url"]
+
+    def _progress(message: str, inline: bool) -> None:
+        # Progress goes to stderr so it never pollutes the doc output. inline
+        # uses a carriage return to update the DMO counter in place.
+        if inline:
+            click.echo(f"\r{message}", nl=False, err=True)
+        else:
+            click.echo(message, err=True)
+
+    try:
+        schema = fetch_metadata(
+            instance_url=org_url,
+            access_token=token,
+            api_version=api_version,
+            timeout=timeout,
+            progress=_progress,
+        )
+    except MetadataError as exc:
+        # Surface a clean one-line error and exit non-zero — never a traceback.
+        raise click.ClickException(str(exc)) from exc
+
+    out_dir = Path(output_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    base = _slug(schema.org_name) or "data360"
+
+    written: list[str] = []
+    want_markdown = output_format in ("markdown", "all")
+    want_json = output_format in ("json", "all")
+    want_pdf = output_format in ("pdf", "all")
+
+    if want_markdown:
+        md_path = out_dir / f"{base}.md"
+        md_path.write_text(render_markdown(schema), encoding="utf-8")
+        written.append(md_path.name)
+        mmd_path = out_dir / f"{base}.mmd"
+        mmd_path.write_text(render_mermaid(schema) + "\n", encoding="utf-8")
+        written.append(mmd_path.name)
+
+    if want_json:
+        json_path = out_dir / f"{base}.json"
+        json_path.write_text(render_json(schema), encoding="utf-8")
+        written.append(json_path.name)
+
+    if want_pdf:
+        click.echo("PDF output is not yet implemented (stub) — skipping.", err=True)
+
+    for name in written:
+        click.echo(f"Wrote {name}")
+    click.echo(
+        f"Generated docs for {len(schema.dmos)} DMOs, "
+        f"{len(schema.dlos)} DLOs, "
+        f"{len(schema.identity_rulesets)} Identity Rulesets"
+    )
+
+
+def _slug(value: str) -> str:
+    """Slugify an org name for use as an output filename stem."""
+    return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
+
+
+if __name__ == "__main__":  # pragma: no cover
+    cli()

data360_autodoc-0.2.0/data360_autodoc/fetcher/__init__.py

@@ -0,0 +1 @@
+"""Salesforce Data 360 API client package."""

data360_autodoc-0.2.0/data360_autodoc/fetcher/_http.py

@@ -0,0 +1,110 @@
+"""Shared HTTP helpers for the Data 360 Connect REST API clients.
+
+All ``/ssot/*`` fetchers share the same needs: a bearer-authenticated GET with
+exponential-backoff retry, and ``nextPageUrl`` pagination with a cycle guard.
+Centralizing them here keeps the per-endpoint clients (``metadata.py``,
+``streams.py``) small and consistent.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, Final, Iterator
+
+import requests
+
+_MAX_ATTEMPTS: Final = 3
+_BACKOFF_BASE_SECONDS: Final = 1.0
+
+
+class FetchError(RuntimeError):
+    """Raised when a Data 360 API request fails (4xx, or retries exhausted)."""
+
+
+def get_json(url: str, *, access_token: str, timeout: float) -> Any:
+    """GET ``url`` with bearer auth, retrying transient failures.
+
+    Retries transport errors and 5xx responses up to three times with
+    exponential backoff (1s, 2s, 4s). 4xx responses are terminal.
+
+    Args:
+        url: Absolute URL to request.
+        access_token: OAuth bearer token.
+        timeout: Per-request timeout in seconds.
+
+    Returns:
+        The decoded JSON body.
+
+    Raises:
+        FetchError: On a 4xx response or after exhausting retries.
+    """
+    headers = {"Authorization": f"Bearer {access_token}", "Accept": "application/json"}
+    last_error: str | None = None
+    for attempt in range(_MAX_ATTEMPTS):
+        try:
+            response = requests.get(url, headers=headers, timeout=timeout)
+        except requests.RequestException as exc:
+            last_error = str(exc)
+        else:
+            if response.status_code == 200:
+                try:
+                    return response.json()
+                except ValueError as exc:
+                    # A 200 with a non-JSON body (e.g. a proxy/login HTML page)
+                    # must become a FetchError, not leak a raw JSONDecodeError
+                    # past the CLI's clean-error handler.
+                    raise FetchError(
+                        f"Non-JSON 200 response from {url}: {exc}"
+                    ) from exc
+            if 400 <= response.status_code < 500:
+                raise FetchError(
+                    f"Request rejected ({response.status_code}) for {url}: "
+                    f"{response.text[:500]}"
+                )
+            last_error = f"HTTP {response.status_code}: {response.text[:500]}"
+        if attempt < _MAX_ATTEMPTS - 1:
+            time.sleep(_BACKOFF_BASE_SECONDS * (2**attempt))
+    raise FetchError(
+        f"Request to {url} failed after {_MAX_ATTEMPTS} attempts: {last_error}"
+    )
+
+
+def iter_pages(
+    first_url: str, *, base_url: str, access_token: str, timeout: float
+) -> Iterator[dict[str, Any]]:
+    """Yield each page of a paginated ``/ssot/*`` response.
+
+    Follows each page's ``nextPageUrl`` (absolute or relative) until exhausted,
+    guarding against a self-referential or repeating link that would otherwise
+    loop forever.
+
+    Args:
+        first_url: Absolute URL of the first page.
+        base_url: Org base URL, used to resolve relative ``nextPageUrl`` values.
+        access_token: OAuth bearer token.
+        timeout: Per-request timeout in seconds.
+
+    Yields:
+        Each page's decoded JSON body.
+
+    Raises:
+        FetchError: On a request failure or a detected pagination cycle.
+    """
+    seen: set[str] = set()
+    next_url: str | None = first_url
+    while next_url:
+        if next_url in seen:
+            raise FetchError(f"Pagination cycle detected at {next_url}")
+        seen.add(next_url)
+        page = get_json(next_url, access_token=access_token, timeout=timeout)
+        yield page
+        next_url = _resolve_next(page.get("nextPageUrl"), base_url)
+
+
+def _resolve_next(raw: Any, base_url: str) -> str | None:
+    """Normalize a ``nextPageUrl`` value to an absolute URL, or ``None``."""
+    if not raw or not isinstance(raw, str):
+        return None
+    if raw.startswith("http://") or raw.startswith("https://"):
+        return raw
+    return f"{base_url.rstrip('/')}/{raw.lstrip('/')}"