dtex 0.2.0__tar.gz → 0.2.2__tar.gz

{dtex-0.2.0 → dtex-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dtex
-Version: 0.2.0
+Version: 0.2.2
 Summary: dtex (data extraction tool) — an open-source Python EL tool: pipelines as configs, connectors as folders, CLI-first.
 Project-URL: Homepage, https://github.com/vej-ai/dtex
 Project-URL: Documentation, https://github.com/vej-ai/dtex/tree/main/docs
@@ -104,7 +104,7 @@ params. Run it with `dtex run -p <config>`. The library equivalent is
 **Sources:** `filesystem` (CSV/JSONL/Parquet from local, GCS, or S3),
 `rest` (paginated REST APIs — 4 pagination strategies, 4 auth modes),
 `postgres` (keyset pagination, no `OFFSET`), `shiphero` (GraphQL),
-`stripe` (resource-as-stream over the REST API).
+`stripe` (REST resource-as-stream + opt-in Sigma SQL-as-stream).
 
 **Destinations:** `duckdb` (zero-config dev default, all 5 capabilities) and
 `bigquery` (production warehouse — Parquet-staged via GCS + LOAD jobs,
@@ -119,6 +119,25 @@ per-destination caps.
 **Secret managers:** GCP Secret Manager, AWS Secrets Manager, HashiCorp
 Vault — each as an opt-in extra.
 
+## Bundled Claude skills
+
+dtex ships three [Claude](https://claude.com) skills inside the wheel:
+`dtex-write-config`, `dtex-write-connector`, `dtex-debug`. They teach
+Claude (or any other agent runtime that reads `.claude/skills/`) the
+post-streams-redesign config schema, the `@stream` decorator pattern, and
+the debugging playbook. Install them into a dtex project with:
+
+```sh
+dtex skills install         # copies into ./.claude/skills/dtex/
+```
+
+The first `dtex` command run inside a project that lacks installed skills
+prints a one-line hint (suppressed thereafter). Run `dtex skills list` to
+see which ship with this dtex version and which are installed locally.
+
+If you're not using Claude, just ignore the hint — the skills are inert
+markdown until installed.
+
 ## Documentation
 
 The full design handbook lives in [`docs/`](https://github.com/vej-ai/dtex/tree/main/docs).

{dtex-0.2.0 → dtex-0.2.2}/README.md

@@ -54,7 +54,7 @@ params. Run it with `dtex run -p <config>`. The library equivalent is
 **Sources:** `filesystem` (CSV/JSONL/Parquet from local, GCS, or S3),
 `rest` (paginated REST APIs — 4 pagination strategies, 4 auth modes),
 `postgres` (keyset pagination, no `OFFSET`), `shiphero` (GraphQL),
-`stripe` (resource-as-stream over the REST API).
+`stripe` (REST resource-as-stream + opt-in Sigma SQL-as-stream).
 
 **Destinations:** `duckdb` (zero-config dev default, all 5 capabilities) and
 `bigquery` (production warehouse — Parquet-staged via GCS + LOAD jobs,
@@ -69,6 +69,25 @@ per-destination caps.
 **Secret managers:** GCP Secret Manager, AWS Secrets Manager, HashiCorp
 Vault — each as an opt-in extra.
 
+## Bundled Claude skills
+
+dtex ships three [Claude](https://claude.com) skills inside the wheel:
+`dtex-write-config`, `dtex-write-connector`, `dtex-debug`. They teach
+Claude (or any other agent runtime that reads `.claude/skills/`) the
+post-streams-redesign config schema, the `@stream` decorator pattern, and
+the debugging playbook. Install them into a dtex project with:
+
+```sh
+dtex skills install         # copies into ./.claude/skills/dtex/
+```
+
+The first `dtex` command run inside a project that lacks installed skills
+prints a one-line hint (suppressed thereafter). Run `dtex skills list` to
+see which ship with this dtex version and which are installed locally.
+
+If you're not using Claude, just ignore the hint — the skills are inert
+markdown until installed.
+
 ## Documentation
 
 The full design handbook lives in [`docs/`](https://github.com/vej-ai/dtex/tree/main/docs).

{dtex-0.2.0 → dtex-0.2.2}/docs/03-connector-contract.md

@@ -420,6 +420,7 @@ objects the function asks for. A function that needs nothing extra can be
 | `state` | `State` | Per-stream persisted state (chapter 04). Free-form key/value scratch space that survives between runs. |
 | `cursor` | `Cursor` | Incremental cursor helper, present only if the stream declares an `incremental` block. See §3.2. |
 | `log` | `Logger` | Structured logger. `log.info(...)`, `log.warning(...)`. Correlation ids attached by the engine. |
+| `stream_def` | `StreamDef` | The parsed `register.yaml` entry for this stream. Lets a connector introspect its own declaration — e.g. the baked `stripe` connector reads `stream_def.sigma.query` to decide which SQL file to submit for Sigma-backed streams. Use when one connector serves more than one extraction surface and the surface is declared in the manifest. |
 
 #### What a `@stream` function yields
 

{dtex-0.2.0 → dtex-0.2.2}/dtex/__init__.py

@@ -70,9 +70,11 @@ from dtex.types import (
     RunStatus,
     Schema,
     SchemaContract,
+    SigmaConfig,
     State,
     StateBackend,
     StateRecord,
+    StreamDef,
     StreamMeta,
     StreamMode,
     StreamResult,
@@ -136,9 +138,11 @@ __all__ = [
     "RunStatus",
     "Schema",
     "SchemaContract",
+    "SigmaConfig",
     "State",
     "StateBackend",
     "StateRecord",
+    "StreamDef",
     "StreamMode",
     "StreamRunConfig",
     "StreamMeta",

{dtex-0.2.0 → dtex-0.2.2}/dtex/destinations/bigquery/client.py

@@ -139,6 +139,56 @@ def _status_code(exc: BaseException) -> int | None:
     return None
 
 
+def _is_retryable_network_error(exc: BaseException) -> bool:
+    """Whether ``exc`` is a transient network-layer failure worth retrying.
+
+    Long-running runs (a backfill against a high-row-count source like
+    RevenueCat that involves hundreds of sequential LOAD jobs) routinely
+    hit stale-TCP-socket failures: a keep-alive connection held open by
+    the google-cloud-bigquery client gets killed by an intermediary,
+    and the next reuse raises a ``requests.exceptions.ConnectionError``
+    (typically wrapping ``http.client.RemoteDisconnected``). These have
+    NO HTTP status code (the connection died before any response
+    arrived), so the status-code path in :func:`run_with_retries` would
+    re-raise them immediately. This helper recognises them so the
+    retry loop covers them.
+
+    Genuine programming errors (``KeyError``, ``TypeError``, etc.) are
+    NOT matched — they should surface immediately, not get masked under
+    a few backoff cycles. The match is by concrete network-exception
+    type, not "any statusless exception."
+    """
+    # requests-level connection/timeout failures (ConnectionError /
+    # Timeout / ConnectTimeout / ReadTimeout / ChunkedEncodingError /
+    # ProxyError / SSLError — all RequestException subclasses).
+    import requests  # noqa: PLC0415 — narrow runtime import
+
+    if isinstance(
+        exc,
+        (
+            requests.exceptions.ConnectionError,
+            requests.exceptions.Timeout,
+            requests.exceptions.ChunkedEncodingError,
+        ),
+    ):
+        return True
+
+    # google-api-core wraps some of the above under its own
+    # ServiceUnavailable / RetryError. These usually DO carry a 503
+    # status code so the existing path catches them, but RetryError
+    # (raised when google's own internal retry exhausts) sometimes
+    # surfaces without one.
+    try:
+        from google.api_core import exceptions as gapic_exc  # noqa: PLC0415
+
+        if isinstance(exc, gapic_exc.RetryError):
+            return True
+    except ImportError:  # pragma: no cover — google-api-core is installed
+        pass
+
+    return False
+
+
 def run_with_retries(
     operation: Any,
     *,
@@ -146,14 +196,16 @@ def run_with_retries(
     backoff_seconds: float,
     sleep: Any = time.sleep,
 ) -> Any:
-    """Run ``operation()`` with exponential backoff on transient BigQuery errors.
+    """Run ``operation()`` with exponential backoff on transient failures.
 
     ``operation`` is a zero-arg callable — typically a ``lambda: job.result(
     timeout=...)`` or a ``lambda: client.load_table_from_uri(...).result(
-    timeout=...)``. A retryable failure (status in :data:`_RETRYABLE_STATUS`)
-    sleeps ``backoff_seconds * 2**attempt`` and retries; a non-retryable
-    failure re-raises immediately. The ``sleep`` parameter is injectable so
-    unit tests can avoid real wall-clock delays.
+    timeout=...)``. A retryable failure (status in :data:`_RETRYABLE_STATUS`
+    OR a connection-class network error matched by
+    :func:`_is_retryable_network_error`) sleeps ``backoff_seconds * 2**attempt``
+    and retries; a non-retryable failure re-raises immediately. The
+    ``sleep`` parameter is injectable so unit tests can avoid real
+    wall-clock delays.
     """
     if max_attempts < 1:
         max_attempts = 1
@@ -163,16 +215,23 @@ def run_with_retries(
             return operation()
         except Exception as exc:  # noqa: BLE001 — re-raises non-retryable below
             status = _status_code(exc)
-            if status is None or status not in _RETRYABLE_STATUS:
+            retryable_by_status = status is not None and status in _RETRYABLE_STATUS
+            retryable_by_network = _is_retryable_network_error(exc)
+            if not (retryable_by_status or retryable_by_network):
                 raise
             last_exc = exc
             if attempt == max_attempts - 1:
                 raise
             delay = backoff_seconds * (2 ** attempt)
+            reason = (
+                f"status={status}"
+                if retryable_by_status
+                else f"network={type(exc).__name__}"
+            )
             logger.warning(
-                "BigQuery transient error (status=%s, attempt=%d/%d): %s — "
+                "BigQuery transient error (%s, attempt=%d/%d): %s — "
                 "retrying in %.1fs",
-                status,
+                reason,
                 attempt + 1,
                 max_attempts,
                 exc,

{dtex-0.2.0 → dtex-0.2.2}/dtex/engine/runner.py

@@ -585,6 +585,7 @@ def _run_one_stream(
         "config": stream_config,
         "state": state,
         "log": log,
+        "stream_def": stream_def,
     }
     if cursor is not None:
         available["cursor"] = cursor

{dtex-0.2.0 → dtex-0.2.2}/dtex/registry.py

@@ -102,12 +102,23 @@ __all__ = [
 # Injection — the parameter names the engine may inject (docs/03 §3.1)
 # ---------------------------------------------------------------------------
 
-STREAM_INJECTABLES: frozenset[str] = frozenset({"config", "state", "cursor", "log"})
+STREAM_INJECTABLES: frozenset[str] = frozenset(
+    {"config", "state", "cursor", "log", "stream_def"}
+)
 """The parameter names the engine injects into a ``@stream`` function — docs/03 §3.1.
 
 The engine inspects the function signature and injects, *by name*, only the
 objects the function declares. A ``@stream`` function may declare any subset of
-these four; declaring anything else is a discovery-time error (docs/03 §7 rule 8).
+these; declaring anything else is a discovery-time error (docs/03 §7 rule 8).
+
+* ``config`` — resolved :class:`dtex.Config` (params + secrets, per-stream layered).
+* ``state`` — the per-stream state blob (a :class:`dtex.State` proxy).
+* ``cursor`` — the per-stream :class:`dtex.Cursor` (incremental streams only).
+* ``log`` — a standard logging-style logger pre-configured with run-correlation.
+* ``stream_def`` — the parsed :class:`dtex.StreamDef` declaration for this
+  stream (its own ``register.yaml`` entry). Lets a connector introspect its
+  own metadata — e.g. the ``stripe`` connector dispatches on
+  ``stream_def.sigma`` to choose REST-vs-Sigma extraction per stream.
 """
 
 # NOTE: docs/03 §4 says a ``@stream_method`` follows "the same injection rules

{dtex-0.2.0 → dtex-0.2.2}/dtex/skills/dtex-write-connector.md

@@ -131,8 +131,8 @@ def customers(config: Config, state: State, cursor: Cursor, log) -> Iterator[Bat
 ### Injected arguments
 
 The engine declares which arguments it can inject — `config`, `state`,
-`cursor`, `log` — and supplies only the ones the function names. List
-only what you use; the engine matches by name.
+`cursor`, `log`, `stream_def` — and supplies only the ones the function
+names. List only what you use; the engine matches by name.
 
 - **`config`** — the resolved `Config` for this run. Read params via
   `config.get("key")` or attribute access. Read secrets via
@@ -143,6 +143,12 @@ only what you use; the engine matches by name.
   that need to remember more than a cursor.
 - **`log`** — a standard logging-style logger. `log.info(...)`,
   `log.warning(...)`.
+- **`stream_def`** — the parsed `StreamDef` for this stream — its own
+  `register.yaml` entry. Lets a connector introspect its own
+  declaration. Useful when one connector serves more than one
+  extraction surface (REST + Sigma SQL, GA + Reporting API, etc.) and
+  the surface choice is declared in the manifest. See "Dual-API
+  connectors" below.
 
 ### Three rules to drill in
 
@@ -162,6 +168,72 @@ but it does not reshape nested dicts into flat columns. If the API returns
 `{"attributes": {"country": "US"}}` and your schema declares
 `country: STRING`, you must flatten in the `@stream` function before yielding.
 
+## Dual-API connectors — when one source serves two surfaces
+
+Some APIs come in two flavors that the same operator pulls from in one
+pipeline. Stripe is the canonical example: a standard REST API
+(resource-as-stream, GA) AND a Sigma SQL API (query-as-stream, paid,
+~3-hour lag). Both belong to one "Stripe" connector, not two.
+
+The dtex pattern is **one source folder, two extraction surfaces, per-stream
+opt-in**. The baked `stripe` connector demonstrates this:
+
+```yaml
+# dtex/sources/stripe/register.yaml — abbreviated
+streams:
+  - name: charges            # REST — no `sigma:` block
+    table: stripe_charges
+    incremental: {cursor_field: created, cursor_type: int}
+    schema: [...]
+
+  - name: charges_daily      # Sigma — opted in via `sigma:` block
+    table: charges
+    sigma:
+      query: queries/charges_daily.sql
+    incremental: {cursor_field: created, cursor_type: timestamp}
+    schema: [...]
+```
+
+In `source.py`, each `@stream` function knows which surface it belongs
+to by which helper it calls. The Sigma functions take the `stream_def`
+injectable, read `stream_def.sigma.query`, and load the SQL from the
+named file:
+
+```python
+@stream(name="charges_daily")
+def charges_daily(
+    stream_def: StreamDef, config: Config, cursor: Cursor, log
+) -> Iterator[Batch]:
+    yield from _extract_sigma_stream(stream_def, config, cursor, log)
+
+# _extract_sigma_stream(stream_def, ...) reads stream_def.sigma.query
+# from the connector folder, binds {since} from the cursor, submits to
+# Sigma, downloads CSV, yields batches.
+```
+
+The decision rules:
+
+* **One connector, not two.** Users binding `source: stripe` see one
+  surface; the `sigma:` block is the opt-in marker per stream. Don't
+  fork into `stripe` and `stripe_sigma`.
+* **Sigma streams declare `sigma: {query: <path>}`.** Path is relative
+  to the connector folder. The `@stream` function reads it from
+  `stream_def.sigma.query` (via the `stream_def` injectable) and
+  loads the SQL.
+* **REST streams don't change.** No `sigma:` block, no `stream_def`
+  arg required — existing connectors stay shaped exactly as they were.
+* **Shared client params, shared auth.** A single Stripe restricted
+  key with both REST + Sigma scopes drives both surfaces from one
+  `api_key` secret. Keep it that way; don't introduce per-surface
+  secrets unless the API genuinely demands separate keys.
+
+The same pattern fits other APIs with a "live REST + analytical SQL"
+shape — Shopify GraphQL + Shopify Bulk Operations, Salesforce REST +
+SOQL, GA Data API + BigQuery export, etc. The `sigma:` block is
+Stripe-specific naming today; future dual-surface connectors should
+either reuse it (if they have similar SQL semantics) or add a parallel
+marker (`bulk:`, `soql:`) following the same opt-in shape.
+
 ## The `client.py` pattern
 
 ```python

dtex-0.2.2/dtex/sources/stripe/README.md

@@ -0,0 +1,188 @@
+# Stripe — baked source connector
+
+One Stripe connector, **two extraction surfaces**:
+
+* **REST** (default) — resource-as-stream over Stripe's GA REST API at
+  `https://api.stripe.com/v1`. One stream per resource type
+  (`charges`, `invoices`, `customers`, `subscriptions`), incremental on
+  the Unix `created` timestamp, paginated via Stripe's cursor model
+  (`starting_after` + `has_more`). Cheap, near-live, no extra
+  subscription.
+* **Sigma SQL** (opt-in per stream) — query-as-stream over Stripe's
+  Sigma API. Each Sigma stream's `register.yaml` entry carries a
+  `sigma: {query: queries/<name>.sql}` block; the connector reads the
+  SQL body, submits it via `POST /v2/data/reporting/query_runs`, polls
+  to completion, downloads the CSV result, and yields batches. Lets a
+  single stream JOIN/aggregate across Stripe's internal tables in
+  Stripe-side compute. Requires a **paid Sigma subscription** and
+  carries a **~3-hour data lag**.
+
+The two surfaces share one source folder, one set of connector params,
+and one `api_key` — a single Stripe restricted key with the relevant
+read scopes drives both. Configs mix REST and Sigma streams freely.
+
+## Authentication
+
+The connector reads a **restricted Stripe secret key** with **read-only
+scopes** from the `STRIPE_API_KEY` environment variable:
+
+```sh
+export STRIPE_API_KEY="rk_live_..."
+```
+
+Recommended restricted-key scopes:
+
+| Surface | Resource | Scope |
+|---|---|---|
+| REST | `/charges` | `charge:read` |
+| REST | `/invoices` | `invoice:read` |
+| REST | `/customers` | `customer:read` |
+| REST | `/subscriptions` | `subscription:read` |
+| Sigma | `POST /v2/data/reporting/query_runs` | `sigma:read`, `reporting:read` |
+
+If your config only uses REST streams you don't need the Sigma scopes,
+and vice versa.
+
+The key is sent as `Authorization: Bearer <key>` on every request. It is
+**never logged** — the `Authorization` header is redacted at each client's
+single logging choke point.
+
+## Supported resources (v1)
+
+Four streams, all `write_disposition: merge` on `primary_key: id`:
+
+- **`charges`** → `stripe_charges`
+- **`invoices`** → `stripe_invoices`
+- **`customers`** → `stripe_customers`
+- **`subscriptions`** → `stripe_subscriptions`
+
+Each declares an explicit schema covering the common columns (`id`, `object`,
+`created`, `livemode`, …) plus the resource-specific scalars. Nested
+sub-objects (e.g. `payment_method_details`, `lines`, `items`, `metadata`)
+land as `JSON` columns — the dtex convention for nested data
+(docs/04 §"Nested data").
+
+## Incremental model
+
+Every stream uses:
+
+```yaml
+incremental:
+  cursor_field: created
+  cursor_type: int          # Stripe `created` is a Unix-timestamp integer
+  lookback: 6h              # declared hint
+  initial_value: "1704067200"   # 2024-01-01 UTC
+```
+
+On each run the connector translates the engine's `cursor.start_value()` into
+a `created[gte]=<ts>` query parameter, walks every page with
+`starting_after=<last_id>`, and `cursor.observe()`s each record so the engine
+can advance the cursor.
+
+Stripe lists are returned **newest first** within the new-records window;
+`cursor.observe()` takes the *max*, so the final cursor is correct regardless
+of arrival order. Per dtex's commit model the cursor is committed only
+after every batch durably lands, so a mid-stream crash safely re-runs the
+whole window — no lost or skipped rows.
+
+`merge` write disposition means re-fetched objects (an invoice whose status
+changed after it was first loaded) upsert in place, so a periodic
+`lookback`-style re-fetch is idempotent.
+
+## Usage
+
+`register.yaml` declares the connector; the project binds it to a
+destination via its `destination:` block or via the project-level
+`default_destination`. Running it is the standard dtex one-liner:
+
+```python
+import dtex
+
+result = dtex.run(
+    connector="stripe",
+    target="prod",
+    # Run a single stream:
+    select=("charges",),
+    # Or override params per invocation:
+    params={"page_size": 100, "requests_per_second": 50},
+)
+print(result.status, result.rows_loaded)
+```
+
+CLI equivalent:
+
+```sh
+dtex run stripe --target prod --select charges
+```
+
+## Operational notes
+
+- **Rate limit.** Default `requests_per_second: 25` is conservative against
+  Stripe's documented ~100/sec live-mode limit. Tune up via `params` if you
+  control the account's throttling budget.
+- **Retries.** `429` honors `Retry-After` exactly; `5xx` retries with
+  exponential backoff up to `max_retries: 5`; other `4xx` (`401`, `403`,
+  `404`) raise immediately — a bad request will not get better on its own.
+- **Dependencies.** `requests>=2.31` (declared in `pyproject.toml`); no
+  Stripe SDK — raw HTTP keeps the dependency surface minimal and avoids
+  version coupling to the SDK's release cadence.
+- **API version pin.** `api_version` defaults to `2024-12-18.acacia`; see
+  [docs/connectors/stripe-research.md](../../../docs/connectors/stripe-research.md)
+  for the source of that pin and where to check for the current GA.
+
+## Sigma SQL streams
+
+Three Sigma streams ship in `register.yaml`:
+
+- **`charges_daily`** → `charges` (incremental on `created`)
+- **`subscriptions_active`** → `subscriptions` (full refresh)
+- **`invoices_paid`** → `invoices` (incremental on `status_transitions_paid_at`)
+
+The SQL bodies live under `queries/<stream_name>.sql`. The connector binds
+the `{since}` placeholder from each stream's `Cursor.start_value()` (or
+the connector-level `sigma_initial_since` for non-incremental streams) as a
+Presto `TIMESTAMP 'YYYY-MM-DD HH:MM:SS'` literal — Sigma rejects ISO-8601's
+`T` separator and timezone suffixes.
+
+### Adding a 4th Sigma stream
+
+Two files + one function:
+
+1. Drop your query at `queries/<new_stream>.sql`. Use `{since}` if it's
+   incremental.
+2. Append a stream to `register.yaml` declaring the schema and the
+   `sigma:` block:
+   ```yaml
+   - name: my_new_stream
+     table: my_new_table
+     primary_key: id
+     write_disposition: merge
+     sigma:
+       query: queries/my_new_stream.sql
+     incremental:
+       cursor_field: <col>
+       cursor_type: timestamp
+       initial_value: "2024-01-01T00:00:00Z"
+     schema:
+       - {name: id,        type: STRING, mode: REQUIRED}
+       - {name: <col>,     type: TIMESTAMP, mode: REQUIRED}
+       - ...
+   ```
+3. Add a one-line `@stream` wrapper in `source.py` that delegates to
+   `_extract_sigma_stream` (the existing functions show the pattern).
+
+### Operational notes (Sigma-specific)
+
+- **Account ID required.** The config must set `params.account_id` to the
+  `acct_...` ID of the Stripe account being queried. Stripe rejects every
+  Sigma request with HTTP 403 otherwise.
+- **API version pin.** `sigma_api_version` defaults to
+  `2026-04-22.preview`; the Query Run API is preview. Bump deliberately.
+- **Polling.** `sigma_poll_interval_seconds` (default `2.0`) is the gap
+  between status polls; `sigma_poll_timeout_seconds` (default `600`) is the
+  hard ceiling per query. Heavy queries on large accounts approach the
+  ceiling; bump if needed.
+- **Download retries.** A truncated CSV mid-body retries the whole download
+  up to `max_retries` (default `5`) with exponential backoff
+  (`retry_backoff_seconds`, default `1.0`). Duplicate-free because no rows
+  are yielded until the full CSV is in hand.

dtex-0.2.2/dtex/sources/stripe/queries/charges_daily.sql

@@ -0,0 +1,24 @@
+-- charges_daily — every Charge created after the cursor floor.
+--
+-- Sigma's dialect is PrestoDB-flavored. The {since} placeholder is
+-- substituted by source.py at run time from the engine's Cursor; it
+-- arrives here as an ISO-8601 string and is compared against `created`
+-- (a Sigma TIMESTAMP column).
+--
+-- Schema is mirrored in register.yaml. If you add a column here, add
+-- it there (and bump the Stripe Sigma API version pin if the new
+-- column is preview-only).
+
+SELECT
+    id,
+    amount,
+    currency,
+    created,
+    status,
+    customer_id,
+    description,
+    paid,
+    refunded
+FROM charges
+WHERE created > TIMESTAMP {since}
+ORDER BY created

dtex-0.2.2/dtex/sources/stripe/queries/invoices_paid.sql

@@ -0,0 +1,21 @@
+-- invoices_paid — paid invoices since the cursor floor.
+--
+-- Sigma's invoices table tracks lifecycle transitions in the
+-- `status_transitions_*` columns. `status_transitions_paid_at` is the
+-- timestamp the invoice transitioned to paid; that's the right
+-- incremental cursor (`created` or `date` would re-load an invoice
+-- whose payment processed long after it was issued).
+
+SELECT
+    id,
+    customer_id,
+    subscription_id,
+    total,
+    amount_paid,
+    currency,
+    status_transitions_paid_at,
+    status,
+    number
+FROM invoices
+WHERE status_transitions_paid_at > TIMESTAMP {since}
+ORDER BY status_transitions_paid_at

dtex-0.2.2/dtex/sources/stripe/queries/subscriptions_active.sql

@@ -0,0 +1,17 @@
+-- subscriptions_active — currently-active subscriptions, full snapshot.
+--
+-- No {since} placeholder: this stream is full-refresh (`write_disposition:
+-- replace` in register.yaml). Sigma's `subscriptions` table is the
+-- authoritative current state of every subscription.
+
+SELECT
+    id,
+    customer_id,
+    status,
+    current_period_start,
+    current_period_end,
+    cancel_at_period_end,
+    created
+FROM subscriptions
+WHERE status = 'active'
+ORDER BY created