dtex 0.2.2__tar.gz → 0.2.4__tar.gz

{dtex-0.2.2 → dtex-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dtex
-Version: 0.2.2
+Version: 0.2.4
 Summary: dtex (data extraction tool) — an open-source Python EL tool: pipelines as configs, connectors as folders, CLI-first.
 Project-URL: Homepage, https://github.com/vej-ai/dtex
 Project-URL: Documentation, https://github.com/vej-ai/dtex/tree/main/docs
@@ -101,14 +101,26 @@ params. Run it with `dtex run -p <config>`. The library equivalent is
 
 ## Pre-baked connectors
 
-**Sources:** `filesystem` (CSV/JSONL/Parquet from local, GCS, or S3),
-`rest` (paginated REST APIs — 4 pagination strategies, 4 auth modes),
-`postgres` (keyset pagination, no `OFFSET`), `shiphero` (GraphQL),
-`stripe` (REST resource-as-stream + opt-in Sigma SQL-as-stream).
+Each connector ships with its own README — click through for the
+auth checklist, scope requirements, schema, and known limitations.
 
-**Destinations:** `duckdb` (zero-config dev default, all 5 capabilities) and
-`bigquery` (production warehouse — Parquet-staged via GCS + LOAD jobs,
-MERGE upserts, cursor-based partitioning).
+**Sources:**
+
+| Connector | What it does |
+|---|---|
+| [`filesystem`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/filesystem/README.md) | CSV / JSONL / Parquet from local, GCS, or S3 |
+| [`rest`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/rest/README.md) | Paginated REST APIs — 4 pagination strategies, 4 auth modes |
+| [`postgres`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/postgres/README.md) | Keyset pagination, no `OFFSET` |
+| [`shiphero`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/shiphero/README.md) | GraphQL |
+| [`stripe`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/stripe/README.md) | REST resource-as-stream + opt-in Sigma SQL-as-stream |
+| [`revenuecat`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/revenuecat/README.md) | v2 API — customers + subscriptions + daily chart metrics |
+
+**Destinations:**
+
+| Connector | What it does |
+|---|---|
+| [`duckdb`](https://github.com/vej-ai/dtex/blob/main/dtex/destinations/duckdb/README.md) | Zero-config dev default, all 5 capabilities |
+| [`bigquery`](https://github.com/vej-ai/dtex/blob/main/dtex/destinations/bigquery/README.md) | Production warehouse — Parquet-staged via GCS + LOAD jobs, MERGE upserts, cursor-based partitioning |
 
 **Engine:** per-stream commit + atomic transactions (rollback on failure),
 state in the destination's `_dtex_state` table, run records in `_dtex_runs`,

{dtex-0.2.2 → dtex-0.2.4}/README.md

@@ -51,14 +51,26 @@ params. Run it with `dtex run -p <config>`. The library equivalent is
 
 ## Pre-baked connectors
 
-**Sources:** `filesystem` (CSV/JSONL/Parquet from local, GCS, or S3),
-`rest` (paginated REST APIs — 4 pagination strategies, 4 auth modes),
-`postgres` (keyset pagination, no `OFFSET`), `shiphero` (GraphQL),
-`stripe` (REST resource-as-stream + opt-in Sigma SQL-as-stream).
-
-**Destinations:** `duckdb` (zero-config dev default, all 5 capabilities) and
-`bigquery` (production warehouse — Parquet-staged via GCS + LOAD jobs,
-MERGE upserts, cursor-based partitioning).
+Each connector ships with its own README — click through for the
+auth checklist, scope requirements, schema, and known limitations.
+
+**Sources:**
+
+| Connector | What it does |
+|---|---|
+| [`filesystem`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/filesystem/README.md) | CSV / JSONL / Parquet from local, GCS, or S3 |
+| [`rest`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/rest/README.md) | Paginated REST APIs — 4 pagination strategies, 4 auth modes |
+| [`postgres`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/postgres/README.md) | Keyset pagination, no `OFFSET` |
+| [`shiphero`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/shiphero/README.md) | GraphQL |
+| [`stripe`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/stripe/README.md) | REST resource-as-stream + opt-in Sigma SQL-as-stream |
+| [`revenuecat`](https://github.com/vej-ai/dtex/blob/main/dtex/sources/revenuecat/README.md) | v2 API — customers + subscriptions + daily chart metrics |
+
+**Destinations:**
+
+| Connector | What it does |
+|---|---|
+| [`duckdb`](https://github.com/vej-ai/dtex/blob/main/dtex/destinations/duckdb/README.md) | Zero-config dev default, all 5 capabilities |
+| [`bigquery`](https://github.com/vej-ai/dtex/blob/main/dtex/destinations/bigquery/README.md) | Production warehouse — Parquet-staged via GCS + LOAD jobs, MERGE upserts, cursor-based partitioning |
 
 **Engine:** per-stream commit + atomic transactions (rollback on failure),
 state in the destination's `_dtex_state` table, run records in `_dtex_runs`,

dtex-0.2.4/dtex/destinations/duckdb/README.md

@@ -0,0 +1,98 @@
+# DuckDB destination — dtex v1
+
+The **zero-config dev default**. Writes batches to a local DuckDB file
+(`append` / `merge` / `replace`) — no external service to set up, no
+credentials, no network. Tier A: DuckDB hosts its own state and audit
+tables (`_dtex_state`, `_dtex_runs`) alongside the loaded data, so a
+fresh checkout of a project resumes correctly with no extra files.
+
+See [docs/05 — Destinations & State](../../../docs/05-destinations-and-state.md)
+for the destination contract; this README is the operator's quick reference.
+
+## Install
+
+```bash
+pip install dtex
+```
+
+DuckDB ships in dtex's base dependencies — no extras needed.
+
+## Config
+
+Default profile is fine for most projects:
+
+```yaml
+# profiles.yml
+duckdb:
+  default_target: dev
+  targets:
+    dev:
+      path: ".dtex/warehouse.duckdb"     # the default — gitignored by `dtex init`
+    prod:
+      path: "/var/data/dtex/warehouse.duckdb"
+```
+
+The `.duckdb` file is created on first run. `dtex init` scaffolds the
+above + a `.gitignore` rule for `.dtex/`. For ephemeral / test runs use
+`path: ":memory:"`.
+
+### Per-pipeline schema override
+
+A config can land each pipeline's tables under a DuckDB schema by setting
+`destination_params.dataset`:
+
+```yaml
+# configs/my_pipeline.yml
+destination_params:
+  dataset: my_pipeline     # creates schema if absent; all tables land here
+```
+
+Without `dataset` the destination uses DuckDB's default schema.
+
+## Capabilities
+
+DuckDB implements all five destination capabilities:
+
+| Capability | What it does |
+|---|---|
+| `append` | INSERT rows as-is. |
+| `merge` | UPSERT on `primary_key` (requires non-empty PK). |
+| `replace` | TRUNCATE then INSERT (full-refresh streams). |
+| `state` | Stores `_dtex_state` rows for cursor resume. |
+| `run_records` | Writes `_dtex_runs` per pipeline invocation. |
+
+## State + run records
+
+Two tables live alongside the loaded data:
+
+- **`_dtex_state`** — one row per (connector, stream) with the committed
+  cursor value, last run id, and cumulative `rows_total`. Read at every
+  run to resume incremental streams.
+- **`_dtex_runs`** — one row per pipeline invocation with start/end time,
+  status, rows loaded, and (for failures) the error type + message.
+  Query history with `dtex runs list` or directly.
+
+## Operational notes
+
+- **Concurrent writes are NOT supported.** DuckDB locks the database file
+  for the duration of a write transaction; two `dtex run` processes
+  targeting the same file will serialize at best, error at worst. For
+  production multi-pipeline use, target BigQuery instead.
+- **Single-machine.** A DuckDB file lives on one disk. dtex doesn't
+  replicate it; if you want a queryable copy on another box, copy the
+  file or switch to BigQuery.
+- **Memory.** DuckDB streams query results, so MERGE operations work fine
+  on large tables — but VERY large in-memory ANALYZE / aggregation jobs
+  can OOM. Adjust `PRAGMA memory_limit` via a connection callback if you
+  hit it.
+
+## What's not in v1
+
+- No clustering / partitioning at the storage layer (DuckDB doesn't
+  expose physical partitioning the way warehouses do; `partition_by`
+  declarations on streams are recorded in metadata but don't influence
+  the physical layout). Use BigQuery if cursor-based partitioning is a
+  hard requirement.
+- No cross-database joins from the warehouse file (a pipeline can read
+  via the `filesystem` source's DuckDB-backed Parquet paths, but the
+  destination itself doesn't host federation).

dtex-0.2.4/dtex/sources/revenuecat/README.md

@@ -0,0 +1,160 @@
+# RevenueCat — baked source connector
+
+Extracts data from [RevenueCat](https://www.revenuecat.com)'s v2 API at
+`https://api.revenuecat.com/v2`. Three streams covering RC's three
+distinct extraction surfaces:
+
+* **`customers`** — every customer in a project. NON-incremental.
+* **`subscriptions`** — per-customer subscriptions. NON-incremental,
+  fan-out across the customer list.
+* **`metrics_daily`** — daily chart metrics (revenue, MRR, churn, etc.).
+  REAL incremental on cohort date, long-format output.
+
+The connector targets v2 specifically. v2 keys are **separate** from v1
+keys and the two APIs are not interchangeable.
+
+## Authentication
+
+The connector reads a **v2 RevenueCat secret key** (`sk_...`) from the
+`REVENUECAT_API_KEY` environment variable by default:
+
+```sh
+export REVENUECAT_API_KEY="sk_..."
+```
+
+Required scopes (set on the key in the RC dashboard):
+
+| Stream | Scope |
+|---|---|
+| `customers` | `customer_information:customers:read` |
+| `subscriptions` | `customer_information:subscriptions:read` |
+| `metrics_daily` | `charts_metrics:overview:read` |
+
+For production, override the secret ref per profile (`profiles.yml`)
+with any resolver-backed `secret://` URL — GCP Secret Manager, AWS
+Secrets Manager, or HashiCorp Vault:
+
+```yaml
+# profiles.yml
+revenuecat:
+  default_target: prod
+  targets:
+    prod:
+      api_key: secret://gcp-secret-manager/projects/<proj>/secrets/<name>/versions/latest
+```
+
+(Requires the matching extra: `pip install 'dtex[gcp-secrets]'` /
+`[aws-secrets]` / `[vault]`.)
+
+The key never appears in log output — it's set on the `requests.Session`
+header once and the connector's logging emits no header contents.
+
+## The three streams
+
+### `customers` — non-incremental
+
+RC v2's `/customers` endpoint has **NO server-side date filter**
+(verified against the docs + the airbyte issue 70315 + the RC community
+forum, June 2026). Every run paginates the full customer list. The
+prior v1 design that filtered client-side had a subtle data-loss bug
+because RC does not guarantee response ordering; the baked connector
+deliberately drops the `incremental:` block to avoid the same trap.
+`write_disposition: merge` on `id` makes re-pulls upsert idempotently;
+the cost is HTTP calls, not duplicate rows.
+
+For a 195k-customer account, expect ~70 minutes wall time at the
+default page_size of 100 (RC's customer-domain rate limit is 480
+req/min; the practical floor is ~3 min/100k rows).
+
+### `subscriptions` — non-incremental, per-customer fan-out
+
+RC has no project-level `/subscriptions` endpoint, only
+`/customers/{id}/subscriptions`. This stream iterates the customer list
+and fetches per-customer subscriptions. Operationally expensive:
+O(N+1) HTTP calls per run for N customers. For Sintra-scale accounts
+this is the heaviest stream by far.
+
+### `metrics_daily` — real incremental
+
+RC v2's `/charts/{chart_name}` endpoint DOES take server-side
+`start_date`/`end_date` filters and tags each per-day value with
+`incomplete=true` when the day is still finalizing. This stream:
+
+1. Computes `start_date = max(cursor.start_value(), today - lookback_days)`.
+2. For each chart in `metrics_charts`, fetches the date range with
+   `resolution=day`.
+3. Flattens the response's `measures[]` × `values[]` arrays into
+   long format: one row per `(cohort_date, chart_name, measure_name)`.
+4. Observes the cursor only against rows where `incomplete=false`, so
+   today's still-incomplete value gets re-pulled on the next run with
+   the corrected (final) value.
+
+Default charts: `revenue`, `mrr`, `actives`, `trials`. Override via
+the `metrics_charts` param (comma-separated). Adding a new chart is
+zero schema migration — it lands as new rows in the same long-format
+table.
+
+The `cohort_explorer` and `prediction_explorer` charts return a
+different shape and would break the flattener; exclude them unless
+you fork the connector.
+
+## Config
+
+A minimum config looks like this:
+
+```yaml
+# configs/revenuecat_bq.yml
+name: revenuecat_bq
+source: revenuecat
+destination: bigquery
+target: prod
+params:
+  project_id: "proj1ab2c3d4"          # required, no default
+destination_params:
+  dataset: revenuecat
+streams: all                          # or list specific streams
+```
+
+To narrow to just the cheap stream:
+
+```yaml
+streams:
+  metrics_daily:
+```
+
+## Rate limits and timeouts
+
+* `/customers` and `/subscriptions` share the **Customer Information**
+  domain with a 480 req/min default cap.
+* `/charts/*` is in the **Charts & Metrics** domain with a 15 req/min
+  cap — much tighter; 4 charts × ~7 days per run hits this fine.
+* Client uses a `(10s connect, 60s read)` timeout to prevent the
+  stale-socket hangs the prior version exhibited.
+* Retries cover: HTTP 5xx (capped, exponential backoff), HTTP 429
+  (honors `Retry-After`, also capped), and the network-exception
+  family (Timeout, ConnectionError, ChunkedEncodingError). All capped
+  by `max_retries`; the prior version had two uncapped retry paths
+  that wedged on persistent rate-limits or dead sockets.
+
+## Operational notes
+
+- **First run on a populated account is long.** Plan for ~70 min wall
+  time on ~200k-customer accounts. Subsequent runs are not faster
+  unless you switch to `metrics_daily` only.
+- **No incremental shortcut exists for customers.** RC's official
+  guidance for "I need fresh data" is their S3/GCS export (a separate
+  paid feature). The connector documents this honestly rather than
+  pretending a client-side filter is incremental.
+- **Long backfills need dtex 0.2.2+.** Earlier dtex BigQuery destinations
+  could fail mid-run on a stale TCP socket; 0.2.2 fixed it.
+
+## What's not in v1 of this connector
+
+- `cohort_explorer` and `prediction_explorer` chart types (different
+  response shape).
+- Webhooks (no v2 endpoint at the time of writing).
+- Transactions / activities / payments lists (RC doesn't expose them
+  at the project level in v2 yet; staff has the feature request
+  open).
+- `non-subscription_purchases` chart (currently excluded from the
+  defaults but should work — try it).

dtex-0.2.4/dtex/sources/revenuecat/__init__.py

@@ -0,0 +1 @@
+# Marker file — makes this folder a Python package so source.py / destination.py can use relative imports for sibling helpers (e.g. `from .client import X`).

dtex-0.2.4/dtex/sources/revenuecat/client.py

@@ -0,0 +1,126 @@
+"""RevenueCat v2 HTTP client — thin wrapper over `requests`.
+
+Two surfaces:
+
+* `paginate(path, params)` — for list endpoints (/customers, per-customer
+  /subscriptions). RC paginates with `starting_after` cursor tokens; the
+  response's `next_page` field is the absolute URL of the next page (with
+  the cursor + original query params baked in), so subsequent calls pass
+  NO params — using the URL as-is is correct per docs.
+* `get(path, params)` — for one-shot endpoints (/charts/{chart_name},
+  /metrics/*). Returns parsed JSON.
+
+Both surfaces share retry + rate-limit handling. Three failure classes
+get bounded, capped retry — the previous version had two real hangs:
+
+* **No socket timeout.** Default `requests.get` waits forever for both
+  connect and read. One dead RC connection mid-walk → infinite sleep
+  inside `_session.get`. Fixed with `timeout=(connect, read)`.
+* **Network errors weren't caught.** `Timeout`, `ConnectionError`,
+  `ChunkedEncodingError`, `ProtocolError` all raise BEFORE a `resp`
+  object exists, so neither status-code branch saw them — they
+  propagated up and failed the stream on the first blip.
+* **429 retry was uncapped + did not increment `_attempt`.** A
+  sustained rate-limit (very real: ~480 req/min on the customer domain
+  vs. tens of thousands of customer pages per run) wedged the run in a
+  permanent sleep-retry loop. Now bounded by `max_retries`, same as
+  the other retry paths.
+
+The Bearer token is set on the session header once; it never appears
+in log output or error messages.
+"""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass, field
+from typing import Any, Iterator
+
+import requests
+
+# (connect_timeout, read_timeout) seconds. The connect leg should be fast
+# — RC's edge is near every common region. The read leg is the dangerous
+# one: a single hung response will block the whole run. 60s is generous
+# enough for slow chart computations without trapping a dead socket.
+_TIMEOUT: tuple[float, float] = (10.0, 60.0)
+
+
+@dataclass
+class RevenueCatClient:
+    api_key: str
+    project_id: str
+    base_url: str = "https://api.revenuecat.com/v2"
+    max_retries: int = 5
+    _session: requests.Session = field(
+        default_factory=requests.Session, init=False, repr=False
+    )
+
+    def __post_init__(self) -> None:
+        self._session.headers.update(
+            {
+                "Authorization": f"Bearer {self.api_key}",
+                "Accept": "application/json",
+            }
+        )
+
+    def paginate(
+        self, path: str, params: dict[str, Any] | None = None
+    ) -> Iterator[dict]:
+        """Yield every item from an RC v2 list endpoint.
+
+        Params apply ONLY to the first request — RC's `next_page` is an
+        absolute URL with the original query string + the `starting_after`
+        cursor token baked in, so subsequent fetches use it verbatim.
+        """
+        next_url: str | None = f"{self.base_url}{path}"
+        first = True
+        while next_url:
+            data = self._get(next_url, params=params if first else None)
+            first = False
+            for item in data.get("items", []):
+                yield item
+            next_url = data.get("next_page")
+
+    def get(self, path: str, params: dict[str, Any] | None = None) -> dict:
+        """One-shot GET on a non-list endpoint. Returns parsed JSON."""
+        url = f"{self.base_url}{path}"
+        return self._get(url, params=params)
+
+    def _get(
+        self,
+        url: str,
+        params: dict | None = None,
+        _attempt: int = 0,
+    ) -> dict:
+        # Network-level failures (timeout, connection reset, broken
+        # chunked encoding) raise BEFORE a `resp` object exists, so we
+        # have to catch them around the .get call and route through the
+        # same capped-backoff path as a 5xx.
+        try:
+            resp = self._session.get(url, params=params, timeout=_TIMEOUT)
+        except requests.exceptions.RequestException as exc:
+            if _attempt < self.max_retries:
+                time.sleep(min(2**_attempt, 60))
+                return self._get(url, params, _attempt + 1)
+            raise RuntimeError(
+                f"revenuecat: network failure after {self.max_retries} retries on {url}: {exc}"
+            ) from exc
+
+        if resp.status_code == 429:
+            # RC enforces a per-domain RPM cap. `Retry-After` is the
+            # server's directive — honor it. But ALSO bound the loop:
+            # the previous version did not increment `_attempt` and a
+            # sustained rate-limit wedged the run forever.
+            if _attempt >= self.max_retries:
+                raise RuntimeError(
+                    f"revenuecat: rate-limited after {self.max_retries} retries on {url}; "
+                    f"Retry-After={resp.headers.get('Retry-After')}"
+                )
+            wait = int(resp.headers.get("Retry-After", 60))
+            time.sleep(wait)
+            return self._get(url, params, _attempt + 1)
+        if resp.status_code in (500, 502, 503, 504) and _attempt < self.max_retries:
+            time.sleep(min(2**_attempt, 60))
+            return self._get(url, params, _attempt + 1)
+        resp.raise_for_status()
+        return resp.json()

dtex-0.2.4/dtex/sources/revenuecat/register.yaml

@@ -0,0 +1,148 @@
+# RevenueCat — baked source connector for the v2 API
+# (https://api.revenuecat.com/v2). Three streams covering RC's three
+# distinct extraction surfaces:
+#
+# * `customers` — every customer in a project. NON-incremental: RC v2 has
+#   no server-side date filter on /customers (verified June 2026 against
+#   the docs, the airbyte issue 70315, and the RC community forum), so
+#   every run paginates the full customer list. `write_disposition: merge`
+#   on `id` makes re-pulls upsert idempotently.
+#
+# * `subscriptions` — per-customer fan-out. RC has no project-level
+#   /subscriptions endpoint, only /customers/{id}/subscriptions, so this
+#   stream iterates the customer list and fetches per-customer
+#   subscriptions. NON-incremental; merge-on-id like customers.
+#   Operationally expensive: O(N+1) HTTP calls per run for N customers.
+#
+# * `metrics_daily` — the third surface. RC v2's Charts API
+#   (/charts/{chart_name}) DOES take server-side start_date/end_date
+#   filters AND tags per-day values with `incomplete=true` when the day
+#   is still finalizing. Long format (one row per cohort_date × chart ×
+#   measure) so adding a chart needs zero schema migration. Real
+#   incremental; the cursor advances only past days RC marked complete,
+#   so partial-today re-pulls cleanly on the next run.
+#
+# Auth: a v2 RC secret key (`sk_...`) with at minimum
+# `customer_information:customers:read`,
+# `customer_information:subscriptions:read`, and
+# `charts_metrics:overview:read`. v2 keys are SEPARATE from v1 keys
+# ("API v1 keys will not work with REST API v2" — RC docs). The default
+# reads it from `${env.REVENUECAT_API_KEY}`; profile-level secret://
+# refs (GCP Secret Manager / AWS / Vault) override per environment.
+
+name: revenuecat
+kind: source
+version: "1.0.0"
+summary: RevenueCat v2 API — customers, subscriptions, and daily chart metrics.
+tags: [revenuecat, saas, subscriptions, analytics, mobile]
+
+requires:
+  - "requests>=2.31"
+
+params:
+  project_id:
+    type: string
+    required: true
+    description: RevenueCat project ID (e.g. proj1ab2c3d4).
+  base_url:
+    type: string
+    default: "https://api.revenuecat.com/v2"
+    description: RevenueCat v2 API base URL — overridable for stubbed tests.
+  page_size:
+    type: int
+    default: 100
+    description: Records per page on list endpoints (RC default is 20).
+
+  # --- metrics_daily-only ---------------------------------------------------
+  # Curated charts the metrics_daily stream pulls. Comma-separated chart_name
+  # values from RC's docs. Each chart can return multiple measures
+  # (e.g. churn = Actives + Churned Actives + Churn Rate); the connector
+  # flattens them all into the long-format output.
+  #
+  # The four defaults are charts that return the standard
+  # {cohort, incomplete, measure, value} shape. cohort_explorer and
+  # prediction_explorer return different shapes and will break the flattener
+  # — exclude unless you custom-handle them.
+  metrics_charts:
+    type: string
+    default: "revenue,mrr,actives,trials"
+    description: Comma-separated chart_name values for the metrics_daily stream.
+
+  # Earliest cohort date for the FIRST metrics_daily run. Once state exists
+  # this is ignored. ISO date, no time component.
+  metrics_initial_since_date:
+    type: string
+    default: "2024-01-01"
+    description: Earliest start_date for metrics_daily's first run.
+
+  # Re-pull window for metrics_daily: every run pulls from
+  # max(cursor, today - lookback_days) up to today. Wider window = more
+  # rate-limit churn but better safety for RC's "most recent day may be
+  # partial" caveat. 7 is safely past RC's same-day finalization window
+  # without burning many calls.
+  metrics_lookback_days:
+    type: int
+    default: 7
+    description: Days of overlap on every metrics_daily run (handles partial-today + late corrections).
+
+secrets:
+  # The RC v2 secret key. The env-var default lets `pip install dtex`
+  # users get going without any extras; profile-level overrides
+  # (`ref: secret://gcp-secret-manager/...`) work for production.
+  - name: api_key
+    ref: ${env.REVENUECAT_API_KEY}
+
+streams:
+  - name: customers
+    table: customers
+    primary_key: [id]
+    write_disposition: merge
+    schema:
+      - {name: id,                    type: STRING,    mode: REQUIRED}
+      - {name: first_seen_at,         type: TIMESTAMP}
+      - {name: last_seen_at,          type: TIMESTAMP}
+      - {name: last_seen_app_version, type: STRING}
+      - {name: last_seen_country,     type: STRING}
+      - {name: last_seen_platform,    type: STRING}
+
+  - name: subscriptions
+    table: subscriptions
+    primary_key: [id]
+    write_disposition: merge
+    schema:
+      - {name: id,                             type: STRING,    mode: REQUIRED}
+      - {name: customer_id,                    type: STRING}
+      - {name: customer_last_seen_at,          type: TIMESTAMP}
+      - {name: product_id,                     type: STRING}
+      - {name: status,                         type: STRING}
+      - {name: gives_access,                   type: BOOLEAN}
+      - {name: auto_renewal_status,            type: STRING}
+      - {name: store,                          type: STRING}
+      - {name: store_subscription_identifier,  type: STRING}
+      - {name: current_period_starts_at,       type: TIMESTAMP}
+      - {name: current_period_ends_at,         type: TIMESTAMP}
+      - {name: total_revenue_gross_usd,        type: FLOAT}
+      - {name: total_revenue_proceeds_usd,     type: FLOAT}
+
+  # metrics_daily — long format (one row per cohort_date × chart × measure).
+  # Adding a new chart to `metrics_charts` requires no schema change.
+  # Cursor on `cohort_date`; the connector observes only complete days
+  # (incomplete=false in the API response) so the next run re-pulls
+  # today AND any day that was still partial last run.
+  - name: metrics_daily
+    table: metrics_daily
+    primary_key: [cohort_date, chart_name, measure_name]
+    write_disposition: merge
+    incremental:
+      cursor_field: cohort_date
+      cursor_type: date
+      initial_value: "2024-01-01"
+    schema:
+      - {name: cohort_date,   type: DATE,      mode: REQUIRED}
+      - {name: chart_name,    type: STRING,    mode: REQUIRED}
+      - {name: measure_name,  type: STRING,    mode: REQUIRED}
+      - {name: value,         type: FLOAT}
+      - {name: incomplete,    type: BOOLEAN}
+      - {name: pulled_at,     type: TIMESTAMP, mode: REQUIRED}
+
+schedule: "0 */6 * * *"