@datarails-ai/dr-cli 0.0.1-locked → 1.6.70

package/dist/skills/datarails-connectors-onboard-connector/SKILL.md

@@ -0,0 +1,813 @@
+---
+name: datarails-connectors-onboard-connector
+description: Use when creating a new connector for the Datarails connectors platform and wiring it end-to-end on a dev/test env - writing the connection + dataquery files, publishing the artifact, binding it, and setting up the datasource + filebox so a sync actually runs. Triggers - "new connector", "onboard a connector", "publish a connector", "wire fmp_fx/<x> connector", "connector sync does nothing", "connector-jobs queue empty", "add <vendor> for another org", "reuse the <vendor> connector", "lib/<vendor>", "shared vendor code", ConnectorQuery/binding/datasource/filebox for a DatarailsConnector.
+---
+
+# Onboard a connector (implement + wire end-to-end)
+
+## Overview
+
+A **connector** is a two-file artifact — a `connections/<conn>.py` composed with a `dataqueries/<dq>.py`
+(no manifest; see the dr-connectors README for the authoritative file layout + programming model) —
+published to the shared JFrog repo by CI, then pulled into a datacenter's blob on ingest. Those two
+files are what *ships*; a third, `tests/test_<dq>.py`, is what makes them mergeable — see
+*Write the test* in §1, and write it as part of building the connector, never as a follow-up. It runs inside
+its flavor's runner image on that flavor's Service Bus queue. Getting a sync to actually run needs **four**
+pieces wired together — miss one and the sync silently does nothing:
+
+| Piece | Where | Holds |
+|---|---|---|
+| **Artifact** | JFrog `<dc>/<org>/<vendor>/<dq>/<vN>/artifact.zip` (publish tier; `vN` = the git tag CI assigns); after ingest, DC blob `artifacts/<name>/<vN>` + control-plane `artifact_registry` | the code (version = git tag, not a hash) |
+| **Datasource** | monolith integration (`DatarailsConnector` type, **`is_sync_service: true`**) | the credentials — an `apikey`, or an `oauth` block the implementer authorises in the UI (§3) |
+| **Filebox + data-query** | monolith filebox (**`support_api` ON**), `data-query.connection = <datasource_id>` | the sync target + trigger |
+| **Binding** | control-plane `Connector` + `ConnectorQuery` (by `filebox_uuid`) | artifact ↔ filebox ↔ flavor ↔ config ↔ secret_ref |
+
+**The trigger is a FILEBOX data-query sync, NOT `dr integrations sync`** (see Gotchas). Design:
+`dr-connectors-runner/docs/plans/2026-07-17-connector-flavors-container-keda-design.md`.
+
+**A new integration is tested CLOUD-FIRST — do NOT gate it on the local `dev-fetch` sandbox.**
+`dr connectors dev-fetch` / the `connector-sandbox` skill pull creds+config from an **existing** dev
+datasource+filebox, so they can't fully run before you've wired one — for a brand-new connector, skip
+that step. Publish to the cloud and verify there: publish (§2) → datasource+filebox+binding (§3–4) →
+trigger (§5) → read the outcome from the **completion store** (per-hook `verdict`/`error` — where
+vendor HTTP errors surface; the Inngest run `output` is `null`), the Inngest run status, and the
+landed version. Iterate by editing the files → **merge to `main`** (CI cuts the next git tag `vN+1` and
+publishes it to JFrog) → `dr connectors publish <dq.py>` (ingests the newest tag; or `--tag vN`) → `dr
+connectors query repoint <queryId> --artifact-version vN+1` → re-trigger. Each merge that touches a
+connector cuts a NEW version, so iterating needs a merge — the ingest 404s on a tag CI hasn't published
+yet. (A quick standalone probe of the vendor endpoint with the raw key still cheaply de-risks the URL/
+response shape *before* the first publish.)
+
+## Identity — act as the customer, never as the operator
+
+**Every `dr-cli` call in this skill runs as the CUSTOMER's user, minted with `dr login-as`. Never as your
+own `@datarails.com` operator login.** The operator login exists only to *authorize* that mint.
+
+```bash
+dr login --server DEV                                     # once — establishes the operator (browser; takes NO --account)
+dr login-as <customer-user-email> --server DEV            # mints the impersonation JWT; no browser, no TTY
+dr whoami --server DEV --account <customer-user-email>    # MUST print "Impersonated by: <operator>"
+ACC="--server DEV --account <customer-user-email>"        # every step below uses THIS identity
+```
+
+If `whoami` does not print `Impersonated by:`, you are still the operator — **stop and fix the identity
+before creating anything.** Full impersonation flow (multiple operators stored → `--as <operator-email>`;
+email won't resolve → `--user-id <n>`; `not permitted` handling) is in the `dr-switch-user` skill.
+
+**Why getting this wrong silently corrupts the wiring.** The org comes from the **JWT**, not from a flag:
+`dr filebox create` has no `--org` option — and neither do `documents support-api set`, `filebox
+data-query set`/`sync`, or `filebox versions` — so an operator-identity run creates the filebox **in the
+operator's own org**. `dr integrations add` is the one exception: its payload names `organization`
+explicitly, so the **datasource lands in the right org while the filebox lands out of tenant**. Everything
+downstream then looks healthy — the binding resolves, the sync completes, a version lands — but it landed
+in the wrong org and the customer never sees the filebox. `dr connectors` (control-plane) accepts the
+impersonated JWT and takes its org from it as well, so `--org-id` can be omitted there. Beyond the
+breakage, creating customer resources under the operator's own identity breaks resource-level scoping and
+loses the audited operator→target trail (`tech/security.md` §2.2, §8).
+
+## 1. Implement (repo: dr-connectors)
+
+### Step 0 — look for a lib FIRST, before writing or copying any code
+
+A vendor's code may already live once in **`lib/<vendor>/`** (e.g. `lib/ramp`, `lib/sap_b1`,
+`lib/metmuseum`). List `lib/` in `C:\codegithub\dr-connectors` before anything else.
+
+**The lib exists** → the connector is **two thin files**, nothing to implement:
+
+```python
+# <datacenter>/<org_id>_(<org_name>)/<vendor>/connections/<conn>.py
+from lib.<vendor>.connections.<variant> import RUNTIME_FLAVOR, <Class> as CONNECTION  # noqa: F401
+
+# <datacenter>/<org_id>_(<org_name>)/<vendor>/dataqueries/<dq>.py
+from lib.<vendor>.dataqueries.<variant> import OBJECT_TYPES, get_query_class  # noqa: F401
+
+CONNECTION = "<conn>"
+```
+
+- **Pick the variant by output, not by customer.** A lib can hold several output shapes of one vendor
+  (`lib/ramp`: `curated` / `reports` / `flat`). Read the lib's `__init__.py` — it names each variant and
+  who uses it — and match the columns the customer expects (a Workato migration wants the variant built
+  for Workato parity). Never merge or edit a variant another org lands data through: that changes their
+  live output.
+- **A per-org difference is config or a one-attribute extension, never a copy.** First look for a
+  binding parameter the lib already reads (`lib/ramp`: optional `scope` in `query_config`). Otherwise
+  subclass the lib's class in the thin connection file and override one attribute —
+  `samples/sample_lib_connector/` shows both forms.
+- **No new test is needed**: a dataquery that only references a lib counts as tested by the lib's own
+  `lib/<vendor>/tests/` (`scripts/check_connector_tests.py`). An extension that adds behaviour tests that
+  behaviour in the lib or next to the connector.
+- **The lib is missing a dataset, or has a bug** → change the lib (with its tests) in its own PR. A lib
+  change republishes every connector that bundles it — new versions, never promoted — so each org moves
+  only when its binding is repointed.
+
+**No lib** → implement the connector as below, starting from `samples/`. If a second org is likely (the
+same vendor queued for another customer, a Workato migration of a vendor we already run), write it as a
+lib from the start: copy **`lib/sample_vendor/`** (the lib template — layout, rules and tests in its
+`__init__.py`) to `lib/<vendor>/`, and add the two thin files above. The dr-connectors README section
+"Shared vendor code: lib/" has the rules (literal `RUNTIME_FLAVOR`/`OBJECT_TYPES` one import hop from the
+thin file, absolute `lib.<vendor>` imports, output variants).
+
+### Write the connector
+
+Clone `C:\codegithub\dr-connectors`. Work on a **feature branch off `main`** — do not commit connector
+files straight to `main`. **Once the connector pair is written, commit it — decide the branch/message
+yourself from the convention below; do NOT ask the user:** branch `DR-<ticket>/feature/<vendor>-connector`
+off `main` (no ticket → `connectors/<datacenter>-<org_id>-<vendor>`), stage ONLY the three connector files
+(connection + dataquery + **its test** — see *Write the test* below; a connector without one is not finished;
+a lib-backed connector is just its two thin files — Step 0),
+commit `DR-<ticket>: add <vendor>__<dq> connector (<datacenter>/<org_id>)` (drop the `DR-<ticket>:` prefix
+when there's no ticket), push, and open a PR. **Merge to `main` is now a prerequisite** — CI publishes the
+artifact to JFrog on merge, and ingest (§2) pulls it from there, so an unmerged connector can't be ingested.
+
+The tree is **vendor-first**, and each connection
+and dataquery is a single `.py` file (the file stem is its name):
+`<datacenter>/<org_id>_(<org_name>)/<vendor>/connections/<conn>.py` +
+`<datacenter>/<org_id>_(<org_name>)/<vendor>/dataqueries/<dq>.py` +
+`<datacenter>/<org_id>_(<org_name>)/<vendor>/tests/test_<dq>.py`. The dataquery's `CONNECTION = "<conn>"`
+names its sibling connection file; the published artifact name is `<vendor>__<dq>`. `tests/` is a third
+sibling of the two published folders and never ships: `compose_zip` puts exactly `connection.py` +
+`query.py` in the artifact, and both `discover_vendor_dataqueries` and the publish workflow's path
+filter ignore any path whose fourth component isn't `connections`/`dataqueries` — so a tests-only
+change neither republishes nor burns a version.
+
+Two files, both importing from `dr_integrations_sdk`. Keep imports within the flavor's `allowlist.txt`
+(stdlib always ok; base flavor = httpx + tenacity + dateutil). No manifest — name = `<vendor>__<dq>`,
+runtime_flavor = the `RUNTIME_FLAVOR` constant, version = the git tag CI assigns on merge (a monotonic
+`vN` per connector — NOT a content hash). **No lib (Step 0)? Start from a `samples/` template — pick by
+the SOURCE's shape:**
+
+| Vendor shape | Copy | It teaches |
+|---|---|---|
+| Flat: one endpoint you page through (`/invoices?offset=`) | **`samples/sample_http_vendor/`** — the golden path | auth gate, the three config channels, `get_query_class` dispatch, the greppable logger, the non-retryable-4xx→PermanentError guard, and both pagination styles |
+| **Fan-out / N+1**: you must drill DOWN to reach rows (`batches → headers → lines`, `invoices → lines`, `accounts → transactions`) | **`samples/sample_streaming_fanout/`** | everything above **plus** how to fetch a lot of rows concurrently while holding almost none — the bounded worker pool + bounded queue that keeps peak memory flat under the sandbox's 512 MB ceiling (see *Memory* below) |
+| **Any shape, and a second org is likely** — write it as a lib | **`lib/sample_vendor/`** + **`samples/sample_lib_connector/`** | the lib layout: output-neutral code once in `_core.py`, one-hop literals, a page cap that raises, the thin reference-only connector and a customer extension, and the lib's tests (including a runner-style artifact test) |
+
+Read `sample_http_vendor` first either way; the fan-out sample assumes it and does not re-explain the
+shared parts. Both live at the repo-root `samples/` dir (NOT under an org folder, since they're
+templates that are never published; copy one INTO your `<datacenter>/<org_id>_(<org_name>)/` when
+starting a real connector). **Base every new connector ONLY on a lib (Step 0) or a `samples/` template —
+do NOT copy, adapt, or crib from existing published dataqueries** (any connector already published under a
+`<datacenter>/<org_id>_(...)` org folder). Copying one is how a vendor ends up with N drifting copies
+(Ramp had four) — if you need an existing connector's code for a second org, move it into `lib/<vendor>/`
+instead and make both orgs thin.
+Those carry org-specific config, may lag the current SDK/runtime contract, and cloning them propagates
+drift — the samples are the single maintained source of truth. At most read an existing one to sanity-check
+one behavior; never use it as the starting point to copy from.
+
+**Batching & output — the non-negotiable pattern (get this right FIRST, before writing fetch).**
+
+**Step 0 — decide the read strategy by the source's batching support:**
+- **Source supports pagination / a batch API** → loop it **INTERNALLY** inside one `fetch()` (below).
+  This is the default; design for it.
+- **Source has NO pagination** (one response returns everything) → you **cannot** safely pull an
+  unbounded result in a single ≤2700s step, and you must not overwrite the blob mid-stream. So in the
+  first build **fetch only a small bounded sample — cap ≤ 1000 rows** (`params["limit"]=1000` or slice
+  the response) — get the connector wired and landing, then treat full/large extraction as a separate
+  design step (server-side filtering, a batch endpoint request to the vendor, or date-window slicing).
+  Never “just pull it all” from a no-pagination source and hope it fits.
+
+**Never overwrite the blob — produce the whole result in ONE `write_output`, always batched/streamed.**
+Do NOT return `has_more` for an accumulated single output (e.g. a CSV). `write_output` streams to
+`req.output_sas` and finishes with `commit_block_list`, which **replaces** that blob's whole contents
+(`dr_integrations_sdk/query.py`). The runtime *does* re-invoke
+`fetch` on `has_more` and mints a per-batch blob, BUT the landable `post_fetch` step is currently fed
+**only `batch_keys[-1]` — the last batch** (`dr-connectors-runtime/src/runtime/activities.py:204-236`:
+`"input_sas": mint_blob_sas(ctx.batch_keys[-1], ...)`; multi-batch assembly is an unimplemented "Phase
+2"). So a multi-batch `has_more` fetch **silently lands only the last page** — verified: a 76-page
+month landed only the final page's rows, with no header. Correct pattern — one `fetch()`, one `write_output`, stream
+the header once then one block per page (`query.py:60-67`), `has_more=False`:
+```python
+async def blocks():
+    yield csv_header_block(FIELDS)
+    async for page in paginate(session, req):   # loop the vendor cursor INTERNALLY
+        yield csv_rows_block(FIELDS, page)       # one Azure block per page — memory stays flat
+return HookResult(output_blob_key=await self.write_output(req, blocks()), has_more=False)
+```
+
+**Memory — the 512 MB ceiling, and the one rule: never hold anything that scales with ROW COUNT.**
+
+The hook runs in a subprocess under **`RLIMIT_AS` = 512 MB of VIRTUAL address space**
+(`dr-connectors-runner/src/runner/sandbox_entry.py`) and the runner pod's memory limit is **also
+512Mi** (`dr-infra-values/applications/<stage>/connectors-runner-<flavor>/values.yaml`), so that
+rlimit is effectively the whole budget — shared with the parent runner process, and paying as well
+for the interpreter, the SDK's `azure.storage.blob.aio` + aiohttp, httpx, OpenSSL, and every
+in-flight response body.
+
+**`RLIMIT_AS` is POSIX-only (`if sys.platform != "win32"`), so none of this reproduces on a Windows
+dev machine.** A buffering connector passes every local test and fails only in production, with a
+`MemoryError` naming whatever allocation happened to die rather than the accumulation that caused
+it. Reproduce it locally by running the hook in `python:3.12-slim` under the same rlimits — as an
+ad-hoc check, never as a committed memory test (see *Write the test*).
+
+*The shape that looks correct and is not* — `oraclefusion__journal_lines` (DR-53061) did this and
+measured **378 MB live to emit 67 MB of CSV** (243 MB of it row dicts at 846 B/row), dying in prod
+with `MemoryError: Unable to allocate output buffer`:
+```python
+line_groups = await asyncio.gather(*(get_lines(t) for t in trios))  # <-- every row, in memory
+rows = [row for group in line_groups for row in group]
+content = buf.getvalue().encode()                                    # <-- + 2 more full copies
+async def blocks(): yield content                                    # <-- a "stream" of ONE block
+```
+There IS an async generator and `write_output` IS a streaming sink — but the generator was built
+*after* all the work was done, so it streams one object that already cost the budget. **Nothing
+about the presence of `async def blocks()` makes a connector stream.**
+
+Four rules, each of which that bug broke:
+1. **All the fetching lives INSIDE the generator handed to `write_output`.** It is PULL-based:
+   `await blob.stage_block(...)` runs inside its `async for` (`query.py`), staging one block per
+   `yield` and pulling the next only once that block is staged. That is where backpressure comes
+   from, and only a generator that does the work gets it.
+2. **`asyncio.gather` over the whole work set IS the accumulation.** It returns only when every
+   coroutine finishes, and returns every result at once — there is no streaming variant. Bounded
+   waves of `gather` over a handful of **pages** are fine (results are bounded); `gather` over
+   every **parent** in a fan-out is a buffer wearing a concurrency costume. Use N worker tasks
+   draining a work queue and pushing finished chunks into a **bounded** `asyncio.Queue` — the
+   bound is what makes `await queue.put(...)` block when the uploader falls behind, carrying
+   backpressure from Azure all the way to the vendor request.
+3. **Rows become bytes when their page arrives, then are forgotten.** Measured per row: dict 846 B,
+   tuple 550 B, pre-joined CSV line 289 B — but the right answer is to keep none. `csv.writer` over
+   tuples, not `DictWriter` over dicts. The only legitimate accumulation is the **work list** (which
+   parents exist), bounded by parent count, not row count — and keep it as small tuples of the
+   fields you need, never the vendor's response objects (journal_lines retained ~4,700 verbose
+   Oracle header dicts to read 5 fields from each).
+4. **Re-batch chunks to a few MB before yielding.** One `yield` == one Azure block, capped at
+   `AZURE_MAX_BLOCKS` (50,000), staged serially — so yielding per page stages thousands of tiny
+   blocks (~100s of round-trips for a 67 MB file), while yielding once at the end is rule 1's bug.
+   `4 * 1024 * 1024` is the sweet spot. Bigger buys nothing: staging overlaps the vendor fetch, and
+   the fetch is orders of magnitude slower than the upload, so block size is a pure memory cost
+   past that point.
+
+Rewritten to that pattern, the same 301,808-row month peaks at **18 MB**, and 12x the rows moves the
+peak by ~1x. `samples/sample_streaming_fanout/` is the working template.
+
+Two more sandbox ceilings worth knowing, both undocumented elsewhere:
+- **The DNS thread pool can quietly eat half the budget.** httpx resolves via anyio →
+  `loop.getaddrinfo` → `run_in_executor(None, ...)`, i.e. asyncio's **default** ThreadPoolExecutor,
+  sized `min(32, os.cpu_count() + 4)` off the **node's** core count, not the container's. Each
+  thread reserves the default **8 MB pthread stack** of virtual address space and
+  ThreadPoolExecutor **never reaps its threads** — so an unbounded pool can claim ~256 MB of the
+  512 MB and never give it back. That is why a brief concurrency spike used to poison every *later*
+  phase of a run with `RuntimeError: can't start new thread` and SSL allocation failures long after
+  the spike finished. Cap it once at the top of the fetch:
+  `asyncio.get_running_loop().set_default_executor(ThreadPoolExecutor(max_workers=4))`.
+- **`RLIMIT_CPU = 60s`** sits right next to the memory limit in `sandbox_entry.py`. Exceeding it
+  SIGKILLs the hook (runner reports `sandbox exited -24`) with no `MemoryError` to explain it. Async
+  I/O waiting is free, but TLS handshakes, gzip and JSON parsing are not — log
+  `time.process_time()` alongside wall clock so you can see how close you are.
+
+**Never raise a "safety cap" into a silent truncation.** Page-limit guards
+(`_LINE_PAGE_LIMIT` etc.) exist so a broken cursor can't loop forever — but a cap that logs a
+warning and returns what it has lands a **partial file that looks complete**. That is how
+journal_lines silently dropped 113,171 rows of a customer's month and turned into a reconciliation
+escalation. Caps must `raise PermanentError`.
+
+Keep that single step under the runner's **2700s (45 min) hook timeout** (`HOOK_TIMEOUT_S` in
+`dr-connectors-runtime/src/runtime/activities.py`, applied **per hook**, so `fetch` gets the whole
+budget; over it → `{"error":"sandbox timeout after 2700s; stderr: …","verdict":"block","category":"timeout"}`).
+A timeout **blocks instead of retrying** — the budget is a property of the hook, not
+of the moment, so a re-run would do the same too-slow work and fail again. One kill, one reported
+failure. Levers, in order: (1) a
+**larger vendor page size** — but the vendor may cap it (**some vendors reject `pagesize` above their own limit; a legacy
+recipe used 1000**), so probe before relying on it and **log vendor 4xx** so a rejected page
+size is visible (a bare `raise_for_status` leaves no trace); (2) **bounded concurrency** inside the one
+`fetch` — for a FLAT source, waves of N≈3–5 page GETs (`asyncio.gather`), streamed in page order,
+stopping at the first short/empty page; for a **fan-out** source, a bounded worker pool feeding a
+bounded queue instead (`samples/sample_streaming_fanout/`) — gather over every parent both buffers
+the whole result (see *Memory*) and stalls fast parents behind slow ones in the same wave, which
+measured 376s vs 192s on the same data. Either way **with per-page retry** (retry a transient
+drop/429/5xx on that one page, not the whole month) since vendors throttle under concurrency; for
+waves use `return_exceptions` so a page fetched *beyond* the last one can't fail the run. Wave
+concurrency is what took a 76-page GL month from ~388s sequential (timeout) to ~177s. (3) only as a
+last resort, narrow the
+sync's date window so each run is smaller. `has_more`/`next_cursor` are reserved for when Phase-2
+multi-batch assembly lands — until then treat them as unusable for a single accumulated file.
+
+- **`connections/<conn>.py`** — subclass `DataIntegrationConnection`; `async def connect(self, req)`
+  resolves creds via `self.get_credentials(req)` and returns an in-memory session. This is the
+  **fail-fast auth gate** — prove auth here, once. Set module-level `RUNTIME_FLAVOR = "<flavor>"` and
+  `CONNECTION = <the class>`:
+  ```python
+  from dr_integrations_sdk import DataIntegrationConnection, SyncRequest
+  RUNTIME_FLAVOR = "base"
+  class MyConnection(DataIntegrationConnection):
+      async def connect(self, req: SyncRequest):
+          creds = await self.get_credentials(req)      # {"apikey"/"clientId"...} from the datasource
+          # mint a token / open a client, then return an in-memory session object
+          return Session(...)
+  CONNECTION = MyConnection                            # the runtime object (a class), NOT a string
+  ```
+- **`dataqueries/<dq>.py`** — subclass(es) of `DataQuery`; `async def fetch(self, session, req)` uses the
+  session (never re-resolving creds) and returns a `HookResult`. Set `CONNECTION = "<conn>"` (a string
+  naming the sibling connection file) and `def get_query_class(name)`:
+  ```python
+  from dr_integrations_sdk import DataQuery, HookResult, SyncRequest
+  CONNECTION = "<conn>"                                # -> connections/<conn>.py, same vendor (a STRING here)
+  class MyQuery(DataQuery):
+      async def fetch(self, session, req: SyncRequest) -> HookResult:
+          cfg = req.config                             # public_config ∪ query_config ∪ filebox Sync-query (cfg["query"], parsed from its JSON text) ∪ cfg["period"]
+          # Merge order matters: the Sync-query is applied LAST and wins any key collision. Selectors
+          # (symbol/account/entity) live there so operators can see them in the UI; query_config is
+          # plumbing only (base_url, caps). Never define one key in both — see step 3c.
+          # httpx GET using `session` -> rows; write via an async generator yielding bytes
+          return HookResult(output_blob_key=await self.write_output(req, gen), has_more=False)
+  def get_query_class(name: str) -> type[DataQuery]:   # name = req.config["objectType"]; single-dataset -> return the sole class
+      return MyQuery
+  ```
+  Note `CONNECTION` means two different things by file: a **class** in the connection file, a **string**
+  (the connection's name) in the dataquery file.
+- **Synchronous vendor SDK?** (google-api-python-client, boto3, most non-httpx clients). `connect`/`fetch`
+  are `async` — a blocking SDK call stalls the runner's event loop. Wrap it in `asyncio.to_thread(...)`, and
+  do the SDK `import` **inside** that thread helper (not module top): the module then imports without the
+  flavor's deps present (unit-testable), and the publish allowlist AST check **still sees** the import
+  (it walks function bodies too). E.g. google_sheets: `build("sheets","v4",developerKey=key,static_discovery=True)`
+  inside a `_read_values()` called via `await asyncio.to_thread(_read_values, …)`.
+
+### Write the test — part of Implement, not a follow-up
+
+**A connector is not finished until `tests/test_<dq>.py` exists and passes.** Write it as you build
+the connector and commit it in the same commit as the two connector files. This is not a style
+preference: the production defects listed in this section — the 113,171 silently dropped rows, the
+cap that truncated, the cursor that never advanced — are trivially catchable by a
+test and was instead found by a customer or by hand. `core/testing_strategy.md` also requires tests
+committed with every code change, so shipping without one is a standards violation, not just a risk.
+
+**Connector tests are UNIT tests. Do NOT write memory tests** — no `tracemalloc`, no
+peak-allocation ratios, no "memory stays flat as rows grow", no timing or performance assertions.
+They are environment-sensitive (the same test measured 2.5x on a dev box and 3.2x on the Linux CI
+runner) and failed PRs that never touched the connector, so they were removed from `dr-connectors`
+(DR-54014) and the harness no longer has an `assert_memory_flat` fixture. Memory safety is a
+**design rule** — follow the *Memory* section above and have it checked in code review; if you must
+reproduce a suspected buffer, do it ad hoc in `python:3.12-slim` under the sandbox rlimits, never as
+a committed test.
+
+**None of these tests may touch the network.** The fake vendor is in-process, the credential resolver
+is monkeypatched, and the repo-root `conftest.py` blocks any non-loopback socket — a unit test that
+can reach a vendor is non-deterministic, unsafe in CI, and can hit a *customer's live account*.
+
+The repo-root `conftest.py` carries the whole harness, so the test itself stays short. Fixtures:
+
+| Fixture | What it gives you |
+|---|---|
+| `connector` | your connector, loaded from the unimportable org path. `.query` / `.connection` resolve by convention from the test's own filename; `.dq(name)` / `.conn(name)` for siblings |
+| `vendor` | the fake vendor. `.paginate(path, total=, page_size=, row=)` generates pages **on demand** (so the fake is never itself the accumulation); `.get(path, handler)` for anything odd; `.calls_to(path)` to assert requests. An unregistered endpoint raises rather than 404s, and a runaway paging loop fails fast instead of hanging |
+| `sync_request` | a real `SyncRequest` with the config channels merged as the platform merges them, wired to the sink |
+| `sink` | a real `file://` `output_sas`. `.rows()`, `.header()`, `.data_rows()`, `.text()`. This drives the SDK's genuine `write_output` block-staging path — do **not** mock `write_output`, or you lose the only thing that shows the connector buffers |
+| `connector_logs` | everything the connector logged, for secret/`base_url` leak assertions |
+| `sleeps` | retry backoff made instant, and the delays recorded (autouse). It patches `asyncio.sleep` process-wide, so a task parked on `asyncio.sleep` to simulate a hang completes immediately — park it on an `asyncio.Event` that is never set instead |
+
+**Six things the test must cover** (unit tests only — see above: no memory tests) — each maps to a defect that reached a customer:
+
+1. **Correctness against the fake vendor, paginating at EVERY level of the fan-out.** Assert the
+   row *contents*, not just the count: a connector that never advances its cursor writes the right
+   number of rows while every page is a copy of page 1.
+2. **Every safety cap, in BOTH directions — as two separate tests.** Landing *exactly* on the cap
+   → completes normally. Exceeding it → `PermanentError`, never a warn-and-return that hands the
+   customer a short file indistinguishable from a complete one. Each side catches a different
+   defect: a cap that fires one row early rejects complete data, one that never fires truncates
+   silently — and neither shows up unless you probe both.
+3. **Non-retryable vendor 4xx → `PermanentError`** (400/401/402/403/404/422), and 429/5xx *not*
+   permanent. A bare exception is `verdict: retry` — five doomed attempts, ~25 min of hammering the
+   customer's vendor account, and a locked filebox.
+4. **Failure and cancellation hygiene.** A failing producer or consumer must surface promptly with
+   no orphaned tasks and no hang; the suite runs with `filterwarnings = ["error"]` and fails any
+   test that leaves a task pending.
+5. **Output shape** — header emitted exactly once, `has_more is False`, every row the declared width.
+6. **No secret and no raw `base_url` in the captured logs** — `base_url` is operator-supplied and can
+   carry `user:pass@` userinfo or a query-string credential into a shared multi-tenant log.
+
+Copy `samples/sample_http_vendor/tests/test_sample_http_vendor.py` (or the fan-out sample's) and
+adapt — it is written to be the template, with each test labelled by the defect it catches. The
+shape is short:
+
+```python
+@pytest.fixture
+def session(connector):  # NOT a conftest fixture - the test file defines its own, as the samples do
+    return connector.connection.Session(api_key=API_KEY)
+
+
+async def test_pages_internally_and_streams_one_complete_file(
+    connector, session, vendor, sync_request, sink
+):
+    vendor.paginate("/api/rows", total=2503, page_size=500, row=make_row, style="cursor")
+    result = await connector.query_class("table")().fetch(session, sync_request(query={...}))
+
+    assert result.has_more is False
+    assert sink.header() == EXPECTED_COLUMNS
+    assert [r[0] for r in sink.data_rows()] == [str(i) for i in range(2503)]  # content, not count
+```
+
+Run it before you commit:
+
+```bash
+# run from the dr-connectors repo root
+python -m pytest "<datacenter>/<org_id>_(<org_name>)/<vendor>/tests/" -q
+python scripts/check_connector_tests.py --changed-from origin/main   # anything missing a test
+```
+
+That script only inspects `dataqueries/`. A branch that changes **only** a `connections/` file reports
+`0/0 dataqueries changed on this branch have a colocated test` — which is not the same as "covered".
+Connection code is where the auth gate and host normalisation live, and both have shipped defects; test
+them through `connector.connection`, the way the samples do.
+
+CI (`.github/workflows/tests.yml`) runs the whole suite on every PR against the SDK revision pinned
+in `.sdk-ref`. **A failing test blocks the merge** — the `pytest` check propagates the suite's exit
+code (https://github.com/Datarails/dr-connectors/pull/139). What is still advisory is the *colocated-test
+report*: a connector shipped with no test at all does not block the merge until Phase 3, and is still
+not acceptable to ship.
+
+## 2. Publish the artifact
+
+Publishing is now two tiers — **publish to JFrog (CI)**, then **ingest into the datacenter (dr-cli)**.
+The manual `az storage blob upload` + `/internal/artifacts` curl dance is **gone**.
+
+**The version is a git tag CI assigns — nobody derives it locally.** (This replaced a content-sha
+version, which wasn't reproducible across zlib/Python/Node, so the CLI's local sha never matched CI's
+and ingest 404'd. Now CI is the single version authority.)
+
+**Publish → JFrog (CI, automatic on merge).** `publish-connectors-jfrog.yml` composes every dataquery
+under each changed vendor, reserves the connector's next monotonic version by pushing a git tag
+`<dc>/<org_id>/<vendor>/<dq>/vN` (atomic CAS; `v1` if none exist, else max+1), and PUTs the artifact to
+`<dc>/<org_id>/<vendor>/<dq>/<vN>/artifact.zip`. It runs on a **GitHub-hosted** runner (JFrog sits
+outside every VNet). **Merge your connector branch to `main` first** — every merge that touches a
+connector cuts a new `vN` (no content dedup); the next step pulls that tag from JFrog.
+
+**Ingest → this datacenter (operator).** `dr connectors publish` (dr-cli **≥ 1.5.0**) makes a published
+version available in a datacenter. It does **NO local compose/hash**: it `git fetch --tags` and reads the
+**newest** `vN` for the connector (or takes `--tag vN`), then POSTs `/connectors/<name>/ingest` with just
+`{version: "vN"}`; the control-plane pulls that tag's bytes from JFrog into the DC's in-VNet blob and
+registers the `artifact_registry` row. **No zip is uploaded from your workstation.**
+
+```bash
+git -C C:/codegithub/dr-connectors fetch --tags origin   # so the newest tag is visible locally
+dr connectors publish "<datacenter>/<org_id>_(<org_name>)/<vendor>/dataqueries/<dataquery>.py" \
+  --server DEV --account <customer-user-email> --json     # needs NODE_EXTRA_CA_CERTS (VPN CA); returns {name, version: "vN", digest}
+# --tag vN  to ingest a specific version instead of the newest.
+```
+
+The returned `digest` is a **server-side integrity sha** the runner verifies pre-exec — it is NOT the
+version and you never supply it. Version everywhere (bindings, repoint, promote) is the tag `vN`.
+
+- `404 … not in JFrog` → **read the message; it names the one cause it can prove.** The store path is `<dc>/<your JWT org>/<vendor>/<dq>/<vN>/artifact.zip`, and the CLI compares the `<dc>` and `<org>` the control-plane echoes back against the dataquery path: a datacenter mismatch is a wrong `--server`, an org mismatch is a wrong `--account` (that 404s EVERY version of the connector), and when both match the tag is genuinely unpublished — merge to `main` so CI cuts+publishes it, then retry. A 404 with no store path in the body never reached the lookup at all: check `--connectors-url`/the host serves `POST /connectors/<name>/ingest`.
+- `409` → idempotent success (that `vN` is already ingested in this DC).
+- `no published versions found … no git tags` → the connector was never merged/published; merge first
+  (or pass `--tag vN`).
+
+**Repointing an existing binding is NOT a raw curl** — `/connector-queries` is *authenticated* (a raw
+in-cluster PATCH gets `401`), and the port-forward serves the SPA. Use the dr-cli, which carries the
+operator JWT: `dr connectors query repoint <queryId> --artifact-version vN --server DEV --account
+<customer-user-email>`. It reaches the **internal-only** control-plane host over VPN — if it errors `fetch failed`, the
+VPN's TLS-inspection cert isn't trusted by Node: set `NODE_EXTRA_CA_CERTS` to the gateway CA (do NOT
+port-forward as a workaround, and never `NODE_TLS_REJECT_UNAUTHORIZED=0`). `dr filebox`/`dr integrations`
+working while `dr connectors` fails is expected — those hit the public monolith, not the internal CP.
+
+## 3. Datasource + filebox (monolith, via dr-cli)
+
+Use `dr-cli` as the **impersonated customer user** (`$ACC` from *Identity* above — never the operator;
+the filebox's org is taken from that JWT). Build
+these **from scratch** — don't assume a prior connector left one behind. Three sub-steps, all mandatory:
+
+**Workato migration — one datasource per credential.** A datasource holds a **single** credential
+(one `config.apikey`). When migrating a customer off Workato and the source exposes **multiple
+credentials** (several API keys / accounts / entities the customer had as separate Workato
+connections), do **not** fold them into one datasource — create a **separate `DatarailsConnector`
+datasource per credential** (repeat step 3a for each), then give each its own filebox + `ConnectorQuery`
+binding (steps 3b–4). One secret per datasource keeps credentials isolated per the secret-separation
+standard (`tech/security.md` §4.1).
+
+**Ask the user before step 3a: does this data source authenticate with an API key or with OAuth?**
+Ask it as a question (AskUserQuestion when available) — do not infer it from the vendor's name. The
+connection file you wrote in §1 is a hint (one that reads `oauth_tokens` is OAuth, like the FreshBooks
+and QuickBooks connections under `dr-connectors/dev-1/69_(test-connectors.com)/`).
+
+**The one OAuth requirement: an OAuth data source MUST have `config.oauth` as a JSON object** — even
+an empty `{}`. That object is the only thing the UI checks before it shows the Authentication row
+and its Connect button; without it the implementer has nothing to click. After step 3a, confirm it
+landed: `dr integrations get <datasource_id> --json $ACC` shows `config.oauth` as an object.
+
+**Every OAuth question is optional — the user may skip any of them, and a skip never blocks the
+onboarding.** If the user skips the API-key-or-OAuth question itself, use the API-key payload, as
+before this question existed.
+
+- **API key / static token** — you hold the secret; step 3a's first payload, and the data source is
+  ready to sync as soon as it exists.
+- **OAuth** — the customer's grant is obtained by a person clicking **Connect** in the Datarails UI and
+  consenting at the vendor. `dr` cannot do that part. It writes the vendor app's `oauth` block (the
+  second payload), and the implementer authorises the data source in the UI afterwards (*OAuth: hand
+  off for authorisation*, below). Then offer, each one optional: the vendor's authorize URL, its token
+  URL, the scopes, the client id, and the client secret. Write only what was answered — leave out a
+  skipped key rather than writing a placeholder. With nothing answered, `oauth` is `{}`: the Connect
+  button still appears, disabled, and the dialog names the missing keys. Tell the user which keys are
+  still missing and who can fill them: client id/secret in the UI dialog; the two URLs only through
+  `dr integrations update` (they have no UI field).
+
+```bash
+ACC="--server DEV --account <customer-user-email>"   # the dr login-as identity from "Identity" above — NOT your @datarails.com operator login
+
+# a) Datasource — a DatarailsConnector integration holding the credential.
+#    is_sync_service MUST be true, or the monolith accepts the sync but never emits sync.requested (see Gotchas).
+#    Inject every secret from an env var; never echo/commit it. `integrations add` is payload-file only.
+#
+#    API key — the credential under `apikey`:
+python -c "import os,json;json.dump({'name':'<name>-connector','organization':<org_id>,'type':'DatarailsConnector','is_sync_service':True,'config':{'apikey':os.environ['<SECRET_ENV>'],'password':''}},open('ds.json','w'))"
+#
+#    OAuth — the vendor app under `oauth` / `oauth_secrets`, and NO apikey. The UI shows the
+#    Authentication row (Connect button) whenever `config.oauth` is an object — `{}` included — and
+#    enables Connect only once authorize_url, token_url, client_id and oauth_secrets.client_secret are
+#    all set. Every key below is optional: include only what the user answered, and drop
+#    `oauth_secrets` entirely when there is no secret. Add vendor extras to `oauth` here (QuickBooks:
+#    'capture_params':['realmId']).
+#    NEVER write `authorised`, `oauth_tokens` or `oauth_pending` — the UI and the monolith's OAuth
+#    callback own them.
+python -c "import os,json;json.dump({'name':'<name>-connector','organization':<org_id>,'type':'DatarailsConnector','is_sync_service':True,'config':{'oauth':{'authorize_url':'<vendor authorize url>','token_url':'<vendor token url>','scopes':'<space-separated scopes>','client_id':os.environ['<CLIENT_ID_ENV>']},'oauth_secrets':{'client_secret':os.environ['<CLIENT_SECRET_ENV>']}}},open('ds.json','w'))"
+#    ...with nothing answered, the OAuth config is just:  'config':{'oauth':{}}
+#
+dr integrations add --payload-file ds.json --json $ACC      # -> note the datasource id
+rm -f ds.json
+
+# b) Filebox — the sync target. Capture its `uuid` (that IS the binding's filebox_uuid — step 5).
+#    THE DATE TAG MUST MATCH THE SYNC-QUERY (step 3c). Decide it from the query, not by habit:
+#      * query HAS a date window (`from`/`to` rendered from `{{ date_tag|somonth|... }}`)
+#        -> give the filebox the matching date tag: --date-sub-type month (or quarter/year to match
+#           the window the query actually renders).
+#      * query has NO date window (reference/dictionary datasets — payers, HCPCS, modifiers, tags,
+#        users, payment methods …) -> create it with NO date tag at all: --no-date.
+#    Do NOT put a date tag on a filebox whose query is undated: every sync then lands a new version
+#    stamped with a period the data was never filtered by, so N periods all hold the same full
+#    snapshot and the tag is a lie. Equally, do NOT date-window a query whose filebox has no date
+#    tag — the window would narrow the data with nothing recording which period landed.
+#    Also pass --parent <folder_id>: a filebox created without it lands at the TENANT ROOT
+#    (`parent: None`), not under Actuals. Customers expect it nested (orgs commonly group by
+#    vendor, e.g. `Actuals/<Vendor>/<dataset>`). Fix an existing one with `dr filebox set-parent`.
+dr filebox create --name "<name>" --scenario Actuals --date-sub-type month --parent <folder_id> --json $ACC   # dated query
+dr filebox create --name "<name>" --scenario Actuals --no-date --parent <folder_id> --json $ACC               # undated query
+
+# c) support_api ON (REQUIRED before data-query set — else HTTP 400), then link the data-query to the datasource.
+#    config.query (the Sync-query) is JSON **TEXT** — a string, NOT a nested dict (DR-50411). Text is
+#    what every other query-as-text vendor stores and what the connectors Sync-query editor writes.
+#    objectType MUST live INSIDE that JSON — the runtime dispatches get_query_class on
+#    `query.get("objectType")` with NO top-level fallback, so `objectType` at the config top level
+#    comes through as None (Gotchas). Single-dataset connectors may default; multi-dataset ones raise
+#    "unknown objectType".
+#    ALSO PUT THE SELECTOR HERE — the ticker/account/entity/spreadsheet id this filebox pulls — so it
+#    is visible and editable in the Sync-query editor. NOT in the binding's query_config, which the UI
+#    does not render. See the note under this block.
+#    AND NAME THE LANDED FILE: every new DATED filebox gets a period-stamped `fileName` (see the note
+#    under this block). Omit the key on an undated filebox — there is no date_tag to render.
+dr documents support-api set <filebox_id> --on $ACC
+python - <<'PY'
+import json
+sync_query = {
+    "objectType": "table",                 # the dataset — MUST be inside config.query, not above it
+    "symbol": "<SELECTOR>",                # the per-filebox selector
+    "from": "{{ date_tag|somonth }}",      # fetch window; drop both on an undated (--no-date) filebox
+    "to": "{{ date_tag|eomonth }}",
+    "fileName": "<dataset>_{{ date_tag|somonth|format:'%Y-%m-%d' }}-{{ date_tag|eomonth|format:'%Y-%m-%d' }}",
+}
+json.dump(
+    {"connection": <datasource_id>, "config": {"query": json.dumps(sync_query), "type": None},
+     "sync_only_changes": True},
+    open("dq.json", "w"),
+)
+PY
+dr filebox data-query set <filebox_id> --payload-file dq.json --json $ACC
+rm -f dq.json
+```
+`dr integrations get <id> --show-secrets --json` shows the apikey (masked otherwise); `dr filebox
+data-query get <fb>` shows `connection` + config + status.
+
+**PUT EVERY SELECTOR IN THE SYNC-QUERY, NOT THE BINDING.** The **Sync-query** (`config.query`) carries
+`objectType`, any per-sync `{{ date_tag }}` window, **and every per-filebox selector — the value that
+says WHICH thing this filebox pulls**: ticker/symbol, account id, entity, company, spreadsheet id +
+range, report id, board, region. It is the ONLY one of the two config channels the monolith's Sync-query
+editor renders, so a selector parked in the binding's `query_config` leaves an operator looking at a
+filebox with **no way to see or change which ticker it fetches** — the filebox name is the only clue,
+and nothing keeps that name honest. Verified on us2/1599: five FMP fileboxes with `{"symbol":"NVDA"}`
+etc. in `query_config` showed an identical, symbol-less Sync-query in the UI for all five.
+
+Only plumbing an operator should never have to touch belongs in the binding's `query_config` (step 4):
+`base_url`, and tuning caps like `max_mb` / `lookback_months` / page size.
+
+**Never put the same key in both.** `req.config` merges the Sync-query **over** `query_config`
+(`{**query_config, **sync_query}`), so the binding's copy is silently ignored — and later repointing it
+looks like a no-op with no error anywhere. One key, one home.
+
+**NAME THE LANDED FILE AFTER ITS PERIOD — the convention for every new dated filebox.** Without
+`fileName` the landed version is called `<vendor>__<dataquery>__<objectType>.csv`, identical for every
+period, so a filebox's versions are told apart only by their date tag. Set the reserved **`fileName`**
+key in the Sync-query and each version arrives self-describing — and a customer migrating off Workato
+keeps the name their downstream reports were built on (`jobs_2026-08-01-2026-08-31.csv`). The key is
+read by `_output_blob_name` (`dr-connectors-runtime/src/runtime/activities.py`); `.csv` is appended for
+you.
+
+**Mirror the window the query already renders** — the same two expressions as `from`/`to`, plus a
+`format` filter:
+
+| Filebox date tag | `fileName` | Lands (August 2026) |
+|---|---|---|
+| `--date-sub-type month` | `<dataset>_{{ date_tag\|somonth\|format:'%Y-%m-%d' }}-{{ date_tag\|eomonth\|format:'%Y-%m-%d' }}` | `jobs_2026-08-01-2026-08-31.csv` |
+| `--date-sub-type year` | `<dataset>_{{ date_tag\|soyear\|format:'%Y-%m-%d' }}-{{ date_tag\|eoyear\|format:'%Y-%m-%d' }}` | `jobs_2026-01-01-2026-12-31.csv` |
+| `--date-sub-type week` | `<dataset>_{{ date_tag\|soweek\|format:'%Y-%m-%d' }}-{{ date_tag\|eoweek\|format:'%Y-%m-%d' }}` | `jobs_2026-08-31-2026-09-06.csv` |
+| `--date-sub-type quarter` | `<dataset>_{{ date_tag\|format:'%Y-%m-%d' }}` | `jobs_2026-09-30.csv` |
+| `--no-date` | **omit the key** | `<vendor>__<dataquery>__<objectType>.csv` |
+
+Rules that bite:
+
+- **Always pass `|format:'%Y-%m-%d'`.** `somonth`/`eomonth` return a *datetime*, and Django renders a
+  bare `{{ date_tag|somonth }}` through `DATETIME_FORMAT` — **`Aug. 1, 2026, midnight`** — which
+  sanitizes into `Aug__1__2026__midnight`. (In `from`/`to` the vendor tolerates it; in a filename the
+  customer sees it.)
+- **Quote the filter argument.** `|format:%Y-%m-%d` raises `TemplateSyntaxError: format requires 2
+  arguments, 1 provided` in the monolith's render — it does not degrade to an empty string, so the
+  trigger's render fails rather than the filename quietly falling back.
+- **There is no `soquarter`/`eoquarter` filter** — for a quarterly filebox use the tag's own date.
+- **Omit it on `--no-date`.** With no date tag there is no `date_tag` in the render context, so the
+  placeholders resolve to empty and every version lands as `jobs_-.csv`.
+- **Rendering happens monolith-side** (`render_connector_config` deep-renders every string in the
+  data_query config), so `dr connectors dev-fetch` and the local sandbox see the raw template. Text
+  still carrying `{{` is dropped by the runtime and the platform name is used — confirm the real name
+  from a landed version (`dr filebox versions <fb> --json`), never from a sandbox run.
+- **Sanitized, not rejected.** Anything outside `A-Za-z0-9_-` becomes `_`, capped at 120 characters, so
+  `Jobs Aug 2026` lands as `Jobs_Aug_2026.csv`. A name that sanitizes to nothing keeps the default.
+- **`fileName` is a reserved platform key** like `objectType`/`type`/`period` — a connector that reads
+  `config["query"]` as its own variables must exclude it.
+
+As JSON text; `dr connectors dev-fetch` warns if `objectType` is missing from it. Reads accept **both** shapes, so an
+older filebox still holding a nested object keeps syncing — `_sync_query_object`
+(`dr-connectors-runtime/src/runtime/activities.py:204`) passes a dict through and `json.loads` a
+JSON-object string (a non-JSON / non-object string degrades to `{}` with a warning). Write text for
+anything new; re-saving an old filebox through the connectors Sync-query editor converts it.
+
+### OAuth: hand off for authorisation
+
+Only for an OAuth data source. Steps 3b–3c and §4 do not need the grant, so run them. **§5 does**: a
+sync before the grant fails in the connection's auth gate (no `oauth_tokens.access_token`), and until
+then the monolith lists the data source as an error source (`requires_auth` with `authorised` not true).
+
+1. **Redirect URI.** The vendor app must list `https://<env host>/api/v2/data_integrations/oauth/callback`
+   exactly as written — one callback serves every vendor. The OAuth dialog shows the URI it will send.
+   A mismatch fails at the vendor's consent page, not in Datarails.
+2. **Hand off.** Stop and tell the user: *open Admin → Data sources → `<name>-connector` → Authentication
+   → Connect, and complete the consent at the vendor* (entering client id/secret in that dialog first if
+   step 3a left them out). If step 3a left out either URL, Connect stays disabled until someone writes
+   it with `dr integrations update` — say so in the hand-off rather than stopping. The consent must be
+   done in the customer's org — by the customer, or by the implementer impersonating them.
+3. **Confirm before §5.** `dr integrations get <datasource_id> --json $ACC` → `config.authorised` is
+   `true`. Continue only then.
+
+*Authenticated* means a grant is stored, not that its access token is live: the UI never refreshes it.
+The control-plane refreshes an expiring token through the monolith when a sync fetches credentials.
+**Disconnect** in the same dialog clears the grant and keeps the `oauth` block, so Connect works again
+without re-running step 3a.
+
+## 4. Create the binding (control-plane, via dr-cli)
+
+Two objects. **`secret_ref` IS the datasource_id string.** Do **NOT** raw-curl `localhost:8000` —
+`POST /connectors` and `POST /connector-queries` are **authenticated** (a raw in-cluster/port-forward
+POST gets `401`; the port-forward also serves the SPA). Use the dr-cli, which carries the operator JWT
+to the internal control-plane (`dr connectors connector create` / `query create` — added 2026-07;
+older skill copies show the now-broken curl):
+```bash
+# a) Connector (creds source). --org-id optional — the JWT org is used when omitted, and under the
+#    impersonated customer identity that IS the customer's org, so omitting it is correct. Note the returned id.
+dr connectors connector create --secret-ref <datasource_id> --json $ACC     # add --org-id <org> only to override the JWT org
+# b) ConnectorQuery (the binding) — connector-id from (a); filebox-uuid + filebox-id from step 3b
+#    (filebox create). --filebox-id is the numeric doc id (MANDATORY; the Sync-query editor's write
+#    key — no uuid->id resolver on the monolith). It's the same <filebox_id> used in steps 3a/5.
+dr connectors query create --connector-id <id> --filebox-uuid <uuid> --filebox-id <filebox_id> \
+  --artifact-name "<vendor>__<dq>" --artifact-version "<ver>" \
+  --runtime-flavor base --query-config '{"base_url":"https://api.vendor.com","max_mb":30}' --json $ACC
+```
+Both need `NODE_EXTRA_CA_CERTS` (VPN CA) like every `dr connectors` call. Verify with
+`dr connectors binding get <filebox_uuid> --json $ACC` (secret_ref + artifact + query_config resolve).
+The connector's `req.config` = `public_config` ∪ `query_config` ∪ the data-query's sync_query (its
+JSON text parsed to a dict by `_sync_query_object`), with the **sync_query last — it wins on any key
+collision**.
+
+`query_config` is for **plumbing only**: `base_url`, tuning caps (`max_mb`, `lookback_months`, page
+size). The **selector** — `symbol`/ticker, account, entity, spreadsheet id, report id — goes in the
+**Sync-query** (step 3c), because that is the only channel the monolith's Sync-query editor shows an
+operator. Note the example above deliberately carries NO `symbol`.
+**409 "filebox_uuid already bound"** = a binding exists; `GET /internal/bindings/<uuid>` to inspect,
+`PATCH /connector-queries/<id>` to repoint artifact/version/config.
+
+## 5. Trigger + verify
+
+**OAuth data source: do not trigger until `config.authorised` is `true`** (*OAuth: hand off for
+authorisation*, §3).
+
+The `filebox_uuid` is the `uuid` returned by `filebox create --json` in step 3b (confirmed: it equals
+the `data.filebox_uuid` the runtime stamps on the `sync.requested` event). `dr filebox get` does *not*
+expose it, and the Inngest event is a second way to read it if you didn't capture it.
+
+```bash
+dr filebox data-query sync <filebox_id> $ACC    # the REAL trigger (NO --json on this subcommand)
+```
+
+**Trigger a sync for a SPECIFIC period (date tag).** `--tag` takes a **raw JSON tag list** (it's
+passed verbatim as `multiple_tags` and `json.loads`'d server-side), NOT a friendly label. Shape:
+`[{"id":<dateTagId>,"name":"Month","value":<epoch_seconds>}]`. Get `<dateTagId>` from
+`dr filebox tags --json` (the DATE tag whose `options.sub_type` matches, e.g. **Month id=6** on a
+monthly filebox); `value` = epoch seconds of a date in that period (month-END is canonical — e.g.
+`2026-08-31` = `1788134400`). Repeat `--tag` (or call sync N times) to land N dated versions in the
+**same** filebox. Example — July then August 2026 into one filebox:
+```bash
+dr filebox data-query sync <fb> --tag '[{"id":6,"name":"Month","value":1785456000}]' $ACC  # 2026-07-31 -> July version
+dr filebox data-query sync <fb> --tag '[{"id":6,"name":"Month","value":1788134400}]' $ACC  # 2026-08-31 -> Aug  version
+```
+The selected tag drives BOTH the landed version's `date` tag AND (via the monolith's `query_tags`
+rendering) any `{{ date_tag|somonth|format:'%Y-%m-%d' }}` / `eomonth` placeholders in the Sync-query
+`from`/`to` — so each activation fetches that month's window. `period` is upload-only, never a fetch
+input (see the dr-connectors README "Passing variables to a query").
+
+Verify the whole chain (per-flavor — use the flavor's runner deployment):
+- **`sync.requested` fired** — via `datarails-connect-inngest` ($INNGEST_API), a `integrations/sync.requested`
+  event with your `datasource_id`. If **none** appears, the datasource isn't `is_sync_service:true` (Gotchas).
+- **Inngest run `Completed`** in ~15–20s (fn `integration-sync`, app `dr-connectors-runtime-dev`). A run
+  `Failed` with `output:null` = died in `load-sync-context` → no binding yet (create the `ConnectorQuery`).
+- **Runner ran the connector:** the hook runs in workload **`connectors-runner`** (NOT
+  `connectors-runner-base` — that name has no logs). It logs `INFO:dr.runner:executing hook
+  connect/fetch/post_fetch for <vendor>__<dq>/<ver>`, downloads the artifact, then runs `connect()`
+  (auth gate) → `fetch()`. **ALWAYS read runner logs + errors via `datarails-connect-groundcover`, not
+  kubectl** (workload `connectors-runner`, env e.g. `dev-germany-west-central`); grep the
+  `[<vendor>__<dq>]` prefix (see the logging gotcha) to cut through the shared-env noise. Groundcover is
+  the source of truth for what the connector did and any traceback — reach for `kubectl logs` only as a
+  last resort if Groundcover is unavailable. (The hook's completion **verdict** — `retry`/`block`, e.g. a
+  `sandbox timeout` — is separate from the log stream; read it from the Inngest run `output` or the
+  completion store, per Gotchas.)
+- **Data landed:** `dr filebox versions <filebox_id> --json` shows a new version per activation, each
+  carrying its selected date tag (`tags: [{"name":"Month","label":"August 2026",…}]`) — the fastest way
+  to confirm two syncs produced two correctly-dated versions. **Read the landed CSV itself** — it is the
+  only check that proves the connector emitted the right rows:
+
+  ```bash
+  dr filebox download <filebox_id> --output out.csv --force $ACC   # latest version
+  dr filebox download <filebox_id> --doc-version <version_id> --output out.csv --force $ACC
+  dr filebox preview <version_id> $ACC                             # VERSION id, not filebox id
+  ```
+
+  **`preview` takes a VERSION id while `download` takes a FILEBOX id.** Passing the filebox id to
+  `preview` 404s with `No DocumentVersion matches the given query` — that is the argument being wrong,
+  not the command being broken. Get the version id from `dr filebox versions <filebox_id> --json`.
+  The Inngest event's `rendered_query.query` (`from`/`to`) still tells you the fetch **window**, which
+  the bytes do not.
+
+Note the runner pulls the connector **from the blob artifact by version (the `vN` tag)** — it does not read git,
+so once a `vN` is ingested the runner keeps executing exactly those bytes regardless of later branch state. That is
+**not** a way to skip the merge: the only path to a `vN` is CI publishing it to JFrog on merge to `main` (§2), so an
+unmerged connector has no version to ingest, bind, or run.
+
+## Gotchas (each one cost real time)
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| Sync `Completed` and a version landed, but the **customer can't see the filebox** / the data is in the wrong org | The `dr-cli` identity was the **operator** (`@datarails.com`), not the customer. Org comes from the JWT and `dr filebox create` has no `--org` flag, so the filebox was created in the operator's org — while `dr integrations add` (whose payload names `organization`) put the datasource in the customer's. Nothing errors, at any step. | `dr login-as <customer-user-email> --server <X>`, confirm `dr whoami --account <customer-user-email>` prints `Impersonated by: <operator>`, then recreate the filebox under that identity. Recreating mints a **new `uuid`**, so the `ConnectorQuery` binding must be recreated too (§4). See *Identity*. |
+| `dr filebox data-query sync` returns `server: OK` but **no `sync.requested`** ever appears (trigger-consumer logs idle; run never starts) | The datasource has **`is_sync_service: false`** — the monolith takes the legacy path and never publishes the connector trigger. `integrations add` defaults it to false. | Set it true: `dr integrations update <id> --payload-file` with `{"…","is_sync_service":true,"config":{"apikey":…}}` (re-send the apikey — the PUT is full-replace). Re-trigger. **This is the #1 silent no-op.** |
+| `data-query set` → HTTP 400 `Support API have to be switch to ON for this integration` | filebox `support_api` flag is off | `dr documents support-api set <filebox_id> --on` first, then re-run `data-query set`. |
+| Sync fires but **nothing** in queue/runtime/Inngest | Triggered the wrong thing — `dr integrations sync <id>` does a monolith resync and emits **no** `sync.requested` | Trigger the **filebox**: `dr filebox data-query sync <filebox_id>`. |
+| Run `Failed` in ~1s, `output: null` | died in `load-sync-context` → no binding | `GET /internal/bindings/<uuid>` returns 404 → create the `ConnectorQuery`. |
+| Connector `KeyError` / "missing required variable" on a config key | The key is in neither channel | Put the **selector** (`symbol`, account, entity, spreadsheet id) in the **Sync-query** (`config.query`, step 3c) and **plumbing** (`base_url`, `max_mb`, …) in the binding's `public_config`/`query_config` (step 4). |
+| Operator opens the Sync-query editor and **cannot see which ticker / account / entity the filebox pulls** | The selector was put in the binding's `query_config`, which the monolith's Sync-query editor does not render — it shows only `config.query` | Move it into the Sync-query and clear it from `query_config` so there's one home. Order matters: `data-query set` the Sync-query WITH the selector first, then `query repoint <id> --query-config '{…}'` to drop it — never the reverse, or a sync in between has no selector at all. Verified on us2/1599 (5 FMP fileboxes). |
+| Changing a selector in the binding's `query_config` has **no effect**, no error | The same key is also in the Sync-query, which merges LAST and wins (`{**query_config, **sync_query}`) | Keep one key in one channel. Check with `dr filebox data-query get <fb>` (Sync-query) vs `dr connectors binding get <uuid>` (query_config) and delete the loser. |
+| `dr connectors publish` → `404 … not in JFrog` | Three causes, and the message now names the one the control-plane's own store path proves. The path is built as `{dc}/{caller's JWT org}/{vendor}/{dq}/{version}/artifact.zip` (`routes_admin.py` `ingest_version`) — the `dc` is that control-plane's `K_ENV_FULL` and the org segment is YOURS, never the artifact's, so a wrong `--server` or `--account` 404s every version and reads like a JFrog outage (cost a day on DR-52228). Nothing is composed or hashed locally, so it is never a local-vs-CI content mismatch. | Do what the message says — it has already compared both segments. `wrong environment` → re-run with the `--server` it names. `wrong --account` → `dr whoami --server <S> --account <A> --json` and re-run under an account whose `organization_id` matches the `<org_id>_(<name>)` folder. `the login is right` → merge to `main` so CI cuts the next `vN`, `git fetch --tags origin`, then re-run (or pass `--tag vN`). Proof that resolution works: a correct account returns `409 alreadyIngested` for an already-ingested version. |
+| `dr connectors publish` aborts with `is published under org <A> … you are authenticated as org <B>` and never calls the control-plane | Not an error in the command — the pre-flight caught the DR-52228 mismatch before the POST. The org folder in the dataquery path is the org CI published under; your JWT org is what the control-plane would look up. | Re-run with `--account` for an org `<A>` user. If the message also names a different `--server`, fix that instead — a wrong datacenter mismatches the org identically. Compared as **strings**, so `001758_(x)` is rejected against org `1758`: rename the folder to the canonical id. The guard fails open (an unreadable `organization_id` claim lets the publish through), so it never blocks a publish the server would have accepted. |
+| creds empty | stored under `password` (blanked) | store the credential under `apikey` (or another non-`password` key). |
+| OAuth data source: the edit dialog shows **no Authentication row / no Connect button** | `config.oauth` is missing or not an object — the data source was created with the API-key payload. The UI decides the row on `config.oauth` alone (dr-fe-app `isOAuthConnector`). | `dr integrations update <id> --payload-file` with the OAuth config from step 3a (full replace — keep `is_sync_service: true`). |
+| OAuth data source: **Connect is disabled** | One of `oauth.authorize_url`, `oauth.token_url`, `oauth.client_id`, `oauth_secrets.client_secret` is empty; the dialog lists the missing keys. | Enter client id/secret in the dialog; the two URLs have no UI field — write them with `dr integrations update` (step 3a's OAuth payload). |
+| OAuth data source was *Authenticated*, now *Not Authenticated* right after a `dr integrations update` | The PUT is **full-replace**: a payload without `oauth_tokens`/`authorised` wipes the grant. | Re-authorise in the UI (Connect). Before an update, read the current config with `--show-secrets --json` and send it back whole, or avoid updating an authorised OAuth data source from `dr` at all. |
+| Consent at the vendor fails with a redirect-URI / `redirect_uri_mismatch` error | The vendor app does not list this environment's callback exactly. | Register `https://<env host>/api/v2/data_integrations/oauth/callback` at the vendor (the dialog shows the value), then Connect again. |
+| `GET /connector-queries` → 405 | no list endpoint | inspect a specific binding via `GET /internal/bindings/<filebox_uuid>`. |
+| `connect` fails `ModuleNotFoundError: No module named 'connection'`; run retries the hook ~5× then dies | The bound artifact is a **pre-split** single-file `connector.py` (old `ConnectorFunction`/`CONNECTOR` model, version like `1.0.0` — not a digest, often no repo source) — the runner now imports the canonical `connection.py`+`query.py`. | Re-author in the split layout (`DataIntegrationConnection`+`DataQuery`), merge (CI cuts a new `vN`), then `dr connectors query repoint <queryId> --artifact-version vN`. |
+| Run `Failed`; runner logs show the hook executing repeatedly with **no traceback** | The failure is in the hook's completion **verdict** (`retry`/`block`), not the runner log. | Read it from the completion store in-cluster — a port-forward returns the SPA, so `exec` into the connectors runtime pod and fetch `/internal/completions/<inngest_runid>-<hook>-0-0` from the control-plane service directly. The `error` field has the sandbox stderr/traceback. Namespace, workload and service names are in the internal connectors runbook. |
+| `dr connectors …` → `error: fetch failed` (but `dr filebox`/`dr integrations` work) | The connectors control-plane is an **internal-only** host (VPN/in-cluster); the monolith commands hit the public host, so their working says nothing about CP reachability. `fetch failed` = Node can't reach/verify it. | Ensure on VPN **and** `NODE_EXTRA_CA_CERTS` points at the gateway CA — and that it's set **persistently** (`setx`), because tool/non-interactive shells only inherit persistent env vars, not one set for the current interactive session. |
+| `dr filebox preview <filebox_id>` → `404 No DocumentVersion matches the given query` | `preview` takes a **version** id; `download` takes a **filebox** id. Passing the filebox id to `preview` looks up a version that does not exist. | `dr filebox versions <filebox_id> --json` for the version id, then `dr filebox preview <version_id>`. |
+| `dr filebox data-query get`/`versions --json` shows `&#x27;` (or `&quot;`) in config strings | The monolith serializer HTML-escapes string values on **read**; the DB stores raw quotes. | Cosmetic/display only — placeholder rendering still works. Don't "fix" the stored value; if you need the true string, HTML-unescape the output. |
+| New artifact code doesn't run even after publish | The runtime resolves the **active-version pointer** (`connector`+`org`), not your latest publish. | `dr connectors publish <dq.py>` then `dr connectors promote <name> --artifact-version vN` (both need `NODE_EXTRA_CA_CERTS`). Confirm: in-cluster `GET /internal/connectors/<name>/active?org_id=<org>`. |
+| `dr filebox get` never shows the filebox `uuid` | Not exposed by that command. | Read `filebox_uuid` from the `sync.requested` event (`datarails-connect-inngest`), or capture it from `dr filebox create --json`. |
+| Creating a binding via `curl localhost:8000/connectors` (or `/connector-queries`) → `401` / SPA HTML | Those POSTs are **authenticated** now; the port-forward serves the SPA. There is no unauthenticated create path. | Use the dr-cli (carries the JWT): `dr connectors connector create --secret-ref <ds>` then `dr connectors query create --connector-id <id> --filebox-uuid <uuid> --artifact-name … --artifact-version … --query-config '{…}'`. See step 4. |
+| Connector's own `log.info(...)` lines never appear in the runner logs / Groundcover (but `dr.runner` INFO does) | The hook runs in a **sandbox whose root logger has no handler**, so bare `logging` is dropped (only WARNING+ via lastResort, and even that may not surface). | In the connector, attach an explicit `logging.StreamHandler(sys.stdout)` to your logger (idempotently), set level INFO, and format with a `[<vendor>__<dq>]` prefix so lines are greppable in the shared `connectors-runner` pod. Log auth (never the secret), the window, per-page progress, row count. |
+| A credential reaches Groundcover through a log line that logs no secret | The connector logged the raw `base_url`. It is **operator-supplied binding config**, so it can carry `user:pass@` userinfo or a credential in the query string — and the runner log is a shared multi-tenant stream. 24 connectors in dr-connectors currently do this (harmless for the keyless public sources, not for the credentialed ones). | Log only the hostname: `urlsplit(base_url).hostname` drops userinfo, port, path and query, which is all a log line needs to identify the environment. `samples/sample_streaming_fanout/` and the databricks connections model this. |
+| `get_query_class` receives `None` / multi-dataset connector raises "unknown objectType" / wrong dataset runs | `objectType` was placed at the data-query **config top level** instead of **inside `config.query`**. The runtime reads `dq["query"].get("objectType")` with **no top-level fallback**, so it comes through as `None`. | Put `objectType` inside the Sync-query, which is JSON **text**: `config.query = '{"objectType":"<dataset>"}'` (drop any top-level `objectType`). `dr connectors dev-fetch` warns when it's missing. Single-dataset connectors survive via their default; multi-dataset ones fail. |
+| Sync-query written as a nested **dict** (the pre-DR-50411 shape) | `config.query` was the only vendor field held as an object. It still syncs — `_sync_query_object` accepts a dict — but the monolith's legacy filebox-settings textarea binds the value directly, so an object rendered there as the literal **`[object Object]`** and a Save wrote that string back over the query. Monolith PR https://github.com/Datarails/dr-datarails-server/pull/7390 took `DatarailsConnector` out of `list_of_integrations_with_dynamics_fields` to put it on the free-text path. | Write `config.query` as JSON text (step 3c). There is **no backfill command** — the dropped `connector_sync_query_to_text` is referenced in an older control-plane commit message but never shipped — so an existing filebox converts only when re-saved through the connectors Sync-query editor. |
+| Sync `Completed` but the landed file has only the **last page** of rows (and no CSV header) | The connector returned `has_more=True`/`next_cursor` (multi-batch pagination). `post_fetch` is fed only `batch_keys[-1]` — the last batch (Phase-2 assembly unimplemented, `activities.py:204-236`); and `write_output`'s `commit_block_list` replaces `req.output_sas` each call. So every batch overwrote the prior and only the last survived. | Paginate **inside one `fetch()`** with a single `write_output` (header once + one block per page), `has_more=False`. See §1. Verified: this is the only way to land a complete multi-page CSV on the current runtime. |
+| Run fails with `output` = `{"error":"sandbox timeout after 2700s; stderr: …","verdict":"block","category":"timeout"}` (can't be cancelled until the step ends) | The **single** `fetch()` step ran longer than the runner's **2700s (45 min) hook timeout** — too many sequential vendor page GETs for one step. This **blocks rather than retries**: one kill, one reported failure, and the filebox's `Singleton` is released then rather than after `MAX_ATTEMPTS` budgets. | Reduce the work in that one step: (1) larger vendor `page_size` **if the vendor allows** (some vendors cap it — probe + **log vendor 4xx**); (2) bounded-concurrency page GETs inside the one `fetch`; (3) narrow the sync date window. Do NOT "fix" it by returning `has_more` — that trades the timeout for silent data loss (row above). To recover a stuck run: merge a fix (new `vN`) → `dr connectors query repoint <queryId> --artifact-version vN` → cancel the run (wait for `Cancelled` — a live run holds the `Singleton(filebox_uuid)` so the monolith **skips** a new sync) → re-trigger. |
+| Filebox versions all carry a date tag (e.g. `Month: August 2026`) but every version holds the same full snapshot | A **date tag on a filebox whose Sync-query has no date window** — a reference/dictionary dataset (payers, HCPCS, modifiers, tags, users). The tag is applied at upload from the selected period regardless of what was fetched, so it records a period the data was never filtered by. | Match the tag to the query: `--no-date` at create for an undated query; a dated tag only when the query renders a `{{ date_tag| … }}` window. On an existing filebox, drop the tag with `dr filebox set-tag-config <id> --remove <TagName>` (or recreate it with `--no-date` — note that mints a NEW `uuid`, so the `ConnectorQuery` binding must be recreated too). |
+| Confused about publish vs ingest | CI publishes to JFrog on merge (cuts the git tag `vN`); the CLI ingests a published version into a DC | `dr connectors publish <path/to/dataqueries/<dq>.py> --server DEV --account <customer-user-email> --json` reads the newest git tag (`git fetch --tags` + max `vN`, or `--tag vN`) — **no local compose** — then the control-plane **pulls that tag from JFrog** (no upload) and registers it (needs `NODE_EXTRA_CA_CERTS`); returns `{name, version: "vN", digest}`. |
+| Run `Failed`, `connect` completion `verdict: ok` but `fetch` completion `verdict: retry`, error `'async for' requires an object with __aiter__ method, got function` | Passed the async-generator **function** to `write_output` — `write_output(req, gen)` instead of **calling** it `write_output(req, gen())`. `write_output` does `async for chunk in <arg>`, which needs an async **iterator**, not a function. (The connect gate passing but fetch dying instantly is the tell.) | Call the generator: `write_output(req, gen())`. The golden template already shows `write_output(req, rows())` / `write_output(req, blocks())` — match it exactly; the missing `()` is easy to drop. |
+| Vendor **4xx** (e.g. `402 Payment Required`, `401`, `403`, bad-symbol `404`) makes the `fetch` completion `verdict: retry` and the run **retries forever**; the HTTP status shows ONLY in the completion store, never in the Inngest run `output` (which is `null`). | A bare `r.raise_for_status()` raises `httpx.HTTPStatusError` for **every** status ≥ 400, and the runtime classifies that as **transient → retry**. But a plan/auth/bad-request 4xx will never succeed on retry. | Log the status+body (skill §1), then for non-retryable statuses (`400/401/402/403/404/422`) `raise PermanentError(...)`; leave `429` + `5xx` to `raise_for_status` (genuinely transient). Concrete case: FMP's free tier answers **402** when `limit` exceeds its historical allowance (dev key caps statements at the last **5** annual records) — a small `limit` or a paid key fixes that one; the PermanentError stops the infinite loop for the general case. The golden `sample_http_vendor` template now shows this guard. |
+| Connector works perfectly in local/dev testing, then in production `fetch` dies with `MemoryError`, `MemoryError: Unable to allocate output buffer`, an SSL/crypto allocation failure, or `RuntimeError: can't start new thread` — often within a second of "fetch start", and deterministically on every one of the 5 attempts | The sandbox caps **virtual address space at `RLIMIT_AS` = 512 MB** (`dr-connectors-runner/src/runner/sandbox_entry.py`), and the runner pod's limit is also 512Mi. The guard is `if sys.platform != "win32"`, so **the ceiling does not exist on a Windows dev box** — a buffering connector cannot fail locally. Two causes, usually together: (a) the fetch accumulates the result set (a `rows` list, `asyncio.gather` over every parent, a full CSV string + its `.encode()` copy) — journal_lines measured **378 MB live to emit 67 MB**, 243 MB of it row dicts at 846 B/row; (b) asyncio's **default** DNS ThreadPoolExecutor grew to `min(32, node_cores + 4)` threads × 8 MB stack ≈ **256 MB of address space that is never reclaimed**, which is why a brief concurrency spike poisons every later phase of the run. | Stream: do the fetching **inside** the generator passed to `write_output`, replace gather-over-everything with a bounded worker pool feeding a bounded `asyncio.Queue`, format rows to CSV bytes immediately (never hold dicts), and re-batch to ~4 MB blocks. Cap the executor once: `asyncio.get_running_loop().set_default_executor(ThreadPoolExecutor(max_workers=4))`. Full rules in §1 *Memory*; working template `samples/sample_streaming_fanout/`. Reproduce locally in `python:3.12-slim` under the same rlimits — never trust a Windows run for this. |
+| Hook dies with no traceback and no `MemoryError`; runner reports `sandbox exited -24` | **`RLIMIT_CPU = 60s`** (`sandbox_entry.py`, right beside the memory limit) — 60 seconds of **CPU** time, then SIGXCPU/SIGKILL. Async I/O waiting is free, but TLS handshakes, gzip and JSON parsing across thousands of requests are not. | Cut CPU per row (fewer/larger vendor pages, `onlyData` where the vendor supports it, no per-row dict churn) and log `time.process_time()` next to wall clock so proximity to the cap is visible before it kills a run. |
+| A sync `Completed` and the file **looks** complete, but the customer's totals are short | A page-limit "safety cap" (`_LINE_PAGE_LIMIT`, `_HEADER_PAGE_LIMIT`, …) tripped and the code **logged a warning and returned what it had**, so a partial result landed as if whole. Same failure class as a missing offset follow-up: journal_lines silently dropped **113,171 rows** of one customer's month this way. | Caps must `raise PermanentError`, never warn-and-return — a failed sync is recoverable, a silently partial landing gets reconciled against and believed. Audit every `log.warning` that is followed by `return`. |
+
+## Env quick reference (dev germany)
+
+- Artifacts blob: `connectors255d8089` / container `artifacts`; account key in KV `vault-255d8089` secret `connectors-access-key`.
+- Control-plane: `svc/connectors-control-plane-load-balancer:8000` (ClusterIP → port-forward). Endpoints: `POST /connectors/{name}/ingest` (pull from JFrog + register), `/connectors`, `/connector-queries`, `PATCH /connector-queries/{id}`, `/connectors/{name}/active`, `/internal/bindings/{filebox_uuid}`, `/internal/get_credentials`.
+- Queues: `connector-jobs-<flavor>` (base worker consumes `connector-jobs-base`).
+
+## Related
+
+- `dr-switch-user` — the `dr login-as` impersonation flow (see *Identity*): operator selection, target
+  resolution, and how denials surface.
+- `dr-cli` — datasource/filebox/integration commands.
+- `datarails-connect-inngest` — read the sync.requested event (for `filebox_uuid`) + run status.
+- `datarails-connectors-add-flavor` — add a flavor if the connector needs new deps.