@syncmatters/connector-sdk 1.0.0 → 1.0.1

package/docs/05-query.md

@@ -0,0 +1,151 @@
+# `query()` — paging, checkpoints and filters
+
+## The lifecycle
+
+The platform calls `query()` repeatedly. First call: `options.queryState` is `undefined`.
+Each return's `queryState` is echoed into the next call, until you return `finalPage: true`
+(returning no rows has the same effect).
+
+```js
+/** @param {SDK.QueryOptions} options @returns {Promise<SDK.QueryPage>} */
+async query(options) {
+  const queryState = options.queryState || { offset: 0, limit: 200, rowCount: 0, startTime: Date.now() };
+
+  const resp = await this.#call(this.#buildRequest(options, queryState));
+
+  /** @type {SDK.Row[]} */
+  const rows = resp.records.map((r) => ({ meta: { key: String(r.id) }, data: r }));
+
+  queryState.offset += rows.length;
+  queryState.rowCount += rows.length;
+
+  /** @type {SDK.QueryPage} */
+  const result = { rows, queryState };
+  const requestedLimit = options.queryOptions?.limit;
+  result.finalPage =
+    rows.length === 0 ||
+    rows.length < queryState.limit ||
+    (requestedLimit != null && queryState.rowCount >= requestedLimit);
+
+  if (options.queryOptions?.checkpointFilter && result.finalPage) {
+    // ALWAYS overlap: emit a checkpoint EARLIER than "now" so rows modified while this
+    // query ran are re-read next time. 60s-5min overlap is the established range.
+    result.checkpoint = new Date(queryState.startTime - 5 * 60 * 1000).toISOString();
+  }
+  return result;
+}
+```
+
+Rules that follow from the lifecycle:
+
+- `queryState` is yours — an opaque cursor (`offset`, `nextPageToken`, `lastId`, ...). Keep it
+  JSON-serializable.
+- Respect `options.queryOptions.limit`: stop at it (`finalPage: true`) and trim overflow rows
+  from the last page.
+- Prefer **keyset pagination** (`WHERE id > lastId ORDER BY id`) over offsets when the API
+  allows — offset/`after` cursors can skip rows when records are deleted mid-pagination.
+
+## `Row`
+
+```js
+{
+  meta: {
+    key: "42",             // REQUIRED - unique row id; the value of the isKey field, as a string
+    deleted: true,          // only on rows reporting a deletion
+    url: "https://app.example.com/contacts/42",  // optional deep link shown in the UI
+    previousKeys: ["41"],  // optional - prior keys (e.g. after a merge)
+  },
+  data: { id: 42, email: "a@b.co" },   // the payload; property names = queryFields ids
+  file: fileProvider,      // LEGACY - do not emit; modern connectors return files as fields of type "file" IN data (13)
+  relationship: { srcRowId: "7", srcObjectId: "companies", relationshipId: "companies-companyId" },
+}
+```
+
+## `QueryOptions`
+
+| Field                                       | Meaning                                                                                                                      |
+| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| `id` / `meta`                               | object id being queried / its `ObjectMeta` (with your `data` payload)                                                        |
+| `log`                                       | platform logger                                                                                                              |
+| `objectMeta(id)`                            | fetch cached `ObjectMeta` of ANY object on the connection (for related/lookup objects)                                       |
+| `queryState`                                | your cursor from the previous page (undefined on the first call)                                                             |
+| `queryOptions.limit`                        | max rows the caller wants                                                                                                    |
+| `queryOptions.fields`                       | field paths the caller needs (used to fetch less when the API supports it)                                                   |
+| `queryOptions.file`                         | LEGACY - ignore; modern connectors include file content when `fields` contains a file-typed field's path (13)                |
+| `queryOptions.idsFilter`                    | `string[]` of row keys — return exactly those rows; row with the key not found? omit it, do not report an error              |
+| `queryOptions.checkpointFilter`             | `{ value?: string }` — rows changed since your last checkpoint (undefined value = first differential run: return everything) |
+| `queryOptions.matchFilter`                  | find rows matching people/records — below                                                                                    |
+| `queryOptions.relatedFilter`                | rows related to another object's rows — below                                                                                |
+| `queryOptions.randomFilter`                 | random sample up to `limit`                                                                                                  |
+| `queryOptions.rowFilter` / `propertyFilter` | your system-specific filters as declared in `meta().queryFilter`                                                             |
+
+Throw `SDK.ConnectorError(SDK.ErrorCode.InvalidFilterParameter, ...)` when asked for a filter
+the object didn't declare — it's the single most-thrown error code in the fleet, and it fails
+fast instead of returning wrong data.
+
+### idsFilter
+
+```js
+const ids = options.queryOptions?.idsFilter ?? [];
+if (ids.length > 0) where.push(`id IN (${ids.map((i) => `'${escape(i)}'`).join(",")})`);
+```
+
+Escape values you interpolate into query languages (backslash first, then quote), and skip
+null/empty ids so you never emit `id = 'undefined'`.
+
+When the row does not exist on the target system (e.g. deleted), do not report an error (e.g. API returns 404), simply do not return the row.
+
+### checkpointFilter (differential sync)
+
+The contract is a round-trip: you emit `checkpoint` (any string — ISO date is conventional) on
+the final page; the platform stores it and hands it back as `checkpointFilter.value` next run.
+
+- **Always subtract an overlap window** (60s–5min) from the query's start time when emitting —
+  rows modified while your query ran would otherwise be lost forever. Re-reading a few rows is
+  free; losing one is a data bug.
+
+### matchFilter
+
+`{ rule, selectOrder?, destFieldPath?, srcData: [{ srcRowId, match: { id?, name?, email?, domain?, firstname?, lastname?, custom? } }], canUse }`.
+
+For each `srcData` entry, search rows matching per the `rule` (e.g. `"email[ci]"` — compare
+case-insensitively). Collect candidates, then call `await options.matchFilter.canUse([{ srcRowId,
+candidates }])` — the platform reserves un-mapped matches and returns the selected rows with
+`row.relationship` already populated; return those. Skip src entries with empty match values.
+
+- **name[ci]** - inspect the `match.name` values and search for rows with the same `name`. The determination of which field holds name is made by the connector, e.g. for a company it may be `companyName`, for a deal it may be `title`.
+- **email[ci]** - inspect the `match.email` values and search for rows with the same `email`. The determination of which field(s) holds email is made by the connector, e.g. for a contact it may search `email` and `alternativeEmails`.
+- **domain[ci]** - inspect the `match.domain` values and search for rows with the same `domain`. The determination of which field(s) holds email is made by the connector, e.g. for a comapny it may search `website`.
+- **first_and_last_name[ci]** - inspect the `match.firstname` and values and `match.lastname` and search for rows with the same name.
+- **id** - inspect the `match.id` value and search in the selected `destFieldPath` (path to the field on the target system) to find rows holding this exact value in this field. This search is case sensitive.
+- **field_value_equals[ci]** - inspect the `match.custom` value and search in the selected `destFieldPath` (path to the field on the target system) to find rows holding this value (case insensitive) in this field.
+
+### relatedFilter
+
+Receives rows from a source object (`otherObjectId`) that has a defined relationship to this object, and collect the related rows from this object.
+
+`{ otherObjectId, otherRelationshipId, otherRows }`. Resolve the rows from a relationship declared in
+`meta()` (your `relationship.data` typically holds the FK field), turn `otherRows` into an id
+lookup, and return rows tagged with
+`relationship: { srcRowId, srcObjectId: otherObjectId, relationshipId: otherRelationshipId }` —
+one row instance per source row it relates to.
+
+## Cross-page scratch state: `kvStore`
+
+When a query needs working data too big for `queryState` (dedup sets across thousands of
+pages, id→row indexes built once and read per page), stage it in a key-value collection:
+
+```js
+const kv = SDK.utilities.kvStore();
+const seen = await kv.open(`seen-${options.id}`);      // get/put/del/iterator
+if (await seen.get(row.meta.key)) continue;            // already emitted on an earlier page
+await seen.put(row.meta.key, true);
+```
+
+Keep `queryState` itself small and JSON-serializable; the kv collection carries the bulk.
+
+## Reporting deletions
+
+Rows that represent deletions carry `meta: { key, deleted: true }` — used in checkpoint streams
+(when the API exposes deletions) and by `handleParsedEvent`
+([07-events-webhooks.md](./07-events-webhooks.md)).

package/docs/06-upsert-delete.md

@@ -0,0 +1,194 @@
+# `upsert()`, `delete()` and write hygiene
+
+## `upsert(options)` → `Promise<SDK.Row[]>`
+
+`options` = `{ log, id, meta, objectMeta, rows }`. `rows` are write payloads shaped by your
+`upsertFields`. Return rows in **query-result format** (`{ meta: { key }, data }`) so keys and
+URLs flow back to the caller.
+
+**The cardinal rule: the returned array is positionally aligned with `options.rows`.**
+`result[3]` is the outcome of `options.rows[3]`, whatever batching you did internally. The
+proven pattern when the API splits inserts and updates:
+
+```js
+const inserts = [],
+  updates = [],
+  where = new Map();
+options.rows.forEach((row, i) => {
+  if (row.id != null) {
+    updates.push(row);
+    where.set(i, { list: "u", at: updates.length - 1 });
+  } else {
+    inserts.push(row);
+    where.set(i, { list: "i", at: inserts.length - 1 });
+  }
+});
+const uOut = updates.length ? await this.#writeBatch("update", updates) : [];
+const iOut = inserts.length ? await this.#writeBatch("insert", inserts) : [];
+return options.rows.map((_, i) => {
+  const w = where.get(i);
+  return w.list === "u" ? uOut[w.at] : iOut[w.at];
+});
+```
+
+If the API's batch response can't be correlated back to an input row (no id echo, no ordering
+guarantee), **throw** rather than guess — silent misalignment corrupts data downstream.
+
+### Batching
+
+- Slice into the API's documented batch size: `rows.slice(i, i + batchSize)` per request.
+- Separate batchable from non-batchable payloads if the API distinguishes them; run both, then
+  re-align by index.
+- Implement `optimalBatchSize()` to tell the platform your preferred sizes per operation
+  (`"upsert"`, `"idsFilter"`, ...) — return `{ size, batchType? }` with `batchType` `"atomic"`,
+  `"concurrent"` or `"splitInsertUpdateAtomic"`.
+
+### add vs update hints
+
+When the caller already knows whether a row exists it stamps a flag on the row (attached via a
+Symbol — invisible to serialization):
+
+```js
+const flags = SDK.utilities.upsertFlags.get(row); // { operation?: "add" | "update" }
+```
+
+### API-reported failures
+
+Inspect the API's per-row results. When a row fails, log the response and throw
+`SDK.ConnectorError(SDK.ErrorCode.ApiReportedError, "<api> <op> reported an error: " + JSON.stringify(detail))`.
+(If your API supports partial success and the platform batch mode is `"atomic"`, failing the
+whole batch by throwing is correct; don't fabricate per-row placeholders.)
+
+## `delete(options)` → `Promise<SDK.Row[]>`
+
+`options.rows` carry the identifying fields you declared in `deleteFields` (usually just the
+key). Return the deleted rows with `meta: { key, deleted: true }`.
+
+```js
+async delete(options) {
+  const ids = options.rows.map((r) => String(r.id));
+  await this.#call({ method: "DELETE", url: `contacts?ids=${ids.join(",")}` });
+  return options.rows.map((r) => ({ meta: { key: String(r.id), deleted: true }, data: { id: r.id } }));
+}
+```
+
+## `upsertClean(options)` — pre-write validation and change detection
+
+Declare `canUpsertClean: true` and implement to let the platform validate + diff a row BEFORE
+writing (skipping no-op writes and surfacing issues early). The platform does the heavy
+lifting; the canonical implementation — and the ORDER matters, see below — is:
+
+```js
+/** @param {SDK.UpsertCleanOptions} options @returns {Promise<SDK.UpsertCleanResponse>} */
+async upsertClean(options) {
+  const issues = [
+    ...(await options.verifyFieldOptions({ cleanAction: "DeleteUpsertField" })),
+    ...(await options.verifyFieldConstraints({ timeZone: "UTC" })),
+  ];
+  const change = options.detectFirstChange();
+  return { hasChanges: change !== undefined, change, issues };
+}
+```
+
+- `verifyFieldOptions` validates picklist values against `optionValues`. It can **mutate the
+  upsert**: `cleanAction: "DeleteUpsertField"` deletes fields holding invalid options,
+  `matchByName` resolves labels to option ids in place, and fields the caller allows may be
+  auto-created via your `upsertFieldOptions()` (resolved ids written back into the upsert).
+  **Because it mutates, always run the verifies BEFORE `detectFirstChange`** — the production
+  fleet does, without exception.
+- `verifyFieldConstraints` enforces your `ObjectField.constraints`.
+- Issues are `{ type, fatal?, fieldPath?, value?, option? }`; `fatal: true` blocks the row.
+- Returning `hasChanges` with no fatal issues queues the row; the platform batches queued rows
+  and calls your `upsert()` with them later (clean order = write order; `upsertClean` calls
+  are serialized per object). **Any throw inside `upsertClean` marks the row as a fatal
+  `RowCleanFailed` and it will not be written.**
+- Your `hasChanges` is not always final: when the caller uses an issues-field strategy, the
+  platform writes the issue text onto the row afterwards and flips `hasChanges` to true if
+  that text differs from the current row's.
+- **Do not call the destination API from `upsertClean`.** It runs per row on the hot path of
+  every sync — validation must work from `meta`, `current` and the upsert alone. (The one
+  sanctioned exception: `verifyFieldOptions` may trigger your `upsertFieldOptions()` to
+  create missing picklist options.)
+- **Not available everywhere:** on-premise (`agent_mode: "always"`) connectors and C#
+  connectors do not support `upsertClean` — don't declare `canUpsertClean` there.
+- For reference, callers (syncs/scripts) drive this per row via
+  `connection.upsertClean.<Object>({ upsert, current, batchSize, ... })` and flush the queued
+  batch by calling it with no arguments — useful to know when writing integration tests.
+
+### `detectFirstChange(options?)` — exact semantics
+
+It walks the **upsert**, comparing against `options.current.data`, and returns the FIRST
+difference as `{ field, oldValue, newValue }` (dot-joined path; non-string values
+JSON-stringified) — or `undefined` when nothing differs. It is a change *detector*, not a
+diff: `change.field` is evidence for logs, never a complete change list.
+
+Comparison rules:
+
+- **Only upsert properties are compared.** Fields on the current row that the upsert omits
+  are ignored — partial updates behave naturally.
+- **The key field never counts as a change** (the `isKey`/`isUserKey` path is skipped).
+- **No `current` row = insert**: it returns the upsert's first property as the change, so
+  `hasChanges` is effectively always true for inserts (some connectors shortcut:
+  `if (!options.current) return { hasChanges: true, change: { field: "*new*" }, issues };`).
+- **Null-equivalence defaults are ON**: `null`/`undefined` equals `""`, `[]` and `{}`. Disable
+  per shape via `overrides: { nullDoesNotEqualEmptyString: true, nullDoesNotEqualEmptyArray:
+  true, nullDoesNotEqualEmptyObject: true }` — needed when your API distinguishes "clear this
+  field" (null) from an empty string/array.
+- **No type coercion**: `"5" !== 5`, `"true" !== true` (`NaN === NaN` is the one special
+  case). If `query()` returns numbers-as-strings but upserts carry numbers, every row reports
+  a change — normalize the values, or veto in `confirmChange`.
+- **Arrays are order-insensitive multisets**: each upsert element must find an equal peer in
+  the current array. An unmatched upsert element = change (added); an unmatched current
+  element = change (removed) unless `overrides.doNotTreatArrayElementRemovalAsChange: true`.
+  The reported `field` is the array's own path. Matching is pairwise (O(n·m)) — very large
+  arrays get slow.
+- **Hard failures (which fail the whole row, per above)**: nested arrays throw; function
+  values throw; circular references throw. Symbols are silently treated as `undefined`.
+- **`FileProvider` values are ALWAYS a change** — no content comparison is attempted, and
+  `confirmChange` is not consulted. A file field in a cleaned upsert forces `hasChanges` on
+  every run; keep files out of cleaned payloads if that matters.
+
+`confirmChange(details)` is a veto callback — return `false` to suppress a detected
+difference. Its limitations are the part nobody guesses:
+
+- It receives `{ objectId, fieldPath, fieldMeta?, oldValue, newValue }`, but **`newValue` is
+  only populated for primitive-vs-primitive comparisons** — null-transition and
+  container-type-mismatch branches pass `newValue: undefined`. Build the callback around
+  `fieldPath` + `oldValue`.
+- It is **not consulted** for array add/remove detections or `FileProvider` changes — those
+  cannot be vetoed.
+- `fieldMeta` can be `undefined` for nested/array paths.
+
+The canonical `confirmChange` use case is API-side value normalization — e.g. an API that
+stores date-only in a datetime field, so the echoed value never equals what was sent:
+
+```js
+const change = options.detectFirstChange({
+  confirmChange: (d) =>
+    dateOnlyFields.has(d.fieldPath[0]) && typeof d.newValue === "string" && typeof d.oldValue === "string"
+      ? d.newValue.slice(0, 10) !== d.oldValue.slice(0, 10)   // compare the date part only
+      : true,
+});
+```
+
+One more ordering trap: APIs that require full-record PUTs need the current row merged into
+the upsert before writing — **detect first, merge after**, or the merged-in fields make every
+row look changed:
+
+```js
+const change = options.detectFirstChange();     // detect on the caller's intent
+Object.assign(options.upsert, this.#mergeCurrent(options.meta, options.current, options.upsert));
+return { hasChanges: !!change, change, issues };
+```
+
+## `upsertFieldOptions(options)` — auto-adding picklist values
+
+Declare `canUpsertFieldOptions: true` on the object AND `canUpdateOptions: true` on the field.
+`options` = `{ fieldPath, options: [{ id, meta? }], ... }`. Create/update the options via the
+API and return `{ resolvedIds, added, updated, unchanged, unresolved, metaUpdate? }` —
+`metaUpdate` lets you return just the changed field metadata so the platform skips a full
+`meta()` refresh.
+
+Real-world trap: many APIs are _eventually consistent_ about new picklist options — an upsert
+referencing a just-created option can fail for ~10-20 seconds. Handle it in your HTTP
+`canRetry` with a long backoff for that specific error ([03](./03-http-auth-ratelimit.md)).

package/docs/07-events-webhooks.md

@@ -0,0 +1,123 @@
+# Events and webhooks
+
+Two sidecar modes (`connector_metadata.webhooks_provider`, see
+[01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md)):
+
+- `{ "mode": "custom.perconnection" }` — each connection registers its own webhook with the
+  external system. `init()` receives `args.webhookUrl` / `args.webhookUid` for this connection.
+  The incoming `webhookUid` identifies the connection, so `parseEvents` runs on a **fully
+  initialized instance**: settings are available, and `eventIdentity` is simply
+  `options.payload.webhookUid`.
+- `{ "mode": "custom.perconnector", "url": ..., "webhook_uid": ... }` — one shared endpoint for
+  ALL connections. The platform cannot know which connection a payload belongs to, so
+  `parseEvents` runs **statically** (no connection settings); your code must derive
+  `eventIdentity` from the payload itself (e.g. an account/portal id), and each connection's
+  `meta().events.identity` must return the matching value so events route correctly.
+
+Declare the event types you emit in `meta()`:
+
+```js
+events: {
+  types: [
+    { id: "record.updated", name: "Record updated" },
+    // dataFields (ObjectField[]) optionally describes the event payload's schema,
+    // so triggers can map event data; [] is common when the payload is free-form
+    { id: "record.deleted", name: "Record deleted", dataFields: [] },
+  ];
+}
+```
+
+Note that the actual webhook UID varues are system generated, you do not manually specify them here.
+
+## `parseEvents(options)` → `{ events: SDK.ParsedEvent[] }`
+
+The execution context follows the `webhooks_provider.mode`:
+
+- **`custom.perconnection` → instance-based.** The connector is initialized for the target
+  connection first, so `parseEvents` can use everything `init()` set up — including a
+  settings-derived signing secret (e.g. a `webhookSecretKey` setting) for signature checks.
+- **`custom.perconnector` → static.** `init()` runs with `mode: "parseEvents"` and no
+  connection settings are guaranteed. Structure `init()` to early-return for that mode and
+  have `parseEvents` build anything it needs (e.g. its own `httpClient`) itself
+  ([02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)); verify signatures with the
+  platform-managed `verifyHash` or a key obtainable from the payload — never from settings,
+  which are absent here.
+
+```js
+/** @param {SDK.ParseEventsOptions} options @returns {Promise<SDK.ParsedEvents>} */
+async parseEvents(options) {
+  /** @type {SDK.ParsedEvent[]} */
+  const events = [];
+  if (!options.payload) return { events };
+
+  const bodyText = options.payload.body.toString("utf-8");   // body is a Buffer
+  const raw = JSON.parse(bodyText);
+
+  if (!(await this.#signatureValid(options, bodyText))) return { events }; // drop, don't throw
+
+  events.push({
+    // per-connection mode: the webhookUid IS the connection identity.
+    // per-connector (shared endpoint) mode: derive this from the payload instead
+    // (e.g. String(raw.accountId)) and return the same value from meta().events.identity.
+    eventIdentity: options.payload.webhookUid,
+    typeId: String(raw.type || "").toLowerCase(),  // must match an id from meta().events.types
+    timestamp: raw.occurredAt ? Date.parse(raw.occurredAt) : Date.now(),
+    uniqueId: raw.eventId,                     // de-dup key (~48h window) - use when available
+    object: raw.recordId ? { id: "contacts", key: String(raw.recordId) } : undefined,
+    data: raw,
+  });
+  return { events };
+}
+```
+
+`options.payload` = `{ webhookUid, uri, method, headers: {[name]: string[]}, body: Buffer }`.
+
+### Verifying signatures
+
+**Always verify** before emitting events — a webhook endpoint is unauthenticated input.
+
+- Platform-managed key: `await options.verifyHash({ strategy: "OAuth2.Sha256.Hex", expression, expected })`
+  — the platform computes the hash with a server-side key and returns `{ success }`.
+- Settings-derived secret (per-connection mode only — settings exist there): HMAC the raw body
+  with the secret the user configured (e.g. a `webhookSecretKey` setting captured in `init()`),
+  compare against the signature header.
+- Manual HMAC with a key obtainable from the payload (works in static mode too):
+
+```js
+const expected = options.payload.headers["x-content-signature"]?.[0];
+const crypto = await import("crypto");
+const computed = crypto.createHmac("sha256", signingKey).update(bodyText).digest("base64");
+if (expected !== computed) return { events: [] };
+```
+
+Failed verification: return no events (and optionally log) — throwing turns an attacker's junk
+into your error noise.
+
+## `handleParsedEvent(options)` — keep the cache current
+
+After your events fire, the platform calls `handleParsedEvent` per affected object so the
+cached data can be updated without a full re-query:
+
+```js
+/** @param {SDK.HandleParsedEventOptions} options */
+async handleParsedEvent(options) {
+  if (options.event.id === "record.deleted" && options.event.key) {
+    await options.reportChanges({
+      id: options.id,
+      page: { rows: [ { meta: { key: options.event.key, deleted: true } } ], finalPage: true },
+    });
+  }
+  // updated-record events: either fetch the row and report it, or report nothing and let the
+  // next differential query pick it up - fetching is better for near-real-time triggers.
+}
+```
+
+`options.event` = `{ id, timestamp, key?, data? }`; `reportChanges({ id, page })` takes a normal
+`QueryPage`.
+
+## Flow summary
+
+external system → platform webhook endpoint → your `parseEvents` (verify, emit
+`ParsedEvent[]`) → platform de-dups (`uniqueId`), resolves connections (`eventIdentity`),
+fires triggers (`typeId`) → your `handleParsedEvent` per changed object → `reportChanges`
+updates the cache.

package/docs/08-errors.md

@@ -0,0 +1,68 @@
+# Error handling
+
+## `SDK.ConnectorError`
+
+Every platform-visible failure is a `ConnectorError`:
+
+```js
+throw new SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, "Setting 'API Key' is required");
+throw new SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, "supplied option not valid",
+  { issues: [{ objectId: "deals", fieldId: "sales_rep", option: "Jane Doe" }] });  // optional data payload
+```
+
+Never throw bare strings or plain `Error` for expected failure modes — the `code` is what the
+platform (and support tooling) branches on. The one exception: `test()` **catches** and returns
+`{ success: false, message, error }` instead of throwing
+([02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)).
+
+## Choosing an `ErrorCode`
+
+Ordered by how often the production fleet actually uses them — the top four cover ~85% of
+throws:
+
+| Code | Throw when |
+|---|---|
+| `InvalidFilterParameter` | asked to run a query/filter/match rule the object didn't declare or with malformed filter values. Fail fast — never silently return unfiltered data |
+| `InvalidOptionValue` | a required connection setting is missing/invalid, or an option/parameter value is not permitted |
+| `InvalidOperation` | "this should never happen" internal-state violations |
+| `InvalidObjectId` | `options.id` names an object this connector doesn't expose |
+| `ApiReportedError` | the external API returned a failure in its response body (include the API's payload in the message) |
+| `InvalidRecordId` | a row key doesn't exist / is malformed |
+| `NotImplimented` | an operation/feature is deliberately unsupported. **Note the spelling — `NotImplimented` is the canonical enum member**, do not "fix" it |
+| `InvalidFieldId` | unknown field/property path, or an option label that can't be resolved |
+| `InvalidApiKey` / `InvalidUserName` / `InvalidPassword` | the specific credential the API rejected |
+| `ApiTokenGenerationTerminated` | token acquisition/refresh was aborted |
+| `ApiUrlNotSupported` / `InvalidFilePath` / `InvalidCacheFilter` / ... | as named — see the `ErrorCode` enum in the type definitions for the full list |
+
+## `HttpError` (thrown by the SDK http client)
+
+Non-2xx responses from `httpClient.execute` throw:
+
+```js
+{ name: "HttpError", code: "HTTP429", statusCode: 429, url,
+  responseHeaders,          // lowercased names
+  responseBodyString, responseBodyJson, responseBodyBuffer, cause }
+```
+
+Idioms:
+
+```js
+} catch (err) {
+  if (err.code === "HTTP401") { /* force OAuth refresh, retry once - see 03 */ }
+  if (err.code === "HTTP404") { return { rows: [], finalPage: true }; }   // when 404 means "no rows"
+  throw err;   // let unexpected errors surface with full context
+}
+```
+
+Prefer the client's built-in retry (`maxRetry` + `canRetry` + `retryBackoffMs`,
+[03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)) over hand-rolled retry loops.
+Translate an `HttpError` into a `ConnectorError` only when you can say something more precise
+(e.g. a 401 during `test()` → `InvalidApiKey`); otherwise let it propagate — it already carries
+the URL, status and response body.
+
+## Logging
+
+`options.log` / `args.log` writes to the platform event store: `log.info(message, props?)`,
+`log.error(messageOrError, props?)`. Log the API's raw response alongside errors you throw for
+supportability, and gate verbose request/response logging behind an advanced `debug` setting —
+never log secrets or tokens.

package/docs/09-testing.md

@@ -0,0 +1,127 @@
+# The connector test harness
+
+Two files per connector, both living beside the entry file (the `"mod "` prefix, with the
+space, is the convention):
+
+| File                         | Sidecar `type`                                                                                   | Origin                                                                        |
+| ---------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
+| `mod ObjectTestFeatures.mjs` | `"module"`                                                                                       | **generated** from a live test connection's cached metadata — never hand-edit |
+| `mod Test.mjs`               | `"connectortest"` (its sidecar lists `mod ObjectTestFeatures.mjs` in `module_script_file_paths`) | hand-written `TestFeatureProvider`                                            |
+
+The platform's connector test suite (script type `"connectorTest"`) drives your connector
+through every capability its `meta()` declares — list/ids/checkpoint/match/related queries,
+upserts, deletes — using these two files to know what to expect and how to prepare data.
+
+## `mod ObjectTestFeatures.mjs` (generated)
+
+Declares, per object, which features the suite should exercise and which fields are usable:
+
+```js
+/** Do not edit. This file is automatically generated using cached metadata from a test connection. */
+import API from "@syncmatters/script-api";
+const toJsonPath = (p) => API.utilities.json.stringToJsonValuePath(p, "all");
+const toJsonPaths = (ps) => API.utilities.json.stringsToJsonValuePaths(ps, "all");
+
+export const objectFeatures = {
+  contacts: {
+    query: {
+      list: true,
+      idsFilter: true,
+      checkpointFilter: true,
+      matchRules: [{ rule: "email[ci]" }],
+      canMatchFields: toJsonPaths(["id", "email", "name"]),
+      relationshipsTo: [{ id: "companies-companyId", relObjectId: "companies" }],
+    },
+    upsert: { canUpsert: true, key: { path: toJsonPath("id") }, canUpsertClean: true },
+    delete: { canDelete: true, key: { path: toJsonPath("id") } },
+  },
+};
+```
+
+It is emitted by the platform's object-features generator from `ObjectMeta[]` cached on a test
+connection — regenerate after `meta()` changes rather than editing.
+
+## `mod Test.mjs` (hand-written)
+
+Implements `SDKTest.TestFeatureProvider` (types from `@syncmatters/script-api/sdk-test`):
+
+```js
+import API from "@syncmatters/script-api";
+import SDKTest from "@syncmatters/script-api/sdk-test";
+import { objectFeatures } from "./mod ObjectTestFeatures";
+
+/** @typedef {API.connections.acme_test.Connection} TestConnection */
+
+export default class Test {
+  expectedIdentity = "Acme Sandbox|integration@example.com"; // must equal meta().identity
+  /** @type {TestConnection | undefined} */ #connection;
+
+  async before(ctx) {
+    this.#connection = /** @type any */ (ctx.connection);
+    await this.#cleanupTestData(); // delete leftovers from crashed runs
+  }
+  objectIds() {
+    return Object.keys(objectFeatures);
+  }
+  objectFeatures(id) {
+    return objectFeatures[id];
+  }
+
+  // prepare hooks - create/arrange the data a given test needs, return what it should expect:
+  // listPrepare, idsFilterPrepare, checkpointFilterStep1Prepare / Step2Prepare,
+  // matchFilterPrepare, relatedFilterPrepare, upsertPrepare
+}
+```
+
+Established conventions:
+
+- **Typed test connection**: `API.connections.<connection_name>.Connection` gives you typed
+  `query`/`upsert`/`delete` accessors for driving real data:
+  `for await (const row of this.#connection.query.contacts({...})) {...}`,
+  `await this.#connection.upsert.contacts([{ id, name }])`.
+- **Test-data hygiene**: prefix every created record (`const testRecordPrefix = "sm_test";`)
+  and have `before()` sweep and delete `sm_test*` leftovers, so crashed runs never pollute the
+  sandbox account.
+- **Scoping out an object**: set `features.cannotTestReason = "<why>"` on an object's features
+  instead of deleting it from the generated file.
+- `expectedIdentity` pins the account: if the sandbox credentials get re-pointed, the suite
+  fails loudly instead of trashing another account's data.
+
+## Fine-grained feature control
+
+`cannotTestReason` has per-feature variants when only part of an object is untestable:
+`query.listCannotTestReason`, `idsFilterCannotTestReason`, `checkpointFilterCannotTestReason`,
+`matchRulesCannotTestReason` (whole or per-rule index), `relationshipsToCannotTestReason`
+(whole or per-relationship index). The `*SameAsObjectId` variants ("this feature behaves
+exactly like object X — test it there") avoid duplicating slow coverage across near-identical
+objects.
+
+## Prepare/cleanup hooks and their data types
+
+Every testable feature has a `*Prepare` hook (and usually a `*Cleanup` counterpart) —
+returning `undefined` from a prepare hook skips that test:
+
+- `listPrepare` / `idsFilterPrepare` / `rowFiltersPrepare` — arrange rows and return what the
+  suite should expect (row-filter testing is easy to forget: provide it when `meta()` exposes
+  `queryFilter.rowFilter`).
+- `checkpointFilterStep1Prepare` / `Step2Prepare` — step 1 seeds data and captures the
+  checkpoint; step 2 mutates data the differential query must then surface.
+- `matchFilterPrepare` — returns `MatchData` including `expectedMatchRowIds`.
+- `upsertPrepare` — returns `UpsertData`; use `withIssueSimulators` / `UpsertIssueSimulator`
+  (`verifyFields`, expected issue `type` + `fatal`) to deliberately exercise your
+  `upsertClean` issue reporting.
+- delete features support soft-delete verification — `isDeleted: { path, valueWhenDeleted }`
+  checks a flag field instead of row absence, and `afterDeleteMaxIndexWaitTimeMs` tolerates
+  APIs whose deletions surface asynchronously.
+
+## Coverage bar
+
+Target **80%+ code coverage** (statements/branches/functions/lines) from the platform's
+connector test runs — reports are downloadable per run and retained ~90 days. Structure
+`mod Test.mjs` so every object with declared capabilities is exercised rather than scoped out.
+
+## Local dev loop
+
+The harness runs platform-side. Locally you get: `sm validate` (workspace structure),
+`checkJs` type checking against the SDK types, `sm push` and a platform-side test run. Keep
+`test()` cheap and `meta()` deterministic so the suite stays fast.