@syncmatters/connector-sdk 1.0.1 → 1.0.2

package/docs/01-workspace-and-meta-json.md

@@ -10,7 +10,7 @@ meta/Connectors/<ConnectorName>/     # one *.meta.json sidecar per source file
 - **Entry file**: `<ConnectorName>.mjs`, default-exporting the connector class. Its sidecar has
   `"type": "connector"` and carries the `connector_metadata` block.
 - **Module files**: additional `.mjs` files in the same folder, imported by the entry (or by
-  other modules). Sidecar `"type": "module"`.
+  other modules). Sidecar `"type": "module"`. Modules **must** be placed in the same folder as the file that imports the module.
 - **Resource files**: e.g. a bundled OpenAPI spec (`acme-api.yaml`). Sidecar type `"yaml"` /
   `"json"`. Load big specs lazily (dynamic `import()`), not at module top level.
 - **Test harness files**: `mod Test.mjs` (sidecar type `"connectortest"`) and the generated
@@ -50,37 +50,40 @@ switch (version) {
     break;
   }
   default:
-    throw new SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, `Connector version '${version}' not recognized`);
+    throw new SDK.ConnectorError(
+      SDK.ErrorCode.InvalidOptionValue,
+      `Connector version '${version}' not recognized`,
+    );
 }
 await this.#core.init(args);
 ```
 
 ## Sidecar top-level fields
 
-| Field | Values / meaning |
-|---|---|
-| `name` | the file name, e.g. `"Acme.mjs"` |
-| `type` | `"connector"` (entry) · `"module"` · `"connectortest"` · `"json"` · `"yaml"` |
-| `runnable` | `false` for connector files (they are invoked by the platform, not run directly) |
-| `engine` | runtime engine version int; leave as scaffolded unless told otherwise |
-| `resource_size` | `""` or `"medium"` — runtime resource class hint |
-| `cpu_architecture` | normally `""` |
-| `module_script_file_paths` | see above |
-| `npm_packages` | name→version map of npm packages this file needs at runtime, e.g. `{"box-node-sdk": "2.4.0"}`. The `sm` CLI unions these into the workspace `package.json` so IntelliSense sees them too. Prefer the SDK's built-in http/utilities over adding packages. |
-| `connector_metadata` | entry sidecar only — everything below |
+| Field                      | Values / meaning                                                                                                                                                                                                                                         |
+| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `name`                     | the file name, e.g. `"Acme.mjs"`                                                                                                                                                                                                                         |
+| `type`                     | `"connector"` (entry) · `"module"` · `"connectortest"` · `"json"` · `"yaml"`                                                                                                                                                                             |
+| `runnable`                 | `false` for connector files (they are invoked by the platform, not run directly)                                                                                                                                                                         |
+| `engine`                   | runtime engine version int; leave as scaffolded unless told otherwise                                                                                                                                                                                    |
+| `resource_size`            | `""` or `"medium"` — runtime resource class hint                                                                                                                                                                                                         |
+| `cpu_architecture`         | normally `""`                                                                                                                                                                                                                                            |
+| `module_script_file_paths` | see above                                                                                                                                                                                                                                                |
+| `npm_packages`             | name→version map of npm packages this file needs at runtime, e.g. `{"box-node-sdk": "2.4.0"}`. The `sm` CLI unions these into the workspace `package.json` so IntelliSense sees them too. Prefer the SDK's built-in http/utilities over adding packages. |
+| `connector_metadata`       | entry sidecar only — everything below                                                                                                                                                                                                                    |
 
 ## `connector_metadata` reference
 
-| Field | Meaning |
-|---|---|
-| `categories` | discovery taxonomy, e.g. `["sales"]`, `["file"]`, `["marketing","sales"]` |
-| `agent_mode` | `"never"` = runs in cloud · `"always"` = runs on the customer's on-premise agent · `"optional"` = per-connection (agent used when the connection selects one) |
-| `oauth2_mode` | `"connector"` = one shared OAuth app for every account · `"connection"` = each connection supplies its own client id/secret. Omit for non-OAuth connectors. |
-| `oauth2_provider` | the OAuth provider definition (below) |
-| `webhooks_provider` | `{"mode": "custom.perconnection"}` (each connection registers its own webhook) or `{"mode": "custom.perconnector", "url": ..., "webhook_uid": ...}` (one shared endpoint) |
-| `settings` | the connection-settings form (below) |
-| `icon_url` | connector logo URL |
-| `connector_name`, `type_id`, `application_name`, `application_icon_url` | platform/branding ids — usually left as scaffolded |
+| Field                                                                   | Meaning                                                                                                                                                                   |
+| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `categories`                                                            | discovery taxonomy, e.g. `["sales"]`, `["file"]`, `["marketing","sales"]`                                                                                                 |
+| `agent_mode`                                                            | `"never"` = runs in cloud · `"always"` = runs on the customer's on-premise agent · `"optional"` = per-connection (agent used when the connection selects one)             |
+| `oauth2_mode`                                                           | `"connector"` = one shared OAuth app for every account · `"connection"` = each connection supplies its own client id/secret. Omit for non-OAuth connectors.               |
+| `oauth2_provider`                                                       | the OAuth provider definition (below)                                                                                                                                     |
+| `webhooks_provider`                                                     | `{"mode": "custom.perconnection"}` (each connection registers its own webhook) or `{"mode": "custom.perconnector", "url": ..., "webhook_uid": ...}` (one shared endpoint) |
+| `settings`                                                              | the connection-settings form (below)                                                                                                                                      |
+| `icon_url`                                                              | connector logo URL                                                                                                                                                        |
+| `connector_name`, `type_id`, `application_name`, `application_icon_url` | platform/branding ids — usually left as scaffolded                                                                                                                        |
 
 ### `settings`
 

package/docs/05-query.md

@@ -124,11 +124,16 @@ candidates }])` — the platform reserves un-mapped matches and returns the sele
 
 Receives rows from a source object (`otherObjectId`) that has a defined relationship to this object, and collect the related rows from this object.
 
-`{ otherObjectId, otherRelationshipId, otherRows }`. Resolve the rows from a relationship declared in
-`meta()` (your `relationship.data` typically holds the FK field), turn `otherRows` into an id
-lookup, and return rows tagged with
+`{ otherObjectId, otherRelationshipId, otherRows }`. DIRECTION: the platform queries the
+relationship's TARGET object and passes the DECLARING object's rows — contacts declare
+contact→company, so a related query arrives on `companies` with `otherObjectId: "contacts"`.
+Resolve the relationship from the OTHER object's metadata
+(`await options.objectMeta(related.otherObjectId)` — it is NOT in `options.meta`; your
+`relationship.data` typically holds the FK field name), collect the FK values from
+`otherRows` (they name THIS object's keys), fetch those rows, and return them tagged with
 `relationship: { srcRowId, srcObjectId: otherObjectId, relationshipId: otherRelationshipId }` —
-one row instance per source row it relates to.
+one row instance per source row it relates to. See the worked implementation in
+[12-example-connector.md](./12-example-connector.md).
 
 ## Cross-page scratch state: `kvStore`
 

package/docs/09-testing.md

@@ -11,6 +11,9 @@ space, is the convention):
 The platform's connector test suite (script type `"connectorTest"`) drives your connector
 through every capability its `meta()` declares — list/ids/checkpoint/match/related queries,
 upserts, deletes — using these two files to know what to expect and how to prepare data.
+The complete worked pair for the Acme connector of
+[12-example-connector.md](./12-example-connector.md) is in
+[15-example-test-file.md](./15-example-test-file.md).
 
 ## `mod ObjectTestFeatures.mjs` (generated)
 
@@ -98,8 +101,12 @@ objects.
 
 ## Prepare/cleanup hooks and their data types
 
-Every testable feature has a `*Prepare` hook (and usually a `*Cleanup` counterpart) —
-returning `undefined` from a prepare hook skips that test:
+Every testable feature has a `*Prepare` hook (and usually a `*Cleanup` counterpart). Most
+prepare hooks skip their test when they return `undefined` — with two exceptions:
+`listPrepare` returning `undefined` still runs the list test (it just expects some rows), and
+`idsFilterPrepare` returning `undefined` makes the harness find ids via a list query itself.
+To disable those tests use `cannotTestReason` instead
+([15-example-test-file.md](./15-example-test-file.md) shows the worked semantics):
 
 - `listPrepare` / `idsFilterPrepare` / `rowFiltersPrepare` — arrange rows and return what the
   suite should expect (row-filter testing is easy to forget: provide it when `meta()` exposes

package/docs/12-example-connector.md

@@ -5,6 +5,9 @@ One canonical, end-to-end connector on a single page. Three objects — `compani
 business unit + order id) — plus a contact→company relationship, static + API-discovered
 custom fields, and full `query` / `upsert` / `delete`. Every pattern here follows the
 production fleet; copy this shape and adapt it. Topic-file cross-references are in comments.
+Its test-file counterpart — the `mod Test.mjs` + `mod ObjectTestFeatures.mjs` pair the
+platform test suite and `sm run test` require — is
+[15-example-test-file.md](./15-example-test-file.md).
 
 The imaginary Acme API: bearer-token REST at `https://api.example.com/v1/`, cursor pagination
 (`?after=<id>`), `modified_since` differential filter, a `/search` endpoint taking field
@@ -211,6 +214,7 @@ export default class Acme {
     }
 
     if (qo.matchFilter) return await this.#queryMatch(options, qo.matchFilter);
+    if (qo.relatedFilter) return await this.#queryRelated(options, qo.relatedFilter);
     // composite-key ids arrive as key STRINGS - parse each and fetch directly (11)
     if (options.id === "salesorders" && qo.idsFilter) {
       return await this.#querySalesOrdersByIds(qo.idsFilter);
@@ -231,13 +235,10 @@ export default class Acme {
       // undefined value = first differential run: return everything (05)
       params.push(`modified_since=${encodeURIComponent(qo.checkpointFilter.value || "1970-01-01T00:00:00Z")}`);
     }
-    if (qo.relatedFilter) params.push(this.#relatedWhere(options, qo.relatedFilter));
-
     const resp = await this.#call({ method: "GET", url: `${options.id}?${params.join("&")}` });
 
     /** @type {SDK.Row[]} */
     let rows = resp.items.map((/** @type {any} */ item) => this.#row(options.id, item));
-    if (qo.relatedFilter) rows = this.#tagRelationships(rows, qo.relatedFilter);
 
     queryState.after = resp.next_cursor || undefined;
     queryState.rowCount += rows.length;
@@ -334,35 +335,52 @@ export default class Acme {
     }
   }
 
-  /** Resolve the FK declared in meta().relationships via its data payload.
-   * @param {SDK.QueryOptions} options @param {SDK.QueryRelatedFilter} related */
-  #relatedWhere(options, related) {
-    const rel = (options.meta.relationships || []).find((r) => r.id === related.otherRelationshipId);
+  /** Fetch rows of THIS object that other-object rows relate to. DIRECTION MATTERS:
+   * the platform queries the relationship's TARGET (here: companies) and passes the
+   * DECLARING object's rows (contacts) - so the relationship is resolved from the
+   * OTHER object's metadata via options.objectMeta, never options.meta (05).
+   * @param {SDK.QueryOptions} options @param {SDK.QueryRelatedFilter} related
+   * @returns {Promise<SDK.QueryPage>} */
+  async #queryRelated(options, related) {
+    const otherMeta = await options.objectMeta(related.otherObjectId);
+    const rel = (otherMeta.relationships || []).find(
+      (r) => r.id === related.otherRelationshipId,
+    );
     if (!rel) {
       throw new SDK.ConnectorError(SDK.ErrorCode.InvalidFilterParameter,
-        `object ${options.id} has no relationship '${related.otherRelationshipId}'`);
+        `object ${related.otherObjectId} has no relationship '${related.otherRelationshipId}'`);
     }
+    // the FK lives on the DECLARING object's rows (contact.company_id names a company);
+    // rel.data is OUR round-trip payload from meta() (04)
     const fk = /** @type {{idField: string}} */ (rel.data).idField;
-    const ids = related.otherRows.map((r) => r.meta.key);
-    return `${fk}=${ids.map(encodeURIComponent).join(",")}`;
-  }
+    /** @type {Map<string, SDK.Row[]>} this-object key -> source rows referencing it */
+    const wanted = new Map();
+    for (const src of related.otherRows) {
+      const key = src.data?.[fk] != null ? String(src.data[fk]) : "";
+      if (!key) continue; // source row without the FK - skip, never guess (10)
+      if (!wanted.has(key)) wanted.set(key, []);
+      /** @type {SDK.Row[]} */ (wanted.get(key)).push(src);
+    }
+    if (wanted.size === 0) return { rows: [], finalPage: true };
 
-  /** One row instance per source row it relates to, tagged with the relationship (05).
-   * @param {SDK.Row[]} rows @param {SDK.QueryRelatedFilter} related @returns {SDK.Row[]} */
-  #tagRelationships(rows, related) {
-    const out = [];
-    for (const row of rows) {
-      for (const src of related.otherRows) {
-        if (String(row.data.company_id) === src.meta.key) {
-          out.push({ ...row, relationship: {
-            srcRowId: src.meta.key,
-            srcObjectId: related.otherObjectId,
-            relationshipId: related.otherRelationshipId,
-          } });
-        }
+    const resp = await this.#call({
+      method: "GET",
+      url: `${options.id}?ids=${[...wanted.keys()].map(encodeURIComponent).join(",")}`,
+    });
+    /** @type {SDK.Row[]} */
+    const rows = [];
+    for (const item of resp.items) {
+      const base = this.#row(options.id, item);
+      // one row INSTANCE per source row it relates to (05)
+      for (const src of wanted.get(base.meta.key) || []) {
+        rows.push({ ...base, relationship: {
+          srcRowId: src.meta.key,
+          srcObjectId: related.otherObjectId,
+          relationshipId: related.otherRelationshipId,
+        } });
       }
     }
-    return out;
+    return { rows, finalPage: true };
   }
 
   // ---------------------------------------------------------------- upsert / delete (06)
@@ -548,8 +566,11 @@ function parseSalesOrderKey(compositeKey) {
    verbatim** (never re-encoded) on update/delete.
 4. **The composite `$compositeKey` pseudo-field carries `isKey`** and is stripped from every
    API body; adds supply the parent part (`business_unit`) as a regular mandatory field.
-5. **`relationships[].data` is the connector's own round-trip payload** — `meta()` stores the
-   FK field name, `query()` reads it back from `options.meta` when resolving `relatedFilter`.
+5. **Related queries run on the relationship's TARGET object** — the platform queries
+   `companies` passing `contacts` rows, so the relationship is resolved from the OTHER
+   object's metadata (`options.objectMeta(otherObjectId)`), never `options.meta`; the FK
+   values on the source rows name the queried object's keys. `relationships[].data` is the
+   connector's own round-trip payload carrying that FK field name.
 6. **`query()` dispatches filters in one place**, throws `InvalidFilterParameter` for anything
    undeclared, and shares the pagination path across list/ids/checkpoint variants — except
    composite-key `idsFilter`, which parses each key and fetches directly.

package/docs/15-example-test-file.md

@@ -0,0 +1,565 @@
+# Acme — the worked test-file pair
+
+The test-file counterpart to [12-example-connector.md](./12-example-connector.md): the complete
+`mod Test.mjs` + `mod ObjectTestFeatures.mjs` pair for the Acme connector, on one page. The
+platform's connector test suite AND `sm run test|meta|query` both require a file of sidecar
+type `"connectortest"` in the connector's folder — **a connector without one cannot be
+run-tested at all**. Harness conventions and hook semantics live in
+[09-testing.md](./09-testing.md); this page shows them applied end to end against Acme's three
+objects: `companies`, `contacts` (email + first/last-name matching, `upsertClean`, custom
+fields) and `salesorders` (the `$compositeKey` composite-key object).
+
+Two files, two sidecars. `mod Test.mjs`'s sidecar is type `"connectortest"` and wires in the
+generated module via `module_script_file_paths`:
+
+`meta/Connectors/Acme/mod ObjectTestFeatures.mjs.meta.json`
+
+```json
+{
+  "runnable": false,
+  "type": "module",
+  "cpu_architecture": "",
+  "engine": 0,
+  "resource_size": "medium",
+  "name": "mod ObjectTestFeatures.mjs"
+}
+```
+
+`meta/Connectors/Acme/mod Test.mjs.meta.json`
+
+```json
+{
+  "runnable": false,
+  "type": "connectortest",
+  "cpu_architecture": "",
+  "engine": 0,
+  "resource_size": "medium",
+  "name": "mod Test.mjs",
+  "module_script_file_paths": ["Connectors/Acme/mod ObjectTestFeatures.mjs"]
+}
+```
+
+Keep exactly one `"connectortest"` file per connector folder — `sm run test` resolves the test
+module from the connector's folder and several candidates are ambiguous.
+
+## `files/Connectors/Acme/mod ObjectTestFeatures.mjs` (generated)
+
+Generated from the sandbox connection's cached `meta()` — regenerate it whenever `meta()`
+changes; never hand-edit it. Note there are **no** `cannotTestReason` / `sameAsObjectId`
+entries here: the generator cannot know those, so the fine-grained scoping knobs are applied
+from `mod Test.mjs`'s `before()` (below), on top of this file.
+
+```js
+/**
+ * Do not edit. This file is automatically generated using cached metadata from a test connection.
+ *
+ * This automatically generated object feature list can be extended in your main test module file; connector metadata
+ * does not contain sufficient detail to fully describe all connector features.
+ */
+import API from "@syncmatters/script-api";
+import SDKTest from "@syncmatters/script-api/sdk-test";
+const toJsonPath = (/** @type string */ fieldPath) => API.utilities.json.stringToJsonValuePath(fieldPath, "all");
+const toJsonPaths = (/** @type string[] */ fieldPaths) => API.utilities.json.stringsToJsonValuePaths(fieldPaths, "all");
+
+/** @type Record<string, SDKTest.ObjectFeatures> */
+export const objectFeatures = {
+  companies: {
+    query: {
+      list: true,
+      idsFilter: true,
+      checkpointFilter: true,
+      matchRules: [{"rule":"id"},{"rule":"name[ci]"}],
+      canMatchFields: toJsonPaths(["id", "name", "domain"])
+    },
+    upsert: {
+      canUpsert: true,
+      key: {
+        path: toJsonPath("id")
+      },
+      constraints: [{"constraints":{"mandatory":"add"},"fieldPath":[{"path":"name"}]}]
+    },
+    delete: {
+      canDelete: true,
+      key: {
+        path: toJsonPath("id")
+      }
+    }
+  },
+  contacts: {
+    query: {
+      list: true,
+      idsFilter: true,
+      checkpointFilter: true,
+      matchRules: [{"rule":"id"},{"rule":"email[ci]"},{"rule":"first_and_last_name[ci]"}],
+      canMatchFields: toJsonPaths(["id", "email", "first_name", "last_name"]),
+      relationshipsTo: [{"id":"companies-company_id","relObjectId":"companies"}]
+    },
+    upsert: {
+      canUpsert: true,
+      key: {
+        path: toJsonPath("id")
+      },
+      canUpsertClean: true,
+      constraints: [{"constraints":{"mandatory":"add","email":true},"fieldPath":[{"path":"email"}]}]
+    },
+    delete: {
+      canDelete: true,
+      key: {
+        path: toJsonPath("id")
+      }
+    }
+  },
+  salesorders: {
+    query: {
+      list: true,
+      idsFilter: true
+    },
+    upsert: {
+      canUpsert: true,
+      key: {
+        path: toJsonPath("$compositeKey")
+      },
+      constraints: [{"constraints":{"mandatory":"add"},"fieldPath":[{"path":"business_unit"}]}]
+    },
+    delete: {
+      canDelete: true,
+      key: {
+        path: toJsonPath("$compositeKey")
+      }
+    }
+  }
+};
+```
+
+## `files/Connectors/Acme/mod Test.mjs` (hand-written)
+
+Implements `SDKTest.TestFeatureProvider`. Every field/type below is from
+`@syncmatters/script-api/sdk-test`; topic-file cross-references are in comments.
+
+```js
+import API from "@syncmatters/script-api";
+import SDKTest from "@syncmatters/script-api/sdk-test";
+// the auto-generated feature list - regenerate after meta() changes, never hand-edit (09)
+import { objectFeatures } from "./mod ObjectTestFeatures";
+
+// Typed test connection: typed query/queryOne/upsert/delete accessors over the sandbox
+// connection. "acme_test" is YOUR test connection's name. (Fleet files that predate the
+// @syncmatters packages import the same types from the legacy "@ihq/script-api" scope -
+// new code always imports @syncmatters/*.)
+/** @typedef {API.connections.acme_test.Connection} TestConnection */
+/** @typedef {TestConnection["upsert"]} Upsert */
+/** @typedef {TestConnection["queryOne"]} Query */
+// With the ^^above^^ typedefs:
+// - upsert row type for "contacts":       {Parameters<Upsert["contacts"]>[0][0]}
+// - query result row type for "contacts": {Awaited<ReturnType<Query["contacts"]>>}
+
+// prefix EVERY record the tests create, so leftovers from crashed runs stay findable
+const testRecordPrefix = "sm_test";
+
+/** @implements SDKTest.TestFeatureProvider */
+export default class TestFeatureProvider {
+  // must equal the sandbox connection's meta().identity - pins the account, so re-pointed
+  // credentials fail loudly instead of trashing another account's data (09, 12)
+  /** @type SDKTest.TestFeatureProvider["expectedIdentity"] */
+  expectedIdentity = "Acme Sandbox|integration@example.com";
+
+  /** @type API.Logger */
+  #log;
+  /** @type TestConnection */
+  #connection;
+  /** @type string */
+  #fixtureCompanyId = ""; // shared parent row for contact test data
+
+  /** @type SDKTest.TestFeatureProvider["before"] */
+  async before(ctx) {
+    this.#log = ctx.log;
+    this.#connection = /** @type any */ (ctx.connection);
+
+    // === extend the GENERATED features here - never by editing the generated file (09).
+    // contacts delete runs the exact same batch/archive code path as companies delete,
+    // so test it once:
+    if (objectFeatures.contacts.delete) {
+      objectFeatures.contacts.delete.sameAsObjectId = "companies";
+    }
+    // per-rule scoping: Acme's /search cannot search by record id - the "id" rule resolves
+    // via get-by-id, which the idsFilter test already covers:
+    for (const objectId of ["companies", "contacts"]) {
+      const idRule = (objectFeatures[objectId].query?.matchRules || []).find((r) => r.rule === "id");
+      if (idRule) idRule.cannotTestReason = "id matches resolve via get-by-id; covered by the idsFilter test";
+    }
+    // Other scoping knobs, when you need them:
+    //   objectFeatures.salesorders.cannotTestReason = "<why the whole object cannot be tested>";
+    //   objectFeatures.salesorders.delete.cannotTestReason = "<why just this feature cannot>";
+    // If Acme's archive endpoint kept returning archived rows (a soft delete), you would
+    // declare the flag field instead of expecting row absence, and allow for slow indexes:
+    //   objectFeatures.companies.delete.isDeleted = { path: [{ path: "archived" }], valueWhenDeleted: true };
+    //   objectFeatures.companies.delete.afterDeleteMaxIndexWaitTimeMs = 30000;
+
+    // === sweep sm_test* leftovers from crashed runs
+    await this.#sweepTestData();
+
+    // === shared fixtures
+    const company = await this.#connection.queryOne.companies();
+    if (!company) throw new Error("Sandbox has no companies - seed at least one before testing");
+    this.#fixtureCompanyId = company.meta.key;
+  }
+
+  /** @type SDKTest.TestFeatureProvider["after"] */
+  async after() {
+    await this.#sweepTestData();
+  }
+
+  /** @type SDKTest.TestFeatureProvider["objectIds"] */
+  async objectIds() {
+    return Object.keys(objectFeatures);
+  }
+
+  /** @type SDKTest.TestFeatureProvider["objectFeatures"] */
+  async objectFeatures(objectId) {
+    return objectFeatures[objectId];
+  }
+
+  /** @type SDKTest.TestFeatureProvider["listPrepare"] */
+  async listPrepare(options) {
+    // returning undefined here would mean "run the list, just verify SOME rows come back";
+    // a small pageSize forces real pagination through Acme's cursor (05)
+    switch (options.meta.id) {
+      case "salesorders":
+        return { pageSize: 25, fields: [[{ path: "$compositeKey" }], [{ path: "status" }]] };
+      default:
+        return { pageSize: 25 };
+    }
+  }
+
+  /** @type SDKTest.TestFeatureProvider["idsFilterPrepare"] */
+  async idsFilterPrepare(options) {
+    // list-capable objects can skip this hook - undefined tells the harness to find two
+    // ids via a list query itself
+    if (options.meta.id !== "salesorders") return undefined;
+
+    // implemented for salesorders to make composite keys concrete: the ids ARE the
+    // connector's JSON key strings (row.meta.key), e.g.
+    //   '{"business_unit":"emea","order_id":1041}'
+    // NEVER hand-assemble them - collect real rows and reuse their keys verbatim (11, 12)
+    /** @type SDKTest.IdsFilterData */
+    const result = { id1: "", id2: "" };
+    for await (const order of this.#connection.query.salesorders()) {
+      if (!result.id1) {
+        result.id1 = order.meta.key;
+      } else if (order.meta.key !== result.id1) {
+        result.id2 = order.meta.key;
+        break;
+      }
+    }
+    return result;
+  }
+
+  /** @type SDKTest.TestFeatureProvider["checkpointFilterStep1Prepare"] */
+  async checkpointFilterStep1Prepare(options) {
+    // step 1: pick a starting checkpoint and report every row the differential query
+    // should return for it - then guarantee at least one by seeding a row
+    const start = new Date(Date.now() - 60000).toISOString();
+    /** @type SDKTest.CheckpointStep1Data */
+    const result = { nextCheckpoint: start, expectedRowIds: [] };
+
+    // a shared sandbox is never quiet: rows already modified inside the window are
+    // expected too, so collect them rather than assuming zero
+    // @ts-ignore - objectId is dynamic
+    for await (const row of this.#connection.query[options.meta.id]({ checkpointFilter: { value: start } })) {
+      result.expectedRowIds.push(row.meta.key);
+    }
+    if (result.expectedRowIds.length === 0) {
+      result.expectedRowIds.push(await this.#createTestRow(options.meta.id));
+    }
+    return result;
+  }
+
+  /** @type SDKTest.TestFeatureProvider["checkpointFilterStep2Prepare"] */
+  async checkpointFilterStep2Prepare(options) {
+    // step 2: the harness hands back the checkpoint the CONNECTOR emitted in step 1;
+    // mutate the data and report what the second differential query must surface
+    /** @type SDKTest.CheckpointStep2Data */
+    const result = { expectedRowIds: [] };
+
+    // checkpoints are emitted with an overlap window (05, 12), so step-1 rows may
+    // legitimately reappear - include everything currently inside the window...
+    // @ts-ignore - objectId is dynamic
+    for await (const row of this.#connection.query[options.meta.id]({ checkpointFilter: { value: options.checkpointReceived } })) {
+      result.expectedRowIds.push(row.meta.key);
+    }
+    // ...then make one fresh modification the query MUST pick up
+    result.expectedRowIds.push(await this.#createTestRow(options.meta.id));
+    return result;
+  }
+
+  /** @type SDKTest.TestFeatureProvider["matchFilterPrepare"] */
+  async matchFilterPrepare(options) {
+    const uid = SDKTest.makeUid();
+    /** @type SDKTest.MatchData[] */
+    const results = [];
+
+    switch (options.meta.id) {
+      case "companies": {
+        if (options.rule !== "name[ci]") return undefined; // undefined = rule cannot be tested
+        const name = `${testRecordPrefix} Match Co ${uid}`;
+        const [company] = await this.#connection.upsert.companies([{ name }]);
+        results.push({
+          // a [ci] rule: deliberately change the case - the match must still land
+          srcData: [{ srcRowId: "src-1", match: { name: name.toUpperCase() } }],
+          destFieldPath: [{ path: "name" }],
+          expectedMatchRowIds: [company.meta.key],
+        });
+        break;
+      }
+      case "contacts": {
+        const email = `${testRecordPrefix}.${uid}@example.com`;
+        const first = `${testRecordPrefix}-First-${uid}`;
+        const last = `Last-${uid}`;
+        const [contact] = await this.#connection.upsert.contacts([
+          { email, first_name: first, last_name: last, company_id: this.#fixtureCompanyId },
+        ]);
+        switch (options.rule) {
+          case "email[ci]": {
+            results.push({
+              srcData: [{ srcRowId: "src-1", match: { email: email.toUpperCase() } }],
+              destFieldPath: [{ path: "email" }],
+              expectedMatchRowIds: [contact.meta.key],
+            });
+            break;
+          }
+          case "first_and_last_name[ci]": {
+            // RowMatchData carries the parts as firstname/lastname - the rule needs BOTH
+            results.push({
+              srcData: [{ srcRowId: "src-1", match: { firstname: first.toUpperCase(), lastname: last.toUpperCase() } }],
+              expectedMatchRowIds: [contact.meta.key],
+            });
+            break;
+          }
+          default:
+            return undefined;
+        }
+        break;
+      }
+      default:
+        return undefined;
+    }
+    return results;
+  }
+
+  /** @type SDKTest.TestFeatureProvider["matchFilterCleanup"] */
+  async matchFilterCleanup(options) {
+    // the rows we created ARE the expected matches - delete them
+    const keys = options.testData.flatMap((d) => d.expectedMatchRowIds);
+    await this.#deleteRows(options.meta.id, keys);
+  }
+
+  /** @type SDKTest.TestFeatureProvider["relatedFilterPrepare"] */
+  async relatedFilterPrepare(options) {
+    // contacts declares relationship "companies-company_id" (12). The harness will
+    // 1) collect the 'from' contact via idsFilter (srcRowId), then 2) query companies with
+    //    relatedFilter { otherObjectId: "contacts", otherRelationshipId, otherRows: [contact] }
+    // and expect EXACTLY expectedRelatedRowIds back.
+    if (options.fromObjectMeta.id !== "contacts" || options.fromRelationshipId !== "companies-company_id") {
+      return undefined;
+    }
+    const uid = SDKTest.makeUid();
+    const [company] = await this.#connection.upsert.companies([{ name: `${testRecordPrefix} Rel Co ${uid}` }]);
+    const [contact] = await this.#connection.upsert.contacts([
+      {
+        email: `${testRecordPrefix}.rel.${uid}@example.com`,
+        first_name: testRecordPrefix,
+        last_name: `Rel ${uid}`,
+        company_id: company.meta.key, // the FK that drives the relationship
+      },
+    ]);
+    // OneToOne cardinality: exactly one related company expected
+    return [{ srcRowId: contact.meta.key, expectedRelatedRowIds: [company.meta.key] }];
+  }
+
+  /** @type SDKTest.TestFeatureProvider["relatedFilterCleanup"] */
+  async relatedFilterCleanup(options) {
+    for (const testData of options.testDatas) {
+      await this.#deleteRows("contacts", [testData.srcRowId]); // child first...
+      await this.#deleteRows("companies", testData.expectedRelatedRowIds); // ...then parent
+    }
+  }
+
+  /** @type SDKTest.TestFeatureProvider["upsertPrepare"] */
+  async upsertPrepare(options) {
+    // This hook also feeds the DELETE test: there is no deletePrepare - the harness calls
+    // upsertPrepare twice, inserts both rows, deletes one and verifies exactly that one
+    // is gone. Keep inserts repeatable and unique (makeUid).
+    const uid = SDKTest.makeUid();
+    /** @type SDKTest.UpsertData */
+    const result = { insert: {}, update: {}, verifyFields: [], issueSimulators: [] };
+
+    switch (options.meta.id) {
+      case "companies": {
+        /** @type Parameters<Upsert["companies"]>[0][0] */
+        const insert = { name: `${testRecordPrefix} Co ${uid}`, domain: `${uid}.example.com` };
+        /** @type Parameters<Upsert["companies"]>[0][0] */
+        const update = { domain: `${uid}-updated.example.com` };
+        result.insert = insert;
+        result.update = update;
+        // the harness re-queries the row after each write and compares these paths
+        result.verifyFields.push([{ path: "domain" }]);
+        break;
+      }
+      case "contacts": {
+        /** @type Parameters<Upsert["contacts"]>[0][0] */
+        const insert = {
+          email: `${testRecordPrefix}.${uid}@example.com`,
+          first_name: testRecordPrefix,
+          last_name: `Contact ${uid}`,
+          company_id: this.#fixtureCompanyId,
+        };
+        /** @type Parameters<Upsert["contacts"]>[0][0] */
+        const update = { last_name: `Contact ${uid} updated` };
+        result.insert = insert;
+        result.update = update;
+        result.verifyFields.push([{ path: "last_name" }]);
+
+        // contacts declares canUpsertClean (12): when asked, supply deliberately-broken
+        // inserts stating EXACTLY which issue upsertClean must report and whether it is
+        // fatal. The harness applies each simulator's fields over the insert row.
+        if (options.withIssueSimulators) {
+          result.issueSimulators = [
+            // email is mandatory on add -> ValueRequired, fatal
+            { type: "ValueRequired", upsert: { email: undefined }, fatal: true },
+            // an invalid email is cleansed (the field is REMOVED)... which then violates
+            // the mandatory-on-add constraint, so this issue escalates to fatal
+            { type: "EmailInvalid", upsert: { email: "not-an-email" }, fatal: true },
+            // unknown option on a sandbox custom picklist field: upsertClean's
+            // cleanAction "DeleteUpsertField" drops the field and the row still
+            // writes -> NOT fatal (06, 12)
+            { type: "OptionNotFound", upsert: { t_shirt_size: "no-such-option" }, fatal: false },
+          ];
+        }
+        break;
+      }
+      case "salesorders": {
+        // composite-key object: the ADD carries the parent part (business_unit) as a
+        // plain mandatory field - there is NO key yet. The harness copies the key the
+        // insert response returns ($compositeKey) onto the update row for us (11, 12).
+        // "emea" is an existing business unit on the Acme sandbox (units are fixtures).
+        /** @type Parameters<Upsert["salesorders"]>[0][0] */
+        const insert = { business_unit: "emea", amount: 42, status: "draft" };
+        /** @type Parameters<Upsert["salesorders"]>[0][0] */
+        const update = { status: "confirmed" };
+        result.insert = insert;
+        result.update = update;
+        result.verifyFields.push([{ path: "status" }]);
+        break;
+      }
+    }
+    return result;
+  }
+
+  /** @type SDKTest.TestFeatureProvider["upsertCleanup"] */
+  async upsertCleanup(options) {
+    // the harness hands back the rows it inserted - delete them right away. This is the
+    // ONLY cleanup path for salesorders test rows: they have no prefixable text field,
+    // so the after() sweep cannot find them.
+    await this.#deleteRows(options.meta.id, options.inserted.map((r) => r.meta.key));
+  }
+
+  // === private helpers ======================================================
+
+  /** Create one prefixed row; creating counts as a modification for checkpoint tests.
+   * @param {string} objectId @returns {Promise<string>} the new row's key */
+  async #createTestRow(objectId) {
+    const uid = SDKTest.makeUid();
+    switch (objectId) {
+      case "companies": {
+        const [row] = await this.#connection.upsert.companies([{ name: `${testRecordPrefix} Co ${uid}` }]);
+        return row.meta.key;
+      }
+      case "contacts": {
+        const [row] = await this.#connection.upsert.contacts([
+          {
+            email: `${testRecordPrefix}.${uid}@example.com`,
+            first_name: testRecordPrefix,
+            last_name: `Contact ${uid}`,
+            company_id: this.#fixtureCompanyId,
+          },
+        ]);
+        return row.meta.key;
+      }
+      default:
+        throw new Error(`#createTestRow not implemented for ${objectId}`);
+    }
+  }
+
+  /** Delete rows by key. @param {string} objectId @param {string[]} keys */
+  async #deleteRows(objectId, keys) {
+    if (keys.length === 0) return;
+    switch (objectId) {
+      case "companies":
+        await this.#connection.delete.companies(keys.map((id) => ({ id })));
+        break;
+      case "contacts":
+        await this.#connection.delete.contacts(keys.map((id) => ({ id })));
+        break;
+      case "salesorders":
+        // composite-key deletes address rows by the SAME key string queries returned (11)
+        await this.#connection.delete.salesorders(keys.map((k) => ({ "$compositeKey": k })));
+        break;
+    }
+  }
+
+  /** Sweep sm_test* leftovers - children (contacts) before parents (companies). */
+  async #sweepTestData() {
+    /** @type string[] */
+    const staleContacts = [];
+    for await (const row of this.#connection.query.contacts()) {
+      if (String(row.data.email || "").startsWith(`${testRecordPrefix}.`)) staleContacts.push(row.meta.key);
+    }
+    /** @type string[] */
+    const staleCompanies = [];
+    for await (const row of this.#connection.query.companies()) {
+      if (String(row.data.name || "").startsWith(testRecordPrefix)) staleCompanies.push(row.meta.key);
+    }
+    if (staleContacts.length || staleCompanies.length) {
+      this.#log.info(`Sweeping ${staleContacts.length} test contact(s), ${staleCompanies.length} test company(ies)`);
+    }
+    await this.#deleteRows("contacts", staleContacts);
+    await this.#deleteRows("companies", staleCompanies);
+  }
+}
+```
+
+## What to notice (the parts assistants most often get wrong)
+
+1. **Every created record is prefixed** (`sm_test`) and `before()`/`after()` sweep the
+   prefix — a crashed run never pollutes the sandbox, and the next run self-heals. Objects
+   with no prefixable text field (salesorders) must be cleaned in `upsertCleanup` via
+   `options.inserted` instead.
+2. **`expectedIdentity` pins the account** — it must equal the sandbox connection's
+   `meta().identity` string exactly; re-pointed credentials then fail loudly instead of
+   letting the suite write into a real account.
+3. **`undefined` from a prepare hook is a decision, not a default.** For
+   `checkpointFilterStep1/2Prepare`, `matchFilterPrepare`, `relatedFilterPrepare`,
+   `rowFiltersPrepare` and `upsertPrepare` it means "this cannot be tested" (the feature is
+   skipped); but for `listPrepare` it means "run the list and just verify some rows", and for
+   `idsFilterPrepare` it means "find two ids via a list query". Returning `undefined` by
+   accident silently drops coverage.
+4. **`mod ObjectTestFeatures.mjs` is generated — regenerate, don't edit.** Scoping
+   (`cannotTestReason`, `sameAsObjectId` and their per-feature/per-rule/per-relationship
+   variants) is layered on from `mod Test.mjs`'s `before()`, so a regeneration never loses it.
+5. **Composite-key objects use the connector's own key strings.** Test ids come from
+   `row.meta.key` verbatim — never hand-assembled; the upsert `insert` has no key at all (the
+   parent part travels as a regular mandatory field) and the harness copies the returned
+   `$compositeKey` onto the `update` row.
+6. **Issue simulators are how `upsertClean` gets exercised.** Each simulator names the exact
+   `UpsertIssueType` and its fatality, and cleansing cascades: an invalid email on a
+   mandatory-on-add field is removed by the cleanse, which then trips `ValueRequired` — so
+   the `EmailInvalid` issue is reported fatal.
+7. **There is no `deletePrepare`.** The delete test calls `upsertPrepare` twice, inserts both
+   rows, deletes one and verifies exactly that one disappeared — so inserts must be
+   repeatable and unique. Soft-deleting APIs declare
+   `delete.isDeleted { path, valueWhenDeleted }` (plus `afterDeleteMaxIndexWaitTimeMs`)
+   instead of expecting row absence.
+8. **Checkpoint tests assume a noisy sandbox.** Both steps collect everything already inside
+   the differential window into `expectedRowIds` before seeding one guaranteed change, and
+   step 2 expects step-1 rows to reappear because the connector's checkpoint deliberately
+   overlaps (05, 12).

package/docs/connector-authoring.md

@@ -25,9 +25,10 @@ locally — the local workspace is for editing, type-checking and pushing via th
 | [09-testing.md](./09-testing.md)                                 | the test harness: `mod Test.mjs` + `mod ObjectTestFeatures.mjs`                                                 |
 | [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)           | style rules, deprecated APIs to avoid, known traps — skim this once regardless                                  |
 | [11-composite-keys.md](./11-composite-keys.md)                   | objects with multi-part identity (child REST resources, junction rows) — key encoding across query/upsert/delete |
-| [12-example-connector.md](./12-example-connector.md)             | **the canonical worked example** — a complete three-object connector (relationships, matching, composite keys, custom fields, upsertClean, query/upsert/delete) on one page |
+| [12-example-connector.md](./12-example-connector.md)             | **the canonical worked connector** — a complete three-object connector (relationships, matching, composite keys, custom fields, upsertClean, query/upsert/delete) on one page; its test-file counterpart is 15 |
 | [13-file-fields.md](./13-file-fields.md)                         | objects carrying file content — fields of `type: "file"`, FileProvider, fields-driven inclusion (do NOT copy the fleet's legacy `row.file`) |
 | [14-vendor-sdks-and-non-http.md](./14-vendor-sdks-and-non-http.md) | vendor npm SDKs (googleapis, AWS, box), database/on-prem `agent_mode: "always"` connectors, SOAP/XML, self-managed rate limiting |
+| [15-example-test-file.md](./15-example-test-file.md)             | **the canonical worked test file** — the complete `mod Test.mjs` + `mod ObjectTestFeatures.mjs` pair for the Acme connector of 12 (TestFeatureProvider, prepare/cleanup hooks, issue simulators, feature scoping) |
 
 ## The 60-second version
 
@@ -151,7 +152,7 @@ Implementing more methods unlocks more platform capability:
 7. **Validate + push**: `sm validate`, `sm status` / `sm diff`, `sm push -m "message"`.
    A push 409 means the platform copy changed: `sm pull`, merge, push again — never force.
 8. **Test harness**: add `mod Test.mjs` (+ generated `mod ObjectTestFeatures.mjs`) so the
-   platform's connector test suite can exercise your objects (09).
+   platform's connector test suite can exercise your objects (09; worked example: 15).
 
 ## Support
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.1",
+  "version": "1.0.2",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,7 +12,7 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "285fcc040f1a62f7472910cf06536de4f33574a6f82c5ac405aa51a6b0797174",
+  "typesContentHash": "1e322fcb4cda8d7a31d350d575470adeb2402009ea1929274efe27d7ccb7e041",
   "dependencies": {
     "@types/node": "*",
     "@syncmatters/script-api": "^1.0.1"