@syncmatters/connector-sdk 1.0.1 → 1.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. package/docs/01-workspace-and-meta-json.md +35 -24
  2. package/docs/03-http-auth-ratelimit.md +5 -1
  3. package/docs/04-meta-objects-fields.md +66 -6
  4. package/docs/05-query.md +71 -6
  5. package/docs/06-upsert-delete.md +11 -0
  6. package/docs/09-testing.md +96 -13
  7. package/docs/10-style-and-pitfalls.md +18 -2
  8. package/docs/11-composite-keys.md +12 -4
  9. package/docs/12-example-connector.md +64 -27
  10. package/docs/15-example-test-file.md +567 -0
  11. package/docs/connector-authoring.md +12 -3
  12. package/lib/connector.d.ts +2 -2
  13. package/package.json +3 -3
package/docs/01-workspace-and-meta-json.md CHANGED
@@ -10,7 +10,7 @@ meta/Connectors/<ConnectorName>/ # one *.meta.json sidecar per source file
10
10
  - **Entry file**: `<ConnectorName>.mjs`, default-exporting the connector class. Its sidecar has
11
11
  `"type": "connector"` and carries the `connector_metadata` block.
12
12
  - **Module files**: additional `.mjs` files in the same folder, imported by the entry (or by
13
- other modules). Sidecar `"type": "module"`.
13
+ other modules). Sidecar `"type": "module"`. Modules **must** be placed in the same folder as the file that imports the module.
14
14
  - **Resource files**: e.g. a bundled OpenAPI spec (`acme-api.yaml`). Sidecar type `"yaml"` /
15
15
  `"json"`. Load big specs lazily (dynamic `import()`), not at module top level.
16
16
  - **Test harness files**: `mod Test.mjs` (sidecar type `"connectortest"`) and the generated
@@ -38,6 +38,12 @@ not exist at runtime.
38
38
  ]
39
39
  ```
40
40
 
41
+ > `sm push` pushes `module_script_file_paths` changes (the CLI maps the paths to platform
42
+ > file ids from the workspace). Every referenced file must be tracked — a reference the CLI
43
+ > can't map holds the whole wiring change back with a warning, so imports never half-wire.
44
+ > A new module and the sidecar edits that wire it can go in ONE push: files are created
45
+ > first, wiring follows automatically.
46
+
41
47
  Import style in code: static `import` for your main module; dynamic `await import("./acme-v3")`
42
48
  for version dispatch so only the selected version's code is loaded:
43
49
 
@@ -50,37 +56,40 @@ switch (version) {
50
56
  break;
51
57
  }
52
58
  default:
53
- throw new SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, `Connector version '${version}' not recognized`);
59
+ throw new SDK.ConnectorError(
60
+ SDK.ErrorCode.InvalidOptionValue,
61
+ `Connector version '${version}' not recognized`,
62
+ );
54
63
  }
55
64
  await this.#core.init(args);
56
65
  ```
57
66
 
58
67
  ## Sidecar top-level fields
59
68
 
60
- | Field | Values / meaning |
61
- |---|---|
62
- | `name` | the file name, e.g. `"Acme.mjs"` |
63
- | `type` | `"connector"` (entry) · `"module"` · `"connectortest"` · `"json"` · `"yaml"` |
64
- | `runnable` | `false` for connector files (they are invoked by the platform, not run directly) |
65
- | `engine` | runtime engine version int; leave as scaffolded unless told otherwise |
66
- | `resource_size` | `""` or `"medium"` — runtime resource class hint |
67
- | `cpu_architecture` | normally `""` |
68
- | `module_script_file_paths` | see above |
69
- | `npm_packages` | name→version map of npm packages this file needs at runtime, e.g. `{"box-node-sdk": "2.4.0"}`. The `sm` CLI unions these into the workspace `package.json` so IntelliSense sees them too. Prefer the SDK's built-in http/utilities over adding packages. |
70
- | `connector_metadata` | entry sidecar only — everything below |
69
+ | Field | Values / meaning |
70
+ | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
71
+ | `name` | the file name, e.g. `"Acme.mjs"` |
72
+ | `type` | `"connector"` (entry) · `"module"` · `"connectortest"` · `"json"` · `"yaml"` |
73
+ | `runnable` | `false` for connector files (they are invoked by the platform, not run directly) |
74
+ | `engine` | runtime engine version int; leave as scaffolded unless told otherwise |
75
+ | `resource_size` | `""` or `"medium"` — runtime resource class hint |
76
+ | `cpu_architecture` | normally `""` |
77
+ | `module_script_file_paths` | see above |
78
+ | `npm_packages` | name→version map of npm packages this file needs at runtime, e.g. `{"box-node-sdk": "2.4.0"}`. The `sm` CLI unions these into the workspace `package.json` so IntelliSense sees them too. Prefer the SDK's built-in http/utilities over adding packages. |
79
+ | `connector_metadata` | entry sidecar only — everything below |
71
80
 
72
81
  ## `connector_metadata` reference
73
82
 
74
- | Field | Meaning |
75
- |---|---|
76
- | `categories` | discovery taxonomy, e.g. `["sales"]`, `["file"]`, `["marketing","sales"]` |
77
- | `agent_mode` | `"never"` = runs in cloud · `"always"` = runs on the customer's on-premise agent · `"optional"` = per-connection (agent used when the connection selects one) |
78
- | `oauth2_mode` | `"connector"` = one shared OAuth app for every account · `"connection"` = each connection supplies its own client id/secret. Omit for non-OAuth connectors. |
79
- | `oauth2_provider` | the OAuth provider definition (below) |
80
- | `webhooks_provider` | `{"mode": "custom.perconnection"}` (each connection registers its own webhook) or `{"mode": "custom.perconnector", "url": ..., "webhook_uid": ...}` (one shared endpoint) |
81
- | `settings` | the connection-settings form (below) |
82
- | `icon_url` | connector logo URL |
83
- | `connector_name`, `type_id`, `application_name`, `application_icon_url` | platform/branding ids — usually left as scaffolded |
83
+ | Field | Meaning |
84
+ | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
85
+ | `categories` | discovery taxonomy, e.g. `["sales"]`, `["file"]`, `["marketing","sales"]` |
86
+ | `agent_mode` | `"never"` = runs in cloud · `"always"` = runs on the customer's on-premise agent · `"optional"` = per-connection (agent used when the connection selects one) |
87
+ | `oauth2_mode` | `"connector"` = one shared OAuth app for every account · `"connection"` = each connection supplies its own client id/secret. Omit for non-OAuth connectors. |
88
+ | `oauth2_provider` | the OAuth provider definition (below) |
89
+ | `webhooks_provider` | `{"mode": "custom.perconnection"}` (each connection registers its own webhook) or `{"mode": "custom.perconnector", "url": ..., "webhook_uid": ...}` (one shared endpoint) |
90
+ | `settings` | the connection-settings form (below) |
91
+ | `icon_url` | connector logo URL |
92
+ | `connector_name`, `type_id`, `application_name`, `application_icon_url` | platform/branding ids — usually left as scaffolded |
84
93
 
85
94
  ### `settings`
86
95
 
@@ -126,7 +135,9 @@ consent UI: items carry `label` plus `read`/`write`/`full` scope strings and opt
126
135
 
127
136
  - `.sm/` (CLI state), and the CLI-generated `package.json` / `jsconfig.json` / `AGENTS.md`
128
137
  managed block at the workspace root. They are editor tooling, never pushed.
129
- - Generated `mod ObjectTestFeatures.mjs` files (regenerate instead).
138
+ - Generated `mod ObjectTestFeatures.mjs` files — regenerate with `sm connector features`
139
+ (or the web UI's feature-generate action) after `meta()` changes; new relationships/match
140
+ rules are not tested until regenerated ([09-testing.md](./09-testing.md)).
130
141
  - System-connector stubs: if a connector's settings include a hidden
131
142
  `systemFlag*` boolean, the real implementation is embedded in the platform runtime and the
132
143
  `.mjs` stub is deliberately a no-op — don't implement it.
package/docs/03-http-auth-ratelimit.md CHANGED
@@ -76,7 +76,7 @@ async #call(request) {
76
76
  to wait before that retry. The proven policy shape:
77
77
 
78
78
  ```js
79
- const canRetry = (err, method, url, httpCode) => {
79
+ const canRetry = (err, method, _url, httpCode) => {
80
80
  if ([400, 403, 404].includes(httpCode)) return false; // client errors don't heal
81
81
  if (method === "GET") return true; // reads are idempotent
82
82
  if ([429, 502, 504].includes(httpCode)) return true; // throttle/gateway blips
@@ -85,6 +85,10 @@ const canRetry = (err, method, url, httpCode) => {
85
85
  };
86
86
  ```
87
87
 
88
+ The parameters are positional — keep every position up to the last one you read, and prefix
89
+ the ones you don't use with `_` (as with `_url` above, or `(_err, method, _url, httpCode)` in
90
+ a shorter policy) so `checkJs` doesn't flag them as unused.
91
+
88
92
  Non-idempotent writes: only retry when you know the API tolerates it. One real-world subtlety:
89
93
  writes that reference a _freshly created picklist option_ can fail transiently while the option
90
94
  propagates — returning ~10000 (10s backoff) from `canRetry` for that specific error is the
package/docs/04-meta-objects-fields.md CHANGED
@@ -31,9 +31,13 @@ async meta() {
31
31
  queryFields: [ /* ObjectField[] returned by query */ ],
32
32
  upsertFields: [ /* ObjectField[] accepted by upsert */ ],
33
33
  deleteFields: [ /* ObjectField[] needed to identify rows to delete */ ],
34
- queryFilter: { // system-specific extra filters
35
- rowFilter: [ { id: "modifiedOnOrAfter", type: "string" } ],
36
- propertyFilter: [ { id: "fields", type: "array", arrayType: "string" } ],
34
+ queryFilter: { // system-specific extra filters (rowFilter only - propertyFilter is deprecated)
35
+ rowFilter: [
36
+ { id: "companyType", name: "Company type", type: "string",
37
+ optionValues: [{ id: "customer" }, { id: "vendor" }, { id: "other" }],
38
+ // where users may combine this filter with the standard filters (05):
39
+ extends: [{ type: "checkpoint" }, { type: "list" }] },
40
+ ],
37
41
  },
38
42
  relationships: [ // enables related-row queries
39
43
  { id: "companies-companyId", name: "Company", relObjectId: "companies",
@@ -50,9 +54,9 @@ Other fields: `settingsMeta`/`settingsValues` (per-object settings), `order` (so
50
54
  rather than setting them `false` — `undefined` and `false` both mean "unsupported", and omitted
51
55
  reads cleaner.
52
56
 
53
- Only make a capability conditional when the API forces it: e.g. objects whose REST path embeds
54
- `{parentId}` can't `canQueryList` (there's no "all parents" call) — give them `canQueryByIds`
55
- and a `relatedFilter` path instead.
57
+ Set flags only for behaviour you implemented — but investigate the workaround patterns below
58
+ before concluding a capability is impossible. "The API has no list-all endpoint" is not the
59
+ same as "no list support". See [the query capability bar](#the-query-capability-bar).
56
60
 
57
61
  ## `ObjectField`
58
62
 
@@ -109,6 +113,11 @@ upsertFields.unshift({
109
113
  `"first_and_last_name[ci]"`, `"field_value_equals[ci]"` (`[ci]` = case-insensitive). Declare
110
114
  only rules `query()` actually implements via `matchFilter` — see [05-query.md](./05-query.md).
111
115
 
116
+ **Commonly forgotten:** an object that supports `idsFilter` can ALWAYS offer
117
+ `"field_value_equals[ci]"` — with `canMatch: true` on its id field, the match is just a
118
+ lookup of the supplied value by id. There is no reason for such an object to declare no
119
+ match rules at all; add this floor even when the API has no search endpoint.
120
+
112
121
  ### Relationships
113
122
 
114
123
  Relationships are defined on the source object and declare that given a row from the source object, the relationship allows row(s) from another object (`relObjectId`) that relate to this row to be collected. The canonical example is collecting the Company record that a Contact relates to.
@@ -117,6 +126,57 @@ Relationships are defined on the source object and declare that given a row from
117
126
  `id` unique per object; `data` is yours (a common pattern stores the FK field name so `query()`
118
127
  can build the related filter). Declaring relationships enables `relatedFilter` queries.
119
128
 
129
+ **Which object declares it?** The one whose row you START from — not necessarily the child:
130
+
131
+ - _Child holds a lookup to its parent_ (Contact → Company): declare on the **child**
132
+ (`contacts` gets `relObjectId: "companies"`, `OneToOne`). Related queries then arrive on
133
+ `companies`.
134
+ - _Parent owns children with no child list endpoint_ (a Flow owns its Executions and the API
135
+ only exposes `/flows/{id}/executions`): declare on the **parent**
136
+ (`flows` gets `relObjectId: "executions"`, `OneToMany`). Related queries then arrive on the
137
+ **child** (`executions`, with `otherObjectId: "flows"`) — that is how consumers reach child
138
+ rows at all, so pair it with `canQueryByIds` on the child
139
+ ([05-query.md](./05-query.md#relatedfilter) has the direction rules and a checklist).
140
+
141
+ Declaring it on the wrong object doesn't fail here — it fails later, when
142
+ `relatedFilter` hands your `query()` keys of the wrong object. Decide direction at `meta()` time.
143
+
144
+ ## The query capability bar
145
+
146
+ Syncs are built from these capabilities — an object missing one is an object syncs can only
147
+ partially use. **For every object, strive to support all five; omit a flag only after the
148
+ workaround patterns below have failed**, and note why in a comment so the next author doesn't
149
+ re-investigate:
150
+
151
+ 1. `canQueryList`
152
+ 2. `canQueryByIds`
153
+ 3. `canQueryByCheckpoint`
154
+ 4. `matchRules` (+ `matchFilter` in `query()`)
155
+ 5. `relationships` (+ `relatedFilter` in `query()`)
156
+
157
+ The recurring "the API is awkward" shapes, and the patterns that beat them:
158
+
159
+ - **No cross-parent list endpoint → fan-out list.** Child resources reachable only via
160
+ `/parents/{id}/children` can still `canQueryList`: enumerate the parents (cache their ids in
161
+ `queryState` or a `kvStore`), then page each parent's children in turn. Pair it with an
162
+ optional parent-id `rowFilter` (`extends: [{ type: "list" }, { type: "checkpoint" }]`) so
163
+ callers can scope to one parent and skip the fan-out.
164
+ - **No "changed-since" cursor → checkpoint from what the API has.** A created/modified date
165
+ param (`DateCreatedFrom`, `modified_since`, an OData `$filter`) mapped from
166
+ `checkpointFilter.value` is a legitimate checkpoint implementation — for insert-only
167
+ objects a creation-date high-water mark is exactly right. Keep the overlap rule (05).
168
+ - **Match via whatever search the API offers.** `matchRules` don't need a dedicated search
169
+ API — a filter param on the list endpoint (`To=`, `ContentName=`, `$filter=email eq ...`)
170
+ implements `email[ci]`/`name[ci]` fine. Fetch candidates, compare case-insensitively
171
+ yourself, hand them to `matchFilter.canUse`. And the floor: `canQueryByIds` alone already
172
+ implements `field_value_equals[ci]` on the id field (see Match rules above).
173
+ - **Parent-scoped children → relationship on the PARENT** (see
174
+ [Relationships](#relationships)) so `relatedFilter` traversal works — in ADDITION to
175
+ fan-out list, not instead of it.
176
+
177
+ The flags-are-promises rule ([10](./10-style-and-pitfalls.md)) still holds: implement first,
178
+ then declare. The bar is about how hard to try before giving up, not about optimistic flags.
179
+
120
180
  ## Building `meta()` maintainably
121
181
 
122
182
  Never hand-maintain a big field list twice. The proven generators, in order of preference:
package/docs/05-query.md CHANGED
@@ -77,7 +77,7 @@ Rules that follow from the lifecycle:
77
77
  | `queryOptions.matchFilter` | find rows matching people/records — below |
78
78
  | `queryOptions.relatedFilter` | rows related to another object's rows — below |
79
79
  | `queryOptions.randomFilter` | random sample up to `limit` |
80
- | `queryOptions.rowFilter` / `propertyFilter` | your system-specific filters as declared in `meta().queryFilter` |
80
+ | `queryOptions.rowFilter` | your system-specific filters as declared in `meta().queryFilter.rowFilter` — may arrive WITH a standard filter when declared `extends` (below) |
81
81
 
82
82
  Throw `SDK.ConnectorError(SDK.ErrorCode.InvalidFilterParameter, ...)` when asked for a filter
83
83
  the object didn't declare — it's the single most-thrown error code in the fleet, and it fails
@@ -104,6 +104,47 @@ the final page; the platform stores it and hands it back as `checkpointFilter.va
104
104
  rows modified while your query ran would otherwise be lost forever. Re-reading a few rows is
105
105
  free; losing one is a data bug.
106
106
 
107
+ ### rowFilter — user-parameterized augmentation of the standard filters
108
+
109
+ The system-specific filters you declared in `meta().queryFilter.rowFilter`
110
+ ([04](./04-meta-objects-fields.md)) arrive as `queryOptions.rowFilter` — an object keyed by
111
+ your declared filter ids. Each declaration's **`extends`** array is the contract for
112
+ COMBINING them with the standard filters:
113
+
114
+ ```js
115
+ // meta(): let users scope companies by type, alone or on top of list/checkpoint queries
116
+ queryFilter: {
117
+ rowFilter: [
118
+ { id: "companyType", name: "Company type", type: "string",
119
+ optionValues: [{ id: "customer" }, { id: "vendor" }, { id: "other" }],
120
+ extends: [{ type: "checkpoint" }, { type: "list" }] },
121
+ ],
122
+ },
123
+ ```
124
+
125
+ `extends: [{ type: "checkpoint" | "list" | "match", mandatory? }]` declares which standard
126
+ filters this rowFilter may arrive WITH — so "vendors changed since the checkpoint" is ONE
127
+ query: `queryOptions.checkpointFilter` and `queryOptions.rowFilter.companyType` together,
128
+ AND semantics. `mandatory: true` means the standard filter never arrives on this object
129
+ without the rowFilter (e.g. a child object whose list endpoint requires a parent id). Without
130
+ `extends` the filter stands alone.
131
+
132
+ In `query()`, treat each declared rowFilter as one more constraint alongside whatever
133
+ standard filter drives the query:
134
+
135
+ ```js
136
+ const companyType = options.queryOptions?.rowFilter?.companyType;
137
+ if (companyType) params.push(`type=${encodeURIComponent(companyType)}`); // AND with modified_since etc.
138
+ ```
139
+
140
+ and keep throwing `InvalidFilterParameter` for filters the object didn't declare. The test
141
+ contract has a `rowFiltersPrepare` hook, but the suite does not exercise rowFilters yet —
142
+ cover yours with a direct query in another prepare hook
143
+ ([09-testing.md](./09-testing.md)).
144
+
145
+ > `queryFilter.propertyFilter` is **deprecated** — the standard `fields` filter supersedes it.
146
+ > Don't declare it in new connectors ([10](./10-style-and-pitfalls.md)).
147
+
107
148
  ### matchFilter
108
149
 
109
150
  `{ rule, selectOrder?, destFieldPath?, srcData: [{ srcRowId, match: { id?, name?, email?, domain?, firstname?, lastname?, custom? } }], canUse }`.
@@ -124,11 +165,35 @@ candidates }])` — the platform reserves un-mapped matches and returns the sele
124
165
 
125
166
  Receives rows from a source object (`otherObjectId`) that has a defined relationship to this object, and collect the related rows from this object.
126
167
 
127
- `{ otherObjectId, otherRelationshipId, otherRows }`. Resolve the rows from a relationship declared in
128
- `meta()` (your `relationship.data` typically holds the FK field), turn `otherRows` into an id
129
- lookup, and return rows tagged with
130
- `relationship: { srcRowId, srcObjectId: otherObjectId, relationshipId: otherRelationshipId }` —
131
- one row instance per source row it relates to.
168
+ `{ otherObjectId, otherRelationshipId, otherRows }`. DIRECTION: the platform queries the
169
+ relationship's TARGET object (`relObjectId`) and passes the DECLARING object's rows — contacts
170
+ declare contact→company, so a related query arrives on `companies` with
171
+ `otherObjectId: "contacts"`. The same rule covers the inverse, parent-owns-children shape:
172
+ when the PARENT declares `flows → executions` (`OneToMany`, because the API only exposes
173
+ `/flows/{id}/executions`), the related query arrives on the CHILD (`executions`) with
174
+ `otherObjectId: "flows"` and `otherRows` = flow rows — your child `query()` reads each parent
175
+ row's key and calls the nested endpoint. Which object declares the relationship is a `meta()`
176
+ decision — see [04-meta-objects-fields.md](./04-meta-objects-fields.md#relationships).
177
+
178
+ The implementation checklist (each step has been gotten wrong in a real connector):
179
+
180
+ 1. Declare the relationship on the SOURCE object in `meta()` (the object rows start from —
181
+ child for lookups, parent for owned children).
182
+ 2. Handle `queryOptions.relatedFilter` in the TARGET object's `query()` (`relObjectId`'s) —
183
+ not the declaring object's.
184
+ 3. Resolve the relationship definition from the OTHER object's metadata:
185
+ `await options.objectMeta(related.otherObjectId)` then find `otherRelationshipId` in its
186
+ `relationships`. It is NOT in `options.meta` — that is this object's metadata, which does
187
+ not declare the relationship. (`relationship.data` typically holds the FK field name.)
188
+ 4. Collect the linking values from `otherRows` — for a lookup they are FK values naming THIS
189
+ object's keys; for parent-owned children they are the parent keys your nested endpoint
190
+ needs.
191
+ 5. Return the related rows tagged with
192
+ `relationship: { srcRowId, srcObjectId: otherObjectId, relationshipId: otherRelationshipId }` —
193
+ one row instance per source row it relates to (the same related row appears once per
194
+ source row).
195
+
196
+ See the worked implementation in [12-example-connector.md](./12-example-connector.md).
132
197
 
133
198
  ## Cross-page scratch state: `kvStore`
134
199
 
package/docs/06-upsert-delete.md CHANGED
@@ -59,6 +59,17 @@ Inspect the API's per-row results. When a row fails, log the response and throw
59
59
  (If your API supports partial success and the platform batch mode is `"atomic"`, failing the
60
60
  whole batch by throwing is correct; don't fabricate per-row placeholders.)
61
61
 
62
+ ### Insert-only objects
63
+
64
+ Some objects create but never update — messages that send on create, execution/run records,
65
+ audit entries. Implement the insert path normally and throw
66
+ `SDK.ConnectorError(SDK.ErrorCode.NotImplimented, ...)` for rows that would update (a row
67
+ arriving with its key set). Two consequences: the object's test spec needs
68
+ `upsert.cannotTestReason` (the harness's upsert test always includes an update phase — see
69
+ [09-testing.md](./09-testing.md#insert-only--immutable-objects)), and if creating a row has
70
+ billable or externally visible side effects (a real message send), make sure test connections
71
+ use sandbox credentials or a no-send setting.
72
+
62
73
  ## `delete(options)` → `Promise<SDK.Row[]>`
63
74
 
64
75
  `options.rows` carry the identifying fields you declared in `deleteFields` (usually just the
package/docs/09-testing.md CHANGED
@@ -11,6 +11,9 @@ space, is the convention):
11
11
  The platform's connector test suite (script type `"connectorTest"`) drives your connector
12
12
  through every capability its `meta()` declares — list/ids/checkpoint/match/related queries,
13
13
  upserts, deletes — using these two files to know what to expect and how to prepare data.
14
+ The complete worked pair for the Acme connector of
15
+ [12-example-connector.md](./12-example-connector.md) is in
16
+ [15-example-test-file.md](./15-example-test-file.md).
14
17
 
15
18
  ## `mod ObjectTestFeatures.mjs` (generated)
16
19
 
@@ -98,15 +101,33 @@ objects.
98
101
 
99
102
  ## Prepare/cleanup hooks and their data types
100
103
 
101
- Every testable feature has a `*Prepare` hook (and usually a `*Cleanup` counterpart) —
102
- returning `undefined` from a prepare hook skips that test:
103
-
104
- - `listPrepare` / `idsFilterPrepare` / `rowFiltersPrepare` — arrange rows and return what the
105
- suite should expect (row-filter testing is easy to forget: provide it when `meta()` exposes
106
- `queryFilter.rowFilter`).
107
- - `checkpointFilterStep1Prepare` / `Step2Prepare` — step 1 seeds data and captures the
108
- checkpoint; step 2 mutates data the differential query must then surface.
104
+ Every testable feature has a `*Prepare` hook (and usually a `*Cleanup` counterpart). Most
105
+ prepare hooks skip their test when they return `undefined` — with two exceptions:
106
+ `listPrepare` returning `undefined` still runs the list test (it just expects some rows), and
107
+ `idsFilterPrepare` returning `undefined` makes the harness find ids via a list query itself.
108
+ To disable those tests use `cannotTestReason` instead
109
+ ([15-example-test-file.md](./15-example-test-file.md) shows the worked semantics):
110
+
111
+ - `listPrepare` / `idsFilterPrepare` — arrange rows and return what the suite should expect.
112
+ (`rowFiltersPrepare` exists in the contract but the suite does NOT run rowFilter tests yet —
113
+ when `meta()` exposes `queryFilter.rowFilter`, cover it yourself with a direct query in a
114
+ hook you already implement.)
115
+ - `checkpointFilterStep1Prepare` / `Step2Prepare` — step 1 returns `expectedRowIds` for a
116
+ query from your chosen starting checkpoint (`nextCheckpoint`); the harness runs it and
117
+ captures the checkpoint the query emits. Step 2 receives that checkpoint
118
+ (`{ meta, step1TestData, checkpointReceived }`) and returns the `expectedRowIds` a query
119
+ FROM it must surface. Mutating data between the steps is one way to produce step-2 rows —
120
+ but not required: on APIs where you can't create rows in-test, pick a step-1 checkpoint far
121
+ enough back that existing rows fall on both sides (remember your connector's overlap window
122
+ re-surfaces recent rows in step 2 — include them in `expectedRowIds`).
109
123
  - `matchFilterPrepare` — returns `MatchData` including `expectedMatchRowIds`.
124
+ - `relatedFilterPrepare({ fromObjectMeta, toObjectMeta, fromRelationshipId })` —
125
+ `fromObjectMeta` is the object where the relationship is DECLARED (the parent, in the
126
+ parent-owns-children shape), `toObjectMeta` is its `relObjectId`. Return
127
+ `{ srcRowId, expectedRelatedRowIds, ... }[]` where `srcRowId` is a row of the DECLARING
128
+ object; the harness fetches it via `idsFilter`, then queries the related object with
129
+ `relatedFilter` and compares against `expectedRelatedRowIds`. The test spec side
130
+ (`relationshipsTo`) also lives on the declaring object's features.
110
131
  - `upsertPrepare` — returns `UpsertData`; use `withIssueSimulators` / `UpsertIssueSimulator`
111
132
  (`verifyFields`, expected issue `type` + `fatal`) to deliberately exercise your
112
133
  `upsertClean` issue reporting.
@@ -114,14 +135,76 @@ returning `undefined` from a prepare hook skips that test:
114
135
  checks a flag field instead of row absence, and `afterDeleteMaxIndexWaitTimeMs` tolerates
115
136
  APIs whose deletions surface asynchronously.
116
137
 
138
+ ## What each test needs from your sandbox data
139
+
140
+ Requirements that only surface as cryptic failures when unmet:
141
+
142
+ - **idsFilter needs two distinct ids.** `idsFilterPrepare` should return `{ id1, id2 }`; ids
143
+ you omit are collected via a list query when the object `canQueryList` — an object with only
144
+ one row (or no list support and no prepare hook) fails with `"No test data"`. Each id is
145
+ then queried alone and must return exactly that one row.
146
+ - **The upsert test is insert THEN update, twice over.** `upsertPrepare` is called twice; both
147
+ rows are inserted, re-queried by `idsFilter` and compared on `verifyFields` (which must be
148
+ non-null on the payload), then BOTH are updated via the declared `upsert.key` and re-queried
149
+ again. There is no insert-only mode — see below for objects that cannot update.
150
+ - **relatedFilterPrepare needs a declaring-object row that actually has related rows** —
151
+ `expectedRelatedRowIds` must be non-empty. In the parent-owns-children shape that means a
152
+ parent with at least one child in the sandbox (create one in `before()` when the API allows;
153
+ otherwise seed the sandbox account once and document it in the test file).
154
+ - **Checkpoint step 2 does not have to create data** — see the hook notes above.
155
+
156
+ ## Insert-only / immutable objects
157
+
158
+ Some objects can never pass the upsert test as written: messages that send on create and are
159
+ immutable after (`update` correctly throws `NotImplimented`), execution/run records, audit
160
+ logs. The harness's upsert test always performs the update phase, so:
161
+
162
+ - Set `upsert.cannotTestReason` on the object's features with a REAL explanation —
163
+ `"Messages are immutable once sent; update is NotImplimented by design"` — so the result
164
+ reads as a documented decision, not a gap. (`delete.cannotTestReason` likewise when rows
165
+ cannot be deleted.)
166
+ - The insert path still deserves coverage: exercise it yourself in another prepare hook you
167
+ already need (e.g. `checkpointFilterStep1Prepare` inserting the row whose id it expects), so
168
+ the code path runs even though the formal upsert test is scoped out.
169
+ - **Billable or externally visible side effects** (a real SMS/WhatsApp send, an email) must
170
+ not fire from tests: point the test connection at sandbox credentials, and when the API has
171
+ no sandbox, gate sends behind a connector setting the test connection sets (a "suppress
172
+ sends" / no-send mode) — then say so in the `cannotTestReason` if that leaves upsert
173
+ untestable.
174
+
117
175
  ## Coverage bar
118
176
 
119
177
  Target **80%+ code coverage** (statements/branches/functions/lines) from the platform's
120
178
  connector test runs — reports are downloadable per run and retained ~90 days. Structure
121
179
  `mod Test.mjs` so every object with declared capabilities is exercised rather than scoped out.
122
180
 
123
- ## Local dev loop
124
-
125
- The harness runs platform-side. Locally you get: `sm validate` (workspace structure),
126
- `checkJs` type checking against the SDK types, `sm push` and a platform-side test run. Keep
127
- `test()` cheap and `meta()` deterministic so the suite stays fast.
181
+ ## Running the suite
182
+
183
+ The harness always executes platform-side. What each entry point actually runs:
184
+
185
+ | Command | What runs |
186
+ | ---------------------------- | ----------------------------------------------------------------------------------------------- |
187
+ | `sm test` | The FULL suite — `connection.test()` + `metaRefresh()` + every object feature above. Identical to the web UI's connector test. |
188
+ | `sm test --only connection` | `connection.test()` only — credentials check, nothing else. The harness has NOT run. |
189
+ | `sm test --only meta` | `connection.metaRefresh()` only. |
190
+ | `sm test --only query` | Every object's plain list query only — no ids/checkpoint/match/related, no writes. |
191
+ | Web UI (connector → test results → run) | Same engine, feature selection via the run dialog. |
192
+
193
+ Do not mistake a passing `--only connection` or `--only query` for suite coverage — only the
194
+ full run exercises the prepare hooks and write features. `sm test` needs a developer token
195
+ with the `scripts:read`, `connections:read` and `scripts:execute` scopes; a 403 from the CLI
196
+ names the missing one. The dev loop around it: `sm validate` (workspace structure), `checkJs`
197
+ type checking against the SDK types, `sm push`, then `sm test`. Keep `test()` cheap and
198
+ `meta()` deterministic so the suite stays fast.
199
+
200
+ Two workspace facts that bite here:
201
+
202
+ - `mod ObjectTestFeatures.mjs` is **generated** — regenerate it with
203
+ `sm connector features` (or the web UI's feature-generate action) after `meta()` changes;
204
+ new relationships/match rules will NOT be tested until you do, because the generated
205
+ `relationshipsTo`/`matchRules` spec still reflects the old metadata. The loop is:
206
+ edit `meta()` → `sm push` → `sm connector features` → `sm test`.
207
+ - The typed test connection (the `API.connections.<connection_name>.Connection` `@typedef` in
208
+ the `mod Test.mjs` example above) resolves from **generated** typings:
209
+ `types/connections/*.d.ts` in a CLI workspace (`sm types` refreshes them; `sm pull` runs it
210
+ automatically) — never hand-write these.
package/docs/10-style-and-pitfalls.md CHANGED
@@ -6,6 +6,10 @@
6
6
  `/** @param {SDK.InitArgs} args @returns {Promise<void>} */`. Fields:
7
7
  `/** @type {SDK.HttpClient | undefined} */ #http;`. Complex local types: `@typedef` blocks at
8
8
  the end of the file. Escape hatches (`/** @type any */ (x)`, `@ts-ignore`) sparingly.
9
+ - **Prefix unused positional parameters with `_`** — callback signatures like
10
+ `canRetry(err, method, url, httpCode)` are positional, so a policy that only reads
11
+ `method`/`httpCode` still declares the earlier slots: `(_err, method, _url, httpCode)`.
12
+ The `_` prefix is what keeps `checkJs`/editors from flagging them as unused.
9
13
  - **Private `#fields`** for all connector state (`#log`, `#http`, `#token`, `#settings`).
10
14
  - First line of `init()`: `SDK.verifyType(this);`.
11
15
  - **Double quotes**, semicolons, CRLF line endings.
@@ -22,6 +26,7 @@
22
26
  | `cacheRefreshOptions()` / `RefreshSpec` / `QueryPage.cacheState` | checkpoint queries ([05-query.md](./05-query.md)) |
23
27
  | `RateLimiter.take()` | `rateLimiter.execute({ cost, exec })` — or just pass the limiter to `httpClient` |
24
28
  | `new SDK.HttpClient(...)` / `new SDK.RateLimiter(...)` classes | `SDK.utilities.httpClient(...)` / `SDK.utilities.rateLimiter(...)` |
29
+ | `queryFilter.propertyFilter` / `queryOptions.propertyFilter` | the standard `fields` filter (`queryOptions.fields`) — don't declare propertyFilter in new connectors |
25
30
 
26
31
  ## The trap list (each one has bitten a real connector)
27
32
 
@@ -55,8 +60,12 @@
55
60
  11. **`NotImplimented` is spelled that way.** The enum member (value `CONN_NOT_IMPLEMENTED`)
56
61
  and several long-standing message strings use the misspelling — it is load-bearing; match
57
62
  it exactly.
58
- 12. **Objects with `{parentId}`-style paths can't support `canQueryList`** — expose
59
- `canQueryByIds` + a relationship for `relatedFilter` traversal instead.
63
+ 12. **Don't give up on `canQueryList`/`canQueryByCheckpoint` for `{parentId}`-scoped
64
+ objects.** The naive reading ("no list-all endpoint → no list") under-delivers: fan out
65
+ over the parents, offer a parent-id `rowFilter` (`extends` list/checkpoint), and map a
66
+ creation/modified-date param to the checkpoint — see
67
+ [the query capability bar](./04-meta-objects-fields.md#the-query-capability-bar). Declare
68
+ the parent relationship as well (`relatedFilter` traversal), not instead.
60
69
  13. **Don't log secrets.** Debug logging of requests/responses must be gated behind an
61
70
  advanced `debug` setting, and auth headers/tokens should be excluded or masked.
62
71
  14. **A hidden `systemFlag*` setting means the connector is a runtime-embedded stub** — the
@@ -65,6 +74,13 @@
65
74
  only when `queryOptions.fields` asks for it ([13-file-fields.md](./13-file-fields.md)) —
66
75
  the `queryFile`/`queryOptions.file`/`Row.file` trio in older connectors is legacy; don't
67
76
  copy it.
77
+ 16. **A relationship declared on the wrong object fails far from the mistake.** Declare it on
78
+ the object rows START from — the child for a lookup (contact→company), the PARENT for
79
+ owned children with no child list endpoint (flow→executions). Get it backwards and
80
+ `relatedFilter` arrives on the wrong object carrying the wrong kind of keys — which
81
+ typically surfaces as a composite-key parse error or "row not found", nowhere near
82
+ `meta()`. Direction rules: [04](./04-meta-objects-fields.md#relationships),
83
+ [05](./05-query.md#relatedfilter).
68
84
 
69
85
  ## Useful utilities you might otherwise reimplement
70
86
 
package/docs/11-composite-keys.md CHANGED
@@ -23,6 +23,11 @@ loudly on garbage, and self-documents which part is which. (A delimiter join lik
23
23
  `5678:1234` is acceptable only when the parts can never contain the delimiter — and it's
24
24
  harder to validate; prefer JSON.)
25
25
 
26
+ The property names need not be field paths — any fixed, self-describing ids satisfy the rules,
27
+ e.g. the API's own resource ids: `{"execution_sid":"FN...","flow_sid":"FW..."}`. What is
28
+ non-negotiable is that the property SET and ORDER are fixed: sort the names, always emit every
29
+ part, so the same row encodes byte-identically forever.
30
+
26
31
  > Changing the encoding of an existing object is a breaking change: every cached row's key
27
32
  > stops matching and the platform sees an entirely new set of rows. Pick the encoding once.
28
33
 
@@ -44,10 +49,13 @@ not conflict with real fielf identifiers on the object. When the caller maps val
44
49
  discrete key fields AND supplies `$compositeKey`, cross-check them against the parsed key parts
45
50
  and throw on mismatch; when the discrete fields are absent, populate them from the parsed key.
46
51
 
47
- Because `{parentId}`-path objects have no "fetch everything" endpoint, they usually can't
48
- declare `canQueryList` without a parent filter — declare a `rowFilter` entry
49
- (`{ id: "parentId", type: "number" }`) and/or a relationship to the parent so rows arrive via
50
- `relatedFilter` (see [04](./04-meta-objects-fields.md), [05](./05-query.md)).
52
+ `{parentId}`-path objects have no "fetch everything" endpoint, but that does NOT mean no
53
+ `canQueryList`: fan out over the parents (enumerate them, page each parent's children), offer
54
+ a parent-id `rowFilter` (`{ id: "parentId", type: "number", extends: [{ type: "list" }, { type: "checkpoint" }] }`)
55
+ so callers can scope to one parent, AND declare the relationship on the parent so rows also
56
+ arrive via `relatedFilter` — see
57
+ [the query capability bar](./04-meta-objects-fields.md#the-query-capability-bar) and
58
+ [05](./05-query.md).
51
59
 
52
60
  ## The helper pair
53
61