@syncmatters/connector-sdk 1.0.19 → 1.0.20

package/docs/09-testing.md

@@ -73,7 +73,8 @@ export default class Test {
 
   // prepare hooks - create/arrange the data a given test needs, return what it should expect:
   // listPrepare, idsFilterPrepare, checkpointFilterStep1Prepare / Step2Prepare,
-  // matchFilterPrepare, relatedFilterPrepare, upsertPrepare
+  // matchFilterPrepare, relatedFilterPrepare, rowFiltersPrepare, upsertPrepare
+  // and, for what is not shaped like a feature, customChecks() (below)
 }
 ```
 
@@ -110,10 +111,14 @@ To disable those tests use `cannotTestReason` instead
 ([15-example-test-file.md](./15-example-test-file.md) shows the worked semantics):
 
 - `listPrepare` / `idsFilterPrepare` — arrange rows and return what the suite should expect.
-  (`rowFiltersPrepare` exists in the contract but the suite does NOT run rowFilter tests yet —
-  when `meta()` exposes `queryFilter.rowFilter`, cover it yourself with a direct query in a
-  hook you already implement. `RowFilterData` accepts an `expect` clause like every other
-  feature's data, but it is not evaluated until the row-filter feature ships.)
+- `rowFiltersPrepare` — one `RowFilterData { rowFilter, expectedRowIds, fields?, expect? }` per
+  case, for the filters `meta()` declares in `queryFilter.rowFilter`. The harness queries each
+  with its `rowFilter` (and `fields`), in order, and the rows must be exactly `expectedRowIds`;
+  the first case that fails decides the object's `rowFilter` status (`Row not returned`,
+  `Returned unexpected row`, or the `expect` failures below), details under `rowFilter`. Row
+  filters have no `ObjectFeatures` flag, so a suite without the hook (or one that returns
+  `undefined`) reads `Not tested` with the reason and the object is NOT made Partial; with
+  neither the hook nor declared row filters the status is `Not supported`.
 - `checkpointFilterStep1Prepare` / `Step2Prepare` — the two-step round-trip test of your
   differential contract; see [Checkpoint testing](#checkpoint-testing-what-the-two-steps-prove)
   below, it is the most-misimplemented pair.
@@ -137,7 +142,7 @@ To disable those tests use `cannotTestReason` instead
 ## Expected values and errors: the `expect` clause
 
 The data every prepare hook returns (`ListData`, `IdsFilterData`, `CheckpointStep1Data`,
-`CheckpointStep2Data`, `MatchData`, `RelationshipData`) takes an optional `expect`, for what a
+`CheckpointStep2Data`, `MatchData`, `RelationshipData`, `RowFilterData`) takes an optional `expect`, for what a
 row-id comparison cannot prove: that a value comes back in the right form, or that a call fails
 the way it must.
 
@@ -212,10 +217,24 @@ succeeded, the rows it returned — is recorded in the details either way. What
 | `CheckpointStep2Data` | the step 2 query, from the checkpoint step 1 emitted                                            |
 | `MatchData`           | that match query                                                                                |
 | `RelationshipData`    | the related query (the 'from' row is still fetched by `idsFilter` and must exist)               |
+| `RowFilterData`       | that row-filter query                                                                           |
 
 `expectedRowIds` / `expectedMatchRowIds` / `expectedRelatedRowIds` may be empty when `error` is
 set.
 
+A timeout is the typical row-filter case: SQL Server's statement filter with a `WAITFOR` and a
+`requestTimeoutOverride` shorter than the wait must fail with the driver's timeout, not hang or
+return rows:
+
+```js
+/** @type {SDKTest.RowFilterData} */
+const timeoutCase = {
+  rowFilter: { where: { clause: "1 = 1; WAITFOR DELAY '00:00:05'" }, requestTimeoutOverride: 1000 },
+  expectedRowIds: [],
+  expect: { error: { code: "ETIMEOUT" } },
+};
+```
+
 **Upsert**: `UpsertData.expect` is `{ insert?, update? }`, each an assertion list for this
 data's own row as queried back after that phase. `upsertPrepare` runs once per row, and the
 row's key is only assigned by the insert, so there is nothing to key by. The asserted paths are
@@ -263,6 +282,59 @@ This catches a connector that keys its rows on one column and tags another, whic
 only a sync finds. A spec without `query.key` (anything generated before it existed) skips
 these checks.
 
+## Custom checks: `customChecks()`
+
+Some tests are not shaped like an object feature: a free statement with typed parameters,
+500 sequential statements to prove connections are released, a connection-level timeout, an API
+quirk. They go in `customChecks()`, which returns named checks the harness runs **after the
+feature tests, in declaration order**, each with its own status, duration and details:
+
+```js
+/** @type {SDKTest.TestFeatureProvider["customChecks"]} */
+async customChecks() {
+  return [
+    {
+      id: "exec-typed-parameters", // stable: keys the result and details, and is what --check names
+      name: "$Exec binds typed parameters",
+      objectId: "$Exec", // gated like a feature; omit for a connection-level check
+      timeoutMs: 30000, // default 60 000
+      run: async ({ connection, meta, record }) => {
+        const rows = await collect(connection.query["$Exec"]({ rowFilter: { statement: "SELECT @p0 AS v", parameters: [{ name: "p0", sqlType: "int", value: 42 }] } }));
+        record("returned", { rows }); // a detail on the check, under the usual caps
+        if (rows[0]?.data.v !== 42) throw new Error(`expected 42, got ${rows[0]?.data.v}`);
+      },
+    },
+  ];
+}
+```
+
+- **Status**: `Passed`, `Failed: <message>` (the first line of what `run` threw; the full error
+  is kept as the check's `fatalError`), or `Not tested` with `notTestedReason`. Results are
+  `CustomCheckResult { id, name, status, notTestedReason?, durationMs, fatalError? }`, on the
+  object's result for a check with an `objectId` and on the run's for one without.
+- **Details**: `record(message, data)` adds a detail under `details["check:<id>"]`; the harness
+  adds one saying how the check ended. Same caps and ~48-hour retention as every feature's.
+- **Selection**: a check with an `objectId` whose object is not on the connection (not selected
+  on a two-phase connection, or not returned by `meta()`) reads `Not tested` with the reason —
+  it does not fail. The object need not be one of `objectIds()`; if it is, the harness's object
+  check fails the run first, as it does for any tested object. A check without an `objectId` is
+  connection-level and always runs. `ctx.meta` is the check object's metadata.
+- **Timeout**: `timeoutMs` (default 60 000) is capped at the run's remaining budget when the run
+  has one. A check still running at its timeout fails (`Failed: Timed out after <n>ms`), what it
+  records afterwards is dropped, and the next check runs.
+- **Roll-up**: a failed check fails its object and the run; a check not tested makes a Passed
+  object Partial and keeps a run that would pass at `Partial: Not all features/objects can be
+  tested`.
+- **Which checks a run reaches**: all of them on a full run; with `--object`, the
+  connection-level checks plus the selected objects'; none with `--only
+  connection|meta|discover|query`; exactly the named ones with `--check <id>` (repeatable),
+  whatever else the run tests. An id the suite does not declare fails as `No such check in the
+  test suite`.
+
+A check is the escape hatch, not the default. If the same check turns up in a second connector,
+it belongs in the harness as a feature; and a value assertion on rows a feature already fetches
+belongs in that feature's `expect`, where it reports on the feature.
+
 ## Fixtures on two-phase connections: fixed names, seeded in `before()`
 
 A two-phase test connection collects only the objects it selected. So a fixture object — a
@@ -274,7 +346,7 @@ connection right after it.
 The procedure:
 
 1. Create the fixtures once, with fixed names (the SQLServer suite's `dbo.sm_test_*` pattern).
-2. Run `sm test --only discover`.
+2. Run `sm connector test --only discover`.
 3. Select the fixtures on the test connection.
 4. Run the suite.
 
@@ -406,25 +478,31 @@ The harness always executes platform-side. What each entry point actually runs:
 
 | Command                      | What runs                                                                                      |
 | ---------------------------- | ----------------------------------------------------------------------------------------------- |
-| `sm test`                    | The FULL suite — `connection.test()` + `metaRefresh()` + every object feature above. Identical to the web UI's connector test. |
-| `sm test --only connection`  | `connection.test()` only — credentials check, nothing else. The harness has NOT run.            |
-| `sm test --only meta`        | `connection.metaRefresh()` only — for a connection with a selection, that is `meta()` over the selected objects. |
-| `sm test --only discover`    | `connection.discover()` only (two-phase connectors) — reports the catalogue size, which settings it returned overrides for, and how long it took, then refreshes the selection and checks every selected object is in the catalogue. |
-| `sm test --only query`       | Every object's plain list query only — no ids/checkpoint/match/related, no writes.              |
+| `sm connector test`                    | The FULL suite — `connection.test()` + `metaRefresh()` + every object feature above + every custom check. Identical to the web UI's connector test. |
+| `sm connector test --only connection`  | `connection.test()` only — credentials check, nothing else. The harness has NOT run.            |
+| `sm connector test --only meta`        | `connection.metaRefresh()` only — for a connection with a selection, that is `meta()` over the selected objects. |
+| `sm connector test --only discover`    | `connection.discover()` only (two-phase connectors) — reports the catalogue size, which settings it returned overrides for, and how long it took, then refreshes the selection and checks every selected object is in the catalogue. |
+| `sm connector test --only query`       | Every object's plain list query only — no ids/checkpoint/match/related, no writes, no custom checks. |
+| `sm connector test --object <id>`      | The full suite for the named object(s), plus the connection-level custom checks and those object(s)' checks. |
+| `sm connector test --check <id>`       | Only the named custom check(s) (repeat the flag); no object features unless `--object` is given too. The fast loop for one failing check. |
 | Web UI (connector → test results → run) | Same engine, feature selection via the run dialog.                                   |
 
+(The old short name `sm test` now prints the replacement and exits 2.)
+
 Each feature test records the inputs it ran with and what the connector returned — pass or
 fail — onto the saved result as diagnostic `details` (the ids an `idsFilter` ran with and the
-rows returned; expected vs actual rows per match rule / relationship; upserted rows and their
-query-back). Review them on the web UI's test results screen instead of re-running with extra
+rows returned; expected vs actual rows per match rule / relationship / row filter; upserted rows
+and their query-back; what each custom check recorded). The web UI shows custom checks in a
+"Custom checks" table under each object and under the connection, and `sm connector test`
+prints one line per check after the object lines. Review them on the web UI's test results screen instead of re-running with extra
 logging. Payloads over the size budget are dropped and flagged `truncated`; details hold real
 connection data and are scrubbed ~48 hours after the run (statuses are kept).
 
 Do not mistake a passing `--only connection` or `--only query` for suite coverage — only the
-full run exercises the prepare hooks and write features. `sm test` needs a developer token
+full run exercises the prepare hooks and write features. `sm connector test` needs a developer token
 with the `scripts:read`, `connections:read` and `scripts:execute` scopes; a 403 from the CLI
 names the missing one. The dev loop around it: `sm validate` (workspace structure), `checkJs`
-type checking against the SDK types, `sm push`, then `sm test`. Keep `test()` cheap and
+type checking against the SDK types, `sm push`, then `sm connector test`. Keep `test()` cheap and
 `meta()` deterministic so the suite stays fast.
 
 Three workspace facts that bite here:
@@ -442,8 +520,79 @@ Three workspace facts that bite here:
   `sm connector features` (or the web UI's feature-generate action) after `meta()` changes;
   new relationships/match rules will NOT be tested until you do, because the generated
   `relationshipsTo`/`matchRules` spec still reflects the old metadata. The loop is:
-  edit `meta()` → `sm push` → `sm connector features` → `sm test`.
+  edit `meta()` → `sm push` → `sm connector features` → `sm connector test`.
 - The typed test connection (the `API.connections.<connection_name>.Connection` `@typedef` in
   the `mod Test.mjs` example above) resolves from **generated** typings:
   `types/connections/*.d.ts` in a CLI workspace (`sm types` refreshes them; `sm pull` runs it
   automatically) — never hand-write these.
+
+## Migrating an old test module to `mod Test.mjs`
+
+Many connectors still carry a second suite in `files/Scripts/Tests/mod <Conn>.mjs`, run by the
+loop in `_tst Connector.mjs`: every `test*` method is called, logged as Success or Fail, and the
+run throws at the end. The harness replaces it. Port each method to the first of these that
+fits, and delete the old module and its `_tst` entry when the last method has moved:
+
+1. **An existing feature.** Most methods (list, by id, by ids, by last-modified, match rules,
+   related rows, insert/update, the `test_*_metadata` methods) are already what the harness
+   tests. Port them as prepare data (`listPrepare`, `idsFilterPrepare`, ...), not as checks;
+   metadata methods are covered by the generated spec and the `fieldMetadata` feature.
+2. **An `expect` clause.** A method that checks a returned value's form (a rowversion as hex, a
+   GUID, a decimal's value, a column that must not be echoed) or a call that must fail (a
+   timeout, a rejected id, a `*_not_exist` query) becomes `expect.rows` or `expect.error` on the
+   feature data that makes that call.
+3. **A custom check**, for what is left: a method that is not shaped like a feature.
+
+Porting a method to a custom check:
+
+```js
+// old: files/Scripts/Tests/mod SQLServer v2.mjs
+async testExec() {
+  const rows = await collect(this.conn.query["$Exec"]({ rowFilter: { statement: "SELECT 1 AS one" } }));
+  if (rows[0].data.one !== 1) throw new Error("exec did not return 1");
+  this.log.info("exec ok");
+}
+
+// new: in mod Test.mjs
+async customChecks() {
+  return [
+    {
+      id: "exec",
+      name: "$Exec returns a statement's rows",
+      objectId: "$Exec",
+      run: async ({ connection, record }) => {
+        const rows = await collect(connection.query["$Exec"]({ rowFilter: { statement: "SELECT 1 AS one" } }));
+        record("returned", { rows });
+        if (rows[0]?.data.one !== 1) throw new Error(`exec returned ${JSON.stringify(rows[0]?.data)}`);
+      },
+    },
+  ];
+}
+```
+
+- The method body moves into `run`; `this.conn` becomes `ctx.connection` (or the suite's own
+  typed connection from `before()`), `this.log.info` of what you want to review later becomes
+  `ctx.record`, and the `throw` stays: throwing is how a check fails.
+- Give it a stable `id` (the old method name without `test` is a good one) and, when it
+  exercises one object, that `objectId`, so an unselected object reads Not tested instead of
+  failing.
+- `onlyTest*` / `skipTest*` renames are gone: `--check <id>` runs one check, and a check that
+  must not run is deleted or given a reason in its own code, not renamed.
+- A method that loops or times (500 sequential statements, a large read) keeps its loop; put
+  the timing in `record`, not in a pass/fail threshold, and set `timeoutMs` above what it needs.
+- Setup the old module did in its constructor or `before()` moves to the suite's `before()`,
+  which runs once for features and checks alike. **Seed rows there, never objects**.
+
+What cannot be ported:
+
+- **Creating a table, custom object or other schema per test** (the `SMTST<random>` pattern).
+  A two-phase test connection only collects the objects it selected, so an object made during
+  the run is never on it. Use a fixed fixture instead ([above](#fixtures-on-two-phase-connections-fixed-names-seeded-in-before)):
+  create it once with a fixed name, discover, select it, and let the check or feature seed rows.
+  Tests that only prove "a table can be created" go.
+- **Anything whose pass/fail is a timing threshold.** A threshold needs a per-environment budget
+  the harness does not have; record the timing as a detail instead.
+- **Methods that depend on another method having run first.** Checks run in declaration order,
+  but each must stand alone (a `--check` run may run only one); share state through `before()`.
+- **Platform tests** (`mod ScriptAPI`, `Storage`, `Webhook`, `Drives`, ...) are not connector
+  tests and stay where they are, with `_tst Main.mjs`.

package/docs/15-example-test-file.md

@@ -2,7 +2,7 @@
 
 The test-file counterpart to [12-example-connector.md](./12-example-connector.md): the complete
 `mod Test.mjs` + `mod ObjectTestFeatures.mjs` pair for the Acme connector, on one page. The
-platform's connector test suite AND the CLI's `sm test` both require a file of sidecar
+platform's connector test suite AND the CLI's `sm connector test` both require a file of sidecar
 type `"connectortest"` in the connector's folder — **a connector without one cannot be
 run-tested at all**. Harness conventions and hook semantics live in
 [09-testing.md](./09-testing.md); this page shows them applied end to end against Acme's three
@@ -39,7 +39,7 @@ generated module via `module_script_file_paths`:
 }
 ```
 
-Keep exactly one `"connectortest"` file per connector folder — `sm test` resolves the test
+Keep exactly one `"connectortest"` file per connector folder — `sm connector test` resolves the test
 module from the folder beside the connector's entry file, and several candidates are ambiguous
 (`<Name>.test.mjs` wins when present).
 
@@ -180,6 +180,8 @@ export default class TestFeatureProvider {
   #connection;
   /** @type string */
   #fixtureCompanyId = ""; // shared parent row for contact test data
+  /** @type string */
+  #fixtureVendorId = ""; // the sandbox's one vendor company, for the companyType row filter
 
   /** @type SDKTest.TestFeatureProvider["before"] */
   async before(ctx) {
@@ -213,6 +215,10 @@ export default class TestFeatureProvider {
     const company = await this.#connection.queryOne.companies();
     if (!company) throw new Error("Sandbox has no companies - seed at least one before testing");
     this.#fixtureCompanyId = company.meta.key;
+    for await (const row of this.#connection.query.companies()) {
+      if (row.data.name === "sm_fixture Vendor") this.#fixtureVendorId = row.meta.key;
+    }
+    if (!this.#fixtureVendorId) throw new Error("Sandbox fixture 'sm_fixture Vendor' is missing - create it once as a vendor");
   }
 
   /** @type SDKTest.TestFeatureProvider["after"] */
@@ -494,6 +500,57 @@ export default class TestFeatureProvider {
     await this.#deleteRows(options.meta.id, options.inserted.map((r) => r.meta.key));
   }
 
+  /** @type SDKTest.TestFeatureProvider["rowFiltersPrepare"] */
+  async rowFiltersPrepare(options) {
+    // companies declares one row filter, companyType (12). Acme's API cannot set a company's
+    // type, so the sandbox holds a fixed fixture instead: exactly one vendor, named
+    // "sm_fixture Vendor", created once and never touched by the suite (the fixed-fixture rule, 09).
+    // before() looked its key up; every other sandbox company is a customer.
+    if (options.meta.id !== "companies") return undefined;
+    return [
+      {
+        rowFilter: { companyType: "vendor" },
+        expectedRowIds: [this.#fixtureVendorId],
+        expect: { rows: { "*": [{ path: [{ path: "name" }], equals: "sm_fixture Vendor" }] } },
+      },
+    ];
+  }
+
+  /** @type SDKTest.TestFeatureProvider["customChecks"] */
+  async customChecks() {
+    // what is not shaped like an object feature; each check reports its own status (09)
+    return [
+      {
+        // connection-level (no objectId): always runs
+        id: "sequential-tests",
+        name: "20 sequential test() calls all succeed",
+        run: async ({ record }) => {
+          const started = Date.now();
+          for (let i = 0; i < 20; i++) {
+            const result = await this.#connection.test();
+            if (!result.success) throw new Error(`test() call ${i + 1} failed: ${result.message}`);
+          }
+          record("20 test() calls", { durationMs: Date.now() - started }); // timing is a detail, not a threshold
+        },
+      },
+      {
+        // object-level: reads Not tested, not Failed, if contacts is not selected on the connection
+        id: "contacts-page-walk",
+        name: "Listing three pages of contacts returns no duplicate keys",
+        objectId: "contacts",
+        timeoutMs: 120000,
+        run: async ({ record }) => {
+          const seen = new Set();
+          for await (const row of this.#connection.query.contacts({ limit: 300 })) {
+            if (seen.has(row.meta.key)) throw new Error(`key ${row.meta.key} returned twice`);
+            seen.add(row.meta.key);
+          }
+          record(`${seen.size} distinct contacts`);
+        },
+      },
+    ];
+  }
+
   // === private helpers ======================================================
 
   /** Create one prefixed row; creating counts as a modification for checkpoint tests.
@@ -570,7 +627,7 @@ Nothing in this pair changes when a connector gains `discover()`
 regenerates `mod ObjectTestFeatures.mjs` without it — it warns about the catalogue entries the
 connection did not select, so read that output rather than assuming a shrunken generated file is
 the connector's doing. Widen the test connection's selection rather than scoping the object out;
-`sm test --only discover` ([09](./09-testing.md#running-the-suite)) checks the catalogue against
+`sm connector test --only discover` ([09](./09-testing.md#running-the-suite)) checks the catalogue against
 the selection and prints what is missing.
 
 The same rule applies to fixture objects a suite owns, such as a sandbox table or a custom
@@ -633,3 +690,12 @@ async before({ connection }) {
 10. **`query.key` is generated, like `upsert.key`.** It names the field whose value IS
    `row.meta.key`, and the list and idsFilter tests check every row against it. If the
    generated file lacks it, it predates the check: regenerate.
+11. **Row filters are tested from a fixed fixture.** `rowFiltersPrepare` has to name the exact
+   rows a filter returns, so it needs a sandbox whose answer it knows: here one vendor company
+   created once and looked up in `before()`, never created by the run. Without the hook
+   companies' row filter reads `Not tested`, which does not make the object Partial.
+12. **Custom checks are for what no feature expresses**, and each reports on its own:
+   `sequential-tests` on the connection, `contacts-page-walk` under contacts (Not tested, not
+   Failed, if contacts is not selected). Throwing fails a check; `record` puts what you want to
+   review later on the result; timings are details, never thresholds. Rerun one with
+   `sm connector test Acme --check contacts-page-walk`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.19",
+  "version": "1.0.20",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,9 +12,9 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "acd8d8f87e1b337a1deddca006f59a875fd740b920d602ed59b3a3775a862332",
+  "typesContentHash": "376c97b772af51a69f5155aff7c80362062a74710e0672b3dd0582679b5cfffb",
   "dependencies": {
     "@types/node": "*",
-    "@syncmatters/script-api": "^1.0.20"
+    "@syncmatters/script-api": "^1.0.21"
   }
 }