@syncmatters/connector-sdk 1.0.4 → 1.0.5

package/docs/04-meta-objects-fields.md

@@ -144,9 +144,10 @@ Declaring it on the wrong object doesn't fail here — it fails later, when
 ## The query capability bar
 
 Syncs are built from these capabilities — an object missing one is an object syncs can only
-partially use. **For every object, strive to support all five; omit a flag only after the
-workaround patterns below have failed**, and note why in a comment so the next author doesn't
-re-investigate:
+partially use ([16-the-end-game-syncs.md](./16-the-end-game-syncs.md) maps each one to the
+sync stage that consumes it). **For every object, strive to support all five; omit a flag
+only after the workaround patterns below have failed**, and note why in a comment so the
+next author doesn't re-investigate:
 
 1. `canQueryList`
 2. `canQueryByIds`

package/docs/05-query.md

@@ -9,22 +9,30 @@ Each return's `queryState` is echoed into the next call, until you return `final
 ```js
 /** @param {SDK.QueryOptions} options @returns {Promise<SDK.QueryPage>} */
 async query(options) {
-  const queryState = options.queryState || { offset: 0, limit: 200, rowCount: 0, startTime: Date.now() };
+  const requestedLimit = options.queryOptions?.limit;
+  const queryState = options.queryState || { offset: 0, pageSize: 200, rowCount: 0, startTime: Date.now() };
 
-  const resp = await this.#call(this.#buildRequest(options, queryState));
+  // honor the caller's limit BOTH ways: request no more rows than still needed...
+  const pageSize = requestedLimit != null
+    ? Math.min(queryState.pageSize, requestedLimit - queryState.rowCount)
+    : queryState.pageSize;
+  const resp = await this.#call(this.#buildRequest(options, queryState, pageSize));
 
   /** @type {SDK.Row[]} */
-  const rows = resp.records.map((r) => ({ meta: { key: String(r.id) }, data: r }));
+  let rows = resp.records.map((r) => ({ meta: { key: String(r.id) }, data: r }));
+  // ...and trim overflow when the API ignores the page size (never return past the limit)
+  if (requestedLimit != null && queryState.rowCount + rows.length > requestedLimit) {
+    rows = rows.slice(0, requestedLimit - queryState.rowCount);
+  }
 
   queryState.offset += rows.length;
   queryState.rowCount += rows.length;
 
   /** @type {SDK.QueryPage} */
   const result = { rows, queryState };
-  const requestedLimit = options.queryOptions?.limit;
   result.finalPage =
     rows.length === 0 ||
-    rows.length < queryState.limit ||
+    rows.length < pageSize ||
     (requestedLimit != null && queryState.rowCount >= requestedLimit);
 
   if (options.queryOptions?.checkpointFilter && result.finalPage) {
@@ -40,8 +48,11 @@ Rules that follow from the lifecycle:
 
 - `queryState` is yours — an opaque cursor (`offset`, `nextPageToken`, `lastId`, ...). Keep it
   JSON-serializable.
-- Respect `options.queryOptions.limit`: stop at it (`finalPage: true`) and trim overflow rows
-  from the last page.
+- **Honor `options.queryOptions.limit` absolutely** — never return more rows than it, across
+  ALL pages combined. Cap the page size you request at the remaining allowance (don't collect
+  rows you'll throw away), trim overflow when the API over-returns anyway, and set
+  `finalPage: true` once the limit is reached. Callers size buffers and previews off this
+  number; over-returning breaks them.
 - Prefer **keyset pagination** (`WHERE id > lastId ORDER BY id`) over offsets when the API
   allows — offset/`after` cursors can skip rows when records are deleted mid-pagination.
 

package/docs/07-events-webhooks.md

@@ -93,10 +93,16 @@ if (expected !== computed) return { events: [] };
 Failed verification: return no events (and optionally log) — throwing turns an attacker's junk
 into your error noise.
 
-## `handleParsedEvent(options)` — keep the cache current
+## `handleParsedEvent(options)` — optional cache hint, usually skipped
 
-After your events fire, the platform calls `handleParsedEvent` per affected object so the
-cached data can be updated without a full re-query:
+**Most connectors do not implement this.** The platform's cache stays correct without it —
+your checkpoint queries pick up changes on the next differential run. `handleParsedEvent` is
+an optional latency optimization the platform calls per affected object after events fire;
+implement it only when one of these actually applies:
+
+- **Deletion events the differential query cannot see** — many APIs' changed-since filters
+  never surface deleted rows, so a `record.deleted` webhook is the only signal. This is the
+  one genuinely load-bearing use:
 
 ```js
 /** @param {SDK.HandleParsedEventOptions} options */
@@ -107,17 +113,20 @@ async handleParsedEvent(options) {
       page: { rows: [ { meta: { key: options.event.key, deleted: true } } ], finalPage: true },
     });
   }
-  // updated-record events: either fetch the row and report it, or report nothing and let the
-  // next differential query pick it up - fetching is better for near-real-time triggers.
+  // updated-record events: usually report NOTHING and let the next differential query pick
+  // the row up; fetch-and-report only when near-real-time trigger freshness matters.
 }
 ```
 
-`options.event` = `{ id, timestamp, key?, data? }`; `reportChanges({ id, page })` takes a normal
-`QueryPage`.
+- **Near-real-time triggers** where waiting for the next differential run is too slow.
+
+(`options.event` = `{ id, timestamp, key?, data? }`; `reportChanges({ id, page })` takes a
+normal `QueryPage`.) If neither case applies, omit the method — an empty or speculative
+implementation is maintenance surface with no benefit.
 
 ## Flow summary
 
 external system → platform webhook endpoint → your `parseEvents` (verify, emit
 `ParsedEvent[]`) → platform de-dups (`uniqueId`), resolves connections (`eventIdentity`),
-fires triggers (`typeId`) → your `handleParsedEvent` per changed object → `reportChanges`
-updates the cache.
+fires triggers (`typeId`) → [optional] your `handleParsedEvent` per changed object →
+`reportChanges` updates the cache.

package/docs/12-example-connector.md

@@ -232,7 +232,11 @@ export default class Acme {
     const queryState = options.queryState ||
       { after: undefined, limit: 200, rowCount: 0, startTime: Date.now() };
 
-    const params = [`limit=${queryState.limit}`];
+    // request no more rows than the caller still needs (05); overflow is trimmed below
+    const pageSize = qo.limit != null
+      ? Math.min(queryState.limit, qo.limit - queryState.rowCount)
+      : queryState.limit;
+    const params = [`limit=${pageSize}`];
     if (queryState.after) params.push(`after=${encodeURIComponent(queryState.after)}`);
 
     if (qo.idsFilter) {

package/docs/16-the-end-game-syncs.md

@@ -0,0 +1,66 @@
+# The end game: how syncs consume your connector
+
+Connectors are not run for their own sake. The platform's flagship consumer is the **sync**:
+a configured pipeline that takes rows from a source system and creates/updates/deletes rows
+in a destination system — differentially, repeatedly, in scheduled group runs, often
+bidirectionally as a pair of syncs. Every capability your `meta()` declares is an input to
+that pipeline; every one you omit removes a sync feature for every customer using your
+connector. This page maps the sync pipeline to the connector contract so you know what each
+method is FOR — and what silently degrades when you skimp.
+
+## The pipeline, from the connector's point of view
+
+A sync run walks each source row through these stages. The left column is the sync engine's
+step; the right column is YOUR code being called (on the source or destination connection):
+
+| Sync stage                                 | Connector capability consumed                                                                                                 |
+| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
+| Collect source rows (differential run)     | source `query()` + `checkpointFilter` — or a FULL re-read every run when `canQueryByCheckpoint` is missing                     |
+| Collect source rows (selected ids / child sync) | source `query()` + `idsFilter` — child syncs resolve referenced rows this way (e.g. the Company a Contact points at)       |
+| Validate previously matched rows           | destination `query()` + `idsFilter` — the match store re-fetches saved destination rows to confirm they still exist            |
+| Collect related source rows                | source `query()` + `relatedFilter` — sync configs select declared relationships to pull rows needed for mapping                |
+| Collect lookup rows                        | source `query()` + `idsFilter` on OTHER objects — field values holding ids of rows from other objects                          |
+| Match rules (no prior match)               | **destination** `query()` + `matchFilter` — link the source row to an existing destination row before ever creating one       |
+| Write to destination                       | destination `upsertClean()` (validate + skip no-op writes) then `upsert()` / `delete()`                                        |
+| Deletion propagation                       | rows with `meta: { key, deleted: true }` from checkpoint streams or `handleParsedEvent`                                        |
+| Near-real-time triggers                    | `parseEvents()` — webhook events fire sync runs without waiting for the schedule                                               |
+| Batching                                   | `optimalBatchSize()` — the engine slices its reads/writes to your preference                                                   |
+
+Two structural notes: match rules run against the **destination** connector (matching finds
+the row to update over there), and a "sync" is usually one direction — a bidirectional setup
+is two syncs consuming both your read AND write paths on the same object.
+
+## What this means for the code you write
+
+- **The [query capability bar](./04-meta-objects-fields.md#the-query-capability-bar) is the
+  sync feature list.** An object without `checkpointFilter` re-reads everything every run
+  (slow, rate-limit hungry — and for big objects, effectively unsyncable on a schedule).
+  Without `idsFilter`, child-sync resolution, lookup collection, and match-store validation
+  all fail. Without `matchRules`, every sync run risks creating duplicates instead of linking
+  to rows that already exist. Without `relationships`, mappings can't reach related data.
+- **Row keys are forever.** The match store persists source-key → destination-key pairs
+  across runs, and checkpoint streams identify rows by key. A key that changes encoding
+  (see [11-composite-keys.md](./11-composite-keys.md)) orphans every stored match and makes
+  cached rows reappear as new — customer-visible duplicates.
+- **The checkpoint overlap rule is a data-integrity rule, not a performance tip.** A missed
+  row in one differential run is missed by every downstream sync until the row happens to
+  change again ([05-query.md](./05-query.md)).
+- **Positional upsert alignment corrupts customer data when wrong.** The engine maps
+  `results[i]` back to the source row it was processing — misalignment stores the WRONG
+  destination id in the match store, and every later run updates the wrong row
+  ([06-upsert-delete.md](./06-upsert-delete.md)).
+- **`upsertClean` is loop insurance.** In bidirectional pairs, writing unchanged values
+  re-triggers the other direction's differential query. `detectFirstChange` skipping no-op
+  writes is what keeps two syncs from ping-ponging a row forever.
+- **`meta: { deleted: true }` rows are how deletions propagate.** If your API exposes
+  deletions and you never emit them, destination systems accumulate rows the source deleted.
+
+## The mindset
+
+When implementing an object, don't ask "what does this API endpoint return?" — ask "can a
+customer schedule a differential, deduplicated, relationship-aware, bidirectional sync of
+this object?" Each "no" in that sentence is a missing capability from the table above, and
+[the capability bar](./04-meta-objects-fields.md#the-query-capability-bar) has the workaround
+patterns for awkward APIs. The platform-side sync configuration (mapping, filters, custom
+`filterFast`/`beforeRun` logic, match rule selection) is the customer's job — giving their
+configuration enough capabilities to work with is yours.

package/docs/17-user-help-pages.md

@@ -0,0 +1,98 @@
+# Writing the connector's user help pages
+
+A finished connector ships with **customer-facing help pages**. They live in the platform
+repo under `docs/help-markdown/connections/<slug>/` (slug = kebab-case product name, e.g.
+`epicor-prophet-21`) and follow a fixed house pattern — copy the structure of an existing
+current page (`greenhouse/` is a clean modern example) rather than inventing one:
+
+| File           | Audience  | Content                                                             |
+| -------------- | --------- | -------------------------------------------------------------------- |
+| `index.md`     | customer  | what it is, how to connect, objects, capabilities & limits           |
+| `scripting.md` | developer | task-based code examples for calling the connection from a script    |
+| `guides/*.md`  | either    | optional deep dives (a complex OAuth setup, webhook configuration)   |
+
+**Derive everything from `meta()` and the sidecar — never from memory.** The settings
+section must mirror `connector_metadata.settings` exactly (same display names, `*(sensitive)*`
+markers on secrets); the objects table lists what `meta()` declares; the operations column is
+computed from the flags (`Q` = query, `U` = `upsertFields`, `D` = `deleteFields`); every
+declared `rowFilter` and match rule deserves a mention or example. If the help page and the
+connector disagree, the page is wrong.
+
+## `index.md` — frontmatter and section order
+
+```yaml
+---
+title: "Greenhouse"
+area: "connections"
+audience: "customer"
+doc_type: "guide"
+summary: "Connect to Greenhouse with a Harvest API key to query and write candidates, jobs, ..."
+auth: "api-key"          # or oauth2-connector / oauth2-connection / basic ...
+agent: "no"              # needs the on-premise agent?
+sync_support: "yes"
+class: "system"
+current_version: "2"     # when the connector has a Version setting
+template_version: 1
+generated: false
+---
+```
+
+Sections, in this order (omit ones that don't apply):
+
+1. **Overview** — one paragraph: what the product is, what the connector does; then the
+   three bullets (`Authentication`, `On-premise agent`, `Sync ready?`).
+2. **Before you begin** — vendor-side prerequisites, each with a link to the OFFICIAL vendor
+   doc plus a one-line click-path ("Configure → Dev Center → API Credential Management →
+   Create New API Key").
+3. **Connection settings** — one bullet per setting, `*(sensitive)*` on secrets, defaults and
+   allowed values stated; `### Advanced settings` for the advanced group.
+4. **Supported objects** — link to the vendor's API reference, then the table
+   `| Name | Operations ^1 | Description |` with the `^1 Q=Query, U=Upsert, D=Delete`
+   footnote. Note that the live per-connection list is in the connection UI.
+5. **Capabilities & limits** — rate limits, timestamp formats, API quirks, insert-only
+   objects, pagination oddities. Start with the standard callout:
+   > In a Sync, source filtering and collecting only changed records (incremental/checkpoint)
+   > are configured in the Sync UI — no code required.
+6. **Versioning & migration** — only when a `Version` setting exists: the version/status
+   table and the "change only after testing" caveat.
+7. **Using this connection in scripts** — the standard `ctx.connections.myX()` snippet plus
+   links to `calling-a-connection.md` and `./scripting.md`.
+8. **See also** — vendor API docs, utility functions.
+
+## `scripting.md` — task-based examples
+
+Frontmatter: `audience: "developer"`, a `slug: "connections/<slug>/scripting"`, one-line
+summary. Open with the standard cross-link paragraph (common pattern lives in
+`calling-a-connection.md`; settings/objects live in `./index.md`). Then one `##` section per
+TASK, each a short runnable snippet — these are integration-SCRIPT examples, so they use the
+script surface, not the connector SDK:
+
+```js
+const greenhouse = await ctx.connections.myGreenhouse();          // ## Connect
+
+const one = await greenhouse.queryOne.Candidates({ idsFilter: ['12345'] });   // by id
+
+for await (const row of greenhouse.query.Candidates({                          // incremental
+  limit: 5,
+  checkpointFilter: { value: '2020-10-08T11:41:28Z' },
+})) { /* ... */ }
+```
+
+Cover, in rough priority: connect · query by id · incremental (checkpoint) query ·
+relatedFilter traversal · matchFilter · your declared rowFilters (show them combined with a
+list/checkpoint query when `extends` allows) · upsert (add and update) · delete · file
+fields when the connector has them. Match the house code style in these pages: single
+quotes, `ctx.log.info(...)` for output, obvious placeholder ids (`'12345'`,
+`test@example.com`), `// do processing here` bodies.
+
+## Rules
+
+- **No secrets, no PII, no real tenant data** — placeholder keys, example.com addresses,
+  obviously fake ids. Vendor click-paths are fine; screenshots are not yours to add.
+- **Don't document deprecated surface** in new pages — `propertyFilter` examples appear in
+  older pages for connectors that still use it; new pages use the standard `fields` filter
+  ([10-style-and-pitfalls.md](./10-style-and-pitfalls.md)).
+- **Insert-only and other contract quirks belong in Capabilities & limits** — "Messages are
+  immutable once sent; updates are not supported" saves every future support ticket.
+- **Keep `summary:` one sentence and verb-led** — it renders in search results and index
+  pages ("Connect to X with Y to query and write Z...").

package/docs/connector-authoring.md

@@ -10,6 +10,12 @@ one external system: how to authenticate, what objects and fields exist, and how
 create, update and delete rows. Your code **executes on the SyncMatters platform**, not
 locally — the local workspace is for editing, type-checking and pushing via the `sm` CLI.
 
+**The end game is syncs**: customers configure differential, deduplicated, relationship-aware
+(often bidirectional) syncs BETWEEN two connectors, and every capability you declare is an
+input to that pipeline — [16-the-end-game-syncs.md](./16-the-end-game-syncs.md) maps each
+sync stage to the connector method it consumes. Build objects to be syncable, not merely
+queryable.
+
 ## Topic files (read what your task touches)
 
 | File                                                             | Read when you are...                                                                                            |
@@ -20,7 +26,7 @@ locally — the local workspace is for editing, type-checking and pushing via th
 | [04-meta-objects-fields.md](./04-meta-objects-fields.md)         | writing `meta()` — objects, fields, flags, keys, relationships, filters                                         |
 | [05-query.md](./05-query.md)                                     | implementing `query()` — paging, checkpoints, ids/match/related filters, `Row`                                  |
 | [06-upsert-delete.md](./06-upsert-delete.md)                     | implementing `upsert()`, `delete()`, `upsertClean()`, `upsertFieldOptions()`, batching                          |
-| [07-events-webhooks.md](./07-events-webhooks.md)                 | webhooks: `parseEvents()`, `handleParsedEvent()`, signature verification                                        |
+| [07-events-webhooks.md](./07-events-webhooks.md)                 | webhooks: `parseEvents()`, signature verification (and the usually-skipped `handleParsedEvent()`)               |
 | [08-errors.md](./08-errors.md)                                   | error handling — `ConnectorError`, `ErrorCode` selection, `HttpError`                                           |
 | [09-testing.md](./09-testing.md)                                 | the test harness: `mod Test.mjs` + `mod ObjectTestFeatures.mjs`                                                 |
 | [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)           | style rules, deprecated APIs to avoid, known traps — skim this once regardless                                  |
@@ -29,6 +35,8 @@ locally — the local workspace is for editing, type-checking and pushing via th
 | [13-file-fields.md](./13-file-fields.md)                         | objects carrying file content — fields of `type: "file"`, FileProvider, fields-driven inclusion (do NOT copy the fleet's legacy `row.file`) |
 | [14-vendor-sdks-and-non-http.md](./14-vendor-sdks-and-non-http.md) | vendor npm SDKs (googleapis, AWS, box), database/on-prem `agent_mode: "always"` connectors, SOAP/XML, self-managed rate limiting |
 | [15-example-test-file.md](./15-example-test-file.md)             | **the canonical worked test file** — the complete `mod Test.mjs` + `mod ObjectTestFeatures.mjs` pair for the Acme connector of 12 (TestFeatureProvider, prepare/cleanup hooks, issue simulators, feature scoping) |
+| [16-the-end-game-syncs.md](./16-the-end-game-syncs.md)           | **why any of this exists** — how the sync pipeline consumes each connector capability, and what silently degrades when one is missing. Read once before designing `meta()` |
+| [17-user-help-pages.md](./17-user-help-pages.md)                 | writing the connector's customer help pages (`connections/<slug>/index.md` + `scripting.md`) — the house structure, frontmatter, and example conventions |
 
 ## The 60-second version
 
@@ -115,7 +123,7 @@ Implementing more methods unlocks more platform capability:
 | `upsert()` / `delete()`                 | write data                                                          |
 | `upsertClean()`                         | pre-validate + change-detect rows before writing (`canUpsertClean`) |
 | `upsertFieldOptions()`                  | auto-add picklist values (`canUpsertFieldOptions`)                  |
-| `parseEvents()` / `handleParsedEvent()` | receive webhooks and keep caches fresh                              |
+| `parseEvents()`                         | receive webhooks and fire triggers (optional `handleParsedEvent()` adds cache hints — usually skipped, see 07) |
 | `optimalBatchSize()`                    | pick batch sizes per operation                                      |
 
 ## Golden rules (every one is explained in a topic file)
@@ -161,6 +169,8 @@ Implementing more methods unlocks more platform capability:
 9. **Run it**: `sm test` = the full suite (identical to the web UI's connector test — including
    write features, so use a sandbox connection); `--only connection|meta|query` narrows. A
    passing `--only` run is NOT suite coverage (09).
+10. **Document it**: write the customer help pages — `connections/<slug>/index.md` +
+    `scripting.md` in the platform repo, derived from `meta()` and the sidecar (17).
 
 ## Support
 

package/module.d.ts

@@ -3,7 +3,7 @@
  * SDK specific elements that would be of no interest to users of the API.
  */
 export { EventScriptComplete } from "@syncmatters/script-api";
-export type { Cache, FileProvider, HttpRequest, HttpClientOptions, KvCollection, KvIterator, Logger, EventContext, TestResult, Row, ObjectMeta, ObjectField, ObjectFieldOption, RateLimiterOptions, ScriptType, } from "@syncmatters/script-api";
+export type { Cache, FileProvider, HttpRequest, HttpClientOptions, KvCollection, KvIterator, Logger, EventContext, TestResult, Row, ObjectMeta, ObjectField, ObjectFieldOption, RateLimiterOptions, ScriptType, JsonValuePath, JsonValuePathPart, QueryMatchFilter, QueryCheckpointFilter, EventTypeMeta, ObjectIndex, ObjectMetaUpsert, ObjectRowFilter, ObjectRelationship, ObjectFieldConstraints, ObjectSettingMeta, UpsertIssue, UpsertFieldOptionMeta, RowMatchRuleType, SyncSession, SyncOptimalBatchSizeOperation, SyncOptimalBatchSizeBatchType, } from "@syncmatters/script-api";
 export * from "./lib/connector.js";
 export * from "./lib/connector-error.js";
 export { envSet } from "./lib/environment.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.4",
+  "version": "1.0.5",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,9 +12,9 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "b47810ca30415f646b1055512fb4e342e48f3cd7459411c2dabcc7ca29959271",
+  "typesContentHash": "f0f72980b64b81d6019dbfd86756b2c4dc6ff60eb44b4d1824bde2f8a689fd6b",
   "dependencies": {
     "@types/node": "*",
-    "@syncmatters/script-api": "^1.0.3"
+    "@syncmatters/script-api": "^1.0.4"
   }
 }