@syncmatters/connector-sdk 1.0.0 → 1.0.1

package/docs/10-style-and-pitfalls.md

@@ -0,0 +1,100 @@
+# Style rules, deprecated APIs, and the trap list
+
+## Style (workspace `.mjs` connector code)
+
+- **JSDoc types everywhere** — workspaces run TypeScript's `checkJs`. Method signatures:
+  `/** @param {SDK.InitArgs} args @returns {Promise<void>} */`. Fields:
+  `/** @type {SDK.HttpClient | undefined} */ #http;`. Complex local types: `@typedef` blocks at
+  the end of the file. Escape hatches (`/** @type any */ (x)`, `@ts-ignore`) sparingly.
+- **Private `#fields`** for all connector state (`#log`, `#http`, `#token`, `#settings`).
+- First line of `init()`: `SDK.verifyType(this);`.
+- **Double quotes**, semicolons, CRLF line endings.
+- Imports: `import SDK from "@syncmatters/connector-sdk";` (and `@syncmatters/script-api` in test files).
+- One central request wrapper per API; separate `httpClient` instances per rate-limit domain.
+- The platform copy is the source of truth — `sm push` early, `sm pull` before big edits, and a
+  push 409 means merge, never force.
+
+## Deprecated — do not use, migrate away on contact
+
+| Deprecated                                                                    | Use instead                                                                                  |
+| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
+| `ObjectMetaSummary` / `ObjectMetaDetail` (objects with a `detail()` callback) | flat `ObjectMeta[]` in `meta().objects` — the summary form doesn't work on on-premise agents |
+| `cacheRefreshOptions()` / `RefreshSpec` / `QueryPage.cacheState`              | checkpoint queries ([05-query.md](./05-query.md))                                            |
+| `RateLimiter.take()`                                                          | `rateLimiter.execute({ cost, exec })` — or just pass the limiter to `httpClient`             |
+| `new SDK.HttpClient(...)` / `new SDK.RateLimiter(...)` classes                | `SDK.utilities.httpClient(...)` / `SDK.utilities.rateLimiter(...)`                           |
+
+## The trap list (each one has bitten a real connector)
+
+1. **Checkpoint without overlap loses rows.** Emit `checkpoint = startTime - overlap`
+   (60s–5min), never `Date.now()` at completion. Rows modified during the run must be re-read
+   next run.
+2. **Offset/`after` cursors skip rows when records are deleted mid-pagination.** Prefer keyset
+   pagination (`WHERE id > lastId ORDER BY id`) when the API allows it.
+3. **Upsert results misaligned with input rows corrupt data silently.** Keep an index map when
+   splitting insert/update batches; throw if an API response row can't be correlated back.
+4. **A rolling rate limiter alone may still trip 429s** — some APIs also punish concurrency.
+   Compose a `concurrent` limiter with the `rolling` one
+   ([03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)).
+5. **Freshly created picklist options are eventually consistent** — upserts referencing them
+   can fail for ~10-20s. Special-case that error in `canRetry` with a long backoff.
+6. **String-interpolated query languages need escaping** — escape backslash first, then the
+   quote character; skip null/empty values so you never emit `id = 'undefined'`.
+7. **External APIs return duplicates where you least expect** (e.g. reference-target lists).
+   `[...new Set(values)]` before mapping into metadata.
+8. **`meta()` flags are promises.** Declaring `canQueryByCheckpoint`/`matchRules` you didn't
+   implement produces runtime failures in someone else's sync. Derive flags from evidence
+   (does the object actually have a last-modified filter?).
+9. **`parseEvents`' context depends on the webhook mode.** With a `custom.perconnector`
+   (shared-endpoint) webhook it runs statically: guard `init()` with an early return for
+   `mode === "parseEvents"`, never assume settings/tokens exist there, and derive
+   `eventIdentity` from the payload. Only `custom.perconnection` connectors get a fully
+   initialized instance (settings-derived signing secrets are legal there). See
+   [07-events-webhooks.md](./07-events-webhooks.md).
+10. **Unverified webhooks are attacker input.** Verify the signature and return zero events on
+    mismatch — don't throw.
+11. **`NotImplimented` is spelled that way.** The enum member (value `CONN_NOT_IMPLEMENTED`)
+    and several long-standing message strings use the misspelling — it is load-bearing; match
+    it exactly.
+12. **Objects with `{parentId}`-style paths can't support `canQueryList`** — expose
+    `canQueryByIds` + a relationship for `relatedFilter` traversal instead.
+13. **Don't log secrets.** Debug logging of requests/responses must be gated behind an
+    advanced `debug` setting, and auth headers/tokens should be excluded or masked.
+14. **A hidden `systemFlag*` setting means the connector is a runtime-embedded stub** — the
+    `.mjs` in the workspace is deliberately a no-op; don't implement it.
+15. **Files are fields now.** Declare file content as a field of `type: "file"` and include it
+    only when `queryOptions.fields` asks for it ([13-file-fields.md](./13-file-fields.md)) —
+    the `queryFile`/`queryOptions.file`/`Row.file` trio in older connectors is legacy; don't
+    copy it.
+
+## Useful utilities you might otherwise reimplement
+
+The two you will use constantly (they appear in nearly every fleet connector):
+
+- `SDK.utilities.clone(value)` — deep clone. Use before mutating anything you received or
+  return (metadata, rows, options) — shared references are how callers get corrupted.
+- `SDK.utilities.tryGet(() => deeply.nested.maybe.missing)` — returns the value or
+  `undefined`, never throws. The idiom for prodding uncertain API payloads:
+  `const sig = tryGet(() => options.payload.headers["x-signature"][0]);`
+
+Frequently used:
+
+- `dateFormat(date, format)` / `dateParse(text, format)` / `dateTransform` — checkpoint and
+  API date math without hand-rolled parsing.
+- `arrayOfStr(value)` / `arrayOfNum` / `setOfStr` / `setOfNum` — normalize comma/CSV-ish
+  settings values ("a, b,c" → `["a","b","c"]` / a `Set`).
+- `emailCleanse(email)` / `emailIsPublic` / `domainParse` / `nameParse` — normalize match-key
+  values before comparing (pairs with `matchRules`).
+- `xmlParse({ string })` — XML APIs without adding a dependency.
+- `kvStore()` — scratch key-value collections for cross-page state (05) or import staging (06).
+- `fileProvider({ string | buffer | file: { path, deleteOnClose } })` — wrap content as a
+  `FileProvider` (13); `tempFile`/`tempDir` for staging.
+- `fieldsFromJSON(sample)` — draft `ObjectField[]` from a sample payload (04).
+- `upsertFlags.get(row)` — the caller's add/update hint (06).
+
+Also available (see the type definitions): `sleep`, `htmlToText`, `csvReader`/`csvWriter`,
+`xlsxReader`/`xlsxWriter`, `gzip`/`zip`, `json.*` (lossless numbers, `valuesReader`/
+`valuesWriter` path-based field access, `changesChecker`), and `serveApi` — which hosts a
+temporary dynamic endpoint from a script:
+`SDK.utilities.serveApi({ path, apiKey, handler: async (req, res) => res.send(200, body) })`
+returns a URL under `/dynamic/{path}`; callers must pass `?apikey=<key>`; the endpoint is
+torn down when the script ends (`req` = `{ params, body, headers, method, path, url }`).

package/docs/11-composite-keys.md

@@ -0,0 +1,164 @@
+# Composite keys
+
+Some objects have no single-field identity. The classic case is a **child REST resource** —
+`/companies/{parentId}/notes/{id}` — where a note's `id` is only unique _within_ its company;
+addressing a row needs both parts. Other cases: association/junction rows (two record ids) and
+multi-part natural keys. This topic is cross-cutting: the same convention drives `meta()`,
+`query()`, `upsert()` and `delete()`.
+
+## The rule
+
+`Row.meta.key` is **one opaque string** — the platform has no concept of a multi-part key. A
+composite key is therefore _encoded into_ the key string, and the encoding must be:
+
+1. **Deterministic** — the platform compares keys by string equality (cache identity, ids
+   filters, dedup). The same row must produce a byte-identical key on every query, forever.
+   Sort the parts; never rely on API response property order.
+2. **Parseable** — `upsert()` and `delete()` receive the key string back and must decode it to
+   rebuild the API path.
+
+**The fleet-standard encoding is a JSON object with the parts keyed by field path, in sorted
+order**: `{"company.id":5678,"id":1234}`. JSON survives any characters in the values, fails
+loudly on garbage, and self-documents which part is which. (A delimiter join like
+`5678:1234` is acceptable only when the parts can never contain the delimiter — and it's
+harder to validate; prefer JSON.)
+
+> Changing the encoding of an existing object is a breaking change: every cached row's key
+> stops matching and the platform sees an entirely new set of rows. Pick the encoding once.
+
+## The `$compositeKey` pseudo-field
+
+The key parts live in separate row properties, so declare a **pseudo-field** that carries the
+encoded key and mark IT as the key — by convention named `$compositeKey` (the `$` prefix
+signals "not a real API field"):
+
+```js
+// meta() for a child object
+queryFields.push({ id: "$compositeKey", name: "Composite key", type: "string", isKey: true });
+upsertFields.push({ id: "$compositeKey", name: "Composite key", type: "string", isKey: true });
+deleteFields = [{ id: "$compositeKey", type: "string", name: "Composite key", isKey: true }];
+```
+
+(`$compositeKey` is a commonly used identifier; the primary requirement is that the identifier
+not conflict with real fielf identifiers on the object. When the caller maps values into the
+discrete key fields AND supplies `$compositeKey`, cross-check them against the parsed key parts
+and throw on mismatch; when the discrete fields are absent, populate them from the parsed key.
+
+Because `{parentId}`-path objects have no "fetch everything" endpoint, they usually can't
+declare `canQueryList` without a parent filter — declare a `rowFilter` entry
+(`{ id: "parentId", type: "number" }`) and/or a relationship to the parent so rows arrive via
+`relatedFilter` (see [04](./04-meta-objects-fields.md), [05](./05-query.md)).
+
+## The helper pair
+
+Keep encode/decode in one place — every operation uses both:
+
+```js
+/** Deterministic composite key: parts sorted by field path. */
+function makeRowKey(keyFieldPaths, data) {
+  const keyObj = {};
+  for (const fieldPath of [...keyFieldPaths].sort((a, b) =>
+    a.join(".").localeCompare(b.join(".")),
+  )) {
+    const value = valueAtPath(data, fieldPath);
+    if (typeof value !== "number") {
+      // ids are numbers in this API - validate hard
+      throw new SDK.ConnectorError(
+        SDK.ErrorCode.InvalidApiKey,
+        `cannot create composite key: '${fieldPath.join(".")}' is not a number, got '${value}'`,
+      );
+    }
+    keyObj[fieldPath.join(".")] = value;
+  }
+  return JSON.stringify(keyObj);
+}
+
+function parseCompositeKey(objectId, compositeKey) {
+  let id, parentId;
+  try {
+    const parsed = JSON.parse(compositeKey);
+    id = parsed?.id;
+    parentId = parsed?.["company.id"];
+  } catch {
+    throw new SDK.ConnectorError(
+      SDK.ErrorCode.InvalidApiKey,
+      `failed to parse composite key '${compositeKey}' for object '${objectId}'`,
+    );
+  }
+  if (id == null || parentId == null) {
+    throw new SDK.ConnectorError(
+      SDK.ErrorCode.InvalidApiKey,
+      `composite key '${compositeKey}' for '${objectId}' did not contain a valid id / parentId`,
+    );
+  }
+  return { id, parentId };
+}
+```
+
+## Per operation
+
+### `query()`
+
+- `meta.key = makeRowKey(...)` for every row, and also expose it in the data:
+  `row.data["$compositeKey"] = key` — so the key is visible and usable as a field.
+- **The row data must contain everything the key encodes.** If the API response omits the
+  parent id (common: you fetched `/companies/5678/notes`, so the notes don't repeat it),
+  inject it into each row (`apiRow.parentId = parentIdFilter`) before building the key.
+- List queries on a child object run per parent: require `rowFilter.parentId` (validate its
+  type, throw `InvalidFilterParameter` otherwise) or arrive via `relatedFilter` from the
+  parent object.
+- `idsFilter` delivers composite key **strings** — parse each one and fetch
+  `/companies/{parentId}/notes/{id}` (in parallel, under your rate limiter).
+
+### `upsert()`
+
+- An **update** row carries `$compositeKey`: parse it, build the API path from the decoded
+  parts, and **remove `$compositeKey` from the body you send** — it is not a real API field.
+- An **add** row has no key yet: require the parent id as a regular upsert field and fail
+  clearly when it's missing:
+
+```js
+const compositeKey = row["$compositeKey"];
+let id, parentId;
+if (compositeKey) {
+  ({ id, parentId } = parseCompositeKey(options.id, compositeKey));
+  delete row["$compositeKey"];
+} else {
+  id = row.id;
+  parentId = valueAtPath(row, parentIdFieldPath);
+}
+if (parentId == null) {
+  throw new SDK.ConnectorError(
+    SDK.ErrorCode.InvalidApiKey,
+    `'${options.id}' is a child resource - adding/updating requires a '$compositeKey' or a parent id`,
+  );
+}
+```
+
+- The parent id belongs in the **URL**, not the request body.
+- Returned rows: `meta.key` must be the composite key — `makeRowKey` over the API's echo (add)
+  or the exact string you received (update).
+
+### `delete()`
+
+Parse the key, DELETE against the decoded path, and echo the **exact key string you
+received** back in `meta: { key, deleted: true }` — the platform matches the cached row by
+string equality, so re-encoding risks a mismatch. Include the decoded parts in `data`.
+
+## Gotchas
+
+1. **Determinism is everything.** Sorted parts, stable stringification, no locale/format
+   drift. A key that encodes differently across runs makes every row look new.
+2. **Echo, don't re-encode.** When an operation received a key string, return that exact
+   string — parse for the API call, but never rebuild the key from parsed parts unless you
+   must.
+3. **Never send `$compositeKey` to the API.** Strip it from bodies after parsing.
+4. **The special case where the child's id IS the parent id** (a 1:1 sub-resource like
+   `/companies/{parentId}/billing`) needs no composite key — the row's key is the parent id;
+   just map it into the URL slot.
+5. **`upsertClean` skips the key field automatically** ([06](./06-upsert-delete.md)) — that
+   applies to `$compositeKey` because it carries `isKey`; the underlying id/parent fields, if
+   also present as upsert fields, are compared like any other field.
+6. **Match/related queries must produce the same keys.** When you build rows from a related
+   lookup (`relatedFilter`) or reconstruct a row to satisfy `idsFilter`, run the same
+   `makeRowKey` — a second code path with its own encoding is how caches get corrupted.