@syncmatters/connector-sdk 1.0.0 → 1.0.1

package/README.md

@@ -1,11 +1,23 @@
 # @syncmatters/connector-sdk
 
-TypeScript type definitions for the **SyncMatters Connector SDK**.
+TypeScript type definitions **and the connector authoring guide** for the **SyncMatters
+Connector SDK**.
 
 This is a **types-only** package: it provides IntelliSense and type checking when developing
 SyncMatters connectors locally (for example in VS Code or Cursor). Connector code executes on the
 SyncMatters platform, which provides the runtime implementation of this SDK.
 
+## Authoring guide (humans and AI assistants)
+
+The complete guide to building SyncMatters connectors ships inside this package, version-locked
+to the types:
+
+- [`docs/connector-authoring.md`](./docs/connector-authoring.md) — start here (cheat sheet +
+  index); focused topic files sit alongside it.
+
+If you are an AI assistant working in a connector workspace: **read that file before writing
+connector code.** Workspaces created by the `sm` CLI also contain an `AGENTS.md` pointing here.
+
 ## Usage
 
 ```js
@@ -28,11 +40,6 @@ export default class MyConnector {
 Add `"checkJs": true` to your `jsconfig.json` to activate the JSDoc type annotations used in
 connector `.mjs` files.
 
-## Legacy package name
-
-`@ihq/connector-sdk` is the legacy name of this package (IntegrateHQ is now SyncMatters).
-Existing connectors importing the legacy name continue to work on-platform.
-
 ## Support
 
 - <https://syncmatters.com>

package/docs/01-workspace-and-meta-json.md

@@ -0,0 +1,132 @@
+# Workspace anatomy and the `*.meta.json` sidecar
+
+## Two parallel trees
+
+```
+files/Connectors/<ConnectorName>/    # source code (what runs on the platform)
+meta/Connectors/<ConnectorName>/     # one *.meta.json sidecar per source file
+```
+
+- **Entry file**: `<ConnectorName>.mjs`, default-exporting the connector class. Its sidecar has
+  `"type": "connector"` and carries the `connector_metadata` block.
+- **Module files**: additional `.mjs` files in the same folder, imported by the entry (or by
+  other modules). Sidecar `"type": "module"`.
+- **Resource files**: e.g. a bundled OpenAPI spec (`acme-api.yaml`). Sidecar type `"yaml"` /
+  `"json"`. Load big specs lazily (dynamic `import()`), not at module top level.
+- **Test harness files**: `mod Test.mjs` (sidecar type `"connectortest"`) and the generated
+  `mod ObjectTestFeatures.mjs` (type `"module"`). The `"mod "` prefix (with a space) is the
+  convention. See [09-testing.md](./09-testing.md).
+
+Naming: entry and class-per-file modules are PascalCase (`Acme.mjs`); shared-logic modules are
+kebab-case (`acme-v3-query.mjs`). Version-split connectors use a `-v<N>` suffix per version core.
+
+## Module wiring: `module_script_file_paths`
+
+The platform stages exactly the files listed in the sidecars — an import that isn't listed will
+not exist at runtime.
+
+- The **entry** file's sidecar lists **every file of the connector, transitively** (all modules
+  and resources).
+- Each **module**'s sidecar lists only **its own direct imports**.
+
+```json
+// meta/Connectors/Acme/Acme.mjs.meta.json (entry - lists everything)
+"module_script_file_paths": [
+  "Connectors/Acme/acme-core.mjs",
+  "Connectors/Acme/acme-query.mjs",
+  "Connectors/Acme/acme-api.yaml"
+]
+```
+
+Import style in code: static `import` for your main module; dynamic `await import("./acme-v3")`
+for version dispatch so only the selected version's code is loaded:
+
+```js
+const version = args.settings.get("connectorVersion") || "1";
+switch (version) {
+  case "3": {
+    const V3 = await import("./acme-v3");
+    this.#core = new V3.AcmeV3();
+    break;
+  }
+  default:
+    throw new SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, `Connector version '${version}' not recognized`);
+}
+await this.#core.init(args);
+```
+
+## Sidecar top-level fields
+
+| Field | Values / meaning |
+|---|---|
+| `name` | the file name, e.g. `"Acme.mjs"` |
+| `type` | `"connector"` (entry) · `"module"` · `"connectortest"` · `"json"` · `"yaml"` |
+| `runnable` | `false` for connector files (they are invoked by the platform, not run directly) |
+| `engine` | runtime engine version int; leave as scaffolded unless told otherwise |
+| `resource_size` | `""` or `"medium"` — runtime resource class hint |
+| `cpu_architecture` | normally `""` |
+| `module_script_file_paths` | see above |
+| `npm_packages` | name→version map of npm packages this file needs at runtime, e.g. `{"box-node-sdk": "2.4.0"}`. The `sm` CLI unions these into the workspace `package.json` so IntelliSense sees them too. Prefer the SDK's built-in http/utilities over adding packages. |
+| `connector_metadata` | entry sidecar only — everything below |
+
+## `connector_metadata` reference
+
+| Field | Meaning |
+|---|---|
+| `categories` | discovery taxonomy, e.g. `["sales"]`, `["file"]`, `["marketing","sales"]` |
+| `agent_mode` | `"never"` = runs in cloud · `"always"` = runs on the customer's on-premise agent · `"optional"` = per-connection (agent used when the connection selects one) |
+| `oauth2_mode` | `"connector"` = one shared OAuth app for every account · `"connection"` = each connection supplies its own client id/secret. Omit for non-OAuth connectors. |
+| `oauth2_provider` | the OAuth provider definition (below) |
+| `webhooks_provider` | `{"mode": "custom.perconnection"}` (each connection registers its own webhook) or `{"mode": "custom.perconnector", "url": ..., "webhook_uid": ...}` (one shared endpoint) |
+| `settings` | the connection-settings form (below) |
+| `icon_url` | connector logo URL |
+| `connector_name`, `type_id`, `application_name`, `application_icon_url` | platform/branding ids — usually left as scaffolded |
+
+### `settings`
+
+Keyed by setting id; the values arrive in `init()` via `args.settings.get("<id>")`.
+
+```json
+"settings": {
+  "api_key":  { "order": "00", "name": "API Key", "type": "string", "control": "singlelinetext", "secret": true },
+  "region":   { "order": "01", "name": "Region", "type": "string", "control": "select",
+                "options": [ { "id": "us", "name": "US", "order": "00" }, { "id": "eu", "name": "EU", "order": "01" } ],
+                "default": { "type": "string", "expression": "us" } },
+  "connectorVersion": { "order": "02", "name": "Connector Version", "defines_version": true,
+                "type": "string", "control": "select",
+                "options": [ { "id": "1", "name": "1 (Superseded)", "order": "00" }, { "id": "2", "name": "2 (Recommended)", "order": "01" } ],
+                "default": { "type": "string", "expression": "2" } },
+  "debug":    { "order": "03", "name": "Debug", "advanced": true, "type": "boolean", "control": "singlelinetext" }
+}
+```
+
+Per-setting fields: `order` (two-digit string, display order), `name`, `type` (`string` |
+`boolean`), `control` (`singlelinetext` | `select`), `secret: true` (masked + encrypted),
+`advanced: true` (collapsed by default), `hidden: true`, `readonly: true`, `placeholder`,
+`options` (for `select`), `default: { "type": "string", "expression": "<value>" }`, and
+`defines_version: true` on the setting whose value picks a version core.
+
+Conventions worth copying: a secret `api_key`/`clientId`+`clientSecret` pair for credentials; an
+advanced boolean `debug` setting that gates verbose request/response logging; a
+`connectorVersion` select with `defines_version` when you need breaking revisions to coexist.
+
+### `oauth2_provider`
+
+Defines the OAuth2 dance the platform performs on the connector's behalf (your code only ever
+sees the resulting token via `args.oauth2Provider` — see
+[03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)). Fields seen in practice:
+`provider_id`, `grant_type` (`"code"`), `default_scopes`, `client_id`, `has_client_secret`,
+`redirect_uri` (the platform's OAuth callback), `authorize_uri`, `authorize_params`,
+`token_uri`, `refresh_uri`, `introspect_uri`, `authorized_domains`. URIs may template settings,
+e.g. `https://{acme_domain}/oauth2/authorize`. An optional `scope_groups` array drives the
+consent UI: items carry `label` plus `read`/`write`/`full` scope strings and optional
+`read_mandatory: true`.
+
+## Do not edit
+
+- `.sm/` (CLI state), and the CLI-generated `package.json` / `jsconfig.json` / `AGENTS.md`
+  managed block at the workspace root. They are editor tooling, never pushed.
+- Generated `mod ObjectTestFeatures.mjs` files (regenerate instead).
+- System-connector stubs: if a connector's settings include a hidden
+  `systemFlag*` boolean, the real implementation is embedded in the platform runtime and the
+  `.mjs` stub is deliberately a no-op — don't implement it.

package/docs/02-lifecycle-and-shape.md

@@ -0,0 +1,116 @@
+# Connector lifecycle and class shape
+
+## The class
+
+The entry file default-exports a class. Only `test()` and `meta()` are required; every other
+method is optional and its presence unlocks a capability (see the table in
+[connector-authoring.md](./connector-authoring.md)).
+
+```js
+import SDK from "@syncmatters/connector-sdk";
+
+export default class Acme {
+  /** @type {SDK.Logger | undefined} */ #log;
+
+  /** @param {SDK.InitArgs} args @returns {Promise<void>} */
+  async init(args) {
+    SDK.verifyType(this);
+    this.#log = args.log;
+    // read settings, build http client(s), wire OAuth refresh
+  }
+
+  /** @returns {Promise<SDK.TestResult>} */
+  async test() {
+    /* ... */
+  }
+
+  /** @returns {Promise<SDK.ConnectorMeta>} */
+  async meta() {
+    /* ... */
+  }
+}
+```
+
+`SDK.verifyType(this)` is a no-op whose only job is to make `checkJs` verify the class against
+the `SDK.Connector` interface — always the first line of `init()`.
+
+## `init(args)` — called exactly once per connector instance
+
+`SDK.InitArgs` fields:
+
+| Field                         | Type                  | Use                                                                                            |
+| ----------------------------- | --------------------- | ---------------------------------------------------------------------------------------------- |
+| `id`                          | `string`              | unique id of this connection instance                                                          |
+| `name`                        | `string`              | display name of the connection                                                                 |
+| `identity?`                   | `string`              | system-specific identity if known (e.g. signed-in email)                                       |
+| `mode`                        | `SDK.ScriptType`      | why the instance was created — see below                                                       |
+| `log`                         | `SDK.Logger`          | `info(message, props?)` / `error(messageOrError, props?)` to the platform event store          |
+| `sourcePath`                  | `string`              | directory holding the connector's staged files                                                 |
+| `settings`                    | `Map<string, any>`    | the connection's configured settings (`settings.get("api_key")`)                               |
+| `oauth2Provider?`             | `OAuth2TokenProvider` | present when the connection has a managed OAuth2 token — see [03](./03-http-auth-ratelimit.md) |
+| `webhookUrl?` / `webhookUid?` | `string`              | this connection's webhook endpoint / its unique id — see [07](./07-events-webhooks.md)         |
+
+`init()` should: validate required settings (throw
+`SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, ...)` naming the missing setting), stash
+what you need in `#fields`, build the HTTP client(s), and register the OAuth refresh callback.
+Do **not** make network calls in `init()` unless unavoidable — `test()` is where connectivity
+is proven.
+
+### `mode` (`SDK.ScriptType`) — early-out for static modes
+
+Values include `"integration"`, `"scriptFile"`, `"parseEvents"`, `"verifyEventIdentity"`,
+`"agentConnection"`, `"syncGroup"`, `"connectorTest"`, `"connectorFeatureGenerate"`,
+`"connectionSemanticTypes"`, `"codeEmbedding"`.
+
+Whether event parsing gets a real connection depends on the sidecar's
+`webhooks_provider.mode` (see [07-events-webhooks.md](./07-events-webhooks.md)): with a
+**per-connection** webhook the connector is initialized normally (settings available); with a
+**per-connector** shared endpoint, `parseEvents` runs **statically** — `init()` is called with
+`mode: "parseEvents"` and no settings/token are guaranteed. Shared-endpoint connectors with
+heavy `init()` setup should return early for static modes:
+
+```js
+const isStatic = args.mode === "parseEvents";
+if (isStatic) return; // static parseEvents only needs the http client it builds itself
+```
+
+## `test()` — prove the credentials
+
+Make the cheapest authenticated call the API offers. Catch, don't throw:
+
+```js
+/** @returns {Promise<SDK.TestResult>} */
+async test() {
+  try {
+    await this.#call({ method: "GET", url: "me" });
+    return { success: true };
+  } catch (err) {
+    return { success: false, message: err instanceof Error ? err.message : String(err), error: /** @type any */ (err) };
+  }
+}
+```
+
+`TestResult` = `{ success: boolean; message?: string; error?: CodedError }`.
+
+## `meta()` — see [04-meta-objects-fields.md](./04-meta-objects-fields.md)
+
+Return `{ identity?, objects: ObjectMeta[], events? }`. `identity` should be a stable string for
+the connected account (e.g. `` `${org.name}|${user.email}` ``) — the platform uses it to detect a
+connection being re-pointed at a different account.
+
+## Optional lifecycle methods
+
+| Method                                                  | Contract                                                                                                                                                  |
+| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `syncSessionBegin(session)` / `syncSessionEnd(session)` | bracket a sync-group run; implement only if you need to bootstrap internal caching for the whole run                                                      |
+| `dispose(options)`                                      | destroy unmanaged resources at process exit — **best-effort only**, may not run on abnormal exit; never rely on it for correctness                        |
+| `base()`                                                | return the underlying third-party client instance for diagnostic scripts                                                                                  |
+| `optimalBatchSize(options)`                             | recommend `{ size, batchType? }` per operation (`"upsert"`, `"idsFilter"`, ...); `batchType` is `"atomic"`, `"concurrent"` or `"splitInsertUpdateAtomic"` |
+
+## Multi-version connectors
+
+When an API revision would break existing connections, add a `connectorVersion` select setting
+(`defines_version: true`, see [01](./01-workspace-and-meta-json.md)) and split the
+implementation into per-version core modules. The entry class reads the setting in `init()` and
+delegates every SDK method to the selected core (`this.#core.query(options)` etc.), loading the
+core with a dynamic `await import("./acme-v3")` so unselected versions never load.

package/docs/03-http-auth-ratelimit.md

@@ -0,0 +1,188 @@
+# HTTP, auth and rate limiting
+
+## The one true HTTP idiom
+
+Every request goes through an `SDK.utilities.httpClient` built in `init()` with a rate limiter.
+Prefer this over `fetch`, `axios`, or `node:https` — the SDK client is what the platform monitors,
+retries and rate-limits.
+
+```js
+this.#http = SDK.utilities.httpClient({
+  limiter: SDK.utilities.rateLimiter({ type: "rolling", count: 10, ms: 1000 }), // 10 req/sec
+  maxRetry: 3,
+  retryBackoffMs: 2000,
+  defaultHeaders: { "Content-Type": "application/json" },
+});
+```
+
+`rateLimiter` types: `{ type: "rolling", count, ms }` (N requests per window) and
+`{ type: "concurrent", count }` (N in flight). Set them from the API's documented limits.
+
+When a window limiter alone is not enough (some APIs also punish burst concurrency), compose
+two limiters:
+
+```js
+const rolling = SDK.utilities.rateLimiter({ type: "rolling", count: 100, ms: 10000 });
+const concurrent = SDK.utilities.rateLimiter({ type: "concurrent", count: 6 });
+const limiter = {
+  execute: async (o) => concurrent.execute({ cost: 1, exec: () => rolling.execute(o) }),
+};
+this.#http = SDK.utilities.httpClient({ limiter, maxRetry: 3, retryBackoffMs: 2000 });
+```
+
+Endpoints with their own (stricter) limits get their **own** client — e.g. a search endpoint
+limited to 4 req/s gets a second `httpClient` with a second limiter.
+
+### Requests
+
+```js
+const response = await this.#http.execute({
+  method: "GET", // GET | POST | PUT | DELETE | PATCH
+  url: "https://api.example.com/v1/contacts",
+  headers: { Authorization: `Bearer ${this.#token}` },
+  body: undefined, // object bodies are JSON-serialized
+  // limiterCost?, fullResponse?, fileResponse?
+});
+```
+
+Non-2xx responses **throw an `HttpError`**: `{ name: "HttpError", code: "HTTP404",
+statusCode, url, responseHeaders, responseBodyString, responseBodyJson, responseBodyBuffer }`.
+Branch on `err.code === "HTTP401"` etc. — see [08-errors.md](./08-errors.md).
+
+### The central call wrapper
+
+Every mature connector funnels requests through one private method that (a) prepends the base
+URL, (b) attaches auth, (c) debug-logs when the `debug` setting is on:
+
+```js
+/** @param {SDK.HttpRequest} request */
+async #call(request) {
+  if (!request.url.startsWith("https://")) request.url = this.#baseUrl + request.url;
+  request.headers = { ...request.headers, Authorization: this.#auth };
+  try {
+    const response = await this.#http.execute(request);
+    this.#debug && this.#log.info(`DEBUG: ${request.method} ${request.url}`, { request, response });
+    return response;
+  } catch (err) {
+    this.#debug && this.#log.info(`DEBUG: ${request.method} ${request.url} failed`, { request, err });
+    throw err;
+  }
+}
+```
+
+### Retry policy: `canRetry`
+
+`canRetry(err, method, url, httpCode)` returns `true`/`false` — or a **number of milliseconds**
+to wait before that retry. The proven policy shape:
+
+```js
+const canRetry = (err, method, url, httpCode) => {
+  if ([400, 403, 404].includes(httpCode)) return false; // client errors don't heal
+  if (method === "GET") return true; // reads are idempotent
+  if ([429, 502, 504].includes(httpCode)) return true; // throttle/gateway blips
+  const msg = String(err.message || "");
+  return msg.includes("socket hang up") || msg.includes("ECONNRESET") || msg.includes("timeout");
+};
+```
+
+Non-idempotent writes: only retry when you know the API tolerates it. One real-world subtlety:
+writes that reference a _freshly created picklist option_ can fail transiently while the option
+propagates — returning ~10000 (10s backoff) from `canRetry` for that specific error is the
+established fix.
+
+## Auth patterns
+
+### API key / basic auth (from settings)
+
+```js
+this.#apiKey = args.settings.get("api_key");
+if (!this.#apiKey)
+  throw new SDK.ConnectorError(SDK.ErrorCode.InvalidOptionValue, "Setting 'API Key' is required");
+// basic:
+this.#auth = `Basic ${Buffer.from(`${user}:${pass}`).toString("base64")}`;
+```
+
+Validate every required setting up front with a clear per-setting message.
+
+### OAuth2 (`args.oauth2Provider`)
+
+The platform performs the OAuth dance (per the sidecar's `oauth2_provider` definition) and
+hands your code a live token. Three obligations:
+
+```js
+// 1. take the current token
+this.#oauth2Provider = args.oauth2Provider;
+this.#token = args.oauth2Provider.token;
+// 2. stay current: the platform refreshes in the background
+args.oauth2Provider.onRefresh((token) => {
+  this.#token = token;
+});
+```
+
+```js
+// 3. refresh-on-401: some APIs invalidate sessions early
+} catch (err) {
+  if (err.code === "HTTP401") {
+    await this.#oauth2Provider.forceRefresh();
+    return this.#callInternal(request); // retry ONCE after refresh
+  }
+  throw err;
+}
+```
+
+Also available: `oauth2Provider.scope` (granted scopes — normalize with
+`.replace(/\+/g, " ").split(" ")` and gate optional objects on the scopes actually granted),
+`introspectToken()` (account metadata — useful for `meta().identity`), `expiry`, `authParams`.
+
+`oauth2_mode` in the sidecar decides who owns the OAuth app: `"connector"` (one shared app) or
+`"connection"` (each connection brings its own client id/secret) — see
+[01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md).
+
+### Self-managed auth (when the managed OAuth2 provider doesn't fit)
+
+Some APIs use schemes the platform provider can't do. The fleet's established shapes:
+
+- **Client-credentials token endpoint you manage yourself** — a small helper that caches the
+  token and refreshes ahead of expiry, called from your request wrapper:
+
+```js
+async #token() {
+  if (this.#expiresAt < Date.now() + 5 * 60 * 1000) { // refresh 5 min early (clock skew)
+    const resp = await this.#http.execute({
+      method: "POST", url: this.#tokenUrl,
+      headers: { "Content-Type": "application/x-www-form-urlencoded" },
+      body: new URLSearchParams({ grant_type: "client_credentials",
+        client_id: this.#clientId, client_secret: this.#clientSecret }).toString(),
+    });
+    this.#expiresAt = Date.now() + resp.expires_in * 1000;
+    this.#auth = `${resp.token_type} ${resp.access_token}`;
+  }
+  return this.#auth;
+}
+```
+
+- **Request-signing schemes** (OAuth 1.0a/TBA HMAC signing, NTLM, AWS SigV4) — encapsulate the
+  signing in its own module (or a vendor package, see
+  [14-vendor-sdks-and-non-http.md](./14-vendor-sdks-and-non-http.md)) exposing the same
+  `execute(request)` shape as the SDK http client, so the rest of the connector can't tell the
+  difference.
+
+## OpenAPI-spec-driven APIs
+
+If the target publishes an OpenAPI 3 spec, bundle it as a resource file and let the SDK build a
+typed client with auth + rate limiting wired in:
+
+```js
+const api = await SDK.OpenApi3.createAsync({
+  specPathYaml: `${args.sourcePath}/acme-api.yaml`,
+  apiKey: { key: this.#apiKey, securityName: "ApiKeyAuth" }, // or oauth2/basicAuth
+  rateLimiterOptions: { type: "rolling", count: 10, ms: 1000 },
+  serverOverride: this.#baseUrl,
+  httpClientOverrides: { maxRetry: 3, defaultHeaders: { Accept: "application/json" } },
+  errLog: args.log,
+});
+// api.apis.<Tag>.<operationId>(...) ; api.httpClient / api.rateLimiter also exposed
+```
+
+Bundled specs are also the proven way to _generate_ `meta()` from the API's schemas at runtime
+— load lazily with `await import()` and map schema properties to `ObjectField[]`.

package/docs/04-meta-objects-fields.md

@@ -0,0 +1,162 @@
+# `meta()` — objects, fields, flags and keys
+
+`meta()` is the contract between your connector and everything the platform does with it. Every
+flag is a promise: set `canQueryByCheckpoint: true` and the platform WILL send checkpoint
+queries. Set nothing you haven't implemented.
+
+```js
+/** @returns {Promise<SDK.ConnectorMeta>} */
+async meta() {
+  return { identity: await this.#accountIdentity(), objects: await this.#buildObjects() };
+}
+```
+
+`ConnectorMeta` = `{ identity?: string; objects: ObjectMeta[]; events?: { identity?, types: EventTypeMeta[] } }`.
+`objects` is an **array**. (`ObjectMetaSummary`/`ObjectMetaDetail` are deprecated — never use.)
+`events` only when you implement webhooks — see [07-events-webhooks.md](./07-events-webhooks.md).
+
+## `ObjectMeta` — the canonical shape
+
+```js
+{
+  id: "contacts",                 // stable object id (what query/upsert receive as options.id)
+  name: "Contacts",               // display name
+  canQueryList: true,             // supports a plain "give me everything" paged query
+  canQueryByIds: true,            // supports queryOptions.idsFilter
+  canQueryByCheckpoint: true,     // supports differential (changed-since) queries
+  canQueryRandoms: true,          // random sample support
+  matchRules: ["id", "field_value_equals[ci]"],   // match rules the object supports
+  canUpsertClean: true,           // you implement upsertClean()
+  canUpsertFieldOptions: true,    // you implement upsertFieldOptions()
+  queryFields: [ /* ObjectField[] returned by query */ ],
+  upsertFields: [ /* ObjectField[] accepted by upsert */ ],
+  deleteFields: [ /* ObjectField[] needed to identify rows to delete */ ],
+  queryFilter: {                  // system-specific extra filters
+    rowFilter: [ { id: "modifiedOnOrAfter", type: "string" } ],
+    propertyFilter: [ { id: "fields", type: "array", arrayType: "string" } ],
+  },
+  relationships: [                // enables related-row queries
+    { id: "companies-companyId", name: "Company", relObjectId: "companies",
+      cardinality: "OneToOne", data: { idField: "companyId" } },
+  ],
+  isCustom: true,                 // object is account-specific (a custom object)
+  data: { /* anything YOUR connector needs back later - options.meta.data */ },
+}
+```
+
+Strive to align the `queryFields` and `upsertFields` shape (same schema). The `deleteFields` should typically contain a single row key field.
+
+Other fields: `settingsMeta`/`settingsValues` (per-object settings), `order` (sort). Omit flags
+rather than setting them `false` — `undefined` and `false` both mean "unsupported", and omitted
+reads cleaner.
+
+Only make a capability conditional when the API forces it: e.g. objects whose REST path embeds
+`{parentId}` can't `canQueryList` (there's no "all parents" call) — give them `canQueryByIds`
+and a `relatedFilter` path instead.
+
+## `ObjectField`
+
+```js
+{ id: "email", name: "Email", type: "string" }
+```
+
+| Field                             | Meaning                                                                                                                                                                           |
+| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `id`                              | property name as it appears in `row.data` - should be unique across all peer fields                                                                                               |
+| `type`                            | `"string"` · `"number"` · `"boolean"` · `"object"` · `"array"` · `"file"` · `"any"`                                                                                               |
+| `isKey: true`                     | this field is the row's unique id — its value becomes `row.meta.key`. Assign this (or `isUserKey`) to exactly one `queryFields`, plus one `upsertFields`, plus one `deleteFields` |
+| `isUserKey: true`                 | user-supplied natural key that can both add and update — see below                                                                                                                |
+| `canMatch: true`                  | field may be referenced by match rules (set on id/name/email-like fields; apply recursively into nested object fields where meaningful)                                           |
+| `optionValues`                    | picklist options `[{ id, name?, group? }]`                                                                                                                                        |
+| `canUpdateOptions: true`          | new picklist options may be added via `upsertFieldOptions()`                                                                                                                      |
+| `objectFields`                    | child fields when `type: "object"`                                                                                                                                                |
+| `arrayType` / `arrayObjectFields` | element type / element fields when `type: "array"`                                                                                                                                |
+| `constraints`                     | validation: `mandatory` (`"always"` · `"add"` · `"update"`), `max/min/maxLen/minLen`, `regex`, `email`, `integer`, `decimals`, `trim`, `dt: { format: "iso8601", "unix", ... }`   |
+| `isCustom`                        | account-specific/custom field                                                                                                                                                     |
+| `data`                            | connector-private payload, echoed back in metadata                                                                                                                                |
+
+### `isKey` vs `isUserKey`
+
+- `isKey`: the system-assigned id (`"id"`, `"Id"`). Updates address rows by it. When an upsert
+  operation omits the field it should be treated as an add operation. Objects whose identity
+  needs MORE than one field (child REST resources like `/parent/{parentId}/notes`) encode the
+  parts into a single `$compositeKey` pseudo-field — see
+  [11-composite-keys.md](./11-composite-keys.md).
+- `isUserKey`: a natural key the USER supplies (e.g. a property name) that can either add or
+  update. It is declared as an **object** field whose children carry the value and the intended
+  operation:
+
+```js
+upsertFields.unshift({
+  id: "name",
+  type: "object",
+  isUserKey: true,
+  objectFields: [
+    { id: "value", name: "Field name", type: "string" },
+    {
+      id: "operation",
+      name: "Upsert operation",
+      type: "string",
+      optionValues: [{ id: "add" }, { id: "update" }],
+    },
+  ],
+});
+```
+
+### Match rules
+
+`matchRules` values: `"id"`, `"name[ci]"`, `"email[ci]"`, `"domain[ci]"`,
+`"first_and_last_name[ci]"`, `"field_value_equals[ci]"` (`[ci]` = case-insensitive). Declare
+only rules `query()` actually implements via `matchFilter` — see [05-query.md](./05-query.md).
+
+### Relationships
+
+Relationships are defined on the source object and declare that given a row from the source object, the relationship allows row(s) from another object (`relObjectId`) that relate to this row to be collected. The canonical example is collecting the Company record that a Contact relates to.
+
+`{ id, name?, relObjectId, cardinality: "OneToOne" | "OneToMany", fkPath?, fkPathOn?, data? }` —
+`id` unique per object; `data` is yours (a common pattern stores the FK field name so `query()`
+can build the related filter). Declaring relationships enables `relatedFilter` queries.
+
+## Building `meta()` maintainably
+
+Never hand-maintain a big field list twice. The proven generators, in order of preference:
+
+1. **From the API's describe/metadata endpoint** — fetch object descriptions (in parallel with
+   `Promise.all`) and map each API field to an `ObjectField`, switch-casing the API's type
+   system into the SDK's. Map reference/lookup fields into `relationships`.
+2. **From a bundled OpenAPI spec** — load the YAML/JSON lazily and derive objects + fields from
+   its schemas.
+3. **From a sample payload** — `SDK.utilities.fieldsFromJSON(sampleJson)` infers a full
+   `ObjectField[]` tree. Its rules: nested objects → `objectFields`; arrays → `arrayType`
+   sampled from the first non-null element (arrays of objects → `arrayObjectFields`); empty
+   `{}`, `null` and `undefined` → type `"any"`. Great for drafting; review the result.
+
+When field lists must be hand-maintained, keep them as **one `Fields<Object>.mjs` module per
+object**, imported by a `MetaData.mjs` that assembles `meta()` — and remember every module
+must appear in the entry sidecar's `module_script_file_paths` (01).
+
+Deduplicate anything the API can duplicate (e.g. `[...new Set(referenceTo)]`) and derive
+capability flags from evidence, not optimism — e.g. `canQueryByCheckpoint` only when a
+last-modified filter actually exists on that object.
+
+## How `constraints` are enforced (what `verifyFieldConstraints` does)
+
+`upsertClean`'s `verifyFieldConstraints` (06) doesn't just report — several constraints
+**cleanse the value in place**:
+
+- `maxLen`: longer strings are truncated to `maxLen - 3` + `"..."` (byte length).
+- `min`/`max`: value parsed via `Number(v)`; out-of-range values are removed.
+- `integer` / `decimals`: `Number(v)` then `Math.round` / `toFixed(decimals)`.
+- `email`: removed when malformed or the TLD is unknown.
+- `trim`: whitespace-trimmed.
+- `dt`: validated/normalized per `format` (`unix`, `unix_ms`, `iso8601`, `yyyy-MM-dd`, ...,
+  or `custom` + `pattern`), plus `future`/`past`/`weekday`/`from`/`to`/`rangeYears` checks.
+- `mandatory` (`"always" | "add" | "update"`) raises a `ValueRequired` issue — and
+  `mandatoryAccepts: ["null" | "blank_string"]` lets a mandatory field treat `null`/`""` as a
+  satisfying value (for APIs where "explicitly blank" is meaningful).
+
+## `identity`
+
+Return a stable account identity string, e.g. `` `${org.name}|${user.username}` ``. The platform
+compares it across refreshes to detect a connection re-pointed at a different account. The test
+harness pins it via `expectedIdentity` ([09-testing.md](./09-testing.md)).