@syncmatters/connector-sdk 1.0.0 → 1.0.2

package/docs/connector-authoring.md

@@ -0,0 +1,159 @@
+# Building SyncMatters connectors — the authoring guide
+
+This is the canonical guide to writing connectors for the SyncMatters platform. It ships inside
+`@syncmatters/connector-sdk` and is version-locked to the type definitions next to it. It is
+written to be consumed by AI coding assistants as much as by humans: read this file first, then
+open only the topic files your task touches.
+
+A **connector** is a JavaScript class (an `.mjs` file) that teaches the platform how to talk to
+one external system: how to authenticate, what objects and fields exist, and how to query,
+create, update and delete rows. Your code **executes on the SyncMatters platform**, not
+locally — the local workspace is for editing, type-checking and pushing via the `sm` CLI.
+
+## Topic files (read what your task touches)
+
+| File                                                             | Read when you are...                                                                                            |
+| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| [01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md) | adding files, wiring modules, editing any `*.meta.json` sidecar, declaring settings/OAuth/webhooks/npm packages |
+| [02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)         | starting a connector; implementing `init`/`test`/`meta`; anything about `InitArgs`                              |
+| [03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)         | making HTTP calls, rate limiting, retries, API keys, basic auth, OAuth2                                         |
+| [04-meta-objects-fields.md](./04-meta-objects-fields.md)         | writing `meta()` — objects, fields, flags, keys, relationships, filters                                         |
+| [05-query.md](./05-query.md)                                     | implementing `query()` — paging, checkpoints, ids/match/related filters, `Row`                                  |
+| [06-upsert-delete.md](./06-upsert-delete.md)                     | implementing `upsert()`, `delete()`, `upsertClean()`, `upsertFieldOptions()`, batching                          |
+| [07-events-webhooks.md](./07-events-webhooks.md)                 | webhooks: `parseEvents()`, `handleParsedEvent()`, signature verification                                        |
+| [08-errors.md](./08-errors.md)                                   | error handling — `ConnectorError`, `ErrorCode` selection, `HttpError`                                           |
+| [09-testing.md](./09-testing.md)                                 | the test harness: `mod Test.mjs` + `mod ObjectTestFeatures.mjs`                                                 |
+| [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)           | style rules, deprecated APIs to avoid, known traps — skim this once regardless                                  |
+| [11-composite-keys.md](./11-composite-keys.md)                   | objects with multi-part identity (child REST resources, junction rows) — key encoding across query/upsert/delete |
+| [12-example-connector.md](./12-example-connector.md)             | **the canonical worked connector** — a complete three-object connector (relationships, matching, composite keys, custom fields, upsertClean, query/upsert/delete) on one page; its test-file counterpart is 15 |
+| [13-file-fields.md](./13-file-fields.md)                         | objects carrying file content — fields of `type: "file"`, FileProvider, fields-driven inclusion (do NOT copy the fleet's legacy `row.file`) |
+| [14-vendor-sdks-and-non-http.md](./14-vendor-sdks-and-non-http.md) | vendor npm SDKs (googleapis, AWS, box), database/on-prem `agent_mode: "always"` connectors, SOAP/XML, self-managed rate limiting |
+| [15-example-test-file.md](./15-example-test-file.md)             | **the canonical worked test file** — the complete `mod Test.mjs` + `mod ObjectTestFeatures.mjs` pair for the Acme connector of 12 (TestFeatureProvider, prepare/cleanup hooks, issue simulators, feature scoping) |
+
+## The 60-second version
+
+A minimal read-only connector (the full-featured canonical example — relationships, matching,
+custom fields, writes — is [12-example-connector.md](./12-example-connector.md)):
+
+```js
+import SDK from "@syncmatters/connector-sdk";
+
+export default class Acme {
+  /** @type {SDK.HttpClient | undefined} */ #http;
+  /** @type {SDK.Logger | undefined} */ #log;
+  /** @type {string} */ #apiKey = "";
+
+  /** @param {SDK.InitArgs} args @returns {Promise<void>} */
+  async init(args) {
+    SDK.verifyType(this); // compile-time check that this class matches SDK.Connector
+    this.#log = args.log;
+    this.#apiKey = args.settings.get("api_key");
+    this.#http = SDK.utilities.httpClient({
+      limiter: SDK.utilities.rateLimiter({ type: "rolling", count: 10, ms: 1000 }),
+      maxRetry: 3,
+    });
+  }
+
+  /** @returns {Promise<SDK.TestResult>} */
+  async test() {
+    try {
+      await this.#call({ method: "GET", url: "me" });
+      return { success: true };
+    } catch (err) {
+      return { success: false, message: err instanceof Error ? err.message : String(err) };
+    }
+  }
+
+  /** @returns {Promise<SDK.ConnectorMeta>} */
+  async meta() {
+    const whoAmI = await this.#call({ method: "GET", url: "me" });
+    return {
+      identity: whoAmi.email,
+      objects: [
+        {
+          id: "contacts",
+          name: "Contacts",
+          canQueryByCheckpoint: true,
+          canQueryList: true,
+          canQueryByIds: true,
+          queryFields: [
+            { id: "id", type: "string", isKey: true },
+            { id: "email", type: "string" },
+          ],
+          matchRules: ["email[ci]", "name[ci]"],
+        },
+      ],
+    };
+  }
+
+  /** @param {SDK.QueryOptions} options @returns {Promise<SDK.QueryPage>} */
+  async query(options) {
+    const queryState = options.queryState || { page: 1 };
+    const resp = await this.#call({ method: "GET", url: `contacts?page=${queryState.page}` });
+    const rows = resp.items.map((item) => ({ meta: { key: String(item.id) }, data: item }));
+    queryState.page += 1;
+    return { rows, queryState, finalPage: rows.length === 0 };
+  }
+
+  /** @param {SDK.HttpRequest} request */
+  async #call(request) {
+    if (!request.url.startsWith("https://")) {
+      request.url = "https://api.example.com/v1/" + request.url;
+    }
+    request.headers = { ...request.headers, Authorization: `Bearer ${this.#apiKey}` };
+    return await this.#http.execute(request);
+  }
+}
+```
+
+Implementing more methods unlocks more platform capability:
+
+| You implement                           | The platform can                                                    |
+| --------------------------------------- | ------------------------------------------------------------------- |
+| `test()` + `meta()` (**required**)      | validate connections, show objects/fields in the UI                 |
+| `query()`                               | read data — lists, differential syncs, lookups                      |
+| `upsert()` / `delete()`                 | write data                                                          |
+| `upsertClean()`                         | pre-validate + change-detect rows before writing (`canUpsertClean`) |
+| `upsertFieldOptions()`                  | auto-add picklist values (`canUpsertFieldOptions`)                  |
+| `parseEvents()` / `handleParsedEvent()` | receive webhooks and keep caches fresh                              |
+| `optimalBatchSize()`                    | pick batch sizes per operation                                      |
+
+## Golden rules (every one is explained in a topic file)
+
+1. Import the SDK exactly as `import SDK from "@syncmatters/connector-sdk";` in workspace `.mjs` files.
+2. **Prefer** calling HTTP through `SDK.utilities.httpClient({ limiter: SDK.utilities.rateLimiter(...) })`.
+   More out of the box than `fetch`, `axios`, or `node:https`.
+3. Rows are `{ meta: { key: string }, data: object }`. `key` must be the value of the field your
+   `meta()` flags with `isKey` (or `isUserKey`). Keys are strings.
+4. `meta().objects` is an **array** of `ObjectMeta`. Every `canQuery*`/`canUpsert*` flag you set
+   is a promise — the platform will call the corresponding operation.
+5. Platform-visible failures are `throw new SDK.ConnectorError(SDK.ErrorCode.<Code>, message)`.
+   `test()` is the exception: it catches and returns `{ success: false, message }`.
+6. `query()` is called repeatedly: return `queryState` to continue, `finalPage: true` to stop.
+   Checkpoint queries must emit a `checkpoint` computed with an **overlap window** (see 05).
+7. `upsert()`/`delete()` results must be **positionally aligned** with `options.rows`.
+8. Every file needs a `meta/**/*.meta.json` sidecar; multi-file connectors are wired by
+   `module_script_file_paths` (entry sidecar lists ALL files, module sidecars list direct deps).
+9. JSDoc-type everything — workspaces run with `checkJs`. Private state in `#fields`. First line
+   of `init()` is `SDK.verifyType(this);`.
+10. Don't use deprecated APIs: `ObjectMetaSummary`/`ObjectMetaDetail`, `cacheRefreshOptions`,
+    `QueryPage.cacheState`, `RateLimiter.take()`. (See 10.)
+
+## Building a new connector, end to end
+
+1. **Scaffold**: `sm connector init` (wizard: settings, agent mode, OAuth mode) — or copy an
+   existing connector's file pair as a pattern.
+2. **Wire auth + HTTP** in `init()` (03). Add connection settings to the sidecar (01).
+3. **Make `test()` real**: cheapest authenticated call the API offers (02).
+4. **Describe objects** in `meta()` (04). Start with one object and its `isKey` field; grow from
+   there. `SDK.utilities.fieldsFromJSON(sample)` can draft field lists from a sample response.
+5. **Implement `query()`** for `canQueryList`, then `idsFilter`, then `checkpointFilter` (05).
+6. **Implement writes** if needed: `upsert()`, `delete()` (06).
+7. **Validate + push**: `sm validate`, `sm status` / `sm diff`, `sm push -m "message"`.
+   A push 409 means the platform copy changed: `sm pull`, merge, push again — never force.
+8. **Test harness**: add `mod Test.mjs` (+ generated `mod ObjectTestFeatures.mjs`) so the
+   platform's connector test suite can exercise your objects (09; worked example: 15).
+
+## Support
+
+- <https://syncmatters.com> · <support@syncmatters.com>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.0",
+  "version": "1.0.2",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,9 +12,9 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "39bdb6461d3d6962e0d284c2f4b6c71f1d08a6cd6a4b422754b3c731414fe6da",
+  "typesContentHash": "1e322fcb4cda8d7a31d350d575470adeb2402009ea1929274efe27d7ccb7e041",
   "dependencies": {
     "@types/node": "*",
-    "@syncmatters/script-api": "^1.0.0"
+    "@syncmatters/script-api": "^1.0.1"
   }
 }