@syncmatters/connector-sdk 1.0.0 → 1.0.1

package/docs/14-vendor-sdks-and-non-http.md

@@ -0,0 +1,115 @@
+# Vendor SDKs and non-HTTP connectors
+
+Not every system speaks plain REST/JSON. When the right client is a vendor npm package
+(`googleapis`, `box-node-sdk`, AWS SDK), a database driver (`pg`, `tedious`), or a protocol
+library (`ssh2` for SFTP, `soap` for WSDL services), the SDK's `httpClient` leaves the picture
+— and with it the rate limiting, retries and monitoring it provided. This topic is how the
+fleet fills those gaps.
+
+## Declaring the dependency
+
+List the package in the entry sidecar's `npm_packages` (name → exact version,
+[01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md)); the platform installs it at
+runtime and the `sm` CLI adds it to the workspace `package.json` so IntelliSense resolves it.
+
+```json
+"npm_packages": { "box-node-sdk": "2.4.0" }
+```
+
+Vendor packages are often untyped or loosely typed under `checkJs`: isolate them behind one
+`#client` field and use `@ts-ignore` sparingly at the boundary, not throughout your code.
+
+## Wiring the client in `init()`
+
+Build the vendor client from settings, exactly where you would have built `httpClient`:
+
+```js
+import { google } from "googleapis";
+
+async init(args) {
+  SDK.verifyType(this);
+  let auth;
+  if (args.settings.get("service_account_json")) {
+    // vendor wants a key FILE: stage the setting's content through a FileProvider
+    const keyFile = await SDK.utilities.fileProvider({
+      string: args.settings.get("service_account_json"),
+    }).save();
+    auth = new google.auth.GoogleAuth({ keyFile, scopes: ["https://www.googleapis.com/auth/drive"] });
+  } else {
+    // platform-managed OAuth2: feed the token in AND keep it fresh (03)
+    auth = new google.auth.OAuth2();
+    auth.setCredentials({ access_token: args.oauth2Provider.token });
+    args.oauth2Provider.onRefresh((token) => auth.setCredentials({ access_token: token }));
+  }
+  this.#drive = google.drive({ version: "v3", auth });
+}
+```
+
+The two patterns to copy: **`oauth2Provider.token` + `onRefresh` feed the vendor auth
+object** (the platform refreshes; you re-inject), and **file-based credentials are staged via
+`fileProvider({string}).save()`** rather than written by hand.
+
+Rarely-used sub-clients load lazily with a dynamic import, so cold starts stay fast:
+
+```js
+const STS = await import("@aws-sdk/client-sts");
+const identity = await new STS.STSClient(config).send(new STS.GetCallerIdentityCommand());
+```
+
+## Rate limiting without `httpClient`
+
+`httpClient` normally owns throttling. With a vendor client, wrap **every** call in a
+rate limiter yourself:
+
+```js
+this.#limiter = SDK.utilities.rateLimiter({ type: "rolling", count: 10, ms: 1000 });
+
+/** all vendor calls flow through here - the vendor SDK does not respect our limits */
+#callApi(exec) {
+  return this.#limiter.execute({ exec });
+}
+// usage:
+const resp = await this.#callApi(() => this.#drive.files.list({ pageSize: 200 }));
+```
+
+Retries are also yours now: vendor SDKs have their own retry options (configure them to match
+[03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)'s policy) — don't stack a hand-rolled
+retry loop on top of one you didn't disable. And map vendor errors into `ConnectorError`s at
+the same boundary ([08-errors.md](./08-errors.md)); a raw vendor stack trace helps nobody.
+
+## The database-connector archetype (`agent_mode: "always"`)
+
+Databases and on-premise systems are a connector shape of their own:
+
+- **`agent_mode: "always"`** in the sidecar — the connector runs on the customer's on-premise
+  agent, because the database is not reachable from the cloud
+  ([01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md)).
+- **No HTTP at all** — the driver connection (`pg`, `tedious`) is built in `init()` purely
+  from discrete settings: host, port, database, credentials, TLS/timeout options.
+- **`meta()` comes from schema introspection** — query the information schema / catalog for
+  tables and columns, map column types to `ObjectField` types, primary keys to `isKey`
+  (composite PKs → [11-composite-keys.md](./11-composite-keys.md)), foreign keys to
+  `relationships`.
+- **`query()` pages with real cursors** — a streaming query held open across pages: keep the
+  live cursor in a `Map` keyed by an id stored in your `queryState`, feed rows out per page,
+  and close it on `finalPage` (and in `dispose()` as a best-effort backstop).
+- **Escape SQL values** exactly as [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)
+  demands for any string-interpolated query language — or better, use the driver's
+  parameterized queries.
+
+Note: on-premise (`agent_mode: "always"`) connectors do not support `upsertClean`
+([06-upsert-delete.md](./06-upsert-delete.md)).
+
+## XML and SOAP systems
+
+- SOAP/WSDL: the `soap` npm package, with the WSDL files bundled as workspace resource files
+  (listed in `module_script_file_paths`) and loaded from `args.sourcePath`.
+- Plain XML responses over HTTP: keep `httpClient` and parse with
+  `await SDK.utilities.xmlParse({ string: responseBody })` — no extra dependency needed.
+
+## C# connectors (out of scope)
+
+A small number of connectors are C# scripts (`.csx`, `agent_mode: "always"`) for
+Windows/COM-interop targets. They use a different SDK surface and carry hard limitations (no
+OAuth2 provider, no webhooks, no `upsertClean`, single file, no npm packages). Do not apply
+this guide's JavaScript patterns to them — they are built by the platform team.

package/docs/connector-authoring.md

@@ -0,0 +1,158 @@
+# Building SyncMatters connectors — the authoring guide
+
+This is the canonical guide to writing connectors for the SyncMatters platform. It ships inside
+`@syncmatters/connector-sdk` and is version-locked to the type definitions next to it. It is
+written to be consumed by AI coding assistants as much as by humans: read this file first, then
+open only the topic files your task touches.
+
+A **connector** is a JavaScript class (an `.mjs` file) that teaches the platform how to talk to
+one external system: how to authenticate, what objects and fields exist, and how to query,
+create, update and delete rows. Your code **executes on the SyncMatters platform**, not
+locally — the local workspace is for editing, type-checking and pushing via the `sm` CLI.
+
+## Topic files (read what your task touches)
+
+| File                                                             | Read when you are...                                                                                            |
+| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| [01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md) | adding files, wiring modules, editing any `*.meta.json` sidecar, declaring settings/OAuth/webhooks/npm packages |
+| [02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)         | starting a connector; implementing `init`/`test`/`meta`; anything about `InitArgs`                              |
+| [03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)         | making HTTP calls, rate limiting, retries, API keys, basic auth, OAuth2                                         |
+| [04-meta-objects-fields.md](./04-meta-objects-fields.md)         | writing `meta()` — objects, fields, flags, keys, relationships, filters                                         |
+| [05-query.md](./05-query.md)                                     | implementing `query()` — paging, checkpoints, ids/match/related filters, `Row`                                  |
+| [06-upsert-delete.md](./06-upsert-delete.md)                     | implementing `upsert()`, `delete()`, `upsertClean()`, `upsertFieldOptions()`, batching                          |
+| [07-events-webhooks.md](./07-events-webhooks.md)                 | webhooks: `parseEvents()`, `handleParsedEvent()`, signature verification                                        |
+| [08-errors.md](./08-errors.md)                                   | error handling — `ConnectorError`, `ErrorCode` selection, `HttpError`                                           |
+| [09-testing.md](./09-testing.md)                                 | the test harness: `mod Test.mjs` + `mod ObjectTestFeatures.mjs`                                                 |
+| [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)           | style rules, deprecated APIs to avoid, known traps — skim this once regardless                                  |
+| [11-composite-keys.md](./11-composite-keys.md)                   | objects with multi-part identity (child REST resources, junction rows) — key encoding across query/upsert/delete |
+| [12-example-connector.md](./12-example-connector.md)             | **the canonical worked example** — a complete three-object connector (relationships, matching, composite keys, custom fields, upsertClean, query/upsert/delete) on one page |
+| [13-file-fields.md](./13-file-fields.md)                         | objects carrying file content — fields of `type: "file"`, FileProvider, fields-driven inclusion (do NOT copy the fleet's legacy `row.file`) |
+| [14-vendor-sdks-and-non-http.md](./14-vendor-sdks-and-non-http.md) | vendor npm SDKs (googleapis, AWS, box), database/on-prem `agent_mode: "always"` connectors, SOAP/XML, self-managed rate limiting |
+
+## The 60-second version
+
+A minimal read-only connector (the full-featured canonical example — relationships, matching,
+custom fields, writes — is [12-example-connector.md](./12-example-connector.md)):
+
+```js
+import SDK from "@syncmatters/connector-sdk";
+
+export default class Acme {
+  /** @type {SDK.HttpClient | undefined} */ #http;
+  /** @type {SDK.Logger | undefined} */ #log;
+  /** @type {string} */ #apiKey = "";
+
+  /** @param {SDK.InitArgs} args @returns {Promise<void>} */
+  async init(args) {
+    SDK.verifyType(this); // compile-time check that this class matches SDK.Connector
+    this.#log = args.log;
+    this.#apiKey = args.settings.get("api_key");
+    this.#http = SDK.utilities.httpClient({
+      limiter: SDK.utilities.rateLimiter({ type: "rolling", count: 10, ms: 1000 }),
+      maxRetry: 3,
+    });
+  }
+
+  /** @returns {Promise<SDK.TestResult>} */
+  async test() {
+    try {
+      await this.#call({ method: "GET", url: "me" });
+      return { success: true };
+    } catch (err) {
+      return { success: false, message: err instanceof Error ? err.message : String(err) };
+    }
+  }
+
+  /** @returns {Promise<SDK.ConnectorMeta>} */
+  async meta() {
+    const whoAmI = await this.#call({ method: "GET", url: "me" });
+    return {
+      identity: whoAmi.email,
+      objects: [
+        {
+          id: "contacts",
+          name: "Contacts",
+          canQueryByCheckpoint: true,
+          canQueryList: true,
+          canQueryByIds: true,
+          queryFields: [
+            { id: "id", type: "string", isKey: true },
+            { id: "email", type: "string" },
+          ],
+          matchRules: ["email[ci]", "name[ci]"],
+        },
+      ],
+    };
+  }
+
+  /** @param {SDK.QueryOptions} options @returns {Promise<SDK.QueryPage>} */
+  async query(options) {
+    const queryState = options.queryState || { page: 1 };
+    const resp = await this.#call({ method: "GET", url: `contacts?page=${queryState.page}` });
+    const rows = resp.items.map((item) => ({ meta: { key: String(item.id) }, data: item }));
+    queryState.page += 1;
+    return { rows, queryState, finalPage: rows.length === 0 };
+  }
+
+  /** @param {SDK.HttpRequest} request */
+  async #call(request) {
+    if (!request.url.startsWith("https://")) {
+      request.url = "https://api.example.com/v1/" + request.url;
+    }
+    request.headers = { ...request.headers, Authorization: `Bearer ${this.#apiKey}` };
+    return await this.#http.execute(request);
+  }
+}
+```
+
+Implementing more methods unlocks more platform capability:
+
+| You implement                           | The platform can                                                    |
+| --------------------------------------- | ------------------------------------------------------------------- |
+| `test()` + `meta()` (**required**)      | validate connections, show objects/fields in the UI                 |
+| `query()`                               | read data — lists, differential syncs, lookups                      |
+| `upsert()` / `delete()`                 | write data                                                          |
+| `upsertClean()`                         | pre-validate + change-detect rows before writing (`canUpsertClean`) |
+| `upsertFieldOptions()`                  | auto-add picklist values (`canUpsertFieldOptions`)                  |
+| `parseEvents()` / `handleParsedEvent()` | receive webhooks and keep caches fresh                              |
+| `optimalBatchSize()`                    | pick batch sizes per operation                                      |
+
+## Golden rules (every one is explained in a topic file)
+
+1. Import the SDK exactly as `import SDK from "@syncmatters/connector-sdk";` in workspace `.mjs` files.
+2. **Prefer** calling HTTP through `SDK.utilities.httpClient({ limiter: SDK.utilities.rateLimiter(...) })`.
+   More out of the box than `fetch`, `axios`, or `node:https`.
+3. Rows are `{ meta: { key: string }, data: object }`. `key` must be the value of the field your
+   `meta()` flags with `isKey` (or `isUserKey`). Keys are strings.
+4. `meta().objects` is an **array** of `ObjectMeta`. Every `canQuery*`/`canUpsert*` flag you set
+   is a promise — the platform will call the corresponding operation.
+5. Platform-visible failures are `throw new SDK.ConnectorError(SDK.ErrorCode.<Code>, message)`.
+   `test()` is the exception: it catches and returns `{ success: false, message }`.
+6. `query()` is called repeatedly: return `queryState` to continue, `finalPage: true` to stop.
+   Checkpoint queries must emit a `checkpoint` computed with an **overlap window** (see 05).
+7. `upsert()`/`delete()` results must be **positionally aligned** with `options.rows`.
+8. Every file needs a `meta/**/*.meta.json` sidecar; multi-file connectors are wired by
+   `module_script_file_paths` (entry sidecar lists ALL files, module sidecars list direct deps).
+9. JSDoc-type everything — workspaces run with `checkJs`. Private state in `#fields`. First line
+   of `init()` is `SDK.verifyType(this);`.
+10. Don't use deprecated APIs: `ObjectMetaSummary`/`ObjectMetaDetail`, `cacheRefreshOptions`,
+    `QueryPage.cacheState`, `RateLimiter.take()`. (See 10.)
+
+## Building a new connector, end to end
+
+1. **Scaffold**: `sm connector init` (wizard: settings, agent mode, OAuth mode) — or copy an
+   existing connector's file pair as a pattern.
+2. **Wire auth + HTTP** in `init()` (03). Add connection settings to the sidecar (01).
+3. **Make `test()` real**: cheapest authenticated call the API offers (02).
+4. **Describe objects** in `meta()` (04). Start with one object and its `isKey` field; grow from
+   there. `SDK.utilities.fieldsFromJSON(sample)` can draft field lists from a sample response.
+5. **Implement `query()`** for `canQueryList`, then `idsFilter`, then `checkpointFilter` (05).
+6. **Implement writes** if needed: `upsert()`, `delete()` (06).
+7. **Validate + push**: `sm validate`, `sm status` / `sm diff`, `sm push -m "message"`.
+   A push 409 means the platform copy changed: `sm pull`, merge, push again — never force.
+8. **Test harness**: add `mod Test.mjs` (+ generated `mod ObjectTestFeatures.mjs`) so the
+   platform's connector test suite can exercise your objects (09).
+
+## Support
+
+- <https://syncmatters.com> · <support@syncmatters.com>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.0",
+  "version": "1.0.1",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,9 +12,9 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "39bdb6461d3d6962e0d284c2f4b6c71f1d08a6cd6a4b422754b3c731414fe6da",
+  "typesContentHash": "285fcc040f1a62f7472910cf06536de4f33574a6f82c5ac405aa51a6b0797174",
   "dependencies": {
     "@types/node": "*",
-    "@syncmatters/script-api": "^1.0.0"
+    "@syncmatters/script-api": "^1.0.1"
   }
 }