@syncmatters/connector-sdk 1.0.0 → 1.0.2

package/docs/13-file-fields.md

@@ -0,0 +1,123 @@
+# File fields
+
+Objects can carry file content (documents, attachments, exports) as **fields of
+`type: "file"`** — declared like any other field, requested like any other field, and written
+like any other field. The value of a file field is an `SDK.FileProvider`.
+
+## Declaring
+
+```js
+// meta() - a document object whose content is a file field
+{
+  id: "documents",
+  name: "Documents",
+  canQueryList: true,
+  canQueryByIds: true,
+  queryFields: [
+    { id: "id", name: "ID", type: "string", isKey: true },
+    { id: "name", name: "File name", type: "string" },
+    { id: "size", name: "Size (bytes)", type: "number" },
+    { id: "content", name: "Content", type: "file" },
+  ],
+  upsertFields: [
+    { id: "name", name: "File name", type: "string", constraints: { mandatory: "add" } },
+    { id: "content", name: "Content", type: "file" },
+    { id: "id", type: "string", isKey: true },
+  ],
+}
+```
+
+## Querying: include the file only when asked
+
+File content is expensive — never fetch it unconditionally. The caller signals what it wants
+through `options.queryOptions.fields` (the field paths it expects, see
+[05-query.md](./05-query.md)); include file content **only when a file-typed field is in that
+list**:
+
+```js
+/** does the caller want any of this object's file fields? */
+#wantsField(options, fieldId) {
+  const fields = options.queryOptions?.fields;
+  if (!fields) return false; // no field selection -> cheap fields only, no file content
+  return fields.some((path) => path[0]?.path === fieldId);
+}
+
+async query(options) {
+  const wantsContent = this.#wantsField(options, "content");
+  // ...fetch the row list as usual...
+  const rows = [];
+  for (const item of resp.items) {
+    /** @type {any} */
+    const data = { id: item.id, name: item.name, size: item.size };
+    if (wantsContent) {
+      // fileResponse: true -> the http client hands back an SDK.FileProvider
+      data.content = /** @type {SDK.FileProvider} */ (await this.#http.execute({
+        method: "GET",
+        url: `documents/${item.id}/content`,
+        fileResponse: true,
+      }));
+    }
+    rows.push({ meta: { key: String(item.id) }, data });
+  }
+  // ...
+}
+```
+
+`fileResponse: true` on an `HttpRequest` is the standard way to download into a provider
+without buffering the whole body in memory. To build a provider from something you already
+have, use the constructor variants:
+
+```js
+SDK.utilities.fileProvider({ buffer: someBuffer });
+SDK.utilities.fileProvider({ string: someText });
+SDK.utilities.fileProvider({ file: { path: tempPath, deleteOnClose: true } });
+```
+
+## Consuming a `FileProvider`
+
+| Method                                | Use                                                                     |
+| ------------------------------------- | ----------------------------------------------------------------------- |
+| `stream(encoding?)`                   | a readable stream — the normal way to upload or parse without buffering |
+| `blob(fileName?)` / `formDataValue()` | for APIs consumed via `fetch`-style multipart bodies                    |
+| `save(path?)`                         | persist to disk, returns the path (pair with `SDK.utilities.tempFile`)  |
+| `length()`                            | size in bytes (e.g. for a `size` field or Content-Length header)        |
+| `close()`                             | release the resource — **always, in a `finally`**                       |
+
+Parsing structured file content: `SDK.utilities.csvReader({ source: { fileProvider } })`
+(same idea for `xlsxReader`).
+
+## Upserting: the file arrives as a field value
+
+The upsert row carries a `FileProvider` at the file field's position. Stream it out and close
+it:
+
+```js
+async upsert(options) {
+  return await Promise.all(options.rows.map(async (/** @type {any} */ row) => {
+    const file = /** @type {SDK.FileProvider} */ (row.content);
+    try {
+      const result = await this.#call({
+        method: "POST",
+        url: `documents${row.id ? `/${row.id}` : ""}`,
+        headers: { "Content-Type": "application/octet-stream", "X-File-Name": row.name },
+        body: await file.stream(),
+      });
+      return { meta: { key: String(result.id) }, data: result };
+    } finally {
+      try { await file.close(); } catch { /* already closed */ }
+    }
+  }));
+}
+```
+
+## Gotchas
+
+1. **No `fields` list means no file content.** When the caller doesn't select fields, return
+   the cheap scalar fields only — a list query that downloads every document is a platform
+   incident waiting to happen.
+2. **`upsertClean` treats every `FileProvider` as a change** and `confirmChange` cannot veto
+   it ([06-upsert-delete.md](./06-upsert-delete.md)) — keep file fields out of cleaned
+   payloads or accept the always-write.
+3. **Close what you open.** Providers can be backed by temp files; `close()` in a `finally`
+   on both the query and upsert paths.
+

package/docs/14-vendor-sdks-and-non-http.md

@@ -0,0 +1,115 @@
+# Vendor SDKs and non-HTTP connectors
+
+Not every system speaks plain REST/JSON. When the right client is a vendor npm package
+(`googleapis`, `box-node-sdk`, AWS SDK), a database driver (`pg`, `tedious`), or a protocol
+library (`ssh2` for SFTP, `soap` for WSDL services), the SDK's `httpClient` leaves the picture
+— and with it the rate limiting, retries and monitoring it provided. This topic is how the
+fleet fills those gaps.
+
+## Declaring the dependency
+
+List the package in the entry sidecar's `npm_packages` (name → exact version,
+[01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md)); the platform installs it at
+runtime and the `sm` CLI adds it to the workspace `package.json` so IntelliSense resolves it.
+
+```json
+"npm_packages": { "box-node-sdk": "2.4.0" }
+```
+
+Vendor packages are often untyped or loosely typed under `checkJs`: isolate them behind one
+`#client` field and use `@ts-ignore` sparingly at the boundary, not throughout your code.
+
+## Wiring the client in `init()`
+
+Build the vendor client from settings, exactly where you would have built `httpClient`:
+
+```js
+import { google } from "googleapis";
+
+async init(args) {
+  SDK.verifyType(this);
+  let auth;
+  if (args.settings.get("service_account_json")) {
+    // vendor wants a key FILE: stage the setting's content through a FileProvider
+    const keyFile = await SDK.utilities.fileProvider({
+      string: args.settings.get("service_account_json"),
+    }).save();
+    auth = new google.auth.GoogleAuth({ keyFile, scopes: ["https://www.googleapis.com/auth/drive"] });
+  } else {
+    // platform-managed OAuth2: feed the token in AND keep it fresh (03)
+    auth = new google.auth.OAuth2();
+    auth.setCredentials({ access_token: args.oauth2Provider.token });
+    args.oauth2Provider.onRefresh((token) => auth.setCredentials({ access_token: token }));
+  }
+  this.#drive = google.drive({ version: "v3", auth });
+}
+```
+
+The two patterns to copy: **`oauth2Provider.token` + `onRefresh` feed the vendor auth
+object** (the platform refreshes; you re-inject), and **file-based credentials are staged via
+`fileProvider({string}).save()`** rather than written by hand.
+
+Rarely-used sub-clients load lazily with a dynamic import, so cold starts stay fast:
+
+```js
+const STS = await import("@aws-sdk/client-sts");
+const identity = await new STS.STSClient(config).send(new STS.GetCallerIdentityCommand());
+```
+
+## Rate limiting without `httpClient`
+
+`httpClient` normally owns throttling. With a vendor client, wrap **every** call in a
+rate limiter yourself:
+
+```js
+this.#limiter = SDK.utilities.rateLimiter({ type: "rolling", count: 10, ms: 1000 });
+
+/** all vendor calls flow through here - the vendor SDK does not respect our limits */
+#callApi(exec) {
+  return this.#limiter.execute({ exec });
+}
+// usage:
+const resp = await this.#callApi(() => this.#drive.files.list({ pageSize: 200 }));
+```
+
+Retries are also yours now: vendor SDKs have their own retry options (configure them to match
+[03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)'s policy) — don't stack a hand-rolled
+retry loop on top of one you didn't disable. And map vendor errors into `ConnectorError`s at
+the same boundary ([08-errors.md](./08-errors.md)); a raw vendor stack trace helps nobody.
+
+## The database-connector archetype (`agent_mode: "always"`)
+
+Databases and on-premise systems are a connector shape of their own:
+
+- **`agent_mode: "always"`** in the sidecar — the connector runs on the customer's on-premise
+  agent, because the database is not reachable from the cloud
+  ([01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md)).
+- **No HTTP at all** — the driver connection (`pg`, `tedious`) is built in `init()` purely
+  from discrete settings: host, port, database, credentials, TLS/timeout options.
+- **`meta()` comes from schema introspection** — query the information schema / catalog for
+  tables and columns, map column types to `ObjectField` types, primary keys to `isKey`
+  (composite PKs → [11-composite-keys.md](./11-composite-keys.md)), foreign keys to
+  `relationships`.
+- **`query()` pages with real cursors** — a streaming query held open across pages: keep the
+  live cursor in a `Map` keyed by an id stored in your `queryState`, feed rows out per page,
+  and close it on `finalPage` (and in `dispose()` as a best-effort backstop).
+- **Escape SQL values** exactly as [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)
+  demands for any string-interpolated query language — or better, use the driver's
+  parameterized queries.
+
+Note: on-premise (`agent_mode: "always"`) connectors do not support `upsertClean`
+([06-upsert-delete.md](./06-upsert-delete.md)).
+
+## XML and SOAP systems
+
+- SOAP/WSDL: the `soap` npm package, with the WSDL files bundled as workspace resource files
+  (listed in `module_script_file_paths`) and loaded from `args.sourcePath`.
+- Plain XML responses over HTTP: keep `httpClient` and parse with
+  `await SDK.utilities.xmlParse({ string: responseBody })` — no extra dependency needed.
+
+## C# connectors (out of scope)
+
+A small number of connectors are C# scripts (`.csx`, `agent_mode: "always"`) for
+Windows/COM-interop targets. They use a different SDK surface and carry hard limitations (no
+OAuth2 provider, no webhooks, no `upsertClean`, single file, no npm packages). Do not apply
+this guide's JavaScript patterns to them — they are built by the platform team.