@syncmatters/script-api 1.0.7 → 1.0.9

package/README.md

@@ -3,10 +3,23 @@
 TypeScript type definitions for the **SyncMatters script API**.
 
 This is a **types-only** package: it provides IntelliSense and type checking when developing
-SyncMatters integration scripts locally (for example in VS Code or Cursor). Script code executes on
-the SyncMatters platform, which provides the runtime implementation of this API.
+SyncMatters integration scripts and sync logic modules locally (for example in VS Code or Cursor).
+Script code executes on the SyncMatters platform, which provides the runtime implementation of
+this API.
 
-## Usage
+## Authoring guide (humans and AI assistants)
+
+The complete guide to integration scripts and sync custom logic ships inside this package,
+version-locked to the types:
+
+- [`docs/script-sync-authoring.md`](./docs/script-sync-authoring.md) — start here (cheat sheet +
+  index); focused topic files sit alongside it.
+
+If you are an AI assistant working in a CLI-pulled workspace: **read that file before writing
+script or sync logic code.** Workspaces created by the `sm` CLI also contain an `AGENTS.md`
+pointing here and at the connector guide in `@syncmatters/connector-sdk`.
+
+## Usage — integration script
 
 ```js
 import API from "@syncmatters/script-api";
@@ -17,6 +30,19 @@ export async function main(ctx) {
 }
 ```
 
+## Usage — sync logic module
+
+```js
+import API from "@syncmatters/script-api";
+
+export default class MySyncLogic {
+  /** @param {API.SyncLogicFilterFastContext} ctx @returns {Promise<boolean>} */
+  async filterFast(ctx) {
+    return true;
+  }
+}
+```
+
 Connector test harnesses additionally use the `sdk-test` entry point:
 
 ```js

package/docs/01-workspace-layout.md

@@ -0,0 +1,41 @@
+# Workspace layout (CLI-pulled tree)
+
+A workspace pulled by `sm` mirrors platform script files locally and adds editor tooling.
+
+```
+files/              # script source (.mjs, .json, …) — EDIT and push via sm push
+meta/               # one *.meta.json sidecar per file under files/ — EDIT and push
+syncs/<sync_uid>/   # sync.config.json — GENERATED read-only snapshots (sm pull sync)
+groups/<group_uid>/ # group.config.json — GENERATED read-only snapshots
+types/              # typed-connection .d.ts — GENERATED (sm types)
+.sm/state.json      # pull pins for conflict detection — do not edit
+package.json        # CLI-generated IntelliSense overlay — do not edit
+jsconfig.json       # CLI-generated — do not edit
+AGENTS.md           # CLI-generated agent guide (managed block + your notes below)
+```
+
+## Push boundary
+
+| Path | Agent may edit? | `sm push`? |
+| --- | --- | --- |
+| `files/**` | Yes (when tasked) | Yes (with sidecar) |
+| `meta/**` | Yes | Yes (meta fields) |
+| `syncs/**/sync.config.json` | **No** | **No** |
+| `groups/**/group.config.json` | **No** | **No** |
+| `types/**`, `.sm/`, `package.json`, `jsconfig.json` | **No** | **No** |
+
+Sync and sync-group **configuration** (mappings, schedules, which module is attached) is changed
+in the **web UI** today. The CLI exports snapshots for **read-only context** so agents know what
+is deployed; a future `sm sync …` push workflow is not available yet.
+
+Generated sync/group files include a marker:
+
+```json
+{
+  "$sm": { "generated_by": "sm pull", "kind": "sync.config" },
+  ...
+}
+```
+
+Refresh snapshots with `sm pull sync` or `sm pull --force` (overwrites dirty generated files
+only). See [05-reading-sync-snapshots.md](./05-reading-sync-snapshots.md).

package/docs/02-script-api-imports.md

@@ -0,0 +1,57 @@
+# Script API imports and integration scripts
+
+## Import style
+
+```js
+import API from "@syncmatters/script-api";
+```
+
+The legacy `@ihq/script-api` npm alias resolves to the same package — match whichever the file
+you are editing already uses; do not mix scopes within one file.
+
+Connector test harnesses additionally use:
+
+```js
+import SDKTest from "@syncmatters/script-api/sdk-test";
+```
+
+(See `@syncmatters/connector-sdk` testing docs — not used for sync logic modules.)
+
+## Integration script entrypoint
+
+Integration scripts export **`main`** receiving **`API.Context`**:
+
+```js
+import API from "@syncmatters/script-api";
+
+/** @param {API.Context} ctx */
+export async function main(ctx) {
+  ctx.log.info("Hello");
+}
+```
+
+### Context highlights (`API.Context`)
+
+| Member | Purpose |
+| --- | --- |
+| `ctx.log` | Platform logger (`info`, `warn`, `error`, …) |
+| `ctx.parameters` | Script parameters configured on the platform |
+| `ctx.connections` | Named connections → async factories returning `API.Connection` |
+| `ctx.syncGroups` | Named sync groups → async factories |
+| `ctx.state` | Persistent key/value (`get` / `set`) across runs |
+| `ctx.feedback` | Progress reporting for long jobs |
+| `ctx.sendEmail` | Platform email helper |
+| `ctx.account.executions.list` | Query past executions |
+| `ctx.environment.engineUid` | Runtime engine identifier |
+
+Use JSDoc (`/** @param {API.Context} ctx */`) so `checkJs` in the workspace validates calls.
+
+## Sync logic modules use the same import
+
+Sync custom logic lives in separate `.mjs` module files and uses `import API from "@syncmatters/script-api"` for hook context types — see [03-sync-logic-modules.md](./03-sync-logic-modules.md).
+
+## Types-only package
+
+`@syncmatters/script-api` is **types-only** locally. At runtime the platform provides the real
+implementations. Do not expect `main()` or sync hooks to run when executing Node locally against
+the pulled tree.

package/docs/03-sync-logic-modules.md

@@ -0,0 +1,58 @@
+# Sync logic modules (`API.SyncLogic`)
+
+Sync **custom logic** is optional JavaScript attached to a sync configuration. The platform
+loads your module, instantiates the **default-exported class** (`new YourClass()`), binds
+methods, and invokes hooks during the sync pipeline.
+
+## Export shape
+
+```js
+import API from "@syncmatters/script-api";
+
+export default class MySyncLogic {
+  /** @param {API.SyncLogicFilterFastContext} ctx @returns {Promise<boolean>} */
+  async filterFast(ctx) {
+    return true; // true = include row, false = exclude
+  }
+}
+```
+
+Use a **class** with async methods (not a plain object). Only implement hooks you need; omitted
+hooks are skipped.
+
+## Hooks (`API.SyncLogic`)
+
+| Hook | When it runs | Return |
+| --- | --- | --- |
+| `beforeRun(ctx)` | Once before the sync processes rows | `Promise<void>` |
+| `filterFast(ctx)` | Early filter on source row (+ optional prior match) | `Promise<boolean>` — keep? |
+| `filterRelated(ctx)` | After related rows are loaded | `Promise<boolean>` |
+| `reviewMatches(ctx)` | When multiple destination matches exist | `Promise<Array<unknown>>` — ordered candidates |
+| `filterMatched(ctx)` | After match + lookups | `Promise<boolean>` |
+| `calculate(ctx)` | Before field mapping | `Promise<void>` — write `ctx.calculated` |
+
+Context types: `SyncLogicBeforeRunContext`, `SyncLogicFilterFastContext`,
+`SyncLogicFilterRelatedContext`, `SyncLogicSelectMatchContext`, `SyncLogicFilterMatchedContext`,
+`SyncLogicCalcContext` (all under `API.*`).
+
+### `beforeRun` state
+
+`SyncLogicBeforeRunContext.state` provides `get()` / `set()` for logic that must persist across
+rows in one group run (use sparingly; prefer stateless filters when possible).
+
+### `calculate`
+
+Populate calculated fields on `ctx.calculated` (map of field id → value) for the mapping phase
+to consume.
+
+## Relationship to connectors
+
+Connectors supply endpoint rows (`query`, `upsert`, …). Sync logic **filters and transforms**
+rows inside a configured sync — it does not replace connector code. Connector authoring:
+`@syncmatters/connector-sdk/docs/connector-authoring.md`.
+
+## Wiring
+
+The module file must be staged on the platform and referenced from sync configuration
+(`script_file_id`). Sidecars and paths: [04-module-wiring-and-sidecars.md](./04-module-wiring-and-sidecars.md).
+Which file is attached: see pulled `sync.config.json` — [05-reading-sync-snapshots.md](./05-reading-sync-snapshots.md).

package/docs/04-module-wiring-and-sidecars.md

@@ -0,0 +1,59 @@
+# Module wiring and sidecars
+
+Script files on the platform always have a parallel **`meta/<same-path>.meta.json`** sidecar.
+The platform stages exactly the files listed in sidecars — an import that is not wired will fail
+at runtime.
+
+This mirrors connector module wiring; see also
+`@syncmatters/connector-sdk/docs/01-workspace-and-meta-json.md` for the full sidecar field
+reference.
+
+## Module files
+
+| Sidecar `type` | Typical use |
+| --- | --- |
+| `"module"` | Shared logic, sync logic classes, helper modules |
+| `"javascript"` / others | Depends on how the file was created on the platform |
+
+Minimum sidecar for a new module pushed from the CLI:
+
+```json
+{ "type": "module", "name": "my-sync-logic.mjs" }
+```
+
+## `module_script_file_paths`
+
+The **entry** script's sidecar lists every file the platform must stage for that tree
+(transitively). Each module's sidecar lists only its **direct** imports.
+
+```json
+"module_script_file_paths": [
+  "Services/AcmeSync/acme-filter.mjs",
+  "Services/AcmeSync/acme-utils.mjs"
+]
+```
+
+`sm push` maps paths to platform file ids from workspace state. Every referenced path must exist
+and be tracked — unresolved references block the wiring change for that file.
+
+## Linking sync logic to a sync
+
+1. Push the module file(s) under `files/` with correct sidecars.
+2. In the **web UI**, open the sync configuration and attach the script file as custom logic
+   (sets `script_file_id` on the platform).
+3. Locally, run `sm pull sync` and read `syncs/<sync_uid>/sync.config.json`:
+   - `script_file_id` — platform id of the attached logic file
+   - `sync_uid` — stable uid used in the snapshot path
+   - field maps, filters, source/destination — full config context
+
+Do **not** edit `sync.config.json` to change wiring or mappings — it is a generated snapshot,
+not a push payload.
+
+## New local files
+
+A new file under `files/` is created on the platform only when:
+
+1. It has a `meta/` sidecar, and
+2. You `sm push` it.
+
+No sidecar → no push.

package/docs/05-reading-sync-snapshots.md

@@ -0,0 +1,45 @@
+# Reading sync and group snapshots
+
+After `sm pull sync` (or default `sm pull` when the token has `syncs:read`), the workspace
+contains read-only JSON snapshots:
+
+```
+syncs/<sync_uid>/sync.config.json    # full ScriptSync configuration + version
+groups/<sync_group_uid>/group.config.json   # ScriptSyncGroup incl. schedule
+```
+
+Each file starts with:
+
+```json
+"$sm": { "generated_by": "sm pull", "kind": "sync.config" }
+```
+
+(or `"kind": "group.config"` for groups).
+
+## What agents should use them for
+
+- **Context:** sync name, uid, source/destination endpoints, field mappings, filters, attached
+  `script_file_id`, group membership, schedule.
+- **Locate code:** map `script_file_id` to a path via `.sm/state.json` `files` entries (match
+  `fileId`) or search `files/` / platform tree — then edit the **module under `files/`**, not
+  the snapshot.
+- **Drift awareness:** `version` (sync) or `modified` (group) plus `.sm/state.json` pins tell
+  you if the platform changed since pull (`sm diff` / re-pull).
+
+## What not to do
+
+- Do **not** hand-edit snapshots to change mappings, schedules, or wiring.
+- Do **not** `sm push` snapshot paths — there is no push route for them today.
+- Do **not** treat snapshots as the source of truth for writes — the web UI (or future CLI sync
+  commands) owns configuration changes.
+
+## Refresh
+
+| Command | Effect |
+| --- | --- |
+| `sm pull sync` | Refresh sync/group snapshots only |
+| `sm pull` | Scripts + snapshots (unless `--no-syncs`) |
+| `sm pull sync --force` | Overwrite locally modified **generated** snapshots |
+
+Missing `syncs:read` on the developer token: default pull warns and skips; `sm pull sync` exits
+2 with mint-token guidance.

package/docs/06-cli-workflow-and-tokens.md

@@ -0,0 +1,56 @@
+# CLI workflow and developer tokens
+
+## Services / sync module workflow
+
+```
+sm pull                    # scripts + meta pins (+ syncs/groups when scoped)
+sm pull sync               # optional: refresh sync/group snapshots only
+→ edit files/ and meta/ only
+→ sm validate
+→ sm status / sm diff
+→ sm push -m "describe change" --yes
+```
+
+- **Scripts only:** `sm push` uploads changed files under `files/` (and meta sidecars). It does
+  **not** upload `syncs/` or `groups/`.
+- **Sync config changes:** web UI (future CLI sync push out of scope).
+- **Reading snapshots:** use `syncs/` and `groups/` for context before editing the wired module
+  in `files/`.
+
+## Connector workflow (when also in the workspace)
+
+Connector work adds `sm test`, `sm connector features`, etc. See
+`@syncmatters/connector-sdk/docs/connector-authoring.md` and workspace `AGENTS.md`.
+
+## Developer token scopes (agents / IDE)
+
+Prefer tokens **without execute scopes** for day-to-day edit/push work:
+
+| Scope | Typical agent use |
+| --- | --- |
+| `scripts:read` | pull, diff incoming |
+| `scripts:write` | push script files |
+| `syncs:read` | `sm pull sync`, sync snapshots |
+| `connections:read` | `sm types`, connection resolution |
+| `scripts:execute` | **Only when running** `sm test` / connector features |
+| `syncs:execute` | Reserved; not needed for authoring |
+
+Default token UI omits `syncs:execute`; keep execute scopes off unless the user explicitly wants
+platform test runs from the agent.
+
+## 403 handling
+
+If the CLI reports **403** with missing scope or "not available to developer token sessions":
+
+1. **Stop** — do not retry other commands hoping one will work.
+2. Tell the user to mint a new token with the required scope (User settings → Developer tokens)
+   and run `sm auth` again.
+
+Examples:
+
+- Missing `syncs:read` → cannot pull sync snapshots (`sm pull sync`).
+- Missing `scripts:execute` → cannot run `sm test` (expected if using a read/write-only token).
+
+## Non-interactive runs
+
+Never answer `[y/N]` prompts in the terminal. Use `--yes` on `sm push`, `sm rm`, etc.

package/docs/07-style-and-pitfalls.md

@@ -0,0 +1,37 @@
+# Style and pitfalls
+
+## Style (match the fleet)
+
+- **Double quotes** for strings in `.mjs` files.
+- **JSDoc** all parameters and return types; workspace `checkJs` is active.
+- Prefix unused callback parameters with `_`.
+- **Private state** in `#private` fields on classes (sync logic modules and connectors).
+- **Import alias:** use `@syncmatters/script-api` or `@ihq/script-api` consistently within a
+  file — never mix.
+
+## Pitfalls
+
+| Trap | Correct approach |
+| --- | --- |
+| Editing `sync.config.json` to fix mappings | Change sync in web UI; re-pull snapshots |
+| Pushing `syncs/` or `groups/` | Not supported — push only `files/` + `meta/` |
+| Plain object default export for sync logic | **Class** default export; platform uses `new` |
+| Missing meta sidecar on new file | Add `meta/.../*.meta.json` before push |
+| Unwired `import` | Add path to `module_script_file_paths` in sidecar |
+| Running sync logic locally | Types check locally; execution is platform-only |
+| Retrying after 403 | Stop; user must fix token scopes |
+| Using `scripts:execute` on every agent token | Reserve execute for explicit test runs |
+
+## Connector vs sync logic
+
+- **Connector** (`@syncmatters/connector-sdk`): talks to an external API — `meta()`, `query()`,
+  `upsert()`, …
+- **Sync logic** (`API.SyncLogic`): filters/calculates rows **inside** a configured sync.
+
+Both can exist in one workspace; edit only the files your task names.
+
+## Further reading
+
+- Connectors: `node_modules/@syncmatters/connector-sdk/docs/connector-authoring.md`
+- Sync pipeline vs connector capabilities:
+  `@syncmatters/connector-sdk/docs/16-the-end-game-syncs.md`

package/docs/script-sync-authoring.md

@@ -0,0 +1,71 @@
+# Integration scripts and sync logic — the authoring guide
+
+This is the canonical guide to writing **integration scripts** and **sync custom-logic modules** for
+the SyncMatters platform. It ships inside `@syncmatters/script-api` and is version-locked to the
+type definitions next to it. Read this file first, then open only the topic files your task
+touches.
+
+**Integration scripts** are scheduled or triggered `.mjs` files that orchestrate work using
+`API.Context` (connections, sync groups, logging, state). **Sync logic modules** are `.mjs`
+classes wired to a sync configuration; the platform instantiates your class and calls optional
+hooks (`filterFast`, `calculate`, …) as rows move through the sync pipeline.
+
+Your code **executes on the SyncMatters platform**, not locally — the workspace pulled by the
+`sm` CLI is for editing, type-checking and pushing script files only.
+
+For **connectors** (external system adapters), use the separate guide in
+`@syncmatters/connector-sdk/docs/connector-authoring.md`.
+
+## Topic files (read what your task touches)
+
+| File | Read when you are… |
+| --- | --- |
+| [01-workspace-layout.md](./01-workspace-layout.md) | understanding `files/`, `meta/`, `syncs/`, `groups/`, `types/`, `.sm/` and what is pushable |
+| [02-script-api-imports.md](./02-script-api-imports.md) | starting an integration script; `import API`, `main(ctx)`, `API.Context` |
+| [03-sync-logic-modules.md](./03-sync-logic-modules.md) | implementing sync custom logic — `API.SyncLogic` hooks and context shapes |
+| [04-module-wiring-and-sidecars.md](./04-module-wiring-and-sidecars.md) | wiring modules via `*.meta.json`, `module_script_file_paths`, linking logic to a sync |
+| [05-reading-sync-snapshots.md](./05-reading-sync-snapshots.md) | using pulled `sync.config.json` / `group.config.json` for context (read-only) |
+| [06-cli-workflow-and-tokens.md](./06-cli-workflow-and-tokens.md) | `sm pull`, `sm pull sync`, validate/status/diff/push; developer token scopes |
+| [07-style-and-pitfalls.md](./07-style-and-pitfalls.md) | JSDoc, quotes, import aliases, common traps |
+
+## The 60-second version — sync logic module
+
+A minimal sync logic class (only `filterFast`; add other hooks as needed):
+
+```js
+import API from "@syncmatters/script-api";
+
+export default class AcmeContactFilter {
+  /**
+   * @param {API.SyncLogicFilterFastContext} ctx
+   * @returns {Promise<boolean>} true = keep row, false = exclude
+   */
+  async filterFast(ctx) {
+    const row = /** @type {{ email?: string }} */ (ctx.src);
+    return typeof row.email === "string" && row.email.endsWith("@example.com");
+  }
+}
+```
+
+Sidecar: `"type": "module"`. Wire the module in the entry sidecar's `module_script_file_paths`.
+The platform links this file to a sync via **sync configuration** (`script_file_id` in the web
+UI) — see [04-module-wiring-and-sidecars.md](./04-module-wiring-and-sidecars.md). Use the pulled
+`syncs/<sync_uid>/sync.config.json` snapshot to see which script file is attached; do **not**
+edit that JSON locally.
+
+## The 60-second version — integration script
+
+```js
+import API from "@syncmatters/script-api";
+
+/** @param {API.Context} ctx */
+export async function main(ctx) {
+  ctx.log.info("Starting integration script");
+  const conn = await ctx.connections["My Connection"]();
+  await conn.test();
+}
+```
+
+Sidecar `"type"` matches the script kind configured on the platform (often `"javascript"` or
+`"integration"` depending on how the file was created). New files need a `meta/` sidecar before
+`sm push` creates them on the platform.

package/lib/connection.d.ts

@@ -192,6 +192,8 @@ export interface QueryOptions {
     randomFilter?: boolean;
     /** filter for rows by row id (key) */
     idsFilter?: Array<string>;
+    /** path of the alternate key field that the idsFilter values refer to, if applicable */
+    idsFilterField?: Array<API.JsonValuePathPart>;
     /** filter for rows that relate to another object on this connection */
     relatedFilter?: QueryRelatedFilter;
     /** filter for rows that meet a specific matching rule */
@@ -239,6 +241,8 @@ export interface QueryOneOptions {
     relationshipFields?: Array<string>;
     /** filter for rows by row id (key) */
     idsFilter?: Array<string>;
+    /** path of the alternate key field that the idsFilter values refer to, if applicable */
+    idsFilterField?: Array<API.JsonValuePathPart>;
     /** filter for rows that relate to another object on this connection */
     relatedFilter?: QueryRelatedFilter;
     /** filter for rows that meet a specific matching rule */
@@ -264,7 +268,8 @@ export interface Row {
     data?: any;
     /** if filters requested files be included in the response, and the if row includes a file, the file will be here */
     file?: FileProvider | null | undefined;
-    /** if filters requested rows filter by 'related', or 'matched' to source rows the relationship details will be here */
+    /** if filters requested rows filter by 'related', or 'matched' to source rows the relationship details will be here;
+     * for idsFilter queries with an idsFilterField, srcRowId holds the requested (alternate key) value this row matched */
     relationship?: {
         relationshipId?: string;
         srcObjectId?: string;
@@ -439,6 +444,8 @@ export interface ObjectField {
     isKey?: boolean;
     /** field holds an object containing the user defined row unique identifier required for add an update operations, and an indication of the operation type */
     isUserKey?: boolean;
+    /** field holds a unique value and the connector honors it as an idsFilterField key in queries */
+    isAlternateKey?: boolean;
     /** field, and any fields below, are custom (account specific) */
     isCustom?: boolean;
     /** can this field be referenced in match rules? */

package/lib/sync.d.ts

@@ -1,4 +1,4 @@
-import { Row, QueryRelatedFilter, QueryMatchFilter, QueryCheckpointFilter, UpsertCleanOptions, UpsertCleanResponse, ObjectFieldConstraints, JsonValuePath, RowMatchRuleType, Logger, Connection } from "../index.js";
+import { Row, QueryRelatedFilter, QueryMatchFilter, QueryCheckpointFilter, UpsertCleanOptions, UpsertCleanResponse, ObjectFieldConstraints, JsonValuePath, JsonValuePathPart, RowMatchRuleType, Logger, Connection } from "../index.js";
 export type GroupRunMode = "Standard" | "SelectedIds" | "Errors";
 export type SyncOptimalBatchSizeOperation = "noFilter" | "idsFilter" | "relatedFilter" | "matchFilter" | "checkpointFilter" | "randomFilter" | "rowFilter" | "upsert" | "delete";
 export type SyncOptimalBatchSizeBatchType = "atomic" | "concurrent" | "splitInsertUpdateAtomic";
@@ -163,6 +163,9 @@ export interface SyncEndpointField {
     /** indicates this field holds an object containing the user defined unique row identifier required
      * for add and update operations, with an indication of the operation type */
     isUserDefinedId?: boolean;
+    /** indicates this field holds a unique value the connector can resolve rows by as an alternate to
+     * the row id (e.g. in lookups) */
+    isAlternateId?: boolean;
     /** indicates this field is custom */
     isCustom?: boolean;
 }
@@ -198,6 +201,8 @@ export interface SyncQueryOptions {
     randomFilter?: boolean;
     /** filter for rows by row id (key) */
     idsFilter?: Array<string>;
+    /** path of the alternate key field that the idsFilter values refer to, if applicable */
+    idsFilterField?: Array<JsonValuePathPart>;
     /** filter for rows that relate to another objects rows on this endpoint */
     relatedFilter?: QueryRelatedFilter;
     /** filter for rows that meet a specific matching rule */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/script-api",
-  "version": "1.0.7",
+  "version": "1.0.9",
   "description": "TypeScript type definitions for the SyncMatters script API (types only - scripts execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -20,7 +20,7 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "cc4a27cd342155f2108b2e8458696b2074548f8d8c93eb2ec51df19daceda1eb",
+  "typesContentHash": "fbf6d9da4649e1b2922ad811c5ebb8c69b7d526bd28c77fcc3a42e0abc9eda59",
   "dependencies": {
     "@types/node": "*"
   }