@syncmatters/script-api 1.0.21 → 1.0.23

package/docs/01-workspace-layout.md

@@ -3,8 +3,9 @@
 A workspace pulled by `sm` mirrors platform script files locally and adds editor tooling.
 
 ```
-files/              # script source (.mjs, .json, …) — EDIT and push via sm push
-meta/               # one *.meta.json sidecar per file under files/ — EDIT and push
+files/Scripts/      # integration scripts + sync logic (.mjs, .json, …) — EDIT and push via sm push
+files/Connectors/   # connector source, one folder per connector — EDIT and push
+meta/               # one *.meta.json sidecar per file under files/ (same sub-path) — EDIT and push
 syncs/<sync_uid>/   # sync.config.json — GENERATED read-only snapshots (sm pull sync)
 groups/<group_uid>/ # group.config.json — GENERATED read-only snapshots
 types/              # typed-connection .d.ts — GENERATED (sm types)
@@ -18,7 +19,9 @@ AGENTS.md           # CLI-generated agent guide (managed block + your notes belo
 
 | Path | Agent may edit? | `sm push`? |
 | --- | --- | --- |
-| `files/**` | Yes (when tasked) | Yes (with sidecar) |
+| `files/Scripts/**` | Yes (when tasked) | Yes (with sidecar) |
+| `files/Connectors/**` | Yes (connector work only) | Yes (with sidecar) |
+| any other top-level folder under `files/` (`Modules/`, `Services/`, …) | **No** — the platform rejects it | **No** |
 | `meta/**` | Yes | Yes (meta fields) |
 | `syncs/**/sync.config.json` | **No** | **No** |
 | `groups/**/group.config.json` | **No** | **No** |

package/docs/04-module-wiring-and-sidecars.md

@@ -28,17 +28,21 @@ The **entry** script's sidecar lists every file the platform must stage for that
 
 ```json
 "module_script_file_paths": [
-  "Services/AcmeSync/acme-filter.mjs",
-  "Services/AcmeSync/acme-utils.mjs"
+  "Scripts/AcmeSync/acme-filter.mjs",
+  "Scripts/AcmeSync/acme-utils.mjs"
 ]
 ```
 
+Paths are FullNames — the path under `files/` without the `files/` prefix. Script modules always
+start with `Scripts/`; only connector code lives under `Connectors/`. The platform rejects any
+other top-level folder.
+
 `sm push` maps paths to platform file ids from workspace state. Every referenced path must exist
 and be tracked — unresolved references block the wiring change for that file.
 
 ## Linking sync logic to a sync
 
-1. Push the module file(s) under `files/` with correct sidecars.
+1. Push the module file(s) under `files/Scripts/` with correct sidecars.
 2. In the **web UI**, open the sync configuration and attach the script file as custom logic
    (sets `script_file_id` on the platform).
 3. Locally, run `sm pull sync` and read `syncs/<sync_uid>/sync.config.json`:
@@ -53,7 +57,8 @@ not a push payload.
 
 A new file under `files/` is created on the platform only when:
 
-1. It has a `meta/` sidecar, and
-2. You `sm push` it.
+1. It sits under `files/Scripts/` (or `files/Connectors/<Name>/` for connector code),
+2. It has a `meta/` sidecar, and
+3. You `sm push` it.
 
 No sidecar → no push.

package/docs/06-cli-workflow-and-tokens.md

@@ -5,7 +5,8 @@
 ```
 sm pull                    # scripts + meta pins (+ syncs/groups when scoped)
 sm pull sync               # optional: refresh sync/group snapshots only
-→ edit files/ and meta/ only
+sm sync fields --source <conn>:<obj> --dest <conn>:<obj>   # both connections must resolve — stop and ask if one is missing
+→ edit files/Scripts/ and meta/Scripts/ only (files/Connectors/ for connector work)
 → sm validate
 → sm status / sm diff
 → sm push -m "describe change" --yes
@@ -13,8 +14,11 @@ sm pull sync               # optional: refresh sync/group snapshots only
 
 - **Scripts only:** `sm push` uploads changed files under `files/` (and meta sidecars). It does
   **not** upload `syncs/` or `groups/`.
-- **Sync config changes:** `changes/*.sync-changes.json` → `sm sync draft` → human review →
-  `sm sync push` (needs `syncs:write`). Agents push only on human instruction.
+- **Sync config changes:** confirm every `connection_uid` with `sm sync fields` first — if it
+  reports an unknown connection or object, stop and ask the human to create the connection,
+  select objects and refresh metadata; do not author mappings from connector tests, OpenAPI
+  specs, another workspace or memory. Then `changes/*.sync-changes.json` → `sm sync draft` →
+  human review → `sm sync push` (needs `syncs:write`). Agents push only on human instruction.
 - **Reading snapshots:** use `syncs/` and `groups/` for context before editing the wired module
   in `files/`.
 

package/docs/07-style-and-pitfalls.md

@@ -16,6 +16,8 @@
 | Editing `sync.config.json` to fix mappings | Change sync in web UI; re-pull snapshots |
 | Pushing `syncs/` or `groups/` | Not supported — push only `files/` + `meta/` |
 | Plain object default export for sync logic | **Class** default export; platform uses `new` |
+| New file outside `files/Scripts/` (e.g. `files/Modules/…`) | Platform rejects it — script code lives under `files/Scripts/`, connectors under `files/Connectors/` |
+| Authoring mappings for a connection that does not exist yet | Stop; ask the human to create the connection, then map only what `sm sync fields` prints |
 | Missing meta sidecar on new file | Add `meta/.../*.meta.json` before push |
 | Unwired `import` | Add path to `module_script_file_paths` in sidecar |
 | Running sync logic locally | Types check locally; execution is platform-only |

package/lib/script-error.d.ts

@@ -15,6 +15,7 @@ export declare enum ErrorCode {
     InvalidRelationshipId = "SCRIPT_INVALID_RELATIONSHIP_ID",
     InvalidRelationshipType = "SCRIPT_INVALID_RELATIONSHIP_TYPE",
     ResourceAlreadyClosed = "SCRIPT_RESOURCE_ALREADY_CLOSED",
+    FileProviderShallowClone = "SCRIPT_FILE_PROVIDER_SHALLOW_CLONE",
     InvalidEmailRecipients = "SCRIPT_INVALID_EMAIL_RECIPIENTS",
     InvalidEmailSubject = "SCRIPT_INVALID_EMAIL_SUBJECT",
     InvalidEmailBody = "SCRIPT_INVALID_EMAIL_BODY",

package/lib/utilities/file-provider/file-provider-buffer.d.ts

@@ -1,5 +1,7 @@
 import { FileProvider } from "./file-provider.js";
+import { kIsClosed } from "./file-provider-internal.js";
 export declare class FileProviderBuffer implements FileProvider {
+    #private;
     constructor(source: Buffer);
     length(): Promise<number>;
     stream(encoding?: "utf8" | "utf16le" | "ucs2" | "latin1"): Promise<NodeJS.ReadableStream>;
@@ -7,5 +9,8 @@ export declare class FileProviderBuffer implements FileProvider {
     blob(fileName?: string): Promise<Blob>;
     save(path?: string): Promise<string>;
     close(): Promise<void>;
+    clone(): FileProviderBuffer;
+    equals(other: FileProvider): Promise<boolean>;
+    [kIsClosed](): boolean;
     toJSON(): string;
 }

package/lib/utilities/file-provider/file-provider-disk.d.ts

@@ -1,4 +1,28 @@
+import fs from "node:fs";
 import { FileProvider } from "./file-provider.js";
+import { kIsClosed } from "./file-provider-internal.js";
+/**
+ * One record per underlying file, shared by a deleteOnClose provider and every clone of it (including
+ * clones of clones). The file is unlinked when the last holder closes.
+ */
+interface SharedFile {
+    path: string;
+    holders: number;
+}
+export interface Privates {
+    file: SharedFile;
+    size?: number;
+    deleteOnClose: boolean;
+    stream: fs.ReadStream | null;
+    closed: boolean;
+}
+type Unlink = (path: string, callback: (err: NodeJS.ErrnoException | null) => void) => void;
+/**
+ * Releases a provider the runtime collected without close(): marks it closed, destroys any open stream and, for a
+ * deleteOnClose file, gives up this holder and unlinks the file when it was the last. Runs in a finalizer, so it
+ * never awaits and swallows every error; the unlink is fire-and-forget.
+ */
+export declare function releaseUnclosed(p: Privates, unlink?: Unlink): void;
 export declare class FileProviderDisk implements FileProvider {
     constructor(path: string, deleteOnClose?: boolean);
     length(): Promise<number>;
@@ -8,5 +32,9 @@ export declare class FileProviderDisk implements FileProvider {
     blob(fileName?: string): Promise<Blob>;
     save(path?: string): Promise<string>;
     close(): Promise<void>;
+    clone(): FileProviderDisk;
+    equals(other: FileProvider): Promise<boolean>;
+    [kIsClosed](): boolean;
     toJSON(): string;
 }
+export {};

package/lib/utilities/file-provider/file-provider-internal.d.ts

@@ -0,0 +1,10 @@
+import type { FileProvider } from "./file-provider.js";
+/**
+ * Internal, non-interface hook shared by the platform FileProvider classes so that equals() can
+ * tell whether the other side is closed without calling it (a closed deleteOnClose disk provider
+ * fails with ENOENT rather than ResourceAlreadyClosed). Not exported from the package.
+ */
+export declare const kIsClosed: unique symbol;
+export declare function providerIsClosed(provider: FileProvider): boolean;
+/** Compares two open providers byte for byte: length first, then both streams chunk by chunk. */
+export declare function providerBytesEqual(a: FileProvider, b: FileProvider): Promise<boolean>;

package/lib/utilities/file-provider/file-provider-shim.d.ts

@@ -0,0 +1,21 @@
+import { FileProvider } from "./file-provider.js";
+import { kIsClosed } from "./file-provider-internal.js";
+/**
+ * What utilities.clone() puts in place of a FileProvider by default. It holds no file and no state: it serialises
+ * like a provider, closes as a no-op, and every read throws a ScriptError (FileProviderShallowClone) naming the
+ * option that makes a real clone. It counts as a FileProvider (isFileProvider() is true) so that a shim which reaches
+ * a connector fails on its first read rather than being uploaded as the text "[object FileProvider]".
+ */
+export declare class FileProviderShim implements FileProvider {
+    length(): Promise<number>;
+    stream(_encoding?: "utf8" | "utf16le" | "ucs2" | "latin1"): Promise<NodeJS.ReadableStream>;
+    close(): Promise<void>;
+    save(_path?: string): Promise<string>;
+    formDataValue(_encoding?: "utf8" | "utf16le" | "ucs2" | "latin1"): Promise<unknown>;
+    blob(_fileName?: string): Promise<Blob>;
+    clone(): FileProvider;
+    equals(_other: FileProvider): Promise<boolean>;
+    /** a shim has no content, so a real provider's equals() treats it as closed and resolves false */
+    [kIsClosed](): boolean;
+    toJSON(): string;
+}

package/lib/utilities/file-provider/file-provider-string.d.ts

@@ -1,5 +1,7 @@
 import { FileProvider } from "./file-provider.js";
+import { kIsClosed } from "./file-provider-internal.js";
 export declare class FileProviderString implements FileProvider {
+    #private;
     constructor(source: string);
     length(): Promise<number>;
     stream(encoding?: "utf8" | "utf16le" | "ucs2" | "latin1"): Promise<NodeJS.ReadableStream>;
@@ -7,5 +9,8 @@ export declare class FileProviderString implements FileProvider {
     blob(fileName?: string): Promise<Blob>;
     save(path?: string): Promise<string>;
     close(): Promise<void>;
+    clone(): FileProviderString;
+    equals(other: FileProvider): Promise<boolean>;
+    [kIsClosed](): boolean;
     toJSON(): string;
 }

package/lib/utilities/file-provider/file-provider.d.ts

@@ -6,6 +6,24 @@ export interface FileProvider {
     formDataValue(encoding?: "utf8" | "utf16le" | "ucs2" | "latin1"): Promise<unknown>;
     blob(fileName?: string): Promise<Blob>;
     toJSON(): string;
+    /**
+     * Returns an independent provider over the same bytes. The clone must be closed like any other
+     * provider: closing the original does not close the clone, and closing the clone does not close
+     * the original. A clone of a string or buffer provider re-wraps the source (no disk is shared).
+     * A clone of a disk provider created with deleteOnClose shares the underlying file by reference
+     * count: the file is removed only when the original and every clone of it have been closed. A
+     * provider dropped without close is released when the runtime collects it, which may be much later.
+     * utilities.clone(value, { withFileProviders: true }) calls this for every provider in value; the
+     * default utilities.clone(value) does not, and puts a shallow-clone shim in its place.
+     * Throws ResourceAlreadyClosed when this provider has been closed.
+     */
+    clone(): FileProvider;
+    /**
+     * Resolves true when both providers hold the same bytes. Compares length() first, then streams both
+     * and compares chunk by chunk without buffering whole files. Resolves false, rather than throwing,
+     * when either side has been closed.
+     */
+    equals(other: FileProvider): Promise<boolean>;
 }
 export interface FileProviderOptions {
     file?: {
@@ -18,4 +36,10 @@ export interface FileProviderOptions {
 export declare function fileProvider(options: FileProviderOptions): FileProvider;
 export declare namespace fileProvider {
     var isFileProvider: (value: unknown) => value is FileProvider;
+    var closeAll: typeof closeFileProviders;
 }
+/**
+ * Closes every FileProvider found in value, walking plain objects and arrays. Close errors are swallowed: this is
+ * for releasing providers the caller owns (for example the clones inside a snapshot) once it is done with them.
+ */
+export declare function closeFileProviders(value: unknown): Promise<void>;

package/lib/utilities/utilities.d.ts

@@ -21,10 +21,24 @@ import * as TypeUtils from "./type-utils/type-utils.js";
 import * as JsonUtils from "./json-utils/index.js";
 import * as UpsertFlags from "./upsert-flags/upsert-flags.js";
 import { KvStore } from "@syncmatters/script-api";
+export interface CloneOptions {
+    /**
+     * Clone each FileProvider with FileProvider.clone(), so the copy can read the file. Each cloned provider must be
+     * closed like any other. Without this, each provider in the copy is a shallow-clone shim.
+     */
+    withFileProviders?: boolean;
+}
+/**
+ * Deep copy. By default each FileProvider in value is replaced with a shallow-clone shim: it serialises like a
+ * provider and needs no close, but any read throws (ErrorCode.FileProviderShallowClone). That suits a snapshot for
+ * logging or tracing. Pass { withFileProviders: true } to clone the providers too (see FileProvider.clone()): the copy
+ * can read the files and each cloned provider must be closed.
+ */
+declare function clone<T>(value: T, options?: CloneOptions): T;
 export declare const utilities: {
     arrayOfStr: typeof arrayOfStrings;
     arrayOfNum: typeof arrayOfNumbers;
-    clone: <T>(value: T) => T;
+    clone: typeof clone;
     csvReader: (options: CsvReaderOptions) => import("./csv-reader/types.js").CsvReader;
     csvWriter: (options: CsvWriterOptions) => CsvWriter;
     dateFormat: typeof dateFormat;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/script-api",
-  "version": "1.0.21",
+  "version": "1.0.23",
   "description": "TypeScript type definitions for the SyncMatters script API (types only - scripts execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -28,7 +28,7 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "3dcf1af8c79b37f9fe75992f52dbbdc70d0f22515d9f65b1123ceb7e4f31de20",
+  "typesContentHash": "a7752b94739b0e0e5ab5d1a50afa574d6ca86c42013470349f048ab296bc4103",
   "dependencies": {
     "@types/node": "*"
   }

package/sdk-test/sdk-test.d.ts

@@ -373,6 +373,13 @@ export interface UpsertOptions {
     meta: API.ObjectMeta;
     withIssueSimulators?: boolean;
     withUpsertFieldOptions?: API.JsonValuePath;
+    /**
+     * Path of a file field (`type: "file"`) on the object's upsert fields that the returned insert and update data
+     * should populate with a FileProvider. The harness sends it on the upsert and delete tests' prepare calls whenever
+     * the object has a file field, preferring one with `constraints.mandatory: "add"`; a hook that ignores it loses
+     * nothing. It is sent on every upsertPrepare call whenever the object has a file field, including the upsertClean
+     * test's (the one with `withIssueSimulators`).
+     */
     withFileField?: API.JsonValuePath;
 }
 /** Options passed when cleaning up after an upsert test. */
@@ -382,13 +389,25 @@ export interface UpsertCleanupOptions {
 }
 /** Data returned when requesting data that will be used for an upsert test. */
 export interface UpsertData {
-    /** data to pass in the upsert call (ideally for an add operation) */
+    /**
+     * data to pass in the upsert call (ideally for an add operation). Return a fresh FileProvider in each call: the
+     * harness passes the connector a clone (`API.utilities.clone(insert, { withFileProviders: true })`, so each provider
+     * is a readable clone) and closes the originals and its clones after verifying.
+     */
     insert?: unknown;
-    /** a simple update that may be made to the added row (the test harness will copy over the key from the add response) */
+    /**
+     * a simple update that may be made to the added row (the test harness will copy over the key from the add response).
+     * FileProviders here follow the same rule as `insert`: fresh per call, cloned before writing, closed after verifying.
+     */
     update?: unknown;
     /** if requested, include 'bad data' examples that will trigger upsertClean issues */
     issueSimulators?: Array<UpsertIssueSimulator>;
-    /** fields to check to verify the insert/update was successful */
+    /**
+     * fields to check to verify the insert/update was successful. The same paths are checked on the update row, so a
+     * listed field must be set in both `insert` and `update`. A file field compares by bytes through
+     * `FileProvider.equals()`: the queried value must be a FileProvider holding the same content. Listing a file field
+     * here (or in `fields`) makes the query-back download its content; the harness never adds it on its own.
+     */
     verifyFields?: Array<API.JsonValuePath>;
     /** optionally specify other fields to include when querying the data back for validation */
     fields?: Array<API.JsonValuePath>;