@syncmatters/connector-sdk 1.0.20 → 1.0.22

package/docs/04-meta-objects-fields.md

@@ -188,6 +188,15 @@ picklist label or differently-cased id to the option id) before filtering.
 field pairs that must ALL be equal (`destA = a AND destB = b ...`). Declare it only when the API
 can apply several equality filters in one query (search filter groups, SOQL/OData `and`, ...).
 
+`"domain[ci]"` matches rows **at the host the sync sends**. The sync cleans the source value
+first (`acme.com`, never `https://www.acme.com/`), and a row holding
+`https://www.acme.com/about` must match it. Declare it only when the API can find such rows:
+equality on a bare-domain field, or contains / `like` (or equality with `IN` / `OR` over the
+common URL variants) on a website field, followed by a `domainCompare` check of each
+candidate. A website field with single-value equality only cannot honour the rule: do not
+declare it.
+See the `domain[ci]` bullet in [05-query.md](./05-query.md).
+
 **Commonly forgotten:** an object that supports `idsFilter` can ALWAYS offer
 `"field_value_equals[ci]"` — with `canMatch: true` on its id field, the match is just a
 lookup of the supplied value by id. There is no reason for such an object to declare no

package/docs/05-query.md

@@ -236,7 +236,12 @@ candidates }])` — the platform reserves un-mapped matches and returns the sele
 
 - **name[ci]** - inspect the `match.name` values and search for rows with the same `name`. The determination of which field holds name is made by the connector, e.g. for a company it may be `companyName`, for a deal it may be `title`.
 - **email[ci]** - inspect the `match.email` values and search for rows with the same `email`. The determination of which field(s) holds email is made by the connector, e.g. for a contact it may search `email` and `alternativeEmails`.
-- **domain[ci]** - inspect the `match.domain` values and search for rows with the same `domain`. The determination of which field(s) holds email is made by the connector, e.g. for a comapny it may search `website`.
+- **domain[ci]** - inspect the `match.domain` values and search for rows **at the host passed**. The determination of which field(s) holds the domain is made by the connector, e.g. for a company it may search `website`. Do not reinterpret the value: passed `software.example.com`, a row at `example.com` is not a match.
+  - What arrives: a sync cleans the source value before sending it. The user chooses how far on the rule: the company domain (the default: `shop.acme.co.uk` arrives as `acme.co.uk`), the full host (`shop.acme.co.uk`: only scheme, `www`, port and path are dropped), or the raw source value. The connector is not told which. In the first two, values that are not domains, or that are a page or subdomain on a shared host such as `facebook.com/acme`, are not sent; a shared host's own domain (`facebook.com`) is, because it belongs to that company. An internationalized domain arrives in the form the source wrote it (`münchen.de` or `xn--mnchen-3ya.de`).
+  - Field holds bare domains (an identity, like a CRM company domain), API offers equality: an equality filter; nothing else to do.
+  - Field holds website addresses, API offers contains or `like`: fetch candidates by contains, then keep the right ones with `domainCompare(storedValue, match.domain)`. It returns `"host"` when the stored value is at exactly the host sought, `"company"` when the domain sought has no subdomain and the stored value is on a subdomain of it (`https://portal.acme.com/login` for `acme.com`), and `undefined` otherwise, including for a stored page on a shared host. **If any candidate is `"host"`, pass only those to `canUse`; otherwise you may pass the `"company"` ones.** The check is required: contains over-matches (`acme.com` is contained in `notacme.com` and `acme.com.au`, and `facebook.com` in every `facebook.com/<page>`).
+  - Field holds website addresses, API offers equality with `IN` / `OR`: query the common variants (`acme.com`, `www.acme.com`, `http(s)://[www.]acme.com`, each with and without a trailing `/`), then keep the right ones as above. This misses deep paths; say so in the connector notes.
+  - Field holds website addresses, API offers single-value equality only: **do not declare `domain[ci]`**.
 - **first_and_last_name[ci]** - inspect the `match.firstname` and values and `match.lastname` and search for rows with the same name.
 - **id** - inspect the `match.id` value and search in the selected `destFieldPath` (path to the field on the target system) to find rows holding this exact value in this field. This search is case sensitive.
 - **field_value_equals[ci]** - inspect the `match.custom` value and search in the selected `destFieldPath` (path to the field on the target system) to find rows holding this value (case insensitive) in this field.

package/docs/09-testing.md

@@ -122,7 +122,13 @@ To disable those tests use `cannotTestReason` instead
 - `checkpointFilterStep1Prepare` / `Step2Prepare` — the two-step round-trip test of your
   differential contract; see [Checkpoint testing](#checkpoint-testing-what-the-two-steps-prove)
   below, it is the most-misimplemented pair.
-- `matchFilterPrepare` — returns `MatchData` including `expectedMatchRowIds`.
+- `matchFilterPrepare` — returns `MatchData` including `expectedMatchRowIds`. For
+  `domain[ci]` the harness does not go through a sync, so send `match.domain` already cleansed
+  (`API.utilities.domainCleanse`, e.g. `acme.com`), exactly as a sync would. When the object's
+  domain field is a website field, store a URL-form value on the test row (e.g.
+  `https://www.acme.com/about`) so the test proves the connector matches on the host rather
+  than on the raw text, and add a case that sends a host with a subdomain
+  (`software.example.com`) and expects a row at `example.com` not to match.
 - `relatedFilterPrepare({ fromObjectMeta, toObjectMeta, fromRelationshipId })` —
   `fromObjectMeta` is the object where the relationship is DECLARED (the parent, in the
   parent-owns-children shape), `toObjectMeta` is its `relObjectId`. Return
@@ -135,6 +141,17 @@ To disable those tests use `cannotTestReason` instead
   contract under "What each test needs" below. Use `withIssueSimulators` /
   `UpsertIssueSimulator` (`verifyFields`, expected issue `type` + `fatal`) to deliberately
   exercise your `upsertClean` issue reporting.
+- **File fields in `upsertPrepare`.** When the object has an upsert field of `type: "file"`,
+  every `upsertPrepare` call (upsert, delete and upsertClean tests) carries `withFileField`
+  set to its path (a field with `constraints: { mandatory: "add" }` is preferred, else the
+  first file field). Put an `SDK.FileProvider` at that path in `insert` (and `update`), and
+  return a **fresh provider on every call**: the harness passes the connector a clone
+  (`SDK.utilities.clone(row, { withFileProviders: true })`), so the connector may close what
+  it is handed, and the harness closes your originals and its clones once the test has
+  verified them ([13-file-fields.md](./13-file-fields.md)). A file field listed in `verifyFields` is
+  compared by bytes with `FileProvider.equals()`: the query-back must return a provider with
+  the same content. The harness never adds the file field to the query-back on its own,
+  since that downloads the content; list it in `verifyFields` or `fields` to opt in.
 - delete features support soft-delete verification — `isDeleted: { path, valueWhenDeleted }`
   checks a flag field instead of row absence, and `afterDeleteMaxIndexWaitTimeMs` tolerates
   APIs whose deletions surface asynchronously.

package/docs/10-style-and-pitfalls.md

@@ -119,7 +119,9 @@
 The two you will use constantly (they appear in nearly every fleet connector):
 
 - `SDK.utilities.clone(value)` — deep clone. Use before mutating anything you received or
-  return (metadata, rows, options) — shared references are how callers get corrupted.
+  return (metadata, rows, options) — shared references are how callers get corrupted. A
+  `FileProvider` in the value becomes an unreadable shim unless you pass
+  `{ withFileProviders: true }` ([13-file-fields.md](./13-file-fields.md)).
 - `SDK.utilities.tryGet(() => deeply.nested.maybe.missing)` — returns the value or
   `undefined`, never throws. The idiom for prodding uncertain API payloads:
   `const sig = tryGet(() => options.payload.headers["x-signature"][0]);`
@@ -132,6 +134,14 @@ Frequently used:
   settings values ("a, b,c" → `["a","b","c"]` / a `Set`).
 - `emailCleanse(email)` / `emailIsPublic` / `domainParse` / `nameParse` — normalize match-key
   values before comparing (pairs with `matchRules`).
+- `domainCompare(storedValue, matchDomain)` — the check for `domain[ci]` candidates: `"host"`,
+  `"company"` or `undefined` ([05-query.md](./05-query.md)). It ignores case and the unicode or
+  punycode form of an internationalized name, and rejects a stored page on a shared host.
+- `domainCleanse(value, level?)` / `domainIsShared(value)` — reduce a hostname or URL to its
+  lowercased company domain (`https://www.shop.acme.co.uk/x` → `acme.co.uk`) or, with
+  `"full_host"`, to its host without `www` (`shop.acme.co.uk`); `undefined` when not a domain.
+  `domainIsShared` spots a page or subdomain on a shared host (`facebook.com/acme`,
+  `sites.google.com/view/acme`; not the host's own `facebook.com`). Both take the original value.
 - `xmlParse({ source: { string | fileProvider, encoding? } })` — preferred full-document XML
   parse. Deprecated: `xmlParse({ string })` (still supported).
 - `xmlParse({ source, emit, skip? })` — stream matching elements as xml2js-shaped chunks (XPath

package/docs/13-file-fields.md

@@ -82,10 +82,34 @@ SDK.utilities.fileProvider({ file: { path: tempPath, deleteOnClose: true } });
 | `save(path?)`                         | persist to disk, returns the path (pair with `SDK.utilities.tempFile`)  |
 | `length()`                            | size in bytes (e.g. for a `size` field or Content-Length header)        |
 | `close()`                             | release the resource — **always, in a `finally`**                       |
+| `clone()`                             | an independent provider over the same bytes; close it like any other    |
+| `equals(other)`                       | `true` when both hold the same bytes; `false` if either side is closed  |
 
 Parsing structured file content: `SDK.utilities.csvReader({ source: { fileProvider } })`
 (same idea for `xlsxReader`).
 
+## Cloning and comparing
+
+`clone()` returns an independent provider: closing the original does not close the clone, and
+the clone **must be closed too**. A clone of a temp-file provider (`deleteOnClose`) shares the
+file by reference count, so the file is removed only when the last holder closes. A provider
+dropped without `close()` is released when the runtime collects it, which may be much later, so
+closing stays the rule. Cloning a closed provider throws `ResourceAlreadyClosed`.
+
+Deep-copying a row that holds a file:
+
+- `SDK.utilities.clone(row)` copies the data and replaces each provider with a **shim**. The
+  shim serialises like a provider (`"[object FileProvider]"`) and needs no close, but every read
+  (`stream()`, `length()`, `save()`, `blob()`, `formDataValue()`, `clone()`, `equals()`) throws
+  `SCRIPT_FILE_PROVIDER_SHALLOW_CLONE`. That is what a snapshot for logging wants.
+- `SDK.utilities.clone(row, { withFileProviders: true })` clones each provider with `clone()`,
+  so the copy can read the files. The code that made the copy owns its providers and must
+  close them.
+
+`equals(other)` compares bytes: lengths first, then both streams chunk by chunk, without
+reading whole files into memory. It resolves `false` rather than throwing when either side has
+been closed.
+
 ## Upserting: the file arrives as a field value
 
 The upsert row carries a `FileProvider` at the file field's position. Stream it out and close
@@ -121,3 +145,9 @@ async upsert(options) {
 3. **Close what you open.** Providers can be backed by temp files; `close()` in a `finally`
    on both the query and upsert paths.
 
+## Testing
+
+The connector test harness exercises file fields through the upsert, delete and upsertClean
+tests: it asks `upsertPrepare` for a file at `withFileField`, passes the connector a clone, and compares a file
+field listed in `verifyFields` by bytes. See
+[09-testing.md](./09-testing.md#preparecleanup-hooks-and-their-data-types).

package/lib/file-provider-buffer.d.ts

@@ -7,5 +7,7 @@ export declare class FileProviderBuffer implements FileProvider {
     blob(fileName?: string): Promise<Blob>;
     save(path?: string): Promise<string>;
     close(): Promise<void>;
+    clone(): FileProvider;
+    equals(other: FileProvider): Promise<boolean>;
     toJSON(): string;
 }

package/lib/file-provider-disk.d.ts

@@ -7,5 +7,7 @@ export declare class FileProviderDisk implements FileProvider {
     blob(fileName?: string): Promise<Blob>;
     save(path?: string): Promise<string>;
     close(): Promise<void>;
+    clone(): FileProvider;
+    equals(other: FileProvider): Promise<boolean>;
     toJSON(): string;
 }

package/lib/file-provider-string.d.ts

@@ -7,5 +7,7 @@ export declare class FileProviderString implements FileProvider {
     blob(fileName?: string): Promise<Blob>;
     save(path?: string): Promise<string>;
     close(): Promise<void>;
+    clone(): FileProvider;
+    equals(other: FileProvider): Promise<boolean>;
     toJSON(): string;
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.20",
+  "version": "1.0.22",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,9 +12,9 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "376c97b772af51a69f5155aff7c80362062a74710e0672b3dd0582679b5cfffb",
+  "typesContentHash": "35909f8b0b17e27f3d540dfe193fd4802c3c595e646e9a64828fd91d49605500",
   "dependencies": {
     "@types/node": "*",
-    "@syncmatters/script-api": "^1.0.21"
+    "@syncmatters/script-api": "^1.0.25"
   }
 }