@syncmatters/connector-sdk 1.0.16 → 1.0.18

package/docs/01-workspace-and-meta-json.md

@@ -87,6 +87,7 @@ await this.#core.init(args);
 | `oauth2_mode`                                                           | `"connector"` = one shared OAuth app for every account · `"connection"` = each connection supplies its own client id/secret. Omit for non-OAuth connectors.               |
 | `oauth2_provider`                                                       | the OAuth provider definition (below)                                                                                                                                     |
 | `webhooks_provider`                                                     | `{"mode": "custom.perconnection"}` (each connection registers its own webhook) or `{"mode": "custom.perconnector", "url": ..., "webhook_uid": ...}` (one shared endpoint) |
+| `discovery`                                                             | `true` = two-phase connection setup: the entry class implements `discover()` and the form collects credentials first, then the settings and objects discovery returns (below) |
 | `settings`                                                              | the connection-settings form (below)                                                                                                                                      |
 | `icon_url`                                                              | connector logo URL                                                                                                                                                        |
 | `connector_name`, `type_id`, `application_name`, `application_icon_url` | platform/branding ids — usually left as scaffolded                                                                                                                        |
@@ -117,6 +118,18 @@ Per-setting fields: `order` (two-digit string, display order), `name`, `type` (`
 `options` (for `select` only — not used for `timezone`), `default: { "type": "string", "expression": "<value>" }`, and
 `defines_version: true` on the setting whose value picks a version core.
 
+A connection setting accepts the same descriptive fields an object setting does — they were
+always stored, they were simply never written down here:
+
+| Field                                                       | Meaning                                                                                                                                 |
+| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
+| `description`                                               | help text shown under the input — the place to say where the value comes from                                                           |
+| `group_id`                                                  | renders the setting inside a named group                                                                                                |
+| `mandatory: true`                                           | the form blocks save until the setting has a value                                                                                      |
+| `display_if` / `display_if_setting` / `display_if_values`   | conditional rendering (`displayIfHasValue` is the deprecated spelling, still accepted)                                                  |
+| `prerequisite: true`                                        | two-phase connectors only — see below                                                                                                   |
+| `affects_meta: true`                                         | `meta()` reads this setting (a scope, a checkpoint column, an API version), so changing its value marks the connection's cached metadata stale until the next full refresh. Independent of `discovery`; implied by `prerequisite`; no effect on a `metaRefreshDisabled` connector |
+
 Use `control: "timezone"` for IANA time zone pickers (e.g. fleet `time_zone` setting). The stored
 value is a plain string (`"America/New_York"`, `"UTC"`, …); the UI renders a searchable list of
 IANA zones — do not list zones in `options`.
@@ -125,6 +138,76 @@ Conventions worth copying: a secret `api_key`/`clientId`+`clientSecret` pair for
 advanced boolean `debug` setting that gates verbose request/response logging; a
 `connectorVersion` select with `defines_version` when you need breaking revisions to coexist.
 
+### Two-phase setup: `discovery` and `prerequisite`
+
+Most connectors' settings are the same for every tenant, and a static `options` list covers the
+selects. When the real choices are tenant-specific — which databases, companies, subsidiaries or
+objects THIS account has — set `"discovery": true` and implement `discover()`
+([02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)). The form then has two steps:
+
+1. Settings marked `"prerequisite": true` — everything needed to reach the tenant, and the only
+   settings `discover()` gets. **Continue** saves the connection and runs `test()` then `discover()`.
+2. Everything else, rendered from the sidecar definition with the per-setting overrides
+   `discover()` returned merged over it, plus the object picker built from the catalogue.
+
+```json
+"connector_metadata": {
+  "discovery": true,
+  "settings": {
+    "host":     { "order": "00", "name": "Host", "type": "string", "control": "singlelinetext", "mandatory": true, "prerequisite": true },
+    "username": { "order": "01", "name": "User name", "type": "string", "control": "singlelinetext", "mandatory": true, "prerequisite": true },
+    "password": { "order": "02", "name": "Password", "type": "string", "control": "singlelinetext", "secret": true, "mandatory": true, "prerequisite": true },
+    "database": { "order": "03", "name": "Database", "type": "string", "control": "singlelinetext", "mandatory": true, "prerequisite": true,
+                  "description": "The catalogue lists this database's tables and views." },
+    "schema":   { "order": "04", "name": "Schema", "type": "string", "control": "select", "affects_meta": true,
+                  "description": "Restrict the catalogue to one schema; the options come from discovery." },
+    "timeZone": { "order": "05", "name": "Time Zone", "type": "string", "control": "timezone" }
+  }
+}
+```
+
+Rules worth knowing before you mark anything:
+
+- `defines_version: true` implies `prerequisite: true` (the version core has to be chosen before
+  anything can be discovered); `prerequisite: true` in turn implies `affects_meta: true`.
+- Changing a `prerequisite` setting on a live connection marks the stored catalogue stale — the
+  previous result is kept and the user is offered a re-collect. Nothing is queued automatically.
+- A setting with no `options`, in the sidecar or in the overrides, renders as free text. The
+  overrides may also supply `default`, `mandatory`, `readonly`, `hidden` and `description`.
+- `prerequisite` is ignored when `discovery` is absent, and `sm validate` rejects
+  `"discovery": true` with no `prerequisite` setting. It warns when it finds no `discover()` on
+  the entry class (and vice versa): the flag is what the platform reads, so a flagged class
+  without the method fails loudly in the wizard with `NotImplimented` rather than being treated
+  as single-phase.
+
+#### A version that cannot discover: `"discovery": false` on its option
+
+`discovery` is connector-wide, but on a multi-version connector one version core may be unable to
+produce a catalogue (a superseded API). Mark that version's option instead of throwing:
+
+```json
+"version": {
+  "order": "01", "name": "Version", "type": "string", "control": "select", "defines_version": true,
+  "default": { "type": "string", "expression": "3b" },
+  "options": [
+    { "id": "1",  "name": "Superseded - legacy API (v1)", "order": "00", "discovery": false },
+    { "id": "3b", "name": "3b (Recommended)",            "order": "03" }
+  ]
+}
+```
+
+A connection is two-phase when `connector_metadata.discovery` is `true` **and** no `prerequisite`
+(or `defines_version`) select setting has its selected option marked `"discovery": false`. An
+option without the field inherits the connector's flag, so a new version is two-phase with
+nothing written; the only thing you ever write is the exception, next to the "Superseded" label.
+A connection on a marked version gets the single-phase form, `discover()` is never called, and
+its metadata is collected on first use like a connector without discovery. The platform reads the
+connection's selected value, not the setting's `default`. Changing a live connection's version
+across the line keeps its cached objects either way: moving to a two-phase version turns them
+into the selection; moving to a marked version clears the selection, so the next refresh collects
+everything. `sm validate` rejects the field on a setting that is not a prerequisite, and on the
+setting's default option.
+
 ### `oauth2_provider`
 
 Defines the OAuth2 dance the platform performs on the connector's behalf (your code only ever

package/docs/02-lifecycle-and-shape.md

@@ -56,6 +56,13 @@ what you need in `#fields`, build the HTTP client(s), and register the OAuth ref
 Do **not** make network calls in `init()` unless unavoidable — `test()` is where connectivity
 is proven.
 
+On a two-phase connector (`discovery: true`, see
+[01](./01-workspace-and-meta-json.md#two-phase-setup-discovery-and-prerequisite)) the first
+`init()` of a connection's life runs with only the `prerequisite` settings entered, so **never
+throw in `init()` for a setting that is not marked `prerequisite`** — discovery would fail before
+it could offer the user the value. Validate those where they are used (`test()`, `meta()`,
+`query()`), as you already do for optional settings.
+
 ### `mode` (`SDK.ScriptType`) — early-out for static modes
 
 Values include `"integration"`, `"scriptFile"`, `"parseEvents"`, `"verifyEventIdentity"`,
@@ -92,6 +99,55 @@ async test() {
 
 `TestResult` = `{ success: boolean; message?: string; error?: CodedError }`.
 
+`test()` is the credential gate in both phases of a two-phase connection: the platform runs it
+before `discover()` (with only the `prerequisite` settings present — it must tolerate every other
+setting being absent) and again before the first `meta()` with the full settings, where the
+business-rule checks an options list cannot express still belong.
+
+## `discover()` — the tenant's catalogue and settings (optional)
+
+Implement `discover()` when the remaining settings and the object list depend on the tenant:
+which databases, companies or subsidiaries this account has, which tables it can see. It runs
+once the user has entered the `prerequisite` settings and returns the choices the rest of the
+form is built from, so a value that used to be free text the user could get silently wrong
+becomes a select of what actually exists.
+
+```js
+/** @param {SDK.DiscoverOptions} options @returns {Promise<SDK.DiscoverResult>} */
+async discover(options) {
+  const [schemas, tables] = await Promise.all([this.#listSchemas(), this.#listTables()]);
+  options.log.info(`Discovered ${tables.length} table(s)`);
+  return {
+    // per-setting overrides, merged over the sidecar definition (01)
+    settingsMeta: {
+      schema: { options: schemas.map((s) => ({ id: s.name })), mandatory: true },
+    },
+    // the catalogue: every object this tenant could describe, WITHOUT field detail
+    objects: tables.map((t) => ({ id: t.id, name: t.name, default: t.isBase })),
+  };
+}
+```
+
+`DiscoverOptions` is `{ log }` — nothing else. `DiscoverResult` is
+`{ settingsMeta?, objects: SDK.DiscoveredObject[], data? }`, where a `DiscoveredObject` is the
+`id`, `name`, `order`, `isCustom`, `data` and `default` subset of `ObjectMeta` (same ids, same
+meanings, no fields) and `data` is a connector-private payload handed back to `meta()` through
+`options.discovery()` ([04](./04-meta-objects-fields.md#metaoptions--selection-aware-refresh)).
+`default: true` pre-ticks an object in the picker on a new connection.
+
+The contract:
+
+- **Stateless and idempotent.** Every call returns the complete catalogue and overrides for the
+  tenant; nothing about earlier results is passed in. It runs on a new connection, when a
+  `prerequisite` setting changes, and whenever a user asks to check for new objects.
+- **Cheap.** Catalogue endpoints, not per-object sampling — the user is sitting in front of the
+  form. The platform gives it 5 minutes and treats a timeout as a discovery failure.
+- **An empty catalogue is a failure, not a result**, and so is a partial one: throw rather than
+  return a short list, or the picker will offer to deselect objects that merely failed to list.
+  Partial success belongs in an omitted override (say why in that setting's `description`).
+- Identity is not part of it — `meta().identity` remains the only identity.
+- It is never called in static modes (`parseEvents`, `verifyEventIdentity`).
+
 ## `meta()` — see [04-meta-objects-fields.md](./04-meta-objects-fields.md)
 
 Return `{ identity?, objects: ObjectMeta[], events? }`. `identity` should be a stable string for
@@ -113,4 +169,10 @@ When an API revision would break existing connections, add a `connectorVersion`
 (`defines_version: true`, see [01](./01-workspace-and-meta-json.md)) and split the
 implementation into per-version core modules. The entry class reads the setting in `init()` and
 delegates every SDK method to the selected core (`this.#core.query(options)` etc.), loading the
-core with a dynamic `await import("./acme-v3")` so unselected versions never load.
+core with a dynamic `await import("./acme-v3")` so unselected versions never load. `discover()`
+delegates like the rest: a `defines_version` setting is implicitly `prerequisite`, so the version
+is chosen before discovery runs and each core can discover its own way. A core that cannot
+discover at all (a superseded API) is marked on its option with `"discovery": false`
+([01](./01-workspace-and-meta-json.md#a-version-that-cannot-discover-discovery-false-on-its-option)):
+connections on that version stay single-phase and `discover()` is never called for them. Keep a
+`NotImplimented` throw in that core's `discover()` as the backstop.

package/docs/04-meta-objects-fields.md

@@ -15,6 +15,69 @@ async meta() {
 `objects` is an **array**. (`ObjectMetaSummary`/`ObjectMetaDetail` are deprecated — never use.)
 `events` only when you implement webhooks — see [07-events-webhooks.md](./07-events-webhooks.md).
 
+## `MetaOptions` — selection-aware refresh
+
+`meta()` takes an optional argument. A connector that ignores it keeps working exactly as it
+always has: the platform filters the full result to whatever the connection selected. Declare
+the parameter when describing every object is expensive — a hundred views parsed, every table
+described — and you would rather describe only the ones the connection uses.
+
+```js
+/** @param {SDK.MetaOptions} [options] @returns {Promise<SDK.ConnectorMeta>} */
+async meta(options) {
+  const wanted = options?.refreshObjectIds ?? options?.selectedObjectIds;   // undefined = everything
+  const tables = (await this.#listTables()).filter((t) => !wanted || wanted.includes(t.id));
+  return { identity: await this.#accountIdentity(), objects: await this.#describe(tables) };
+}
+```
+
+| Field                          | Meaning                                                                                                                  |
+| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
+| `log`                          | `SDK.Logger` for this call                                                                                               |
+| `selectedObjectIds?`           | the objects the connection has selected. `undefined` = all (a connector with no `discover()`, or an older platform)      |
+| `refreshObjectIds?`            | the subset to (re)build — objects just added to the selection. `undefined` = all selected                                |
+| `objectMeta(id)`               | `Promise<ObjectMeta \| undefined>` — the cached metadata for any object, `undefined` when nothing is cached              |
+| `events()`                     | `Promise<ConnectorMeta["events"] \| undefined>` — the cached event types, for merging on a partial call                  |
+| `discovery()`                  | `Promise<SDK.DiscoverResult \| undefined>` — the stored catalogue, including your own `data`. Fetched only when you call it |
+
+Read it as `options?.selectedObjectIds`: `options` is `undefined` on an older platform, and
+`selectedObjectIds` is `undefined` whenever the connection has no selection. An empty selection
+is `[]`, not absent.
+
+What the platform does with the result:
+
+- Every returned object that is in the selection is upserted, **whether or not it was asked
+  for** — adding object A may legitimately change object B (a new relationship, say). Objects
+  outside the selection are ignored with a log line.
+- A selected object your result omits keeps its cached entry — its fields, its user-edited
+  per-object settings and its refresh time are all left alone. Only a selection change evicts
+  an object, and the connector is not involved in that.
+- Without a selection the old rule still stands: every cached object your result omits is
+  deleted. That is why a connector that ignores `options` is safe, and why a connector that
+  scopes its result MUST only do so from `options`.
+- `identity` is returned on every call, partial or not, and must equal the pinned identity or
+  the refresh is rejected as a re-pointed connection (see [`identity`](#identity)). A connector
+  whose identity is scoped by a discovered setting appends it: `` `${user}|${host}|${database}` ``.
+
+### `default` and `unavailable`
+
+Two optional `ObjectMeta` properties exist for the selection:
+
+- `default: true` — pre-tick this object in the picker on a new connection. It is also a
+  `DiscoveredObject` field, so the catalogue can carry it before any metadata exists (02).
+- `unavailable: "deleted" | "unauthorized"` — a selected object you can no longer describe,
+  because it is gone from the tenant or the API user may no longer read it. Return the object
+  with the flag set instead of omitting it: the platform keeps the cached copy, shows the state
+  and leaves the decision to deselect to the user. Throwing, by contrast, fails the whole
+  refresh.
+
+### Relationships reach objects the connection did not select
+
+Keep emitting every relationship the object has — the selection is the user's, and trimming
+metadata to match it hides the traversal from the sync designer. A `relatedFilter` query against
+an object that is not selected fails with `InvalidObjectId`, which is the honest outcome; the
+platform warns the user at deselect time, naming the syncs that would break.
+
 ## `ObjectMeta` — the canonical shape
 
 ```js
@@ -50,7 +113,9 @@ async meta() {
 
 Strive to align the `queryFields` and `upsertFields` shape (same schema). The `deleteFields` should typically contain a single row key field.
 
-Other fields: `settingsMeta`/`settingsValues` (per-object settings), `order` (sort). Omit flags
+Other fields: `settingsMeta`/`settingsValues` (per-object settings), `order` (sort), and
+`default`/`unavailable` for connections that select their objects
+([above](#default-and-unavailable)). Omit flags
 rather than setting them `false` — `undefined` and `false` both mean "unsupported", and omitted
 reads cleaner.
 

package/docs/07-events-webhooks.md

@@ -29,6 +29,15 @@ events: {
 
 Note that the actual webhook UID varues are system generated, you do not manually specify them here.
 
+**Partial metadata refreshes leave events alone.** When a connection selects its objects, the
+platform may call `meta()` for only some of them (`options.refreshObjectIds`, see
+[04-meta-objects-fields.md](./04-meta-objects-fields.md#metaoptions--selection-aware-refresh)).
+Omitting `events` from such a result keeps the cached event types and their identity — the right
+default, since event types rarely depend on which objects are selected. Return `events` only when
+you mean to replace the whole set; if you need to add to it on a partial call, merge over what is
+already cached with `await options.events()`. Per-connector static webhook modes are unaffected:
+`parseEvents` still runs statically and never triggers discovery.
+
 ## `parseEvents(options)` → `{ events: SDK.ParsedEvent[] }`
 
 The execution context follows the `webhooks_provider.mode`:

package/docs/09-testing.md

@@ -244,7 +244,8 @@ The harness always executes platform-side. What each entry point actually runs:
 | ---------------------------- | ----------------------------------------------------------------------------------------------- |
 | `sm test`                    | The FULL suite — `connection.test()` + `metaRefresh()` + every object feature above. Identical to the web UI's connector test. |
 | `sm test --only connection`  | `connection.test()` only — credentials check, nothing else. The harness has NOT run.            |
-| `sm test --only meta`        | `connection.metaRefresh()` only.                                                                |
+| `sm test --only meta`        | `connection.metaRefresh()` only — for a connection with a selection, that is `meta()` over the selected objects. |
+| `sm test --only discover`    | `connection.discover()` only (two-phase connectors) — reports the catalogue size, which settings it returned overrides for, and how long it took, then refreshes the selection and checks every selected object is in the catalogue. |
 | `sm test --only query`       | Every object's plain list query only — no ids/checkpoint/match/related, no writes.              |
 | Web UI (connector → test results → run) | Same engine, feature selection via the run dialog.                                   |
 
@@ -262,7 +263,16 @@ names the missing one. The dev loop around it: `sm validate` (workspace structur
 type checking against the SDK types, `sm push`, then `sm test`. Keep `test()` cheap and
 `meta()` deterministic so the suite stays fast.
 
-Two workspace facts that bite here:
+Three workspace facts that bite here:
+
+- **A two-phase test connection must SELECT every object the test module covers.** On a
+  connector with `discover()` the connection caches only its selected objects, so
+  `sm connector features` generates from those alone and the suite can only test those — an
+  object in `mod Test.mjs` that nobody selected reports "not present on connection". The CLI
+  says which ids are missing rather than skipping them quietly; fix it in the connection's
+  object picker, not in the test file. `sm connector features` also warns about catalogue
+  entries the connection did not select, which is the reminder to widen the test connection
+  when `meta()` grows a new object.
 
 - `mod ObjectTestFeatures.mjs` is **generated** — regenerate it with
   `sm connector features` (or the web UI's feature-generate action) after `meta()` changes;

package/docs/10-style-and-pitfalls.md

@@ -81,6 +81,38 @@
     typically surfaces as a composite-key parse error or "row not found", nowhere near
     `meta()`. Direction rules: [04](./04-meta-objects-fields.md#relationships),
     [05](./05-query.md#relatedfilter).
+17. **Returning a partial catalogue from `discover()` is worse than failing.** When some
+    catalogue call errors or a permission is missing, THROW. A short list looks like a tenant
+    that lost objects: the picker offers to deselect them and the user acts on it. Same for an
+    empty catalogue — the platform treats it as a failure. Partial success is expressed by
+    omitting a setting's override (and saying why in that setting's `description`), never by
+    trimming `objects` ([02](./02-lifecycle-and-shape.md#discover--the-tenants-catalogue-and-settings-optional)).
+18. **`meta(options)` must work when `options` is `undefined`.** `options` is absent on an older
+    platform, and `options.selectedObjectIds` is absent whenever the connection has no
+    selection — always read it as `options?.selectedObjectIds`. A `meta()` that indexes
+    `options` directly fails the very first refresh of every connection that has no selection
+    yet, which includes every existing connection on the day you add the parameter. And scope
+    the result ONLY from what `options` says: without a selection the platform still deletes
+    every cached object the result omits
+    ([04](./04-meta-objects-fields.md#metaoptions--selection-aware-refresh)).
+19. **Throwing in `init()` for a setting that is not `prerequisite`** breaks two-phase setup:
+    the first `init()` of a connection's life runs with only the prerequisite settings entered,
+    so a connector that demands the rest up front fails discovery before the user can be shown
+    the values ([02](./02-lifecycle-and-shape.md)). Validate them where they are used.
+20. **An unmarked version whose core cannot discover fails loudly in the wizard.** That is the
+    design: with `"discovery": true` every version is two-phase unless its option says
+    `"discovery": false`, so a new connection on an unmarked legacy version hits the core's
+    `NotImplimented` throw on Continue (shown inline, with Retry) instead of being silently
+    downgraded. Mark the option
+    ([01](./01-workspace-and-meta-json.md#a-version-that-cannot-discover-discovery-false-on-its-option)).
+    Marking the setting's default option is a `sm validate` error: every new connection would
+    be single-phase.
+21. **Reading another object's metadata through `objectMeta()` works on legacy connections and
+    fails on selected ones.** Keep an object self-contained: put what its own query/upsert needs
+    on its `data`; treat any other object's metadata as optional and fail with a clear
+    `ConnectorError` when it is absent. A two-phase connection caches only the objects the user
+    selected, so a companion object the parent reads its association ids or aliases from may
+    simply not be there.
 
 ## Useful utilities you might otherwise reimplement
 

package/docs/12-example-connector.md

@@ -572,6 +572,86 @@ function parseSalesOrderKey(compositeKey) {
 }
 ```
 
+## If Acme were two-phase: `discover()` and a scoped `meta()`
+
+Acme as written is single-phase, which is right for it: two settings, three objects, every
+account the same. Suppose instead that an Acme account exposes its own set of business units and
+its own custom objects — the values the user has to type today (the business unit) and the
+objects worth caching are then tenant-specific, which is exactly what `discover()` is for
+([02](./02-lifecycle-and-shape.md#discover--the-tenants-catalogue-and-settings-optional)).
+Three deltas, nothing else changes.
+
+**Sidecar** — turn the flow on, mark what discovery needs, and add the setting it fills in:
+
+```json
+"connector_metadata": {
+  "categories": ["sales"],
+  "agent_mode": "never",
+  "discovery": true,
+  "settings": {
+    "api_key":      { "order": "00", "name": "API Key", "type": "string", "control": "singlelinetext", "secret": true, "mandatory": true, "prerequisite": true },
+    "businessUnit": { "order": "01", "name": "Business Unit", "type": "string", "control": "select", "affects_meta": true,
+                      "description": "Sales orders are scoped to this unit; the options come from your account." },
+    "debug":        { "order": "02", "name": "Debug", "advanced": true, "type": "boolean", "control": "singlelinetext" }
+  }
+}
+```
+
+**`discover()`** — catalogue endpoints only, no per-object sampling:
+
+```js
+  /** Tenant catalogue + the options for businessUnit (02). @param {SDK.DiscoverOptions} options @returns {Promise<SDK.DiscoverResult>} */
+  async discover(options) {
+    const [units, catalogue] = await Promise.all([
+      this.#call({ method: "GET", url: "businessunits" }),
+      this.#call({ method: "GET", url: "objects" }),
+    ]);
+    options.log.info(`Acme account exposes ${catalogue.items.length} object(s)`);
+    return {
+      settingsMeta: {
+        businessUnit: {
+          options: units.items.map((/** @type any */ u) => ({ id: String(u.id), name: u.name })),
+          mandatory: true,
+        },
+      },
+      // no field detail here - just the ids meta() will be asked to describe.
+      // 'default' pre-ticks the three built-ins in the picker on a new connection
+      objects: catalogue.items.map((/** @type any */ o) => ({
+        id: o.id, name: o.label, isCustom: o.custom === true, default: o.custom !== true,
+      })),
+    };
+  }
+```
+
+**`meta(options)`** — scope the expensive work, not just the result. Acme's cost is one
+`/fields` call per object, so the filter goes in front of those, and the `objects` array is
+filtered on the way out:
+
+```js
+  /** @param {SDK.MetaOptions} [options] @returns {Promise<SDK.ConnectorMeta>} */
+  async meta(options) {
+    const me = await this.#call({ method: "GET", url: "me" });
+    // undefined = no selection (legacy connection, or an older platform): describe everything
+    const scope = options?.refreshObjectIds ?? options?.selectedObjectIds;
+    const wanted = (/** @type string */ id) => !scope || scope.includes(id);
+    const [companyCustom, contactCustom] = await Promise.all([
+      wanted("companies") ? this.#customFields("companies") : [],
+      wanted("contacts") ? this.#customFields("contacts") : [],
+    ]);
+
+    /** @type {SDK.ObjectMeta[]} */
+    const objects = [ /* ...unchanged... */ ];
+
+    // identity is returned on EVERY call, partial or not, and must still match the pin (04)
+    return { identity: `${me.account_name}|${me.email}`, objects: objects.filter((o) => wanted(o.id)) };
+  }
+```
+
+Note what does NOT change: the relationships stay on every object even when the other end is
+unselected, `test()` still runs before both `discover()` and `meta()`, and `init()` still
+throws for a missing `api_key` — it is `prerequisite`, so it is always there — but it must
+not start throwing for `businessUnit`, which is only entered after discovery.
+
 ## What to notice (the parts assistants most often get wrong)
 
 1. **One `#call` wrapper** owns base URL, auth and debug logging; nothing else touches HTTP.

package/docs/14-vendor-sdks-and-non-http.md

@@ -96,10 +96,28 @@ Databases and on-premise systems are a connector shape of their own:
 - **Escape SQL values** exactly as [10-style-and-pitfalls.md](./10-style-and-pitfalls.md)
   demands for any string-interpolated query language — or better, use the driver's
   parameterized queries.
+- **`discover()` earns its keep here** — the database name, the schema and the table list are
+  exactly the settings a static sidecar cannot know. Set `"discovery": true`, mark the
+  connection settings (host, credentials, database) `prerequisite`, and return the schemas as
+  `settingsMeta` options and the tables/views as the catalogue from a handful of catalog
+  queries ([01](./01-workspace-and-meta-json.md#two-phase-setup-discovery-and-prerequisite),
+  [02](./02-lifecycle-and-shape.md#discover--the-tenants-catalogue-and-settings-optional)).
+  `meta({ selectedObjectIds })` then describes only the tables the connection uses instead of
+  every table in the database, which is the difference between a refresh of seconds and one of
+  minutes.
 
 Note: on-premise (`agent_mode: "always"`) connectors do not support `upsertClean`
 ([06-upsert-delete.md](./06-upsert-delete.md)).
 
+Everything in this guide applies unchanged on the agent: `discover()` and `meta(options)` are
+plain calls with JSON in and JSON out, and the host-provided accessors on `MetaOptions`
+(`objectMeta`, `events`, `discovery`) are proxied across the boundary the way
+`QueryOptions.objectMeta` already is — `await options.discovery()` costs a round trip, so call
+it once and keep the result, and don't call it at all if you don't need it. The one thing that
+does differ is version skew: an agent runtime too old to honour `MetaOptions` fails the refresh
+with "agent upgrade required" rather than quietly collecting every object, so a customer on a
+stale agent sees a clear error instead of a slow one.
+
 ## XML and SOAP systems
 
 - SOAP/WSDL: the `soap` npm package, with the WSDL files bundled as workspace resource files

package/docs/15-example-test-file.md

@@ -532,6 +532,20 @@ export default class TestFeatureProvider {
 }
 ```
 
+## If Acme were two-phase: the test connection's selection
+
+Nothing in this pair changes when a connector gains `discover()`
+([12](./12-example-connector.md#if-acme-were-two-phase-discover-and-a-scoped-meta)) — but the
+**test connection** does. A connection that selects its objects caches only those, so
+`objectIds()` above returns three ids only if the sandbox connection has `companies`,
+`contacts` AND `salesorders` ticked in its object picker. Miss one and it reports
+`Failed: Object from test spec not present on connection`, and `sm connector features`
+regenerates `mod ObjectTestFeatures.mjs` without it — it warns about the catalogue entries the
+connection did not select, so read that output rather than assuming a shrunken generated file is
+the connector's doing. Widen the test connection's selection rather than scoping the object out;
+`sm test --only discover` ([09](./09-testing.md#running-the-suite)) checks the catalogue against
+the selection and prints what is missing.
+
 ## What to notice (the parts assistants most often get wrong)
 
 1. **Every created record is prefixed** (`sm_test`) and `before()`/`after()` sweep the

package/docs/17-user-help-pages.md

@@ -59,10 +59,19 @@ Sections, in this order (omit ones that don't apply):
    doc plus a one-line click-path ("Configure → Dev Center → API Credential Management →
    Create New API Key").
 3. **Connection settings** — one bullet per setting, `*(sensitive)*` on secrets, defaults and
-   allowed values stated; `### Advanced settings` for the advanced group.
+   allowed values stated; `### Advanced settings` for the advanced group. On a two-phase
+   connector (`connector_metadata.discovery`, see
+   [01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md#two-phase-setup-discovery-and-prerequisite))
+   split the list the way the form does: the `prerequisite` settings the customer enters to
+   reach their tenant, then the settings the connector fills in for them — say where the
+   options come from ("the databases your login can see") rather than listing values that
+   differ per account, and name the ones that mark metadata stale when changed.
 4. **Supported objects** — link to the vendor's API reference, then the table
    `| Name | Operations ^1 | Description |` with the `^1 Q=Query, U=Upsert, D=Delete`
-   footnote. Note that the live per-connection list is in the connection UI.
+   footnote. Note that the live per-connection list is in the connection UI. On a two-phase
+   connector the table is the catalogue the connector CAN describe; the connection caches only
+   the objects the customer selected, so also say how to add one later (the object picker's
+   "check for new objects" re-runs discovery) and that metadata is refreshed for the selection.
 5. **Capabilities & limits** — rate limits, timestamp formats, API quirks, insert-only
    objects, pagination oddities. Start with the standard callout:
    > In a Sync, source filtering and collecting only changed records (incremental/checkpoint)

package/docs/connector-authoring.md

@@ -21,7 +21,7 @@ queryable.
 | File                                                             | Read when you are...                                                                                            |
 | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
 | [01-workspace-and-meta-json.md](./01-workspace-and-meta-json.md) | adding files, wiring modules, editing any `*.meta.json` sidecar, declaring settings/OAuth/webhooks/npm packages |
-| [02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)         | starting a connector; implementing `init`/`test`/`meta`; anything about `InitArgs`                              |
+| [02-lifecycle-and-shape.md](./02-lifecycle-and-shape.md)         | starting a connector; implementing `init`/`test`/`discover`/`meta`; anything about `InitArgs`                              |
 | [03-http-auth-ratelimit.md](./03-http-auth-ratelimit.md)         | making HTTP calls, rate limiting, retries, API keys, basic auth, OAuth2                                         |
 | [04-meta-objects-fields.md](./04-meta-objects-fields.md)         | writing `meta()` — objects, fields, flags, keys, relationships, filters                                         |
 | [05-query.md](./05-query.md)                                     | implementing `query()` — paging, checkpoints, ids/match/related filters, `Row`                                  |
@@ -119,6 +119,8 @@ Implementing more methods unlocks more platform capability:
 | You implement                           | The platform can                                                    |
 | --------------------------------------- | ------------------------------------------------------------------- |
 | `test()` + `meta()` (**required**)      | validate connections, show objects/fields in the UI                 |
+| `discover()` (+ `"discovery": true`)    | two-phase connection setup: offer the tenant's real settings options and let the customer pick the objects to cache (01, 02) |
+| `meta(options)` (the optional argument) | refresh metadata for only the selected objects instead of every object, every time (04) |
 | `query()`                               | read data — lists, differential syncs, lookups                      |
 | `upsert()` / `delete()`                 | write data                                                          |
 | `upsertClean()`                         | pre-validate + change-detect rows before writing (`canUpsertClean`) |
@@ -155,6 +157,9 @@ Implementing more methods unlocks more platform capability:
 3. **Make `test()` real**: cheapest authenticated call the API offers (02).
 4. **Describe objects** in `meta()` (04). Start with one object and its `isKey` field; grow from
    there. `SDK.utilities.fieldsFromJSON(sample)` can draft field lists from a sample response.
+   If the settings or the object list depend on the tenant — a database, a company, a
+   subsidiary, a hundred views of which a connection uses six — add `discover()` and
+   `"discovery": true` (02, 01) and take `meta(options)`'s selection into account (04).
 5. **Implement `query()` capabilities in order**: list → ids → checkpoint → match → related
    (05). For EVERY object aim for all five — syncs are built from them. When the API is
    awkward (parent-scoped paths, no changed-since cursor), work the patterns in
@@ -167,7 +172,8 @@ Implementing more methods unlocks more platform capability:
 8. **Test harness**: add `mod Test.mjs` (+ generated `mod ObjectTestFeatures.mjs`) so the
    platform's connector test suite can exercise your objects (09; worked example: 15).
 9. **Run it**: `sm test` = the full suite (identical to the web UI's connector test — including
-   write features, so use a sandbox connection); `--only connection|meta|query` narrows. A
+   write features, so use a sandbox connection); `--only connection|meta|query|discover`
+   narrows. A
    passing `--only` run is NOT suite coverage (09).
 10. **Document it**: write the customer help pages IN the connector —
     `Connectors/<Name>/help/index.md` + `help/scripting.md` (sidecar type `"markdown"`),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@syncmatters/connector-sdk",
-  "version": "1.0.16",
+  "version": "1.0.18",
   "description": "TypeScript type definitions for the SyncMatters connector SDK (types only - connectors execute on the SyncMatters platform)",
   "types": "./index.d.ts",
   "exports": {
@@ -12,9 +12,9 @@
   "license": "MIT",
   "author": "SyncMatters",
   "homepage": "https://syncmatters.com",
-  "typesContentHash": "07fac99bb193fe5a9bff764336c8de1815d3729720589c2d893dd45a987e6eab",
+  "typesContentHash": "2d30d816e8bb71a9fe2d9390945f4aa62c8eff1f4f86e69ec8f965e8a42cd6c8",
   "dependencies": {
     "@types/node": "*",
-    "@syncmatters/script-api": "^1.0.17"
+    "@syncmatters/script-api": "^1.0.19"
   }
 }