@elastic.io/mcp 1.0.0-beta

package/resources/docs/flow-building-guide.md

@@ -0,0 +1,581 @@
+# elastic.io Flow Building Guide
+
+This guide describes how to construct and update integration flows on the elastic.io iPaaS platform.
+
+## Flow Architecture
+An elastic.io flow contains a directed graph of steps (nodes) and connections (edges).
+
+### Step ID Format
+Every `step_id` must follow the format `step_N`, where `N` is a positive integer starting from `1`. Use the exact prefix `step_` followed by the numeric index, with no spaces or additional characters.
+
+**Valid examples**: `step_1`, `step_2`, `step_3`, ..., `step_N`
+
+**Invalid examples**: `step1`, `Step_1`, `step_0`, `step_01`, `my_step_1`, `step_lk_from`
+
+> [!CAUTION]
+> Always use this format when specifying a `step_id` in flow-related API requests. The platform rejects non-conforming step IDs and they cause errors in the UI.
+
+## Flow Scheduling (Cron)
+To run a flow on a periodic schedule (e.g. every 10 minutes, daily, etc.):
+1. Set the flow type to `"ordinary"` (scheduled).
+2. Set the `"cron"` property under attributes to a standard 5-field cron expression:
+   * **Default**: `"cron": "*/10 * * * *"` (every 10 minutes) — applied automatically if not specified
+   * `"cron": "*/15 * * * *"` (every 15 minutes)
+   * `"cron": "0 * * * *"` (hourly)
+   * `"cron": "0 0 * * *"` (daily at midnight)
+3. The first step of the flow graph must be a scheduling/trigger component, such as the Simple Trigger.
+
+## Common Core Components
+When building flows, you can look up these components or use their IDs:
+1. **Simple Trigger (Timer)**
+   * **Component ID**: `aaa1111111bbb2222222ccc01`
+   * **Command**: `elasticio/simple-trigger-component:timer`
+2. **Email Component**
+   * **Component ID**: `aaa1111111bbb2222222ccc02`
+   * **Command**: `elasticio/email-component:send`
+3. **Splitter**
+   * **Component ID**: `aaa1111111bbb2222222ccc03`
+   * **Command**: `elasticio/splitter:splitOnJsonataExpression` (preferred) or `elasticio/splitter:split` (deprecated)
+   * **Key field**: `expression` — a JSONata expression that MUST use `$getPassthrough().step_id.body` to reference the array to split. Do NOT use bare `body`.
+4. **Node.js Code Component (Custom Logic)**
+   * **Component ID**: Discover this using list_components (commonly `aaa1111111bbb2222222ccc04` or `aaa1111111bbb2222222ccc05`).
+   * **Command**: Format using the standard template below, with the action ID `execute`.
+
+## Node (Step) Schema & Object-to-String Normalization
+When retrieving a flow config via `get_flow`, the API returns steps with object metadata. However, when patching or creating a flow via `create_flow` or `update_flow`, you MUST normalize these objects into plain strings:
+
+1. **`component_id`**:
+   * *Retrieved (get_flow)*: `{ "componentId": "aaa1111111bbb2222222ccc06", ... }`
+   * *Constructed (update/create)*: Plain string, e.g., `"aaa1111111bbb2222222ccc06"`
+2. **`command`**:
+   * *Retrieved (get_flow)*: `{ "actionOrTrigger": "receive", "componentVersion": "..." }`
+   * *Constructed (update/create)*: Plain string. For all components (like Salesforce v2, Email, Code, HubSpot, etc.), it MUST be formatted dynamically using the fields returned by the `get_component` tool: `"{commandPrefix}:{action_or_trigger_id}@latestVersionId"` (e.g. `"elasticio/email-component:send@latestVersionId"`). Specifying only the action/trigger name (e.g. `"getUpdatedObjectsPolling"` or `"execute"`) will fail validation and result in a backend 500 error.
+
+> [!CAUTION]
+> **Deprecated Actions/Triggers**: Before selecting an action or trigger, you MUST check the component descriptor (returned by `get_component`) for the `"deprecated": true` flag on the action/trigger definition. If an action/trigger is deprecated, you MUST NOT use it — select the non-deprecated alternative instead. Common examples:
+> * **REST API V2**: Use `httpRequestAxios` (Axios Library), NOT `httpRequestAction` (Request Library, deprecated)
+> * **REST API V2 trigger**: Use `httpRequestAxiosTrigger`, NOT `httpRequestTrigger` (deprecated)
+> * **Slack**: Use `makeRawRequest` or `upsertObject`, NOT `postMessage` (deprecated)
+> * **Splitter**: Use `splitOnJsonataExpression`, NOT `split` (deprecated)
+>
+> Using deprecated actions causes a "Deprecated" badge in the UI and may lead to removal in future component versions. When updating existing flows that use deprecated actions, replace them with the non-deprecated equivalent.
+
+> [!CAUTION]
+> **Trigger vs Action mismatch**: When recommending a component for a flow step, you MUST verify that the component actually has the right type of operations for the intended role:
+> * For the **first step** (trigger position): the component must have at least one **trigger** defined in its descriptor.
+> * For **subsequent steps** (action positions): the component must have at least one **action** defined in its descriptor.
+>
+> Many components are specialized — e.g., `hubspot-webhook-component` only has triggers (it receives webhook events), while `hubspot-component` has both triggers and actions. Similarly, other `-webhook-` or `-listener-` components typically lack actions.
+>
+> Before presenting component options to the user, call `get_component` for each candidate and filter out components that lack the required operation type. Do NOT list components that cannot serve the intended role — this confuses users and wastes time.
+
+> [!IMPORTANT]
+> **Webhook Component Credentials**: Although the Webhook component (`elasticio/webhook:receive`) does not require authentication to receive payloads, you MUST always propose attaching credentials to the Webhook step when building flows. Webhook credentials enable HMAC signature verification, API key validation, basic auth, and other security features. Even if the user says "just a simple webhook", explicitly ask whether they want to secure it and offer to set up or attach credentials. Never silently omit credentials on a Webhook step without asking the user.
+
+3. **`credentials_id`** (for Credential-based components):
+   * *Retrieved (get_flow)*: `{ "credentialId": "CRED_ID", "credentialName": "..." }`
+   * *Constructed (update/create)*: Plain string, e.g., `"CRED_ID"`
+4. **`secret_id`** (for Secret-based/OAuth2 components, e.g., Salesforce v2, REST API v2, Google Sheets v2, Zoho):
+   * *Retrieved (get_flow)*: `{ "secretId": "SECRET_ID", "secretName": "..." }`
+   * *Constructed (update/create)*: Plain string, e.g., `"SECRET_ID"`
+5. **`fields`**: Key-value pairs containing static component-level configuration parameters (e.g. selecting a direction, input formats, output formats, or a code string for the Code component). These are defined by the component descriptor or select models. Do NOT put input metadata fields here (those must be configured in the incoming connection edge's mapper instead).
+
+> [!IMPORTANT]
+> **JSONata Fields in Node `fields` (e.g. Splitter `expression`)**: Some component fields use `viewClass: "JSONataView"` and contain JSONata expressions (e.g., the Splitter's `expression` field). These fields follow the **same passthrough path rules** as edge mappers. You MUST use `$getPassthrough().step_id.body` (or `$getPassthrough().step_id.body.fieldName`) to reference data from prior steps — do NOT use bare references like `body` or field names. For example, to split an array received by a Webhook in step_1:
+> * **INCORRECT**: `"expression": "body"` — resolves to empty/undefined
+> * **CORRECT**: `"expression": "$getPassthrough().step_1.body"` — correctly references the Webhook's body
+
+### Data Samples
+Data samples store example input/output data for flow steps, used for building mappings between steps in the UI.
+
+**Where data sample IDs are stored:**
+- In the flow draft, each node can have a `selected_data_samples` array containing data sample IDs:
+  ```json
+  {
+    "id": "step_1",
+    "component_id": "55ba18e35d04040500000004",
+    "command": "elasticio/webhook:receive@...",
+    "selected_data_samples": ["6aa3f14f03c95f00125ec437"]
+  }
+  ```
+- This field is visible via `get_flow_draft` but NOT via `get_flow` (where `nodes_config` may appear empty).
+
+**Updating a data sample:**
+- Use `update_data_sample` with the `dataSampleId` and new `body` to replace the sample's `result.body` content. The UI picks up changes immediately — no need to touch the draft.
+
+**Creating and attaching a data sample:**
+- Use `create_data_sample` to create a new sample, then include its ID in the node's `selected_data_samples` array when calling `update_flow_draft`.
+
+### Dynamic Dropdowns (Select Models)
+Some component settings (fields) are dropdowns where the user selects from a list of options (e.g., choosing a Salesforce Object Type or a database). These fields are defined under `fields` with `viewClass: "SelectView"` in the descriptor.
+- **Static Select Models**: If the `model` property of the field in the descriptor is a JSON object (e.g., `{"available": "Available", "pending": "Pending"}`), the keys in that object are the static options you can use in `fields`.
+- **Dynamic Select Models**: If the `model` property is a string (e.g., `"getDatabasesModel"` or `"objectTypes"`), it is a dynamic function. To retrieve the valid dropdown options:
+  1. Call the `init_dynamic_select_model` tool, passing the `workspaceId`, `componentId`, `actionOrTriggerName`, and `methodName` (which is the string value of `model`), along with any credential/secret and preceding dependent field configurations (e.g. if selecting a collection depends on the database `db` field).
+  2. Call the `get_scheduled_execution_result` tool with the returned `executionId` (with default `poll: true`) to wait and fetch the populated options object. Use the keys from this object to populate the setting in `fields`.
+
+### Dynamic Metadata
+Some components define their input/output schemas dynamically based on other step configuration settings (e.g., Salesforce or database operations where fields depend on the selected object type). These triggers or actions are defined with `"dynamicMetadata": true` in the component descriptor.
+- To retrieve the dynamic input and output schemas:
+  1. Call the `init_dynamic_metadata` tool, passing the `workspaceId`, `componentId`, and `actionOrTriggerName` (which maps to the module), along with any credential/secret and preceding dependent fields configuration (e.g., `{"sobject": "Account"}`).
+  2. Call the `get_scheduled_execution_result` tool with the returned `executionId` (with default `poll: true`) to wait and retrieve the completed schema.
+  3. The returned schemas are automatically pruned of large enum lists (e.g., country/state codes) to save context tokens. Use these schemas to build JSONata mappings on the step's incoming edge.
+
+### Credential Verification
+Before verifying credentials for a component step:
+- Call the `init_verify_credentials` tool, passing the `workspaceId`, `componentId`, and optional `versionId`.
+- To verify a credential or secret connection, pass its `credentialId` or `secretId`. The tool will automatically retrieve its configured fields and initiate verification. You can optionally pass overrides in the `fields` parameter.
+- The tool polls internally and returns the final verification result directly.
+
+There are two primary styles of credential/secret verification based on component authorization types:
+1. **OAuthFieldView (e.g. Salesforce v2)**: The credentials/tokens must be nested inside an `oauth` object (e.g. containing `access_token`, `refresh_token`, `expires_in`, and a nested `undefined_params` object with `instance_url`, etc.). The tool handles this structure automatically when verifying secrets of type `oauth2`.
+2. **HTTPAuthView (e.g. REST API v2, Code Component with credentials)**: The fields are flat at the root of the `fields` object. These secrets have a top-level type of `auth-secret`. The sub-types are:
+   - **No Auth (`noauth`)**: Represents an empty `credentials` object.
+   - **Basic Auth (`basic`)**: Contains flat `username` and `password` fields.
+   - **API Key (`api_key`)**: Contains flat `headerName` and `headerValue` fields.
+   - **OAuth2 (`oauth2`)**: Contains flat `access_token`, `refresh_token`, `expires_in`, and optionally `scope` and `additional_params` fields.
+3. **Plain Credentials**: Passed flat at the root of the `fields` object (mapping to `keys` on the credential object, e.g. `apiKey`).
+
+
+### Example: Node.js Code Component Step
+To execute custom JavaScript code:
+```json
+{
+  "id": "step_3",
+  "component_id": "aaa1111111bbb2222222ccc05",
+  "command": "elasticio/code:execute@latestVersionId",
+  "name": "Custom JavaScript Code Block",
+  "fields": {
+    "code": "async function run(msg, cfg) {\n  console.log('Input message:', msg);\n  const data = msg.body;\n  if (!data) return;\n  await this.emit('data', { body: { success: true, processed: data } });\n}"
+  }
+}
+```
+
+### Example: Splitter Component Step (Split on JSONata Expression)
+The Splitter's `expression` field is a JSONata expression that MUST use `$getPassthrough()` to reference data from prior steps. Assuming the expression evaluates to an array, each element is emitted as a separate message.
+
+#### Splitting a Webhook body (step_1 is Webhook receiving an array)
+```json
+{
+  "id": "step_2",
+  "component_id": "aaa1111111bbb2222222ccc03",
+  "command": "elasticio/splitter:splitOnJsonataExpression@latestVersionId",
+  "name": "Split Items",
+  "fields": {
+    "expression": "$getPassthrough().step_1.body"
+  }
+}
+```
+
+#### Splitting a nested array from REST API response (step_1 is REST API V2 trigger)
+The REST API V2 trigger emits `{ statusCode, HTTPHeaders, responseBody }`. To split the `responseBody` array:
+```json
+{
+  "id": "step_2",
+  "component_id": "aaa1111111bbb2222222ccc03",
+  "command": "elasticio/splitter:splitOnJsonataExpression@latestVersionId",
+  "name": "Split API Response",
+  "fields": {
+    "expression": "$getPassthrough().step_1.body.responseBody"
+  }
+}
+```
+
+#### Splitting a deeply nested array (e.g. `responseBody.data.items`)
+```json
+{
+  "id": "step_2",
+  "component_id": "aaa1111111bbb2222222ccc03",
+  "command": "elasticio/splitter:splitOnJsonataExpression@latestVersionId",
+  "name": "Split Nested Items",
+  "fields": {
+    "expression": "$getPassthrough().step_1.body.responseBody.data.items"
+  }
+}
+```
+
+#### Splitting after a Code component (step_2 is Code that emits an array in `body.results`)
+```json
+{
+  "id": "step_3",
+  "component_id": "aaa1111111bbb2222222ccc03",
+  "command": "elasticio/splitter:splitOnJsonataExpression@latestVersionId",
+  "name": "Split Code Results",
+  "fields": {
+    "expression": "$getPassthrough().step_2.body.results"
+  }
+}
+```
+
+> [!WARNING]
+> **Common Splitter mistakes**:
+> * `"expression": "body"` — bare reference, resolves to empty/undefined
+> * `"expression": "responseBody"` — bare reference, same problem
+> * `"expression": "$getPassthrough().step_1.responseBody"` — missing `.body.` segment; passthrough always wraps in `body`
+> * **CORRECT**: `"expression": "$getPassthrough().step_1.body.responseBody"`
+
+> [!CAUTION]
+> **UI validation limitation for deep JSONata paths**: The UI form validator for the Splitter's `expression` field only accepts paths up to `$getPassthrough().step_X.body` (one level after `body`). Deeper paths like `$getPassthrough().step_1.body.data.items` work correctly at runtime but cause a **red configuration indicator** and **blocked Continue button** in the UI.
+>
+> **Workaround**: Instead of splitting a deeply nested array directly, restructure the flow:
+> 1. Have the sender send the array directly as the body (e.g., `[item1, item2, ...]` instead of `{ "data": { "items": [...] } }`), OR
+> 2. Add a **Code component** before the Splitter to extract the nested array and emit it as the body, then use `$getPassthrough().step_X.body` in the Splitter.
+>
+> **Do NOT use deep paths** like `$getPassthrough().step_1.body.responseBody.data.items` even though they are valid JSONata — they will cause UI issues for users editing the flow.
+
+### REST API V2 `reader` Field Format (RESTAPIView)
+Components using `viewClass: "RESTAPIView"` (e.g., REST API V2) store HTTP request configuration in a `reader` field object.
+
+#### Headers Format
+The `headers` property inside `reader` **MUST** be an array of `{key, value}` objects with `frozen: true`, NOT a plain object. Header values that are string literals **MUST** use escaped double quotes (JSONata string syntax):
+
+**INCORRECT** (causes `TypeError: r.map is not a function` in the frontend):
+```json
+"headers": {"Content-Type": "application/json"}
+```
+
+**CORRECT:**
+```json
+"headers": [
+  {"key": "Content-Type", "value": "\"application/json\"", "frozen": true}
+]
+```
+
+#### Body Format for POST/PUT/PATCH
+For HTTP methods that send a request body (POST, PUT, PATCH), the `body` property inside `reader` **MUST** be an object with the following structure — NOT a plain string:
+
+```json
+"body": {
+  "raw": "<JSONata expression or static JSON string>",
+  "formData": [],
+  "urlencoded": [],
+  "contentType": "application/json"
+}
+```
+
+* `raw`: The request body content. Can be a JSONata expression (e.g., `$getPassthrough().step_1.body`) or a static JSON string (e.g., `{"city":"London"}`).
+* `formData`: Must be `[]` (empty array) unless using multipart/form-data.
+* `urlencoded`: Must be `[]` (empty array) unless using application/x-www-form-urlencoded.
+* `contentType`: The body content type (e.g., `"application/json"`).
+
+> [!WARNING]
+> Omitting `body` for POST/PUT/PATCH methods causes a red validation indicator in the UI. Passing `body` as a plain string instead of an object causes `TypeError: Cannot create property 'raw' on string` in the frontend.
+
+#### Complete POST Example
+```json
+{
+  "reader": {
+    "url": "\"https://httpbin.org/post\"",
+    "method": "POST",
+    "headers": [
+      {"key": "Content-Type", "value": "\"application/json\"", "frozen": true}
+    ],
+    "body": {
+      "raw": "{\n  \"data\": $getPassthrough().step_1.body\n}",
+      "formData": [],
+      "urlencoded": [],
+      "contentType": "application/json"
+    }
+  }
+}
+```
+
+## Separation of Concerns: Node Fields vs. Edge Mapper
+An integration step has two separate kinds of configurations that must not be mixed:
+
+1. **Static Configuration (Node Fields)**:
+   * Configured under the step's `fields` object.
+   * Defines how the component behaves globally (e.g. choosing `"jsonToHjson"` vs `"hjsonToJson"` or selecting an attachment format).
+   * Do NOT include input schema fields (fields that receive data from other steps) in `node.fields`.
+
+2. **Data Mapping (Edge Mapper)**:
+   * Configured under the incoming edge's `config.mapper` object.
+   * Maps outputs of previous steps to the current step's input schema fields (as defined by the component's input metadata schema).
+   * **CRITICAL: Respect field nesting from input metadata.** If the component's input metadata defines fields nested inside an object (e.g., Marketo's `values` object containing `email`, `firstName`, etc.), the mapper MUST mirror that nesting structure. Always call `init_dynamic_metadata` first to discover the exact field hierarchy, then build the mapper accordingly. Do NOT flatten nested fields to the top level. If `init_dynamic_metadata` fails (e.g., returns 500 due to broken credentials), do NOT guess the mapping structure — inform the user that the metadata is unavailable and that valid credentials are required to determine the correct field structure.
+   * **CRITICAL: Use property keys, not display names.** The UI shows human-readable titles (e.g., "E-mail Address", "First Name") from the JSON Schema `title` property, but the mapper MUST use the actual property keys from the metadata schema (e.g., `email`, `firstName`). These often differ — they may be in camelCase, snake_case, or use completely different words. Always get the correct keys from `init_dynamic_metadata` output.
+   * **CRITICAL: Mapper values MUST use real output field names from source steps — NEVER guess or hallucinate field names.** See the "Correct Mapping Algorithm" section below for the required procedure.
+   * For example, if the component's input metadata defines a field `jsonToConvert`, the mapper on the incoming edge must contain: `"mapper": { "jsonToConvert": "body" }`. This mapping is NEVER placed inside the step's `fields` object.
+
+### Correct Mapping Algorithm (MANDATORY)
+A mapper has two parts: **keys** (what the target step expects as input) and **values** (where the data comes from in previous steps). Both MUST be derived from real metadata — NEVER guessed or hallucinated.
+
+> [!CAUTION]
+> **Do NOT invent or guess field names for mapper values.** The most common and critical mistake is assuming what fields a previous step outputs based on general knowledge of the system (e.g., assuming Salesforce returns `Name`, `Id`, `Phone`). The actual output fields depend on the specific component version, action configuration, and object type — they may differ from what you expect. ALWAYS retrieve the real output schema.
+
+**Step-by-step procedure for building a mapper on an edge from `step_X` (source) to `step_Y` (target):**
+
+1. **Retrieve TARGET input schema (mapper keys):**
+   * Call `init_dynamic_metadata` for the TARGET step (`step_Y`) with its `componentId`, `actionOrTriggerName`, credentials, and configured `fields`.
+   * Poll `get_scheduled_execution_result` to get the result.
+   * The `in` (input) schema properties define the mapper **keys** — the field names the target step expects to receive.
+
+2. **Retrieve SOURCE output schema (mapper values):**
+   * Call `init_dynamic_metadata` for the SOURCE step (`step_X`) with its `componentId`, `actionOrTriggerName`, credentials, and configured `fields`.
+   * Poll `get_scheduled_execution_result` to get the result.
+   * The `out` (output) schema properties define what field names the source step **actually emits**. Use ONLY these field names in `$getPassthrough().step_X.body.<field>` paths.
+   * If the source step has `dynamicMetadata: true`, you MUST call `init_dynamic_metadata` for it — do NOT skip this step.
+   * If the source step does NOT have `dynamicMetadata: true`, check the action/trigger's static `metadata.out` schema from `get_component` instead.
+
+3. **Build the mapper object:**
+   * For each field in the target's input schema (`in.properties`), create a mapper entry:
+     * **Key**: the property name from the target's input schema
+     * **Value**: a JSONata expression using `$getPassthrough().step_X.body.<output_field>` where `<output_field>` is a real field name from the source's output schema
+   * If a required target input field has no corresponding source output field, inform the user and ask how to handle it (e.g., use a static literal value, a Code component to transform data, or skip the field).
+
+**Example:**
+```
+Source step (step_1): Salesforce "getUpdatedObjectsPolling" with sobject="Account"
+  → init_dynamic_metadata returns out.properties: { "Id": {...}, "Name": {...}, "BillingCity": {...} }
+
+Target step (step_2): Email "send"
+  → init_dynamic_metadata returns in.properties: { "to": {...}, "subject": {...}, "textBody": {...} }
+
+Mapper on edge step_1 → step_2:
+{
+  "to": "\"admin@example.com\"",
+  "subject": "\"Account Updated: \" & $getPassthrough().step_1.body.Name",
+  "textBody": "\"ID: \" & $getPassthrough().step_1.body.Id & \"\\nCity: \" & $getPassthrough().step_1.body.BillingCity"
+}
+```
+
+> [!WARNING]
+> **When updating an existing flow's mapping**, always re-retrieve the output metadata of the source step. Do NOT reuse field names from the existing mapper without verification — the source step's configuration may have changed (e.g., different object type selected), which changes the output schema.
+
+## Edge (Connection) Schema & JSONata Mapping
+Each edge in the `graph.edges` array connects two steps:
+* `id`: Unique identifier (typically `source_id:target_id` or `mapper:source_id:target_id`).
+* `source`: The source step ID.
+* `target`: The target step ID.
+* `config`: Optional mapping config:
+  * `mapper_type`: Must be `"jsonata"`
+  * `mapper`: A JSONata expression string.
+    * > [!IMPORTANT]
+    * > **CRITICAL MAPPING RULE (PASSTHROUGH PATHS)**: You MUST ALWAYS use the explicit passthrough path format: `$getPassthrough().step_id.body.field` (e.g., `$getPassthrough().step_1.body.Name`) to reference any properties/fields from prior steps, where `step_id` is the ID of the node that originally produced or fetched the field (e.g., `step_1` for Salesforce). This applies even if it's the immediate predecessor. NEVER reference fields by their plain name (like `Name` or `Id`) or implicit context. They will fail to resolve.
+    * > **JSONATA QUOTING AND ESCAPING RULES**:
+    * > 1. **Use escaped double quotes (`\"`) for literal strings in JSONata, but do NOT wrap the entire expression or concatenation in outer quotes.**
+    * >    * **INCORRECT**: `"subject": "\"'Salesforce Account Imported: ' & Name\""`
+    * >    * **INCORRECT**: `"to": "\"'user@example.com'\""`
+    * >    * **CORRECT**: `"to": \"user@example.com\"` (escaped double quotes for the string literal)
+    * >    * **CORRECT**: `"subject": \"Salesforce Account Imported: \" & $getPassthrough().step_1.body.Name` (concatenation, escaped double quotes only around the literal text, not the whole expression!)
+    * >    * **CORRECT**: `"textBody": \"ID: \" & $getPassthrough().step_1.body.Id & \"\\nName: \" & $getPassthrough().step_1.body.Name`
+    * > 2. **Double Escaping Newlines**: To include a newline in a JSONata string literal within the JSON payload, write it as `\\n` (double backslash).
+    * > 3. **No Quotes on Path/Variable Expressions**: Never put quotes around path expressions like `$getPassthrough().step_1.body.Name`.
+    * > 4. **Preceding Step Output Schema**: Standard message payload emitted is `{ "body": { ... } }`. Therefore, always start with `$getPassthrough().step_id.body.field`.
+    * > 5. **Array-typed mapper fields MUST be JSONata strings, NOT JSON arrays.** When a component's input metadata defines a field with `"type": "array"` (e.g., the ChatGPT component's `messages` field), the mapper value MUST be a **JSONata expression string** that evaluates to an array — NOT a native JSON array. The UI cannot render native JSON arrays in the mapper and will show empty fields.
+    * >    * **INCORRECT** (UI shows empty fields):
+    * >      ```json
+    * >      "messages": [{"role": "\"system\"", "content": [{"type": "\"text\"", "text": "\"Hello\""}]}]
+    * >      ```
+    * >    * **CORRECT** (UI renders properly):
+    * >      ```json
+    * >      "messages": "[{\"role\": \"system\", \"content\": [{\"type\": \"text\", \"text\": \"Hello\"}]}]"
+    * >      ```
+    * >    Note: Inside the JSONata string, literal values like `"system"` use standard JSONata double quotes (escaped as `\"` in JSON). Dynamic values use passthrough paths without quotes: `$getPassthrough().step_1.body.field`.
+
+> [!CAUTION]
+> **UI visual mapper limitation with array-typed input fields**: When a component's input schema defines a field with `"type": "array"` (e.g., `file_urls`, `messages`, `recipients`), the UI visual mapper **cannot map properties from prior steps into array items**. The mapper dropdown shows "Please note that it is impossible to map properties that are not inside the array property" and no properties are selectable — even if the previous step (e.g. Webhook) has valid output data.
+>
+> **This is a UI-only limitation** — the JSONata mapping set via API works correctly at runtime. However, users editing the flow in the UI will see red validation indicators and empty fields.
+>
+> **When building flows via API with array-typed input fields, you MUST**:
+> 1. Always set the mapper on the edge — never leave the edge without a `config.mapper` for steps that have array-typed inputs.
+> 2. Use a JSONata array expression string (starting with `[`) that constructs the array inline, e.g.:
+>    ```
+>    "[{\"file_url\": $getPassthrough().step_1.body.file_url, \"filename\": $getPassthrough().step_1.body.filename}]"
+>    ```
+> 3. Do NOT use a plain passthrough like `$getPassthrough().step_1.body.file_urls` for array fields — even if the webhook sends an array, the UI cannot render this mapping and shows red indicators.
+>
+> **Inform the user** that the flow will work correctly at runtime, but the UI may show red indicators on array fields. They can verify the mapping by clicking the `<>` (JSONata) toggle in the UI.
+
+### Example: Edge with Array-typed Mapper Field (e.g. ChatGPT messages)
+```json
+{
+  "id": "step_1:step_2",
+  "source": "step_1",
+  "target": "step_2",
+  "config": {
+    "mapper_type": "jsonata",
+    "mapper": {
+      "messages": "[{\"role\": \"system\", \"content\": [{\"type\": \"text\", \"text\": \"Classify this support ticket.\"}]}, {\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": $getPassthrough().step_1.body.description}]}]"
+    }
+  }
+}
+```
+
+### Example: Edge with Correct JSONata Mapper
+```json
+{
+  "id": "step_2:step_3",
+  "source": "step_2",
+  "target": "step_3",
+  "config": {
+    "mapper_type": "jsonata",
+    "mapper": {
+      "to": "\"user@example.com\"",
+      "subject": "\"Salesforce Account Imported: \" & $getPassthrough().step_1.body.Name",
+      "textBody": "\"Salesforce Account Details:\\nID: \" & $getPassthrough().step_1.body.Id & \"\\nName: \" & $getPassthrough().step_1.body.Name & \"\\nPhone: \" & $getPassthrough().step_1.body.Phone & \"\\nWebsite: \" & $getPassthrough().step_1.body.Website"
+    }
+  }
+}
+```
+
+> [!NOTE]
+> **Supported Mapper Formats**:
+> Elastic.io supports two mapper formats for `edge.config.mapper`:
+> 1. **Per-field Object Mapper** (`{"fieldName": "expression"}`): Standard format for simple/flat component mappings where each property is mapped individually in the visual UI. Nested objects and arrays of objects/expressions are fully supported.
+> 2. **Root JSONata String Mapper** (`"{\n  \"field\": expression\n}"` or `"(\n  $var := ...;\n  {\n    \"field\": ...\n  }\n)"`): Ideal for complex payloads, nested objects/arrays, or developer-mode mappings requiring JSONata block variables. Ensure the expression starts with `{`, `(`, or `[`, and is NOT wrapped in outer literal quotes.
+
+## Read-Only Properties to Strip
+When sending an `update_flow` (PATCH) payload, the API will fail if read-only properties are present in the request. You MUST strip the following keys from `attributes` and `relationships`:
+
+| Section | Properties to Delete/Exclude |
+|---|---|
+| **attributes** | `created_at`, `updated_at`, `status`, `current_status`, `active`, `status_changed_at`, `type`, `container_state`, `error_count`, `api_version`, `links`, `stats_enabled` |
+| **relationships** | `user`, `contract`, `versions`, `latest_version`, `workspace` (exclude workspace on update, keep only on create) |
+
+## Safe Flow Modification Lifecycle
+To change the graph of the flow or replace one component with another:
+1. **Stop if Active**: If the flow is active, stop it using `stop_flow` (POST `/flows/:id/stop`).
+2. **Wait for Shutdown**: Query the flow configuration using `get_flow` and check that it is fully inactive:
+   `data.attributes.status === data.attributes.current_status === "inactive"`
+   (Wait/poll since container cleanup can take 10-30 seconds).
+3. **Check for Existing Draft**: Call `get_flow_draft` (GET `/flows/:id/draft`) to retrieve the current draft. If 404, no draft exists — the first `update_flow_draft` call will create one.
+4. **Apply Changes to Draft**: Use `update_flow_draft` (PUT `/flows/:id/draft`) to make modifications on the draft. For multiple changes, call `update_flow_draft` iteratively — one change at a time or in logical batches. Do NOT modify the flow directly via `update_flow` until the draft is finalized.
+5. **Publish Draft**: Once the draft is confirmed, call `update_flow` (PATCH `/flows/:id`) to publish the draft and promote it to the active flow configuration.
+6. **Start Flow**: Restart the flow by calling `start_flow` (POST `/flows/:id/start`).
+
+## Recommended Flow Creation Strategy (Empty → Skeleton → Mappers → Publish)
+When building flows, use a three-stage draft approach. The key principle: **`data` (published config) stays empty until the final Publish step** — all construction happens on the **draft**, just like in the UI.
+
+### Stage 1: Create Empty Flow
+Use `create_flow` with **minimal attributes only** — no nodes, no edges, no graph:
+- Include only: `name`, `type` (e.g. `"ordinary"`), `cron` (if scheduled), and `workspace_id`.
+- Do NOT include `graph`, `nodes`, `edges`, or step configurations.
+- This creates an empty flow with an empty `data` section and a blank draft — exactly like clicking "Add New Flow" in the UI.
+
+### Stage 2: Build Skeleton on Draft
+Use `update_flow_draft` to add the flow structure (nodes and edges without mappers):
+1. Call `get_flow(flowId)` to confirm the flow exists.
+2. Call `get_flow_draft(flowId)` to retrieve the current (empty) draft.
+3. Call `update_flow_draft(flowId, body)` with all nodes (`component_id`, `command`, `credentials_id`/`secret_id`, `name`, static `fields`) and edges (`source`, `target` only — no `config.mapper`).
+4. This validates the structure (components, credentials, graph topology) independently of JSONata mappings.
+
+### Stage 3: Add Mappers on Draft
+Use `update_flow_draft` to iteratively add mappers to edges:
+1. Call `get_flow_draft(flowId)` to retrieve the current draft (now containing the skeleton).
+2. For each edge that needs a mapper:
+   - Retrieve the **target step's input schema** — call `init_dynamic_metadata` (if `dynamicMetadata: true`) or use static `metadata.in` from `get_component`.
+   - Retrieve the **source step's output schema** — call `init_dynamic_metadata` or use static `metadata.out` from `get_component`.
+   - Build the mapper object using real field names via `$getPassthrough().step_id.body.field` paths.
+3. Call `update_flow_draft(flowId, body)` with the updated graph containing `config.mapper` and `config.mapper_type: "jsonata"` on the relevant edges.
+4. Repeat per edge or in batches — each `update_flow_draft` call updates the draft in place.
+
+> [!TIP]
+> The draft is visible in the UI throughout all stages. The user can open the flow and see it being built step by step — just like editing in the UI.
+
+> [!IMPORTANT]
+> **Data Sample Updates**: After changing a step's configuration (`fields`) or an edge's mapper, you MUST ask the user whether they want to update/regenerate the data sample for the affected step(s). Stale data samples cause incorrect field previews in the UI, making the flow confusing for users who later edit it manually. Use `debug_flow_step` or `create_data_sample` to regenerate samples.
+
+### Publish
+Once all mappers are configured and verified on the draft:
+- Use `update_flow(flowId, body)` (PATCH) to publish the draft — this promotes the draft to `data` (the active flow configuration).
+- Before this step, `data` remains empty. After publishing, `data` contains the full flow configuration from the draft.
+- Optionally call `start_flow` to activate.
+
+### Benefits
+- **Matches UI behavior**: `data` stays empty until publish, just like manual flow creation in the UI.
+- **Separation of concerns**: Stage 2 validates structure; Stage 3 validates mappings. If `update_flow_draft` fails in Stage 2 → structural issue. If it fails in Stage 3 → mapper issue.
+- **Iterative**: The skeleton and mappers can be built incrementally, verified against real schemas from `init_dynamic_metadata`.
+- **UI-friendly**: The flow appears in the UI immediately after Stage 1; the user can inspect progress in real time as the draft evolves.
+- **Metadata-driven**: Dynamic metadata calls work better when the flow already exists and step dependencies are resolvable.
+- **Safe**: Nothing is published until explicitly confirmed — all work stays on the draft.
+
+## UI Configuration vs. REST API Payload Separation
+When helping users or explaining component configurations, always distinguish between **what the user configures on the UI** and **what the LLM sends to the elastic.io REST API**. These are entirely different concepts:
+
+> [!WARNING]
+> DO NOT confuse UI Settings/Fields and Input Mappings with REST API Nodes and Edges when explaining component configuration to a user. Always present UI-oriented instructions (e.g., "Set the Object Type setting to 'Account' and map the 'First Name' input field to...") unless the user is asking how to build/update a flow via the REST API or using tools like create_flow / update_flow.
+
+### 1. What the User Configures in the Browser UI
+When a user sets up a component (e.g. Salesforce or Webhook) in the browser, they configure:
+* **Fields / Settings**: The static configuration inputs (e.g., selecting "Account" as the object type or setting "Direction" to "jsonToHjson"). These are metadata/descriptor-driven.
+* **Mappings (Input Schema Fields)**: Mappings connecting previous steps' output fields to the current step's input fields (e.g., mapping an incoming "company_name" from a Webhook to the Salesforce "Name" field). These input fields are defined dynamically by the component's input metadata schema.
+
+### 2. What the LLM Sends to the REST API (Request Body Payload)
+Under the hood, the REST API does not have a separate "settings" and "mappings" screen. Instead, everything is serialized into a single JSON API `graph` payload:
+* **User's Static Settings/Fields** are placed inside the **Node's `fields` object** (`nodes[i].fields`):
+  ```json
+  {
+    "id": "step_3",
+    "component_id": "salesforce-component-id",
+    "command": "upsertObject",
+    "fields": {
+      "object": "Account" // Static settings go in the node fields
+    }
+  }
+  ```
+* **User's Input Mappings** are placed inside the **Edge's `config.mapper` object** (`edges[j].config.mapper`):
+  ```json
+  {
+    "id": "step_2:step_3",
+    "source": "step_2",
+    "target": "step_3",
+    "config": {
+      "mapper_type": "jsonata",
+      "mapper": "{\n  \"Name\": $getPassthrough().step_2.body.company_name\n}" // Mappings go in the edge mapper, NEVER the node fields
+    }
+  }
+  ```
+
+## Content-Based Router (Branching)
+The Content-Based Router component (`elasticio/router`, action `route`) examines message content and routes messages to different flow branches based on conditions.
+
+> [!CAUTION]
+> **Do NOT use the Router for single-branch filtering.** The Router component is only justified when there are **2 or more outgoing branches** with different conditions. If you only need to conditionally pass or drop messages, put the `condition` directly on the edge between the source and target steps — no Router node needed. A Router with a single outgoing edge is redundant and adds unnecessary complexity to the flow.
+
+### Router Node
+The router node itself has no special fields — just set the `command` to `route`:
+```json
+{
+  "id": "step_2",
+  "component_id": "<router_component_id>",
+  "command": "elasticio/router:route@<latestVersionId>",
+  "name": "Router",
+  "fields": {}
+}
+```
+
+### Router Edge Conditions
+Each outgoing edge from the router MUST have a `condition` property in its `config`. The condition is a JSONata expression that evaluates to `true` or `false`:
+```json
+{
+  "id": "step_2:step_3",
+  "source": "step_2",
+  "target": "step_3",
+  "config": {
+    "condition": "$getPassthrough().step_1.body.status = \"paid\""
+  }
+}
+```
+
+If the edge also has a mapper, include both `condition` and `mapper`/`mapper_type` in `config`.
+
+> [!WARNING]
+> **Restricted JSONata Functions in Router Conditions**: Router conditions run in a restricted JSONata context. The following built-in JSONata functions are **NOT available** and will cause "Attempted to invoke a non-function" errors:
+> - `$now()`, `$millis()` — time generation functions
+> - `$hour()`, `$minute()`, `$second()`, `$year()`, `$month()`, `$day()` — time component extraction functions
+> - `$toMillis()`, `$fromMillis()` — time conversion functions
+> - `$formatTimestamp()`, `$parseTimestamp()`
+>
+> **Workaround**: Use `$substring()` to extract parts from ISO 8601 date strings available in passthrough data. For example, to extract the hour from a `fireTime` like `"2026-08-26T14:30:00.000Z"`:
+> ```
+> $substring($getPassthrough().step_1.body.fireTime, 11, 2) < "12"
+> ```
+>
+> **Available functions** in conditions include: `$getPassthrough()`, `$substring()`, `$string()`, `$number()`, `$length()`, `$contains()`, `$split()`, `$join()`, `$lowercase()`, `$uppercase()`, `$trim()`, `$replace()`, `$match()`, and standard comparison/logic operators.
+
+## Abstract Flow Topology Templates (Few-Shot Reference)
+When you need to construct a new flow:
+1. **Query Patterns**: Call `search_flow_samples` with semantic queries (e.g. "polling sync timer", "splitter loop", "webhook reply", "batch collect") to retrieve the matching abstract pattern topology.
+2. **Template Decoupling**: The retrieved templates contain generic dummy component IDs (e.g. `generic_source_component_id`), placeholder credentials (`TARGET_CREDENTIAL_ID_PLACEHOLDER`), and version hashes (`@latestVersionId`).
+3. **Resolve Real Properties**: Discover the actual component IDs by calling `list_components` and `get_component`, and discover credentials using `list_credentials`/`list_secrets`.
+4. **Swap and Connect**: Swap the real component IDs, credentials/secrets, commands, and fields into the abstract pattern layout, and map properties using standard JSONata mapping rules (`$getPassthrough().step_id.body.field`).

package/resources/docs/rest-api-pagination.md

@@ -0,0 +1,58 @@
+# Implementing Pagination with the REST API Component and Any Component That Uses the makeRawRequest Action
+
+The REST API component's `httpRequestAxios` action (and the `makeRawRequest` action available in other components) does not provide automatic pagination or automatically retrieve subsequent pages. To retrieve all available records, implement pagination logic directly in your integration flow using a recursive flow execution pattern.
+
+The pagination mechanism depends on the target REST API. It may provide a next-page URL, a cursor, a page number, an offset, or another pagination parameter. Always follow the pagination mechanism documented by the target API.
+
+## 1. Flow Linking: Receive pagination parameters
+
+Configure a Flow Linking trigger to receive the input required to execute a paginated request.
+
+The input should contain the initial request URL or parameters for the first execution. For subsequent executions, it should contain the pagination information extracted from the previous response.
+
+## 2. REST API: httpRequestAxios
+
+Configure the REST API component (or any component that uses the `makeRawRequest` action) with the `httpRequestAxios` action to execute the HTTP request.
+
+- Set the HTTP method, URL, headers, authentication, and other required request parameters.
+- Use the initial URL or pagination parameters received from the Flow Linking trigger.
+- Execute the request and retrieve the response.
+
+The action provides the HTTP status code, response headers (`HTTPHeaders`), and response body (`responseBody`), unless the response is configured to be downloaded as an attachment.
+
+## 3. Filter or Router: Check whether another page is available
+
+Inspect the response body and, if applicable, the response headers to determine whether more records are available.
+
+Use the pagination mechanism supported by the target API. For example:
+
+- **Next-page URL**: Check whether the response contains a next-page URL or link.
+- **Cursor-based pagination**: Check whether the response contains a continuation cursor or token.
+- **Offset-based pagination**: Check whether the API indicates that additional records are available and calculate the next offset.
+- **Page-based pagination**: Check whether another page exists and increment the page number.
+
+If another page is available, continue to the next step. Otherwise, stop pagination because the last page has been reached.
+
+## 4. Flow Linking: Trigger the next execution
+
+If another page is available, use Flow Linking to trigger the same flow again.
+
+Pass the pagination information extracted from the previous response, such as the next-page URL, cursor, offset, or page number.
+
+The next execution must use this information to configure the subsequent REST API request.
+
+## 5. Repeat until all pages are retrieved
+
+Repeat the request and pagination-checking steps until the target API indicates that no more pages are available.
+
+Ensure that all retrieved records are processed or stored as required by the integration flow. If necessary, use additional steps to aggregate records from multiple responses.
+
+## Important considerations
+
+- Do not assume that all REST APIs use the same pagination mechanism or response structure.
+- Use the pagination parameters and continuation indicators documented by the target API.
+- Do not construct or modify pagination URLs when the API provides a complete next-page URL. Follow the returned URL as specified by the API.
+- Ensure that the pagination condition correctly identifies the last page to prevent unnecessary requests or infinite execution loops.
+- Consider API rate limits and execution constraints when implementing recursive requests. Configure appropriate delays between requests when necessary.
+
+This approach allows you to retrieve all available records from REST APIs that use pagination, even when automatic pagination is not supported by the REST API component or other components that use the `makeRawRequest` action.