@elastic.io/mcp 1.0.0

package/resources/docs/credential-validation-gaps.md

@@ -0,0 +1,35 @@
+# Credential Validation Gaps
+
+## Problem
+When building or updating a flow, the MCP server does not validate whether credentials/secrets are actually functional before allowing them to be used. This leads to flows being created with broken credentials that fail at runtime.
+
+## Discovered on flow `6aaa5f922fbba1001240ee87`
+- Step 7: Azure Data Lake Storage (`makeRawRequest`)
+- Secret used: `68655543cb48720012860817` ("My New Azure Data Lake Storage component Credential") — state `ready` but account `sparrow1test` does not exist (DNS failure)
+- Second secret: `6842d77f5758c500127a90b1` ("QA") — state `error`, OAuth refresh_token expired since July 2025
+
+## Fix applied (partial)
+Changed `discoveryState.secrets` from `Set<string>` to `Map<string, string>` (id → state) so that `checkDiscoveryState()` can reject secrets in `error` state **before** submitting to the API.
+
+### Files changed
+- `src/tools/types.ts` — `secrets` type changed to `Map<string, string>`
+- `src/tools/handlers.ts` — initialization updated
+- `src/tools/handlers/auth.ts` — `list_secrets` now stores state
+- `src/tools/validation.ts` — `checkDiscoveryState()` rejects non-ready secrets
+- `src/tools/__tests__/handlers.test.ts` — updated to use Map
+
+## Additional fix: verification tracking and pre-submission blocking
+
+Added `credentialVerifications: Map<string, { ok: boolean; error?: string }>` to `DiscoveryState`. 
+
+### Changes
+- `src/tools/types.ts` — added `credentialVerifications` field
+- `src/tools/handlers.ts` — initialization updated
+- `src/tools/handlers/auth.ts` — `init_verify_credentials` now polls verification result internally and stores it in `discoveryState.credentialVerifications`. Returns final result instead of pending executionId.
+- `src/tools/validation.ts` — `checkDiscoveryState()` checks verification results for all nodes and blocks credentials/secrets that failed verification
+- `src/tools/__tests__/handlers.test.ts` — 6 new tests + 4 updated existing tests
+
+### What this covers
+1. **Secret in `ready` state but actually broken** — caught if `init_verify_credentials` was called (verification result stored and checked pre-submission)
+2. **Plain credentials** — same mechanism via `credentialVerifications` Map
+3. **Test coverage** — secret state check + verification result check for both secrets and plain credentials

package/resources/docs/debug-issues.md

@@ -0,0 +1,71 @@
+# Debug Issues — Component Testing
+
+Issues discovered during debug_flow_step testing on flow `6aad4e79acd7d1001240041f`.
+
+---
+
+## Apptivo — Invalid API endpoint
+
+- **Component**: `PS_component_testing/apptivo-component` (`689b49481890650012d6e5b9`)
+- **Action**: `makeRawRequest`
+- **Credential**: `QA_test` (`689b54ef1890650012d72e17`) — verifies OK
+- **Problem**: All `makeRawRequest` calls return `500 Internal Server Error` with `{"status_code":"1503","status_reason":"Invalid API end point","status":"failure"}`
+- **Tried URLs**:
+  - `/contacts?a=getAllContacts&countPerPage=1`
+  - `/contacts?a=getContactById&contactId=1`
+  - `contacts?a=getAllContacts&countPerPage=1` (without leading slash)
+- **Base URL**: Component docs say base is `https://app.apptivo.com/app/dao/v6`, so the raw request URL should be relative to that. None of the tried paths worked.
+- **Status**: Unresolved. Needs investigation of correct Apptivo API v6 URL format.
+
+---
+
+## Square — URL requires no `/v2` prefix
+
+- **Component**: `PS_component_testing/square-component` (`667be8681ed49100120a1197`)
+- **Action**: `makeRawRequest`
+- **Credential**: `QA` (`68babd3faa954a00120aabbf`) — verifies OK
+- **Problem**: `/v2/locations` returns 404 (`INVALID_REQUEST_ERROR / NOT_FOUND`), but `/locations` works fine (200 with location data).
+- **Root cause**: The component's base URL already includes `/v2`, so raw request paths should NOT include the `/v2` prefix. This is not obvious from the component descriptor.
+- **Status**: Resolved. Use paths without `/v2` prefix (e.g. `/locations` instead of `/v2/locations`).
+
+---
+
+## Phantom data sample / dynamic_metadata deletion
+
+- **Flow**: `6aad4e79acd7d1001240041f`
+- **Problem**: After building 18 steps with debug samples, UI showed "Failed to load the samples" for all steps. Investigation found 4 missing resources (404):
+  - `6ab260c7adc8720012aacb24` — `dynamic_metadata` on step_8 (AWS S3)
+  - `6ab2771aab80c900126621e0` — `dynamic_metadata` on step_13 (Gemini)
+  - `6ab2771badc8720012adf2dc` — `dynamic_metadata` on step_14 (Grok)
+  - `6ab28333ab80c9001267dd0a` — `selected_data_samples` on step_18 (Stripe)
+- **UI behavior**: `src/frontend/src/modules/store/actions/data-samples.js:70-75` uses `Promise.all` to load all samples; if any returns 404, the entire load fails with "Failed to load the samples".
+- **What dynamic_metadata actually is**: Stored in MongoDB collection `DynamicMetadata` (`src/commons/lib/models/dynamic_metadata.js`), NOT in `DataSample`. Created by `/v2/components/{id}/versions/{ver}/dynamic-metadata`. The `dynamic_metadata` field on a node references a DynamicMetadata record, not a data sample — so a 404 on these should NOT cause data sample loading to fail (unless the UI tries to load them as data samples).
+- **No TTL/GC exists**: Neither `DataSample` nor `DynamicMetadata` models have `expireAfterSeconds` indexes. Wiper does not clean them. Both are only deleted on workspace deletion or recipe re-export.
+- **Root cause**: Unknown. The records disappeared without any known cleanup mechanism. Possible causes:
+  - Manual deletion via API or DB
+  - Debug process resetting the draft and orphaning/deleting associated records
+  - Race condition during concurrent draft modifications (UI + API)
+- **Fix applied**: Removed broken `dynamic_metadata` references from step_8/13/14, re-debugged Stripe to create new data sample `6ab2851cadc8720012b015e0`.
+- **Status**: Fixed but root cause unknown. Needs investigation of what deletes data samples and dynamic metadata records outside of workspace deletion.
+
+---
+
+## SAP Business One — Debug timeout (30s)
+
+- **Component**: `PS_component_testing/sap-business-one-component` (`676e7cb9d6b1100012f5e9be`)
+- **Action**: `makeRawRequest`
+- **Credential**: (`676ea7e5d6b1100012f6d05b`)
+- **Input**: `{"method": "GET", "url": "BusinessPartners?$top=1"}`
+- **Problem**: `debug_flow_step` times out after 30s. On-premise SAP system too slow to respond within the debug timeout window.
+- **Status**: Unresolved. Synthetic data sample used instead. Live retrieval not possible with current 30s timeout.
+
+---
+
+## MS Dynamics CRM V2 — DNS resolution failure
+
+- **Component**: `PS_component_testing/microsoft-dynamics-crm-v2` (`64592579991bbb00128be3e4`)
+- **Action**: `makeRawRequest`
+- **Secret**: (`686fc9f14109aa0012d32f07`)
+- **Input**: `{"method": "GET", "url": "accounts?$top=1"}`
+- **Problem**: `500 Internal Server Error` — `getaddrinfo ENOTFOUND org15005212.crm4.dynamics.com`. The Dynamics CRM instance hostname cannot be resolved (instance may be decommissioned or offline).
+- **Status**: Unresolved. Synthetic data sample used instead. Credential/instance needs verification.

package/resources/docs/deployment-and-scaling.md

@@ -0,0 +1,136 @@
+# Deployment and Scaling Guide (SSE Mode)
+
+This guide documents the architecture, horizontal scaling requirements, and security controls for deploying the elastic.io MCP server in multi-tenant cloud environments.
+
+---
+
+## 1. The Stateful Nature of SSE Mode
+
+When running in Server-Sent Events (SSE) mode, the server operates on a hybrid architecture:
+1. **Persistent SSE Stream (GET /sse)**: Keeps an open, persistent HTTP response channel to push events from the server to the client.
+2. **Client-to-Server Messages (POST /sse or POST /messages)**: Short-lived HTTP requests sent by the client to invoke tools or send notifications.
+
+Under the hood, each user session creates a live Node.js `McpServer` object and an active REST client tied to the user's platform credentials. **This connection state and session memory exist solely in the RAM of the specific Node.js process/container that initialized it.**
+
+---
+
+## 2. Horizontal Scaling & Clustering (Sticky Sessions)
+
+Because the session manager is in-memory, horizontal scaling across multiple container instances (e.g. behind AWS ALB, Kubernetes NGINX Ingress, or GCP Cloud Load Balancing) requires **Sticky Sessions (Session Affinity)**. 
+
+Without Sticky Sessions:
+* **The Issue**: A client connects to `GET /sse` and routes to *Instance A* (creating the session in memory). The client's subsequent tool call `POST /messages` gets routed by the load balancer to *Instance B*. *Instance B* does not have this session ID in its memory, returning a `404 Session not found` error.
+* **The Solution**: The load balancer must bind a client's session to the same backend container instance for the duration of the session.
+
+### Load Balancer Configuration Patterns
+
+Below are configuration patterns for major cloud providers:
+
+#### A. AWS Application Load Balancer (ALB)
+Configure **Duration-Based Stickiness** or **Application-Controlled Stickiness** on your Target Group:
+* **Stickiness Type**: `lb_cookie` (Load Balancer Generated Cookie)
+* **Cookie Duration**: Match your session idle timeout (e.g., `1800 seconds` / 30 minutes).
+* **Terraform Example**:
+  ```hcl
+  resource "aws_lb_target_group" "mcp_server" {
+    name     = "mcp-server-tg"
+    port     = 3000
+    protocol = "HTTP"
+    vpc_id   = var.vpc_id
+
+    stickiness {
+      type            = "lb_cookie"
+      cookie_duration = 1800
+      enabled         = true
+    }
+  }
+  ```
+
+#### B. Kubernetes NGINX Ingress Controller
+Add the session affinity annotations to your Ingress resource. NGINX will automatically inject a cookie (default: `INGRESSCOOKIE`) to route subsequent requests:
+* **Ingress YAML Example**:
+  ```yaml
+  apiVersion: networking.k8s.io/v1
+  kind: Ingress
+  metadata:
+    name: mcp-server-ingress
+    annotations:
+      nginx.ingress.kubernetes.io/affinity: "cookie"
+      nginx.ingress.kubernetes.io/session-cookie-name: "route"
+      nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
+      nginx.ingress.kubernetes.io/session-cookie-expires: "1800"
+      nginx.ingress.kubernetes.io/session-cookie-max-age: "1800"
+  spec:
+    rules:
+    - host: mcp.yourdomain.com
+      http:
+        paths:
+        - path: /
+          pathType: Prefix
+          backend:
+            service:
+              name: mcp-server-service
+              port:
+                number: 3000
+  ```
+
+#### C. Google Cloud Platform (GCP) HTTP(S) Load Balancing
+Set the backend service's session affinity to `GENERATED_COOKIE`:
+* **gcloud Command**:
+  ```bash
+  gcloud compute backend-services update mcp-server-backend-service \
+      --session-affinity=GENERATED_COOKIE \
+      --affinity-cookie-ttl=1800 \
+      --global
+  ```
+
+---
+
+## 3. Production Resource & Rate Limit Considerations
+
+### Redis-Backed Rate Limiting
+The default rate limiter (`sseRateLimiter` in `src/index.ts`) records IP connections in an in-memory Map. 
+* **Limitation**: In a cluster, each container rate-limits independently. An attacker could bypass rate limits by hitting different containers.
+* **Production Fix**: Replace the custom rate limiter with `express-rate-limit` using the `rate-limit-redis` store:
+  ```typescript
+  import rateLimit from 'express-rate-limit';
+  import RedisStore from 'rate-limit-redis';
+  import Redis from 'ioredis';
+
+  const redisClient = new Redis(config.redisUrl);
+
+  const limiter = rateLimit({
+    windowMs: 60 * 1000, // 1 minute
+    max: 10, // Limit each IP to 10 connections per window
+    standardHeaders: true,
+    legacyHeaders: false,
+    store: new RedisStore({
+      sendCommand: (...args: string[]) => redisClient.call(args[0], ...args.slice(1)),
+    }),
+  });
+  ```
+
+---
+
+## 4. Architectural Alternatives: Transitioning to Mature Solutions
+
+The custom Node.js Express/SSE wrapper (`SessionManager`) acts as an in-memory multiplexer, which introduces maintenance overhead (writing custom reapers, session state checks, and sticky load balancer affinity).
+
+For enterprise production setups, you are highly encouraged to move the session, transport, and process management out of the application codebase using one of two mature alternatives:
+
+### Alternative A: Third-Party MCP Gateway (Protocol Bridging)
+Deploy this repository strictly as a stateless `stdio` (stdin/stdout) service (which is already implemented in the code), and wrap it using a production-ready, off-the-shelf gateway:
+* **ContextForge Gateway**: An open-source gateway designed to act as a centralized, multi-tenant proxy that handles credentials, logging, and exposes a unified SSE/REST interface.
+* **Supergateway**: A lightweight CLI/Docker wrapper that exposes any `stdio` server over SSE with zero custom code:
+  ```bash
+  supergateway --port 3000 --stdio "node dist/index.js"
+  ```
+* **Apache APISIX (`mcp-bridge` plugin)**: Bridges standard stdio processes and scales them dynamically as SSE streams, managing routing, rate-limiting, and credentials centrally.
+
+### Alternative B: Infrastructure-Level Multi-Tenancy (Micro-Containers)
+Achieve strict security and process-level isolation by delegating execution lifecycles to cloud infrastructure:
+1. **Stateless Stdio Container**: Package the MCP server strictly in `stdio` mode inside a Docker image.
+2. **On-Demand Orchestration**: Spin up isolated, single-tenant container instances (or micro-VMs) on-demand for active users (using AWS ECS, GCP Cloud Run, or Fly.io).
+3. **Gateway Routing**: An API Gateway authenticates the user, spins up their dedicated container task, and proxies the connection.
+4. **Benefits**: Wipes out custom session reapers, memory leak risks, and session state serialization logic from the application repository.
+