@tangle-network/agent-integrations 0.24.0 → 0.25.1

package/docs/external-product-integration.md

@@ -0,0 +1,203 @@
+# External Product Integration
+
+This guide is for product teams using `@tangle-network/agent-integrations`
+outside this repository. The package gives you stable contracts and runtime
+helpers; your product supplies persistence, secret storage, UI, and deployment
+policy.
+
+## Mental Model
+
+```txt
+user connects account
+  -> product stores IntegrationConnection + secret refs
+  -> app/agent declares IntegrationManifest
+  -> product creates IntegrationGrant
+  -> sandbox/app receives short-lived capability bundle
+  -> sandbox/app invokes product /integrations/invoke endpoint
+  -> IntegrationHub validates capability, policy, approval, idempotency
+  -> provider backend executes action
+  -> product stores audit event and returns normalized result
+```
+
+The invariant: generated apps and agents call the same Tangle integration tools
+no matter whether a connector is backed by a native adapter, hosted gateway, or
+catalog runtime.
+
+## Product-Owned Pieces
+
+You must provide:
+
+- `IntegrationConnection` storage in your database.
+- `IntegrationGrant` storage mapping user-owned connections to apps, agents, or
+  sandboxes.
+- Approval, audit, healthcheck, workflow, and event stores.
+- `IntegrationSecretStore` backed by your vault or KMS.
+- OAuth/API-key connect UI built from `IntegrationSpec` renderers.
+- Tenant policy for which connectors are enabled.
+- Human approval UX for write/destructive actions.
+- A deployed invocation endpoint such as `/v1/integrations/invoke`.
+
+The package provides interfaces and helpers for all of these. It does not store
+your secrets or run your product UI.
+
+## Setup Flow
+
+1. Build the registry.
+
+```ts
+import { buildDefaultIntegrationRegistry } from '@tangle-network/agent-integrations'
+
+const registry = buildDefaultIntegrationRegistry({
+  tangleCatalogRuntimeExecutable: process.env.TANGLE_CATALOG_RUNTIME === '1',
+})
+```
+
+2. Render setup UI from specs.
+
+```ts
+import {
+  listIntegrationSpecs,
+  renderConsoleSteps,
+  validateCredentialSet,
+} from '@tangle-network/agent-integrations'
+
+const spec = listIntegrationSpecs().find((candidate) => candidate.kind === 'google-calendar')
+const steps = renderConsoleSteps(spec!, { host: 'id.example.com' })
+const validation = validateCredentialSet(spec!, submittedCredentials)
+```
+
+3. Store provider credentials in your vault, then persist an
+   `IntegrationConnection` with secret refs, scopes, owner, connector id, and
+   status.
+
+4. Run `runIntegrationHealthchecks()` after setup and on a schedule.
+
+## Runtime Flow
+
+Create one hub per product runtime.
+
+```ts
+import {
+  createConnectorAdapterProvider,
+  createDefaultIntegrationActionGuard,
+  IntegrationHub,
+  createPlatformIntegrationPolicyPreset,
+} from '@tangle-network/agent-integrations'
+
+const hub = new IntegrationHub({
+  providers: [
+    createConnectorAdapterProvider({
+      adapters,
+      resolveDataSource: (connection) => credentialResolver.resolve(connection),
+    }),
+    // createHttpIntegrationProvider(...) or createTangleCatalogExecutorProvider(...)
+  ],
+  store: connections,
+  capabilitySecret: process.env.INTEGRATION_CAPABILITY_SECRET!,
+  policy: createPlatformIntegrationPolicyPreset(),
+  guard: createDefaultIntegrationActionGuard({
+    audit,
+    idempotency,
+  }),
+})
+```
+
+Your `/v1/integrations/invoke` route should:
+
+1. Parse the invocation envelope.
+2. Validate the capability token.
+3. Resolve the user's connection and credentials.
+4. Run policy and approval checks.
+5. Execute through the matching provider.
+6. Store an audit event.
+7. Return a normalized result or `approval_required`.
+
+## Generated Apps And Sandboxes
+
+Generated apps declare needs through `IntegrationManifest`.
+
+```ts
+const manifest = {
+  id: 'calendar-workout-planner',
+  requirements: [
+    {
+      connectorId: 'google-calendar',
+      mode: 'read',
+      reason: 'Find open workout windows from calendar events.',
+      requiredActions: ['events.list'],
+      scopes: ['calendar.read'],
+    },
+  ],
+}
+```
+
+When a user previews or installs the app:
+
+1. Resolve the manifest against the user's existing connections.
+2. Ask the user to connect missing accounts.
+3. Show consent using `renderConsentSummary()`.
+4. Create grants from the user's connections to the app.
+5. Build a sandbox bundle with short-lived capabilities.
+6. Inject the bridge environment into the sandbox.
+
+Generated app code should use `createTangleIntegrationsClient()` or the product
+equivalent. It should not call Google, Slack, GitHub, or any provider directly
+with user credentials.
+
+## Workflows And Triggers
+
+Use `createIntegrationWorkflowRuntime()` for non-agent automations:
+
+- GitHub issue or pull-request sync.
+- Slack message triggers.
+- Calendar event updates.
+- CRM record changes.
+- Webhook-to-sandbox workflows.
+
+Inbound provider webhooks should go through `receiveIntegrationWebhook()` for
+signature verification, provider-event dedupe, and normalized event dispatch.
+
+## Long-Tail Connectors
+
+The registry distinguishes connector contracts from executable backend state.
+Products can expose long-tail connectors only when they have configured a
+backend:
+
+- native adapter
+- hosted integration gateway
+- Tangle catalog runtime
+- product-specific provider
+
+Use `buildDefaultIntegrationRegistry({ tangleCatalogRuntimeExecutable: true })`
+only after the catalog runtime is deployed and audited with
+`auditTangleCatalogRuntimePackages()`.
+
+## Security Requirements
+
+- Never pass provider refresh tokens or API keys into a sandbox.
+- Use short-lived capability tokens scoped to connector, action, subject, and
+  connection.
+- Require approval for writes by default.
+- Deny destructive actions unless the product explicitly enables them.
+- Use idempotency keys for state-changing actions.
+- Store audit events for connect, grant, invoke, approve, revoke, rotate, and
+  webhook flows.
+- Redact provider credentials from logs, traces, errors, and generated app
+  payloads.
+
+## Launch Checklist
+
+- [ ] Connection, grant, approval, audit, healthcheck, workflow, and event stores
+      are backed by production persistence.
+- [ ] Secret refs resolve through vault/KMS and never serialize raw credentials.
+- [ ] OAuth/API-key setup UI renders from `IntegrationSpec`.
+- [ ] Connect, revoke, rotate, approve, and audit-log screens exist.
+- [ ] Generated app manifests feed into `resolveManifest()` and
+      `createGrants()`.
+- [ ] Sandbox launches receive `buildIntegrationBridgeEnvironment()` output.
+- [ ] Sandbox invocations route through your product integration endpoint.
+- [ ] Writes require approval and idempotency.
+- [ ] Webhooks verify signatures and dedupe provider event ids.
+- [ ] Healthchecks run after setup and on a schedule.
+- [ ] Browser E2E covers connect, consent, preview, invoke, approval, revoke,
+      and failure recovery.

package/docs/generated-integration-coverage-checklist.md

@@ -1,6 +1,6 @@
 # Agent Integrations Coverage Checklist
 
-Generated from `listIntegrationCoverageSpecs()`. Catalog presence means the product can plan/request/connect the integration; executable first-party adapters are promoted separately behind the same provider contract.
+Generated from `listIntegrationCoverageSpecs()`. Catalog presence means the product can plan/request/connect the integration; native adapters and runtime backends execute behind the same provider contract.
 
 ## Summary
 
@@ -135,8 +135,8 @@ Generated from `listIntegrationCoverageSpecs()`. Catalog presence means the prod
 - [ ] tier_1 / database / Qdrant (qdrant) - vector, database, ai
 - [ ] tier_1 / workflow / Zapier (zapier) - automation, workflow
 - [ ] tier_1 / workflow / Make (make) - automation, workflow
-- [ ] tier_1 / workflow / Hosted Integration Gateway (hosted-integration-gateway) - integration-platform, oauth
-- [ ] tier_1 / workflow / Workflow Runtime Gateway (workflow-runtime-gateway) - integration-platform, workflow
+- [ ] tier_1 / workflow / Nango (nango) - integration-platform, oauth
+- [ ] tier_1 / workflow / Pipedream (pipedream) - integration-platform, workflow
 - [ ] tier_1 / workflow / Open Automation Catalog (open-automation-catalog) - automation, workflow, open-source
 - [ ] tier_0 / webhook / Generic Webhook (webhook) - webhook, http, events
 - [ ] tier_0 / workflow / HTTP Request (http) - http, api, webhook

package/docs/integration-coverage-checklist.md

@@ -6,7 +6,7 @@ Goal: cover the integrations that make agents useful for 99% of practical produc
 
 - Use `buildIntegrationCoverageConnectors()` for broad planning/catalog coverage.
 - Use hosted gateways, imported connector catalogs, and custom HTTP providers for long-tail OAuth/API coverage.
-- Promote high-volume, sensitive, or moat-critical connectors into first-party `ConnectorAdapter`s.
+- Implement high-volume, sensitive, or moat-critical connectors as native `ConnectorAdapter`s.
 - Do not claim a connector is executable until it has auth, scope, action, error, rate-limit, idempotency, and approval-path tests.
 - Keep every connector behind `IntegrationHub`, capability tokens, policy checks, and sandbox invocation envelopes.
 
@@ -16,7 +16,7 @@ Goal: cover the integrations that make agents useful for 99% of practical produc
 - [x] Central sandbox invocation envelope validation.
 - [x] Coverage catalog for 100+ high-value integrations.
 - [x] Builder API consumes the coverage catalog for app planning.
-- [x] Declarative REST adapter factory for fast first-party promotion of REST-shaped APIs.
+- [x] Declarative REST adapter factory for fast native implementation of REST-shaped APIs.
 - [x] Provider gateway catalog adapter can normalize 500+ external catalog connectors.
 - [x] Canonical registry dedupes overlapping catalogs, aliases, support tiers, and conflict diagnostics.
 - [ ] Generated sandbox apps can request missing connections from the catalog.
@@ -25,7 +25,7 @@ Goal: cover the integrations that make agents useful for 99% of practical produc
 - [ ] Tier 0 write actions have idempotency and approval tests.
 - [x] Provider gateway adapters can import/sync catalog metadata from external registries.
 - [x] Adapter execution triage is documented in [`adapter-triage.md`](./adapter-triage.md).
-- [x] Tangle Integrations Catalog entries can be promoted to executable actions through `createTangleCatalogExecutorProvider`.
+- [x] Tangle Integrations Catalog contracts can execute through `createTangleCatalogExecutorProvider`.
 - [x] Tangle Integrations Catalog executable actions can dispatch through a signed HTTP runtime executor protocol.
 
 ## Tier 0 First-Party Promotion Queue

package/docs/integration-execution-audit.md

@@ -17,8 +17,8 @@ This audit separates product contracts from implementation backends:
 | Catalog connectors with runtime package names | 669 |
 | Catalog actions | 3790 |
 | Catalog triggers | 998 |
-| Catalog triggers with verified upstream names in this repo | 998 |
-| Catalog actions with verified upstream action names in this repo | 3790 |
+| Catalog triggers with upstream names | 998 |
+| Catalog actions with upstream action names | 3790 |
 | Catalog connectors with auth field metadata | 648 |
 | Custom-auth connectors with auth field metadata | 11 |
 | Runtime package dependencies declared by this package | 0 |
@@ -33,6 +33,7 @@ This audit separates product contracts from implementation backends:
 | Native adapter backends | 10 |
 | Native adapter surfaces shipped | 16 |
 | Package-runtime backends | 659 |
+| Runtime manifest dependencies | 670 |
 | Tangle catalog connectors exposable behind runtime | 669 |
 | Tangle catalog actions exposable behind runtime | 3970 |
 
@@ -108,7 +109,9 @@ Executable setup specs:
 | Connector discovery/catalog search | Done | 669 catalog connectors, 3790 actions, 998 triggers normalized into Tangle catalog shapes. |
 | Native adapter execution | Done for listed native backends | 16 reviewed native adapter surfaces ship from this package; 10 overlap the 669 catalog contracts. |
 | OAuth/API-key setup metadata | Partial | 142 setup specs exist; 14 are executable setup specs and 128 are catalog/setup-only. |
-| Package-runtime action execution | Wiring done; runtime deployment/smoke pending | 659 contracts use package-runtime backends with package names and 3790 mapped upstream action names. |
+| Package-runtime action execution | Wiring done; runtime deployment/smoke pending | 659 contracts use package-runtime backends with package names and 3790 catalog upstream action names. |
+| Runtime dependency manifest | Done | `buildTangleCatalogRuntimePackageManifest()` emits 670 dependencies for a complete package-runtime worker install. |
+| Runtime package coverage audit | Done | `auditTangleCatalogRuntimePackages()` and `tangle-catalog-runtime --audit-packages` verify installed packages, piece exports, exact action mappings, and trigger surfaces in a deployed worker. |
 | Long-tail credential mapping | Mostly mapped | 648 connectors have auth field metadata. 0 custom-auth connectors still need exact manual auth fields. |
 | Trigger provider flow | Done structurally | 998 triggers are cataloged, 998 have upstream names, and catalog providers can route subscribe/unsubscribe/normalize hooks. Runtime services still need package-specific trigger hosting. |
 | Sandbox/app invocation envelope | Done | The library has capability bundles, invocation envelopes, policy checks, guard hooks, signed catalog runtime HTTP calls, and generated-app client helpers. |
@@ -119,7 +122,7 @@ Executable setup specs:
 | Bucket | Count | What it means |
 | --- | ---: | --- |
 | Package-runtime contracts needing deployed runtime smoke verification | 659 | Connector has a Tangle contract and package backend; deployed runtime still needs package-load/live-smoke proof. |
-| Catalog connectors with zero verified action mappings | 0 | We normalized action labels, but have not checked the exact runtime action export names into the catalog. |
+| Catalog connectors with zero upstream action names | 0 | These entries need catalog action-name mapping before exact package-runtime invocation can work. |
 | Custom-auth catalog connectors needing manual credential-field mapping | 0 | These are still custom auth and no field names were extracted from source. |
 | Catalog connectors with triggers needing runtime-service hosting | 288 | Trigger metadata and provider hooks exist; runtime services still need package-specific webhook/polling hosting. |
 
@@ -166,17 +169,15 @@ Examples needing deployed runtime smoke verification:
 - `assembled` -> `@activepieces/piece-assembled`
 - `assemblyai` -> `@activepieces/piece-assemblyai`
 
-Examples needing manual custom auth mapping:
+Manual custom auth mapping gap: none.
 
-
-
-## What Is Not Done
+## Completion Claims And Remaining Proof Gates
 
 1. **Tangle first-class connector contracts are complete.**
    All 669 catalog entries have Tangle-owned contracts. 10 use native adapter backends; 659 use package-runtime backends.
 
-2. **Action-name mapping is complete for cataloged actions.**
-   Done for cataloged actions: the catalog currently has 3790 actions and 3790 verified upstream action-name mappings in the checked-in catalog. The runtime executor uses those names automatically and still accepts explicit `actionAliases` for overrides.
+2. **Action-name mapping exists for cataloged actions.**
+   Done for cataloged actions: the catalog currently has 3790 actions and 3790 upstream action-name mappings in the checked-in catalog. The runtime executor uses those names automatically and still accepts explicit `actionAliases` for overrides. Deployed smoke verification proves those names against the installed packages.
 
 3. **Credential field mapping is complete for catalog auth setup.**
    Auth shapes are api_key: 519, oauth2: 118, none: 21, custom: 11. The catalog now includes auth field metadata for all 648 connectors that require credentials. 0 custom-auth connectors need manual auth-field mapping.
@@ -193,14 +194,11 @@ Examples needing manual custom auth mapping:
 - It is accurate to say: **all product code can use one IntegrationHub/tool contract across native and package-runtime backends.**
 - It is accurate to say: **deployed runtime smoke verification is the remaining proof step for package-runtime connectors.**
 
-## Next Gap To Close
-
-Build a runtime coverage generator that installs/imports each package in isolation, extracts real action names, writes `actionAliases`, and emits a pass/fail matrix per connector:
+## Runtime Proof Gate
 
-- package loads
-- package installed in the runtime service
-- package load verified
-- normalized action maps to real action
-- auth shape identified or marked as manual
-- dry-run invocation possible
-- live smoke credential available
+Run `tangle-catalog-runtime --audit-packages` inside the deployed runtime image
+after installing the manifest from `--print-package-json` or
+`--print-pnpm-add`. That produces the concrete package-load/action-map/trigger
+surface matrix for the exact runtime image products will call. Live provider
+smoke tests still require real OAuth/API-key credentials from the product
+environment.