@trayio/cdk-dsl 5.29.0 → 5.31.0

package/README.md

@@ -642,6 +642,601 @@ Finally, for DDL operations, we need to add an extra `type` property in the `ope
 
 This will categorise this operation as a DDL and will exclude it from the list of visible operations of the connector.
 
+## Trigger Connectors
+
+Trigger connectors enable workflows to start automatically when specific events occur in external systems.
+Unlike regular connectors that execute operations on-demand, trigger connectors listen for events and initiate workflow execution when those events are detected.
+
+The CDK supports two types of trigger connectors:
+
+- **Webhook triggers** (also called "normal" triggers): Receive real-time events via HTTP webhooks from external services
+- **Polling triggers**: Periodically check external services for new data or changes
+
+To create a trigger connector, the `connector.json` file must include the `isTrigger` property set to `true`:
+
+```json
+{
+	"name": "my_trigger_connector",
+	"title": "My Trigger Connector",
+	"version": "1.0.0",
+	"isTrigger": true
+}
+```
+
+Setting `isTrigger: true` disables the automatic injection of the raw HTTP operation that regular connectors receive.
+This is necessary because trigger connectors have their own specialized lifecycle operations.
+
+Each trigger operation must specify its `triggerType` in the `operation.json` file:
+
+```json
+{
+	"name": "new_records",
+	"title": "New Records",
+	"description": "Triggers when new records are created",
+	"triggerType": "polling" // or "normal" for webhook triggers
+}
+```
+
+## Webhook Triggers
+
+Webhook triggers receive real-time events from external services via HTTP callbacks.
+They are ideal when the external service supports webhooks and you need immediate notification of events.
+
+A webhook trigger connector implements four lifecycle handlers:
+
+### Create Handler
+
+The create handler registers the webhook with the external service:
+
+```typescript
+export const newOrderCreateHandler =
+	OperationHandlerSetup.configureTriggerCreateHandler<AuthType, CreateInput>(
+		(handler) =>
+			handler.usingHttp((http) =>
+				http
+					.post('https://api.example.com/webhooks')
+					.handleRequest((ctx, input, request) =>
+						request
+							.withBearerToken(ctx.auth!.user.access_token)
+							.withBodyAsJson({
+								url: input.webhookUrl,
+								events: input.eventTypes,
+								secret: input.webhookSecret,
+							})
+					)
+					.handleResponse((ctx, input, response) =>
+						response.parseWithBodyAsJson()
+					)
+			)
+	);
+```
+
+### Destroy Handler
+
+The destroy handler unregisters the webhook when the trigger is disabled:
+
+```typescript
+export const newOrderDestroyHandler =
+	OperationHandlerSetup.configureTriggerDestroyHandler<AuthType, DestroyInput>(
+		(handler) =>
+			handler.usingHttp((http) =>
+				http
+					.delete('https://api.example.com/webhooks/:webhookId')
+					.handleRequest((ctx, input, request) =>
+						request
+							.withBearerToken(ctx.auth!.user.access_token)
+							.addPathParameter('webhookId', input.webhookId)
+							.withoutBody()
+					)
+					.handleResponse((ctx, input, response) =>
+						response.parseWithBodyAsJson()
+					)
+			)
+	);
+```
+
+### Request Handler
+
+The request handler processes incoming webhook HTTP requests and extracts the event data:
+
+```typescript
+export const newOrderRequestHandler =
+	OperationHandlerSetup.configureTriggerRequestHandler<
+		AuthType,
+		RequestInput,
+		OrderEvent
+	>((handler) =>
+		handler.usingComposite(async (ctx, input) => {
+			// input.request is a RegularTriggerOperationHttpRequest (hasLargeBody: false, body: string)
+			// or a LargeTriggerOperationHttpRequest (hasLargeBody: true, body_url instead of body)
+			if (input.request.hasLargeBody) {
+				return OperationHandlerResult.failure(
+					OperationHandlerError.userInputError(
+						'Unexpected large webhook payload'
+					)
+				);
+			}
+
+			// Verify webhook signature - header values are string[]
+			const [signature] = input.request.headers['x-webhook-signature'] ?? [];
+			if (
+				!verifySignature(
+					input.request.body,
+					signature,
+					input.input.webhookSecret
+				)
+			) {
+				return OperationHandlerResult.failure(
+					OperationHandlerError.userInputError('Invalid webhook signature')
+				);
+			}
+
+			// Parse and return the event
+			const order = JSON.parse(input.request.body);
+			return OperationHandlerResult.success({
+				output: {
+					event: order,
+					eventId: order.id,
+					eventTimestamp: order.created_at,
+				},
+			});
+		})
+	);
+```
+
+### Response/Event Handler
+
+The response handler builds the HTTP response to send back to the webhook sender, or the event handler emits the event to the workflow:
+
+```typescript
+// For synchronous response
+export const newOrderResponseHandler =
+	OperationHandlerSetup.configureTriggerResponseHandler<
+		AuthType,
+		ResponseInput,
+		OrderEvent
+	>((handler) =>
+		handler.usingComposite(async (ctx, input) =>
+			OperationHandlerResult.success({
+				status_code: 200,
+				headers: { 'Content-Type': ['application/json'] },
+				body: JSON.stringify({ received: true }),
+			})
+		)
+	);
+
+// Or for asynchronous event emission
+export const newOrderEventHandler =
+	OperationHandlerSetup.configureTriggerEventHandler<
+		AuthType,
+		EventInput,
+		OrderEvent
+	>((handler) =>
+		handler.usingComposite(async (ctx, input) =>
+			OperationHandlerResult.success({
+				id: input.response.eventId,
+				eventType: 'order.created',
+				data: JSON.stringify(input.response.event),
+			})
+		)
+	);
+```
+
+## Polling Triggers
+
+Polling triggers periodically check external services for new data or changes.
+They are useful when the external service doesn't support webhooks or when you need to batch-process changes.
+
+Polling triggers use the Tray Polling Service to manage the polling schedule, cursor state persistence, and failure handling.
+The connector implements four lifecycle handlers that work together with the Polling Service.
+
+### Polling Configuration
+
+Polling triggers accept configuration for the polling interval and failure handling:
+
+```typescript
+import { PollingTriggerInput } from '@trayio/cdk-dsl/connector/operation/PollingTypes';
+
+export type NewRecordsInput = PollingTriggerInput & {
+	// Platform-managed fields (injected automatically):
+	// - pollingInterval: { value: number, unit: 'minutes' | 'hours' }
+	// - maxFailureCount: number (1-150, default 30)
+	// - publicUrl: string (the workflow trigger URL)
+
+	// Your connector-specific fields:
+	objectType: string;
+	includeDeleted?: boolean;
+	fields?: string[];
+};
+```
+
+The platform enforces these validation rules:
+
+- Minimum polling interval: 5 minutes or 1 hour depending on the unit
+- Maximum failure count: between 1 and 150 (default is 30)
+
+### Create Handler
+
+The create handler registers the polling subscription with the Polling Service.
+Unlike webhook triggers, the handler body is a no-op - the runtime intercepts this operation and calls the Polling Service directly:
+
+```typescript
+export const newRecordsCreateHandler =
+	OperationHandlerSetup.configureTriggerPollCreateHandler<
+		AuthType,
+		NewRecordsInput
+	>();
+```
+
+The runtime automatically attaches validation for `pollingInterval` and `maxFailureCount` to ensure they meet the platform requirements.
+
+### Destroy Handler
+
+The destroy handler deregisters the polling subscription.
+Like the create handler, it has a no-op body - the runtime handles the Polling Service communication:
+
+```typescript
+export const newRecordsDestroyHandler =
+	OperationHandlerSetup.configureTriggerPollDestroyHandler<
+		AuthType,
+		NewRecordsInput
+	>();
+```
+
+The runtime always sends `preserveCursor: true` when deregistering, so if the trigger is re-enabled later, it will resume from where it left off rather than starting over.
+
+### Poll Request Handler
+
+The poll request handler is where your actual polling logic lives.
+It receives the current cursor and returns new events along with the next cursor:
+
+```typescript
+import { OperationHandlerResult } from '@trayio/cdk-dsl/connector/operation/OperationHandler';
+import { DynamicObject } from '@trayio/commons/dynamictype/DynamicType';
+
+type RecordEvent = {
+	id: string;
+	type: string;
+	created: string;
+	data: Record<string, unknown>;
+};
+
+export const newRecordsPollHandler =
+	OperationHandlerSetup.configureTriggerPollRequestHandler<
+		AuthType,
+		NewRecordsInput,
+		RecordEvent
+	>((handler) =>
+		handler.usingHttp((http) =>
+			http
+				.get('https://api.example.com/records')
+				.handleRequest((ctx, input, request) => {
+					// Input contains: { input: NewRecordsInput, cursor: string | null }
+					const since =
+						input.cursor || new Date(Date.now() - 86400000).toISOString();
+
+					return request
+						.withBearerToken(ctx.auth!.user.access_token)
+						.addQueryString('type', input.input.objectType)
+						.addQueryString('since', since)
+						.addQueryString('limit', '100')
+						.withoutBody();
+				})
+				.handleResponse((ctx, input, response) =>
+					// handleResponse must return an HttpOperationResponse, so the
+					// result is built inside the parseWithBodyAsJson callback
+					response.parseWithBodyAsJson((body: DynamicObject) => {
+						const records = body.records as unknown as RecordEvent[];
+
+						// Find the latest timestamp to use as next cursor
+						const latestTimestamp =
+							records.length > 0
+								? Math.max(...records.map((r) => new Date(r.created).getTime()))
+								: Date.parse(input.cursor ?? new Date().toISOString());
+
+						return OperationHandlerResult.success({
+							events: records,
+							nextCursor: new Date(latestTimestamp).toISOString(),
+							canPollMore: body.hasMore === true,
+						});
+					})
+				)
+		)
+	);
+```
+
+The output must include:
+
+- `events`: Array of events to emit to the workflow
+- `nextCursor`: The cursor to use for the next poll (persisted by the Polling Service)
+- `canPollMore`: If true, the Polling Service will immediately poll again (useful for pagination)
+
+### Deduplication Handler
+
+The deduplication handler computes a unique ID for each event to prevent duplicate processing:
+
+```typescript
+import { OperationHandlerResult } from '@trayio/cdk-dsl/connector/operation/OperationHandler';
+
+export const newRecordsDedupHandler =
+	OperationHandlerSetup.configureTriggerPollDedupHandler<
+		AuthType,
+		NewRecordsInput,
+		RecordEvent
+	>((handler) =>
+		handler.usingComposite(async (ctx, input) => {
+			// Input contains: { event: RecordEvent }
+			const event = input.event;
+
+			// Generate a stable, unique ID for this event
+			const dedupId = `${event.type}_${event.id}_${event.created}`;
+
+			return OperationHandlerResult.success({
+				triggerDeduplicationId: dedupId,
+			});
+		})
+	);
+```
+
+The platform uses the `triggerDeduplicationId` to drop events that have already been processed, ensuring exactly-once delivery semantics even if the same data is polled multiple times.
+
+### Cursor Management
+
+The Polling Service persists the cursor between poll cycles.
+Your poll handler should:
+
+1. Use the provided cursor to determine where to start fetching data
+2. Return a `nextCursor` that represents the point to resume from next time
+3. Design cursors to be stable and resumable (timestamps, IDs, or offset values work well)
+
+Example cursor strategies:
+
+```typescript
+// Timestamp-based cursor for time-ordered data
+const since = input.cursor || new Date(Date.now() - 3600000).toISOString(); // Default to 1 hour ago
+// ... fetch records created after 'since'
+const nextCursor = latestRecord.created_at;
+
+// ID-based cursor for sequentially-numbered records
+const afterId = input.cursor || '0';
+// ... fetch records with ID > afterId
+const nextCursor = latestRecord.id;
+
+// Offset-based cursor for paginated APIs
+const offset = parseInt(input.cursor || '0');
+// ... fetch records with OFFSET offset LIMIT 100
+const nextCursor = (offset + recordCount).toString();
+```
+
+### Error Handling
+
+The Polling Service tracks consecutive failures and stops polling after `maxFailureCount` failures (default 30).
+When the failure threshold is reached:
+
+1. The Polling Service stops scheduling new poll cycles
+2. A synthetic error event is emitted to the workflow with details about the failure
+3. The error is emitted before the deduplication handler runs
+
+You can use `OperationHandlerError.skipTriggerError()` to indicate non-critical errors that shouldn't count toward the failure limit:
+
+```typescript
+.handleResponse((ctx, input, response) => {
+	if (response.getStatusCode() === 429) {
+		// Rate limit - skip this poll but don't count as failure.
+		// parseWithoutBody keeps the return type an HttpOperationResponse
+		// even though this branch never reads the response body.
+		return response.parseWithoutBody(() =>
+			OperationHandlerResult.failure(
+				OperationHandlerError.skipTriggerError('Rate limited, will retry next interval')
+			)
+		);
+	}
+	// ... normal processing, returned via response.parseWithBodyAsJson(...)
+})
+```
+
+### Complete Polling Trigger Example
+
+Here's a complete example of a polling trigger connector that monitors for new records in an external system:
+
+**Directory structure:**
+
+```
+my-polling-connector/
+├── connector.json
+├── src/
+│   ├── Auth.ts
+│   └── new_records/
+│       ├── operation.json
+│       ├── input.ts
+│       ├── output.ts
+│       ├── handler.ts
+│       └── handler.test.ts
+```
+
+**connector.json:**
+
+```json
+{
+	"name": "my_polling_connector",
+	"title": "My Polling Connector",
+	"version": "1.0.0",
+	"description": "Polls for new records",
+	"isTrigger": true
+}
+```
+
+**src/new_records/operation.json:**
+
+```json
+{
+	"name": "new_records",
+	"title": "New Records",
+	"description": "Triggers when new records are created",
+	"triggerType": "polling"
+}
+```
+
+**src/new_records/input.ts:**
+
+```typescript
+import { PollingTriggerInput } from '@trayio/cdk-dsl/connector/operation/PollingTypes';
+
+export type NewRecordsInput = PollingTriggerInput & {
+	/**
+	 * @title Record Type
+	 * @description The type of records to monitor
+	 */
+	recordType: 'contact' | 'account' | 'opportunity';
+
+	/**
+	 * @title Include Archived
+	 * @description Include archived records in results
+	 * @default false
+	 */
+	includeArchived?: boolean;
+};
+```
+
+**src/new_records/output.ts:**
+
+```typescript
+export type NewRecordsOutput = {
+	id: string;
+	type: string;
+	name: string;
+	createdAt: string;
+	data: Record<string, unknown>;
+};
+```
+
+**src/new_records/handler.ts:**
+
+```typescript
+import { OperationHandlerSetup } from '@trayio/cdk-dsl/connector/operation/OperationHandlerSetup';
+import {
+	OperationHandlerResult,
+	OperationHandlerError,
+} from '@trayio/cdk-dsl/connector/operation/OperationHandler';
+import { DynamicObject } from '@trayio/commons/dynamictype/DynamicType';
+import { AuthType } from '../Auth';
+import { NewRecordsInput } from './input';
+import { NewRecordsOutput } from './output';
+
+// Create handler - registers the polling subscription
+export const newRecordsCreateHandler =
+	OperationHandlerSetup.configureTriggerPollCreateHandler<
+		AuthType,
+		NewRecordsInput
+	>();
+
+// Destroy handler - deregisters the polling subscription
+export const newRecordsDestroyHandler =
+	OperationHandlerSetup.configureTriggerPollDestroyHandler<
+		AuthType,
+		NewRecordsInput
+	>();
+
+// Poll request handler - fetches new records since last cursor
+export const newRecordsPollHandler =
+	OperationHandlerSetup.configureTriggerPollRequestHandler<
+		AuthType,
+		NewRecordsInput,
+		NewRecordsOutput
+	>((handler) =>
+		handler.usingHttp((http) =>
+			http
+				.get('https://api.example.com/v1/records')
+				.handleRequest((ctx, input, request) => {
+					// Default to 24 hours ago if no cursor
+					const since =
+						input.cursor || new Date(Date.now() - 86400000).toISOString();
+
+					return request
+						.withBearerToken(ctx.auth!.user.access_token)
+						.addQueryString('type', input.input.recordType)
+						.addQueryString('created_after', since)
+						.addQueryString(
+							'include_archived',
+							input.input.includeArchived ? 'true' : 'false'
+						)
+						.addQueryString('limit', '100')
+						.addQueryString('sort', 'created_at:asc')
+						.withoutBody();
+				})
+				.handleResponse((ctx, input, response) => {
+					// Handle rate limiting gracefully. parseWithoutBody keeps the
+					// return type an HttpOperationResponse without reading the body.
+					if (response.getStatusCode() === 429) {
+						return response.parseWithoutBody(() =>
+							OperationHandlerResult.failure(
+								OperationHandlerError.skipTriggerError(
+									'API rate limit reached, will retry at next interval'
+								)
+							)
+						);
+					}
+
+					return response.parseWithBodyAsJson((body: DynamicObject) => {
+						const records = body.data as unknown as NewRecordsOutput[];
+
+						// No new records - keep current cursor
+						if (records.length === 0) {
+							return OperationHandlerResult.success({
+								events: [],
+								nextCursor: input.cursor || new Date().toISOString(),
+								canPollMore: false,
+							});
+						}
+
+						// Use the latest record's timestamp as next cursor
+						const latestTimestamp = records[records.length - 1].createdAt;
+
+						return OperationHandlerResult.success({
+							events: records,
+							nextCursor: latestTimestamp,
+							canPollMore: body.has_more === true,
+						});
+					});
+				})
+		)
+	);
+
+// Deduplication handler - generates unique IDs to prevent duplicate events
+export const newRecordsDedupHandler =
+	OperationHandlerSetup.configureTriggerPollDedupHandler<
+		AuthType,
+		NewRecordsInput,
+		NewRecordsOutput
+	>((handler) =>
+		handler.usingComposite(async (ctx, input) => {
+			const record = input.event;
+
+			// Create a stable unique ID for this record
+			const dedupId = `${record.type}:${record.id}:${record.createdAt}`;
+
+			return OperationHandlerResult.success({
+				triggerDeduplicationId: dedupId,
+			});
+		})
+	);
+```
+
+### CLI Limitations
+
+Currently, the `tray-cdk add-operation` command only supports scaffolding `http` and `composite` operation types.
+Trigger and polling trigger operations must be created manually following the patterns shown above.
+
+To create a trigger operation:
+
+1. Create the operation folder manually under `src/`
+2. Add the `operation.json` file with the appropriate `triggerType`
+3. Create `input.ts` and `output.ts` type definitions
+4. Implement the required handlers in `handler.ts`
+5. Add tests in `handler.test.ts`
+
+Future versions of the CLI may add support for scaffolding trigger operations.
+
 ## Dynamic Output Schema
 
 Some operations may need to generate their output schema dynamically at runtime based on the actual API response structure. This is useful when different inputs result in different output structures, or when the API response schema varies by resource type.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@trayio/cdk-dsl",
-  "version": "5.29.0",
+  "version": "5.31.0",
   "description": "A DSL for connector development",
   "exports": {
     "./*": "./dist/*.js"
@@ -14,7 +14,7 @@
     "access": "public"
   },
   "dependencies": {
-    "@trayio/commons": "5.29.0"
+    "@trayio/commons": "5.31.0"
   },
   "typesVersions": {
     "*": {