@walkeros/server-source-gcp 4.0.0 → 4.0.1-next-1778183328892

package/README.md

@@ -1,7 +1,8 @@
 # @walkeros/server-source-gcp
 
-Google Cloud Platform server sources for walkerOS - lightweight, single-purpose
-runtime adapters for GCP services.
+Google Cloud Platform server sources for walkerOS, lightweight runtime adapters
+for GCP services. Ships two surfaces: the Cloud Functions HTTP handler and
+Pub/Sub (pull subscriber and push webhook).
 
 ## Installation
 
@@ -262,3 +263,216 @@ This source follows the walkerOS patterns:
 
 The source's `push` function accepts HTTP requests, transforms them into walker
 events, and forwards them to the collector for processing by destinations.
+
+---
+
+## Pub/Sub source
+
+Subscribes to a Google Cloud Pub/Sub topic and forwards each message to the
+walkerOS collector. Ships in two delivery models:
+
+- `sourcePubSubPull`: long-running streaming pull subscriber. Use in containers,
+  VMs, or any process that stays alive. Highest throughput.
+- `sourcePubSubPush`: HTTP push webhook handler. Use in serverless deployments
+  (Cloud Run, Cloud Functions, Lambda) where there is no long-running process to
+  keep a streaming subscriber alive.
+
+### Installation
+
+Pub/Sub support is part of `@walkeros/server-source-gcp`. The Pub/Sub SDK is
+declared as a runtime dependency:
+
+```bash
+npm install @walkeros/server-source-gcp
+```
+
+### Quickstart, pull subscriber
+
+```typescript
+import { sourcePubSubPull } from '@walkeros/server-source-gcp';
+import { startFlow } from '@walkeros/collector';
+
+await startFlow({
+  sources: {
+    pubsub: {
+      code: sourcePubSubPull,
+      config: {
+        settings: {
+          projectId: 'my-gcp-project',
+          subscription: 'events-sub',
+          // Optional: tighten flow control beyond SDK defaults
+          flowControl: { maxMessages: 100, maxBytes: 10 * 1024 * 1024 },
+        },
+      },
+    },
+  },
+  destinations: {
+    // Your destinations
+  },
+});
+```
+
+The pull source is event-driven: `init()` opens the streaming subscription and
+forwards each delivered message to the collector via `env.push`. The source's
+`push` is a deliberate no-op stub (the framework never calls it). `destroy()`
+closes the subscriber gracefully, honoring `shutdownTimeoutMs` (default 30000).
+
+### Quickstart, push webhook
+
+```typescript
+import express from 'express';
+import { sourcePubSubPush } from '@walkeros/server-source-gcp';
+import { startFlow } from '@walkeros/collector';
+
+const { sources } = await startFlow({
+  sources: {
+    pubsub: {
+      code: sourcePubSubPush,
+      config: {
+        settings: { decoder: 'json' },
+      },
+    },
+  },
+  destinations: {
+    // Your destinations
+  },
+});
+
+const app = express();
+app.use(express.json());
+app.post('/pubsub-push', sources.pubsub.push);
+app.listen(8080);
+```
+
+Pub/Sub POSTs each message envelope to `/pubsub-push`. The source decodes the
+envelope, base64-decodes the data field, runs the configured decoder, and
+forwards to the collector. Returns 200 on success, 400 on malformed envelope,
+401 when OIDC verification fails, 500 on push failure (Pub/Sub will retry per
+the subscription's retry policy).
+
+### Authentication
+
+Three modes, in precedence order:
+
+1. **Pre-configured client** (`settings.client`). When you bring your own
+   `PubSub` instance with custom auth, this bypasses credentials resolution.
+2. **Service account JSON** (`settings.credentials`). Pass an object
+   `{ client_email, private_key }` or a JSON string the source will parse.
+3. **Application Default Credentials (ADC)**. The default. Works on GCP compute
+   (Cloud Run, GCE, GKE, Cloud Functions) and locally via
+   `gcloud auth application-default login`.
+
+```typescript
+// Service account (object form)
+config: {
+  settings: {
+    projectId: 'my-project',
+    subscription: 'events-sub',
+    credentials: {
+      client_email: 'sa@my-project.iam.gserviceaccount.com',
+      private_key: '-----BEGIN PRIVATE KEY-----\n...',
+    },
+  },
+}
+```
+
+### Setup, idempotent subscription provisioning
+
+`walkeros setup source.<id>` provisions the subscription. Idempotent: safe to
+re-run; `ALREADY_EXISTS` is non-fatal. Drift detection emits `setup.drift`
+warnings without auto-mutating.
+
+```typescript
+config: {
+  setup: {
+    createTopic: true,           // Create the topic if missing (default: false)
+    ackDeadlineSeconds: 60,
+    deadLetterPolicy: {
+      deadLetterTopic: 'events-dlq',
+      maxDeliveryAttempts: 5,
+      createDeadLetterTopic: true,
+    },
+    retryPolicy: {
+      minimumBackoff: { seconds: 10 },
+      maximumBackoff: { seconds: 600 },
+    },
+  },
+  settings: {
+    projectId: 'my-project',
+    subscription: 'events-sub',
+    topic: 'events',
+  },
+}
+```
+
+Auto-created topics use the EU multi-region storage policy (`eu-west1`,
+`eu-west3`, `eu-west4`). Operators in non-EU regions should override
+`messageStoragePolicy` per organisation policy.
+
+The destination side (`@walkeros/server-destination-gcp`) provisions the topic
+itself; you can use either side's setup for the topic. Subscription provisioning
+is owned exclusively by this source.
+
+### Decoders
+
+| Decoder          | Behavior                                                                   |
+| ---------------- | -------------------------------------------------------------------------- |
+| `json` (default) | `JSON.parse(data.toString('utf8'))`. Throws DecoderError on parse failure. |
+| `text`           | `data.toString('utf8')`. The text becomes the event payload.               |
+| `raw`            | The raw `Buffer` is forwarded as-is.                                       |
+
+### Backpressure and flow control
+
+The pull subscriber has built-in flow control:
+
+- `maxMessages` (default 100): max concurrent in-flight messages.
+- `maxBytes` (default 10 MB): max concurrent in-flight bytes.
+
+The subscriber automatically slows down when the collector pushes back. Tune via
+`settings.flowControl`.
+
+### OIDC verification (push mode)
+
+Off by default. Enable per push subscription when your endpoint is publicly
+reachable:
+
+```typescript
+config: {
+  settings: {
+    verifyOidc: true,
+    audience: 'https://my-service.example/pubsub-push',
+  },
+}
+```
+
+The source verifies the `Authorization: Bearer <jwt>` header against GCP public
+keys via `google-auth-library`. Misconfigured OIDC silently rejects all
+messages, so leave it off when running behind a private network and rely on
+network isolation instead.
+
+### Emulator
+
+Both pull and push honor `PUBSUB_EMULATOR_HOST`. For explicit configuration, set
+`settings.apiEndpoint`:
+
+```typescript
+config: {
+  settings: {
+    projectId: 'my-project',
+    subscription: 'events-sub',
+    apiEndpoint: 'localhost:8085',
+  },
+}
+```
+
+### Troubleshooting
+
+- **Subscription not found / unauthorized.** The pull source logs a canonical
+  hint:
+  `Pub/Sub subscription "X" not found or unauthorized in project "Y". Run "walkeros setup source.<id>" to create it.`
+  Run setup or grant `roles/pubsub.subscriber` to the runtime service account.
+- **JSON decode failures.** Default `onPushError: 'nack'` redelivers; set
+  `onPushError: 'ack'` to drop instead. Switch to `decoder: 'raw'` if your
+  payloads are binary (Avro, Protobuf, etc.) and decode in a transformer.
+- **Push 401 with OIDC enabled.** Verify the audience matches your endpoint URL
+  exactly. Pub/Sub signs tokens with the configured audience.

package/dist/dev.d.mts

@@ -1,6 +1,7 @@
 import { Source, Flow, Trigger } from '@walkeros/core';
 import * as _walkeros_core_dev from '@walkeros/core/dev';
 import { z } from '@walkeros/core/dev';
+import { PubSub } from '@google-cloud/pubsub';
 
 /**
  * HTTP methods enum
@@ -46,7 +47,7 @@ type CorsOptions = z.infer<typeof CorsOptionsSchema>;
 /**
  * GCP CloudFunction source settings schema
  */
-declare const SettingsSchema: z.ZodObject<{
+declare const SettingsSchema$2: z.ZodObject<{
     cors: z.ZodOptional<z.ZodUnion<readonly [z.ZodBoolean, z.ZodObject<{
         origin: z.ZodOptional<z.ZodUnion<readonly [z.ZodString, z.ZodArray<z.ZodString>, z.ZodLiteral<"*">]>>;
         methods: z.ZodOptional<z.ZodArray<z.ZodEnum<{
@@ -64,19 +65,16 @@ declare const SettingsSchema: z.ZodObject<{
     }, z.core.$strip>]>>;
     timeout: z.ZodOptional<z.ZodNumber>;
 }, z.core.$strip>;
-type Settings = z.infer<typeof SettingsSchema>;
+type Settings$2 = z.infer<typeof SettingsSchema$2>;
 
-declare const settings: _walkeros_core_dev.JSONSchema;
+declare const settings$2: _walkeros_core_dev.JSONSchema;
 
-type index$1_CorsOptions = CorsOptions;
-declare const index$1_CorsOptionsSchema: typeof CorsOptionsSchema;
-declare const index$1_CorsOrigin: typeof CorsOrigin;
-declare const index$1_HttpMethod: typeof HttpMethod;
-type index$1_Settings = Settings;
-declare const index$1_SettingsSchema: typeof SettingsSchema;
-declare const index$1_settings: typeof settings;
-declare namespace index$1 {
-  export { type index$1_CorsOptions as CorsOptions, index$1_CorsOptionsSchema as CorsOptionsSchema, index$1_CorsOrigin as CorsOrigin, index$1_HttpMethod as HttpMethod, type index$1_Settings as Settings, index$1_SettingsSchema as SettingsSchema, index$1_settings as settings };
+type index$5_CorsOptions = CorsOptions;
+declare const index$5_CorsOptionsSchema: typeof CorsOptionsSchema;
+declare const index$5_CorsOrigin: typeof CorsOrigin;
+declare const index$5_HttpMethod: typeof HttpMethod;
+declare namespace index$5 {
+  export { type index$5_CorsOptions as CorsOptions, index$5_CorsOptionsSchema as CorsOptionsSchema, index$5_CorsOrigin as CorsOrigin, index$5_HttpMethod as HttpMethod, type Settings$2 as Settings, SettingsSchema$2 as SettingsSchema, settings$2 as settings };
 }
 
 declare module '@walkeros/core' {
@@ -87,21 +85,21 @@ declare module '@walkeros/core' {
         };
     }
 }
-interface Request {
+interface Request$1 {
     method: string;
     body?: unknown;
     headers: Record<string, string | string[]>;
     get(name: string): string | undefined;
 }
-interface Response {
-    status(code: number): Response;
-    json(body: unknown): Response;
-    send(body?: unknown): Response;
-    set(key: string, value: string): Response;
+interface Response$1 {
+    status(code: number): Response$1;
+    json(body: unknown): Response$1;
+    send(body?: unknown): Response$1;
+    set(key: string, value: string): Response$1;
 }
-interface Env extends Source.Env {
-    req?: Request;
-    res?: Response;
+interface Env$2 extends Source.Env {
+    req?: Request$1;
+    res?: Response$1;
 }
 
 /**
@@ -110,28 +108,27 @@ interface Env extends Source.Env {
  * Use this for testing HTTP event ingestion and request/response handling
  * without requiring a real GCP Cloud Function environment.
  */
-declare const push: Env;
+declare const push$2: Env$2;
 
-declare const env_push: typeof push;
-declare namespace env {
-  export { env_push as push };
+declare namespace env$2 {
+  export { push$2 as push };
 }
 
 declare const postEvent: Flow.StepExample;
 declare const orderEvent: Flow.StepExample;
 
-declare const step_orderEvent: typeof orderEvent;
-declare const step_postEvent: typeof postEvent;
-declare namespace step {
-  export { step_orderEvent as orderEvent, step_postEvent as postEvent };
+declare const step$2_orderEvent: typeof orderEvent;
+declare const step$2_postEvent: typeof postEvent;
+declare namespace step$2 {
+  export { step$2_orderEvent as orderEvent, step$2_postEvent as postEvent };
 }
 
-interface Content {
+interface Content$2 {
     method: string;
     body: Record<string, unknown>;
     headers?: Record<string, string>;
 }
-interface Result {
+interface Result$2 {
     status: number;
     body: unknown;
 }
@@ -143,13 +140,273 @@ interface Result {
  * This is the realistic approach for GCP Cloud Functions - the Functions Framework
  * synthesizes these objects in production too.
  */
+declare const createTrigger$2: Trigger.CreateFn<Content$2, Result$2>;
+
+declare namespace index$4 {
+  export { createTrigger$2 as createTrigger, env$2 as env, step$2 as step };
+}
+
+/**
+ * GCP Pub/Sub pull source settings schema.
+ *
+ * Required: projectId, subscription. All other fields optional with sensible
+ * defaults applied at runtime by `getConfig`.
+ */
+declare const SettingsSchema$1: z.ZodObject<{
+    client: z.ZodOptional<z.ZodAny>;
+    projectId: z.ZodString;
+    subscription: z.ZodString;
+    topic: z.ZodOptional<z.ZodString>;
+    credentials: z.ZodOptional<z.ZodAny>;
+    apiEndpoint: z.ZodOptional<z.ZodString>;
+    decoder: z.ZodOptional<z.ZodEnum<{
+        json: "json";
+        text: "text";
+        raw: "raw";
+    }>>;
+    flowControl: z.ZodOptional<z.ZodObject<{
+        maxMessages: z.ZodOptional<z.ZodNumber>;
+        maxBytes: z.ZodOptional<z.ZodNumber>;
+    }, z.core.$strip>>;
+    ackDeadline: z.ZodOptional<z.ZodNumber>;
+    shutdownTimeoutMs: z.ZodOptional<z.ZodNumber>;
+    onPushError: z.ZodOptional<z.ZodEnum<{
+        nack: "nack";
+        ack: "ack";
+    }>>;
+}, z.core.$strip>;
+type Settings$1 = z.infer<typeof SettingsSchema$1>;
+
+/**
+ * GCP Pub/Sub pull source setup schema.
+ *
+ * Provisioning options for `walkeros setup source.<id>`. Idempotent
+ * subscription creation, plus optional topic and dead-letter topic
+ * auto-create. Triggered only by the explicit CLI command.
+ */
+declare const SetupSchema: z.ZodObject<{
+    createTopic: z.ZodOptional<z.ZodBoolean>;
+    ackDeadlineSeconds: z.ZodOptional<z.ZodNumber>;
+    messageRetentionDuration: z.ZodOptional<z.ZodObject<{
+        seconds: z.ZodNumber;
+    }, z.core.$strip>>;
+    filter: z.ZodOptional<z.ZodString>;
+    deadLetterPolicy: z.ZodOptional<z.ZodObject<{
+        deadLetterTopic: z.ZodString;
+        maxDeliveryAttempts: z.ZodNumber;
+        createDeadLetterTopic: z.ZodOptional<z.ZodBoolean>;
+    }, z.core.$strip>>;
+    retryPolicy: z.ZodOptional<z.ZodObject<{
+        minimumBackoff: z.ZodObject<{
+            seconds: z.ZodNumber;
+        }, z.core.$strip>;
+        maximumBackoff: z.ZodObject<{
+            seconds: z.ZodNumber;
+        }, z.core.$strip>;
+    }, z.core.$strip>>;
+    enableMessageOrdering: z.ZodOptional<z.ZodBoolean>;
+    labels: z.ZodOptional<z.ZodRecord<z.ZodString, z.ZodString>>;
+    expirationPolicy: z.ZodOptional<z.ZodObject<{
+        ttl: z.ZodOptional<z.ZodNullable<z.ZodObject<{
+            seconds: z.ZodNumber;
+        }, z.core.$strip>>>;
+    }, z.core.$strip>>;
+}, z.core.$strip>;
+type Setup = z.infer<typeof SetupSchema>;
+
+declare const settings$1: _walkeros_core_dev.JSONSchema;
+declare const setup: _walkeros_core_dev.JSONSchema;
+
+type index$3_Setup = Setup;
+declare const index$3_SetupSchema: typeof SetupSchema;
+declare const index$3_setup: typeof setup;
+declare namespace index$3 {
+  export { type Settings$1 as Settings, SettingsSchema$1 as SettingsSchema, type index$3_Setup as Setup, index$3_SetupSchema as SetupSchema, settings$1 as settings, index$3_setup as setup };
+}
+
+declare module '@walkeros/core' {
+    interface SourceMap {
+        'pubsub-pull': {
+            type: 'pubsub-pull';
+            platform: 'server';
+        };
+    }
+}
+interface Env$1 extends Source.Env {
+    PubSub?: typeof PubSub;
+}
+
+/**
+ * Standard mock environment for the pull source.
+ *
+ * `PubSub` is intentionally absent: the canonical pattern is module-level
+ * `jest.mock('@google-cloud/pubsub')`, not env-injection.
+ */
+declare const push$1: Env$1;
+declare const simulation$1: string[];
+
+declare namespace env$1 {
+  export { push$1 as push, simulation$1 as simulation };
+}
+
+declare const pageView: Flow.StepExample;
+declare const orderComplete: Flow.StepExample;
+declare const decoderText: Flow.StepExample;
+declare const malformedJson: Flow.StepExample;
+declare const malformedJsonAck: Flow.StepExample;
+
+declare const step$1_decoderText: typeof decoderText;
+declare const step$1_malformedJson: typeof malformedJson;
+declare const step$1_malformedJsonAck: typeof malformedJsonAck;
+declare const step$1_orderComplete: typeof orderComplete;
+declare const step$1_pageView: typeof pageView;
+declare namespace step$1 {
+  export { step$1_decoderText as decoderText, step$1_malformedJson as malformedJson, step$1_malformedJsonAck as malformedJsonAck, step$1_orderComplete as orderComplete, step$1_pageView as pageView };
+}
+
+/**
+ * Content shape for the pull-source trigger.
+ *
+ * The trigger synthesizes a message from this partial input and dispatches
+ * it through the source's `push()` (the same pipeline the SDK subscriber
+ * callback uses). `dataString` is the user-friendly way to provide the
+ * message body; the trigger encodes it to a Buffer.
+ */
+interface Content$1 {
+    id: string;
+    dataString: string;
+    attributes?: Record<string, string>;
+    orderingKey?: string;
+}
+/**
+ * Result shape for the pull-source trigger.
+ *
+ * Returns the recorded ack/nack as `[method, id]` entries, mirroring the
+ * destination's `[method, ...args]` recording shape used elsewhere.
+ */
+type Result$1 = Array<[string, ...unknown[]]>;
+/**
+ * Pub/Sub pull source createTrigger.
+ *
+ * Boots the collector via startFlow, finds the registered pubsub-pull source,
+ * and invokes its `push()` with a synthesized message. The source dispatches
+ * the synthetic message through the same handler the SDK subscriber callback
+ * uses, exercising the full decode / forward / ack-nack pipeline without
+ * touching real Pub/Sub.
+ *
+ * Matches the convention used by other source triggers (express, lambda,
+ * cloudfunction): find the source by type from the collector and call its
+ * public `push()` interface.
+ */
+declare const createTrigger$1: Trigger.CreateFn<Content$1, Result$1>;
+
+declare namespace index$2 {
+  export { type Content$1 as Content, type Result$1 as Result, createTrigger$1 as createTrigger, env$1 as env, step$1 as step };
+}
+
+/**
+ * GCP Pub/Sub push source settings schema.
+ *
+ * No required fields. The push source is a request-handler; configuration
+ * tunables live here, but operators wire `source.push` into their own
+ * express/lambda runtime.
+ */
+declare const SettingsSchema: z.ZodObject<{
+    projectId: z.ZodOptional<z.ZodString>;
+    decoder: z.ZodOptional<z.ZodEnum<{
+        json: "json";
+        text: "text";
+        raw: "raw";
+    }>>;
+    verifyOidc: z.ZodOptional<z.ZodBoolean>;
+    audience: z.ZodOptional<z.ZodString>;
+}, z.core.$strip>;
+type Settings = z.infer<typeof SettingsSchema>;
+
+declare const settings: _walkeros_core_dev.JSONSchema;
+
+type index$1_Settings = Settings;
+declare const index$1_SettingsSchema: typeof SettingsSchema;
+declare const index$1_settings: typeof settings;
+declare namespace index$1 {
+  export { type index$1_Settings as Settings, index$1_SettingsSchema as SettingsSchema, index$1_settings as settings };
+}
+
+declare module '@walkeros/core' {
+    interface SourceMap {
+        'pubsub-push': {
+            type: 'pubsub-push';
+            platform: 'server';
+        };
+    }
+}
+/**
+ * Minimal request shape for the push handler. Mirrors the cloudfunction
+ * source so operators can wire `sourcePubSubPush` into express, lambda,
+ * or any framework that delivers a similar contract.
+ */
+interface Request {
+    method: string;
+    body?: unknown;
+    headers: Record<string, string | string[]>;
+    get(name: string): string | undefined;
+}
+interface Response {
+    status(code: number): Response;
+    json(body: unknown): Response;
+    send(body?: unknown): Response;
+    set(key: string, value: string): Response;
+}
+interface Env extends Source.Env {
+    req?: Request;
+    res?: Response;
+    /** Optional override for OIDC verification. Tests can inject a stub. */
+    verifyOidcToken?: (token: string, audience: string) => Promise<{
+        sub?: string;
+        email?: string;
+    }>;
+}
+
+declare const push: Env;
+declare const simulation: string[];
+
+declare const env_push: typeof push;
+declare const env_simulation: typeof simulation;
+declare namespace env {
+  export { env_push as push, env_simulation as simulation };
+}
+
+declare const validEnvelope: Flow.StepExample;
+declare const orderEnvelope: Flow.StepExample;
+declare const malformedEnvelope: Flow.StepExample;
+
+declare const step_malformedEnvelope: typeof malformedEnvelope;
+declare const step_orderEnvelope: typeof orderEnvelope;
+declare const step_validEnvelope: typeof validEnvelope;
+declare namespace step {
+  export { step_malformedEnvelope as malformedEnvelope, step_orderEnvelope as orderEnvelope, step_validEnvelope as validEnvelope };
+}
+
+/** Content for the push trigger: a Pub/Sub envelope or any POST body. */
+type Content = Record<string, unknown>;
+/** Result: the recorded HTTP response (`status` + `json` body). */
+type Result = Array<[string, ...unknown[]]>;
+/**
+ * Pub/Sub push source createTrigger.
+ *
+ * Boots the collector via startFlow, builds a fake Request/Response pair
+ * from the trigger Content, and invokes the source's HTTP handler. Returns
+ * the recorded response as a Result entry.
+ */
 declare const createTrigger: Trigger.CreateFn<Content, Result>;
 
+type index_Content = Content;
+type index_Result = Result;
 declare const index_createTrigger: typeof createTrigger;
 declare const index_env: typeof env;
 declare const index_step: typeof step;
 declare namespace index {
-  export { index_createTrigger as createTrigger, index_env as env, index_step as step };
+  export { type index_Content as Content, type index_Result as Result, index_createTrigger as createTrigger, index_env as env, index_step as step };
 }
 
-export { index as examples, index$1 as schemas };
+export { index$4 as examples, index$2 as pubsubPullExamples, index$3 as pubsubPullSchemas, index as pubsubPushExamples, index$1 as pubsubPushSchemas, index$5 as schemas };