@walkeros/collector 4.7.0-next-1790245733116 → 4.7.0

package/CHANGELOG.md

@@ -1,50 +1,72 @@
 # @walkeros/collector
 
-## 4.7.0-next-1790245733116
+## 4.7.0
 
 ### Minor Changes
 
-- f5a1cfd: New `collector.next`: a transformer chain that runs once per event,
-  before the event is handed to the destinations. Everything it produces reaches
-  every destination, and a `stop` in it drops the event for all of them;
-  per-destination filtering stays on `destination.before`. The explorer flow map
-  draws this chain, and the MCP guidance describes it.
-- f5a1cfd: Inside an array, a transformer's own `next` now runs right after it,
-  then the array continues. `many` works in every chain field: each match
-  becomes its own copy with a derived `event.id` that finishes the rest of the
-  path. Unknown transformer ids log a warning and the chain continues.
-  `@walkeros/collector` no longer exports `walkChain` or
-  `extractTransformerNextMap`.
-- f5a1cfd: Routes can drop an event with `{ stop: true }`, optionally gated by
-  `match`, in every chain field. A route `match` now reads `{ ingest, event }`
-  and resolves only when the event reaches it, so it sees values earlier steps
-  loaded. `getNextSteps(spec, root)` now requires that root and returns
-  `NextSteps`; the new `getRouteGraph` lists every branch.
-- 2aebc53: `state` paths now resolve against `{ event, ingest }` and need an
-  `event.` or `ingest.` prefix, so a store can be keyed by request context. A
-  `get` can write into `ingest`, keeping the value off the event. An
-  unresolvable key logs a warning instead of failing silently.
-  `buildCacheContext` is renamed `createMappingRoot`.
+- 74821ed: Inside an array, a transformer's own `next` runs right after it, then
+  the array continues. `many` works in every chain field, each match becoming a
+  copy with a derived `event.id`. Unknown transformer ids warn and the chain
+  continues, and a throwing transformer counts in `status.failed`. `walkChain`
+  and `extractTransformerNextMap` are no longer exported.
+- b9668bf: A destination's `on()` handler now receives the current state before
+  any event is sent to it, including with `require`, `walker destination` at
+  runtime, and state passed to `walker run`. Handlers are awaited up to
+  `config.timeout`; a destination whose handler does not respond receives no
+  events until it does. The exported `callDestinationOn` now returns a promise.
+- 74821ed: New `collector.next`: a transformer chain run once per event before
+  the destinations; a `stop` there drops it for all of them. Routes in every
+  chain field can drop an event with `{ stop: true }`, optionally gated by
+  `match`. A route `match` reads `{ ingest, event }` when the event reaches it.
+  New `getRouteGraph`; `getNextSteps(spec, root)` requires its root.
+- 74821ed: `state` paths resolve against `{ event, ingest }` and need an
+  `event.` or `ingest.` prefix; a `get` can write into `ingest`, and an
+  unresolvable key warns. Every step field now survives bundling: transformers
+  with only `state` or `mapping`, and code steps with `cache` or `state`, were
+  silently dropped before. `buildCacheContext` is renamed `createMappingRoot`.
 
 ### Patch Changes
 
-- 2aebc53: A transformer whose `push` throws still stops the chain and drops the
-  event, and is now also counted in `status.failed`, so diagnostics and health
-  checks show the failure instead of only a log line.
-- Updated dependencies [57788bc]
+- 786a860: AWS destinations now deliver in bundled flows without extra setup.
+  Firehose batches by default, ends records with a newline (`newline: false`
+  turns it off), sends mapped `data` and accepts `config.credentials`. Rejected
+  records now count as failed. SNS works with `topicArn` alone and no longer
+  creates topics at start. `walkeros setup` resolves `$secret` values; partial
+  batch failures log their causes.
+- 7e5e3de: A destination added at runtime now keeps the defaults its code ships,
+  such as a default `batch`, just like a destination configured at startup.
+  Values you pass in the destination config still win over those defaults, and a
+  `require` set in the code's defaults now gates the destination as well.
+- 74821ed: Simulate records every vendor call through one recorder, `observeEnv`
+  from `@walkeros/core`, and runs offline: BigQuery, Data Manager and the
+  Sheets, GCS and S3 stores reach the network through `env`. Client and `Env`
+  types of BigQuery, Firehose, PostHog, SQS and Pub/Sub are structural.
+  `@walkeros/cli` exports `resolveExportName` and `selectDevExamples` and drops
+  `findExample` and `compareOutput`.
+- e860006: `state` gains `mapping` to shape what a lookup returns or a write
+  stores, merged into the target. An undeclared `state.store`, or a `file: true`
+  store, is now a validation error in `walkeros validate` and deploy preflight;
+  at runtime the entry logs an error or is skipped with a warning. The s3 and
+  gcs stores report their `file` mode.
+- Updated dependencies [74821ed]
 - Updated dependencies [06b498a]
-- Updated dependencies [f5a1cfd]
-- Updated dependencies [ef02916]
+- Updated dependencies [74821ed]
+- Updated dependencies [64b06de]
+- Updated dependencies [4b4937f]
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
 - Updated dependencies [06b498a]
-- Updated dependencies [ef02916]
-- Updated dependencies [2aebc53]
-- Updated dependencies [f5a1cfd]
-- Updated dependencies [f5a1cfd]
-- Updated dependencies [2aebc53]
-- Updated dependencies [8f8aea6]
-- Updated dependencies [2aebc53]
-- Updated dependencies [2aebc53]
-  - @walkeros/core@4.7.0-next-1790245733116
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
+- Updated dependencies [e860006]
+- Updated dependencies [74821ed]
+- Updated dependencies [74821ed]
+- Updated dependencies [4f89234]
+  - @walkeros/core@4.7.0
 
 ## 4.6.1
 

package/dist/index.d.mts

@@ -213,6 +213,146 @@ declare function resolveBreakerConfig(breaker: Destination.Config['breaker']): B
  */
 declare function isBreakerProbePermitted(breaker: BreakerState | undefined, now: number): boolean;
 
+/**
+ * State-delivery event types: the reactive-state commands that bump
+ * `collector.stateVersion` (see handle.ts). These are the only deliveries
+ * subject to the per-subscriber high-water-mark + `allowed` gate. Lifecycle
+ * types (ready/run/session) and non-reactive config keep their own gating
+ * (onReady/onRun check `allowed`, onSession checks `session`).
+ */
+declare function isStateDelivery(type: On.Types): boolean;
+/**
+ * Is a recorded state CELL present (non-empty)? The single source of truth for
+ * "cell X has a value", shared by `isRequireSatisfied` (require gating) and
+ * `redeliverStateAtRun` (run-barrier re-delivery) so the presence semantics
+ * never drift between the two. PRESENCE, not grant: a denied consent
+ * (`{marketing:false}`) counts as present. Non-cell types return `false` here;
+ * their satisfaction (session/run/ready/arbitrary) is handled by the callers.
+ *
+ * Note: `globals` is seeded from `config.globalsStatic` at construction, so it
+ * reads present whenever a static global exists, before any `command('globals')`
+ * fires. That is intentional and presence-based.
+ */
+declare function isStatePresent(collector: Collector.Instance, type: On.Types): boolean;
+/**
+ * Predicate: is a `require` entry satisfied by the collector's CURRENT recorded
+ * state? This is the level-not-edge core of order-independent activation: a
+ * step's gate is checked against the present state, not against whether the
+ * required type fired before or after the step registered.
+ *
+ * Cell-backed types defer to `isStatePresent` (presence, not grant — the
+ * destination send-gate `getGrantedConsent` remains a separate concern).
+ * `run`/`ready` map to `allowed`. Any other type (including `session`) is
+ * satisfied once it has been broadcast (recorded in `seenEvents`), which also
+ * recovers a broadcast that fired before the requiring step registered.
+ *
+ * `session` deliberately uses the `seenEvents` path, not a cell check:
+ * `collector.session` is vestigial (never written), so gating on it would park
+ * a `require:["session"]` step forever. The session source signals via
+ * `command('session', …)`, which records `session` in `seenEvents`, so this
+ * keeps `session` order-independent like every other type.
+ */
+declare function isRequireSatisfied(collector: Collector.Instance, type: On.Types): boolean;
+/**
+ * Advance a subscriber's mark for a cell to that cell's current version after
+ * an invocation. Only the delivered cell's mark moves; other cells stay owed.
+ */
+declare function setMark(collector: Collector.Instance, subscriber: object, type: On.Types): void;
+/**
+ * A subscriber is invoked for a state delivery iff that CELL has advanced past
+ * its per-cell mark AND the collector is allowed. While `!allowed`, deliveries
+ * are deferred (not fired, mark not advanced) so the subscriber stays "owed".
+ */
+declare function shouldDeliver(collector: Collector.Instance, subscriber: object, type: On.Types): boolean;
+/**
+ * Open the cascade-tracking structure for the OUTERMOST top-level state command
+ * (or an init flush) and return a teardown. Nested commands emitted by
+ * reacting callbacks find `collector.cascade` already set and join it, so the
+ * counters are scoped to the originating command; the cascade is cleared when
+ * its last holder leaves. Re-entrancy is detected by the presence of
+ * `collector.cascade`.
+ *
+ * Assumes top-level state commands run serially on a given collector;
+ * concurrent overlapping state commands on one shared collector are not
+ * supported (web is serial; the server per-request path is event push, not
+ * state commands).
+ */
+declare function enterCascade(collector: Collector.Instance): () => void;
+/**
+ * Registers a callback for a specific event type.
+ *
+ * @param collector The walkerOS collector instance.
+ * @param type The type of the event to listen for.
+ * @param option The callback function or an array of callback functions.
+ */
+declare function on(collector: Collector.Instance, type: On.Types, option: WalkerOS.SingleOrArray<On.Subscription>): Promise<void>;
+/**
+ * Calls a destination's on() handler with proper context and waits for it to
+ * settle. Used by onApply() for live deliveries and by destinationInit() for
+ * flushing queued deliveries; both await it, so a handler's side effect (a
+ * vendor consent update, an SDK opt-out) lands before the destination's next
+ * push. The wait is bounded by the destination's `config.timeout`, the same
+ * window as a push.
+ *
+ * Returns whether the handler settled. A state delivery holds the destination
+ * while it runs, and keeps holding it when it does not settle:
+ * `pushToDestinations` then keeps its events in `queuePush` rather than
+ * sending them to a vendor whose consent state was not set. The hold clears
+ * once no state delivery to the destination is in flight and every present
+ * state cell has reached the handler.
+ *
+ * One state handler call runs per destination at a time, so the state the
+ * vendor applies last is always the newest. A state delivery that arrives
+ * while a call runs does not call the handler: its cell stays owed, and the
+ * current cells are delivered right after the running call ends. A call that
+ * timed out stays in flight until its handler settles; its late settle marks
+ * nothing, and the current cells, then the held events, follow from there.
+ * Deferring rather than waiting keeps a handler that issues a state command
+ * itself from waiting on its own call.
+ *
+ * A state delivery carries the command's delta when the destination has
+ * received every earlier change of that cell, and the whole cell otherwise
+ * (after a failed delivery, or when its mark trails by more than one change),
+ * so a settled delivery never marks content it did not carry.
+ */
+declare function callDestinationOn(collector: Collector.Instance, destination: Destination.Instance, destId: string, type: On.Types, data: unknown): Promise<boolean>;
+/**
+ * Fire a set of registered `on` callbacks against current collector state.
+ *
+ * Used by both `on()` (when registering a new callback, to fire it against
+ * current state) and `onApply()` (when dispatching a state-change command to
+ * every registered callback of that type). Separating this from `onApply`
+ * ensures `on()` does NOT trigger `onApply`'s source/destination broadcast,
+ * which would cause infinite recursion if a source's `on` handler registers
+ * another callback of the same type.
+ */
+declare function fireCallbacks(collector: Collector.Instance, type: On.Types, options: Array<On.Subscription>, config?: unknown): void;
+/**
+ * Run-barrier re-delivery. Called once from `runCollector` after the collector
+ * becomes `allowed` and the RunState merge has bumped `stateVersion` for any
+ * merged cells. Re-broadcasts each non-empty recorded state cell to its OWED
+ * subscribers (mark < stateVersion) exactly once, so reactions deferred while
+ * `!allowed` now emit into the open, consent-gated pipeline.
+ *
+ * Narrow path: it fires `collector.on` rules/fns via `fireCallbacks`, the
+ * gated `source.on` handlers via `deliverStateToSource`, and the gated
+ * destination `on` handlers via `deliverStateToDestination` (queued for an
+ * uninitialized destination's init flush). It deliberately skips the
+ * `require`-decrement and `queueOn`-flush machinery in `onApply` (those are
+ * live-command concerns). Exactly-once is free from the `shouldDeliver` gate:
+ * already-delivered subscribers (mark == stateVersion) are skipped.
+ */
+declare function redeliverStateAtRun(collector: Collector.Instance): Promise<void>;
+/**
+ * Applies all registered callbacks for a specific event type.
+ *
+ * @param collector The walkerOS collector instance.
+ * @param type The type of the event to apply the callbacks for.
+ * @param options The options for the callbacks.
+ * @param config The consent configuration.
+ */
+declare function onApply(collector: Collector.Instance, type: On.Types, options?: Array<On.Subscription>, config?: unknown): Promise<boolean>;
+
 /**
  * Adds a new destination to the collector.
  *
@@ -337,119 +477,6 @@ declare function enrichEvent(collector: Collector.Instance, event: WalkerOS.Deep
  */
 declare function runCollector(collector: Collector.Instance, state?: RunState): Promise<Elb.PushResult>;
 
-/**
- * State-delivery event types: the reactive-state commands that bump
- * `collector.stateVersion` (see handle.ts). These are the only deliveries
- * subject to the per-subscriber high-water-mark + `allowed` gate. Lifecycle
- * types (ready/run/session) and non-reactive config keep their own gating
- * (onReady/onRun check `allowed`, onSession checks `session`).
- */
-declare function isStateDelivery(type: On.Types): boolean;
-/**
- * Is a recorded state CELL present (non-empty)? The single source of truth for
- * "cell X has a value", shared by `isRequireSatisfied` (require gating) and
- * `redeliverStateAtRun` (run-barrier re-delivery) so the presence semantics
- * never drift between the two. PRESENCE, not grant: a denied consent
- * (`{marketing:false}`) counts as present. Non-cell types return `false` here;
- * their satisfaction (session/run/ready/arbitrary) is handled by the callers.
- *
- * Note: `globals` is seeded from `config.globalsStatic` at construction, so it
- * reads present whenever a static global exists, before any `command('globals')`
- * fires. That is intentional and presence-based.
- */
-declare function isStatePresent(collector: Collector.Instance, type: On.Types): boolean;
-/**
- * Predicate: is a `require` entry satisfied by the collector's CURRENT recorded
- * state? This is the level-not-edge core of order-independent activation: a
- * step's gate is checked against the present state, not against whether the
- * required type fired before or after the step registered.
- *
- * Cell-backed types defer to `isStatePresent` (presence, not grant — the
- * destination send-gate `getGrantedConsent` remains a separate concern).
- * `run`/`ready` map to `allowed`. Any other type (including `session`) is
- * satisfied once it has been broadcast (recorded in `seenEvents`), which also
- * recovers a broadcast that fired before the requiring step registered.
- *
- * `session` deliberately uses the `seenEvents` path, not a cell check:
- * `collector.session` is vestigial (never written), so gating on it would park
- * a `require:["session"]` step forever. The session source signals via
- * `command('session', …)`, which records `session` in `seenEvents`, so this
- * keeps `session` order-independent like every other type.
- */
-declare function isRequireSatisfied(collector: Collector.Instance, type: On.Types): boolean;
-/**
- * Advance a subscriber's mark for a cell to that cell's current version after
- * an invocation. Only the delivered cell's mark moves; other cells stay owed.
- */
-declare function setMark(collector: Collector.Instance, subscriber: object, type: On.Types): void;
-/**
- * A subscriber is invoked for a state delivery iff that CELL has advanced past
- * its per-cell mark AND the collector is allowed. While `!allowed`, deliveries
- * are deferred (not fired, mark not advanced) so the subscriber stays "owed".
- */
-declare function shouldDeliver(collector: Collector.Instance, subscriber: object, type: On.Types): boolean;
-/**
- * Open the cascade-tracking structure for the OUTERMOST top-level state command
- * and return a teardown that clears it. Nested commands emitted by reacting
- * callbacks find `collector.cascade` already set and reuse it (teardown is a
- * no-op for them), so the counters are scoped to the originating command and
- * reset when it returns. Re-entrancy is detected by the presence of
- * `collector.cascade`.
- *
- * Assumes top-level state commands run serially on a given collector;
- * concurrent overlapping state commands on one shared collector are not
- * supported (web is serial; the server per-request path is event push, not
- * state commands).
- */
-declare function enterCascade(collector: Collector.Instance): () => void;
-/**
- * Registers a callback for a specific event type.
- *
- * @param collector The walkerOS collector instance.
- * @param type The type of the event to listen for.
- * @param option The callback function or an array of callback functions.
- */
-declare function on(collector: Collector.Instance, type: On.Types, option: WalkerOS.SingleOrArray<On.Subscription>): Promise<void>;
-/**
- * Calls a destination's on() handler with proper context.
- * Used by both onApply() for immediate calls and destinationInit() for flushing queued events.
- */
-declare function callDestinationOn(collector: Collector.Instance, destination: Destination.Instance, destId: string, type: On.Types, data: unknown): void;
-/**
- * Fire a set of registered `on` callbacks against current collector state.
- *
- * Used by both `on()` (when registering a new callback, to fire it against
- * current state) and `onApply()` (when dispatching a state-change command to
- * every registered callback of that type). Separating this from `onApply`
- * ensures `on()` does NOT trigger `onApply`'s source/destination broadcast,
- * which would cause infinite recursion if a source's `on` handler registers
- * another callback of the same type.
- */
-declare function fireCallbacks(collector: Collector.Instance, type: On.Types, options: Array<On.Subscription>, config?: unknown): void;
-/**
- * Run-barrier re-delivery. Called once from `runCollector` after the collector
- * becomes `allowed` and the RunState merge has bumped `stateVersion` for any
- * merged cells. Re-broadcasts each non-empty recorded state cell to its OWED
- * subscribers (mark < stateVersion) exactly once, so reactions deferred while
- * `!allowed` now emit into the open, consent-gated pipeline.
- *
- * Narrow path: it fires `collector.on` rules/fns via `fireCallbacks` and the
- * gated `source.on` loop via `deliverStateToSource`. It deliberately skips the
- * `require`-decrement and `queueOn`-flush machinery in `onApply` (those are
- * live-command concerns). Exactly-once is free from the `shouldDeliver` gate:
- * already-delivered subscribers (mark == stateVersion) are skipped.
- */
-declare function redeliverStateAtRun(collector: Collector.Instance): Promise<void>;
-/**
- * Applies all registered callbacks for a specific event type.
- *
- * @param collector The walkerOS collector instance.
- * @param type The type of the event to apply the callbacks for.
- * @param options The options for the callbacks.
- * @param config The consent configuration.
- */
-declare function onApply(collector: Collector.Instance, type: On.Types, options?: Array<On.Subscription>, config?: unknown): Promise<boolean>;
-
 /**
  * Flush a source's queueOn buffer. Called when the source becomes "started"
  * (config.init === true AND config.require is empty/absent). Idempotent:
@@ -459,7 +486,9 @@ declare function onApply(collector: Collector.Instance, type: On.Types, options?
  * A throw inside `source.on` is treated as a pipeline failure: log via the
  * scoped 'source' logger and increment `status.failed`. The flush itself
  * is walkerOS-orchestrated startup; the throw represents the source's
- * inability to consume a buffered state-change event.
+ * inability to consume a buffered state-change event. A handler that does
+ * not settle within the default destination timeout is logged the same way
+ * but not counted, matching the live delivery path.
  *
  * State-delivery entries (consent/user/globals/custom) are gated through the
  * same per-source high-water mark as the direct `onApply` broadcast: while
@@ -530,28 +559,28 @@ declare function runTransformerChain(collector: Collector.Instance, transformers
 declare function runTransformerBefore(collector: Collector.Instance, transformers: Transformer.Transformers, transformerId: string, event: WalkerOS.DeepPartialEvent, ingest?: Ingest, respond?: RespondFn, chainContext?: string): Promise<Transformer.ChainResult>;
 
 interface WrapResult {
-    /** Env with tracked paths wrapped by recording functions */
+    /** Env view whose tracked paths record each call */
     wrappedEnv: Record<string, unknown>;
     /** Mutable array — calls are pushed here during step execution */
     calls: Simulation.Call[];
     /**
-     * Well-formed declared paths this env could not wrap: a missing or
-     * non-object root/intermediate, an absent/non-function leaf, or a parent
-     * beyond the clone cap (wrapping there would mutate the caller's env). Each
-     * entry is the post-`call:` stripped dot-path (what the recorder records as
-     * `fn`) — this list feeds `EnvObserve.paths` directly, so malformed paths
-     * (`parseCallPath` returned `[]`) are skipped entirely: the shared grammar
-     * makes them unresolvable for the fallback wrapper too. A path that
-     * navigates part-way then stops belongs here, never wrapped at a wrong level.
+     * Well-formed declared paths whose object intermediates or leaf are absent
+     * at wrap time, as the post-`call:` dot-path (what the recorder records as
+     * `fn`). This list feeds `EnvObserve.paths` directly, so malformed paths
+     * (`parseCallPath` returned `[]`) are skipped entirely. A path that
+     * continues through a function (constructor, factory, method result)
+     * resolves at call time and is not listed.
      */
     unresolved: string[];
 }
 /**
- * Wrap tracked paths in a destination env with recording wrappers.
+ * Record calls on tracked paths of a destination env.
  *
  * The env object must include a `simulation: string[]` declaring which
- * dot-paths to intercept. Returns a cloned env (without `simulation`)
- * where those paths record every call into the `calls` array.
+ * dot-paths to intercept. Returns a view of a cloned env (without
+ * `simulation`) where those paths record every call into the `calls` array.
+ * Recording is the shared core recorder `observeEnv`, so constructors,
+ * method results and Promises on a path work the same as in web-core `getEnv`.
  */
 declare function wrapEnv(env: Record<string, unknown> & {
     simulation: string[];