@walkeros/cli 4.0.0 → 4.1.0-next-1778155282668

package/CHANGELOG.md

@@ -1,5 +1,108 @@
 # @walkeros/cli
 
+## 4.1.0-next-1778155282668
+
+### Minor Changes
+
+- abfb0bb: @walkeros/cli: Server bundles now use @vercel/nft to trace
+  dependencies and copy only files actually used into dist/node_modules/. Pacote
+  remains the install layer (driven by flow.json's config.bundle.packages field;
+  users do not run npm install for step packages). The walkerOS.bundle.external
+  annotation field on package manifests is no longer recognized (deprecation
+  warning if seen). The flow.<name>.config.bundle.external sub-field on flow
+  configs is also no longer supported (warned and stripped during load). The
+  flow.<name>.config.bundle.traceInclude field is the escape hatch for cases nft
+  cannot statically trace. Server output is always a directory: dist/{flow.mjs,
+  package.json, node_modules/}. Default output filename changed from bundle.mjs
+  to flow.mjs. The runtime image expects /app/flow/flow.mjs. flow.json schema is
+  unchanged (still v4); only @walkeros/cli bumps. Migration: see
+  https://walkeros.io/docs/migrate/cli-4x.
+
+  @walkeros/server-destination-gcp: removed obsolete walkerOS.bundle.external
+  annotation from package manifest. nft handles externalization automatically.
+  No behavior change for consumers.
+
+### Patch Changes
+
+- ed304b4: Bundler honors `walkerOS.bundle.external` declared in step-package
+  package.json files. Listed packages are externalized from the ESM bundle and
+  the bundler always installs them (plus their full transitive deps) into
+  `<outputDir>/node_modules/` via pacote — no `npm install` shell-out, no manual
+  deploy step. When externals is empty, output remains a single `bundle.mjs`
+  (backward compatible). When non-empty, output is a self-contained directory:
+  `bundle.mjs`, `package.json`, `package-lock.json`, `node_modules/`.
+
+  The bundler reads npm config (registry, scope tokens) from `.npmrc`,
+  parallelizes manifest fetches with retry, atomically stages each package
+  extraction (no half-populated `node_modules/` on failure), and reuses the
+  closure resolution from the existing `collectAllSpecs` BFS so peerDependencies
+  are honored.
+
+  Hard-errors when:
+  - A package in the install closure declares a `pre/install/postinstall` script
+    (pacote.extract does not run them).
+  - A step package names an external in `walkerOS.bundle.external` but does not
+    list it in `dependencies` or `peerDependencies`.
+  - Two step packages declare the same external and the resolved version does
+    not satisfy all consumers' constraints.
+
+  Warns (not errors) when:
+  - Bundle output contains unresolved `__dirname` / `__filename` references
+    (with package attribution by hit count).
+  - A step package's `walkerOS.bundle.*` block contains unknown keys (typo
+    guard).
+
+  New sibling export `downloadPackagesWithResolution` returns both the package
+  paths and the full `ResolutionResult`. Existing `downloadPackages` keeps its
+  return shape unchanged.
+
+- e4b6cf4: Fix `walkeros bundle` failing on Windows when stage 2 import paths
+  contained backslashes that JS parsed as escape sequences.
+- 381dfe7: Add an optional `setup` lifecycle to destinations, sources, and
+  stores.
+
+  Each package may now implement `setup?: SetupFn` to provision external
+  resources (BigQuery datasets and tables, Pub/Sub topics and subscriptions,
+  SQLite tables, webhook registrations, etc.). Setup is triggered only by the
+  new `walkeros setup <kind>.<name>` CLI command, never automatically by the
+  runtime, push, or deploy. Idempotency, ordering, and error semantics are the
+  package's responsibility; the framework provides the type slot, the CLI
+  invocation, and a `resolveSetup(value, defaults)` helper.
+
+  `LifecycleContext<C, E>` is the new shared context type used by both `setup`
+  and `destroy`. `DestroyContext` remains as a deprecated type alias for one
+  minor cycle. The `Types` bundle on `Destination`, `Source`, and `Store` gains
+  a 5th/6th/4th positional slot for setup options; existing aliases compile
+  unchanged because the slot defaults to `unknown`.
+  `Config<T>.setup?: boolean | SetupOptions<T>` is added across all three kinds
+  and validated by the corresponding Zod `ConfigSchema` plus the flow component
+  schemas in `@walkeros/core/schemas/flow.ts`.
+
+  CLI:
+  - `walkeros setup <kind>.<name>` runs a single component's `setup()` function.
+  - `<kind>` is `source`, `destination`, or `store` (transformers have no
+    provisioning).
+  - `--config <path>` (default `./flow.json`), `--flow <name>` for multi-flow
+    configs, plus standard `--json` / `--verbose` / `--silent`.
+  - Exit 0 on success or skip; non-zero on failure. Skip narration covers three
+    cases: no `setup()` on the package, `config.setup === false`, or
+    `config.setup` unset.
+  - When the package's `setup()` returns a non-undefined value, the CLI emits it
+    as JSON on stdout for `jq` piping.
+
+- 03d7055: Merge `definitions` into `variables`. Single concept, single syntax
+  `$var.name(.deep.path)?`. Whole-string references preserve native type; inline
+  interpolation requires scalars. Deep paths and recursive resolution with cycle
+  detection now supported. `Flow.Definitions`, `Flow.Primitive`, and the `$def.`
+  reference syntax are removed.
+- Updated dependencies [381dfe7]
+- Updated dependencies [1524275]
+- Updated dependencies [03d7055]
+  - @walkeros/core@4.1.0-next-1778155282668
+  - @walkeros/collector@4.1.0-next-1778155282668
+  - @walkeros/server-core@4.1.0-next-1778155282668
+  - @walkeros/server-destination-api@4.1.0-next-1778155282668
+
 ## 4.0.0
 
 ### Major Changes

package/README.md

@@ -52,7 +52,7 @@ walkeros push flow.json --event '{"name":"product view"}' --simulate destination
 walkeros push flow.json --event '{"name":"product view"}'
 
 # Run a collection server locally
-walkeros run dist/bundle.mjs --port 3000
+walkeros run dist/flow.mjs --port 3000
 ```
 
 ## Commands
@@ -79,7 +79,6 @@ walkeros bundle https://example.com/config.json                  # Remote URL
 - `--stats` - Show bundle statistics
 - `--json` - Output as JSON (implies --stats)
 - `--no-cache` - Disable package caching
-- `--dockerfile [file]` - Generate Dockerfile (or copy custom file) to dist/
 - `-v, --verbose` - Verbose output
 - `-s, --silent` - Suppress output
 
@@ -89,15 +88,31 @@ walkeros bundle https://example.com/config.json                  # Remote URL
 # Bundle with stats
 walkeros bundle examples/server-collect.json --stats
 
-# Bundle with auto-generated Dockerfile
-walkeros bundle flow.json --dockerfile
+# Bundle to a custom output directory
+walkeros bundle flow.json -o ./build/
+```
+
+### Server bundles use nft tracing
+
+Server flows (`platform: "server"`) are bundled with `@vercel/nft`. The CLI
+externalizes every step package, traces the entry to discover which files are
+actually reachable at runtime, and copies only those files into
+`dist/node_modules/`. Step packages are installed via pacote, driven by the
+`config.bundle.packages` field in `flow.json`. **You do not need to run
+`npm install` for step packages**: only `@walkeros/cli` belongs in your
+`package.json` devDependencies.
 
-# Bundle with custom Dockerfile
-walkeros bundle flow.json --dockerfile Dockerfile.custom
+The output is always a directory:
+
+```
+dist/
+├── flow.mjs        # ESM entry point
+├── package.json     # informational sidecar
+└── node_modules/    # only the files nft traced
 ```
 
-The output path uses convention-based defaults: `./dist/bundle.mjs` for server,
-`./dist/walker.js` for web.
+For web flows the output stays a single self-contained file (default
+`dist/walker.js`).
 
 ### push
 
@@ -162,23 +177,85 @@ walkeros push flow.json -e event.json --mock destination.ga4='{"status":"ok"}'
 
 ```bash
 # Push with pre-built bundle
-walkeros push dist/bundle.mjs --event '{"name":"order complete"}'
+walkeros push dist/flow.mjs --event '{"name":"order complete"}'
 
 # Override platform detection
-walkeros push dist/bundle.js --platform server --event '{"name":"order complete"}'
+walkeros push dist/walker.js --platform web --event '{"name":"order complete"}'
 ```
 
 **Push modes:**
 
-| Mode | Flag | API Calls | Use Case |
-| ---- | ---- | --------- | -------- |
-| Real | (none) | Real HTTP requests | Integration testing |
-| Simulate | `--simulate` | Mocked (captured) | Safe local testing |
-| Mock | `--mock` | Returns mock value | Controlled testing |
+| Mode     | Flag         | API Calls          | Use Case            |
+| -------- | ------------ | ------------------ | ------------------- |
+| Real     | (none)       | Real HTTP requests | Integration testing |
+| Simulate | `--simulate` | Mocked (captured)  | Safe local testing  |
+| Mock     | `--mock`     | Returns mock value | Controlled testing  |
 
 Use `--simulate` first to validate safely, then push without flags for real
 integrations.
 
+### setup
+
+Run the optional `setup()` lifecycle on a single component to provision external
+resources (BigQuery datasets, Pub/Sub topics, SQLite tables, webhook
+registrations).
+
+```bash
+walkeros setup <target> [options]
+```
+
+**Target format:** `<kind>.<name>` matching `walkeros push --simulate`. Valid
+kinds: `source`, `destination`, `store`. Transformers are pure functions and
+have no setup.
+
+**Options:**
+
+- `-c, --config <path>` - Flow config file (default: `./flow.json`)
+- `-f, --flow <name>` - Flow name for multi-flow configs
+- `--json` - Output as JSON
+- `-v, --verbose` - Verbose output
+- `-s, --silent` - Suppress output
+
+**Behavior:**
+
+- Loads the flow config, resolves the named component, imports its package, and
+  calls `setup({ id, config, env, logger })`. No collector boot, no event
+  pipeline, no destinations are pushed to.
+- Skips with an explanatory message in three cases: the package has no `setup`
+  function, `config.setup === false`, or `config.setup` is unset.
+- When `setup()` returns a non-undefined value, the CLI emits it as JSON on
+  stdout for `jq`-style scripting.
+- Exit code `0` on success or skip, non-zero on failure.
+
+**Operator-time, never automatic.** Setup is explicit only. It is never
+triggered by `push`, `simulate`, `deploy`, or the long-running runtime.
+Operators run it once when provisioning, and again only when external resources
+need to be re-provisioned.
+
+**IAM note:** Setup typically needs higher permissions than runtime push (for
+example, "create dataset" vs "write rows"). Operators commonly run setup with a
+separate service account or different credentials than the runtime uses.
+
+**Examples:**
+
+```bash
+# Provision the BigQuery dataset and table for the destination named "bigquery"
+# in the default flow file.
+walkeros setup destination.bigquery
+
+# Same but pointing to a specific flow in a multi-flow config.
+walkeros setup destination.bigquery --flow analytics
+
+# Custom config path.
+walkeros setup destination.bigquery --config ./flows/prod.json
+
+# Pipe the structured result to jq.
+walkeros setup destination.bigquery --json | jq .datasetCreated
+
+# Provision a Pub/Sub source's subscription.
+walkeros setup source.events-in
+```
+
 ### run
 
 Run flows locally (no Docker daemon required).
@@ -201,8 +278,8 @@ walkeros run <config-file> [options]
 # Run collection server (auto-bundles JSON)
 walkeros run examples/server-collect.json --port 3000
 
-# Run with pre-built bundle
-walkeros run examples/server-collect.mjs --port 3000
+# Run with a pre-built server bundle
+walkeros run dist/flow.mjs --port 3000
 ```
 
 **How it works:**
@@ -328,18 +405,22 @@ walkeros bundle flow.json --no-cache
 
 ## Flow Configuration
 
-Flow configs use the `Flow.Config` format with `version` and `flows`:
+Flow configs use the `Flow.Json` format with `version` and `flows`:
 
 ```json
 {
-  "version": 3,
+  "version": 4,
   "flows": {
     "default": {
-      "server": {},
-      "packages": {
-        "@walkeros/collector": { "imports": ["startFlow"] },
-        "@walkeros/server-source-express": {},
-        "@walkeros/destination-demo": {}
+      "config": {
+        "platform": "server",
+        "bundle": {
+          "packages": {
+            "@walkeros/collector": { "imports": ["startFlow"] },
+            "@walkeros/server-source-express": {},
+            "@walkeros/destination-demo": {}
+          }
+        }
       },
       "sources": {
         "http": {
@@ -363,7 +444,9 @@ Flow configs use the `Flow.Config` format with `version` and `flows`:
 }
 ```
 
-Platform is determined by the `web: {}` or `server: {}` key presence.
+Platform is set via `config.platform` (`"web"` or `"server"`). The
+`config.bundle.packages` field declares what pacote should install.
+`config.bundle.overrides` pins transitive dependency versions when needed.
 
 ### Package Configuration Patterns
 
@@ -509,10 +592,7 @@ const result = await push(
 // result.usage = API call tracking data
 
 // Push for real
-await push(
-  './flow.json',
-  { name: 'page view', data: { title: 'Test' } },
-);
+await push('./flow.json', { name: 'page view', data: { title: 'Test' } });
 
 // Run
 await runCommand({
@@ -566,7 +646,7 @@ walkeros push \
 walkeros bundle my-flow.json --stats
 
 # 4. Run locally
-walkeros run dist/bundle.mjs --port 3000
+walkeros run dist/flow.mjs --port 3000
 
 # 5. In another terminal, test it
 curl -X POST http://localhost:3000/collect \
@@ -577,13 +657,16 @@ curl -X POST http://localhost:3000/collect \
 ## Architecture
 
 ```
-CLI (downloads packages + bundles with esbuild)
- ├─ Bundle → optimized .mjs file
- ├─ Push → execute bundle (with optional --simulate for testing)
- └─ Run → execute bundle with built-in runtime
+CLI
+ ├─ Pacote installs flow.json packages (no user-side npm install)
+ ├─ esbuild externalizes step packages, emits ESM entry
+ ├─ @vercel/nft traces entry, copies only used files
+ └─ Output: dist/{flow.mjs, package.json, node_modules/}  (server)
+            dist/walker.js                                  (web)
 ```
 
-**Key principle**: CLI handles both build-time and runtime operations.
+**Key principle**: CLI handles both build-time install/trace/bundle and runtime
+execution.
 
 ## Runner (Docker)
 
@@ -625,7 +708,7 @@ Run the bundle directly with the CLI:
 walkeros bundle flow.json
 
 # Run in production
-walkeros run dist/bundle.mjs --port 8080
+walkeros run dist/flow.mjs --port 8080
 ```
 
 This runs the flow in the current Node.js process, suitable for deployment on