@defrex/autobuild-hosted-dispatcher 1.4.0 → 1.4.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -7,8 +7,9 @@ tick per configured repository per invocation. It drives the same core
7
7
  pass runs — claim, provision, launch, observe, settle, merge PRs, janitor,
8
8
  lease sweep — and reuses `@defrex/autobuild-hosted-store-service`'s validated
9
9
  store environment for the signing secret. See the
10
- [operator procedure](../../docs/hosted-dispatcher.md) for the schedule,
11
- incident pauses, and behavior details.
10
+ [self-hosting guide](https://github.com/defrex/autobuild/blob/main/packages/hosted-store-service/SELF-HOSTING.md#dispatcher-enabled-deployment)
11
+ for the schedule, function duration, and pausing. Self-hosting is unsupported
12
+ and buyer-beware.
12
13
 
13
14
  A deployment opts into building by installing the package and mounting the
14
15
  route: the Next.js app's `app/api/dispatch/route.ts` (in the hosted store
@@ -20,17 +21,50 @@ it drives) and `@defrex/autobuild-hosted-store-service` (the environment
20
21
  parser and types); a deployment that installs it adds
21
22
  `bun add @defrex/autobuild-hosted-dispatcher`.
22
23
 
23
- Besides the endpoint driver, the package owns two deploy-time helpers:
24
+ Besides the endpoint driver, the package owns the deploy-time
25
+ `ab-hosted-dispatcher` bin, also exported as
26
+ `@defrex/autobuild-hosted-dispatcher/bin`. A bundled deployment has neither
27
+ `bun` nor a source tree at runtime, so the build has to carry everything the
28
+ dispatch function installs and loads. The bin has two commands for that:
24
29
 
25
- - the `ab-hosted-dispatcher` bin with the `pack-distribution` command, which
26
- packs the running distribution into `<root>/.autobuild-dist/autobuild-<version>.tgz`
27
- during the deployment build (the hosted store service's `deploy:build` runs
28
- it by path from the package directory:
29
- `bun ../../packages/hosted-dispatcher/src/bin.ts pack-distribution --root .`); and
30
- - the `ship-packed-distribution` trace helper, which appends that archive to
31
- the dispatch route's Next.js trace file so the function bundle carries it
32
- (the same `deploy:build` runs it by path from the package directory as its
33
- final step: `bun ../../packages/hosted-dispatcher/src/ship-packed-distribution.ts`).
30
+ - `pack-distribution [--root DIR]`, run before `next build`, packs the installed
31
+ `@defrex/autobuild` into `<root>/.autobuild-dist/autobuild-<version>.tgz`.
32
+ It removes archives left over from other versions.
33
+ - `ship-deployment [--root DIR] [--dist-dir DIR] [--plugin NAME]...`, run after
34
+ `next build`, appends the archive and the distribution manifest to the
35
+ `/api/dispatch` route's Next.js trace. It then stages each selected provider
36
+ plugin into the function bundle. It fails the build, naming the missing
37
+ piece, when it cannot ship the archive, the manifest or a selected plugin,
38
+ and when the archive's version differs from the installed `@defrex/autobuild`.
39
+
40
+ The hosted store service's published `ab-hosted-service-build` runs both
41
+ commands automatically when this package is installed.
42
+
43
+ ### Which provider plugins ship
44
+
45
+ The first of these rules that applies decides:
46
+
47
+ 1. `ship-deployment --plugin NAME` flags.
48
+ 2. The deployment `package.json` field `"autobuild": { "plugins": [...] }`.
49
+ An explicit `[]` ships none.
50
+ 3. By default, each entry of the deployment's `dependencies` whose installed
51
+ `package.json` declares `@defrex/autobuild` in `peerDependencies`, as
52
+ provider plugins do. The default skips Autobuild's own service packages:
53
+ `@defrex/autobuild-hosted-store-service`,
54
+ `@defrex/autobuild-hosted-dispatcher` and `@defrex/autobuild-postgres-store`.
55
+
56
+ Each selected package must be installed in the deployment and must bundle.
57
+ Its bundle must also evaluate to a valid plugin manifest. Any failure fails
58
+ the build, naming the package and the rule that selected it. The build
59
+ imports each bundle once to validate it, so top-level code in a plugin runs
60
+ during the build.
61
+
62
+ A plugin ships as a self-contained `bun build` bundle staged at
63
+ `node_modules/<plugin>` inside the installed `@defrex/autobuild` directory.
64
+ That is the first place the dispatch runtime looks when it resolves a plugin
65
+ from its installation. The deployment's own installed copy of the plugin is
66
+ never modified. A local build without `VERCEL` set removes the staged bundle
67
+ afterwards, so `node_modules` stays exactly as `bun install` left it.
34
68
 
35
69
  ## Dispatcher environment variables
36
70
 
@@ -53,7 +87,7 @@ Dispatcher variables (server-only, like every secret; none reach the browser):
53
87
  `hosted: false` without requiring an origin.
54
88
  - `AB_DISPATCHER_BUDGET_SECONDS`: per-invocation work budget (default 240,
55
89
  clamped 10–780); pair it with the route's `maxDuration` as described in the
56
- operator procedure.
90
+ [self-hosting guide](https://github.com/defrex/autobuild/blob/main/packages/hosted-store-service/SELF-HOSTING.md#function-duration-and-budget).
57
91
  - `AB_DISPATCHER_TOKEN_TTL_SECONDS`: an upper bound on stored cron credential
58
92
  lifetime (default 604800 — 7 days; minimum 3600). Actual expiry is capped at
59
93
  the invocation deadline plus 60 seconds. Successful and failed ticks delete
@@ -86,7 +120,7 @@ Dispatcher variables (server-only, like every secret; none reach the browser):
86
120
  archive. Unset, the kernel uses the archive
87
121
  `ab-hosted-dispatcher pack-distribution` wrote to `.autobuild-dist/` during
88
122
  the deployment build (required for a bundled deployment, which has no `bun`
89
- to pack with at runtime — see the operator procedure).
123
+ to pack with at runtime).
90
124
 
91
125
  The Sandbox SDK authenticates with the deployment's own OIDC identity inside
92
126
  Vercel functions (the `x-vercel-oidc-token` request header, forwarded to the
@@ -102,7 +136,7 @@ deposit time, overridable with `AB_ARTIFACT_RETENTION_MAX_REVISIONS`.
102
136
 
103
137
  Manage projects through the deployment's database-authorized
104
138
  `ab-hosted-store projects list|add|remove` commands (mutations require
105
- `--user IDENTITY`); see [connected projects](../../docs/setup.md#connected-projects).
139
+ `--user IDENTITY`); see [connected projects](https://github.com/defrex/autobuild/blob/main/packages/hosted-store-service/README.md#connected-projects).
106
140
  Migrate PostgreSQL before deployment. Deprecated `AB_WEB_REPOSITORIES` seeds
107
141
  registrations once per database, including standalone dispatcher deployments;
108
142
  subsequent changes to it are not administration. Forge token override keys are
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@defrex/autobuild-hosted-dispatcher",
3
- "version": "1.4.0",
3
+ "version": "1.4.2",
4
4
  "type": "module",
5
5
  "license": "Apache-2.0",
6
6
  "engines": {
@@ -22,6 +22,10 @@
22
22
  "./distribution": {
23
23
  "types": "./src/ship-packed-distribution.ts",
24
24
  "import": "./src/ship-packed-distribution.ts"
25
+ },
26
+ "./bin": {
27
+ "types": "./src/bin.ts",
28
+ "import": "./src/bin.ts"
25
29
  }
26
30
  },
27
31
  "bin": {
package/src/bin.ts CHANGED
@@ -1,17 +1,42 @@
1
1
  #!/usr/bin/env bun
2
- import { writePrebuiltDistributionArchive } from '@defrex/autobuild/distribution'
2
+ import { basename } from 'node:path'
3
+ import { readFile } from 'node:fs/promises'
4
+ import {
5
+ distributionManifestPath,
6
+ writePrebuiltDistributionArchive,
7
+ } from '@defrex/autobuild/distribution'
8
+ import { selectDeploymentPlugins } from './deployment-plugins'
9
+ import { ensureDistributionArchiveInTrace, findArchive } from './ship-packed-distribution'
10
+ import { restoresInstallIn, shipProviderPlugins } from './ship-provider-plugin'
3
11
 
4
12
  const USAGE = `Usage:
5
- ab-hosted-dispatcher pack-distribution [--root DIR]`
13
+ ab-hosted-dispatcher pack-distribution [--root DIR]
14
+ ab-hosted-dispatcher ship-deployment [--root DIR] [--dist-dir DIR] [--plugin NAME]...`
15
+
16
+ function options(args: string[], name: string): string[] {
17
+ return args.flatMap((value, index) => {
18
+ if (value !== name) return []
19
+ const next = args[index + 1]
20
+ if (next === undefined || next.startsWith('--')) throw new Error(`${name} requires a value`)
21
+ return [next]
22
+ })
23
+ }
6
24
 
7
25
  function option(args: string[], name: string): string | undefined {
8
- const indexes = args.flatMap((value, index) => (value === name ? [index] : []))
9
- if (indexes.length > 1) throw new Error(`${name} may only be supplied once`)
10
- const index = indexes[0]
11
- if (index === undefined) return undefined
12
- const value = args[index + 1]
13
- if (value === undefined || value.startsWith('--')) throw new Error(`${name} requires a value`)
14
- return value
26
+ const values = options(args, name)
27
+ if (values.length > 1) throw new Error(`${name} may only be supplied once`)
28
+ return values[0]
29
+ }
30
+
31
+ /** Reject anything that is not the command or one of its `--name value`
32
+ * pairs. */
33
+ function rejectUnknown(args: string[], command: string, names: string[]): void {
34
+ if (args[0] !== command) throw new Error(`unknown argument: ${args[0] ?? '(none)'}`)
35
+ for (let index = 1; index < args.length; index++) {
36
+ const arg = args[index]!
37
+ if (!names.includes(arg)) throw new Error(`unknown argument: ${arg}`)
38
+ index++
39
+ }
15
40
  }
16
41
 
17
42
  /** `pack-distribution`: pack the running distribution into
@@ -25,8 +50,7 @@ export async function runPackDistribution(
25
50
  ): Promise<number> {
26
51
  try {
27
52
  const root = option(args, '--root')
28
- const allowed = new Set(['pack-distribution', '--root', root])
29
- for (const arg of args) if (!allowed.has(arg)) throw new Error(`unknown argument: ${arg}`)
53
+ rejectUnknown(args, 'pack-distribution', ['--root'])
30
54
  const path = await writePrebuiltDistributionArchive(root)
31
55
  write(path)
32
56
  return 0
@@ -36,10 +60,93 @@ export async function runPackDistribution(
36
60
  }
37
61
  }
38
62
 
63
+ export interface ShipDeploymentArguments {
64
+ root: string
65
+ distDir: string
66
+ plugins: string[]
67
+ }
68
+
69
+ export function parseShipDeployment(args: string[], cwd = process.cwd()): ShipDeploymentArguments {
70
+ const root = option(args, '--root') ?? cwd
71
+ const distDir = option(args, '--dist-dir') ?? '.next'
72
+ const plugins = options(args, '--plugin')
73
+ rejectUnknown(args, 'ship-deployment', ['--root', '--dist-dir', '--plugin'])
74
+ return { root, distDir, plugins }
75
+ }
76
+
77
+ /** The packed archive must be the running distribution's version: a stale
78
+ * archive would install a different Autobuild into guests. */
79
+ export async function assertArchiveMatchesDistribution(
80
+ archive: string,
81
+ manifest = distributionManifestPath(),
82
+ ): Promise<void> {
83
+ const archived = /^autobuild-(.+)\.tgz$/.exec(basename(archive))?.[1]
84
+ let version: unknown
85
+ try {
86
+ version = (JSON.parse(await readFile(manifest, 'utf8')) as { version?: unknown }).version
87
+ } catch {
88
+ throw new Error(
89
+ `missing distribution manifest ${manifest} — provisioning reads the version from it`,
90
+ )
91
+ }
92
+ if (archived !== version) {
93
+ throw new Error(
94
+ `distribution archive ${basename(archive)} is version ${archived ?? '(unknown)'} but the ` +
95
+ `installed @defrex/autobuild is ${String(version)} — rerun pack-distribution`,
96
+ )
97
+ }
98
+ }
99
+
100
+ /** `ship-deployment`: after `next build`, carry the packed archive, the
101
+ * distribution manifest, and the selected provider plugins into the
102
+ * `/api/dispatch` function bundle. */
103
+ export async function runShipDeployment(
104
+ args: string[],
105
+ write: (text: string) => void = (text) => console.log(text),
106
+ writeError: (text: string) => void = (text) => console.error(text),
107
+ env: Record<string, string | undefined> = process.env,
108
+ ): Promise<number> {
109
+ let parsed: ShipDeploymentArguments
110
+ try {
111
+ parsed = parseShipDeployment(args)
112
+ } catch (error) {
113
+ writeError(`${error instanceof Error ? error.message : String(error)}\n${USAGE}`)
114
+ return 2
115
+ }
116
+ try {
117
+ await assertArchiveMatchesDistribution(await findArchive(parsed.root))
118
+ await ensureDistributionArchiveInTrace({
119
+ root: parsed.root,
120
+ distDir: parsed.distDir,
121
+ log: write,
122
+ })
123
+ const selection = await selectDeploymentPlugins(parsed.root, parsed.plugins)
124
+ write(
125
+ selection.packages.length === 0
126
+ ? 'no provider plugins selected'
127
+ : `provider plugins (${selection.rule}): ${selection.packages.join(', ')}`,
128
+ )
129
+ await shipProviderPlugins({
130
+ cwd: parsed.root,
131
+ distDir: parsed.distDir,
132
+ packages: selection.packages,
133
+ rule: selection.rule,
134
+ restoreInstall: restoresInstallIn(env),
135
+ log: write,
136
+ })
137
+ return 0
138
+ } catch (error) {
139
+ writeError(error instanceof Error ? error.message : String(error))
140
+ return 1
141
+ }
142
+ }
143
+
39
144
  if (import.meta.main) {
40
145
  const args = process.argv.slice(2)
41
146
  if (args[0] === 'pack-distribution') {
42
147
  process.exitCode = await runPackDistribution(args)
148
+ } else if (args[0] === 'ship-deployment') {
149
+ process.exitCode = await runShipDeployment(args)
43
150
  } else {
44
151
  console.error(`unknown command: ${args[0] ?? '(none)'}\n${USAGE}`)
45
152
  process.exitCode = 2
@@ -0,0 +1,126 @@
1
+ /**
2
+ * Which provider plugins a hosted deployment ships into its `/api/dispatch`
3
+ * function. The first rule that applies decides:
4
+ *
5
+ * 1. `ship-deployment --plugin NAME` flags;
6
+ * 2. the deployment `package.json` field `"autobuild": { "plugins": [...] }`
7
+ * (an explicit `[]` ships none);
8
+ * 3. every `dependencies` entry whose installed `package.json` declares
9
+ * `@defrex/autobuild` in `peerDependencies` — the plugin-authoring
10
+ * convention — except the Autobuild service packages.
11
+ *
12
+ * Selection only names packages; the shipper fails the build when a selected
13
+ * package cannot be resolved, bundled, or evaluated to a plugin manifest.
14
+ */
15
+ import { readFile } from 'node:fs/promises'
16
+ import { dirname, join, resolve } from 'node:path'
17
+
18
+ export type PluginSelectionRule = 'flag' | 'manifest' | 'dependencies'
19
+
20
+ export interface DeploymentPluginSelection {
21
+ packages: string[]
22
+ rule: PluginSelectionRule
23
+ }
24
+
25
+ /** Packages that peer on `@defrex/autobuild` but are Autobuild's own service
26
+ * packages, never provider plugins. */
27
+ export const SERVICE_PACKAGES: readonly string[] = [
28
+ '@defrex/autobuild-hosted-store-service',
29
+ '@defrex/autobuild-hosted-dispatcher',
30
+ '@defrex/autobuild-postgres-store',
31
+ ]
32
+
33
+ /** Human-readable description of the rule that selected a package, used in
34
+ * shipping errors. */
35
+ export function describeSelectionRule(rule: PluginSelectionRule): string {
36
+ switch (rule) {
37
+ case 'flag':
38
+ return 'the --plugin flag'
39
+ case 'manifest':
40
+ return 'package.json autobuild.plugins'
41
+ case 'dependencies':
42
+ return (
43
+ 'the default rule (a dependency peering on @defrex/autobuild); ' +
44
+ 'list plugins explicitly in package.json autobuild.plugins to override'
45
+ )
46
+ }
47
+ }
48
+
49
+ async function readJson(path: string): Promise<Record<string, unknown>> {
50
+ return JSON.parse(await readFile(path, 'utf8')) as Record<string, unknown>
51
+ }
52
+
53
+ /** `node_modules/<name>` found by walking up from `start`, as Node and Bun
54
+ * resolve bare specifiers. */
55
+ export async function findInstalledPackage(start: string, name: string): Promise<string | null> {
56
+ let current = resolve(start)
57
+ while (true) {
58
+ const candidate = join(current, 'node_modules', ...name.split('/'))
59
+ try {
60
+ await readFile(join(candidate, 'package.json'))
61
+ return candidate
62
+ } catch {
63
+ // Not installed here: keep walking up.
64
+ }
65
+ const parent = dirname(current)
66
+ if (parent === current) return null
67
+ current = parent
68
+ }
69
+ }
70
+
71
+ function explicitList(value: unknown, source: string): string[] {
72
+ if (
73
+ !Array.isArray(value) ||
74
+ value.some((entry) => typeof entry !== 'string' || entry.trim() === '') ||
75
+ new Set(value).size !== value.length
76
+ ) {
77
+ throw new Error(`${source} must be an array of unique, non-blank package names`)
78
+ }
79
+ return value as string[]
80
+ }
81
+
82
+ export async function selectDeploymentPlugins(
83
+ projectRoot: string,
84
+ flags: readonly string[] = [],
85
+ ): Promise<DeploymentPluginSelection> {
86
+ if (flags.length > 0) return { packages: explicitList([...flags], '--plugin'), rule: 'flag' }
87
+ const manifestPath = join(projectRoot, 'package.json')
88
+ let manifest: Record<string, unknown>
89
+ try {
90
+ manifest = await readJson(manifestPath)
91
+ } catch (error) {
92
+ throw new Error(
93
+ `cannot read the deployment manifest ${manifestPath}: ` +
94
+ (error instanceof Error ? error.message : String(error)),
95
+ )
96
+ }
97
+ const autobuild = manifest.autobuild
98
+ if (autobuild !== undefined) {
99
+ if (typeof autobuild !== 'object' || autobuild === null || Array.isArray(autobuild)) {
100
+ throw new Error(`${manifestPath}: "autobuild" must be an object`)
101
+ }
102
+ const plugins = (autobuild as Record<string, unknown>).plugins
103
+ if (plugins !== undefined) {
104
+ return {
105
+ packages: explicitList(plugins, `${manifestPath}: "autobuild.plugins"`),
106
+ rule: 'manifest',
107
+ }
108
+ }
109
+ }
110
+ const dependencies = manifest.dependencies
111
+ const names =
112
+ typeof dependencies === 'object' && dependencies !== null ? Object.keys(dependencies) : []
113
+ const packages: string[] = []
114
+ for (const name of names.sort()) {
115
+ if (SERVICE_PACKAGES.includes(name)) continue
116
+ const directory = await findInstalledPackage(projectRoot, name)
117
+ // An uninstalled dependency cannot be recognized as a plugin.
118
+ if (directory === null) continue
119
+ const installed = await readJson(join(directory, 'package.json')).catch(() => undefined)
120
+ const peers = installed?.peerDependencies
121
+ if (typeof peers === 'object' && peers !== null && '@defrex/autobuild' in peers) {
122
+ packages.push(name)
123
+ }
124
+ }
125
+ return { packages, rule: 'dependencies' }
126
+ }
@@ -4,7 +4,7 @@
4
4
  * ship in the hosted deployment.
5
5
  *
6
6
  * Next 16's default Turbopack builds never apply `outputFileTracingIncludes`
7
- * (see docs/hosted-dispatcher.md), so a `next build` alone cannot be relied on
7
+ * (see this package's README), so a `next build` alone cannot be relied on
8
8
  * to carry `.autobuild-dist/autobuild-<version>.tgz` into the `/api/dispatch`
9
9
  * function bundle. This post-build step appends the archive to the dispatch
10
10
  * route's `route.js.nft.json` — the trace file Vercel's Next builder consumes
@@ -12,7 +12,7 @@
12
12
  * Next itself computes `outputFileTracingIncludes` entries (relative from the
13
13
  * trace file's directory). It fails loudly when pack-distribution has not run
14
14
  * or the trace file is missing, so a deployment can never silently ship
15
- * without the archive.
15
+ * without the archive. `ab-hosted-dispatcher ship-deployment` runs it.
16
16
  *
17
17
  * It appends the distribution manifest the same way. Provisioning reads the
18
18
  * running version from the repository root's package.json, which no import
@@ -62,7 +62,8 @@ const ROUTE_DIRECTORY = join('server', 'app', 'api', 'dispatch')
62
62
  /** The operator HTTP route; retained legacy skill trace entry. */
63
63
  const OPERATOR_ROUTE_DIRECTORY = join('server', 'app', 'operator', '[[...path]]')
64
64
 
65
- async function findArchive(root: string): Promise<string> {
65
+ /** The single packed archive under `<root>/.autobuild-dist/`. */
66
+ export async function findArchive(root: string): Promise<string> {
66
67
  const dist = join(root, DIST_DIRECTORY)
67
68
  let entries: string[]
68
69
  try {
@@ -70,14 +71,14 @@ async function findArchive(root: string): Promise<string> {
70
71
  } catch {
71
72
  throw new Error(
72
73
  `no ${DIST_DIRECTORY}/ directory under ${root} — run ` +
73
- '`bun ../../packages/hosted-dispatcher/src/bin.ts pack-distribution --root .` before building',
74
+ '`ab-hosted-dispatcher pack-distribution --root <project>` before building',
74
75
  )
75
76
  }
76
77
  const archives = entries.filter((name) => /^autobuild-.+\.tgz$/.test(name)).sort()
77
78
  if (archives.length === 0) {
78
79
  throw new Error(
79
80
  `no autobuild-*.tgz in ${DIST_DIRECTORY}/ — run ` +
80
- '`bun ../../packages/hosted-dispatcher/src/bin.ts pack-distribution --root .` before building',
81
+ '`ab-hosted-dispatcher pack-distribution --root <project>` before building',
81
82
  )
82
83
  }
83
84
  if (archives.length > 1) {
@@ -96,7 +97,7 @@ async function readTrace(path: string): Promise<string[]> {
96
97
  } catch {
97
98
  throw new Error(
98
99
  `missing Next.js trace file ${path} — run this tool after \`next build\`; ` +
99
- 'if the dispatch route was renamed, update packages/hosted-dispatcher/src/ship-packed-distribution.ts',
100
+ 'if the dispatch route was renamed, update ship-packed-distribution.ts in @defrex/autobuild-hosted-dispatcher',
100
101
  )
101
102
  }
102
103
  const parsed: unknown = JSON.parse(raw)
@@ -1,72 +1,90 @@
1
- #!/usr/bin/env bun
2
1
  /**
3
- * Ship the provider plugin package into the hosted deployment (AUT-517).
2
+ * Ship provider plugin packages into the hosted deployment's `/api/dispatch`
3
+ * function (AUT-517, defrex/autobuild#820).
4
4
  *
5
5
  * A bare package specifier resolves only through a `node_modules/` path, so
6
- * appending the plugin's source files to the dispatch route's trace alone
7
- * cannot make `@defrex/autobuild-vercel-sandbox` resolvable inside the
8
- * deployed `/api/dispatch` function (build extract-the-vercel,
9
- * finding f_0b59ab99). The runtime resolution root is `distributionRoot()`,
10
- * which — derived from this repository's own compiled Turbopack output — is
11
- * the REPOSITORY ROOT, not the Next project directory: the chunk holding
12
- * `distribution.ts` synthesizes `import.meta.url` from the source path and
13
- * resolves its absolute root six file-relative ups from the runtime chunk
14
- * (`<repoRoot>/packages/hosted-store-service/.next/server/chunks/`), which is
15
- * the repository root. `loadPlugins` therefore resolves bare specifiers
16
- * against `<repoRoot>/node_modules`.
6
+ * appending a plugin's source files to the dispatch route's trace alone cannot
7
+ * make it resolvable inside the deployed function (finding f_0b59ab99). At
8
+ * runtime `loadPlugins` falls back to the installation root,
9
+ * `distributionRoot()` — the installed `@defrex/autobuild` directory — and
10
+ * walks up from it looking for `node_modules/<pkg>`; the first directory it
11
+ * checks is `<distributionRoot>/node_modules/<pkg>`. This step stages a
12
+ * self-contained `bun build` bundle of each plugin exactly there and appends
13
+ * the staged files to the dispatch route's `route.js.nft.json`, relative to
14
+ * the trace file's directory. The bundle inlines the plugin-sdk closure and
15
+ * every dependency; node builtins stay external.
17
16
  *
18
- * The deployment's repo-root `node_modules` carries no `@defrex/autobuild`
19
- * either (the trace has zero `@defrex` entries — core is Turbopack-inlined),
20
- * so this step stages a self-contained `bun build` bundle of the plugin: the
21
- * bundle inlines the plugin-sdk closure (the shared capability object, its
22
- * validators, and transitive dependencies including `@vercel/sandbox`) and
23
- * evaluates with zero external module imports; node builtins stay external.
17
+ * In this source checkout `distributionRoot()` is the repository root, so the
18
+ * staging path is the workspace symlink `bun install` left in the root
19
+ * `node_modules`; it is replaced and, on a local run, put back. In an
20
+ * installed deployment the staging path is a nested directory `bun install`
21
+ * never creates, so the consumer's installed package is never touched.
24
22
  *
25
- * Staging targets `<repoRoot>/node_modules/@defrex/autobuild-vercel-sandbox/`
26
- * and every staged file is appended to the dispatch route's
27
- * `route.js.nft.json` computed relative to the trace file's directory —
28
- * seven `'..'` components up to the repository root (what `path.relative()`
29
- * yields from `…/.next/server/app/api/dispatch`), the same depth class as
30
- * the trace's existing `node_modules/.bun/**` entries. It fails loudly when the package, bundle
31
- * input, or trace is missing, so a deployment can never silently ship
32
- * without the plugin. Idempotent ACROSS PROCESSES: the bundle input is
33
- * located through the workspace tree, never through `node_modules`, so a
34
- * re-run in a fresh process still finds the source after a previous run
35
- * replaced the workspace symlink with the staged directory (finding
36
- * f_5b9ade16).
23
+ * Each staged `package.json` carries `"autobuildStagedBundle": true`. The
24
+ * plugin's source is found from the deployment's own installation
25
+ * (`node_modules/<pkg>` walking up from the project). Only the source
26
+ * checkout can resolve onto a staged bundle — a Vercel-cached re-run after
27
+ * staging replaced the workspace symlink (finding f_5b9ade16) — and then the
28
+ * source is located through the root `workspaces` tree instead. Bundles
29
+ * staged before the marker existed are recognized by their two-file shape,
30
+ * but only at the staging path: elsewhere that shape is an ordinary plugin.
37
31
  *
38
- * The npm published package remains the real `src` — only the deployment
39
- * stages the bundle. In a Vercel deployment build the staged directory
40
- * replaces the workspace symlink and persists there (the deployment never
41
- * runs `bun install` again). A developer-local run of `deploy:build` (no
42
- * `VERCEL` environment) removes the staged directory afterwards and restores
43
- * the workspace symlink, so a local run leaves the checkout exactly as
44
- * `bun install` had it and local test runs keep resolving the workspace
45
- * source. Staged files are build-time artifacts under `node_modules`
32
+ * Every staged bundle is imported once at build time and must evaluate to a
33
+ * valid plugin manifest; a failure at any step fails the build naming the
34
+ * package. A deployment build (`VERCEL` set) keeps the staged bundle — the
35
+ * function needs it. A developer-local run restores the installation as
36
+ * `bun install` had it. Staged files are build artifacts under `node_modules`
46
37
  * (gitignored, never committed).
47
38
  */
48
- import { lstat, mkdir, readdir, readFile, rm, symlink, writeFile } from 'node:fs/promises'
39
+ import {
40
+ lstat,
41
+ mkdir,
42
+ readdir,
43
+ readFile,
44
+ readlink,
45
+ realpath,
46
+ rm,
47
+ rmdir,
48
+ symlink,
49
+ writeFile,
50
+ } from 'node:fs/promises'
49
51
  import { dirname, join, relative, resolve } from 'node:path'
52
+ import { pathToFileURL } from 'node:url'
50
53
  import type { Dirent } from 'node:fs'
54
+ import { distributionRoot as installedDistributionRoot } from '@defrex/autobuild/distribution'
55
+ import { parsePluginManifest } from '@defrex/autobuild/plugin-sdk'
56
+ import {
57
+ describeSelectionRule,
58
+ findInstalledPackage,
59
+ type PluginSelectionRule,
60
+ } from './deployment-plugins'
51
61
 
52
- export interface ShipProviderPluginOptions {
53
- /** Working directory the repository root is derived from; deploy:build
54
- * runs in the Next project directory. */
62
+ export interface ShipProviderPluginsOptions {
63
+ /** The Next project directory. */
55
64
  cwd?: string
56
65
  /** Next build output directory under the project directory. */
57
66
  distDir?: string
58
- /** The plugin package specifier to stage. */
59
- packageName?: string
60
- /** Remove the staged directory after the trace append and re-create the
61
- * workspace symlink it replaced. The deployment build (the default) must
62
- * keep the staged bundle — its runtime resolution needs it; only a
63
- * developer-local run (no `VERCEL` environment) should restore the link. */
64
- restoreWorkspaceLink?: boolean
67
+ /** Plugin package specifiers to stage. */
68
+ packages: readonly string[]
69
+ /** The rule that selected `packages`, named in shipping errors. */
70
+ rule?: PluginSelectionRule
71
+ /** The installed `@defrex/autobuild` directory; staging goes under its
72
+ * `node_modules`. Defaults to `distributionRoot()` of this process. */
73
+ distributionRoot?: string
74
+ /** The Next output-file-tracing root; staged files must lie under it.
75
+ * Defaults to `AB_SERVICE_BUILD_ROOT`, else the workspace root above the
76
+ * project, else the project. */
77
+ traceRoot?: string
78
+ /** Restore the installation after the trace append. A deployment build
79
+ * must keep the staged bundles; only a developer-local run restores. */
80
+ restoreInstall?: boolean
65
81
  log?: (message: string) => void
66
82
  }
67
83
 
68
84
  export interface ShipProviderPluginResult {
69
- repoRoot: string
85
+ packageName: string
86
+ /** Directory the bundle was built from (absolute). */
87
+ source: string
70
88
  /** Directory the bundle was staged into (absolute). */
71
89
  staging: string
72
90
  /** Staged files appended to (or already present in) the trace, relative to
@@ -74,21 +92,20 @@ export interface ShipProviderPluginResult {
74
92
  traceEntries: string[]
75
93
  appended: number
76
94
  bytes: number
77
- /** Whether the staged directory was removed and the workspace symlink
78
- * restored after staging (local runs only). */
79
- restoredWorkspaceLink?: boolean
95
+ /** Whether the installation was restored after staging (local runs). */
96
+ restored: boolean
80
97
  }
81
98
 
82
99
  /** True when the run is a developer-local build: Vercel sets `VERCEL` in
83
100
  * every build environment — including a local `vercel build` — so an absent
84
- * `VERCEL` identifies exactly the plain local runs this tool's restore is
85
- * for. */
86
- export function restoresWorkspaceLinkIn(env: Record<string, string | undefined>): boolean {
101
+ * `VERCEL` identifies exactly the plain local runs the restore is for. */
102
+ export function restoresInstallIn(env: Record<string, string | undefined>): boolean {
87
103
  return env.VERCEL === undefined
88
104
  }
89
105
 
90
106
  const TRACE_FILE = 'route.js.nft.json'
91
107
  const ROUTE_DIRECTORY = join('server', 'app', 'api', 'dispatch')
108
+ const STAGED_MARKER = 'autobuildStagedBundle'
92
109
 
93
110
  /** Nearest ancestor directory whose package.json declares `workspaces` — the
94
111
  * same root manifest `readWorkspaceManifests` reads. Never a hardcoded
@@ -115,14 +132,10 @@ export async function findWorkspaceRoot(start: string): Promise<string> {
115
132
  }
116
133
 
117
134
  /** Directory of the plugin package's SOURCE, located through the repository
118
- * root's `workspaces` tree — never through `node_modules`. Resolving through
119
- * the installation would find the workspace link on the first run but the
120
- * STAGED bundle directory on every later run (staging replaces the symlink
121
- * and `bun install` does not relink a pre-existing directory, so the checkout
122
- * never self-heals), making the second deploy build target the staged
123
- * directory's nonexistent `src/` (finding f_5b9ade16). The workspace tree is
124
- * stable across runs. Pattern support mirrors tools/workspace-manifest-check.ts:
125
- * `directory/*` globs only. */
135
+ * root's `workspaces` tree. The fallback for a source checkout whose
136
+ * `node_modules` link was replaced by a staged bundle (finding f_5b9ade16);
137
+ * pattern support mirrors tools/workspace-manifest-check.ts: `directory/*`
138
+ * globs only. */
126
139
  async function resolvePackageSourceDirectory(
127
140
  repoRoot: string,
128
141
  packageName: string,
@@ -183,6 +196,60 @@ async function resolvePackageSourceDirectory(
183
196
  )
184
197
  }
185
198
 
199
+ /** Whether `directory` holds a bundle a previous run staged: the marker, or,
200
+ * with `legacyShape`, the exact two-file shape bundles had before the marker
201
+ * existed. Only the staging path itself may be read by shape — an installed
202
+ * plugin elsewhere can legitimately ship exactly those two files. */
203
+ async function isStagedBundle(directory: string, legacyShape: boolean): Promise<boolean> {
204
+ let manifest: Record<string, unknown>
205
+ try {
206
+ manifest = JSON.parse(await readFile(join(directory, 'package.json'), 'utf8'))
207
+ } catch {
208
+ return false
209
+ }
210
+ if (manifest[STAGED_MARKER] === true) return true
211
+ if (!legacyShape) return false
212
+ const exports = manifest.exports as Record<string, unknown> | undefined
213
+ if (exports?.['.'] !== './dist/index.js') return false
214
+ const entries = (await readdir(directory).catch(() => [])).sort()
215
+ const dist = await readdir(join(directory, 'dist')).catch(() => [])
216
+ return entries.join(',') === 'dist,package.json' && dist.join(',') === 'index.js'
217
+ }
218
+
219
+ /** The package's import entry: `exports["."]` (the `import` condition, then
220
+ * `default`), else `module`, else `main`. */
221
+ async function entryFile(directory: string, packageName: string): Promise<string> {
222
+ const manifest = JSON.parse(await readFile(join(directory, 'package.json'), 'utf8')) as {
223
+ exports?: unknown
224
+ module?: unknown
225
+ main?: unknown
226
+ }
227
+ let target: unknown
228
+ const root =
229
+ typeof manifest.exports === 'object' && manifest.exports !== null
230
+ ? (manifest.exports as Record<string, unknown>)['.']
231
+ : manifest.exports
232
+ if (typeof root === 'string') target = root
233
+ else if (typeof root === 'object' && root !== null) {
234
+ const conditions = root as Record<string, unknown>
235
+ target = conditions.import ?? conditions.default
236
+ }
237
+ target ??= manifest.module ?? manifest.main
238
+ if (typeof target !== 'string' || target === '') {
239
+ throw new Error(`${join(directory, 'package.json')}: ${packageName} declares no import entry`)
240
+ }
241
+ const path = resolve(directory, target)
242
+ if (relative(directory, path).startsWith('..')) {
243
+ throw new Error(`${packageName}'s entry ${target} lies outside the package`)
244
+ }
245
+ try {
246
+ await lstat(path)
247
+ } catch {
248
+ throw new Error(`${packageName}'s entry ${path} does not exist`)
249
+ }
250
+ return path
251
+ }
252
+
186
253
  /** Single self-contained ESM bundle of the plugin's entrypoint for the Bun
187
254
  * runtime. The plugin-sdk closure inlines; node builtins stay external. */
188
255
  async function buildBundle(entry: string): Promise<string> {
@@ -212,8 +279,8 @@ async function traceFiles(trace: string): Promise<string[]> {
212
279
  raw = await readFile(trace, 'utf8')
213
280
  } catch {
214
281
  throw new Error(
215
- `missing Next.js trace file ${trace} — run this tool after \`next build\`; ` +
216
- 'if the dispatch route was renamed, update packages/hosted-dispatcher/src/ship-provider-plugin.ts',
282
+ `missing Next.js trace file ${trace} — run this step after \`next build\`; ` +
283
+ 'if the dispatch route was renamed, update ship-provider-plugin.ts in @defrex/autobuild-hosted-dispatcher',
217
284
  )
218
285
  }
219
286
  const parsed: unknown = JSON.parse(raw)
@@ -227,97 +294,164 @@ async function traceFiles(trace: string): Promise<string[]> {
227
294
  return (parsed as { files: string[] }).files
228
295
  }
229
296
 
230
- /** Bundle the provider plugin, stage it into the repository-root
231
- * `node_modules`, and append the staged files to the dispatch route's trace. */
232
- export async function shipProviderPlugin({
297
+ /** The default tracing root: `AB_SERVICE_BUILD_ROOT` (what the published
298
+ * build passes Next), else the workspace root above the project, else the
299
+ * project itself. */
300
+ export async function defaultTraceRoot(
301
+ cwd: string,
302
+ env: Record<string, string | undefined> = process.env,
303
+ ): Promise<string> {
304
+ const explicit = env.AB_SERVICE_BUILD_ROOT?.trim()
305
+ if (explicit) return resolve(explicit)
306
+ return findWorkspaceRoot(cwd).catch(() => resolve(cwd))
307
+ }
308
+
309
+ async function exists(path: string): Promise<boolean> {
310
+ return lstat(path).then(
311
+ () => true,
312
+ () => false,
313
+ )
314
+ }
315
+
316
+ /** Bundle each plugin, stage it under the installed distribution's
317
+ * `node_modules`, validate it, and append the staged files to the dispatch
318
+ * route's trace. */
319
+ export async function shipProviderPlugins({
233
320
  cwd = process.cwd(),
234
321
  distDir = '.next',
235
- packageName = '@defrex/autobuild-vercel-sandbox',
236
- restoreWorkspaceLink = false,
322
+ packages,
323
+ rule = 'flag',
324
+ distributionRoot = installedDistributionRoot(),
325
+ traceRoot,
326
+ restoreInstall = false,
237
327
  log = (message) => console.log(message),
238
- }: ShipProviderPluginOptions = {}): Promise<ShipProviderPluginResult> {
239
- const repoRoot = await findWorkspaceRoot(cwd)
240
- const packageDirectory = await resolvePackageSourceDirectory(repoRoot, packageName)
241
- const packageManifest = JSON.parse(
242
- await readFile(join(packageDirectory, 'package.json'), 'utf8'),
243
- ) as { name?: unknown; version?: unknown }
244
- if (typeof packageManifest.name !== 'string' || typeof packageManifest.version !== 'string') {
245
- throw new Error(`${join(packageDirectory, 'package.json')} must declare name and version`)
246
- }
247
-
248
- const bundle = await buildBundle(join(packageDirectory, 'src', 'index.ts'))
328
+ }: ShipProviderPluginsOptions): Promise<ShipProviderPluginResult[]> {
329
+ const project = await realpath(cwd)
330
+ const tracing = await realpath(traceRoot ?? (await defaultTraceRoot(project)))
331
+ const installation = await realpath(distributionRoot)
332
+ const trace = join(project, distDir, ROUTE_DIRECTORY, TRACE_FILE)
333
+ // Fail on a missing trace before touching node_modules.
334
+ await traceFiles(trace)
335
+ const results: ShipProviderPluginResult[] = []
336
+ for (const packageName of packages) {
337
+ const selected = `selected by ${describeSelectionRule(rule)}`
338
+ const staging = join(installation, 'node_modules', ...packageName.split('/'))
339
+ if (relative(tracing, staging).startsWith('..')) {
340
+ throw new Error(
341
+ `cannot ship ${packageName}: its staging directory ${staging} lies outside the ` +
342
+ `output-file-tracing root ${tracing}`,
343
+ )
344
+ }
249
345
 
250
- // Stage into the repository-root node_modules. bun install leaves a
251
- // workspace symlink there for the root devDependency; it must be replaced
252
- // by a real directory (the deployment never runs bun install again). A
253
- // pre-existing real directory — a previous run's staging, which bun install
254
- // does not relink — is replaced too, so re-runs are clean.
255
- const staging = join(repoRoot, 'node_modules', ...packageName.split('/'))
256
- const existing = await lstat(staging).catch(() => undefined)
257
- if (existing !== undefined) {
258
- await rm(staging, { recursive: true })
259
- }
260
- await mkdir(join(staging, 'dist'), { recursive: true })
261
- const stagedManifest = {
262
- name: packageManifest.name,
263
- version: packageManifest.version,
264
- type: 'module',
265
- exports: { '.': './dist/index.js' },
266
- }
267
- const stagedFiles = [
268
- { name: 'package.json', contents: `${JSON.stringify(stagedManifest, null, 2)}\n` },
269
- { name: join('dist', 'index.js'), contents: bundle },
270
- ]
271
- for (const file of stagedFiles) {
272
- await writeFile(join(staging, file.name), file.contents)
273
- }
274
- const bytes = stagedFiles.reduce((total, file) => total + Buffer.byteLength(file.contents), 0)
346
+ // Source: the deployment's own installation, unless that resolves onto a
347
+ // staged bundle (a source checkout re-run).
348
+ const installed = await findInstalledPackage(project, packageName)
349
+ if (installed === null) {
350
+ throw new Error(
351
+ `provider plugin ${packageName} is not installed in the deployment (${selected})`,
352
+ )
353
+ }
354
+ let source = await realpath(installed)
355
+ const installedAtStaging = resolve(installed) === staging
356
+ let healLink = false
357
+ if (await isStagedBundle(source, installedAtStaging)) {
358
+ source = await resolvePackageSourceDirectory(await findWorkspaceRoot(project), packageName)
359
+ healLink = installedAtStaging
360
+ }
361
+ const sourceManifest = JSON.parse(await readFile(join(source, 'package.json'), 'utf8')) as {
362
+ name?: unknown
363
+ version?: unknown
364
+ }
365
+ if (typeof sourceManifest.name !== 'string' || typeof sourceManifest.version !== 'string') {
366
+ throw new Error(`${join(source, 'package.json')} must declare name and version`)
367
+ }
368
+ const bundle = await buildBundle(await entryFile(source, packageName))
275
369
 
276
- const trace = join(cwd, distDir, ROUTE_DIRECTORY, TRACE_FILE)
277
- const files = await traceFiles(trace)
278
- const entries = stagedFiles.map((file) => relative(dirname(trace), join(staging, file.name)))
279
- const missing = entries.filter((entry) => !files.includes(entry))
280
- if (missing.length > 0) {
281
- const parsed = JSON.parse(await readFile(trace, 'utf8'))
282
- await writeFile(
283
- trace,
284
- `${JSON.stringify({ ...parsed, files: [...files, ...missing] }, null, 2)}\n`,
370
+ // What may already sit at the staging path: nothing, a staged bundle, or
371
+ // the source checkout's workspace symlink. Never an installed package.
372
+ const existing = await lstat(staging).catch(() => undefined)
373
+ let linkTarget: string | undefined
374
+ if (existing?.isSymbolicLink()) {
375
+ linkTarget = await readlink(staging)
376
+ } else if (existing !== undefined && !(await isStagedBundle(staging, true))) {
377
+ throw new Error(`refusing to replace installed package at ${staging} (${packageName})`)
378
+ }
379
+ const created = [join(installation, 'node_modules'), dirname(staging)].filter(
380
+ (path, index, all) => all.indexOf(path) === index && path !== staging,
285
381
  )
286
- }
287
- log(
288
- `plugin package ${packageName} (${bytes} bytes, ${stagedFiles.length} file(s)) ` +
289
- `${missing.length > 0 ? `appended ${missing.length} entr${missing.length === 1 ? 'y' : 'ies'} to` : 'already present in'} ${trace}`,
290
- )
291
- if (restoreWorkspaceLink) {
292
- // A local run must leave the checkout as `bun install` had it: the staged
293
- // directory would otherwise satisfy the manifest and keep `bun install`
294
- // from ever relinking the workspace, so later local runs would resolve
295
- // the plugin to the staged bundle instead of the workspace source. The
296
- // link target is computed from the link's OWN directory (a relative
297
- // symlink resolves from the directory containing the link):
298
- // `<repoRoot>/node_modules/@defrex` + `../../packages/...` — exactly what
299
- // `bun install` leaves.
300
- const target = relative(dirname(staging), packageDirectory)
301
- await rm(staging, { recursive: true })
302
- await symlink(target, staging, 'dir')
303
- log(`restored workspace symlink ${staging} -> ${target} (local run, no VERCEL environment)`)
304
- return {
305
- repoRoot,
306
- staging,
307
- traceEntries: entries,
308
- appended: missing.length,
309
- bytes,
310
- restoredWorkspaceLink: true,
382
+ const createdHere: string[] = []
383
+ for (const path of created) if (!(await exists(path))) createdHere.push(path)
384
+ if (existing !== undefined) await rm(staging, { recursive: true, force: true })
385
+
386
+ const stagedManifest = {
387
+ name: sourceManifest.name,
388
+ version: sourceManifest.version,
389
+ type: 'module',
390
+ exports: { '.': './dist/index.js' },
391
+ [STAGED_MARKER]: true,
311
392
  }
312
- }
313
- return { repoRoot, staging, traceEntries: entries, appended: missing.length, bytes }
314
- }
393
+ const stagedFiles = [
394
+ { name: 'package.json', contents: `${JSON.stringify(stagedManifest, null, 2)}\n` },
395
+ { name: join('dist', 'index.js'), contents: bundle },
396
+ ]
397
+ try {
398
+ await mkdir(join(staging, 'dist'), { recursive: true })
399
+ for (const file of stagedFiles) await writeFile(join(staging, file.name), file.contents)
400
+ const bytes = stagedFiles.reduce((total, file) => total + Buffer.byteLength(file.contents), 0)
315
401
 
316
- if (import.meta.main) {
317
- try {
318
- await shipProviderPlugin({ restoreWorkspaceLink: restoresWorkspaceLinkIn(process.env) })
319
- } catch (error) {
320
- console.error(error instanceof Error ? error.message : error)
321
- process.exit(1)
402
+ let namespace: { default?: unknown }
403
+ try {
404
+ namespace = await import(pathToFileURL(join(staging, 'dist', 'index.js')).href)
405
+ parsePluginManifest(namespace.default)
406
+ } catch (error) {
407
+ throw new Error(
408
+ `${packageName} is not an Autobuild plugin (${selected}): ` +
409
+ (error instanceof Error ? error.message : String(error)),
410
+ )
411
+ }
412
+
413
+ const files = await traceFiles(trace)
414
+ const entries = stagedFiles.map((file) => relative(dirname(trace), join(staging, file.name)))
415
+ const missing = entries.filter((entry) => !files.includes(entry))
416
+ if (missing.length > 0) {
417
+ const parsed = JSON.parse(await readFile(trace, 'utf8'))
418
+ await writeFile(
419
+ trace,
420
+ `${JSON.stringify({ ...parsed, files: [...files, ...missing] }, null, 2)}\n`,
421
+ )
422
+ }
423
+ log(
424
+ `plugin package ${packageName} (${bytes} bytes, ${stagedFiles.length} file(s)) staged at ${staging}; ` +
425
+ `${missing.length > 0 ? `appended ${missing.length} entr${missing.length === 1 ? 'y' : 'ies'} to` : 'already present in'} ${trace}`,
426
+ )
427
+ results.push({
428
+ packageName,
429
+ source,
430
+ staging,
431
+ traceEntries: entries,
432
+ appended: missing.length,
433
+ bytes,
434
+ restored: false,
435
+ })
436
+ } finally {
437
+ if (restoreInstall) {
438
+ // A local run leaves the installation as `bun install` had it: the
439
+ // workspace symlink back in place, or no trace of the staging.
440
+ await rm(staging, { recursive: true, force: true })
441
+ if (linkTarget !== undefined || healLink) {
442
+ const target = linkTarget ?? relative(dirname(staging), source)
443
+ await symlink(target, staging, 'dir')
444
+ log(
445
+ `restored workspace symlink ${staging} -> ${target} (local run, no VERCEL environment)`,
446
+ )
447
+ } else {
448
+ for (const path of createdHere.reverse()) await rmdir(path).catch(() => {})
449
+ log(`removed staged ${staging} (local run, no VERCEL environment)`)
450
+ }
451
+ const last = results.at(-1)
452
+ if (last?.packageName === packageName) last.restored = true
453
+ }
454
+ }
322
455
  }
456
+ return results
323
457
  }