@base44/functions-compiler 0.1.0

package/README.md

@@ -0,0 +1,137 @@
+# @base44/functions-compiler
+
+Production compiler for Base44 backend functions. Takes function sources as
+data and returns a single Cloudflare Workers module — the generated worker
+entry, the Deno shim, the runtime and private-data-source modules, and npm/jsr
+dependency resolution through `@deno/loader` + esbuild.
+
+Two consumers share this one engine:
+
+- the **Base44 CLI**, which compiles locally (and inside the platform's build
+  sandbox) with no network service in the path;
+- apper's **`base44-userapp-bundler`** HTTP service, which keeps its own
+  endpoints, auth, worker pool and telemetry around it.
+
+## Status and scope
+
+Internal to Base44 — published **public** so apper's bundler service can install
+it, but it is not a supported public API: the CLI bundles it at build time so
+end users never install it, and it carries no compatibility promise to anyone
+outside this repo. It compiles functions and nothing else: shard planning, size
+splitting, artifact writing, version creation and deploy all live above it.
+
+The package carries no credentials and reads no configuration of its own. It
+names the environment variables and headers the generated worker will use at
+runtime (`BASE44_*`, `X-Base44-*`) but holds none of their values, and the code
+it ships is the same code already compiled into every deployed user worker.
+
+## What is missing: shards, and whole-app CFW bundles
+
+Today this package compiles **one module per call**. It does not decide which
+functions belong in which module, how large the result may be, or what to do
+when it is too large. That work is still Python in apper's
+`backend/app/cloudflare_functions/` and moves here next — it is what turns
+"compile this set of sources" into "produce the deployable Cloudflare Workers
+bundles for this app".
+
+| Missing piece | Where it lives today | What it has to do |
+|---|---|---|
+| Source assembly | `function_bundle.py` — `cfw_bundle_input`, `collect_reachable_backend_files` | Turn a function's directory plus the shared files it reaches into the `entry` + `files` this package takes, keeping the flat single-file case and the refusal to escape into the frontend tree |
+| Fresh shard planning | `shard_planning.py` — `full_repartition`, `target_shard_count` | Group an app's functions into shards deterministically, and refuse a set that exceeds the supplied product capacity |
+| Size measurement | `cloudflare_wfp_runtime.py` — `measure_bundle_bytes`, `judge_bundle_size` | Raw UTF-8 bytes and level-6 gzip, against Cloudflare's 64 MiB uncompressed limit and our compressed cap |
+| Split on overflow | `cloudflare_wfp_runtime.py` — `_build_shard_with_split` | Halve an oversized multi-function shard deterministically and recompile; fail the build when one function alone is too big |
+| Whole-build validation | apper PR #23460 | Reject a partial result: every declared function in exactly one successful shard, or no build at all |
+
+Only the **fresh-build** slice comes across. Everything that remembers a
+previous deploy stays in apper: incremental shard reuse (it needs the previous
+deployment map), the per-app `shard_size_override` ratchet, entitlement and
+settings reads, provider upload, binding resolution and secret delivery. Policy
+numbers — shard size, shard count, the gzip cap, whether the gate is enforced —
+arrive as inputs; the package never reads them itself.
+
+Two details from the Python that must survive the port, both verified against
+the current code:
+
+- The single-shard path builds in caller order while the multi-shard path sorts
+  by name. Function order changes the emitted bytes, so both branches carry
+  over as they are — normalising them is a byte change dressed as a cleanup.
+- Capacity is judged at the *global* shard size while packing may use a smaller
+  ratcheted one, so a legal plan can hold more shards than `max_shards`. A
+  final "shard count ≤ max_shards" assertion would lock out apps that deploy
+  fine today.
+
+Also deliberately out of scope for the new lane: Deno deployment targets,
+existing-Worker reuse, incremental deploy state, and actors — though the engine
+keeps its actor support for the legacy service that still uses it.
+
+## Using it
+
+```ts
+import { bundle, bundleApp } from "@base44/functions-compiler";
+
+const result = await bundle({
+  entry: "main.ts",
+  files: { "main.ts": "export default { fetch: () => new Response('hi') }" },
+});
+if (result.ok) console.log(result.module);
+```
+
+`bundleApp` compiles several functions into one combined module and reports
+per-function status; a successful response can still contain failed functions,
+so a caller that needs a whole-app build must reject those itself.
+
+### Process isolation
+
+`installFetchGuard()` blocks outbound `fetch` other than the dependency
+resolution the compiler itself performs. Install it in the worker or child
+process that runs a compile — not in a process that later needs the network.
+
+### Diagnostics
+
+The compiler ships no tracer and no service credentials. Hosts that want spans
+or structured logs register their own, in the thread that runs the compile:
+
+```ts
+import { setCompilerTracer, setLogSink } from "@base44/functions-compiler";
+
+setCompilerTracer({ withSpan, setSpanTags }); // e.g. a dd-trace adapter
+setLogSink((level, event, fields) => myLogger[level](event, fields));
+```
+
+Without a sink, `logEvent` writes the Datadog-shaped JSON line it always has.
+
+## Layout
+
+| Path | What it is |
+|---|---|
+| `src/bundler.ts` | Single-function and combined-app compilation, error attribution |
+| `src/deno-bundle.ts`, `src/esbuild/` | esbuild execution and the resolver/virtual-module plugins |
+| `src/worker-entry.ts`, `src/actor-compat.ts` | Generated worker entry and the Actor wrapper |
+| `src/shim/`, `src/static-egress.ts` | Shim sources — esbuild inputs for `build:shim`, never imported |
+| `src/runtime/`, `src/runtime-context.ts`, `src/private-data-sources/` | Compile-time assets read as **text** and injected into the user bundle |
+
+The compile-time assets must stay TypeScript: the virtual plugins load them
+with esbuild's `ts` loader. `scripts/copy-assets.ts` copies them into `lib/`
+next to the compiled JS so the published package resolves them the same way.
+
+**Do not reformat an asset.** Their text goes into the user's bundle, so
+whitespace is part of the emitted worker bytes — running Biome over
+`src/shim/`, `src/private-data-sources/`, `src/runtime/`, `runtime-context.ts`,
+`static-egress.ts` or `static-egress-marker.ts` changes what every compiled
+function hashes to. `biome.json` excludes those paths for that reason. The
+`test/` directory is likewise held byte-identical to apper's copy until that
+copy is deleted; it sits outside the repo's `packages/*/src` lint glob.
+
+## Commands
+
+```bash
+bun run build:shim   # regenerate dist/{deno-shim,activation-shim,actor}.mjs
+bun run test         # vitest (builds the shims first)
+bun run typecheck    # tsc --noEmit over src/, test/, scripts/
+bun run build        # shims + tsc -> lib/ + assets; what gets published
+```
+
+`exports` points only at `lib/`, so anything consuming this package — including
+a sibling workspace — needs `bun run build` here first. There is deliberately
+no source-resolving export condition: the tarball ships `lib/` alone, and a
+second resolution path would mean two answers to "which code ran".

package/lib/dist/activation-shim.mjs

@@ -0,0 +1,237 @@
+// GENERATED by scripts/build-shim.ts from src/shim/activation.ts — do not edit.
+// Regenerate with `npm run build:shim`.
+
+// src/shim/activation.ts
+import { inflateSync } from "node:zlib";
+import { setRuntimeManifest } from "base44:private-data-sources/runtime-manifest-store";
+var NEEDS_ACTIVATION_HEADER = "X-Base44-Needs-Activation";
+var RUNTIME_SECRETS_HEADER = "Base44-Runtime-Secrets";
+var HKDF_INFO = "base44-runtime-secrets-v1";
+var BACKEND_PUBKEY_LEN = 65;
+var NONCE_LEN = 12;
+var DATA_KEY_LEN = 32;
+var WRAP_BLOCK_LEN = NONCE_LEN + DATA_KEY_LEN + 16;
+var BLOB_BINDING_PREFIX = "BASE44_SECRETS_BLOB";
+var BLOB_KEY_ID_LEN = 1;
+var MANIFEST_KEY = "BASE44_PRIVATE_DATA_SOURCES";
+var _subtle = crypto.subtle;
+var _subtleGenerateKey = _subtle.generateKey.bind(_subtle);
+var _subtleExportKey = _subtle.exportKey.bind(_subtle);
+var _subtleImportKey = _subtle.importKey.bind(_subtle);
+var _subtleDeriveBits = _subtle.deriveBits.bind(_subtle);
+var _subtleDecrypt = _subtle.decrypt.bind(_subtle);
+var _JSONparse = JSON.parse;
+var _TextDecoder = TextDecoder;
+var _TextEncoder = TextEncoder;
+var _decodeUtf8 = TextDecoder.prototype.decode;
+var _encodeUtf8 = TextEncoder.prototype.encode;
+var _Uint8Array = Uint8Array;
+var _atob = atob;
+var _btoa = btoa;
+var _DOMException = DOMException;
+var _ObjectEntries = Object.entries;
+var _Headers = Headers;
+var _Request = Request;
+var _Response = Response;
+var _headersGet = Headers.prototype.get;
+var _headersAppend = Headers.prototype.append;
+var _headersEntries = Headers.prototype.entries;
+var _iterNext = Object.getPrototypeOf(new Headers().entries()).next;
+function accessor(proto, name) {
+  for (let o = proto; o; o = Object.getPrototypeOf(o)) {
+    const d = Object.getOwnPropertyDescriptor(o, name);
+    if (d?.get) return d.get;
+  }
+  return void 0;
+}
+var _reqHeaders = accessor(Request.prototype, "headers");
+var _resHeaders = accessor(Response.prototype, "headers");
+var _resStatus = accessor(Response.prototype, "status");
+var _resStatusText = accessor(Response.prototype, "statusText");
+var _resBody = accessor(Response.prototype, "body");
+var _resWebSocket = accessor(Response.prototype, "webSocket");
+var NEEDS_ACTIVATION_HEADER_LC = "x-base44-needs-activation";
+var RUNTIME_SECRETS_HEADER_LC = "base44-runtime-secrets";
+function headersWithout(src, dropLowercased) {
+  const out = new _Headers();
+  const it = _headersEntries.call(src);
+  for (; ; ) {
+    const step = _iterNext.call(it);
+    if (step.done) break;
+    if (step.value[0] !== dropLowercased) _headersAppend.call(out, step.value[0], step.value[1]);
+  }
+  return out;
+}
+var keyPairPromise = null;
+var installed = false;
+var pendingInstall = null;
+var installChain = Promise.resolve();
+function getKeyPair() {
+  keyPairPromise ??= _subtleGenerateKey(
+    { name: "ECDH", namedCurve: "P-256" },
+    false,
+    ["deriveBits"]
+  ).catch((e) => {
+    keyPairPromise = null;
+    throw e;
+  });
+  return keyPairPromise;
+}
+async function needsActivationResponse() {
+  const { publicKey } = await getKeyPair();
+  const raw = await _subtleExportKey("raw", publicKey);
+  return new Response(null, {
+    status: 503,
+    headers: {
+      [NEEDS_ACTIVATION_HEADER]: toBase64Url(new _Uint8Array(raw)),
+      "Cache-Control": "no-store"
+    }
+  });
+}
+async function install(envelopeB64, env) {
+  const envelope = fromBase64Url(envelopeB64);
+  const backendPub = envelope.slice(0, BACKEND_PUBKEY_LEN);
+  const recipientCount = envelope[BACKEND_PUBKEY_LEN];
+  const { privateKey } = await getKeyPair();
+  const peer = await _subtleImportKey(
+    "raw",
+    backendPub,
+    { name: "ECDH", namedCurve: "P-256" },
+    false,
+    []
+  );
+  const shared = await _subtleDeriveBits({ name: "ECDH", public: peer }, privateKey, 256);
+  const hkdfKey = await _subtleImportKey("raw", shared, "HKDF", false, ["deriveBits"]);
+  const keyBits = await _subtleDeriveBits(
+    {
+      name: "HKDF",
+      hash: "SHA-256",
+      salt: new _Uint8Array(0),
+      info: _encodeUtf8.call(new _TextEncoder(), HKDF_INFO)
+    },
+    hkdfKey,
+    256
+  );
+  const kek = await _subtleImportKey("raw", keyBits, "AES-GCM", false, ["decrypt"]);
+  const appId = env?.BASE44_APP_ID;
+  const aad = _encodeUtf8.call(new _TextEncoder(), typeof appId === "string" ? appId : "");
+  let dataKeyBytes = null;
+  for (let i = 0; i < recipientCount && dataKeyBytes === null; i++) {
+    const blockStart = BACKEND_PUBKEY_LEN + 1 + i * WRAP_BLOCK_LEN;
+    const wrapNonce = envelope.slice(blockStart, blockStart + NONCE_LEN);
+    const wrapped = envelope.slice(blockStart + NONCE_LEN, blockStart + WRAP_BLOCK_LEN);
+    try {
+      dataKeyBytes = await _subtleDecrypt(
+        { name: "AES-GCM", iv: wrapNonce, additionalData: aad },
+        kek,
+        wrapped
+      );
+    } catch {
+    }
+  }
+  if (dataKeyBytes === null) {
+    throw new _DOMException("no wrap block for this isolate", "OperationError");
+  }
+  const bindings = env ?? {};
+  let blobB64 = "";
+  for (let i = 0; ; i++) {
+    const part = bindings[`${BLOB_BINDING_PREFIX}_${i}`];
+    if (typeof part !== "string") break;
+    blobB64 += part;
+  }
+  if (blobB64 === "") {
+    throw new _DOMException("no secret blob bindings on this script", "OperationError");
+  }
+  const blob = fromBase64Url(blobB64);
+  const blobNonce = blob.slice(BLOB_KEY_ID_LEN, BLOB_KEY_ID_LEN + NONCE_LEN);
+  const blobCiphertext = blob.slice(BLOB_KEY_ID_LEN + NONCE_LEN);
+  const dataKey = await _subtleImportKey("raw", dataKeyBytes, "AES-GCM", false, ["decrypt"]);
+  const plaintext = await _subtleDecrypt(
+    { name: "AES-GCM", iv: blobNonce, additionalData: aad },
+    dataKey,
+    blobCiphertext
+  );
+  const payload = _JSONparse(
+    _decodeUtf8.call(new _TextDecoder(), inflateSync(new _Uint8Array(plaintext)))
+  );
+  const secrets = payload && typeof payload === "object" && payload.secrets ? payload.secrets : {};
+  const manifestValue = secrets[MANIFEST_KEY];
+  setRuntimeManifest(typeof manifestValue === "string" ? manifestValue : "");
+  for (const [key, value] of _ObjectEntries(secrets)) {
+    if (key !== MANIFEST_KEY && typeof value === "string") process.env[key] = value;
+  }
+  installed = true;
+}
+function installOnce(envelopeB64, env) {
+  if (installed) return Promise.resolve();
+  const afterPrevious = installChain.then(void 0, () => {
+  });
+  const run = afterPrevious.then(() => installed ? void 0 : install(envelopeB64, env));
+  installChain = run.then(void 0, () => {
+  });
+  pendingInstall = run;
+  const clear = () => {
+    if (pendingInstall === run) pendingInstall = null;
+  };
+  run.then(clear, clear);
+  return run;
+}
+async function ensureActivation(request, env) {
+  const header = _headersGet.call(_reqHeaders.call(request), RUNTIME_SECRETS_HEADER);
+  if (header) {
+    try {
+      await installOnce(header, env);
+      return null;
+    } catch (e) {
+      console.error(
+        `Base44 runtime-secrets activation failed, re-signaling: ${e?.name ?? "error"}`
+      );
+      return needsActivationResponse();
+    }
+  }
+  if (installed) return null;
+  if (pendingInstall) {
+    try {
+      await pendingInstall;
+    } catch {
+    }
+    if (installed) return null;
+  }
+  return needsActivationResponse();
+}
+function withoutRuntimeSecretsHeader(request) {
+  const raw = _reqHeaders.call(request);
+  if (_headersGet.call(raw, RUNTIME_SECRETS_HEADER) === null) return request;
+  return new _Request(request, { headers: headersWithout(raw, RUNTIME_SECRETS_HEADER_LC) });
+}
+function withoutActivationSignal(response) {
+  const raw = _resHeaders.call(response);
+  if (_headersGet.call(raw, NEEDS_ACTIVATION_HEADER) === null) return response;
+  return new _Response(_resBody.call(response), {
+    status: _resStatus.call(response),
+    statusText: _resStatusText.call(response),
+    headers: headersWithout(raw, NEEDS_ACTIVATION_HEADER_LC),
+    // Preserved or a 101 upgrade would fail to reconstruct.
+    webSocket: _resWebSocket ? _resWebSocket.call(response) : void 0
+  });
+}
+function toBase64Url(bytes) {
+  let bin = "";
+  for (const b of bytes) bin += String.fromCharCode(b);
+  return _btoa(bin).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
+}
+function fromBase64Url(value) {
+  const b64 = value.replace(/-/g, "+").replace(/_/g, "/");
+  const padded = b64 + "=".repeat((4 - b64.length % 4) % 4);
+  const bin = _atob(padded);
+  const bytes = new _Uint8Array(bin.length);
+  for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
+  return bytes;
+}
+export {
+  NEEDS_ACTIVATION_HEADER,
+  RUNTIME_SECRETS_HEADER,
+  ensureActivation,
+  withoutActivationSignal,
+  withoutRuntimeSecretsHeader
+};