@quadnix/octo-event-listeners 0.1.0 → 0.2.0

package/README.md

@@ -1,35 +1,122 @@
 # Octo
 
-https://octo.quadnix.com/
+Infrastructure as code written as intent: you describe an app and what it needs, modules turn that
+into Terraform, and your own Terragrunt applies it. Users start at
+[octo.quadnix.com](https://octo.quadnix.com/).
 
-The open-source infrastructure management and CICD tool.
+> Octo is in `0.x`. Version `1.x` will mark it ready for production.
 
-> Disclaimer,
-> This library is still under development and is currently in version `0.x.x`.
-> We plan to release version `1.x.x` when it is deemed ready for production use.
+**This README is a set of procedures for working on the repository.** `docs/ARCHITECTURE.md`
+explains what exists, and `docs/CODING-STANDARDS.md` how to add to it.
 
-# Local Setup
+Requires Node 20 or newer, Terraform or OpenTofu 1.6 or newer, and Terragrunt 0.78 or newer. The
+integration suites also need Docker, and the e2e suites AWS credentials.
 
-## Prerequisites
+---
 
-- [Node.js](https://nodejs.org) 22 (LTS). This ships with Corepack, which provisions the
-  exact package manager the project expects — you do **not** need to install pnpm yourself.
+## A — Set up this machine
 
-## Install
-
-This repository is a [pnpm](https://pnpm.io) workspace. Corepack reads the `packageManager`
-field in the root `package.json` and uses the pnpm version pinned there, so everyone (and CI)
-runs the same version.
+Once.
 
 ```sh
-# One-time per machine: activate Corepack so the `pnpm` command uses the pinned version.
 corepack enable
-
-# Install all workspace dependencies (packages/*).
 pnpm install
+pnpm run prepare
 ```
 
-> **Note:** The documentation site under `apps/octo-docs` is an intentionally separate
-> project — it consumes the published `@quadnix/*` packages from the registry rather than the
-> local source. It is not part of the root workspace and is only installed (with its own
-> `pnpm install`, run from inside that directory) when you are working on the docs.
+`prepare` installs the pre-commit hook, which runs the format check, lint, the build with its type
+check, the circular-dependency check, the secrets spec and the docs code-block check.
+
+## B — Build, lint and test
+
+```sh
+pnpm run build
+pnpm run lint
+pnpm test
+```
+
+`octo-aws-cdk`, `octo-event-listeners`, `octo-build` and `octo-templates` import `octo` from its
+`dist`, and `octo-templates` imports `octo-aws-cdk` the same way. **Rebuild what you changed before
+testing what depends on it**, or the suite runs against the old code and still passes.
+
+`pnpm run build` also type-checks every package, tests included, through each package's
+`check:types`.
+
+`pnpm run format` writes what `pnpm run format:check` checks. Lint does not check formatting.
+
+`pnpm run lint:fix` sorts imports, members and keys. A flag passed through `pnpm run` is written
+without `--`: pnpm forwards the separator, and the script then drops the flag.
+
+## C — Run integration tests
+
+Integration specs run real Terraform; `octo-templates`' start LocalStack in Docker.
+
+```sh
+pnpm run test:integration
+```
+
+## D — Run an e2e suite
+
+**Every e2e suite reaches a live AWS account.** `octo-aws-cdk`'s plan and apply nothing;
+`octo-templates`' create real resources and tear them down.
+
+1. Copy the `.env.defaults` beside the suite to `.env` and fill in real values. `.env` is ignored.
+2. Build the packages the suite imports:
+
+   ```sh
+   npx nx run-many -t build --projects=octo,octo-event-listeners,octo-aws-cdk
+   ```
+
+3. Run one spec, by path, from its package:
+
+   ```sh
+   cd packages/octo-templates
+   NODE_OPTIONS=--experimental-vm-modules npx jest --runTestsByPath src/<template>/main.e2e-spec.ts
+   ```
+
+Never `pnpm run test:e2e` — it runs every suite live at once. Never combine `-u` with `-t`: jest then
+deletes the snapshots of every test it skipped. A template's own `README.md`, where it has one,
+holds what is particular to it.
+
+## E — Work in a worktree
+
+For a large change or a hands-free agent run:
+
+```sh
+git worktree add ../ai-worktrees/octo-worktree-<branch> -b ai/<branch>
+WORKTREE=../ai-worktrees/octo-worktree-<branch>
+find . -maxdepth 4 -name node_modules -type d ! -path "*/node_modules/*/node_modules" | \
+  while read src; do
+    mkdir -p "$WORKTREE/$(dirname $src)"
+    ln -s "$(pwd)/$src" "$WORKTREE/$src"
+  done
+```
+
+The symlinks let Nx and Jest resolve packages. Run lint and tests from inside the worktree.
+
+## F — Release
+
+1. Merge to `main` with PR titles per `.github/jobs/validate-pr-title.job.sh`. The commit type sets
+   the bump: `chore` releases nothing.
+2. Run the **Manual Release** workflow on `main`. It versions every changed package, tags, writes
+   the changelog, and publishes to npm.
+
+## Environment variables
+
+Everything outside the code that governs how this repository behaves. A suite's `.env` is read
+through `env-smart`, falling back to the `.env.defaults` beside it.
+
+| Variable                                                                                            | Set by                                        | What it does                                                                                                                                                                                                        |
+|-----------------------------------------------------------------------------------------------------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `AWS_ACCOUNT_ID`                                                                                    | a suite's `.env`                              | The account every e2e suite plans or applies into.                                                                                                                                                                  |
+| `AWS_REGION_ID`                                                                                     | `packages/octo-aws-cdk/test/.env`             | The region the module e2es plan into. Template suites hardcode `us-east-1`.                                                                                                                                         |
+| `OCTO_DEFINITION_FILE_PATH`                                                                         | the user, or a `create-app` project's scripts | The `octo.yaml` every lifecycle command boots from. `OctoCliUtility` sets it for the CLI-driven template suites. A relative path is resolved once against the working directory, so `run-action` callbacks find it. |
+| `OCTO_JOURNAL_COMMAND_ID`                                                                           | `JournalEventListener`                        | The running command's id, set for every process it starts, so a `run-action` that Terraform starts mid-apply names that apply as `parentCommandId` in its journal.                                                  |
+| `TF_VAR_OCTO_PROJECT_ROOT`                                                                          | `TerraformUtility`, unless already set        | The working directory, so `terraform.localPath()` paths resolve inside Terragrunt's cache copy of a folder.                                                                                                         |
+| `TG_LOG_CUSTOM_FORMAT`                                                                              | `TerraformUtility`, always                    | Terragrunt's log line layout, its default without the time, since the journal records when each line arrived.                                                                                                       |
+| `TG_LOG_FORMAT`                                                                                     | `TerraformUtility`, always unset              | Unset for every Terragrunt process, so it cannot override `TG_LOG_CUSTOM_FORMAT`.                                                                                                                                   |
+| `TG_PROVIDER_CACHE`                                                                                 | `TerraformUtility`, unless already set        | `1`, so Terragrunt shares one provider cache across folders.                                                                                                                                                        |
+| `TG_TF_PATH`                                                                                        | `TerraformUtility`, from `--terraformBinary`  | Which Terraform or OpenTofu binary Terragrunt runs.                                                                                                                                                                 |
+| `NODE_OPTIONS`                                                                                      | every package's test scripts                  | `--experimental-vm-modules`, which Jest needs to run ESM. Set it yourself when running `jest` directly.                                                                                                             |
+| `OCTO_STAGE`, `OCTO_PHASE`, `OCTO_NAT`, `OCTO_TARGET`, `OCTO_WORK`, `OCTO_CASE`, `OCTO_PROBE_STATE` | you, by hand                                  | Drive the `aws-ecs-server` debugging harnesses; that template's `README.md` describes each.                                                                                                                         |
+| `GH_TOKEN`                                                                                          | a repository secret                           | Checkout and push in **Manual Release**, and the tags and GitHub releases `nx release` creates.                                                                                                                     |

package/dist/index.d.ts

@@ -1,2 +1,2 @@
-export { HtmlReportEventListener } from './html-report.event.listener.js';
+export { JournalEventListener } from './journal.event.listener.js';
 export { LoggingEventListener } from './logging.event.listener.js';

package/dist/index.js

@@ -1,3 +1,3 @@
-export { HtmlReportEventListener } from './html-report.event.listener.js';
+export { JournalEventListener } from './journal.event.listener.js';
 export { LoggingEventListener } from './logging.event.listener.js';
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,uBAAuB,EAAE,MAAM,iCAAiC,CAAC;AAC1E,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AACnE,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC"}

package/dist/index.type.d.ts

@@ -0,0 +1,2 @@
+export type { EventPayloads, JournalEntry, JournalFile, JournalHeader, TerraformLogFile, TerraformLogRecord, } from './journal.event.listener.js';
+export type { RecordedError } from './utilities/journal/journal.utility.js';

package/dist/index.type.js

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=index.type.js.map

package/dist/index.type.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.type.js","sourceRoot":"","sources":["../src/index.type.ts"],"names":[],"mappings":""}

package/dist/journal.event.listener.d.ts

@@ -0,0 +1,317 @@
+import { Event } from '@quadnix/octo';
+import { JournalUtility, type Plain } from './utilities/journal/journal.utility.js';
+/**
+ * Each event a journal records, by its class name, mapped to its payload as the journal holds it. An
+ * event without a payload has none in its entry.
+ *
+ * @group Event Listeners/Journal
+ */
+export type EventPayloads = {
+    [K in keyof typeof JournalUtility.EVENT_CLASSES]: K extends 'StateFilesSavedEvent' ? {
+        /**
+         * Every state file the save wrote, but the journal's own; an entry is recorded only when one is
+         * left. A file copied from a path, or deleted, has no `md5`.
+         */
+        stateFiles: {
+            md5?: string;
+            stateFileName: string;
+        }[];
+    } : Plain<InstanceType<(typeof JournalUtility.EVENT_CLASSES)[K]>['payload']>;
+};
+/**
+ * What every journal file of one command starts with.
+ *
+ * @group Event Listeners/Journal
+ */
+export type JournalHeader = {
+    /**
+     * The CLI command, such as `validate`, or the mode where no command ran it.
+     */
+    command: string;
+    /**
+     * Unix milliseconds when the command started, then 8 random hex: `1767225600000-0a1b2c3d`.
+     */
+    commandId: string;
+    host: string;
+    /**
+     * The version of `@quadnix/octo` the command ran, absent when it could not be read.
+     */
+    octoVersion?: string;
+    /**
+     * The command that started this one, such as the `apply` a `run-action` ran inside.
+     */
+    parentCommandId?: string;
+    /**
+     * When the command started, in Unix milliseconds.
+     */
+    startedAt: number;
+    /**
+     * The format of the journal files.
+     */
+    version: 1;
+};
+/**
+ * One event of a command, as its journal holds it. Narrow on `type` to read its payload.
+ *
+ * @group Event Listeners/Journal
+ */
+export type JournalEntry = {
+    [K in keyof EventPayloads]: {
+        /**
+         * The event's subject, where its type alone does not say, such as a module's class name; empty
+         * otherwise.
+         */
+        name: string;
+        /**
+         * The event's place among every event the command's process created, shared with the terraform log.
+         */
+        sequence: number;
+        /**
+         * When the event was created, in Unix milliseconds.
+         */
+        time: number;
+        type: K;
+    } & (EventPayloads[K] extends undefined ? {
+        payload?: undefined;
+    } : {
+        payload: EventPayloads[K];
+    });
+}[keyof EventPayloads];
+/**
+ * `journal.<commandId>.<command>.json`: every event of one command, in the order it happened.
+ *
+ * Beside it, `journal.<commandId>.<command>.<md5>.json` keeps each version of `models.json`,
+ * `resources-actual.json`, `resources-old.json` or an external resource's record the command wrote,
+ * named by the MD5 of its contents, as its `StateFilesSavedEvent` entry names it.
+ *
+ * @group Event Listeners/Journal
+ */
+export type JournalFile = JournalHeader & {
+    entries: JournalEntry[];
+};
+/**
+ * One record of a terraform log: a process starting, a chunk it wrote, or its exit. `process` tells
+ * apart the command's processes, and `sequence` places the record among the command's events.
+ *
+ * @group Event Listeners/Journal
+ */
+export type TerraformLogRecord = {
+    args: string[];
+    binary: string;
+    dir: string;
+    process: number;
+    sequence: number;
+    time: number;
+    type: 'started';
+} | {
+    process: number;
+    sequence: number;
+    stream: 'stderr' | 'stdout';
+    text: string;
+    time: number;
+    type: 'output';
+} | {
+    exitCode: number;
+    process: number;
+    sequence: number;
+    time: number;
+    type: 'exited';
+};
+/**
+ * `journal.<commandId>.<command>.terraform.json`: everything terraform and terragrunt wrote during one
+ * command, chunk by chunk, in the order it arrived.
+ *
+ * @group Event Listeners/Journal
+ */
+export type TerraformLogFile = JournalHeader & {
+    outputs: TerraformLogRecord[];
+};
+/**
+ * Writes the journal: every event of this command, in the order it happened, into Octo's state.
+ *
+ * A command is one process, named after the CLI command it runs, or the mode where no command runs
+ * it. Its events go to `journal.<commandId>.<command>.json`, and what terraform and terragrunt wrote to
+ * `journal.<commandId>.<command>.terraform.json`, each chunk as it arrives.
+ * A copy of every model and resource state file written is kept as `journal.<commandId>.<command>.<md5>.json`,
+ * so each version a command wrote survives the next command overwriting it.
+ * Ids start with the time they were made, so files sort in the order their commands started.
+ *
+ * The listener only creates files: {@link StateManagementService} archives them when the lock is
+ * released. A command that runs under a lock, run when none is held, writes nothing, because its
+ * files would belong to no lock. They are written with every other state file written, at the end of a command,
+ * and before the lock is released, after which nothing more is recorded. It never fails a command: a write that
+ * fails is reported on stderr. The one write that fails loudly is the one before the release, which
+ * keeps the lock held.
+ *
+ * The terraform log is written to a temporary file as each chunk arrives, and holds nothing in memory.
+ * That file is valid JSON after every chunk. Each state save takes a copy of it on disk and hands the
+ * state provider that copy's path, so an apply's output reaches the state provider at every terraform
+ * state check-in without ever being read into memory. It is deleted when the command ends.
+ *
+ * A command that another one started, such as a `run-action` terraform starts mid-apply, names that
+ * command as `parentCommandId`.
+ *
+ * @group Event Listeners
+ */
+export declare class JournalEventListener {
+    /**
+     * The CLI command, or the mode, this process runs. Undefined until one begins.
+     */
+    private command;
+    /**
+     * A UTC timestamp to the millisecond, then 8 random hex, so ids sort by when commands started.
+     */
+    private commandId;
+    /**
+     * Every event recorded so far.
+     */
+    private entries;
+    /**
+     * Whether an `octo` command announced itself, which then decides when the journal is written. A
+     * driver that announces no command, such as `runModules`, runs each mode as a command of its own.
+     */
+    private isInCommand;
+    /**
+     * Whether the lock was released, after which nothing is recorded or written.
+     */
+    private isReleased;
+    /**
+     * Whether this command writes at all, settled at its first write.
+     */
+    private isWritten;
+    /**
+     * The version of `@quadnix/octo` this command uses.
+     */
+    private readonly octoVersion;
+    /**
+     * The state file lists this listener saves itself, which it adds nothing to.
+     */
+    private readonly ownStateFiles;
+    /**
+     * The command of the process that started this one, when there is one.
+     */
+    private readonly parentCommandId;
+    /**
+     * The last write queued, so the next one waits for it.
+     */
+    private pending;
+    /**
+     * State file contents not copied yet, by MD5.
+     */
+    private readonly pendingCopies;
+    /**
+     * When this command started, in Unix milliseconds.
+     */
+    private startedAt;
+    /**
+     * The terraform log file this command is writing, opened at its first record.
+     */
+    private terraformLog;
+    /**
+     * The last change to the terraform log queued, so the next one waits for it.
+     */
+    private terraformLogQueue;
+    /**
+     * The snapshots of the terraform log a save carries and has not finished with.
+     */
+    private readonly terraformLogSnapshotPaths;
+    /**
+     * The MD5 of every copy this command already wrote.
+     */
+    private readonly writtenCopies;
+    /**
+     * Names this command to every process it starts.
+     */
+    constructor();
+    /**
+     * A new command id: the Unix time in milliseconds, then 8 random hex.
+     */
+    private static createCommandId;
+    /**
+     * Records every event but registrations and serializations, adds the journal to every state save,
+     * and writes it at the end of a command, or a mode outside one, and before the lock is released.
+     */
+    onEvent(event: Event<unknown>): Promise<void>;
+    /**
+     * Adds a record to the terraform log, opening it at the first. Each is written over the end of the
+     * file, followed by the end again, so the file stays valid JSON. Nothing is recorded before a command
+     * is named, since a journal is only written for a command.
+     */
+    private appendTerraformLog;
+    /**
+     * What every journal file of this command starts with.
+     */
+    private buildHeader;
+    /**
+     * The command's journal, its terraform log, and the state file copies not written yet. Empty before
+     * a command begins.
+     */
+    private buildStateFiles;
+    /**
+     * Closes the terraform log and deletes it, once every record queued is written.
+     */
+    private closeTerraformLog;
+    /**
+     * The name of one of this command's journal files.
+     */
+    private fileName;
+    /**
+     * The state management service, or `undefined` when no state provider was booted, and so there is
+     * nowhere to write.
+     */
+    private getStateManagementService;
+    /**
+     * Holds the contents of each model and resource state file, and each external resource record, a
+     * save wrote, until a journal write copies them. Marks the copies the save itself carried as written.
+     */
+    private holdCopies;
+    /**
+     * Whether this command writes at all. One that runs under a lock writes nothing when no lock is
+     * held. A lock that cannot be read counts as held, so a failure is still recorded.
+     */
+    private isWrittenAtAll;
+    /**
+     * Opens a new terraform log in a temporary directory, holding the header and no records, and
+     * removes it if the process exits before it is closed.
+     */
+    private openTerraformLog;
+    /**
+     * The version of `@quadnix/octo` this command uses, read from its `package.json`.
+     */
+    private readOctoVersion;
+    /**
+     * Adds one event to the journal, as plain data taken the moment it arrives. An event of a class the
+     * journal does not record is left out.
+     */
+    private record;
+    /**
+     * Copies the terraform log as it stands, once every record queued is written, beside it, and returns
+     * the copy's path, or `undefined` when nothing was recorded. A save uploads the copy while the log
+     * keeps growing; the copy is deleted once the save is done, or with the log.
+     */
+    private snapshotTerraformLog;
+    /**
+     * Starts a new command in this process, for a driver that runs each mode as one. The first mode
+     * names the command the process already started.
+     */
+    private startCommand;
+    /**
+     * Reports a journal failure on stderr, which stays free of a command's own output.
+     */
+    private warn;
+    /**
+     * Writes the command's journal, its terraform log, and the state file copies not written yet, as one
+     * change.
+     */
+    private write;
+}
+/**
+ * @internal
+ */
+export declare class JournalEventListenerFactory {
+    private static instance;
+    /**
+     * @internal
+     */
+    static create(): Promise<JournalEventListener>;
+}