@harperfast/integration-testing 0.1.0

package/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -0,0 +1,291 @@
+# @harperfast/integration-testing
+
+Integration testing utilities for Harper-based projects. Provides framework-agnostic Harper instance lifecycle management, a concurrent loopback address pool, directory compression utilities, and an optional preconfigured Node.js test runner script.
+
+## Testing Ethos
+
+There are three distinct levels of testing in the Harper ecosystem:
+
+- **Unit** tests validate isolated logic against source or built code directly, relying on mocks or stubs in place of real dependencies.
+- **Integration** tests run against the actual Harper software, but locally and ephemerally. Harper is started as a child process for each test suite, given a temporary install directory, and torn down when the suite finishes. There is no remote infrastructure to manage.
+- **End-to-end** tests go further: they target a deployment that mirrors production as closely as possible — think a staging environment with real infrastructure, remote systems, and operational configuration.
+
+This project is strictly for **integration** testing.
+
+The distinction matters because integration tests give you confidence in Harper's actual behavior without the cost and complexity of managing a deployed environment. You get the real software, not mocks, but with fast local iteration.
+
+Regardless of the framework or runner used, integration tests should strive to be:
+
+- **Independent**: No dependency on execution order or shared state with other tests
+- **Hermetic**: Self-contained with no external side-effects
+- **Deterministic**: The same input always produces the same output
+
+These properties are what make safe concurrent execution possible — without them, parallelization produces flaky, order-dependent failures.
+
+## Installation
+
+```sh
+npm install --save-dev @harperfast/integration-testing
+```
+
+## Framework Agnosticism
+
+The core lifecycle and loopback pool APIs in this package — `startHarper`, `teardownHarper`, `killHarper`, and the loopback utilities — are **not tied to any specific test runner or framework**. They spawn and manage Harper child processes and allocate loopback addresses safely across concurrent processes using a file-based locking mechanism. This means they can be used with any testing framework that supports async setup and teardown: Node.js `node:test`, Vitest, Playwright, or anything else.
+
+The key responsibility these APIs place on you is **thinking critically about Harper instance scope**. Starting a Harper instance has real cost — it spawns a process, allocates a loopback address from the pool, and creates a temporary install directory. How you scope instances depends on your framework and what you're testing. A single instance shared across many assertions in one logical test block may be perfectly appropriate; spinning up a fresh instance for every individual assertion would be wasteful. What matters is that concurrently executing scopes each have their own instance and loopback address — the pool handles cross-process allocation safely, but your setup and teardown hooks determine the boundaries.
+
+This package also includes a **preconfigured Node.js test runner script** (`harper-integration-test-run`) as a recommended, batteries-included way to run tests written with the `node:test` API. It is documented below, and its [source](./src/run.ts) serves as a reference implementation for anyone integrating these utilities with a different runner.
+
+## Setup: Loopback Addresses
+
+Running Harper integration tests concurrently requires multiple loopback addresses so that each Harper instance can bind to a distinct address without port conflicts.
+
+Linux Ubuntu systems have `127.0.0.1–127.255.255.255` enabled by default. macOS and Windows do not.
+
+Use the included script to configure them:
+
+```sh
+npx harper-integration-test-setup-loopback
+```
+
+This script requires `sudo` and respects the `HARPER_INTEGRATION_TEST_LOOPBACK_POOL_COUNT` environment variable (defaults to 32).
+
+## API
+
+The lifecycle and utility APIs below are framework-agnostic. They manage Harper child processes and a cross-process loopback address pool. Use them in the setup/teardown hooks of whichever test framework you prefer.
+
+### `startHarper(ctx, options?)`
+
+Allocates a loopback address from the pool, creates a temporary install directory, starts a Harper process, and waits for it to be ready. Populates `ctx.harper` with the instance details. Call in a setup/`before()` hook.
+
+The Harper binary is resolved in the following order:
+
+1. `harperBinPath` option passed directly to `startHarper()`
+2. `HARPER_INTEGRATION_TEST_INSTALL_SCRIPT` environment variable (path to `dist/bin/harper.js`)
+3. Auto-resolved from a `harper` package installed as a project dependency
+
+**Options:**
+
+```ts
+interface StartHarperOptions {
+  startupTimeoutMs?: number;   // Default: 30000 or HARPER_INTEGRATION_TEST_STARTUP_TIMEOUT_MS
+  config?: object;             // Harper config overrides (passed via HARPER_SET_CONFIG)
+  env?: object;                // Additional environment variables for the Harper process
+  harperBinPath?: string;      // Explicit path to dist/bin/harper.js
+}
+```
+
+**Environment Variables:**
+
+- `HARPER_INTEGRATION_TEST_STARTUP_TIMEOUT_MS` - Default startup timeout
+- `HARPER_INTEGRATION_TEST_INSTALL_PARENT_DIR` - Parent directory for temp Harper install dirs (default: OS tmpdir)
+- `HARPER_INTEGRATION_TEST_INSTALL_SCRIPT` - Path to Harper CLI script
+
+### `setupHarperWithFixture(ctx, fixturePath, options?)`
+
+Like `startHarper()`, but copies a component directory into the Harper install before starting, so it's available on first boot without a deploy.
+
+### `killHarper(ctx)`
+
+Sends SIGTERM to the Harper process and waits for it to exit. Does not release the loopback address or clean up the install directory. Useful for restart scenarios where the test will call `startHarper` again.
+
+### `teardownHarper(ctx)`
+
+Kills Harper, releases the loopback address back to the pool, and removes the install directory. Call in a teardown/`after()` hook.
+
+### `sendOperation(context, operation)`
+
+Helper to POST an operation to the Operations API and assert HTTP 200.
+
+### `ContextWithHarper`
+
+TypeScript interface for the test context populated by `startHarper()`. When using `node:test`, pass it as the suite context type:
+
+```ts
+interface HarperContext {
+  dataRootDir: string;          // Harper install directory
+  admin: { username: string; password: string };
+  httpURL: string;              // e.g. 'http://127.0.0.2:9926'
+  operationsAPIURL: string;     // e.g. 'http://127.0.0.2:9925'
+  hostname: string;             // e.g. '127.0.0.2'
+  process: ChildProcess;
+  logDir?: string;
+}
+```
+
+### Server Log Capture
+
+When `HARPER_INTEGRATION_TEST_LOG_DIR` is set, each Harper instance writes its logs to a per-suite subdirectory. Logs from passing suites are automatically cleaned up; only failing suite logs are retained.
+
+```sh
+HARPER_INTEGRATION_TEST_LOG_DIR=/tmp/harper-test-logs npx harper-integration-test-run "integrationTests/**/*.test.ts"
+```
+
+### `targz(dirPath)`
+
+Packs and compresses a directory into a base64-encoded tar.gz string. Useful for `deploy_component` Operations API calls.
+
+### Loopback Pool Utilities
+
+These are used internally by `startHarper` and `teardownHarper`, but are exported for advanced use cases or custom runner integrations.
+
+- `validateLoopbackAddressPool(): Promise<{ successful: string[]; failed: { loopbackAddress: string; error: Error }[] }>` - Validates all pool addresses can be bound to
+- `getNextAvailableLoopbackAddress(): Promise<string>` - Allocates an address from the pool
+- `releaseLoopbackAddress(address: string): Promise<void>` - Returns an address to the pool
+- `releaseAllLoopbackAddressesForCurrentProcess(): Promise<void>` - Releases all addresses held by this process
+
+## Node.js Test Runner
+
+> Tests executed by this runner must use the `node:test` API. If you're using a different test framework, use the [lifecycle APIs](#api) directly and refer to the [runner source](./src/run.ts) as a reference implementation.
+
+The included `harper-integration-test-run` script is the recommended way to run Harper integration tests. It is built on the Node.js `node:test` API and handles the concurrency configuration described in the [original parallelization analysis](https://github.com/Ethan-Arrowood/node-test-runner-parallelization-analysis): because each integration test spawns a resource-intensive Harper process, the safe default concurrency is roughly half of available system parallelism rather than Node.js's default of `availableParallelism() - 1`.
+
+The runner executes each test **file** in its own process, which means files are the unit of isolation. For this to work safely and predictably, each test file must adhere to the independent, hermetic, and deterministic properties described in the [Testing Ethos](#testing-ethos). Strictly following these properties enables files to run concurrently without interfering with each other. When a test breaks, that file can be run in isolation to iterate on a fix without re-running everything.
+
+The [Harper integration tests](https://github.com/HarperFast/harper/tree/main/integrationTests) are an open source example of this runner and framework in production use.
+
+### Running Tests
+
+```sh
+npx harper-integration-test-run "integrationTests/**/*.test.ts"
+```
+
+Or add to your `package.json`:
+
+```json
+{
+  "scripts": {
+    "test:integration": "harper-integration-test-run integrationTests/**/*.test.ts"
+  }
+}
+```
+
+### Runner Options
+
+All CLI arguments can be overridden using the associative `HARPER_INTEGRATION_TEST_*` environment variable.
+
+| CLI Argument | Environment Variable | Default | Description |
+|---|---|---|---|
+| `--concurrency=N` | `HARPER_INTEGRATION_TEST_CONCURRENCY` | `floor(parallelism/2)+1` | Number of concurrent test processes |
+| `--isolation=mode` | `HARPER_INTEGRATION_TEST_ISOLATION` | `process` | `process` or `none` |
+| `--shard=index/total` | `HARPER_INTEGRATION_TEST_SHARD` | `1/1` | Test sharding for CI |
+| `--only` | `HARPER_INTEGRATION_TEST_ONLY` | `false` | Run only tests marked with `.only` |
+
+Example - run sequentially without loopback pool:
+
+```sh
+npx harper-integration-test-run --isolation=none "integrationTests/**/*.test.ts"
+```
+
+Example - CI sharding across 4 jobs:
+
+```sh
+npm run test:integration -- --shard=1/4
+npm run test:integration -- --shard=2/4
+npm run test:integration -- --shard=3/4
+npm run test:integration -- --shard=4/4
+```
+
+### Writing Tests
+
+Test files should be written in ESM TypeScript, end in `.test.ts`, and begin with a comment block describing what they verify. File names should be short and hyphen-separated — e.g., `install.test.ts` or `application-management.test.ts`.
+
+Tests should use `suite`/`describe`, `test`/`it`, and lifecycle methods (`before`, `beforeEach`, `after`, `afterEach`) from `node:test`, with assertions from `node:assert/strict`.
+
+A test file can contain multiple suites. Suites always run **sequentially**. Tests within a suite run **sequentially by default**, but can be made concurrent with `{ concurrency: true }` — in which case each individual test within that suite must itself be independent, hermetic, and deterministic.
+
+Since these tests interact with a running Harper instance, they often validate actual application output: standard streams (`stdout`/`stderr`), network responses, or file system state.
+
+#### Suite Concurrency Example
+
+This file contains two suites with two 1-second tests each. The first suite enables `{ concurrency: true }`, the second does not.
+
+```ts
+import { suite, test } from 'node:test';
+import { setTimeout as sleep } from 'node:timers/promises';
+
+suite('Concurrency Enabled', { concurrency: true }, () => {
+  test('1 second', async () => {
+    await sleep(1000);
+  });
+
+  test('1 second', async () => {
+    await sleep(1000);
+  });
+});
+
+suite('Concurrency Disabled', () => {
+  test('1 second', async () => {
+    await sleep(1000);
+  });
+
+  test('1 second', async () => {
+    await sleep(1000);
+  });
+});
+```
+
+- The first suite's tests run concurrently → ~1 second
+- The second suite's tests run sequentially → ~2 seconds
+- Suites always run sequentially → **total ~3 seconds**
+
+```
+▶ Concurrency Enabled
+  ✔ 1 second (1001.359083ms)
+  ✔ 1 second (1001.860375ms)
+✔ Concurrency Enabled (1002.26125ms)
+▶ Concurrency Disabled
+  ✔ 1 second (1001.122166ms)
+  ✔ 1 second (1000.5495ms)
+✔ Concurrency Disabled (2001.850625ms)
+ℹ tests 4
+ℹ duration_ms 3087.786041
+```
+
+#### Template
+
+```ts
+/**
+ * Description of what this test file verifies.
+ */
+import { suite, test, before, after } from 'node:test';
+import { strictEqual } from 'node:assert/strict';
+import { startHarper, teardownHarper, type ContextWithHarper } from '@harperfast/integration-testing';
+
+suite('short description', (ctx: ContextWithHarper) => {
+  before(async () => {
+    await startHarper(ctx);
+  });
+
+  after(async () => {
+    await teardownHarper(ctx);
+  });
+
+  test('test description', async () => {
+    const response = await fetch(ctx.harper.httpURL);
+    strictEqual(response.status, 200);
+  });
+});
+```
+
+### Using Other Frameworks
+
+The lifecycle APIs work with any framework that supports async setup and teardown. The [runner source](./src/run.ts) is a concrete reference for how the Node.js `node:test` integration is wired up, including concurrency configuration and shard support. Use it as a starting point when building an integration for another framework.
+
+## GitHub Actions Workflow Parallelization
+
+Integration tests should run efficiently and reliably on CI, not just locally. As the number of test files grows — further compounded by verifying multiple operating systems or runtime versions — even a modest test suite can become a large CI operation. The goal is to strike a balance between cost and thoroughness. Tests must always run reliably; correctness is never compromised for speed. Overloaded runners cause flaky failures.
+
+**Default GitHub Actions runners have limited performance.** They can reasonably handle a single integration test, but parallelizing many of them will often exceed the runner's capabilities. A recommended strategy is to **parallelize across multiple runners using workflow matrix jobs** combined with **test sharding** — distributing test files across runners rather than running them all on one.
+
+Larger runners are available but come at a meaningful cost increase. The right trade-off depends on your organization's total CI job volume, since parallel job execution is typically capped at the org level. When wait time for available runners starts to exceed the time saved by parallelization, fewer jobs on larger runners may be more effective.
+
+It's also worth being deliberate about *when* the full integration suite runs. Running every test across all supported configurations on every commit of every open PR can quickly exhaust concurrency limits. Strategies like file path filter triggers, manually triggered workflows, and limiting the full matrix to merge-time checks can keep CI responsive without sacrificing coverage.
+
+A **merge queue** is worth considering for this last point — running the complete matrix only after a PR passes initial checks (e.g., a single configuration), then verifying the full matrix before merge. This allows for faster PR iteration while keeping the main branch thoroughly tested.
+
+The [Harper integration tests workflow](https://github.com/HarperFast/harper/blob/main/.github/workflows/integration-tests.yml) is a real-world example of these strategies in practice.
+
+## License
+
+Apache-2.0