@peachlife/artisan 0.1.0

package/README.md

@@ -0,0 +1,220 @@
+# Artisan
+
+> Artifact-level dogfood tests for command-line tools.
+
+Artisan runs the **compiled CLI artifact your users will run** inside clean Docker
+containers, then lets you assert its real stdout, stderr, exit code, config
+handling, and filesystem side effects with familiar JavaScript tests.
+
+Unit tests can say your code works. Artisan proves the binary works.
+
+## What is Artisan?
+
+Artisan is a zero-configuration dogfood testing framework for CLI binaries. It
+mounts your artifact into isolated Linux containers across a distro matrix such
+as Debian, Alpine, and Arch, then runs `*.artisan.test.mjs` tests against it.
+
+It is built on:
+
+- **Testcontainers** for disposable Docker sandboxes.
+- **Execa** for command execution and stdout/stderr/exit-code capture.
+- **Vitest** for the `test()` and `expect()` API.
+
+## Who is it for?
+
+### Agent-assisted development
+
+LLM agents are good at making unit tests pass. They are also good at missing the
+thing users actually execute: the built artifact.
+
+Artisan makes “done” mean:
+
+- the binary exists;
+- it is executable;
+- it starts in a clean sandbox;
+- it handles real arguments, config, paths, and files;
+- it exits with the expected code.
+
+That catches the classic agent failure mode: “all green” while the CLI is broken
+in a real environment.
+
+### CLI authors
+
+If you ship CLIs in Go, Rust, C/C++, Node, Python, Swift, or any other language,
+Artisan helps verify runtime behavior across Linux environments without
+hand-writing Dockerfiles or CI matrices.
+
+It is especially useful for catching:
+
+- `glibc` vs `musl` issues;
+- missing runtime packages;
+- hardcoded host paths;
+- broken config discovery;
+- bad permissions;
+- incorrect stdout/stderr/exit codes;
+- filesystem side effects that only work on your machine.
+
+## Example
+
+Create test file:
+
+```javascript
+// tests/artisan/cli-version.artisan.test.mjs
+import { expect, test } from "@peachlife/artisan";
+
+test("CLI outputs correct version", async ({ run }) => {
+  const { stdout, exitCode } = await run("--version");
+
+  expect(exitCode).toBe(0);
+  expect(stdout).toContain("1.0.0"); // Replace with your real version.
+});
+```
+
+Run it:
+
+```bash
+npx @peachlife/artisan test
+```
+
+If your project has exactly one executable artifact in `package.json#bin`, `./dist`,
+or `./bin`, Artisan can usually discover it without config.
+
+## Add explicit config when needed
+
+Use `artisan.config.json` when you want to choose the artifact, distros, setup
+commands, or config fixtures:
+
+```json
+{
+  "artifact": "./dist/mycli",
+  "distros": ["debian:stable-slim", "alpine:latest", "archlinux/archlinux:latest"],
+  "testMatch": "**/*.artisan.test.mjs",
+  "parallel": true
+}
+```
+
+Then run:
+
+```bash
+npx @peachlife/artisan test
+```
+
+## Test real behavior, not internals
+
+Artisan tests should assert what a user can observe.
+
+```javascript
+import { expect, test } from "@peachlife/artisan";
+
+test("CLI writes an output file", async ({ run, setup }) => {
+  await setup(["rm -f /tmp/output.txt"]);
+
+  const { stdout, exitCode } = await run([
+    "write-file",
+    "/tmp/output.txt",
+    "hello from the sandbox",
+  ]);
+
+  expect(exitCode).toBe(0);
+  expect(stdout).toContain("wrote /tmp/output.txt");
+
+  await setup([
+    "test -f /tmp/output.txt",
+    "grep -q '^hello from the sandbox$' /tmp/output.txt",
+  ]);
+});
+```
+
+## Test config safely
+
+Config bugs are where many CLIs break: `~` paths, missing files, leaked
+credentials, wrong XDG paths, and “works on my machine” assumptions.
+
+After `artisan.config.json` exists, import a sanitized fixture from a real
+config directory:
+
+```bash
+npx @peachlife/artisan add config ~/.config/mycli --name mycli-config
+```
+
+Artisan copies it into `./fixtures/mycli-config`, scrubs secret-like files, and
+wires the fixture into `artisan.config.json`.
+
+Inside the container, fixture directories are copied under:
+
+```text
+$XDG_CONFIG_HOME/<artifact-name>/
+```
+
+Now your tests can prove the artifact reads config in a clean sandbox instead of
+depending on your host machine.
+
+## Common commands
+
+```bash
+# Initialize Artisan in a project.
+npx @peachlife/artisan init -y --artifact ./dist/mycli --distros debian:stable-slim,alpine:latest
+
+# Run every artifact test.
+npx @peachlife/artisan test
+
+# Run one test file.
+npx @peachlife/artisan test ./tests/artisan/cli-version.artisan.test.mjs
+
+# Filter by test name.
+npx @peachlife/artisan test -t "version"
+
+# Debug one distro serially with verbose output.
+npx @peachlife/artisan test --distros alpine:latest --serial --verbose
+
+# Import a sanitized config fixture.
+npx @peachlife/artisan add config ~/.config/mycli --name mycli-config
+```
+
+Run `npx @peachlife/artisan --help` or `npx @peachlife/artisan <command> --help` for the full
+CLI reference.
+
+## Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| `0` | All tests passed. |
+| `1` | One or more tests failed, including timed-out commands. |
+| `2` | Usage or configuration error. |
+| `125` | Docker/container startup error. |
+| `130` | Interrupted by the user. |
+
+
+## CI shape
+
+A minimal GitHub Actions job looks like this:
+
+```yaml
+name: Artifact tests
+on: [push, pull_request]
+
+jobs:
+  artisan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+      - run: npm ci
+      - run: npx @peachlife/artisan test --no-color
+```
+
+Docker must be available on the runner.
+
+## Install
+
+Run without installing:
+
+```bash
+npx @peachlife/artisan test
+```
+
+## License
+
+MIT

package/bin/artisan.mjs

@@ -0,0 +1,18 @@
+#!/usr/bin/env node
+import { createProgram } from "../src/cli/parser.mjs";
+import { ArtisanError } from "../src/utils/errors.mjs";
+
+const program = createProgram();
+
+try {
+	await program.parseAsync(process.argv);
+} catch (error) {
+	if (error instanceof ArtisanError) {
+		console.error(`artisan: ${error.message}`);
+		process.exit(error.exitCode);
+	}
+	const message =
+		error instanceof Error ? error.message : String(error ?? "unknown error");
+	console.error(`artisan: unexpected error: ${message}`);
+	process.exit(1);
+}

package/package.json

@@ -0,0 +1,84 @@
+{
+	"name": "@peachlife/artisan",
+	"version": "0.1.0",
+	"type": "module",
+	"bin": {
+		"artisan": "./bin/artisan.mjs"
+	},
+	"exports": {
+		".": {
+			"types": "./dist/api/types.d.ts",
+			"import": "./src/api/injection.mjs"
+		}
+	},
+	"scripts": {
+		"build": "npx biome check src bin examples && npm test && npm run test:dogfood",
+		"test": "vitest run",
+		"test:watch": "vitest",
+		"test:integration": "npm run test:dogfood",
+		"test:dogfood": "npm run test:dogfood:fixture && npm run test:dogfood:self",
+		"test:dogfood:fixture": "node bin/artisan.mjs test tests/artisan-dogfood.artisan.test.mjs",
+		"test:dogfood:self": "node bin/artisan.mjs test tests/artisan-self.artisan.test.mjs --artifact ./bin/artisan.mjs --distros debian:stable-slim,alpine:latest,archlinux/archlinux:latest",
+		"test:all": "npm run test && npm run test:integration && npm run test:dogfood && npm run test:dogfood:fixture && npm run test:dogfood:self",
+		"eslint": "eslint .",
+		"eslint:fix": "eslint . --fix",
+		"lint": "biome check .",
+		"lint:fix": "biome check --write .",
+		"lint:dup": "jscpd",
+		"lint:arch": "depcruise src --config .dependency-cruiser.cjs",
+		"lint:dead": "knip --include-entry-exports",
+		"lint:secrets": "gitleaks detect --redact --source .",
+		"lint:sbom": "cyclonedx-npm --output-file ./sbom.cdx.json --output-format json",
+		"lint:semgrep": "semgrep ci --config p/javascript --config p/security-audit",
+		"typecheck": "tsc --noEmit --allowImportingTsExtensions --allowJs --checkJs false --noResolve --skipLibCheck --module nodenext --moduleResolution nodenext --target esnext src/**/*.mjs bin/**/*.mjs || true",
+		"lint:all": "npm run lint && npm run eslint && npm run lint:dup && npm run lint:arch && npm run lint:dead",
+		"fix:all": "biome check --write . && eslint . --fix && knip --fix",
+		"ci": "npm run lint:all && npm run lint:secrets && npm run lint:semgrep && npm run lint:sbom && npm run test:all"
+	},
+	"lint-staged": {
+		"*.mjs": [
+			"biome check --write",
+			"eslint --fix",
+			"eslint"
+		]
+	},
+	"dependencies": {
+		"commander": "^15.0.0",
+		"execa": "^9.6.1",
+		"fast-glob": "^3.3.3",
+		"testcontainers": "^12.0.3"
+	},
+	"devDependencies": {
+		"@biomejs/biome": "^2.5.1",
+		"@cyclonedx/cyclonedx-npm": "^5.0.0",
+		"@eslint/js": "^10.0.1",
+		"@typescript-eslint/eslint-plugin": "^8.62.0",
+		"@typescript-eslint/parser": "^8.62.0",
+		"dependency-cruiser": "^18.0.0",
+		"eslint": "^10.5.0",
+		"eslint-plugin-security": "^4.0.1",
+		"eslint-plugin-sonarjs": "^4.1.0",
+		"eslint-plugin-unicorn": "^69.0.0",
+		"jscpd": "^5.0.11",
+		"knip": "^6.21.0",
+		"lint-staged": "^17.0.8",
+		"vitest": "^4.1.9"
+	},
+	"files": [
+		"src",
+		"bin",
+		"templates"
+	],
+	"publishConfig": {
+		"access": "public"
+	},
+	"engines": {
+		"node": ">=22"
+	},
+	"allowScripts": {
+		"cpu-features@0.0.10": true,
+		"fsevents@2.3.3": true,
+		"protobufjs@7.6.4": true,
+		"ssh2@1.17.0": true
+	}
+}

package/src/api/injection.mjs

@@ -0,0 +1,87 @@
+export {
+	afterAll,
+	afterEach,
+	beforeAll,
+	beforeEach,
+	describe,
+	expect,
+} from "vitest";
+
+import { test as vitestTest } from "vitest";
+import { DEFAULT_XDG_CONFIG_HOME } from "../core/constants.mjs";
+import { ContainerManager } from "../core/container-manager.mjs";
+
+export function createTestContext(container) {
+	return {
+		async run(arguments_ = "", options = {}) {
+			return container.exec(arguments_, options);
+		},
+		async copyFixture(localPath, containerPath) {
+			return container.copyFile(localPath, containerPath);
+		},
+		async setup(commands, options = {}) {
+			for (const command of commands) {
+				const result = await container.shell(command, options);
+				if (result.exitCode !== 0) {
+					throw new Error(
+						`setup cmd failed: ${command}\n${result.stderr || result.stdout}`,
+					);
+				}
+			}
+		},
+	};
+}
+
+function createContainer(
+	distro,
+	artifact,
+	setupCommands,
+	configs,
+	xdgConfigHome,
+) {
+	return new ContainerManager(distro, artifact, {
+		setupCommands,
+		configs,
+		env: {
+			HOME: "/root",
+			XDG_CONFIG_HOME: xdgConfigHome,
+		},
+	});
+}
+
+// Build fixture-extended test from env vars (set by artisan test command).
+function buildExtendedTest() {
+	const distro = process.env.ARTISAN_DISTRO;
+	const artifact = process.env.ARTISAN_ARTIFACT;
+	if (!distro || !artifact) return vitestTest;
+
+	const setupCommands = JSON.parse(process.env.ARTISAN_SETUP ?? "[]");
+	const configs = JSON.parse(process.env.ARTISAN_CONFIGS ?? "[]");
+	const xdgConfigHome =
+		process.env.ARTISAN_XDG_CONFIG_HOME ?? DEFAULT_XDG_CONFIG_HOME;
+
+	return vitestTest
+		.extend("artisanContext", async ({ task: _task }, { onCleanup }) => {
+			const container = createContainer(
+				distro,
+				artifact,
+				setupCommands,
+				configs,
+				xdgConfigHome,
+			);
+			await container.start();
+			onCleanup(() => container.stop());
+			return createTestContext(container);
+		})
+		.extend("run", async ({ artisanContext }) =>
+			artisanContext.run.bind(artisanContext),
+		)
+		.extend("copyFixture", async ({ artisanContext }) =>
+			artisanContext.copyFixture.bind(artisanContext),
+		)
+		.extend("setup", async ({ artisanContext }) =>
+			artisanContext.setup.bind(artisanContext),
+		);
+}
+
+export const test = buildExtendedTest();

package/src/cli/commands/add-config.mjs

@@ -0,0 +1,91 @@
+import { rm, stat, writeFile } from "node:fs/promises";
+import { basename, join } from "node:path";
+import { readJsonConfig } from "../../core/config.mjs";
+import { sanitizeConfigDirectory } from "../../core/sanitizer.mjs";
+import { UsageError } from "../../utils/errors.mjs";
+import { fileExists } from "../../utils/fs.mjs";
+import { expandUserPath } from "../../utils/path.mjs";
+import { validateSinglePathName } from "../../utils/validation.mjs";
+
+export async function runAddConfig(options = {}) {
+	const cwd = options._cwd ?? process.cwd();
+	const source = options.source;
+	if (!source) {
+		throw new UsageError(
+			"Usage: artisan add config <source> [--name <name>] [--raw] [--force]",
+		);
+	}
+
+	const configPath = join(cwd, "artisan.config.json");
+	const config = await readConfig(configPath);
+	const sourcePath = expandUserPath(source, cwd);
+	await assertSourceDirectory(sourcePath, source);
+
+	const name = options.name ?? basename(sourcePath);
+	validateSinglePathName(name, "fixture directory name");
+	const fixturePath = join(cwd, "fixtures", name);
+
+	if (!options.force && (await fileExists(fixturePath))) {
+		console.log(
+			`Skipped ${fixturePath} (already exists, use --force to overwrite)`,
+		);
+		return;
+	}
+
+	await rm(fixturePath, { recursive: true, force: true });
+	const result = await sanitizeConfigDirectory(sourcePath, fixturePath, {
+		raw: options.raw ?? false,
+	});
+	await appendToConfigs(configPath, config, `./fixtures/${name}`);
+
+	console.log(`Created ${fixturePath}`);
+	if (result.raw) {
+		console.log("  copied: all entries (raw)");
+	} else {
+		console.log(`  copied: ${result.copied.length} entries`);
+	}
+	if (result.scrubbed.length > 0) {
+		const preview = result.scrubbed.slice(0, 10).join(", ");
+		const suffix = result.scrubbed.length > 10 ? "…" : "";
+		console.log(`  scrubbed (${result.scrubbed.length}): ${preview}${suffix}`);
+	}
+	if (result.manifestPath) console.log(`  manifest: ${result.manifestPath}`);
+	if (result.raw)
+		console.warn(
+			`  ⚠️  --raw: secrets NOT scrubbed. Review ${fixturePath} before committing.`,
+		);
+}
+
+async function assertSourceDirectory(sourcePath, displaySource) {
+	let sourceStat;
+	try {
+		sourceStat = await stat(sourcePath);
+	} catch {
+		throw new UsageError(
+			`config source "${displaySource}" not found at ${sourcePath}`,
+		);
+	}
+
+	if (!sourceStat.isDirectory()) {
+		throw new UsageError(
+			`config source "${displaySource}" must be a directory`,
+		);
+	}
+}
+
+async function readConfig(configPath) {
+	return readJsonConfig(configPath);
+}
+
+async function appendToConfigs(configPath, config, entry) {
+	if (config.configs !== undefined && !Array.isArray(config.configs)) {
+		throw new UsageError('"configs" in artisan.config.json must be an array');
+	}
+	config.configs ??= [];
+	if (!config.configs.includes(entry)) config.configs.push(entry);
+	await writeFile(
+		configPath,
+		`${JSON.stringify(config, undefined, 2)}\n`,
+		"utf8",
+	);
+}

package/src/cli/commands/add.mjs

@@ -0,0 +1,64 @@
+import { mkdir, writeFile } from "node:fs/promises";
+import { join } from "node:path";
+import { UsageError } from "../../utils/errors.mjs";
+import { fileExists } from "../../utils/fs.mjs";
+import {
+	readTemplate,
+	resolveTemplatesDirectory,
+} from "../../utils/template.mjs";
+import { validateSinglePathName } from "../../utils/validation.mjs";
+import { runAddConfig } from "./add-config.mjs";
+
+const TEMPLATES_DIR = resolveTemplatesDirectory(
+	import.meta.url,
+	"../../../templates/tests",
+);
+
+const TYPE_MAP = {
+	version: {
+		template: "version.test.mjs.tmpl",
+		filename: "cli-version.artisan.test.mjs",
+	},
+	help: {
+		template: "help.test.mjs.tmpl",
+		filename: "cli-help.artisan.test.mjs",
+	},
+	custom: { template: "custom.test.mjs.tmpl", filename: undefined },
+};
+
+export async function runAdd(type, options) {
+	if (type === "config") return runAddConfig(options);
+
+	const spec = TYPE_MAP[type];
+	if (!spec)
+		throw new UsageError(
+			`Unknown test type: ${type}. Valid types: version, help, custom, config`,
+		);
+
+	let filename = spec.filename;
+	if (type === "custom") {
+		if (!options.name)
+			throw new UsageError("--name is required for custom type");
+		validateSinglePathName(options.name, "test name");
+		filename = `${options.name}.artisan.test.mjs`;
+	}
+
+	const targetDirectory = join(process.cwd(), options.dir ?? "./tests/");
+	await mkdir(targetDirectory, { recursive: true });
+
+	const outPath = join(targetDirectory, filename);
+	if (!options.force && (await fileExists(outPath))) {
+		console.log(
+			`Skipped ${outPath} (already exists, use --force to overwrite)`,
+		);
+		return;
+	}
+
+	let content = await readTemplate(TEMPLATES_DIR, spec.template);
+	if (type === "custom") {
+		// eslint-disable-next-line unicorn/no-unsafe-string-replacement
+		content = content.replaceAll("{{TEST_NAME}}", options.name ?? "my test");
+	}
+	await writeFile(outPath, content, "utf8");
+	console.log(`Created ${outPath}`);
+}

package/src/cli/commands/init.mjs

@@ -0,0 +1,71 @@
+import { mkdir, writeFile } from "node:fs/promises";
+import { join } from "node:path";
+import { DEFAULT_DISTROS } from "../../core/constants.mjs";
+import { UsageError } from "../../utils/errors.mjs";
+import { fileExists } from "../../utils/fs.mjs";
+import {
+	readTemplate,
+	renderTemplate,
+	resolveTemplatesDirectory,
+} from "../../utils/template.mjs";
+
+const TEMPLATES_DIR = resolveTemplatesDirectory(
+	import.meta.url,
+	"../../../templates",
+);
+
+function parseDistros(distrosRaw) {
+	if (typeof distrosRaw !== "string") {
+		throw new UsageError("Distros must be a comma-separated string");
+	}
+	const distros = distrosRaw
+		.split(",")
+		.map((d) => d.trim())
+		.filter(Boolean);
+	if (distros.length === 0) {
+		throw new UsageError("At least one distro is required");
+	}
+	return distros;
+}
+
+export async function runInit(options) {
+	const cwd = process.cwd();
+	const artifact = options.artifact ?? "./dist/mycli";
+	const distrosRaw = options.distros ?? DEFAULT_DISTROS.join(",");
+	const distros = parseDistros(distrosRaw);
+	const force = options.force ?? false;
+
+	// Write artisan.config.json
+	const configPath = join(cwd, "artisan.config.json");
+	if (!force && (await fileExists(configPath))) {
+		console.log(
+			`Skipped artisan.config.json (already exists, use --force to overwrite)`,
+		);
+	} else {
+		const tmpl = await readTemplate(TEMPLATES_DIR, "config.json.tmpl");
+		const distrosJson = distros.map((d) => JSON.stringify(d)).join(", ");
+		const content = renderTemplate(tmpl, {
+			ARTIFACT: artifact,
+			DISTROS: distrosJson,
+		});
+		await writeFile(configPath, content, "utf8");
+		console.log(`Created artisan.config.json`);
+	}
+
+	// Write starter test file
+	const testsDirectory = join(cwd, "tests/artisan");
+	await mkdir(testsDirectory, { recursive: true });
+	const testPath = join(testsDirectory, "cli-version.artisan.test.mjs");
+	if (!force && (await fileExists(testPath))) {
+		console.log(
+			`Skipped tests/artisan/cli-version.artisan.test.mjs (already exists)`,
+		);
+	} else {
+		const tmpl = await readTemplate(
+			TEMPLATES_DIR,
+			"tests/version.test.mjs.tmpl",
+		);
+		await writeFile(testPath, tmpl, "utf8");
+		console.log(`Created ./tests/artisan/cli-version.artisan.test.mjs`);
+	}
+}