@monster-orange/den 0.0.0 → 0.2.0

package/CHANGELOG.md

@@ -0,0 +1,204 @@
+# Changelog
+
+All notable changes to Den are documented here. Each release lists the
+config and manifest changes a distribution must make to move to it under
+**Migrations**.
+
+## [0.2.0] - 2026-11-04
+
+### Added
+
+- **Manifest overrides.** An optional `manifests/<repo>.override.yml` in the
+  distribution is deep-merged onto the repo's base manifest (its own, or the
+  held one): maps merge by key, lists and scalars replace, and an explicit
+  `null` deletes a key. An override may only name services and keys the base
+  has, and an override with no base is an error. See "Overriding a manifest"
+  in `docs/distributions.md`.
+- **Manifest layers are visible.** `/api/services` returns `manifestLayers`
+  per service (`["repo"]`, `["held", "override"]`, …) and the Services cards
+  show them; `GET /api/manifest/validate?path=<repo dir>` returns each layer's
+  result.
+- **`validate` checks each layer.** For a repo folder it validates the base
+  and the override separately, each error naming its file, then the merged
+  result.
+- **`<cli> mcp`: an MCP server over stdio** for agents, a thin client of the
+  running web server (it never starts it, and says how when it is down). Read
+  tools are always on, including the new `stacks` and `edge_status`; the
+  actions `service_up`, `service_down`, `service_restart`, `workspace_sync`
+  and `run_tests` are opt-in per org with `mcp.actions`. Guards: lifecycle
+  commands and env=local only, locally run services only, `run_tests` limited
+  to `mcp.run_tests.envs` (default `[local]`, never production), security runs
+  must pass the owned-host plan and `full` is local only, log tools read
+  local sources unless `mcp.logs.remote: true`, and all text returned is
+  secret-redacted. No deploy tool. Pinned `@modelcontextprotocol/sdk` 1.32.1.
+  See "MCP server" in `docs/distributions.md`.
+- **Agent audit trail.** Every service exec, test run and sync is recorded
+  with who asked: an agent's actions carry actor `agent`, the MCP client's
+  self-reported name and version, and the MCP pid and session id, and show as
+  `[agent]` lines in the web server log on the Logs page.
+- **Read-only status endpoints:** `/api/stacks` (what `stack ls/health` show),
+  `/api/edge` (what `edge status` shows) and `/api/security/plan` (the
+  security CLI's own dry run). `stack ls` and `edge status` now print from the
+  same code.
+- **Graph: cards and a details panel.** The Local Dev graph is laid out by
+  ELK (layered, compound org or application › repo › service, orthogonal
+  edges; elkjs vendored and served by Den, so it works offline). Cards are
+  full size (detailed or compact) on a canvas you pan and zoom; Fit zooms out
+  to 35%. A card shows status, name, repo, placement and health, and its main
+  actions on hover. Clicking a card or a repo opens a side panel: one primary
+  action (Start when down, Open when up, else Logs), two common ones, and More
+  (manifest commands, tools, and careful actions that confirm on a second
+  click); tabs Overview, Logs (live tail), Config and Activity (the audit
+  trail, yours and agents'). ⤢ pops the panel out as a dialog. Layout
+  direction, grouping, card density and panel width are remembered per user
+  (`/api/settings` `graph`). Find highlights and centres on matches. The
+  panel is built from registries (actions, tabs, sections) a distribution or
+  later release can add to.
+- `GET /api/activity`: the audit trail, filtered by service or repo.
+- **One secret redactor** (`src/shared/redact.ts`), shared by the journeys and
+  security pillars and the MCP server.
+- **Placement (R13): what runs here and what runs in the cloud.** A
+  manifest's service may say `runs: { local: remote, dev: remote }` (per env,
+  `local` or `remote`) and `target: { env, account?, url?, expect? }`.
+  Without `runs`, a service is local when it or any service of its repo has
+  local commands (or the repo is a compose stack), remote when it has only an
+  HTTP health URL and nothing in its repo runs here, and otherwise has no
+  placement. `/api/services`
+  and `/api/health` rows carry `runs` and `target`; `/api/applications`
+  carries `placements` per service address, and per application `placement`
+  (`local | remote | hybrid | none`) and a `summary`
+  (`shop-frontend local · shop-core remote (dev)`). `<cli> status`,
+  `services` and `apps` suffix remote services `☁ remote (dev)`, and
+  `<cli> graph` marks remote and hybrid nodes. See "Placement" in
+  `docs/distributions.md`.
+- **Remote health.** A remote service is checked by the health declared for
+  its `target.env`; only when there is none does Den GET `target.url`, up when
+  the status is one of `target.expect` (a code or a list, 100–599; default
+  any 2xx). Its health row's label names the target (`dev · AWS`).
+- **`production_envs`** in the org config: the envs that count as production
+  (default `[prod, production]`).
+
+### Changed
+
+- **`/api/health` rows: `url` is only ever a URL.** A shell health check's
+  command (a compose stack's `<cli> stack health …`, or any declared shell
+  check) is now in a new `check` field and `url` is null; the UI shows the
+  same tooltips as before. Clients that read the command from `url` should
+  read `check`.
+- `/api/server-info` returns Den's `version`.
+- **Nothing runs a command on a remote service.** `/api/service/exec`,
+  `<cli> up|down|restart`, workspace up and down, and `<cli> up`'s plan refuse
+  it (`runs in dev (AWS); nothing to start here`; a `down` succeeds with
+  nothing to stop). The MCP guard `runsLocally()` reads `runs`. Security
+  `auto_targets` excludes remote services explicitly, a remote service is
+  never scanned as if it were local, and one whose `target.env` counts as
+  production is never scanned at all.
+- **Overrides may add placement.** `runs` and `target` are the one thing a
+  `manifests/<repo>.override.yml` may add to a service its base has.
+- `validate` checks env names: a `runs` key or `target.env` that is not one of
+  the org's `environments` is an error.
+- `<cli> status` keeps a hybrid repo's remote service as its own row instead
+  of rolling it into the entrypoint.
+
+### Fixed
+
+- **Logs of compose-stack services.** The live tail (`/api/service/taillog`,
+  so the UI's log view and `logs_tail`), log history and search
+  (`logs_search`) and `<cli> logs` now find a stack service's container,
+  `<stack>-<workspace>-<service>-1`, through the stacks naming helpers. They
+  looked only for opctl containers, so they fell back to the startup log
+  ("container log unavailable") or found nothing.
+- **`<cli> mcp` against an older web server.** A tool whose route the web
+  server predates (`stacks`, `edge_status`, a security `run_tests`) now says
+  `web server is Den <x>; <tool> needs >= <y>: run <cli> web restart`
+  instead of a bare 404; other 404s name the tool and the route.
+- Sync never touches a dev-managed checkout: a managed entry whose path is,
+  holds or sits inside one is refused with the reason, and naming a
+  dev-managed repo says so instead of doing nothing.
+- The What's new dialog showed every changelog bullet cut off mid-sentence.
+- With nothing run yet, the empty output pane no longer takes 200px from the
+  Local Dev page.
+- A security run's mode chosen on the Tests page (`api`, `full`) now reaches
+  the scan; it always ran `baseline` before.
+
+### Security
+
+- Dependency advisories: `npm audit fix` plus `@cursor/sdk` 1.0.23 → 1.0.40
+  (drops its `undici` 5 / `zod` 3 tree) clear every critical and high
+  advisory. Left: a `minimatch` ReDoS under `openapi-typescript` (only fix is
+  a downgrade to 6.x; it matches local config globs only) and dev-only
+  tooling (jest, eslint) moderates.
+
+### Migrations
+
+- None required: with no override files, manifests resolve as in 0.1.0.
+- To let agents act through `<cli> mcp`, add `mcp.actions` to the org config.
+- Placement needs no change: without `runs`, services keep working as
+  before. A service with an HTTP health URL, no commands, and nothing else
+  in its repo that runs here now shows as remote; add `runs: { local: local }`
+  if it is really on this machine. To
+  mark more envs as production than `prod` and `production`, set
+  `production_envs`.
+
+
+## [0.1.0] - 2026-11-04
+
+The first Den release: one engine for local development, tests and releases
+across an organisation's repos, shipped as `@monster-orange/den` and used
+through thin distributions that pin an exact version.
+
+### Added
+
+- **`den create <name> [dir]`** (also `npx @monster-orange/den create`)
+  scaffolds a distribution: `den.yml`, a `package.json` pinning this exact
+  Den, a `bin/<cli>` shim, a Makefile including `make/den.mk`, an org config
+  with a brand, and empty `manifests/` and `tests/pillars/`.
+  `examples/distribution/` is its output for `acme`. See
+  `docs/distributions.md`.
+- **Den runs as an installed package.** Den's own files resolve from the
+  engine root (`node_modules/@monster-orange/den`), the distribution's from
+  its root; scripts run with Den's tsx, never through `npx` or the current
+  directory. The package ships `src/` and runs it with tsx (a dependency
+  now); CI proves it with `scripts/dist-smoke.sh`.
+- **Org > Application > Repo > Services.** Declared applications (config
+  `applications:` or a manifest's `application:`) group repos; the Local Dev
+  graph boxes each one inside its org's frame, next to the connected groups of
+  repos that are their own application.
+- **Security scans keep a latest record.** Every run writes
+  `reports/security/<env>/latest-<mode>.json`; the Tests page shows it, with
+  findings marked new since the previous run.
+- **Repo suites** (`tests:` in a manifest) run from the Tests page and CLI.
+- **Edge opt-ins**: a project CA and route certificates are opt-in; with none
+  set, the rendered edge is unchanged.
+
+- **launchd service from a distribution.** `service/install.sh` ships in the
+  package and runs from `DEN_ROOT`; `DEN_SERVICE_LABEL` and `DEN_SERVICE_LOGS`
+  keep an existing label and log folder; `print` shows the plist.
+
+### Migrations
+
+From the engine this release was generalised from:
+
+- Run Den from the package: a `bin/<cli>` shim that sets `DEN_ROOT` to the
+  distribution and execs `node_modules/@monster-orange/den/bin/den`, and a
+  Makefile that does `include node_modules/@monster-orange/den/make/den.mk`
+  (`den create` writes both; `docs/distributions.md` shows them).
+- Add `den.yml` at the distribution root with `product: { name: <your name> }`.
+  Env vars, the config and manifest file names, the state, managed-clone and
+  workspace directories all derive from that name, so an existing install
+  keeps every on-disk and env name by naming itself as before.
+- Workspace mode `<name>-managed` is now `managed`. Old values are still
+  read, and rewritten on the next save.
+- The CLI command `use-<name>` is now `use-managed`.
+- `github_org` has no default: set it in the org config.
+- The shared local-infra repo is configuration: `graph.infra: { repo,
+  signals }`. Without it no infra edges are derived. The default-workspace
+  entry repo is `defaults.workspace.entry`.
+- Perf suites come from the distribution (`tests/pillars/perf/<module>/test.js`)
+  or from repo manifests; Den ships none.
+- Service and setup make targets are `<prefix>-up`, `-status`, … where the
+  prefix is `DEN_TARGET_PREFIX` (set before including `make/den.mk`; `den
+  create` sets it to the CLI name, so `make <name>-up` keeps working).
+- `service_env` has no built-in entries: a shared-infra secrets file is
+  configured as `service_env: { <KEY>: "{repo:<infra repo>}/<path>" }`.
+- The smoke and browser make targets have no default repo: pass `repo=<repo>`.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Monster Orange
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,167 @@
 # Den
 
-A config-driven local development appliance. This is a name placeholder; the first release is 0.1.0.
+Den is a config-driven local development appliance. It starts the repos you
+work on in dependency order, puts them behind a local HTTPS edge, gives you a
+web UI and a CLI over all of it, and runs smoke, browser, API, journey,
+security and performance suites against the environments you declare.
+
+Den knows nothing about any particular organisation. Everything specific to
+yours — product name, repos, environments, hosts, secrets backend, test
+accounts — lives in **your distribution**, not in Den.
+
+> **Status: 0.2.0**: `npm install @monster-orange/den`. Den is pre-1.0; each release lists the changes a distribution makes under **Migrations** in [CHANGELOG.md](CHANGELOG.md).
+
+## Make it your own
+
+1. **Create** a distribution named after your product:
+
+   ```sh
+   npx @monster-orange/den create acme      # → ./acme
+   cd acme && npm install
+   ```
+
+2. **Name** it: `den.yml` holds `product: { name: acme }`, so the CLI is
+   `./bin/acme`, env vars are `ACME_*`, state lives in `~/.acme`. Pass
+   `--cli`, `--env-prefix` or `--title` to `create`, or edit `den.yml`.
+3. **Brand** it: `brand:` in `config/acme.yml` sets the name, tagline and
+   banner people see in the CLI and web UI.
+4. **Point it at your repos**: set `github_org` in `config/acme.yml`, then
+   `./bin/acme register ../shop-api` (or add a manifest per repo under
+   `manifests/`).
+5. **Run** it: `./bin/acme up`, `./bin/acme web`, `make test`.
+
+Every field, the bin shim, the make fragment and upgrading are in
+[docs/distributions.md](docs/distributions.md);
+[examples/distribution/](examples/distribution) is what `create acme` writes.
+
+## Distributions
+
+You run Den through a small repo of your own (a _distribution_):
+
+```
+my-dev/
+├── den.yml               # who this distribution is (below)
+├── package.json          # pins an exact @monster-orange/den version
+├── config/<name>.yml     # your org config
+├── config/environments/  # one <env>.json per environment
+├── manifests/            # manifests for repos that don't carry their own
+├── tests/pillars/<pillar>/   # your own suites (e.g. tests/pillars/perf/<module>/test.js)
+├── bin/<cli>             # shim: DEN_ROOT=<this dir> exec node_modules/@monster-orange/den/bin/den
+└── Makefile              # include node_modules/@monster-orange/den/make/den.mk
+```
+
+`den.yml` names the product. Every runtime name derives from it:
+
+```yaml
+product:
+  name:
+    acme # CLI "acme", env vars ACME_*, config/acme.yml,
+    # manifests .acme.yml, state ~/.acme,
+    # managed clones .acme-services/, workspaces acme-workspaces/
+  # Override any of them: cli, env_prefix, config, manifest, state_dir,
+  # services_dir, workspaces_dir
+```
+
+Den's own checkout is a distribution named `den`, so in this repo the CLI is
+`./bin/den`, the config is `config/den.yml`, and env vars are `DEN_*`.
+
+## Repos opt in with a manifest
+
+A repo joins by carrying a manifest at its root (`.<name>.yml`, e.g.
+`.den.yml`) whose commands delegate to the repo's own targets. Den never
+reaches inside a repo beyond that. When a repo has no manifest, the
+distribution can hold one for it in `manifests/<repo>.<name>.yml`; the repo's
+own manifest always wins.
+
+```yaml
+# demo-api: a self-contained service. Nothing it calls, so it starts alone.
+version: 1
+name: demo-api
+application: shop # the application this repo belongs to (den apps)
+compose:
+  files: [docker-compose.yml]
+  routes:
+    api: api:8080 # https://api.demo-api.<edge domain>
+services:
+  demo-api:
+    commands:
+      local:
+        setup: make setup
+        preflight: make test
+# OWASP ZAP target for the security pillar (`make test-sec-api env=local`).
+security:
+  api:
+    hosts:
+      local: ["https://api.demo-api.globex.localhost"]
+```
+
+That is `examples/demo/demo-api/.den.yml`; the other demo repos are in
+[`examples/demo/`](examples/demo/).
+
+## What it does
+
+- **Workspaces.** A workspace is a set of repos. Each is either
+  _dev-managed_ (your checkout, your branch) or _managed_ (cloned by Den and
+  kept on a tracking branch).
+- **Derived dependency graph.** Start and stop order comes from real signals in
+  each repo — SDK imports, `*_URL` env vars, IaC references, event rules,
+  readiness probes — tuned in config with `graph.aliases`, `graph.edges` and
+  an optional shared local-infra repo (`graph.infra`).
+- **Local HTTPS edge.** One local proxy routes `https://<route>.<domain>` to
+  each service, with an optional project CA.
+- **Test pillars.** `smoke`, `browser` (synthetic Playwright), `api`,
+  `journeys` (agent-driven), `perf` (k6), `security` (OWASP ZAP against hosts
+  you own) and `suites` (a repo's own commands).
+- **Releases.** A release view over the repos and deploy workflows your config
+  lists.
+- **Web UI and CLI** over the same state.
+
+## Strategies
+
+Concerns that vary by org are strategies selected in config, never hard-wired:
+
+| Concern          | Built-in choices                        | Chosen in                                                                                                            |
+| ---------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
+| Secrets backend  | `env`, `doppler`, `1password`           | `secrets.provider` in the org config; per engineer in `config/local/user.yml`; per shell `<PREFIX>_SECRETS_PROVIDER` |
+| Test-user tokens | `local-jwt`, `cognito`, `auth0`, `none` | `auth.provider` in `config/environments/<env>.json`                                                                  |
+
+A new secrets backend implements `SecretsProvider`
+(`src/shared/secrets/types.ts`); a new identity provider implements a token
+provider in `src/shared/auth/token-providers/`.
+
+## CLI
+
+```bash
+den up [repo]          # start the repo's derived dependency closure (-x deps only, -n dry run)
+den down [repo]        # stop it cleanly
+den status             # service health for the active workspace
+den apps               # Org > Application > Repo > Services
+den graph [repo]       # the derived dependency graph
+den logs [repo]        # tail a repo's local logs
+den use-local [repo]   # use your checkout · use-managed [repo] to switch back
+den workspace list     # workspaces (activate · deactivate · use · create · delete)
+den run                # run test suites (interactive, or with options)
+den web                # the web UI
+den --help             # everything else
+```
+
+In a distribution the command is that distribution's `cli` name.
+
+## Developing Den
+
+```bash
+npm ci
+npm run typecheck
+npm run lint
+npm test
+DEN_ORG_NAMES_FILE=<patterns file> node scripts/org-name-gate.mjs
+```
+
+CI runs the same gates. The org-name gate fails the build when any file names
+an organisation, product, host or account; its patterns are kept out of this
+repo (a CI secret) so the list itself is never published. See
+[CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+[MIT](LICENSE)

package/bin/den

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Two roots (see src/shared/paths.ts):
+#   DEN_ENGINE_ROOT  Den's own package: this script's parent directory, which
+#                    in a distribution is node_modules/@monster-orange/den.
+#   DEN_ROOT         the distribution: the directory holding den.yml. A
+#                    distribution's bin/<cli> shim sets it; in Den's own
+#                    checkout (where the engine dir holds den.yml) it is set
+#                    here. Otherwise it is left unset and Den walks up from
+#                    the entry script to find den.yml.
+DEN_ENGINE_ROOT="$(cd "$(dirname "$(realpath "$0")")/.." && pwd)"
+export DEN_ENGINE_ROOT
+if [ -z "${DEN_ROOT:-}" ] && [ -f "$DEN_ENGINE_ROOT/den.yml" ]; then
+  DEN_ROOT="$DEN_ENGINE_ROOT"
+  export DEN_ROOT
+fi
+export DEN_INVOKE_CWD="$PWD"
+export OP_ACCOUNT="${OP_ACCOUNT:-example.1password.com}"
+
+# tsx: Den's own (a checkout) or hoisted into the distribution's node_modules.
+find_tsx() {
+  local dir="$DEN_ENGINE_ROOT"
+  while :; do
+    if [ -x "$dir/node_modules/.bin/tsx" ]; then
+      printf '%s\n' "$dir/node_modules/.bin/tsx"
+      return 0
+    fi
+    [ "$dir" = "/" ] && return 1
+    dir="$(dirname "$dir")"
+  done
+}
+
+# `create` scaffolds a new distribution: it needs no den.yml, so it runs on
+# its own entry point rather than the full CLI.
+entry="src/ui/cli/index.ts"
+built="dist/src/ui/cli/index.js"
+if [ "${1:-}" = "create" ]; then
+  shift
+  entry="src/ui/cli/create-main.ts"
+  built="dist/src/ui/cli/create-main.js"
+fi
+# `mcp` speaks the MCP protocol on stdout: its own entry point, so nothing
+# the full CLI prints can reach the protocol stream.
+if [ "${1:-}" = "mcp" ]; then
+  shift
+  entry="src/ui/cli/mcp-main.ts"
+  built="dist/src/ui/cli/mcp-main.js"
+fi
+
+# Escape hatch: force the compiled build (e.g. to test the prod artifact).
+if [ "${DEN_USE_DIST:-}" = "1" ] && [ -f "$DEN_ENGINE_ROOT/$built" ]; then
+  exec node "$DEN_ENGINE_ROOT/$built" "$@"
+fi
+
+# Prefer live TypeScript source (the package ships src/ and runs it with tsx).
+# A built dist/ is gitignored and only refreshed by `npm run build`, so
+# preferring it would silently shadow local source edits with a stale build.
+if tsx="$(find_tsx)"; then
+  exec "$tsx" "$DEN_ENGINE_ROOT/$entry" "$@"
+fi
+
+if [ -f "$DEN_ENGINE_ROOT/$built" ]; then
+  exec node "$DEN_ENGINE_ROOT/$built" "$@"
+fi
+
+echo "den: no runnable entrypoint: run 'npm install'" >&2
+exit 1