kitsune-kit 0.4.1 → 0.6.0

data/docs/security.md

@@ -0,0 +1,116 @@
+# Security model
+
+Kitsune Kit changes remote security policy and can destroy infrastructure. Its design favors explicit ownership, verified transitions and refusal over implicit adoption.
+
+## Trust boundaries
+
+Kitsune Kit trusts:
+
+- the local machine/user running the gem;
+- configuration committed by the project;
+- secrets supplied through the process environment;
+- the independently verified first SSH host-key fingerprint;
+- provider and package repositories over TLS;
+- local state as the record of which resources Kitsune Kit owns.
+
+Compromise of the local account, repository, environment, private key or state can invalidate these assumptions. Kitsune Kit is not a secrets vault and does not encrypt local files.
+
+## Secrets
+
+Never place API tokens or service passwords in YAML, command arguments, state or Git. Configuration stores only environment-variable names. Secret values are registered with a central redactor before event reporting/logging.
+
+```bash
+export DO_API_TOKEN="..."
+export POSTGRES_PASSWORD="..."
+export REDIS_PASSWORD="..."
+```
+
+Generated service `.env` files exist only on the remote server with mode `0600`. Compose output, fingerprints and local state do not contain plaintext passwords. Newlines, carriage returns and NUL bytes are rejected.
+
+Process environments may be observable by privileged local processes and CI administrators. Use the secret store of your CI platform, limit token scope, rotate credentials and avoid shell tracing.
+
+## SSH
+
+Kitsune Kit never uses `verify_host_key: :never` in production adapters. A new key raises a prompt containing the exact SHA256 fingerprint. Acceptance records it in `.kitsune/known_hosts`; later sessions use strict matching.
+
+`--trust-host-key` accepts only an exact fingerprint and is intended for independently verified non-interactive bootstrap. A mismatch fails. Key changes are not automatically accepted.
+
+SSH hardening preserves a recovery route:
+
+1. bootstrap using deploy or root access;
+2. create/configure the deploy user;
+3. verify a second deploy-user connection;
+4. validate generated `sshd_config` with `sshd -t`;
+5. reload first with key-based root recovery still enabled;
+6. verify a new deploy-user connection, then disable root and verify again;
+7. add the new firewall route before removing a stale route owned by Kitsune Kit and verify deploy access once more.
+
+The transport keeps the already authenticated session open across each safety-critical transition. Fresh verification uses a separate connection; if it fails before or after finalization, the preserved session runs the script rollback before it closes. A recovery failure is surfaced as a verification error with provider-console guidance. Kitsune Kit does not modify unrelated SSH/firewall configuration during rollback.
+
+## Command and file safety
+
+Remote commands are a validated executable plus separately Shellwords-escaped arguments. File uploads are base64 data over stdin, not interpolated shell source. Absolute remote paths and modes are validated before connecting.
+
+Generated Compose documents are built as Ruby hashes and serialized with YAML. Resource names, users, ports, CIDRs, images, domains and environment names have allow-list validation. Tests include metacharacters, substitutions, traversal and embedded newline attacks.
+
+Versioned shell scripts use `set -Eeuo pipefail`, explicit action dispatch and verification. ShellCheck runs in CI.
+
+## Firewall and data services
+
+PostgreSQL and Redis do not publish host ports by default. When publishing is enabled:
+
+- at least one valid allowed CIDR is required;
+- Compose binds only the configured IPv4 address/port;
+- matching allow and terminal drop rules are installed in Docker's `DOCKER-USER` chain because Docker-published ports can bypass UFW forwarding policy;
+- only rules recorded as created by Kitsune Kit are reconciled or removed.
+
+Use private networking or an application-local Docker network whenever possible. Never publish a database to `0.0.0.0` with a broad CIDR unless the exposure is deliberate and independently audited.
+
+## Ownership and destructive actions
+
+Kitsune Kit records provider IDs, DNS record IDs, remote fingerprints, file backups and firewall ownership. Same-name resources without the expected tags/state are not adopted or deleted.
+
+Semantics:
+
+| Operation | Server | Containers | Data volume | Configuration |
+| --- | --- | --- | --- | --- |
+| `rollback` | Keep | Restore/remove managed | Keep | Restore captured state |
+| service `remove` | Keep | Remove | Keep | Keep ownership record |
+| service `destroy-data` | Keep | Remove | Delete | Delete managed service files/state |
+| server `destroy` | Delete | Delete with server | Delete with server | Restore/remove managed DNS after provider deletion |
+
+`--yes` never authorizes permanent service data or server destruction. Exact confirmation strings are required.
+Interactive service-data destruction offers a restricted archive first. In automation,
+`--backup-before-destroy` requests it explicitly; copy that same-server archive to independent storage.
+
+The deploy user belongs to the `docker` group and has passwordless sudo, so it is effectively a privileged
+administrator. Protect its private key as a root credential, restrict SSH CIDRs where possible and rotate the
+key after suspected exposure.
+
+## State, logs and diagnostics
+
+State/log/support directories and files use restrictive permissions. State writes use file locks, fsync, atomic rename and backups. Logs are structured, redacted and rotated. Support bundles are local JSON and never transmitted.
+
+Back up `.kitsune/state/` securely. If state is lost, recover it rather than recreating ownership from names. Inspect support bundles before sharing; redaction is defense in depth, not proof that arbitrary user-provided text contains no sensitive information.
+
+When both state copies are lost, `kit server import` is the limited recovery path. It requires an exact numeric provider ID and exact-name confirmation, verifies the complete configured server identity, and imports only server metadata. It deliberately does not claim remote resources or DNS records.
+
+## Supply chain
+
+- Docker is installed from Docker's official signed apt repository.
+- Metrics installation is off by default. Enabling it requires an explicitly configured SHA256 for the downloaded script; no `curl | sh` pipeline is used.
+- Ruby dependencies are locked for development, audited with Bundler Audit and updated through Dependabot.
+- Release metadata requires MFA on RubyGems. Releases should use trusted publishing and protected tags when configured.
+
+Container service tags are mutable unless pinned by digest. For production reproducibility, use an image reference ending in `@sha256:...`, test it in staging and update deliberately.
+
+The internal Alpine image used only to archive service volumes is multi-platform and pinned to the published Docker Official Image manifest digest. Its source is the [Docker Hub `alpine:3.20` manifest](https://hub.docker.com/layers/library/alpine/3.20/images/sha256-ac77ebc035f69184acb2660028580c9053f6d0f892de7933e1456d8b5e0ac085). Updating it requires reviewing the new manifest and running backup/restore tests.
+
+Kitsune Kit contains no telemetry or automatic upload path. Logs and support bundles remain local unless the
+operator deliberately shares them.
+
+The latest manual review and its limitations are recorded in [Security audit](security-audit.md).
+
+## Reporting vulnerabilities
+
+Follow [SECURITY.md](../SECURITY.md). Do not include real credentials, addresses or domains in a public issue or reproduction.

data/docs/services/compose.md

@@ -0,0 +1,164 @@
+# Compose customization
+
+Kitsune Kit can manage PostgreSQL and Redis with a generated Compose document, a user overlay, or a complete
+custom document. All three modes use the same `plan`, `apply`, state, recovery, rollback and data-safety workflow.
+The TUI is optional; these features are fully available through ordinary CLI commands.
+
+## Choosing a mode
+
+| Mode | Kitsune Kit generates the base | Your file | Best use |
+| --- | --- | --- | --- |
+| `generated` | Yes | None | Secure defaults with image, port and password settings controlled by `config.yml`. |
+| `overlay` | Yes | Compose override | Add labels, resource limits, logging, environment options or other supported overrides. |
+| `custom` | No | Complete Compose document | Take direct ownership of the service definition while retaining Kitsune Kit lifecycle management. |
+
+Start with `generated`. Prefer `overlay` while the base service, volume, network and health check still fit. Use
+`custom` when you need to replace those assumptions. There is no legacy-blueprint compatibility layer.
+
+## Generated mode
+
+```yaml
+services:
+  postgres:
+    compose:
+      mode: generated
+      file:
+      allow_unsafe: false
+```
+
+The generated document is deterministic. Passwords appear only as `${POSTGRES_PASSWORD}` or
+`${REDIS_PASSWORD}` references; their values are uploaded separately in a mode-`0600` `.env` file.
+
+## Overlay mode
+
+Create a project file such as `.kitsune/compose/postgres.override.yml`:
+
+```yaml
+services:
+  postgres:
+    environment:
+      LOG_STATEMENT: ddl
+    logging:
+      options:
+        max-size: 20m
+        max-file: "5"
+    deploy:
+      resources:
+        limits:
+          memory: 1G
+```
+
+Reference it from configuration:
+
+```yaml
+services:
+  postgres:
+    compose:
+      mode: overlay
+      file: .kitsune/compose/postgres.override.yml
+      allow_unsafe: false
+```
+
+Kitsune Kit uploads `compose.yml` followed by `compose.override.yml` and supplies both to every Docker Compose
+command in that order. Changes to either document change the plan fingerprint.
+
+## Custom mode and eject
+
+Create a complete starting document automatically:
+
+```bash
+kit service postgres compose eject
+```
+
+This operation:
+
+1. writes `.kitsune/compose/postgres.yml` from the current generated definition;
+2. copies `.kitsune/config.yml` to `.kitsune/config.yml.backup`;
+3. changes the service to `compose.mode: custom` and references the new file.
+
+Existing output or backup files are not overwritten. Review both and use `--force` only when replacement is
+intentional. YAML comments in `config.yml` may be reformatted by ejection; the byte-for-byte backup is retained.
+
+A custom PostgreSQL file must contain `services.postgres`; Redis must contain `services.redis`. Configuration
+fields such as `image` no longer alter a custom document, but `password_env`, publication/firewall policy and
+lifecycle state remain part of Kitsune Kit's safety model.
+
+## Inspecting and validating
+
+```bash
+kit service postgres compose show
+kit service postgres compose validate
+kit service postgres compose diff
+kit service postgres compose show --format json
+```
+
+- `show` displays the exact ordered documents and their sources.
+- `validate` parses YAML, validates the required service and applies local security policy.
+- `diff` compares desired content with the last successfully applied Compose fingerprint.
+- `plan` includes mode, ordered files, findings and the overall service fingerprint.
+- `apply` uploads restricted files and runs `docker compose config --quiet` remotely before `up`.
+
+Local validation does not replace Docker's own validation. A customization can pass local checks but fail because
+an option is unsupported by the Docker Compose version on the VPS; `apply` then recovers the previous managed
+files and attempts to restart the previous file set.
+
+## File boundary
+
+Customization files must:
+
+- be inside `--root` after path expansion and resolution;
+- be regular files, not symlinks;
+- be at most 256 KiB;
+- contain a top-level YAML mapping;
+- avoid YAML aliases.
+
+These constraints keep the deployment input reviewable and prevent a project configuration from reading an
+arbitrary local file.
+
+## Secrets
+
+Never put a secret value in Compose YAML. Sensitive mapping keys and `KEY=value` environment-list entries are
+rejected unless the value is a `${VARIABLE_NAME}` reference.
+
+```yaml
+services:
+  postgres:
+    environment:
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
+```
+
+Export the variable before `doctor`, `plan`, `apply` or `install`. Compose display/JSON output never resolves its
+value. The remote `.env`, Compose documents and backups use restricted permissions.
+
+## Unsafe options
+
+The local policy flags options that cross Kitsune Kit's normal isolation boundary, including:
+
+- privileged containers and host PID/IPC/network namespaces;
+- `ALL`, `SYS_ADMIN`, `SYS_PTRACE` or `NET_ADMIN` capabilities;
+- host devices, absolute bind mounts and the Docker socket;
+- ports outside the exact configured bind/port/firewall model;
+- remote `build` contexts and unmanaged `env_file` references.
+
+By default, any finding stops `show`, `validate`, `plan` and `apply`. If the configuration is intentionally outside
+the standard boundary, review every finding and set:
+
+```yaml
+compose:
+  mode: custom
+  file: .kitsune/compose/postgres.yml
+  allow_unsafe: true
+```
+
+This is an escape hatch, not a promise that Kitsune Kit manages or reverses the extra host resources. Findings
+remain visible in metadata and plan details. Inline secrets are always rejected and cannot be enabled by
+`allow_unsafe`.
+
+## Rollback behavior
+
+State records the mode, ordered filenames and content fingerprint after a verified apply. Before an update,
+Kitsune Kit backs up all currently tracked and desired Compose files plus `.env`. A failed apply removes the new
+file set, restores the backup and starts the prior ordered file set. `kit rollback` uses the same recorded set.
+
+Docker volumes remain governed by the existing service contract: `remove` preserves data and `destroy-data`
+requires exact confirmation. Compose customization does not broaden deletion ownership.

data/docs/services/postgres.md

@@ -0,0 +1,78 @@
+# PostgreSQL service
+
+PostgreSQL is an optional Docker Compose service managed by Kitsune Kit on the target server. It is disabled and private by default.
+
+## Enable
+
+```yaml
+services:
+  postgres:
+    enabled: true
+    image: postgres:17
+    publish: false
+    bind: 127.0.0.1
+    allowed_cidrs: []
+    port: 5432
+    password_env: POSTGRES_PASSWORD
+```
+
+```bash
+export POSTGRES_PASSWORD="$(ruby -rsecurerandom -e 'print SecureRandom.base64(36)')"
+kit plan
+kit service postgres install
+```
+
+The Compose project is `kitsune-ENV-postgres`, attached to the external `kitsune-private` network. Data is stored in its named `data` volume. The database name is `app_ENV` with `-` converted to `_`; the user is `postgres`.
+
+## Connectivity
+
+With `publish: false`, there is no host port mapping. Applications on `kitsune-private` can connect using Docker service/network addressing and the configured credentials.
+
+Publishing is an explicit exception:
+
+```yaml
+publish: true
+bind: 0.0.0.0
+allowed_cidrs:
+  - 203.0.113.10/32
+```
+
+Kitsune Kit requires at least one CIDR and installs owned `DOCKER-USER` allow/drop rules. Validate the result externally; UFW alone does not reliably restrict Docker-published ports.
+
+## Status, backup and removal
+
+```bash
+kit service postgres status
+kit service postgres backup
+kit service postgres remove --yes
+```
+
+Backup pauses the service, archives the named volume into a restricted remote backup directory and unpauses in an `ensure` path. The returned path is on the server. Copy it to independent storage and test restoration; creating an archive on the same Droplet is not disaster recovery.
+
+`remove` runs Compose down without `--volumes`, removes only firewall rules owned by Kitsune Kit and marks the service removed. Data remains.
+
+## Permanent destruction
+
+```bash
+kit service postgres backup
+kit service postgres destroy-data --backup-before-destroy --confirm-destroy postgres@production
+```
+
+In an interactive terminal, `destroy-data` offers to create the archive and then requires typing
+`postgres@ENV`. Automation uses `--backup-before-destroy` when desired and must always provide the exact
+`--confirm-destroy` value. The data archive is retained, while Compose volumes, managed configuration
+backups/files, markers and service state are removed. Copy the archive off the server before relying on it;
+`--yes` cannot replace the exact confirmation.
+
+## Updates and rollback
+
+Changing the image/configuration/secret changes the service fingerprint. Before an update, Kitsune Kit backs up managed Compose/env files. It validates Compose, waits for health, reconciles firewall rules and only then records the new state. Failed updates restore previous files and restart the prior service where possible.
+
+`kit rollback` restores captured managed configuration and preserves the volume. It does not promise database-level downgrade compatibility; test image upgrades and rollback in staging.
+
+## External PostgreSQL
+
+Set `enabled: true`, `mode: external`, `host` and `port` to represent a provider-managed database without
+installing it on the VPS. `kit service postgres status` reports the endpoint; install, backup, remove and
+destroy-data are intentionally unavailable. Kitsune Kit does not test provider TLS, create users/databases or
+manage external backups. Supply those through the database provider and application deployment system.

data/docs/services/redis.md

@@ -0,0 +1,53 @@
+# Redis service
+
+Redis is an optional Docker Compose service managed by Kitsune Kit on the target server. It is disabled, password-protected and private by default.
+
+## Enable
+
+```yaml
+services:
+  redis:
+    enabled: true
+    image: redis:7.2
+    publish: false
+    bind: 127.0.0.1
+    allowed_cidrs: []
+    port: 6379
+    password_env: REDIS_PASSWORD
+```
+
+```bash
+export REDIS_PASSWORD="$(ruby -rsecurerandom -e 'print SecureRandom.base64(36)')"
+kit plan
+kit service redis install
+```
+
+The generated service enables append-only persistence, requires the password and attaches to `kitsune-private`. Data lives in the Compose named `data` volume.
+
+## Connectivity and publishing
+
+With `publish: false`, only containers on the private Docker network can connect. If TCP access from outside the host is unavoidable, set `publish: true`, choose a bind address and provide narrow `allowed_cidrs`. Kitsune Kit installs matching `DOCKER-USER` rules and a final drop rule it owns.
+
+Never treat the Redis password as an adequate substitute for network isolation. Validate port 6379 from an untrusted external network after any firewall/Docker change.
+
+## Lifecycle
+
+```bash
+kit service redis status
+kit service redis backup
+kit service redis remove --yes
+kit service redis destroy-data --backup-before-destroy --confirm-destroy redis@production
+```
+
+- `backup` pauses Redis, archives its volume remotely, then unpauses even if archiving fails.
+- `remove` removes the container and owned firewall rules but preserves the data volume.
+- `destroy-data` removes the volume, service files, backups and state after exact confirmation.
+- `kit rollback` restores captured managed files/settings while preserving data.
+
+Copy backups away from the Droplet, control access to them and regularly test restoration. Image downgrades may not understand data written by a newer Redis version; operational rollback and data-format compatibility are separate concerns.
+
+## External Redis
+
+Set `enabled: true`, `mode: external`, `host` and `port` for a provider-managed endpoint. Status is available,
+but Kitsune Kit refuses install, backup, remove and destroy-data because those belong to the external provider. TLS,
+availability and backup policy remain the operator/provider responsibility.

data/docs/testing.md

@@ -0,0 +1,200 @@
+# Testing and quality gates
+
+Kitsune Kit uses layers because no single test environment accurately represents provider APIs, SSH, systemd, UFW, Docker and public network exposure.
+
+## Local commands
+
+```bash
+bin/setup                         # install gems and run the local gate
+bundle exec rake test             # complete normal RSpec suite + coverage gate
+bundle exec rake lint             # RuboCop + ShellCheck
+bundle exec rake security         # Bundler Audit
+bundle exec rake build            # build the gem
+bundle exec rake artifact_smoke   # install the gem with fresh dependencies and verify its CLI
+bundle exec rake ci               # lint, tests and gem build
+bundle exec rake integration      # Docker-backed SSH/script suite
+```
+
+Run an individual spec without applying whole-suite coverage thresholds:
+
+```bash
+bundle exec rspec spec/unit/configuration_spec.rb
+```
+
+Set `KITSUNE_COVERAGE_GATE=1` to force the full threshold for a custom selection or `0` for diagnostic subsets. CI always runs the complete gate.
+
+## Test layers
+
+### Unit/domain
+
+Fast tests cover configuration precedence/validation, plans, errors/exit codes, redaction, reporters, state locking/writes, run journals, operation idempotence, rollback calculations, workflow resume/failure and TUI state/rendering. No network is used.
+
+### Fakes and contracts
+
+`FakeProvider`, `FakeTransport`, `FakeStateStore` and `FakeReporter` are deterministic memory adapters with
+recorded calls or captured events. Shared examples define provider, transport and state-store behavior. Real
+adapters execute those same contracts against simulated SDK/SSH boundaries, including timeouts, exact IDs,
+status channels and hostile arguments.
+
+### CLI process tests
+
+The executable runs as a subprocess to verify global/help/version behavior, invalid syntax, exit codes, JSON parsing, non-TTY behavior and environment selection. Core behavior is separately tested through workflows so subprocess coverage is not mistaken for domain coverage.
+
+### Shell scripts
+
+Every remote script passes ShellCheck and `bash -n`. CI mounts scripts read-only into both supported Ubuntu images to ensure the distributed Bash parses there.
+
+A simple container does not prove systemd/UFW behavior. The project does not claim otherwise; those effects are checked on a VPS in the E2E layer.
+
+### Ephemeral SSH integration
+
+`rake integration` builds an Ubuntu OpenSSH container with a temporary Ed25519 key. It verifies real public-key authentication, stdout/stderr/status separation, safe quoting, exact-byte upload/mode, host-key rejection, authentication failure and command timeout. It also uses a pseudo-terminal to prove that the TUI restores the main screen and console mode after normal exit and `SIGINT`. CI runs it on Ubuntu 22.04 and 24.04 base images.
+
+Docker must be installed and running. If unavailable, examples report why they were skipped locally; the CI integration job first requires `docker info`, so it cannot silently pass without Docker.
+
+### DigitalOcean E2E
+
+The E2E suite is billable and never runs from a normal test command. Required variables:
+
+```text
+KITSUNE_E2E=1
+DO_API_TOKEN
+KITSUNE_E2E_SSH_KEY_ID
+KITSUNE_E2E_KEY_PATH
+POSTGRES_PASSWORD (generated by the test workflow)
+REDIS_PASSWORD (generated by the test workflow)
+```
+
+Use a dedicated DigitalOcean token and SSH key pair for this suite. The private key must not have a passphrase
+because the GitHub runner cannot answer an unlock prompt. Generate a test-only Ed25519 pair locally:
+
+```bash
+ssh-keygen -t ed25519 -f ~/.ssh/kitsune_e2e_ed25519 -C kitsune-e2e -N ""
+chmod 0600 ~/.ssh/kitsune_e2e_ed25519
+```
+
+Upload only `~/.ssh/kitsune_e2e_ed25519.pub` in DigitalOcean under **Settings → Security → SSH keys → Add SSH
+Key**. Never upload or paste the private file into DigitalOcean. `KITSUNE_E2E_SSH_KEY_ID` is the numeric ID
+assigned to that uploaded public key, not its fingerprint or path. With `DO_API_TOKEN` already exported, list
+the available IDs without printing the token:
+
+```bash
+curl --fail --silent --show-error \
+  -H "Authorization: Bearer ${DO_API_TOKEN}" \
+  https://api.digitalocean.com/v2/account/keys |
+  ruby -rjson -e 'JSON.parse(STDIN.read).fetch("ssh_keys").each { |key| puts [key["id"], key["name"], key["fingerprint"]].join("\t") }'
+```
+
+The minimum custom token scopes used by this suite are `account:read`, `droplet:read`, `droplet:create`,
+`droplet:delete`, `regions:read`, `sizes:read`, `actions:read`, `image:read`, `ssh_key:read`, `tag:read` and
+`tag:create`. The dynamic expiry tag makes `tag:create` necessary. DNS scopes are not required by the current
+E2E configuration. DigitalOcean does not allow changing an existing token's scopes; create a replacement token
+if any permission is missing.
+
+Before authorizing a billable run, confirm that the token can access the endpoint used by `doctor`:
+
+```bash
+curl --output /dev/null --silent --show-error --write-out '%{http_code}\n' \
+  -H "Authorization: Bearer ${DO_API_TOKEN}" \
+  https://api.digitalocean.com/v2/account
+```
+
+The expected status is `200`; `403` means the token lacks `account:read` or the selected team role does not grant
+it.
+
+For a local run, keep the token out of shell history and export the remaining values:
+
+```bash
+printf 'DigitalOcean token: '
+IFS= read -r -s DO_API_TOKEN
+printf '\n'
+export DO_API_TOKEN
+export KITSUNE_E2E_SSH_KEY_ID="12345678"
+export KITSUNE_E2E_KEY_PATH="$HOME/.ssh/kitsune_e2e_ed25519"
+export KITSUNE_E2E=1
+bundle exec rake e2e
+```
+
+The optional `KITSUNE_E2E_REGION` and `KITSUNE_E2E_SIZE` variables override the defaults `sfo3` and
+`s-1vcpu-1gb` when that combination is unavailable.
+
+For GitHub Actions, add these repository secrets under **Settings → Secrets and variables → Actions**:
+
+| GitHub secret | Value |
+| --- | --- |
+| `DO_API_TOKEN` | Dedicated DigitalOcean token |
+| `KITSUNE_E2E_SSH_KEY_ID` | Numeric ID of the uploaded public key |
+| `KITSUNE_E2E_KEY` | Complete contents of the matching private key file |
+
+GitHub writes `KITSUNE_E2E_KEY` to a restricted temporary file and sets `KITSUNE_E2E_KEY_PATH` itself. After
+the workflow exists on GitHub, run **Actions → Real infrastructure → Run workflow**. It has no schedule and
+therefore cannot create billable infrastructure unless someone dispatches it explicitly.
+
+Run only in a dedicated account/project:
+
+```bash
+KITSUNE_E2E=1 bundle exec rake e2e
+```
+
+It creates a unique TTL-tagged Droplet, applies the complete configuration including PostgreSQL/Redis, runs CLI doctor/plan, demands a zero-change second plan, checks data ports from outside, rolls back and destroys in `ensure`.
+
+Before creating anything, the suite derives the public half of `KITSUNE_E2E_KEY_PATH` and compares it with the
+public key returned by DigitalOcean for `KITSUNE_E2E_SSH_KEY_ID`, then validates account API access. A key
+mismatch or missing `account:read` permission fails without creating a billable Droplet. After creation, Kitsune Kit
+waits up to two minutes for Ubuntu SSH readiness because an `active` API status and public IP do not guarantee
+that `sshd` has finished starting.
+
+The manually dispatched workflow serializes E2E runs and has a 45-minute limit. Its always-run cleanup job invokes
+`script/cleanup-ci-resources`, which is dry-run by default and only considers `kitsune-ci` Droplets with an expired
+`kitsune-expires-EPOCH` tag:
+
+```bash
+bundle exec ruby script/cleanup-ci-resources
+bundle exec ruby script/cleanup-ci-resources --execute
+```
+
+Unit tests inject a fake Droplet API into the cleanup class and prove that dry-run never deletes, untagged or
+unexpired resources are ignored, and `--execute` uses only the filtered provider IDs.
+
+## Coverage
+
+SimpleCov records line and branch coverage. The gate is raised with the new core and must not be improved by excluding difficult production files. Destructive paths require explicit examples even when another path happens to cover the same line.
+
+The complete suite enforces at least 80% line/50% branch coverage across all production code and 85% branch coverage in the domain core. For this gate, the core is `lib/kitsune/kit` excluding presentation (`cli.rb`, `tui/`, `reporters/`) and external adapters; operations/workflows/configuration/state remain included. `script/verify-core-coverage` calculates this directly from SimpleCov's current result.
+
+Coverage is evidence of exercised control flow, not proof of provider/system correctness; contract, integration, security and E2E tests remain mandatory.
+
+The dated manual source review, remediated findings and residual infrastructure risks are in
+[Security audit](security-audit.md).
+
+## CI jobs
+
+Pull requests run:
+
+- unit tests on every declared Ruby family;
+- CLI process tests;
+- adapter contracts;
+- Ruby/shell lint;
+- dependency audit;
+- gem build, isolated installation and executable smoke test;
+- Docker SSH/script integration for both supported Ubuntu releases.
+
+The credential-gated, manually dispatched workflow runs the DigitalOcean lifecycle and cleanup. Secrets are
+never available to untrusted pull-request jobs, and no calendar trigger creates billable infrastructure.
+
+The artifact job deliberately resolves runtime dependencies without the repository lockfile, runs from outside
+the checkout, verifies the exact public command set, and initializes a temporary project. This catches undeclared
+standard-library gems and framework commands that a locked development bundle could hide.
+
+## Writing a new operation test
+
+At minimum cover:
+
+1. absent resource plan/apply/verification;
+2. already-correct no-change behavior;
+3. allowed update and immutable/destructive drift;
+4. failure before and after mutation;
+5. resume after a persisted partial step;
+6. rollback of owned state and refusal to touch pre-existing state;
+7. timeout/cancellation;
+8. hostile input and secret redaction.