@a5c-ai/krate 5.0.1-staging.00fa5317c

package/docs/tests/observability-reliability-tests.md

@@ -0,0 +1,54 @@
+# Observability and reliability tests
+
+## Observability coverage
+
+Required signals:
+
+- API request latency and errors;
+- controller reconcile counts, durations, retries, and failures;
+- watch connection counts and reconnects;
+- Git operation latency/errors;
+- webhook queue depth and delivery status;
+- runner queue/wait/runtime metrics;
+- memory query latency and import validation status;
+- Agent Mux session binding and event stream status;
+- audit event counts by action/outcome.
+
+## Reliability tests
+
+| Failure | Expected behavior |
+| --- | --- |
+| Kubernetes API temporary failure | retry with backoff, status condition, no duplicate side effects. |
+| Gitea unavailable | repository status degraded, no data loss, UI warning. |
+| Postgres unavailable | aggregated API degraded/read-only where possible. |
+| object storage unavailable | artifact writes fail safely with retry. |
+| webhook receiver fails | retry and replay available. |
+| watch disconnects | UI reconnects and resumes from list state. |
+| memory repo unavailable | required-memory dispatch blocks, optional memory warns. |
+| Agent Mux unavailable | dispatch shows pending/failed handoff and retry/recover action. |
+| redaction failure | memory import blocks and no content leaks. |
+
+## Chaos and load
+
+Nightly/staging tests should eventually cover:
+
+- burst webhook deliveries;
+- many repository list queries;
+- concurrent dispatches;
+- runner pool exhaustion;
+- memory grep/query bounds;
+- large context truncation;
+- controller restart during reconciliation;
+- duplicate event delivery idempotency.
+
+## Audit assertions
+
+Every mutating or denied action should emit audit with:
+
+- org and namespace;
+- actor;
+- resource ref;
+- action and outcome;
+- source event/run/session when applicable;
+- digest fields for artifacts/context/memory;
+- no secret values.

package/docs/tests/product-test-matrix.md

@@ -0,0 +1,145 @@
+# Product test matrix
+
+## Purpose
+
+This matrix maps Krate product areas to required automated test coverage. It covers existing functionality and future agent/company-brain functionality so implementation work can add tests in the right layer instead of relying on one broad E2E path.
+
+## Matrix legend
+
+| Mark | Meaning |
+| --- | --- |
+| Required | must exist before feature is considered complete. |
+| Recommended | should exist when the feature reaches production or staging. |
+| Nightly | acceptable in slower scheduled/live suites. |
+| Future | planned once the underlying feature exists. |
+
+## Product-area coverage
+
+| Product area | Unit | Integration/API | Browser/UI | E2E/scenario | Security | Package/install |
+| --- | --- | --- | --- | --- | --- | --- |
+| Resource model and schemas | Required | Required | Recommended | Recommended | Recommended | Required |
+| Organization and namespace scoping | Required | Required | Required | Required | Required | Required |
+| Repository data plane | Required | Required | Required | Required | Required | Required |
+| Pull requests and reviews | Required | Required | Required | Required | Recommended | Recommended |
+| Issues and inbox | Required | Required | Required | Recommended | Recommended | Recommended |
+| Pipelines and jobs | Required | Required | Required | Required | Required | Required |
+| Runner pools and job isolation | Required | Required | Recommended | Required | Required | Required |
+| Webhook subscriptions and deliveries | Required | Required | Required | Required | Required | Recommended |
+| Identity, auth, teams, invites | Required | Required | Required | Recommended | Required | Required |
+| RBAC and policy | Required | Required | Recommended | Required | Required | Required |
+| Secrets and config grants | Required | Required | Required | Required | Required | Recommended |
+| Deployments and environments | Required | Required | Required | Required | Recommended | Required |
+| Argo CD and KubeVela/OAM | Required | Required | Recommended | Nightly | Recommended | Required |
+| Operations install/readiness | Required | Required | Required | Required | Required | Required |
+| Web UI shell and navigation | Recommended | Recommended | Required | Required | Recommended | Recommended |
+| Advanced YAML/resource panels | Required | Required | Required | Recommended | Required | Recommended |
+| Agent stacks and capabilities | Future | Future | Future | Future | Future | Future |
+| Agent dispatch and Agent Mux | Future | Future | Future | Future | Future | Future |
+| Agent triggers | Future | Future | Future | Future | Future | Future |
+| Agent workspaces and sessions | Future | Future | Future | Future | Future | Future |
+| Company brain memory | Future | Future | Future | Future | Future | Future |
+| `.a5c` run memory imports | Future | Future | Future | Future | Future | Future |
+| Artifacts and write-back | Future | Future | Future | Future | Future | Future |
+| Packaging and release | Required | Required | Recommended | Required | Required | Required |
+
+## Existing command mapping
+
+| Command | Covers | Gaps |
+| --- | --- | --- |
+| `npm test` | unit/integration tests in `tests/*.test.js` | not yet split by subsystem; no coverage report. |
+| `npm run e2e` | current deterministic package/minikube E2E tests | no browser automation or live cluster path. |
+| `npm run validate:docs` | docs/source/ontology coverage | does not validate all `docs/tests` requirements yet. |
+| `npm run package:check` | package/chart/example coverage | not yet aware of future agent/memory CRDs. |
+| `npm run smoke` | runtime smoke | should expand as APIs/routes grow. |
+| `npm run ui:validate` | static UI validation | not a browser test. |
+| `npm run ui:build` | Next production build | not behavioral UI coverage. |
+| `npm run check` | all current gates | should remain required as new gates are added. |
+
+## Future suite mapping
+
+| Future suite | Product areas |
+| --- | --- |
+| `test:unit` | resource model, route helpers, redaction, context assembly, ref resolution, validators. |
+| `test:integration` | API controller, controller fakes, memory import, Gitea/K8s/Agent Mux fakes. |
+| `test:api` | org-scoped endpoints, stable errors, resource actions, watch filters. |
+| `test:browser` | org navigation, repository flows, deployments, run detail, agent/memory flows. |
+| `test:coverage` | coverage thresholds and untested-file reporting. |
+| `test:security` | auth/RBAC/no-secret/cross-org/secret-grant checks. |
+| `test:charts` | Helm render, CRD examples, kubeconform, APIService/RBAC. |
+| `test:agents` | agent dispatch, context, memory, Agent Mux, imports, triggers. |
+| `test:live` | real cluster/Gitea/Argo/KubeVela/NATS/ARC/Object storage. |
+
+## Required negative coverage
+
+Every relevant product area must include negative tests for:
+
+- missing or mismatched `organizationRef`;
+- wrong namespace for org;
+- missing RBAC permission;
+- missing Secret/ConfigMap grant;
+- untrusted fork or untrusted runner tries privileged action;
+- cross-org resource reference;
+- invalid or stale Git ref;
+- invalid webhook signature;
+- resource deleted while a controller is reconciling;
+- secret-like value appears in input and must not appear in output;
+- watch reconnect after disconnect;
+- duplicate event delivery and idempotency.
+
+## Release readiness matrix
+
+A release candidate is blocked if any of these are missing:
+
+- package/chart validation;
+- CRD/example coverage for every shipped kind;
+- at least one install smoke path;
+- auth/RBAC/no-secret tests;
+- UI build and route smoke;
+- repository/PR/CI core E2E;
+- deployment/OAM smoke when deployment features ship;
+- agent/company-brain vertical slice when agent features ship;
+- documented known gaps and quarantined tests.
+
+## External backend coverage
+
+External provider support adds required coverage for:
+
+| Area | Required tests |
+| --- | --- |
+| Provider auth | GitHub App Secret metadata, installation access, no-token leak. |
+| Webhooks | signature validation, dedupe, replay, enqueue, malformed payload. |
+| Issue interface | issue/comment/label sync, PR-backed issue handling, conflicts. |
+| CI/CD interface | workflow run/job/check sync, rerun/cancel permissions, lazy logs. |
+| Git forge interface | repo/PR/ref/key/branch protection sync and drift. |
+| Bidirectional writes | write intent, approval, provider failure, confirmation, conflict. |
+| Rate limits | backoff, degraded status, resume. |
+| Cross-org | provider binding and native object references cannot cross orgs. |
+
+## Pluggable provider contract tests
+
+Each provider adapter should run a shared contract suite for every supported interface:
+
+| Contract suite | Providers |
+| --- | --- |
+| Issue tracking contract | GitHub, GitLab, Bitbucket when enabled, Azure DevOps, Jira, Linear, Gitea, custom. |
+| CI/CD contract | GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines, Buildkite, CircleCI, Jenkins, custom. |
+| Git forge contract | GitHub, GitLab, Bitbucket, Azure Repos, Gitea, Gerrit, raw Git partial, custom. |
+| Webhook contract | any provider with webhooks. |
+| Write-intent contract | any provider with mutating operations. |
+| Conflict contract | any bidirectional provider. |
+
+Contract tests should use fake provider adapters first, then provider-specific fixtures and optional live tests.
+
+## External UX flow tests
+
+Browser and E2E tests should cover:
+
+- connect GitHub provider;
+- connect Jira issue-only provider;
+- combine GitHub forge with Buildkite CI;
+- resolve a sync conflict;
+- approve an agent-proposed external write;
+- replay a dead-lettered webhook;
+- show provider rate-limit degraded state.
+
+These flows are specified in `docs/external/external-backend-ux-flows.md`.

package/docs/tests/qa-adoption-roadmap.md

@@ -0,0 +1,130 @@
+# QA adoption roadmap
+
+## Purpose
+
+This roadmap sequences QA automation work so Krate can improve coverage without blocking every product change on the final-state toolchain.
+
+## Stage 0: current baseline
+
+Status: available now.
+
+Required gates:
+
+- `npm run validate:docs`;
+- `npm test`;
+- `npm run e2e`;
+- `npm run package:check`;
+- `npm run smoke`;
+- `npm run ui:validate`;
+- `npm run ui:build`;
+- `npm run check` before release-like changes.
+
+## Stage 1: suite organization
+
+Add:
+
+- `tests/fixtures` with org/repository/resource fixtures;
+- test helper modules for fake Kubernetes and API controller setup;
+- metadata comments for owner/gate/area;
+- docs check that `docs/tests` exists and is linked.
+
+Exit criteria:
+
+- existing tests still pass;
+- new fixture policy is followed;
+- no behavior change required.
+
+## Stage 2: browser route smoke
+
+Add:
+
+- Playwright dependency and config;
+- route smoke for org dashboard, repositories, repo code/issues/runs/settings, deployments, and operations pages;
+- screenshot/trace capture on failure;
+- accessibility smoke on primary routes.
+
+Exit criteria:
+
+- browser gate runs in CI for UI changes;
+- route failures show useful artifacts;
+- no test relies on live external services.
+
+## Stage 3: coverage and API suites
+
+Add:
+
+- coverage command and reporting;
+- split API/controller tests;
+- stable error-code assertions;
+- org mismatch tests;
+- no-secret response assertions;
+- watch filter tests.
+
+Exit criteria:
+
+- coverage report generated in CI;
+- minimum thresholds set for critical modules;
+- cross-org denial is tested for resource APIs.
+
+## Stage 4: security and package hardening
+
+Add:
+
+- dependency/secret/license checks;
+- rendered chart schema validation;
+- action/workflow lint;
+- Docker build smoke;
+- SBOM/signature plan for release.
+
+Exit criteria:
+
+- release gate publishes security/package artifacts;
+- chart regressions fail before release.
+
+## Stage 5: agent/company-brain vertical slice
+
+Add:
+
+- org memory fixtures;
+- fake Agent Mux;
+- fake memory Git repo;
+- dispatch with memory snapshot tests;
+- summary-only `.a5c` import tests;
+- cross-org memory denial tests;
+- browser journey for memory preview/import review.
+
+Exit criteria:
+
+- `docs/agents/org-memory-vertical-slice-spec.md` acceptance paths are automated;
+- no raw `.a5c` secret-like content leaks;
+- retry uses pinned memory snapshot.
+
+## Stage 6: live/staging reliability
+
+Add:
+
+- live cluster smoke profiles;
+- Gitea, NATS, Argo CD, KubeVela, ARC, object storage checks;
+- controller restart/idempotency tests;
+- performance smoke for API/UI;
+- webhook burst and retry tests.
+
+Exit criteria:
+
+- staging gates prove install, core workflows, and rollback;
+- failure artifacts are actionable.
+
+## Stage 7: continuous quality intelligence
+
+Add:
+
+- flaky test dashboard;
+- coverage trend dashboard;
+- failure signature clustering;
+- ownership routing;
+- QA metrics in release notes;
+- automated gap reminders when new resources/routes lack tests.
+
+Exit criteria:
+
+- QA reports guide prioritization instead of only blocking merges.

package/docs/tests/qa-automation-plan.md

@@ -0,0 +1,101 @@
+# QA automation plan
+
+## Scope
+
+The QA plan covers:
+
+- core resource model and CRDs;
+- aggregated API and Postgres-backed resources;
+- Kubernetes controller and gateway behavior;
+- Gitea-backed repository data plane;
+- web UI and route flows;
+- CI/runners and pipeline/job lifecycle;
+- hooks and webhook delivery;
+- identity, RBAC, secrets, config, and policy;
+- deployments and KubeVela/OAM integration;
+- packaging, Helm chart, Docker image, install, smoke, and upgrade;
+- future agents, Agent Mux integration, company brain memory, `.a5c` run imports, triggers, tools, skills, subagents, and orchestration.
+
+## Test pyramid
+
+| Layer | Goal | Examples | Required speed |
+| --- | --- | --- | --- |
+| Static checks | catch broken contracts before execution | docs coverage, package validation, schema lint, import checks | seconds. |
+| Unit tests | verify pure functions and module contracts | resource schema, route helpers, auth helpers, redaction, ref resolution | seconds. |
+| Integration tests | verify module boundaries with fakes | API controller, Kubernetes gateway, UI model, memory import normalizer | seconds to minutes. |
+| Component/UI tests | verify rendered components and interactions | org switcher, repo tabs, dispatch composer, memory import panel | minutes. |
+| Browser E2E | verify critical user journeys | create repo, run CI, dispatch agent, import memory | minutes. |
+| Package/install tests | verify release artifact shape | Helm template, CRD coverage, minikube dry-run, Docker build | minutes. |
+| Live cluster tests | verify real integrations | Gitea, NATS, ARC, Argo CD, KubeVela, webhooks | longer/nightly. |
+| Chaos/reliability tests | verify failure behavior | watch reconnect, controller retry, Git outage, redaction failure | nightly/staging. |
+
+## Definition of done
+
+A feature is not done until:
+
+- resource/API/schema docs are updated;
+- unit or integration tests cover the core logic;
+- at least one acceptance or E2E path covers the user-visible behavior;
+- cross-org/RBAC/secret negative tests exist where relevant;
+- docs validation and package validation pass;
+- UI changes pass browser or component checks where relevant;
+- release-impacting changes update chart/package tests;
+- future agent/memory changes update the fixture plan and acceptance matrix.
+
+## Rollout phases
+
+### Phase 1: codify current gates
+
+- Keep `npm run check` as the local all-up gate.
+- Make `npm run validate:docs`, `npm test`, `npm run e2e`, `npm run package:check`, `npm run smoke`, `npm run ui:validate`, and `npm run ui:build` visible in CI docs.
+- Add test ownership labels by subsystem.
+
+### Phase 2: add browser automation
+
+- Add Playwright for route-level browser tests.
+- Cover org navigation, repository code/issues/runs/settings, deployments, and advanced plans.
+- Add accessibility checks for primary routes.
+
+### Phase 3: add API/controller contract tests
+
+- Add table-driven tests for org-scoped API routes, resource apply/list/delete, watch, and errors.
+- Add fake Kubernetes/Gitea/NATS/Agent Mux adapters.
+- Add no-secret response tests.
+
+### Phase 4: add agent/company-brain tests
+
+- Add fixtures for org memory, `.a5c` run imports, historical memory refs, and Agent Mux session binding.
+- Add cross-org denial and redaction tests.
+- Add browser E2E for dispatch with memory and import review.
+
+### Phase 5: staging and live integration
+
+- Add nightly cluster tests for Gitea, Argo CD, KubeVela, ARC, NATS, webhooks, and object storage.
+- Add upgrade/rollback tests.
+- Add reliability and failure injection scenarios.
+
+## Ownership model
+
+| Area | Owner role | Required evidence |
+| --- | --- | --- |
+| Resource/API contracts | platform/backend | schema tests, API tests, docs coverage. |
+| Controllers | platform/backend | reconciliation tests, idempotency tests, events/audit. |
+| UI/UX | frontend/product | browser/component tests, accessibility, route guards. |
+| CI/runners | platform/runtime | lifecycle E2E, isolation, ServiceAccount/RBAC tests. |
+| Security | security/platform | auth/RBAC/secret/no-leak tests. |
+| Packaging | release/platform | package/chart/install/smoke tests. |
+| Agents/memory | agents/platform | dispatch, context, memory, Agent Mux, import, trigger tests. |
+
+## Reporting
+
+Every CI run should publish:
+
+- command summary;
+- pass/fail by suite;
+- coverage by subsystem;
+- flaky test list;
+- failed test artifacts;
+- browser traces/screenshots for UI failures;
+- package/chart validation summary;
+- security/secret-scan findings;
+- links to run logs and relevant resources.

package/docs/tests/security-compliance-tests.md

@@ -0,0 +1,57 @@
+# Security and compliance tests
+
+## Authentication and authorization
+
+Required tests:
+
+- OIDC/delegated identity config parsing;
+- unauthenticated request rejection;
+- user/group mapping;
+- Kubernetes SubjectAccessReview invocation;
+- org namespace authorization;
+- route guard denies resource with wrong org label;
+- admin-only actions require admin/RBAC.
+
+## RBAC and policy
+
+Required tests:
+
+- runner ServiceAccount scoped to repo/ref/trust tier;
+- agent ServiceAccount cannot mount another org Secret/ConfigMap;
+- untrusted fork has no secrets and no cluster write access;
+- missing `AgentSecretGrant` or `AgentConfigGrant` blocks stack readiness;
+- policy/audit mode surfaces warnings without mutating resources;
+- cross-org refs require `OrgSharingPolicy`.
+
+## Secret and data leakage
+
+No secret-like values may appear in:
+
+- API responses;
+- UI rendered text;
+- context bundles;
+- prompt previews;
+- memory imports;
+- logs and watch events;
+- artifacts;
+- audit records;
+- browser traces.
+
+## Supply chain
+
+Release gates should eventually include:
+
+- dependency vulnerability scan;
+- license policy scan;
+- Docker image scan;
+- SBOM generation;
+- image/chart provenance or signatures;
+- GitHub Actions workflow lint.
+
+## Agent-specific security
+
+- Memory records are untrusted prompt content.
+- Tool calls are admitted by Krate, not Agent Mux alone.
+- Historical memory runs cannot read current memory without refresh/approval.
+- Agent write-back requires artifact digest and approval.
+- `.a5c` imports are redacted and validated before entering company brain.

package/docs/tests/test-framework-tools.md

@@ -0,0 +1,88 @@
+# Test framework and tools
+
+## Existing baseline
+
+Krate currently uses Node.js with ESM and the built-in `node:test` runner for unit and E2E tests. The package scripts are the source of truth for current gates.
+
+Existing tools:
+
+| Tool | Use |
+| --- | --- |
+| Node `node:test` | unit, integration, and current E2E tests. |
+| `assert/strict` | assertions. |
+| Next.js build | production UI build validation. |
+| custom scripts | docs, package, smoke, UI, minikube dry-run validation. |
+| Helm/minikube dry-run plans | install command validation without a live cluster. |
+
+## Recommended additions
+
+| Tool | Add when | Use |
+| --- | --- | --- |
+| Playwright | first browser suite | browser E2E, traces, screenshots, route assertions, accessibility hooks. |
+| Testing Library / React test utilities | component-level UI tests | component interaction tests without full browser cost. |
+| Istanbul/c8 or Node coverage | coverage reporting | line/branch/function coverage for `src` and critical scripts. |
+| `axe-core` or Playwright accessibility assertions | UI accessibility gate | WCAG smoke checks on primary pages. |
+| Helm unittest or template assertions | chart complexity grows | focused Helm render checks beyond current string tests. |
+| kubeconform/kubeval | CRD/chart validation | Kubernetes schema validation for rendered manifests. |
+| actionlint | workflow validation | GitHub Actions YAML checks. |
+| secret scanner | before memory imports and release | ensure fixtures/logs/artifacts do not leak secrets. |
+| dependency/license scanner | release gate | supply-chain checks. |
+| k6 or autocannon | performance stage | API/web smoke load tests. |
+
+## Tool selection principles
+
+- Prefer fast built-in Node tests for pure logic and contracts.
+- Use browser automation only for routes and interactions that cannot be validated below the browser layer.
+- Use deterministic fakes for Kubernetes, Gitea, Agent Mux, NATS, Argo CD, and object storage in PR gates.
+- Use live integration only in nightly/staging or explicit release gates.
+- Keep tests runnable on Windows and Linux.
+- Store fixtures in repo; avoid network calls in deterministic CI unless the suite is explicitly live.
+
+## Proposed npm scripts
+
+Future scripts should be additive and keep current scripts stable:
+
+```json
+{
+  "test:unit": "node --test tests/unit/**/*.test.js",
+  "test:integration": "node --test tests/integration/**/*.test.js",
+  "test:api": "node --test tests/api/**/*.test.js",
+  "test:e2e": "node --test tests/e2e/**/*.test.js",
+  "test:browser": "playwright test",
+  "test:coverage": "node --test --experimental-test-coverage tests/**/*.test.js",
+  "test:security": "node scripts/security-check.mjs",
+  "test:charts": "node scripts/validate-package.mjs",
+  "test:all": "npm run check && npm run test:browser"
+}
+```
+
+The exact script names can change during implementation, but the suite split should remain recognizable.
+
+## Test doubles
+
+Required fakes/mocks:
+
+| Adapter | Fake behavior |
+| --- | --- |
+| Kubernetes API | list/get/apply/delete/watch resources, SubjectAccessReview, events. |
+| Gitea/Git | repository create, refs, commits, clone URL, protected branches, webhook callbacks. |
+| Postgres | aggregated resources and migrations, preferably in-memory or isolated test DB. |
+| Object storage | artifact put/get by digest. |
+| NATS/webhook queue | enqueue, deliver, retry, replay. |
+| Agent Mux | create run/session, stream events, accept chat continuation, cancel/resume. |
+| Memory Git repo | resolve refs, read files, grep, write branch/PR, merge, diff. |
+| Argo CD/KubeVela | Application status, sync plan, rollout state. |
+
+## Artifacts
+
+Test failures should preserve:
+
+- assertion output;
+- API request/response body with secrets redacted;
+- generated YAML/resource plans;
+- browser trace and screenshot;
+- console/network logs for browser failures;
+- rendered Helm manifests;
+- memory import validation report;
+- `.a5c` fixture redaction report;
+- coverage report.