deepdebug-local-agent 1.0.23 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (178) hide show
  1. package/.claude/context.md +76 -0
  2. package/.claude/skills/README.md +25 -0
  3. package/.claude/skills/tdd-debugging/SKILL.md +53 -0
  4. package/.claude/skills/tdd-debugging/references/anti-patterns.md +43 -0
  5. package/.claude/skills/tdd-debugging/references/command-reference.md +46 -0
  6. package/.claude/standards/README.md +51 -0
  7. package/.claude/standards/clean-code.md +75 -0
  8. package/.claude/standards/code-review-checklist.md +61 -0
  9. package/.claude/standards/layered-architecture.md +163 -0
  10. package/.claude/standards/node.md +45 -0
  11. package/.claude/standards/security-owasp.md +114 -0
  12. package/.claude/standards/simplicity.md +42 -0
  13. package/.claude/standards/solid.md +64 -0
  14. package/.claude/standards/testing.md +59 -0
  15. package/.cursor/rules/standards.mdc +17 -0
  16. package/.github/workflows/build-byoc-image.yml +315 -0
  17. package/.github/workflows/deploy-gcp.yml +102 -0
  18. package/.github/workflows/npm-publish.yml +76 -8
  19. package/AGENTS.md +21 -0
  20. package/CHANGELOG.md +76 -0
  21. package/CLAUDE.md +13 -0
  22. package/CLAUDE_CODE_AGENT_INFINITE.md +5 -3
  23. package/Dockerfile +4 -0
  24. package/Dockerfile.full +55 -3
  25. package/bin/install.js +16 -5
  26. package/cloudbuild-full.yaml +57 -0
  27. package/docker-compose.yml +5 -3
  28. package/docs/OBSERVABILITY.md +164 -0
  29. package/docs/byoc-agent-image.md +70 -0
  30. package/docs/dlp.md +78 -0
  31. package/docs/graphify-packaging-plan.md +133 -0
  32. package/docs/graphify-validation-run-2026-10-28.md +243 -0
  33. package/docs/releasing.md +37 -0
  34. package/docs/signed-commands.md +163 -0
  35. package/env.example +34 -0
  36. package/helm/Chart.yaml +2 -2
  37. package/helm/templates/deployment.yaml +13 -3
  38. package/helm/templates/networkpolicy.yaml +22 -0
  39. package/helm/templates/secret.yaml +1 -0
  40. package/helm/values.yaml +13 -14
  41. package/mcp-server.js +4 -1
  42. package/package.json +9 -4
  43. package/scripts/next-version.mjs +75 -0
  44. package/scripts/upgrade-observability-push.sh +227 -0
  45. package/src/ai-engine.js +22 -22
  46. package/src/config/trusted-signing-keys.json +3 -0
  47. package/src/detectors/build-tool-detector.js +50 -0
  48. package/src/detectors/multi-project-detector.js +22 -22
  49. package/src/detectors/service-detector.js +24 -24
  50. package/src/dlp/engine.js +111 -0
  51. package/src/dlp/index.js +137 -0
  52. package/src/dlp/rules.js +97 -0
  53. package/src/dlp/validators.js +123 -0
  54. package/src/exec-utils.js +26 -4
  55. package/src/git/base-git-provider.js +117 -117
  56. package/src/git/bitbucket-provider.js +2 -2
  57. package/src/git/git-provider-registry.js +18 -18
  58. package/src/git/github-provider.js +13 -13
  59. package/src/git/multi-repo-pr.js +161 -0
  60. package/src/mcp-http-server.js +32 -16
  61. package/src/observability/bundle-files.js +418 -0
  62. package/src/observability/detect.js +151 -0
  63. package/src/observability/provision.js +212 -0
  64. package/src/observability/render-cli.js +59 -0
  65. package/src/observability/routes.js +88 -0
  66. package/src/patch/patch-engine.js +28 -28
  67. package/src/patch-manager.js +85 -85
  68. package/src/patch_bkp.js +14 -14
  69. package/src/ports.js +4 -4
  70. package/src/routes/workspace.route.js +47 -16
  71. package/src/runtimes/base-runtime.js +75 -75
  72. package/src/runtimes/java/gradle-runtime.js +17 -17
  73. package/src/runtimes/java/java-integrations.js +25 -25
  74. package/src/runtimes/java/maven-runtime.js +34 -34
  75. package/src/runtimes/node/node-integrations.js +1 -1
  76. package/src/runtimes/node/npm-runtime.js +14 -14
  77. package/src/runtimes/node/yarn-runtime.js +4 -4
  78. package/src/runtimes/runtime-registry.js +25 -25
  79. package/src/server-local.js +4 -4
  80. package/src/server.js +1366 -333
  81. package/src/siblings/burst-suppressor.js +89 -0
  82. package/src/siblings/capabilities.js +273 -0
  83. package/src/siblings/env-compat.cjs +58 -0
  84. package/src/siblings/incident-forwarder.js +322 -0
  85. package/src/siblings/logger.js +15 -0
  86. package/src/siblings/metrics.js +94 -0
  87. package/src/utils/agent-auth.js +184 -0
  88. package/src/utils/approval-token.js +56 -0
  89. package/src/utils/bind-host.js +29 -0
  90. package/src/utils/branch-refspec.js +98 -0
  91. package/src/utils/child-env.js +34 -0
  92. package/src/utils/environment-diagnostics.js +55 -55
  93. package/src/utils/exec-policy.js +962 -0
  94. package/src/utils/exec-utils.js +15 -15
  95. package/src/utils/file-count.js +46 -0
  96. package/src/utils/fs-utils.js +10 -10
  97. package/src/utils/multi-repo-open.js +132 -0
  98. package/src/utils/open-gate.js +183 -0
  99. package/src/utils/ranged-read.js +65 -0
  100. package/src/utils/request-guards.js +77 -0
  101. package/src/utils/request-signing.js +342 -0
  102. package/src/utils/root-recovery.js +244 -0
  103. package/src/utils/safe-exec-args.js +58 -0
  104. package/src/utils/sandbox.js +182 -0
  105. package/src/utils/search-outcome.js +56 -0
  106. package/src/utils/workspace-root.js +80 -0
  107. package/src/validators/syntax-validator.js +149 -0
  108. package/src/workspace/detect-port.js +16 -16
  109. package/src/workspace/file-reader.js +1 -1
  110. package/src/workspace/process-manager.js +46 -46
  111. package/src/workspace-manager.js +86 -24
  112. package/test/adapters/registry.test.js +9 -2
  113. package/test/agent-auth.test.js +194 -0
  114. package/test/bind-host.test.js +26 -0
  115. package/test/branch-refspec.test.js +224 -0
  116. package/test/burst-suppressor.test.js +60 -0
  117. package/test/capabilities-access.test.js +47 -0
  118. package/test/capabilities.test.js +87 -0
  119. package/test/child-env.test.js +23 -0
  120. package/test/dlp/engine.test.js +173 -0
  121. package/test/dlp/fixtures.js +126 -0
  122. package/test/dlp/index.test.js +166 -0
  123. package/test/env-compat.test.js +71 -0
  124. package/test/exec-policy.test.js +463 -0
  125. package/test/file-count.test.js +64 -0
  126. package/test/incident-forwarder.test.js +231 -0
  127. package/test/multi-repo-open.test.js +180 -0
  128. package/test/multi-repo-pr.test.js +129 -0
  129. package/test/next-version.test.js +48 -0
  130. package/test/observability-detect.test.js +199 -0
  131. package/test/open-gate.test.js +341 -0
  132. package/test/ranged-read.test.js +60 -0
  133. package/test/request-guards.test.js +91 -0
  134. package/test/request-signing.test.js +284 -0
  135. package/test/root-recovery-sidecar.test.js +171 -0
  136. package/test/root-recovery.test.js +274 -0
  137. package/test/safe-exec-args.test.js +58 -0
  138. package/test/sandbox.test.js +98 -0
  139. package/test/search-outcome.test.js +50 -0
  140. package/test/workspace-root.test.js +64 -0
  141. package/tools/graphify-validation/README.md +92 -0
  142. package/tools/graphify-validation/config/questions.json +31 -0
  143. package/tools/graphify-validation/config/repos.json +9 -0
  144. package/tools/graphify-validation/config/symbols.json +6 -0
  145. package/tools/graphify-validation/fixtures/spring/expected.json +34 -0
  146. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/AppConfig.java +29 -0
  147. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/AuditLogger.java +15 -0
  148. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/Cache.java +7 -0
  149. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/CheckoutService.java +25 -0
  150. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/InMemoryCache.java +18 -0
  151. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/OrderController.java +22 -0
  152. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/PaymentGateway.java +5 -0
  153. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/PaypalGateway.java +11 -0
  154. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/RefundService.java +14 -0
  155. package/tools/graphify-validation/fixtures/spring/src/main/java/com/example/fx/StripeGateway.java +11 -0
  156. package/tools/graphify-validation/lib/affected_vs_rg.py +81 -0
  157. package/tools/graphify-validation/lib/audit_hook/sitecustomize.py +22 -0
  158. package/tools/graphify-validation/lib/graph_metrics.py +96 -0
  159. package/tools/graphify-validation/lib/gv_common.py +130 -0
  160. package/tools/graphify-validation/lib/licenses.py +47 -0
  161. package/tools/graphify-validation/lib/pip_audit_summary.py +18 -0
  162. package/tools/graphify-validation/lib/recall.py +115 -0
  163. package/tools/graphify-validation/lib/report.py +226 -0
  164. package/tools/graphify-validation/lib/run_all.py +237 -0
  165. package/tools/graphify-validation/lib/run_graphify.py +144 -0
  166. package/tools/graphify-validation/lib/rusage_helper.py +17 -0
  167. package/tools/graphify-validation/lib/sample_edges.py +246 -0
  168. package/tools/graphify-validation/lib/spring_fixture.py +81 -0
  169. package/tools/graphify-validation/requirements-py311.lock +508 -0
  170. package/tools/graphify-validation/requirements-tools.in +1 -0
  171. package/tools/graphify-validation/requirements-tools.lock +371 -0
  172. package/tools/graphify-validation/requirements.in +1 -0
  173. package/tools/graphify-validation/requirements.lock +502 -0
  174. package/tools/graphify-validation/run.sh +43 -0
  175. package/tools/graphify-validation/tests/test_harness.py +151 -0
  176. package/cloudbuild-agent-qa.yaml +0 -43
  177. package/cloudbuild-agent-staging.yaml +0 -40
  178. package/cloudbuild.yaml.deprecated +0 -42
@@ -0,0 +1,164 @@
1
+ # Client Observability — method & runbook
2
+
3
+ How the DeepDebug local agent brings **observability to a client that has none**.
4
+ It is the client-side delivery of the same stack we run for ourselves in
5
+ `vh-observability` (Grafana + Loki + Alloy + Alertmanager) — same building
6
+ blocks, different plumbing.
7
+
8
+ | | `vh-observability` (ours) | This bundle (client) |
9
+ | --- | --- | --- |
10
+ | Runs on | Cloud Run (multi-container) | Client VPS/host (docker-compose) |
11
+ | Log source | Cloud Logging → Pub/Sub (`gcplog`) | **`docker.sock`** (every container) |
12
+ | Loki storage | GCS bucket | **local filesystem volume** |
13
+ | Grafana login | Google Workspace OAuth | local admin (loopback-only) |
14
+ | Alerts → | gateway `/webhooks/alertmanager` (Bearer secret) | **same gateway webhook** |
15
+
16
+ Everything downstream of Loki is identical: the ruler evaluates a LogQL error
17
+ rule, Alertmanager posts firing alerts to the gateway (tenant resolved from the
18
+ trigger secret, sent as a Bearer header), and Grafana's logs dashboard has an **"Open in DeepDebug
19
+ Workbench"** data link into `/api/v1/webhooks/from-log`.
20
+
21
+ ---
22
+
23
+ ## Architecture
24
+
25
+ ```
26
+ All containers on the host
27
+ │ (docker.sock, read-only)
28
+ ▼
29
+ Alloy ──push──▶ Loki (filesystem) ◀──query── Grafana (127.0.0.1:3000)
30
+ (discovery.docker, (chunks + tsdb in (provisioned datasource +
31
+ service = name, a local volume) dashboard, loopback only)
32
+ level parsed) │ ruler
33
+ ▼
34
+ Alertmanager ──webhook──▶ gateway /api/v1/webhooks/alertmanager (Bearer <secret>)
35
+ │
36
+ incident (tenant from the secret) + Slack notify
37
+ ```
38
+
39
+ Four containers, one compose project (`deepdebug-observability`). Only Grafana
40
+ publishes a port, and only on `127.0.0.1` — reach it over an SSH tunnel, never
41
+ expose it to the internet.
42
+
43
+ ## The error → workbench loop
44
+
45
+ - **Automatic** — Loki rule `ClientServiceErrors` fires on error rate → Alertmanager
46
+ → `POST /api/v1/webhooks/alertmanager` (`Authorization: Bearer <secret>`) → incident → Slack.
47
+ - **Manual** — the Logs panel data link → `GET /api/v1/webhooks/from-log?s=<secret>&…`
48
+ → incident → 302 redirect to `<frontend>/incidents/<id>` (the PG-persisted id).
49
+
50
+ The tenant is always resolved from the trigger secret, never from the
51
+ caller — the secret IS the auth.
52
+
53
+ ## Logs into the platform (Live logs + services)
54
+
55
+ Loki stays on the host for Grafana, but the platform cannot reach it (no inbound
56
+ port). Alloy therefore **also pushes** the same Docker stream to the gateway:
57
+
58
+ ```
59
+ loki.process ──▶ otelcol.receiver.loki ──▶ batch ──▶ otelcol.exporter.otlphttp
60
+ POST <gateway>/api/v1/webhooks/logs/otlp (JSON, gzip, Authorization: Bearer <secret>)
61
+ ──▶ Redis list vh.logs.push.<tenant> ──▶ monitor-service (provider SELF_HOSTED)
62
+ ──▶ log_entries + live tail ──▶ Live logs (one service per container)
63
+ ```
64
+
65
+ - Outbound HTTPS only; same `vht_` secret as the Alertmanager.
66
+ - The first push creates the tenant's `SELF_HOSTED` row in `tenant_log_configs`.
67
+ - Pushed logs are **not** promoted to incidents by the monitor (the Loki rule above
68
+ already does that) unless `MONITOR_PUSH_DETECT_ENABLED=true`.
69
+ - The gateway accepts OTLP **JSON** only (protobuf gets 415).
70
+
71
+ **Existing installs** (provisioned before this change): run
72
+ `scripts/upgrade-observability-push.sh` on the server (`--dry-run` first). It finds the
73
+ bundle, reads the gateway URL and secret from `alertmanager.yml`, probes the push endpoint,
74
+ adds the `otelcol.*` blocks, moves the secret to a Bearer header, validates both configs
75
+ with the stack's own images, restarts only `alloy` + `alertmanager` and rolls back on
76
+ failure. Idempotent.
77
+
78
+ ---
79
+
80
+ ## The zero-observability method
81
+
82
+ The agent only offers to provision when the host has **nothing** already shipping
83
+ logs/metrics. The decision is deliberately conservative: a false "they have it"
84
+ just means we skip; a false "they have nothing" would deploy on top of a real
85
+ stack. So we skip on ANY match.
86
+
87
+ **Signals** (matched case-insensitively against container image/name AND process
88
+ command lines): promtail, fluent-bit, fluentd, vector, filebeat, logstash,
89
+ otel-collector, telegraf, loki, elasticsearch, opensearch, victoriametrics,
90
+ prometheus, thanos, mimir, signoz, grafana, kibana, datadog, newrelic, splunk,
91
+ dynatrace, instana, sumologic, … (see `OBSERVABILITY_SIGNALS` in
92
+ `src/observability/detect.js`).
93
+
94
+ **Our own stack never counts** as the client already having observability: any
95
+ container whose compose project is `deepdebug-observability` is excluded, so
96
+ re-running detection after we provision still reports the client's baseline and
97
+ `up -d` stays idempotent.
98
+
99
+ ### Step-by-step the agent performs (`provisionObservability`)
100
+
101
+ 1. **Probe** the host — `docker ps` + process list → classify (above).
102
+ 2. **Guard** — if the client already has observability and `force` is not set,
103
+ **stop** and report what was found. Do not touch their stack.
104
+ 3. **Pre-flight** — verify `docker` and the compose plugin are present
105
+ (else return an actionable error).
106
+ 4. **Render** the bundle for this tenant: inject `gatewayUrl`, the alertmanager
107
+ `triggerSecret`, and a static `environment` label into every template.
108
+ 5. **Write** files to `~/.deepdebug/observability/deepdebug-observability/`
109
+ plus a `.env` (Grafana admin creds, `DDO_ENVIRONMENT`).
110
+ 6. **Start** — `docker compose up -d --remove-orphans`.
111
+ 7. **Report** — container status + the loopback Grafana URL and admin password.
112
+
113
+ ---
114
+
115
+ ## Agent API
116
+
117
+ Called by the gateway (Setup/Onboarding agent) or by an operator directly.
118
+
119
+ ### `GET /observability/status`
120
+ Probe + classify. No side effects.
121
+ ```json
122
+ { "ok": true, "hasObservability": false, "recommendation": "provision",
123
+ "found": [], "ownStackRunning": false,
124
+ "docker": { "docker": true, "compose": true }, "containersScanned": 6 }
125
+ ```
126
+
127
+ ### `POST /observability/provision`
128
+ ```jsonc
129
+ {
130
+ "gatewayUrl": "https://api.visionhammer.io", // no trailing slash needed
131
+ "triggerSecret": "vht_….", // tenant alertmanager secret
132
+ "environment": "production", // static label for this client
133
+ "grafanaUser": "admin", // optional
134
+ "grafanaPassword": "…", // optional (else generated + returned)
135
+ "force": false, // deploy even if a stack exists
136
+ "start": true // run `docker compose up -d`
137
+ }
138
+ ```
139
+ Returns `409 { skipped: true, found: [...] }` when the client already has a stack
140
+ and `force` is not set. On success returns `installDir`, container `status`, and
141
+ `grafana: { url, user, password }`.
142
+
143
+ ### `POST /observability/teardown`
144
+ `{ "purge": false }` → `docker compose down` (`purge:true` also removes volumes).
145
+
146
+ ---
147
+
148
+ ## Access & security
149
+
150
+ - **Grafana** is bound to `127.0.0.1:3000`. Reach it with
151
+ `ssh -L 3000:127.0.0.1:3000 <host>` then open `http://localhost:3000`.
152
+ - The **trigger secret** is the only credential in the alertmanager webhook URL
153
+ and it authorises exactly one tenant. Rotate via the gateway trigger source.
154
+ - Alloy mounts `docker.sock` **read-only** and only tails logs.
155
+ - The generated Grafana admin password is reported once — rotate it, or pass
156
+ your own `grafanaPassword`.
157
+
158
+ ## Where the bundle lives
159
+
160
+ The bundle is **embedded** in the agent (`src/observability/bundle-files.js`) so
161
+ it ships inside the `pkg` binary — there are no loose asset files to lose. Editing
162
+ that file changes what the agent deploys. `renderBundle()` is a pure function
163
+ (templated in → final files out) and is covered by
164
+ `test/observability-detect.test.js`.
@@ -0,0 +1,70 @@
1
+ # BYOC agent image (`local-agent-full`)
2
+
3
+ Customer-cloud agents (BYOC GCP Cloud Run and BYOC Azure Container Instances)
4
+ run the public image built from `Dockerfile.full`:
5
+
6
+ ```
7
+ europe-west1-docker.pkg.dev/visionhammer/visionhammer-docker-public-eu/local-agent-full:<semver>
8
+ europe-west1-docker.pkg.dev/visionhammer/visionhammer-docker-public-eu/local-agent-full:<git-sha>
9
+ ```
10
+
11
+ The managed per-tenant Cloud Run agent (gateway `AgentProvisioningService`) is
12
+ built separately per tenant and does not use this image.
13
+
14
+ ## Publishing a version
15
+
16
+ 1. Actions → **Build BYOC Agent Image** → *Run workflow* on `main` (version
17
+ defaults to `package.json`; or push a `v*` tag / publish a release on a
18
+ commit that is on `main`).
19
+ 2. The workflow builds `Dockerfile.full`, smoke-tests it (`/health` public,
20
+ API returns 401 without `X-Agent-Token` when `LOCAL_AGENT_TOKEN` is set),
21
+ then pushes `:<semver>` and `:<git-sha>`. It refuses to overwrite an
22
+ existing `:<semver>` built from a different commit, and never moves an
23
+ existing `:<git-sha>` (a commit rebuilt under a new semver keeps its first
24
+ image under `:<git-sha>`; always pin `:<semver>@sha256:`). There is no floating tag.
25
+ 3. Copy the pinned reference from the run summary
26
+ (`...local-agent-full:<semver>@sha256:<digest>`).
27
+
28
+ 4. Optional supply chain (flag, off by default): tick `supply_chain` on the
29
+ dispatch (or set the repository variable `BYOC_SUPPLY_CHAIN=true` for tag and
30
+ release runs). The `supply-chain` job signs `:<semver>@sha256:<digest>` with
31
+ cosign keyless (GitHub OIDC), attaches an SPDX SBOM (syft) and a SLSA v1
32
+ provenance attestation, and verifies all three. Consumers verify with:
33
+
34
+ ```
35
+ cosign verify \
36
+ --certificate-identity-regexp '^https://github.com/VISIONHAMMER/deepdebug-local-agent/' \
37
+ --certificate-oidc-issuer https://token.actions.githubusercontent.com \
38
+ <image>@sha256:<digest>
39
+ ```
40
+
41
+ Use `cosign verify-attestation --type spdxjson` (or `slsaprovenance1`) for
42
+ the SBOM and the provenance. The provenance is generated inside this
43
+ workflow and signed with the workflow identity; it is not a
44
+ slsa-github-generator (isolated builder) attestation.
45
+
46
+ Fallback without GitHub Actions: `cloudbuild-full.yaml` (see header).
47
+
48
+ ## Rolling it out (gateway)
49
+
50
+ Set `deepdebug.byoc.agent-image` in vh-api-gateway (env `BYOC_AGENT_IMAGE`) to
51
+ the pinned reference, or bump its default in `application.yml` via PR. Only
52
+ newly provisioned agents pick it up; existing ones keep their image until
53
+ re-provisioned from the platform (Cloud Engineer → provision again). Azure
54
+ container groups created before the TLS/token spec are recreated on re-provision.
55
+
56
+ ## Runtime knobs relevant to BYOC
57
+
58
+ | Env | Meaning |
59
+ | --- | --- |
60
+ | `LOCAL_AGENT_TOKEN` | Enforce `X-Agent-Token` on every non-public route (gateway sets it = the per-workspace shared secret on ACI). |
61
+ | `AGENT_SHARED_SECRET` | Same secret; used for the agent's own in-process calls. |
62
+ | `LOCAL_AGENT_BIND_HOST` | Listen address. Default: `0.0.0.0` when a token (`LOCAL_AGENT_TOKEN`/`AGENT_SHARED_SECRET`) or Cloud Run is present, otherwise `127.0.0.1`. ACI sets `127.0.0.1`: only the Caddy TLS sidecar in the same container group can reach it. |
63
+
64
+ ## One-time operator setup
65
+
66
+ - `github-deployer@visionhammer.iam.gserviceaccount.com` needs
67
+ `roles/artifactregistry.writer` on the `visionhammer-docker-public-eu` repository (europe-west1; today granted at project level)
68
+ (and `artifactregistry.reader` for the overwrite guard).
69
+ - The repository must stay public-read (`allUsers` → `artifactregistry.reader`) so
70
+ customer projects / ACI pull without credentials.
package/docs/dlp.md ADDED
@@ -0,0 +1,78 @@
1
+ # DLP: masking secrets and PII before content leaves the agent (T4-09)
2
+
3
+ Status: module + first integration point. Flag `LOCAL_AGENT_DLP_MODE`, default **off**.
4
+
5
+ ## Flags
6
+
7
+ | Variable | Values | Default | Effect |
8
+ |---|---|---|---|
9
+ | `LOCAL_AGENT_DLP_MODE` | `off`, `warn`, `enforce` | `off` | `off`: nothing scanned, output identical to before. `warn`: output unchanged, counters and local log report what enforce would redact. `enforce`: hits replaced by `[REDACTED:<rule_id>]`. Any other value means `off`. |
10
+ | `LOCAL_AGENT_DLP_BLOCK_CLASSES` | comma list of classes | empty | Enforce only. If a hit belongs to a listed class the whole text is replaced by `[BLOCKED:<class>]`. |
11
+ | `LOCAL_AGENT_DLP_MAX_BYTES` | integer | `2000000` | Larger inputs are not scanned; enforce fails closed (`[REDACTED:dlp_oversize]`). |
12
+
13
+ Classes: `private-key`, `cloud-credential`, `vcs-token`, `payment-key`, `saas-token`, `token`,
14
+ `credential`, `connection-string`, `url-credential`, `pii-email`, `pii-phone`, `pii-id`,
15
+ `pii-financial`, `high-entropy`.
16
+
17
+ ## API (reusable by the WebSocket work, task M)
18
+
19
+ ```js
20
+ import { maskSensitive, maskDeep, protectEgress, protectEgressAsync, protectDeep,
21
+ getDlpStats, rulesetHash, RULESET_VERSION } from './src/dlp/index.js';
22
+ ```
23
+
24
+ - `maskSensitive(text, { mode, blockClasses, maxBytes })`: **pure** function (no env, no state).
25
+ Returns `{ text, redactions: {rule_id: n}, total, blocked, blockedClasses, oversize, rulesetVersion }`.
26
+ - `maskDeep(value, opts)`: same over every string of a JSON-like value (keys untouched, input not mutated).
27
+ - `protectEgress(text, { source, log, env })`: reads the flag from the environment, updates counters,
28
+ writes a local record with counts per rule (never the value). Returns `{ text, ... }`.
29
+ - `protectEgressAsync(text, { policyDecision, timeoutMs, ... })`: as above, plus a per-action
30
+ PolicyGate decision (D19). In `enforce`, anything other than an explicit `'ALLOW'` (DENY, unknown
31
+ answer, throw, timeout, PolicyGate offline) returns `[DLP_DENIED]`.
32
+ - `protectDeep(value, opts)`: env-driven `maskDeep`.
33
+ - `getDlpStats()`: `{ mode, ruleset_version, ruleset_hash, scanned, blocked, denied, dlp_redactions_total: {rule_id: n} }`.
34
+ Counts only; intended for the heartbeat (`dlp_redactions_total` by rule). `rulesetHash()` is what
35
+ `config.update` can carry to prove which ruleset is active.
36
+
37
+ Fail-closed rules in `enforce`: internal error, oversize input, policy not ALLOW = content withheld.
38
+ In `warn`/`off` the original text is always returned.
39
+
40
+ ## Where it is applied today
41
+
42
+ - `src/siblings/incident-forwarder.js`: `message` of the incident POSTed to the gateway (log excerpt).
43
+
44
+ Not applied yet (follow-ups, outside this PR's boundary): `content.request` responses and `job.event`
45
+ (WebSocket channel, task M), prompts of the loop, error messages returned by routes. They can call
46
+ `protectEgress`/`protectDeep`/`maskSensitive` directly.
47
+
48
+ ## Ruleset (version `2026.10.28-1`, 40 rules)
49
+
50
+ Order matters (specific before generic). Highlights: PEM/OpenSSH/PGP private keys; AWS key id and secret;
51
+ GCP API key, OAuth token and client secret; Azure storage key; DigitalOcean; GitHub (classic, fine-grained);
52
+ GitLab; npm; PyPI; Stripe (keys, webhook secret); Shopify; Slack (token, webhook); Discord webhook;
53
+ SendGrid; Twilio; Mailgun; Hugging Face; Telegram; Anthropic; OpenAI; JWT; Bearer and Basic headers;
54
+ DB connection strings (postgres, mysql, mongodb, redis, amqp, mssql, cockroach); JDBC and ADO passwords;
55
+ credentials in URLs; generic `name = literal` assignments (with a validator that ignores identifiers,
56
+ calls, templates and placeholders); e-mail (ignores asset names and documentation domains);
57
+ IBAN (mod-97); card numbers (Luhn); PT NIF (checksum and keyword context); international phone
58
+ (leading `+`/`00`, 9 to 15 digits); opaque high-entropy blobs (40+ chars, ignores hex digests, UUIDs,
59
+ SRI hashes and paths). Patterns are reimplemented from public token formats; no third-party rule
60
+ files were copied. The PiiEgressMasker of the agents repo was used as a behavioural reference only.
61
+
62
+ Bump `RULESET_VERSION` in `src/dlp/rules.js` on any change.
63
+
64
+ ## Tests
65
+
66
+ `test/dlp/`: every rule has at least one synthetic positive (fixtures are built at runtime from split
67
+ prefixes and a deterministic hash stream, no real secrets); 40 negative lines (code, hashes, UUIDs,
68
+ SSH remotes, templates) must be unchanged; mode, block, oversize, idempotence, performance on
69
+ pathological 1 MB inputs, policy fail-closed and the forwarder integration (flag off = byte-identical).
70
+
71
+ Indicative false-positive check (not a measured product metric): running `enforce` over this repo
72
+ (182 text files, ~41k lines) altered about 40 lines, almost all true positives (e-mail addresses in
73
+ docs and test fixtures, literal tokens in tests). The plan's target (<= 2 % FP on a real code
74
+ corpus) still needs to be measured on a customer-like corpus during the pilot.
75
+
76
+ ## Rollback
77
+
78
+ Unset `LOCAL_AGENT_DLP_MODE` (or set `off`) and restart the agent. No data, schema or protocol changes.
@@ -0,0 +1,133 @@
1
+ # Graphify no `local-agent-full`: plano de empacotamento e checklist de licenças
2
+
3
+ Estado: **proposta** (tarefa T4-06 do plano GA do Local Agent, decisões D8, D11, D12, D14 e D18 do ADR-0017). Nada neste
4
+ documento está aplicado: nenhum Dockerfile, workflow ou `package.json` foi alterado. A implementação é a tarefa T4-07
5
+ (PR separado). Marcação: **[V]** verificado nesta sessão (comando e saída em `docs/graphify-validation-run-2026-10-28.md`),
6
+ **[N]** não verificado.
7
+
8
+ ## 1. Decisões que este plano executa
9
+
10
+ | Decisão | Consequência técnica |
11
+ |---|---|
12
+ | D8: Graphify via PyPI, versão fixada, com hash | dependência `graphifyy==X.Y.Z`, sem fork; instalação só com `--require-hashes` |
13
+ | D11: GA 1.1.0 | a imagem `local-agent-full:1.1.0` é a primeira com o runtime; o `Dockerfile.full` e o Helm deixam de apontar para `1.0.0` e `:latest` (tarefa T4-07) |
14
+ | D12: cosign e SBOM obrigatórios | a imagem e o SBOM saem assinados na mesma release (secção 6) |
15
+ | D14: fallback ripgrep | a imagem inclui `ripgrep`; se o índice do grafo faltar ou estiver velho, o contexto vem de `rg` |
16
+ | D18: runtime pack multi-runtime | o runtime Graphify é um pack como os outros (Java, Swift): instalável e removível sem tocar no agente Node |
17
+
18
+ ## 2. O que vai fixado, e como
19
+
20
+ ### 2.1 Ficheiros de lock (já no harness)
21
+
22
+ `tools/graphify-validation/requirements.in` tem **uma** linha, `graphifyy[sql,terraform]==0.9.93`. Dois locks com hashes SHA-256
23
+ de todos os artefactos de cada versão:
24
+
25
+ - `requirements.lock`, Python 3.12 (venv do harness) [V];
26
+ - `requirements-py311.lock`, Python 3.11, a versão do Debian bookworm que o `Dockerfile.full` já instala (`python3`, `libpython3.11`) [V no ficheiro].
27
+
28
+ **Os locks não são intermutáveis [V]:** em 3.11 o resolvedor escolhe `networkx 3.6.1` e `numpy 2.4.6`; em 3.12, `networkx 3.7` e
29
+ `numpy 2.5.4`. A imagem usa o lock do Python que tiver. Para a mesma versão de Python, o conjunto de pacotes resolvido é o mesmo em
30
+ Linux x86_64, Linux aarch64 e macOS arm64 (`uv pip compile --python-platform ...`, comparado) [V]. O lock 3.11 foi instalado com
31
+ `--require-hashes --no-deps --only-binary=:all:` num venv Python 3.11 em macOS arm64 e deu o mesmo grafo (52 nós, 84 arestas) na fixture Spring [V];
32
+ a instalação em Linux (glibc, musl) **não foi corrida** [N] (sem Docker local).
33
+
34
+ Extras: só `sql` (`tree-sitter-sql`) e `terraform` (`tree-sitter-hcl`). **Não** entram `mcp`, `leiden` (puxa `graspologic`), nem extras de LLM
35
+ (`anthropic`, `openai`, `gemini`, `bedrock`, `ollama`...). Sem `leiden`, a deteção de comunidades usa o fallback do próprio Graphify;
36
+ as comunidades não são identificadores estáveis de qualquer modo (nota do piloto). O pacote instala o script `graphify-mcp`
37
+ mesmo sem o extra `mcp` [V, `entry_points.txt`]: **apagar `bin/graphify-mcp`** na imagem.
38
+
39
+ ### 2.2 Instalação
40
+
41
+ ```dockerfile
42
+ # proposta para Dockerfile.full (T4-07); não aplicada
43
+ ARG GRAPHIFY_LOCK=requirements-py311.lock
44
+ COPY tools/graphify-validation/${GRAPHIFY_LOCK} /tmp/graphify.lock
45
+ RUN python3 -m venv /opt/vh-graphify \
46
+ && /opt/vh-graphify/bin/pip install --no-cache-dir --disable-pip-version-check \
47
+ --require-hashes --no-deps --only-binary=:all: -r /tmp/graphify.lock \
48
+ && /opt/vh-graphify/bin/python -m compileall -q /opt/vh-graphify/lib \
49
+ && rm -f /opt/vh-graphify/bin/graphify-mcp /tmp/graphify.lock
50
+ ```
51
+
52
+ Porquê cada flag: `--require-hashes` recusa qualquer ficheiro cujo hash não esteja no lock; `--no-deps` impede que o pip resolva
53
+ dependências por fora do lock; `--only-binary=:all:` proíbe sdists, logo **nenhum `setup.py` de terceiros corre no build** (a nota do piloto já
54
+ tinha visto que o `pyproject` do Graphify é setuptools puro [C]); `compileall` deixa o bytecode feito para o sistema de ficheiros só de leitura
55
+ (o Python não escreve `.pyc` com `PYTHONDONTWRITEBYTECODE=1`).
56
+
57
+ Tamanho medido do `site-packages`: 154 MB (Python 3.12, macOS arm64, com `pip`) e 124 MB (3.11, sem `pip`) [V]. Em Linux pode diferir [N]; o
58
+ limite proposto de delta de imagem é 400 MB (G9).
59
+
60
+ ### 2.3 Política de subida
61
+
62
+ 1. Só versões com mais de 7 dias no PyPI (cooldown contra pacote comprometido). A fixada, 0.9.93, tinha 8 dias a 2026-10-28 [V, data de upload no PyPI]; o
63
+ upstream já ia na 0.9.97 (publicada a 2026-10-27, **não escolhida de propósito**).
64
+ 2. PR de subida: editar `requirements.in`, regenerar os dois locks (comandos no README do harness), **rever o diff do código Python do Graphify entre versões**
65
+ (o PyPI mostra 13 versões em 17 dias, de 0.9.84 a 0.9.97 [V]), repetir o harness (G1, G2, G7, G8) e anexar o `REPORT.md`.
66
+ 3. Nunca `pip install graphifyy` sem lock, nem `--upgrade`, nem `latest`, em Dockerfile, instalador, Helm ou CI.
67
+ 4. Mesma regra para o `pip-audit` (`requirements-tools.in/.lock`).
68
+
69
+ ## 3. `pip-audit` e cadeia de fornecimento
70
+
71
+ - Local: `run.sh --pip-audit` corre `pip-audit --require-hashes --disable-pip --no-deps -r requirements.lock`. Resultado de 2026-10-28: **0 vulnerabilidades conhecidas em
72
+ 32 pacotes** (base PyPI/OSV nessa data) [V]. O mesmo audit tem de correr para `requirements-py311.lock` [N: só o lock 3.12 foi auditado].
73
+ - CI (T4-07): job no PR que toca nos locks e job semanal agendado; falha com qualquer vulnerabilidade com versão corrigida disponível. Exceções só com ID, motivo e data de expiração
74
+ num ficheiro `pip-audit-ignore.txt` revisto no PR. Nota: o audit só vê vulnerabilidades **publicadas**; não substitui a revisão de diff da secção 2.3.
75
+ - Proveniência: confirmar se o `graphifyy` publica atestações PyPI (Trusted Publishing) [N]. Se publicar, verificar no job de subida; se não, registar o risco no ADR.
76
+ - A imagem corre **sem rede para o Graphify**: o `entrypoint` do pack não define proxies nem chaves, e o contentor de teste usa `--network none` (G9) [N].
77
+
78
+ ## 4. Superfície permitida em runtime
79
+
80
+ O Local Agent chama o Graphify por **um wrapper** (`vh-graphify`, a escrever em T4-07), nunca diretamente:
81
+
82
+ | Permitido | Proibido (o wrapper recusa com exit 64) |
83
+ |---|---|
84
+ | `update <path>`, `query`, `affected`, `explain`, `path`, `god-nodes` | `label`, `cluster-only` sem `--no-label`, `add`, `clone`, `prs`, `watch`, `install`, `global`, `merge-graphs`, `graphify-mcp` |
85
+
86
+ - Ambiente do processo filho: `HOME` num diretório do volume de dados, `PATH`, `LANG`, `PYTHONHASHSEED=0`, `PYTHONDONTWRITEBYTECODE=1`. **Sem** chaves LLM, tokens nem proxies
87
+ (a CLI auto-deteta chaves e envia nomes de comunidade ao LLM, nota do piloto). O harness corre assim e o relatório regista as chaves passadas [V].
88
+ - Rede: o harness provou 0 eventos `socket.*` com a rede bloqueada nos três repos VH, a rede estava de facto bloqueada (sonda TCP falhou) e os únicos `subprocess.Popen` foram `git rev-parse HEAD` [V].
89
+ O wrapper deve manter o `git` no `PATH`.
90
+ - Um índice de cada vez (fila), limite de memória 1,5 GB e timeout por job. Medido: o `update` incremental chega a 1,13-1,17 GB no maior processo filho nos repos de ~1 700 ficheiros [V]; num VPS de 4 GB cabe, num de 2 GB não.
91
+ Em 2 vCPU 4 GB **não medido** [N].
92
+ - `graphify-out/` fica no volume de dados do cliente, fora do git, e entra na lista de limpeza/erasure. `git ls-files graphify-out` tem de dar 0; hoje o `visionhammer-agents@release/qa` tem 57 ficheiros versionados [V] (remoção = PR próprio no agents, `git rm -r --cached graphify-out`).
93
+ - Utilizador não-root, FS só de leitura exceto o volume de dados e `/tmp`.
94
+
95
+ ## 5. Checklist de licenças
96
+
97
+ Inventário da instalação do harness (32 distribuições, `lib/licenses.py`) [V]: **32 permissivas, nenhuma copyleft, nenhuma desconhecida**.
98
+
99
+ | Componente | Licença (metadados) | Ficheiro de licença no wheel | Ação |
100
+ |---|---|---|---|
101
+ | graphifyy 0.9.93 | Apache-2.0 (+ `LICENSE-MIT`, `NOTICE`) | sim (`LICENSE`, `LICENSE-MIT`, `NOTICE`) | copiar os três para `/usr/share/doc/vh-graphify/`; incluir o `NOTICE` no `THIRD_PARTY_NOTICES` do release |
102
+ | networkx 3.7 / 3.6.1 | BSD-3-Clause | sim | notice |
103
+ | numpy 2.5.4 / 2.4.6 | `BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0` (libs incluídas) | sim | notice com a lista das licenças incluídas |
104
+ | RapidFuzz 3.14.6, tree-sitter 0.25.2 (core) | MIT | sim | notice |
105
+ | 27 gramáticas `tree-sitter-*` instaladas | MIT (2 em Apache-2.0: `tree-sitter-elixir`, `tree-sitter-hcl`) | sim, **exceto `tree-sitter-groovy`** | `tree-sitter-groovy` é MIT nos metadados mas **o wheel não traz ficheiro de licença**: obter o texto do repositório upstream e anexar [N] |
106
+ | ripgrep (Debian) | Unlicense/MIT | pacote Debian | notice (D14) |
107
+ | Python 3.11 (Debian) | PSF | pacote Debian | notice |
108
+
109
+ Checklist antes de publicar:
110
+
111
+ - [ ] Gerar o SBOM CycloneDX da imagem final (inclui hashes e licenças de cada wheel) e anexar à release; é a fonte do `THIRD_PARTY_NOTICES`.
112
+ - [ ] Reter `LICENSE`, `LICENSE-MIT` e `NOTICE` do Graphify, sem os alterar (exigência do Apache-2.0 para redistribuição; o `package.json` do agente diz `Proprietary`, o que não impede redistribuir um componente Apache-2.0 com notice).
113
+ - [ ] Obter o texto de licença de `tree-sitter-groovy` e conferir que a gramática distribuída corresponde ao MIT declarado.
114
+ - [ ] Não incluir os extras `leiden` (graspologic), `neo4j`, `falkordb`, `postgres`, `google`, `video`, `office`, `pdf` sem nova revisão: não foram inventariados [N].
115
+ - [ ] Marca: o upstream promove o "Graphify Cloud" no README; a imagem e a documentação VH não usam o nome nem o logótipo comerciais, só o nome do pacote.
116
+ - [ ] Confirmar com quem decide (jurídico) que a redistribuição dentro de uma imagem proprietária cumpre os contratos de cliente. Isto é uma checklist técnica, **não parecer jurídico**.
117
+ - [ ] Repetir o inventário (`lib/licenses.py`) a cada subida de versão; qualquer licença nova fora de MIT/BSD/Apache/PSF/ISC bloqueia o PR.
118
+ - [ ] Decisão pendente do Edu: abandonar o fork `VISIONHAMMER/visionhammer-graphify` (recomendado no plano GA); se se mantiver, manter os cabeçalhos de licença e o `NOTICE` do upstream.
119
+
120
+ ## 6. Assinatura e SBOM (D12)
121
+
122
+ - `syft <imagem> -o cyclonedx-json` no workflow de build; o SBOM vai como asset da release e é anexado à imagem com `cosign attach sbom` ou atestação `cosign attest --type cyclonedx`.
123
+ - `cosign sign` da imagem por digest (keyless via OIDC do GitHub Actions, ou chave no KMS; escolha por decidir [N]). O workflow `build-byoc-image.yml` hoje não assina [N, não verificado nesta tarefa].
124
+ - Verificação no instalador/Helm: `cosign verify` do digest antes de arrancar; o Helm deixa de usar `:latest` e passa a referenciar o digest.
125
+ - O hash do lock e o hash do SBOM do pack Graphify ficam no `metadata` do heartbeat para o gateway poder detetar versões não aprovadas (desenho em T4-07/T4-08) [N].
126
+
127
+ ## 7. O que falta provar (antes de declarar G9)
128
+
129
+ 1. `docker build -f Dockerfile.full` com o bloco da secção 2.2 e medir o delta de imagem (limite 400 MB).
130
+ 2. `docker run --read-only --network none --user 10001 ... vh-graphify update /workspace` com sucesso, mesmo grafo que o harness (comparar `signature`).
131
+ 3. Lock 3.11 em Linux x86_64 e aarch64; `pip-audit` sobre `requirements-py311.lock`.
132
+ 4. `tree-sitter-groovy`: licença anexada.
133
+ 5. Wrapper `vh-graphify` com teste de que cada comando proibido sai com 64 e que nenhuma chave LLM é herdada.