ghost-dragon 4.2.1

package/docs/SYSTEM-PREMORTEM.md

@@ -0,0 +1,397 @@
+# Dragon Stack · System Pre-Mortem
+
+Scope: the **whole stack** under Ghost Protocol's offensive-security
+banner — PhantomDragon AI (engine), PhantomDragon Control (operator
+dashboard), DragonNet (OSINT investigation platform), Phantom Memory
+(persistent engagement memory), Wyrm (cross-tool memory backbone),
+and Dragon CLI (unified shell). This pre-mortem imagines it's 12
+months from now and the integrated stack has been abandoned.
+
+This is a *system* pre-mortem. Individual products have their own
+under `~/Git Projects/<product>/docs/PRE-MORTEM.md`. Here we focus on
+what breaks **between** them.
+
+> Re-read at every quarterly review, before any cross-tool refactor,
+> and after any incident that crosses product boundaries. Update
+> probabilities as we learn.
+
+---
+
+## #1 · Cross-system schema drift
+
+**Probability: HIGH · Impact: SEVERE**
+
+Each product evolves on its own cadence. PhantomDragon's
+`findings.json` shape, DragonNet's `entities` table, Phantom Memory's
+SQLite schema, and Wyrm's `data` records all carry implicit
+contracts. There is no single source of truth and no contract test.
+
+**How it kills us:** PhantomDragon v3.6 renames `meta.target` to
+`meta.target_url`. Phantom Memory still reads `meta.target`. The
+`/v1/memory` endpoint silently records empty targets for new scans.
+Three months later, the operator notices "no memory for upalis.com"
+even though they scanned it last week. Trust evaporates.
+
+**Controls in v1:**
+- Phantom Memory uses defensive `data.get("meta", {}).get(...)` so a
+  field rename produces nulls, not crashes.
+- The dragon CLI's `dragon status` could compare versions across
+  products (not yet implemented).
+
+**Mitigations deferred:**
+- A single canonical schema package (e.g. `@dragon/shared-schemas`)
+  imported by every product. Engine emits via Pydantic, dashboards
+  consume via TypeScript types generated from JSON Schema.
+- `dragon doctor` contract test that runs sample data from each
+  product's outputs through the other products' parsers; fails
+  CI when shapes mismatch.
+- A version field in every persisted record (`meta.schema_version`)
+  with explicit upgrade paths.
+
+**Death-avoidance metric:** Days since a schema field rename
+without coordinating updates across all consumers. >0 days =
+yellow; >7 days = red.
+
+---
+
+## #2 · The unified-CLI mirage
+
+**Probability: HIGH · Impact: MEDIUM**
+
+`dragon` exists. But if operators settle into Tool A's native UI
+(PhantomDragon Control's dashboard) and Tool B's native CLI
+(PhantomDragon's `phantomdragon.py`), the unified CLI never gets the
+exercise it needs. Bug rot sets in. The cross-tool flows that only
+the CLI knows about silently break.
+
+**How it kills us:** Six months in, `dragon scan` has rotted because
+the PhantomDragon CLI has new flags the wrapper doesn't pass through.
+`dragon osint` doesn't exist because no one needed it badly enough
+to add it. Operators conclude the CLI was a nice idea that didn't
+land.
+
+**Controls in v1:**
+- Dragon CLI is the *recommended* entry point in the onboarding
+  doc.
+- `dragon scan` flags mirror the engine's flag surface 1:1.
+
+**Mitigations deferred:**
+- The dragon CLI should be the **only** documented entry point in
+  external docs (README, blog posts, customer-facing). Native
+  binaries remain available but are internal.
+- `dragon up` one-command bootstrap so new operators never know
+  the underlying ports.
+- `dragon status` becomes the canonical "is everything OK" view —
+  no operator should ever need to `ss -tlnp | grep`.
+
+**Death-avoidance metric:** Ratio of CLI invocations to native
+binary invocations across operator sessions. Target ≥3:1 by month 3.
+
+---
+
+## #3 · Two memory systems, one operator
+
+**Probability: HIGH · Impact: HIGH**
+
+Phantom Memory (`~/.copilot/phantom-memory.db`, SQLite) and Wyrm
+(`~/.copilot/session-store.db`, also SQLite + HTTP :3333) both claim
+to be the cross-tool memory store. Today they don't overlap — Phantom
+Memory holds scan/OSINT/credential data, Wyrm holds session/quest/
+context data. But the boundary is fuzzy.
+
+**How it kills us:** A future feature wants to query "what targets
+has Ryan worked on this quarter?" — both DBs have a partial answer.
+The query is built against Wyrm. Two months later someone builds
+the same query against Phantom Memory because they didn't know
+Wyrm had it. Two truths, no reconciliation, and now the AI brain
+gets *different* context depending on which path it took.
+
+**Controls in v1:**
+- Phantom Memory is documented as "scan + OSINT + creds";
+  Wyrm is "sessions + quests + decisions". Different concerns.
+
+**Mitigations deferred:**
+- One-paragraph "where data lives" doc in `dragon-cli/docs/`.
+- A federated query endpoint (`dragon memory query "..."` that hits
+  both stores and merges results with provenance).
+- A long-term decision: either fold Phantom Memory *into* Wyrm
+  as a new entity kind, or extract Wyrm's session/quest layer
+  into Phantom Memory. One store, two access paths.
+
+**Death-avoidance metric:** Number of queries that get a different
+answer from Phantom Memory vs Wyrm for the same logical question.
+Should be 0.
+
+---
+
+## #4 · No multi-tenant isolation
+
+**Probability: MEDIUM · Impact: SEVERE (for PTaaS)**
+
+PhantomDragon Control's `reports/` is single-operator. Phantom Memory
+is single-DB. DragonNet has investigation membership and roles
+(good) but no account isolation at the DB level. The day Ghost
+Protocol takes its second PTaaS customer, this becomes a problem.
+
+**How it kills us:** Customer A logs in and sees scans that include
+PII from Customer B's penetration test (because Phantom Memory's
+"all targets" view doesn't know about tenancy). One incident →
+contract terminated, GDPR exposure, reputational hit.
+
+**Controls in v1:**
+- Deploys are explicitly single-operator. No customer is live yet.
+- PhantomDragon PTaaS (separate spec-kit project at
+  `~/Git Projects/phantomdragon-ptaas/`) does account-level isolation
+  in its own DB.
+
+**Mitigations deferred:**
+- Phantom Memory gains an `operator_id` / `tenant_id` foreign key
+  on every table.
+- PhantomDragon Control gains auth + per-operator reports/ dir
+  before any external deploy.
+- The dragon CLI carries a `--operator` context that gates queries.
+
+**Death-avoidance metric:** Days until first non-Ryan operator gets
+access to the stack. Becomes red the moment that's <30 days away
+without multi-tenant work shipped.
+
+---
+
+## #5 · Single-laptop disaster
+
+**Probability: MEDIUM · Impact: SEVERE**
+
+The entire engagement memory — every scan finding, every OSINT
+entity, every credential exposure across years — lives in one
+SQLite file on Ryan's laptop. Disk failure = years gone.
+
+**How it kills us:** Laptop dies. Ryan reformats. Phantom Memory
+was "going to be backed up next week." Engagements with paying
+customers reference finding IDs that are now permanently lost.
+
+**Controls in v1:**
+- Wyrm has R2 cloud backup (Wyrm task #171).
+- `reports/` is a git-tracked directory in some cases.
+
+**Mitigations deferred:**
+- `dragon backup` command that snapshots Phantom Memory + reports/
+  + DragonNet's Postgres dump + Wyrm DB to encrypted R2.
+- Nightly cron via systemd timer (operator opt-in).
+- `dragon restore` for round-trip verification.
+
+**Death-avoidance metric:** Days since last successful Phantom
+Memory backup to a different physical drive. <7 = green, 7-30 =
+yellow, >30 = red.
+
+---
+
+## #6 · Auth fragmentation
+
+**Probability: MEDIUM · Impact: MEDIUM**
+
+DragonNet: magic-link + JWT + TOTP. PhantomDragon Control: no auth
+(binds to localhost). Wyrm: bearer token from `~/.wyrm/http-config.json`.
+The dragon CLI: trusts the operator's shell.
+
+**How it kills us:** First deploy to a LAN — PhantomDragon Control
+suddenly readable by anyone in the office network. The cross-link
+from PhantomDragon Control → DragonNet does *not* pass auth state,
+so the operator has to re-login every time they click "OSINT this".
+That friction means they stop using the integration.
+
+**Controls in v1:**
+- Documented as "localhost-only" until auth ships.
+- DragonNet's auth is solid — magic-link + TOTP + JWT session.
+
+**Mitigations deferred:**
+- Adopt DragonNet's auth as the canonical layer; PhantomDragon
+  Control consumes its sessions via shared cookie.
+- A common origin (`http://localhost:4000` reverse-proxied to all
+  services) so the same session cookie applies everywhere.
+- The dragon CLI does an auth handshake on `dragon up`.
+
+**Death-avoidance metric:** Days since the stack ran with any
+service bound to 0.0.0.0 without auth. Becomes red the moment any
+such bind exists in production.
+
+---
+
+## #7 · Darkweb collectors create legal exposure
+
+**Probability: MEDIUM · Impact: HIGH**
+
+TORWATCH crawls operator-supplied .onion seeds. PASTEDRAGNET probes
+pastebin.com mirrors. TELEGAZER scrapes Telegram public channels.
+NIGHTGLASS reads operator-supplied breach corpora. Each of these is
+defensible on its own — they touch only public previews or operator-
+provided data. But the *combination* could be read by a hostile
+prosecutor as "systematic collection of breach data."
+
+**How it kills us:** Sri Lanka's Computer Crimes Act 2007 is broad
+enough that a determined regulator could call any of this "unauthorized
+access" if the seed list includes a credential market URL or the
+breach corpus came from somewhere borderline. Defense cost alone
+sinks Ghost Protocol.
+
+**Controls in v1:**
+- Every darkweb collector has an explicit operator-supplied input
+  (env var or breach DB).
+- TORWATCH refuses to fetch without an explicit Tor proxy set.
+- No collector invents seeds, channels, or breach data.
+
+**Mitigations deferred:**
+- A `dragon legal pre-flight <jurisdiction>` command that prints
+  the legal posture for each enabled collector in that
+  jurisdiction.
+- Explicit per-investigation consent flag: "I represent that the
+  target authorizes this investigation."
+- Tamper-evident audit log of every collector run (already in
+  DragonNet's `audit` table; surface it via UI).
+- Geofenced collectors that auto-disable in jurisdictions where
+  they're risky (operator-configurable).
+
+**Death-avoidance metric:** Number of investigations where a
+darkweb collector ran without an authorization record in the
+audit log. Should be 0.
+
+---
+
+## #8 · AI brain context bloat
+
+**Probability: MEDIUM · Impact: MEDIUM**
+
+`phantom_memory.ai_brain_context()` truncates at 4000 chars. For a
+target with 3 scans this is generous. For a target with 50 scans
+across 2 years, 500 OSINT entities, and 1000 credential exposures,
+4000 chars is nowhere near enough. The brain ends up with a
+truncated, randomly-ordered view of history.
+
+**How it kills us:** A heavy long-term target's scan produces worse
+results in month 18 than in month 1 because the context system can't
+prioritize what matters. Operator concludes "the AI got dumber"
+when actually we got smarter — we just don't know how to use the
+extra knowledge.
+
+**Controls in v1:**
+- 4000-char cap is documented as MVP.
+- Persistent findings + last-resolved are surfaced first
+  (highest signal).
+
+**Mitigations deferred:**
+- Tiered context: critical findings → high → medium-recency tech →
+  recent OSINT → credential exposures. Each tier has its own cap.
+- Vector embeddings on findings so the brain pulls semantically
+  relevant prior findings, not just chronological ones.
+- A `dragon memory context <target> --max-tokens N` flag for
+  larger budgets.
+
+**Death-avoidance metric:** Average char usage of brain context as
+% of cap. >85% means we're saturating; should expand or tier.
+
+---
+
+## #9 · Cross-tool data flow has no replay
+
+**Probability: LOW · Impact: HIGH**
+
+When PhantomDragon Control pushes a scan to Phantom Memory and
+Phantom Memory pushes to Wyrm, the flow is fire-and-forget. If any
+step fails (Phantom Memory disk full, Wyrm offline, network blip),
+data silently goes missing. No queue. No retry.
+
+**How it kills us:** Disk-full or Wyrm offline for an hour during a
+scan. The scan completes successfully in PhantomDragon's reports/.
+Phantom Memory ingestion fails silently. Operator notices weeks
+later that "memory has gaps." Cannot reconstruct which scans were
+ingested without parsing every reports/ dir against memory.
+
+**Controls in v1:**
+- `phantom_memory.ingest_reports_dir()` is idempotent — re-running
+  recovers any missed scans.
+- The `control_api.py` startup auto-runs `ingest_reports_dir()`.
+
+**Mitigations deferred:**
+- A persistent retry queue (`~/.copilot/phantom-memory-queue.json`)
+  for failed pushes to Wyrm.
+- `dragon memory verify` command that diffs reports/ against
+  Phantom Memory and reports any gaps.
+- `dragon memory reingest --since 2026-04-01` for time-bounded
+  recovery.
+
+**Death-avoidance metric:** Count of reports/ scans not present in
+Phantom Memory after a `dragon status memory` check. Should be 0.
+
+---
+
+## #10 · Operator-onboarding cliff
+
+**Probability: LOW · Impact: HIGH (if hiring)**
+
+The stack assumes the operator (Ryan) built it. A new operator
+joining tomorrow would face: 3 repos to clone, 5 services to start,
+2 database systems to migrate, 1 unified CLI to install but with
+spotty docs, and a mental model that lives only in Ryan's head.
+
+**How it kills us:** Ghost Protocol hires a second operator. They
+spend 3 days getting the local stack running. They never quite
+understand which features go through which dashboard. They produce
+slower, worse engagements for the first month. Either Ryan refuses
+to hire again, or the new operator quits.
+
+**Controls in v1:**
+- `dragon init` auto-detects product paths.
+- This pre-mortem + the skills under `~/.copilot/skills/` exist.
+
+**Mitigations deferred:**
+- `dragon up` one-command bootstrap (clones missing repos, runs
+  migrations, starts services, opens the dashboard).
+- A 1-page operator playbook ("here's how an engagement flows
+  through the stack").
+- Onboarding video / Loom (operator-facing, 10 min).
+- `dragon doctor` to verify the local install matches the
+  reference shape.
+
+**Death-avoidance metric:** Time-to-first-real-scan for a new
+operator. Target < 30 minutes from `git clone dragon-cli` to a
+successful `dragon scan example.com`.
+
+---
+
+## Death-avoidance dashboard
+
+Track monthly. Anything red triggers a remediation session.
+
+| Metric | Green | Yellow | Red |
+|---|---|---|---|
+| Days since cross-tool schema rename without coordination | 0 | 1-7 | >7 |
+| Ratio of dragon-CLI to native-CLI invocations | ≥3:1 | 1-3:1 | <1:1 |
+| Phantom Memory vs Wyrm divergence for same logical query | 0 | 1-3 | >3 |
+| Days until first non-Ryan operator | >30 | 7-30 | <7 |
+| Days since last Phantom Memory backup to second drive | <7 | 7-30 | >30 |
+| Days running 0.0.0.0 without auth | 0 (localhost only) | — | any |
+| Investigations with darkweb collectors lacking authorization record | 0 | 1-3 | >3 |
+| Avg brain-context char usage as % of cap | <70% | 70-90% | >90% |
+| Reports not present in Phantom Memory after status check | 0 | 1-3 | >3 |
+| Time-to-first-scan for new operator | <30min | 30-90min | >90min |
+
+## Re-read schedule
+
+- Every quarterly review
+- Before any cross-tool refactor
+- After any incident crossing product boundaries
+- When any metric crosses into yellow
+
+## The single thread
+
+Eight of these ten causes-of-death converge on the same root: **the
+stack treats its products as independent, but operators experience
+them as one system.** Schema drift, the CLI mirage, the two-memory
+problem, auth fragmentation, AI context bloat, the data-flow replay
+gap, and the onboarding cliff all stem from missing federation
+discipline.
+
+The fix is structural, not feature-by-feature: **the dragon CLI must
+be the federation layer.** Every cross-tool flow goes through it.
+Every health check, every backup, every schema contract, every auth
+handshake. The CLI is not a convenience — it's the integration
+substrate.

package/dragon-manifest.toml

@@ -0,0 +1,241 @@
+# Ghost Protocol · Dragon Stack Manifest
+# Declares every product in the stack so `dragon up`, `dragon update`,
+# `dragon install`, and `dragon stop` can manage the whole portfolio
+# from a single command.
+#
+# Copyright 2026 Ghost Protocol (Pvt) Ltd. All Rights Reserved.
+# Author: Ryan Sebastian <ryan@ghosts.lk>
+
+manifest_version = "1.0"
+default_root = "~/Git Projects"
+
+# Each [[product]] entry declares:
+#   key          - short slug used by `dragon install <key>` etc.
+#   name         - display name
+#   repo         - GitHub repo (owner/name)
+#   dir          - local checkout dir name (under default_root)
+#   stack        - "operator" | "commercial" | "daemon" | "library"
+#   kind         - "rust" | "node" | "python" | "monorepo" | "docker"
+#   install      - command(s) to bring up the local install
+#   start        - command to start the runtime (optional)
+#   stop         - command/signal to stop it (optional)
+#   port         - service port if applicable
+#   requires     - list of other product keys that MUST be installed first
+
+# ── Operator stack ────────────────────────────────────────────────
+
+[[product]]
+key = "phantom-dragon-ai"
+name = "PhantomDragon AI"
+repo = "Ghosts-Protocol-Pvt-Ltd/phantom-dragon-ai"
+dir = "Phantom Dragon AI"
+stack = "operator"
+kind = "python"
+install = ["pip install -e ."]
+start = "python3 -m phantom_dragon_ai.control_api"
+port = 4091
+
+[[product]]
+key = "dragon-console"
+name = "Dragon Console"
+repo = "ghosts-lk/dragon-console"
+dir = "phantomdragon-control"
+stack = "operator"
+kind = "node"
+install = ["npm install"]
+start = "npm run dev"
+port = 4090
+requires = ["phantom-dragon-ai"]
+# Dragon Console is the GUI sibling of the `dragon` CLI — houses
+# PhantomDragon (offensive), Phantom Memory, DragonNet (cross-link),
+# DragonKeep (cross-link + defensive queue view), and commercial
+# modules. Local dir stays "phantomdragon-control" until the repo is
+# renamed on GitHub (gh repo rename ghosts-lk/dragon-console).
+
+[[product]]
+key = "dragonhunt"
+name = "DragonHunt (lead-gen)"
+repo = "Ghosts-Protocol-Pvt-Ltd/DragonHunt"
+dir = "DragonHunt"
+stack = "operator"
+kind = "node"
+install = ["npm install"]
+# Reads ./.env (JWT_SECRET, PD_API, port). The 'safe' scan posture calls
+# control_api at :4091, so phantom-dragon-ai must be up first. Dragon Console
+# proxies to this at :3334 with DRAGONHUNT_TOKEN (see its .env.local).
+start = "npm run dev"
+port = 3334
+requires = ["phantom-dragon-ai"]
+
+[[product]]
+key = "dragonnet"
+name = "DragonNet OSINT API"
+repo = "ghosts-lk/dragonnet"
+dir = "DragonNet"
+stack = "operator"
+kind = "monorepo"
+install = ["pnpm install"]
+# Dashboard at :4081 is deprecated as of 2026-05-23 — its UI lives
+# inside Dragon Console under /investigations now. We only start the
+# Fastify API at :4080 going forward.
+start = "pnpm --filter @dragonnet/api dev"
+port = 4080
+
+[[product]]
+key = "dragonkeep"
+name = "DragonKeep"
+repo = "Ghosts-Protocol-Pvt-Ltd/DragonKeep"
+dir = "DragonKeep"
+stack = "operator"
+kind = "rust"
+install = ["cargo build --release"]
+start = "./target/release/dragonkeep monitor"
+
+[[product]]
+key = "dragon-cli"
+name = "Dragon CLI"
+repo = "ghosts-lk/dragon-cli"
+dir = "dragon-cli"
+stack = "operator"
+kind = "node"
+install = ["npm install", "npm run build", "npm link"]
+
+[[product]]
+key = "dragonchronicle"
+name = "DragonChronicle"
+repo = "ghosts-lk/dragonchronicle"
+dir = "DragonChronicle"
+stack = "library"
+kind = "python"
+install = ["pip install -e ."]
+
+[[product]]
+key = "dragonscan-sdk"
+name = "DragonScan SDK"
+repo = "ghosts-lk/dragonscansdk"
+dir = "DragonScanSDK"
+stack = "library"
+kind = "python"
+install = ["pip install -e ."]
+
+[[product]]
+key = "dragonagent"
+name = "DragonAgent"
+repo = "ghosts-lk/dragonagent"
+dir = "DragonAgent"
+stack = "library"
+kind = "python"
+install = ["pip install -e ."]
+requires = ["dragonscan-sdk"]
+
+[[product]]
+key = "dragonpilot"
+name = "DragonPilot"
+repo = "ghosts-lk/dragonpilot"
+dir = "DragonPilot"
+stack = "library"
+kind = "node"
+install = ["npm link"]
+
+[[product]]
+key = "dragonlens"
+name = "DragonLens VS Code"
+repo = "ghosts-lk/dragonlens"
+dir = "DragonLens"
+stack = "library"
+kind = "node"
+install = ["npm install", "npm run compile"]
+
+# ── MCP servers ───────────────────────────────────────────────────
+
+[[product]]
+key = "phantom-memory-mcp"
+name = "Phantom Memory MCP"
+repo = "ghosts-lk/phantom-memory-mcp"
+dir = "phantom-memory-mcp"
+stack = "daemon"
+kind = "python"
+install = ["pip install -e ."]
+
+[[product]]
+key = "dragonkeep-mcp"
+name = "DragonKeep MCP"
+repo = "ghosts-lk/dragonkeep-mcp"
+dir = "dragonkeep-mcp"
+stack = "daemon"
+kind = "python"
+install = ["pip install -e ."]
+
+[[product]]
+key = "dragonnet-mcp"
+name = "DragonNet MCP"
+repo = "ghosts-lk/dragonnet-mcp"
+dir = "dragonnet-mcp"
+stack = "daemon"
+kind = "python"
+install = ["pip install -e ."]
+
+[[product]]
+key = "dragon-meta-mcp"
+name = "Dragon Meta MCP"
+repo = "ghosts-lk/dragon-meta-mcp"
+dir = "dragon-meta-mcp"
+stack = "daemon"
+kind = "python"
+install = ["pip install -e ."]
+requires = ["phantom-memory-mcp", "dragonkeep-mcp", "dragonnet-mcp"]
+
+# ── Commercial stack ──────────────────────────────────────────────
+
+[[product]]
+key = "phantomdragon-ptaas"
+name = "PhantomDragon PTaaS"
+repo = "ghosts-lk/phantomdragon-ptaas"
+dir = "phantomdragon-ptaas"
+stack = "commercial"
+kind = "monorepo"
+install = ["pnpm install"]
+start = "pnpm dev"
+port = 4092
+
+[[product]]
+key = "dragonscribe"
+name = "DragonScribe"
+repo = "ghosts-lk/dragonscribe"
+dir = "dragonscribe"
+stack = "commercial"
+kind = "monorepo"
+install = ["pnpm install"]
+start = "pnpm dev"
+port = 4093
+
+[[product]]
+key = "dragonreef"
+name = "DragonReef"
+repo = "ghosts-lk/dragonreef"
+dir = "dragonreef"
+stack = "commercial"
+kind = "monorepo"
+install = ["pnpm install"]
+start = "pnpm dev"
+port = 4094
+
+# ── Memory backbone ───────────────────────────────────────────────
+
+[[product]]
+key = "wyrm"
+name = "Wyrm MCP"
+repo = "ghosts-lk/Wyrm"
+dir = "Wyrm"
+stack = "daemon"
+kind = "monorepo"
+# Wyrm has no root package.json or pnpm workspace — install + build run
+# inside packages/mcp-server. (Earlier `pnpm install` / `pnpm build`
+# entries silently no-op'd here and made `dragon up` look successful
+# while leaving Wyrm uninstalled.)
+install = [
+  "npm install --prefix packages/mcp-server",
+  "npm run build  --prefix packages/mcp-server",
+]
+start = "node packages/mcp-server/dist/http-server.js"
+port = 3333