cloudmap 1.3.2__tar.gz → 1.3.4__tar.gz

cloudmap-1.3.4/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,43 @@
+name: Bug report
+description: Something broke, or the map is wrong
+labels: [bug]
+body:
+  - type: input
+    id: version
+    attributes:
+      label: cloudmap version
+      description: "`pip show cloudmap` or `cloudmap --help` header"
+      placeholder: "1.3.2"
+    validations:
+      required: true
+  - type: input
+    id: command
+    attributes:
+      label: The exact command you ran
+      placeholder: "cloudmap trace my-app --live --allow-live --single-sub"
+    validations:
+      required: true
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened, and what did you expect
+      description: Paste the output. If the map itself is wrong (a missing or bogus edge), say which edge.
+    validations:
+      required: true
+  - type: textarea
+    id: fixture
+    attributes:
+      label: Minimal synthetic fixture (for wrong-map bugs)
+      description: >
+        The perfect report reproduces the wrong edge with a minimal synthetic
+        fixture - see fixtures/contoso.json for the shape. NEVER paste a real
+        capture; run `cloudmap scrub` on it first and read the file before
+        posting.
+      render: json
+  - type: checkboxes
+    id: scrubbed
+    attributes:
+      label: Safety check
+      options:
+        - label: Anything I pasted above contains no real resource names, subscription ids or credentials.
+          required: true

cloudmap-1.3.4/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Security issue
+    url: https://github.com/KatsaounisThanasis/cloudmap/security/advisories/new
+    about: Anything that could expose a user's infrastructure data - report it privately, never as a public issue.

cloudmap-1.3.4/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,20 @@
+name: Feature request
+description: A resource type mapped poorly, a missing edge kind, a workflow that needs a flag
+labels: [enhancement]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: The situation
+      description: What were you trying to learn from the map, and what did it give you instead?
+    validations:
+      required: true
+  - type: textarea
+    id: proposal
+    attributes:
+      label: What you would expect
+      description: >
+        If it is a missing dependency for a resource type: which property of
+        which type expresses it? (That is usually a small extractor rule plus a
+        fixture test.) Keep in mind the rules in CONTRIBUTING.md - read-only,
+        local-first, every edge carries evidence.

cloudmap-1.3.4/.github/workflows/ci.yml

@@ -0,0 +1,51 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        # 3.9 is the floor the README promises; 3.13 catches what is about to break.
+        # Windows and macOS are here because the package claims OS Independent and
+        # an ubuntu-only matrix let a Windows-breaking bug ship: `az` installs as
+        # az.cmd there, and subprocess without shell=True never finds it. macOS is
+        # the cheap third data point for path and encoding assumptions.
+        include:
+          - {os: ubuntu-latest,  python-version: "3.9"}
+          - {os: ubuntu-latest,  python-version: "3.13"}
+          - {os: windows-latest, python-version: "3.13"}
+          - {os: macos-latest,   python-version: "3.13"}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check .
+      - run: pytest
+      # The suite includes tests/test_fixtures_safe.py, which fails if a fixture
+      # ever carries a credential or an unscrubbed GUID. That guard is the reason
+      # CI runs on pull requests too.
+
+      # A smoke test of the path a stranger actually takes first, on every OS:
+      # writing all five artifacts to disk exercises the file encodings and the
+      # path joining that a pure-unit suite never touches.
+      - name: demo trace writes every artifact
+        run: >
+          cloudmap trace contoso-web --from demo -d smoke
+          --json smoke/x.json --html smoke/x.html --csv smoke/x.csv --mermaid smoke/x.mmd
+      - name: the artifacts exist and are not empty
+        shell: python
+        run: |
+          import pathlib, sys
+          missing = [str(p) for p in [pathlib.Path("smoke", f"x.{e}")
+                                      for e in ("json", "html", "csv", "mmd")]
+                     if not pathlib.Path(p).exists() or pathlib.Path(p).stat().st_size == 0]
+          if missing:
+              sys.exit("empty or missing: " + ", ".join(missing))

cloudmap-1.3.4/.gitignore

@@ -0,0 +1,30 @@
+__pycache__/
+*.pyc
+.venv/
+venv/
+build/
+dist/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+
+# generated diagram output
+*.blast.drawio
+*.mmd
+out/
+
+# never commit anything pulled from a live cloud
+live/
+*.live.json
+.team/
+
+# A live trace writes artifacts named after the resource it traced, so their
+# names cannot be known in advance. Point --out-dir at live/ or out/ above.
+# These root-anchored rules are the belt-and-braces for the ones that land in
+# the repo root anyway - no tracked file at the root has these extensions, and
+# the committed sample artifacts live under tests/ and docs/, which are unaffected.
+/*.drawio
+/*.json
+/*.html
+/*.csv
+/*.mmd

cloudmap-1.3.4/ARCHITECTURE.md

@@ -0,0 +1,179 @@
+# Architecture
+
+> Written in English to match the rest of the repo (README, code, comments).
+> `file.py:NN` references are indicative - they drift as the code moves. Trust the
+> function and file names; grep for the symbol rather than jumping to the line.
+
+## Overview
+
+cloudmap is a local-first CLI (Python 3.9+; two runtime deps, `rich` and
+`questionary`, both for the terminal UI) that takes the **name of one Azure
+resource** and produces its full **dependency graph (blast radius)** as an
+**editable draw.io diagram**, plus an interactive HTML viewer, Mermaid, JSON and
+CSV. It is a clean four-stage pipeline: **ingest** (fixture or live `az`) →
+**extract** (properties / hostnames / RBAC / secrets → typed edges) → **graph**
+(direction-consistent blast-radius BFS) → **render**. The core value is that
+Azure Resource Graph has no "dependencies" table, so `extract/extractors.py`
+infers dependencies and **verifies** each one - anything referenced but not
+verified becomes an explicit `external` node instead of being silently dropped.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    CLI["__main__.py → cli.main()<br/>argparse: trace / capture / scrub / ask"] --> TRACE["_cmd_trace"]
+    CLI -->|"no args, tty"| WIZ["interactive.py<br/>wizard: sub → rg → resource"]
+    WIZ --> TRACE
+
+    TRACE -->|"--from (or 'demo')"| FIX["adapters/load_graph()<br/>raw export or neutral map"]
+    TRACE -->|"--live"| AZ["ingest/azure.py<br/>query_live()"]
+
+    AZ --> GUARD["_guard()<br/>optional ALLOW_SUB pin"]
+    AZ --> SUBS["_target_subscriptions()<br/>tenant-wide by default"]
+    AZ --> PAGE["_graph_paged()<br/>ALL types, skip_token paging"]
+
+    FIX --> BUILD["graph.build_graph()"]
+    AZ --> BUILD
+    BUILD --> EXTRACT["extract_edges()<br/>typed rules + 4 generic passes"]
+
+    TRACE --> ENRICH["_enrich_live() per seed type<br/>enrich_webapps() / enrich_aks_clusters()<br/>appsettings+RBAC+diag, secrets in-memory"]
+    ENRICH --> EXTERNAL["seed_external_dependencies()<br/>never-drop → external nodes"]
+
+    BUILD --> BLAST["blast_radius() graph.py<br/>BFS, direction consistency"]
+    EXTERNAL --> BLAST
+    BLAST --> COLLAPSE["collapse_high_level()<br/>singletons keep their name"]
+    COLLAPSE --> R1["render/drawio.py<br/>native Azure icons"]
+    COLLAPSE --> R2["render/mermaid.py"]
+    COLLAPSE --> R3["render/json_out.py<br/>meta: truncated/gaps/blind spots"]
+    COLLAPSE --> R4["render/html.py<br/>self-contained viewer"]
+    COLLAPSE --> R5["render/csv_export.py<br/>edge list + evidence"]
+
+    CLI --> ASK["_cmd_ask"]
+    R3 -.->|"saved map, reloaded"| FIX
+    ASK --> INTENT["ask/intent.py<br/>rules first, model only as<br/>validated fallback (--llm)"]
+    INTENT --> Q["ask/queries.py<br/>impact / depends / paths /<br/>shared / guesses / summary<br/>COMPUTED from edges"]
+    Q --> NARR["ask/narration.py (--explain)<br/>prose from the computed facts"]
+    INTENT -.-> LM["local_model.py<br/>the only outbound call:<br/>ollama or OpenAI-compatible, local"]
+    NARR -.-> LM
+```
+
+## File Map
+
+| File | Role | Why it's built this way |
+|---|---|---|
+| `cloudmap/model.py` | `Node`/`Edge`/`Graph` dataclasses | Provider-neutral core. The `external`+`note` fields exist for the "never silent drop" principle - a referenced-but-unverified target still needs a place in the model. |
+| `cloudmap/ingest/fixture.py` | Load synthetic/captured JSON | Accepts both a bare list and `{"data":[...]}` so the same file works as a fixture and as captured `az graph` output. |
+| `cloudmap/ingest/azure.py` | Live `az` ingest + enrichment | The most sensitive file, so it concentrates every guard: the single `subprocess.run` boundary with timeouts, the optional `CLOUDMAP_ALLOW_SUBSCRIPTION` pin (never injected into `az account` commands, so a mismatch is reported as a mismatch), `classify_gap()` so artifacts carry categories instead of raw stderr, and the rule that Kubernetes secret text never reaches an export. |
+| `cloudmap/extract/extractors.py` | **The heart**: properties/hosts/RBAC/secrets → edges | Typed rules for ~17 services plus four generic passes that make "any type" true: ARM-id references (with containment property paths excluded, not whole types), host mentions, nested child-of, and generic host indexing (a host counts as identity only when it carries the resource's own name - two apps on one App Service scale unit share `ftpsHostName`). Observers (alert rules, dashboards) emit `observes` instead of vanishing. |
+| `cloudmap/graph.py` | Build graph + blast-radius BFS + the high-level collapse | Direction consistency in `blast_radius` (never reverse once you have stepped, tracked per direction) is why a shared VNet/plan does not pull in unrelated apps. `collapse_high_level` folds instances into one box per type, keeps a singleton's real name, and carries kinds AND evidence across. |
+| `cloudmap/interactive.py` | The zero-argument wizard | Offers every resource in the subscription (no type allowlist - the engine has none), delegates all `az` I/O to `ingest.azure`, and pins the chosen subscription before handing off to `_cmd_trace`. |
+| `cloudmap/scrub.py` | Global consistent pseudonymisation for captures | Identifier positions are renamed consistently (a reference and its target stay correlated); credentials are redacted, and its docstrings carry the postmortems of the two leak classes that shipped and were caught. |
+| `cloudmap/render/drawio.py` | `.drawio` with Azure2 icons | Verified icon paths; falls back to a box, external nodes render dashed, so an icon is never broken. |
+| `cloudmap/render/html.py` | One self-contained interactive viewer | No server, no CDN; the INCOMPLETE badge opens its "why" in the side panel, grouped ×N boxes list their instances on click. |
+| `cloudmap/render/{mermaid,json_out,csv_export}.py` | Secondary outputs | Preview, machine-readable inventory (with the honesty meta), and a flat evidence-bearing edge list for auditors. |
+| `cloudmap/ask/queries.py` | **The Ask layer's heart**: the six queries, each computed by traversal | An answer must be auditable, so it is derived from edges, never generated. `_trust()` grades a whole path by its weakest hop. |
+| `cloudmap/ask/intent.py` | Question → one query | Rules first; the model is a fallback that may only name a query from a fixed list and a resource, both validated against the graph - it routes, it never answers. |
+| `cloudmap/ask/narration.py` | Optional prose (`--explain`) | Handed the computed facts only, printed *below* them. |
+| `cloudmap/local_model.py` | The single outbound model call | One module = one auditable promise: localhost (ollama's API by default, any OpenAI-compatible server via `CLOUDMAP_LLM_URL`), and failure returns an empty value. |
+| `cloudmap/cli.py` | Orchestration + argparse | `_cmd_trace` wires the stages; `_cmd_capture`/`_cmd_scrub` produce committable fixtures; `_cmd_ask` loads a saved map and prints the computed answer, proof lines included. |
+| `cloudmap/data/contoso.json` | The packaged demo estate | `--from demo` must work for a pip-only install - the first command a stranger runs cannot depend on cloning the repo. |
+| `fixtures/` | Synthetic + scrubbed-golden estates | Fixture-first development → zero cloud contact in tests. |
+| `tests/` | 536 tests | Behaviour-named; includes a conformance sweep of all 4,687 published Azure resource types (`tests/data/azure_resource_types.txt`) through every dependency channel, a realistic-estate suite with the property names Azure genuinely returns, `test_fixtures_safe.py` (fails CI if a fixture ever carries a credential), `test_deterministic_output.py` (subprocesses with different hash seeds - the in-process suite cannot see that class of bug) and `test_sovereign_clouds.py`. |
+| `.gitignore` | Safety | `live/`, `*.live.json`, `*.blast.drawio` and every artifact extension at the repo root are ignored → output from a real tenant cannot be committed by accident. |
+
+## Execution Flow
+
+**Fixture path** (`cloudmap trace contoso-web --from demo`):
+
+1. `__main__.py` → `cli.main()` - argparse, or the wizard when run bare on a tty.
+2. `_cmd_trace` - fixture branch → `adapters.load_graph()` (raw export or neutral
+   map, auto-detected; a reloaded capture keeps its truncated/gaps meta).
+3. `graph.build_graph()` → builds `Node`s and calls `extract_edges()`.
+4. `extract_edges()` - constructs a `Resolver` and derives edges: typed rules
+   (`serverFarmId`→hosted-on, VM `has-nic`/`uses-disk`, NIC `in-subnet`, config
+   hostnames/vault refs/IK, RBAC via system AND user-assigned identity
+   principals, `managedBy`), then the generic passes for every other shape.
+5. `find_seeds()` - exact id, exact name, else substring; >1 → each match is
+   printed with its resource group and full id.
+6. `blast_radius()` - BFS from the seed in both directions, never reversing.
+7. Render: draw.io / HTML / Mermaid / JSON / CSV, `_print_summary` shows
+   "Depends on" and "What depends on it" as separate panels.
+
+**Live path** (`--live --allow-live [--resolve-secrets]`) - additionally:
+
+8. `query_live()` → `_guard()` (optional pin check) → `_target_subscriptions()`
+   → `_graph_paged()` (ALL resource types + tenant-wide role assignments,
+   `skip_token` paging, loud warning at the page cap).
+9. `_enrich_live()` picks targets by seed type: a workload seed enriches itself;
+   a config-referenced data service (Key Vault, SQL, storage, Redis, ...)
+   enriches every app in scope, because only their config reveals its
+   dependents; compute/network seeds stay seed-only - ARM already has their
+   relationships. `enrich_webapps()`/`enrich_aks_clusters()` run concurrently
+   (`CLOUDMAP_ENRICH_WORKERS`, default 12). Secrets resolve **in-memory** only.
+10. Rebuild, `seed_external_dependencies()` resurfaces unresolved references as
+    dashed external nodes, and whatever was *not* enriched is recorded as a
+    blind spot in `meta` - so the artifact, and every `ask` answer drawn from
+    it, repeats it.
+
+## Design Decisions
+
+- **extract vs ingest separation** (`extractors.py` knows nothing about `az`).
+  All dependency logic is pure Python over dicts, so the full suite runs in ~2s
+  without Azure.
+- **Direction consistency** (`blast_radius`). Real-world testing showed shared
+  plan/VNet connect dozens of unrelated apps. "From the seed go both ways, but
+  never reverse afterwards" solves it without a hub-type list; the visited set
+  is tracked per direction so a node reachable both ways loses neither cone.
+- **Containment is not dependency** (`_CONTAINMENT_PATHS`). A VNet's
+  `subnets[].ipConfigurations` enumerate what is plugged INTO it; following them
+  made a shared VNet a hub. But its DDoS plan and peerings are real
+  dependencies, so the exclusion is on property paths, not on types - an
+  earlier type-wide exclusion silently lost all of those edges.
+- **Generic before hand-written, resolver as the safety net.** A type with no
+  rule still maps through ARM ids, advertised hosts, child ids and RBAC; a
+  reference that names nothing scanned yields no edge, so the generic passes
+  cannot invent a dependency. This is what a 4,687-type conformance sweep pins.
+- **Enrichment is scoped by seed type, and the gap is declared**
+  (`_enrichment_targets`). Tracing a VM must not deep-read 289 web apps to
+  learn what ARM already says; tracing a vault must, because config is the only
+  place its dependents exist. Whatever is skipped becomes a `blind_spot`.
+- **Scrub preserves structure, not identity** (`scrub.py`), and a capture never
+  carries Kubernetes secret text at all - the field is stripped before any
+  export, under every flag combination.
+- **Never silent drop** (`seed_external_dependencies` + `model.py`).
+- **The model proposes, code verifies** (`extract/llm.py`): only proposals that
+  resolve to a scanned resource survive, drawn dashed and marked; `ask` may use
+  the model to route a phrasing or narrate a computed answer, never to state a
+  fact.
+
+## Open Questions / Risks
+
+- **Live validation breadth**: the engine is conformance-tested against every
+  published type, but has been exercised against one real tenant, in the public
+  cloud, on Linux. Property shapes elsewhere may surface new false-positive
+  classes like the shared `ftpsHostName` one (found live, fixed,
+  regression-pinned). The sovereign clouds are covered by construction rather
+  than by a live run: an edge's kind now comes from the resolved target's ARM
+  type, not from its DNS suffix (`extractors.kind_for_target`), so
+  `vault.usgovcloudapi.net` and `vault.azure.cn` need no table entry. Windows
+  and macOS are covered by CI rather than by a table of assumptions.
+- **Deep config readers exist for App Service and AKS only** - the two common
+  types whose config Resource Graph withholds. Checked, not assumed: Logic Apps,
+  Container Apps, ML workspaces, Cognitive Services and API Management all return
+  theirs in `properties`. What remains hidden is a different shape - child
+  resources a provider never registered with Resource Graph (Data Factory and
+  Synapse linked services, Automation runbooks, Stream Analytics inputs) and
+  data planes with their own API (Databricks secret scopes). Those types are
+  listed in `cli._CONFIG_HIDDEN_TYPES` and declared as a blind spot on every
+  map that contains one, live or fixture, so an empty result cannot pass for an
+  answer. A reader for them is roadmap, not present.
+- **Cost of `--enrich all`** on large tenants: one `az` round-trip per app,
+  `CLOUDMAP_ENRICH_WORKERS` (default 12) at a time. No cache between runs yet -
+  every trace pays the scan again.
+- **`az graph --skip-token`** depends on the resource-graph extension version.
+- **`az aks command invoke`** starts a short-lived pod in the cluster to run
+  its read-only kubectl - documented in the README as the one asterisk on
+  "read-only".
+- **Secret resolution reachability**: a vault behind an unreachable private
+  endpoint fails the in-memory resolve and the dependency shows as external
+  rather than resolved.