vmware-debug 1.8.4__tar.gz → 1.8.5__tar.gz

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vmware-debug
-Version: 1.8.4
+Version: 1.8.5
 Summary: VMware diagnostic brain — read-only incident triage, log/event correlation, and root-cause routing across the VMware skill family
 Author-email: Wei Zhou <wei-wz.zhou@broadcom.com>
 License-Expression: MIT
@@ -13,7 +13,7 @@ Requires-Python: >=3.10
 Requires-Dist: mcp[cli]<2.0,>=1.10
 Requires-Dist: rich<15.0,>=13.0
 Requires-Dist: typer<1.0,>=0.12
-Requires-Dist: vmware-policy<2.0,>=1.8.4
+Requires-Dist: vmware-policy<2.0,>=1.8.5
 Description-Content-Type: text/markdown
 
 <!-- mcp-name: io.github.zw008/vmware-debug -->

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/RELEASE_NOTES.md

@@ -1,3 +1,91 @@
+## v1.8.5 (2026-07-20) — the two fixes v1.8.4 announced now actually work
+
+Four adversarial reviews of v1.8.4 found that both of its headline fixes were
+incomplete in ways the release notes did not reflect. This release makes them
+real. If you are on 1.8.4, this is the one to take.
+
+### Fixed — a failure that was *returned* was still audited as a success
+
+vmware-policy 1.8.4 added `report_tool_failure()` for tools that catch an
+exception and return an error payload instead of raising. **No skill called it.**
+
+Every string-returning tool therefore kept doing exactly what 1.8.4 said it had
+stopped doing: writing `status=ok` to `~/.vmware/audit.db` for an operation that
+failed, recording an undo token for a change that never happened, and telling the
+circuit breaker the call succeeded so repeated failures never tripped it.
+
+The surface this covered is not marginal:
+
+| Skill | What was mis-audited |
+|---|---|
+| vmware-aiops | 25 of 49 tools, including **every undo-bearing write** — a failed `vm_power_on` left an undo token saying "power it back off" |
+| vmware-avi | all 28 tools, including `vs_toggle` and `ako_restart` |
+| vmware-storage | all 4 write tools |
+| vmware-nsx | the 5 delete tools |
+
+vmware-avi is worth calling out: before 1.8.4 its exceptions propagated and the
+audit was correct. 1.8.4 caught them and returned a string, so **that release made
+its audit trail worse than it had been.**
+
+Skills whose tools already return dict payloads (vmware-monitor, vmware-vks,
+vmware-aria, vmware-log-insight, vmware-harden, vmware-debug, vmware-pilot) were
+already detected correctly. They gained a test proving it rather than a redundant
+call.
+
+### Fixed — narrowing `OSError` did not close the leak it was meant to close
+
+1.8.4 narrowed the `_safe_error` passthrough because bare `OSError` let TLS and
+DNS failures reach the agent with hostnames and certificate subjects in them.
+That narrowing had no effect on the error it was written for:
+
+```
+ssl.SSLCertVerificationError → ssl.SSLError → OSError, ValueError
+```
+
+`ValueError` has been on every allowlist since long before 1.8.4, so a
+certificate failure kept passing through — the commonest self-signed-certificate
+failure in this family, carrying the hostname it was checked against. An
+allowlist structurally cannot express "not this one".
+
+Where `ssl.SSLError` can actually surface — the pyVmomi skills — it is now
+reduced *ahead* of the allowlist. In the httpx skills TLS arrives wrapped as
+`httpx.ConnectError`, and in vmware-avi as `requests.exceptions.SSLError`, so the
+guard cannot fire there; in those skills the leak was the raw exception
+interpolated into an already-allowlisted `*ApiError`, and that is now authored
+text naming the config target and `verify_ssl` instead of the exception.
+
+The missing-password error — this family's most common first-run failure, whose
+entire remedy is the environment variable name it carries — keeps its message
+through a narrow `ConfigError(OSError)` rather than the base class. Connection
+failures are translated at the connection layer into an authored remedy that
+names the target and the setting to change, with the raw detail left on
+`__cause__` for the server log.
+
+### Also fixed
+
+- **vmware-vks**: the quickstart documented a password variable the code never
+  reads — following `README.md` verbatim produced "Password not found". Five
+  places, plus six references to a `doctor` command this CLI has never had, two
+  descriptions promising fields the tools do not return, and eight teaching
+  messages that `RuntimeError` was masking.
+- **vmware-nsx**: an error cited `--route-advertisement`; the flag is `--advertise`.
+- **vmware-pilot**: `get_workflow_status` told the model to call `approve` — a
+  tool the read-only gate withholds — as the required next step; and a hint
+  pointed at a filename that could never appear in that message.
+- **vmware-aiops**: `vm_task_status` polling a *failed task* returned
+  `{"state": "error", "error": ...}` from a successful read, which the new
+  detection read as the call itself failing. The field is now `task_error`.
+  **This is a breaking change for anything parsing that payload.**
+- Several remedies that were still being cut by the 300-character cap the 1.8.4
+  notes claimed to have addressed.
+
+### Known and not fixed
+
+`ConnectionError` remains one type from two sources in several skills — a
+skill's own authored message and urllib3's `HTTPSConnectionPool(host=..., port=...)`
+share it, and an allowlist cannot separate them. vmware-vks is converted; the
+rest need their own domain type and are deferred rather than half-done.
+
 ## v1.8.4 (2026-07-20) — errors that teach, and tool descriptions a small model can route from
 
 A capability eval was rolled out across the family and asked two open questions:

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "vmware-debug"
-version = "1.8.4"
+version = "1.8.5"
 description = "VMware diagnostic brain — read-only incident triage, log/event correlation, and root-cause routing across the VMware skill family"
 readme = "README.md"
 license = "MIT"
@@ -21,7 +21,7 @@ dependencies = [
     "typer>=0.12,<1.0",
     "rich>=13.0,<15.0",
     "mcp[cli]>=1.10,<2.0",
-    "vmware-policy>=1.8.4,<2.0",
+    "vmware-policy>=1.8.5,<2.0",
 ]
 
 [project.scripts]

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/server.json

@@ -7,12 +7,12 @@
     "url": "https://github.com/zw008/VMware-Debug",
     "source": "github"
   },
-  "version": "1.8.4",
+  "version": "1.8.5",
   "packages": [
     {
       "registryType": "pypi",
       "identifier": "vmware-debug",
-      "version": "1.8.4",
+      "version": "1.8.5",
       "transport": {
         "type": "stdio"
       }

vmware_debug-1.8.5/tests/eval/capability/_family.py

@@ -0,0 +1,204 @@
+"""What every skill in the family exposes — generated, do not edit.
+
+Written by scripts/install_capability_evals.sh, which is the only place
+that can see all twelve repos at once. Error messages and tool
+descriptions route across skills, so checking those citations needs the
+sibling surfaces; deriving them here beats a hand-kept list that goes
+stale without saying so.
+"""
+
+from __future__ import annotations
+
+FAMILY_TOOLS: dict[str, frozenset[str]] = {
+    'vmware-aiops': frozenset({
+        'acknowledge_vcenter_alarm', 'attach_iso_to_vm', 'batch_clone_vms',
+        'batch_deploy_from_spec', 'batch_linked_clone_vms', 'browse_datastore',
+        'cluster_add_host', 'cluster_configure', 'cluster_create', 'cluster_delete',
+        'cluster_health_summary', 'cluster_info', 'cluster_remove_host',
+        'convert_vm_to_template', 'cross_vcenter_attention',
+        'datastore_investigation_bundle', 'deploy_linked_clone', 'deploy_vm_from_ova',
+        'deploy_vm_from_template', 'host_investigation_bundle', 'list_vcenter_alarms',
+        'reset_vcenter_alarm', 'scan_datastore_images', 'vm_apply_plan', 'vm_cancel_ttl',
+        'vm_clean_slate', 'vm_clone', 'vm_create', 'vm_create_plan', 'vm_create_snapshot',
+        'vm_delete', 'vm_delete_snapshot', 'vm_guest_download', 'vm_guest_exec',
+        'vm_guest_exec_output', 'vm_guest_provision', 'vm_guest_upload',
+        'vm_investigation_bundle', 'vm_list_plans', 'vm_list_snapshots', 'vm_list_ttl',
+        'vm_migrate', 'vm_power_off', 'vm_power_on', 'vm_reconfigure', 'vm_revert_snapshot',
+        'vm_rollback_plan', 'vm_set_ttl', 'vm_task_status',
+    }),
+    'vmware-aria': frozenset({
+        'acknowledge_alert', 'cancel_alert', 'create_alert_definition',
+        'delete_alert_definition', 'delete_report', 'generate_report', 'get_alert',
+        'get_aria_health', 'get_capacity_overview', 'get_remaining_capacity', 'get_report',
+        'get_resource', 'get_resource_health', 'get_resource_metrics',
+        'get_resource_riskbadge', 'get_time_remaining', 'get_top_consumers',
+        'investigate_alert', 'list_alert_definitions', 'list_alerts', 'list_anomalies',
+        'list_collector_groups', 'list_report_definitions', 'list_reports',
+        'list_resources', 'list_rightsizing_recommendations', 'list_symptom_definitions',
+        'set_alert_definition_state',
+    }),
+    'vmware-avi': frozenset({
+        'ako_amko_status', 'ako_clusters', 'ako_config_diff', 'ako_config_show',
+        'ako_config_upgrade', 'ako_ingress_check', 'ako_ingress_diagnose',
+        'ako_ingress_map', 'ako_logs', 'ako_restart', 'ako_status', 'ako_sync_diff',
+        'ako_sync_force', 'ako_sync_status', 'ako_version', 'pool_list',
+        'pool_member_disable', 'pool_member_enable', 'pool_members', 'se_health', 'se_list',
+        'ssl_expiry_check', 'ssl_list', 'vs_analytics', 'vs_error_logs', 'vs_list',
+        'vs_status', 'vs_toggle',
+    }),
+    'vmware-debug': frozenset({
+        'incident_timeline', 'list_symptom_categories',
+    }),
+    'vmware-harden': frozenset({
+        'get_baseline_rules', 'get_remediation', 'list_baselines', 'list_drift_events',
+        'list_violations', 'scan_target',
+    }),
+    'vmware-log-insight': frozenset({
+        'alert_get', 'alert_history', 'alert_list', 'log_aggregate', 'log_fields',
+        'log_search', 'log_version',
+    }),
+    'vmware-monitor': frozenset({
+        'active_sessions', 'active_tasks', 'certificate_status', 'cluster_health_summary',
+        'cross_vcenter_attention', 'datastore_capacity', 'datastore_investigation_bundle',
+        'get_alarms', 'get_events', 'get_host_sensors', 'get_host_services',
+        'host_investigation_bundle', 'host_log_scan', 'host_performance', 'license_status',
+        'list_all_clusters', 'list_all_datastores', 'list_all_networks', 'list_esxi_hosts',
+        'list_virtual_machines', 'ntp_status', 'resource_pool_usage', 'snapshot_aging',
+        'vm_info', 'vm_investigation_bundle', 'vm_list_snapshots', 'vm_performance',
+    }),
+    'vmware-nsx': frozenset({
+        'configure_tier0_bgp', 'create_ip_pool', 'create_nat_rule', 'create_segment',
+        'create_static_route', 'create_tier1_gateway', 'delete_ip_pool', 'delete_nat_rule',
+        'delete_segment', 'delete_static_route', 'delete_tier1_gateway',
+        'get_bgp_neighbors', 'get_edge_cluster_status', 'get_ip_pool_usage',
+        'get_logical_port_status', 'get_nsx_manager_status', 'get_segment',
+        'get_segment_port_for_vm', 'get_tier0_gateway', 'get_tier1_gateway',
+        'get_transport_node_status', 'list_edge_clusters', 'list_ip_pools',
+        'list_nat_rules', 'list_nsx_alarms', 'list_segments', 'list_static_routes',
+        'list_tier0_gateways', 'list_tier1_gateways', 'list_transport_nodes',
+        'list_transport_zones', 'update_segment', 'update_tier1_gateway',
+    }),
+    'vmware-nsx-security': frozenset({
+        'apply_vm_tag', 'create_dfw_policy', 'create_dfw_rule', 'create_group',
+        'delete_dfw_policy', 'delete_dfw_rule', 'delete_group', 'get_dfw_policy',
+        'get_dfw_rule_stats', 'get_group', 'get_idps_status', 'get_traceflow_result',
+        'list_dfw_policies', 'list_dfw_rules', 'list_groups', 'list_idps_profiles',
+        'list_vm_tags', 'remove_vm_tag', 'run_traceflow', 'update_dfw_policy',
+        'update_dfw_rule',
+    }),
+    'vmware-pilot': frozenset({
+        'approve', 'cancel_workflow', 'confirm_draft', 'create_workflow', 'design_workflow',
+        'get_skill_catalog', 'get_workflow_status', 'list_workflows', 'plan_workflow',
+        'review_workflow', 'rollback', 'run_workflow', 'update_draft',
+    }),
+    'vmware-storage': frozenset({
+        'browse_datastore', 'list_all_datastores', 'list_cached_images',
+        'scan_datastore_images', 'storage_iscsi_add_target', 'storage_iscsi_enable',
+        'storage_iscsi_remove_target', 'storage_iscsi_status', 'storage_rescan',
+        'vsan_capacity', 'vsan_health',
+    }),
+    'vmware-vks': frozenset({
+        'check_vks_compatibility', 'create_namespace', 'create_tkc_cluster',
+        'delete_namespace', 'delete_tkc_cluster', 'get_harbor_info', 'get_namespace',
+        'get_supervisor_kubeconfig', 'get_supervisor_status', 'get_tkc_available_versions',
+        'get_tkc_cluster', 'get_tkc_kubeconfig', 'list_namespace_storage_usage',
+        'list_namespaces', 'list_supervisor_storage_policies', 'list_tkc_clusters',
+        'list_vm_classes', 'scale_tkc_cluster', 'update_namespace', 'upgrade_tkc_cluster',
+    }),
+}
+
+FAMILY_COMMANDS: dict[str, frozenset[str]] = {
+    'vmware-aiops': frozenset({
+        'alarm acknowledge', 'alarm list', 'alarm reset', 'attention', 'cluster add-host',
+        'cluster configure', 'cluster create', 'cluster delete', 'cluster info',
+        'cluster remove-host', 'daemon start', 'daemon status', 'daemon stop',
+        'datastore browse', 'datastore scan-images', 'deploy batch', 'deploy batch-clone',
+        'deploy iso', 'deploy linked-clone', 'deploy mark-template', 'deploy ova',
+        'deploy template', 'doctor', 'hub status', 'init', 'investigate datastore',
+        'investigate host', 'investigate vm', 'mcp', 'mcp-config generate',
+        'mcp-config install', 'mcp-config list', 'plan list', 'scan now', 'summary',
+        'vm cancel-ttl', 'vm clean-slate', 'vm clone', 'vm create', 'vm delete',
+        'vm guest-download', 'vm guest-exec', 'vm guest-upload', 'vm list-ttl',
+        'vm migrate', 'vm power-off', 'vm power-on', 'vm reconfigure', 'vm set-ttl',
+        'vm snapshot-create', 'vm snapshot-delete', 'vm snapshot-list',
+        'vm snapshot-revert', 'vm task-status',
+    }),
+    'vmware-aria': frozenset({
+        'alert acknowledge', 'alert cancel', 'alert definitions', 'alert get', 'alert list',
+        'anomaly list', 'anomaly risk', 'capacity overview', 'capacity remaining',
+        'capacity rightsizing', 'capacity time-remaining', 'doctor', 'health collectors',
+        'health status', 'init', 'mcp', 'report definitions', 'report delete',
+        'report generate', 'report get', 'report list', 'resource get', 'resource health',
+        'resource list', 'resource metrics', 'resource top',
+    }),
+    'vmware-avi': frozenset({
+        'ako amko-status', 'ako clusters', 'ako config-diff', 'ako config-show',
+        'ako config-upgrade', 'ako ingress-check', 'ako ingress-diagnose',
+        'ako ingress-map', 'ako logs', 'ako restart', 'ako status', 'ako sync-diff',
+        'ako sync-force', 'ako sync-status', 'ako version', 'analytics', 'config', 'doctor',
+        'init', 'logs', 'mcp', 'pool disable', 'pool enable', 'pool members', 'se health',
+        'se list', 'ssl expiry', 'ssl list', 'vs disable', 'vs enable', 'vs list',
+        'vs status',
+    }),
+    'vmware-debug': frozenset({
+        'categories', 'mcp', 'triage', 'version',
+    }),
+    'vmware-harden': frozenset({
+        'advise', 'apply', 'baseline import', 'baseline list', 'baseline validate',
+        'doctor', 'drift', 'mcp', 'report', 'scan', 'web',
+    }),
+    'vmware-log-insight': frozenset({
+        'aggregate', 'alert get', 'alert history', 'alert list', 'doctor', 'fields', 'mcp',
+        'search', 'version',
+    }),
+    'vmware-monitor': frozenset({
+        'activity sessions', 'activity tasks', 'attention', 'capacity datastores',
+        'capacity pools', 'daemon start', 'daemon status', 'daemon stop', 'doctor',
+        'health alarms', 'health events', 'health sensors', 'health services',
+        'infra certs', 'infra licenses', 'infra ntp', 'init', 'inventory clusters',
+        'inventory datastores', 'inventory hosts', 'inventory networks', 'inventory vms',
+        'investigate datastore', 'investigate host', 'investigate vm', 'mcp',
+        'mcp-config generate', 'mcp-config list', 'perf hosts', 'perf vms', 'scan now',
+        'snapshots aging', 'summary', 'vm info', 'vm snapshot-list',
+    }),
+    'vmware-nsx': frozenset({
+        'doctor', 'gateway configure-tier0-bgp', 'gateway create-tier1',
+        'gateway delete-tier1', 'gateway update-tier1', 'health alarms',
+        'health edge-cluster-status', 'health manager-status',
+        'health transport-node-status', 'init', 'inventory get-segment',
+        'inventory get-tier0', 'inventory get-tier1', 'inventory list-edge-clusters',
+        'inventory list-segments', 'inventory list-tier0s', 'inventory list-tier1s',
+        'inventory list-transport-nodes', 'inventory list-transport-zones',
+        'ip-pool create', 'ip-pool delete', 'mcp', 'mcp-config generate',
+        'mcp-config install', 'mcp-config list', 'nat create-rule', 'nat delete-rule',
+        'networking bgp-neighbors', 'networking ip-pool-usage', 'networking list-ip-pools',
+        'networking list-nat-rules', 'networking list-static-routes', 'route create-static',
+        'route delete-static', 'segment create', 'segment delete', 'segment update',
+        'troubleshoot port-status', 'troubleshoot vm-segment',
+    }),
+    'vmware-nsx-security': frozenset({
+        'doctor', 'group delete', 'group get', 'group list', 'idps profiles', 'idps status',
+        'init', 'mcp', 'policy create', 'policy delete', 'policy get', 'policy list',
+        'rule delete', 'rule list', 'rule stats', 'tag apply', 'tag list', 'tag remove',
+        'traceflow run',
+    }),
+    'vmware-pilot': frozenset({
+        'mcp', 'version',
+    }),
+    'vmware-storage': frozenset({
+        'datastore browse', 'datastore list', 'datastore scan-images', 'doctor', 'init',
+        'iscsi add-target', 'iscsi enable', 'iscsi remove-target', 'iscsi rescan',
+        'iscsi status', 'mcp', 'vsan capacity', 'vsan health',
+    }),
+    'vmware-vks': frozenset({
+        'check', 'harbor', 'init', 'kubeconfig get', 'kubeconfig supervisor', 'mcp',
+        'namespace create', 'namespace delete', 'namespace get', 'namespace list',
+        'namespace update', 'namespace vm-classes', 'preflight-auth', 'storage',
+        'supervisor status', 'supervisor storage-policies', 'tkc create', 'tkc delete',
+        'tkc get', 'tkc list', 'tkc scale', 'tkc upgrade', 'tkc versions',
+    }),
+}
+
+#: Every tool name anywhere in the family, for citation checks that do
+#: not care which skill owns the name.
+ALL_TOOLS: frozenset[str] = frozenset().union(*FAMILY_TOOLS.values())

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/tests/eval/capability/_scores.json

@@ -1,5 +1,5 @@
 {
-  "_comment": "Capability eval scores. Regenerate with: pytest -m capability. These are tracked trends, not pass/fail gates \u2014 see _scoring.py.",
+  "_comment": "Capability eval scores. Regenerate with: pytest -m capability. These are tracked trends, not pass/fail gates \u2014 see _scoring.py. Entries marked stale=true were carried over from an earlier run because this run did not measure them.",
   "scores": {
     "entity_reachability_full": {
       "detail": {

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/tests/eval/capability/_scoring.py

@@ -99,14 +99,38 @@ class ScoreBoard:
         return {s.name: s.as_dict() for s in sorted(self.records, key=lambda s: s.name)}
 
     def write(self, path: Path = SCORES_PATH) -> None:
+        """Persist this run's scores, never shrinking the recorded baseline.
+
+        A partial selection — ``pytest tests/eval/capability/test_x.py`` — used
+        to rewrite the file with only the metrics that run collected, silently
+        deleting the rest. Running one measurement across the family reduced all
+        twelve baselines from thirteen metrics to three, and one ``git add -A``
+        would have made that permanent: the next release would have had nothing
+        to diff against, which is the mixed-provenance corruption this file
+        exists to prevent.
+
+        Metrics from earlier runs are carried forward and marked ``stale`` so a
+        reader can tell a fresh measurement from an inherited one. Nothing is
+        lost, and nothing pretends to be newer than it is.
+        """
         if not self.records:
             return
+        fresh = self.as_dict()
+        merged = dict(fresh)
+        for name, value in previous_scores(path).items():
+            if name not in merged:
+                carried = dict(value) if isinstance(value, dict) else value
+                if isinstance(carried, dict):
+                    carried["stale"] = True
+                merged[name] = carried
         payload = {
             "_comment": (
                 "Capability eval scores. Regenerate with: pytest -m capability. "
-                "These are tracked trends, not pass/fail gates — see _scoring.py."
+                "These are tracked trends, not pass/fail gates — see _scoring.py. "
+                "Entries marked stale=true were carried over from an earlier run "
+                "because this run did not measure them."
             ),
-            "scores": self.as_dict(),
+            "scores": merged,
         }
         path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
 

{vmware_debug-1.8.4 → vmware_debug-1.8.5}/tests/eval/capability/test_error_actionability.py

@@ -65,6 +65,7 @@ from collections.abc import Callable
 import pytest
 
 from ._scoring import Score
+from ._family import ALL_TOOLS, FAMILY_COMMANDS
 from ._skill import CLI_NAME, COMPANION_SKILLS, PACKAGE, SERVER_MODULE
 
 pytestmark = pytest.mark.capability
@@ -213,6 +214,49 @@ def _cli_commands() -> frozenset[str] | None:
     return None
 
 
+#: A token cited in an imperative position — "run `foo_bar`", "call foo_bar".
+#: Only these are treated as claims about the surface. A snake_case word
+#: elsewhere in a sentence is usually a parameter or a response field, and
+#: flagging those would drown the real signal.
+#: ``see`` and ``use`` are excluded: "see RELEASE_NOTES.md 'Known limitations'"
+#: read as a call to a tool named ``release_notes``. The trailing guard keeps a
+#: filename from matching at all — a citation ends at the word, not at a dot.
+CITED_TOOL = re.compile(
+    r"\b(?:run|re-run|rerun|call|invoke)\s+['\"`]?([a-z][a-z0-9]*(?:_[a-z0-9]+)+)(?![.\w])"
+)
+
+
+def _citable_tool_names(tools) -> frozenset[str]:
+    """Every tool name a message in this repo may legitimately cite.
+
+    This repo's own surface plus every sibling's, because cross-skill routing is
+    the family's promoted pattern — "run vmware-monitor's ``list_esxi_hosts``"
+    is a documented hand-off, not a phantom. ``ALL_TOOLS`` is generated from all
+    twelve live registries by ``scripts/install_capability_evals.sh``.
+
+    A separate function so the union is testable on its own: driven only through
+    the live check, dropping the sibling half is invisible in any repo whose own
+    messages happen not to route outward.
+    """
+    own = frozenset(t.name.lower() for t in tools)
+    assert own, "no tools on the surface — this check would pass vacuously"
+    assert ALL_TOOLS, "_family.py holds no tools — regenerate it before trusting this"
+    return own | frozenset(n.lower() for n in ALL_TOOLS)
+
+
+def _phantom_tool_citations(text: str, known: frozenset[str]) -> list[str]:
+    """Tool-shaped names cited as callable that are not on the surface.
+
+    ``names_artifact`` is an any-of predicate: one real artifact anywhere in the
+    message earns the point. So a message reading "Run list_vms to see VMs.
+    Also check .env." scored full marks *and* steered the model at a tool that
+    does not exist — the registry rebuild closed only the case where the phantom
+    was the sole artifact. Phantom citations are a regression-class property, not
+    a trend, so they are asserted rather than scored.
+    """
+    return sorted({m.group(1) for m in CITED_TOOL.finditer(text)} - known)
+
+
 _UNSET = object()
 
 
@@ -256,44 +300,60 @@ def _artifact_matcher(tools, cli_commands=_UNSET) -> Callable[[str], bool]:
     #: tools are snake_case, so a token containing ``_`` is a tool reference,
     #: not a subcommand claim — it is checked against the registry below rather
     #: than being mistaken for a phantom command named "get".
-    cli_call = re.compile(
-        r"(?:['\"`]|\b(?:run|re-run|rerun|via|using|with)\s+['\"`]?)"
-        + re.escape(CLI_NAME.lower())
-        + r"((?:\s+[a-z][a-z0-9_-]*){1,3})"
-    )
-    cli_mention = re.compile(re.escape(CLI_NAME.lower()))
+    def _cites_cli(low: str, cli: str, known_commands) -> bool:
+        """``<cli> <subcommand>`` counts only when the subcommand is registered.
 
-    def _valid(tokens: list[str]) -> bool:
-        """True if some prefix of the cited tokens is a runnable command path.
+        Used for this skill's own CLI *and* for every companion it routes to, so
+        the two rules cannot drift — the companion path was previously exempt
+        and credited any mention outright.
 
-        A prefix, because real messages trail prose — "run 'vmware-nsx init' to
-        rebuild it" cites `init` followed by words that are not arguments. A
-        *group* on its own is not a prefix that runs, so `pool` alone never
-        satisfies this even though `pool members` does.
-        """
-        return any(" ".join(tokens[: i + 1]) in commands for i in range(len(tokens)))
+        A prefix of the cited tokens is enough, because real messages trail
+        prose: "run 'vmware-nsx init' to rebuild it" cites ``init`` followed by
+        words that are not arguments. A *group* alone is not a runnable prefix,
+        so ``pool`` never satisfies this even though ``pool members`` does.
 
-    def _cites_this_cli(low: str) -> bool:
-        if not cli_mention.search(low):
+        ``known_commands is None`` means that CLI's tree could not be resolved —
+        unverifiable, which is not the same as refuted.
+        """
+        if cli not in low:
             return False
+        pattern = re.compile(
+            r"(?:['\"`]|\b(?:run|re-run|rerun|via|using|with)\s+['\"`]?)"
+            + re.escape(cli)
+            + r"((?:\s+[a-z][a-z0-9_-]*){1,3})"
+        )
         claims = [
-            [t for t in m.group(1).split() if "_" not in t] for m in cli_call.finditer(low)
+            [t for t in m.group(1).split() if "_" not in t] for m in pattern.finditer(low)
         ]
         claims = [c for c in claims if c]
         if not claims:
             return True  # names the executable, or a tool the registry check reads
-        if commands is None:
-            return True  # app not introspectable: unverifiable, not refuted
-        return any(_valid(c) for c in claims)
+        if known_commands is None:
+            return True  # tree not introspectable: unverifiable, not refuted
+        return any(
+            any(" ".join(c[: i + 1]) in known_commands for i in range(len(c)))
+            for c in claims
+        )
+
+    def _cites_this_cli(low: str) -> bool:
+        return _cites_cli(low, CLI_NAME.lower(), commands)
 
     def names_artifact(text: str) -> bool:
         low = text.lower()
         if any(m in low for m in STATIC_ARTIFACT_MARKERS if m != CLI_NAME):
             return True
-        # A companion hand-off counts, but not when the "companion" is this
-        # skill's own name wearing a shorter prefix.
-        if any(m.group(0) != CLI_NAME.lower() for m in COMPANION_PATTERN.finditer(low)):
-            return True
+        # A companion hand-off counts — but its subcommand is verified exactly
+        # like our own. Crediting any companion mention outright reopened the
+        # phantom-command hole one hop away: "run 'vmware-storage not-a-command'"
+        # scored full marks in every repo, which is the same defect the CLI
+        # check was built for. FAMILY_COMMANDS is generated from every repo's
+        # live Typer tree by scripts/install_capability_evals.sh.
+        for match in COMPANION_PATTERN.finditer(low):
+            companion = match.group(0)
+            if companion == CLI_NAME.lower():
+                continue
+            if _cites_cli(low, companion, FAMILY_COMMANDS.get(companion)):
+                return True
         if _cites_this_cli(low):
             return True
         if FLAG_PATTERN.search(low) or ENV_VAR_PATTERN.search(text):
@@ -474,6 +534,29 @@ def test_error_actionability_index(board, graded_errors):
     )
 
 
+def test_no_error_message_cites_a_tool_that_does_not_exist(tools, graded_errors):
+    """Every tool an error tells the model to run must be on the surface.
+
+    Separate from the scored dimensions on purpose. ``names_artifact`` asks "is
+    there something to act on?" and stops at the first yes; this asks "is
+    everything it points at real?", which no aggregate can express — a message
+    can be 3/3 actionable and still strand the model, as long as it names one
+    valid artifact alongside the phantom.
+    """
+    known = _citable_tool_names(tools)
+
+    phantoms = [
+        {"where": f"{f}:{ln}", "cites": bad, "message": text[:120]}
+        for f, ln, text, _c, _g in graded_errors
+        for bad in [_phantom_tool_citations(text.lower(), known)]
+        if bad
+    ]
+    assert not phantoms, (
+        f"{len(phantoms)} error message(s) tell the model to run a tool this "
+        f"surface does not expose — a remedy it cannot carry out: {phantoms[:5]}"
+    )
+
+
 def test_teaching_error_rate(board, graded_errors):
     """Share of errors carrying *both* a remedy and something concrete to act on.
 
@@ -661,7 +744,7 @@ def test_remedies_survive_the_truncation_cap(board, graded_errors):
     )
 
 
-def _error_returns_in_server():
+def _error_returns_in_server(server_dir=None, module=None):
     """Yield ``(lineno, is_dict, has_hint, hint_text)`` for each caught-error return.
 
     Scanned statically out of the MCP server module rather than by calling a
@@ -671,8 +754,15 @@ def _error_returns_in_server():
     *exhaustively* — which is the one that drifts, since each site is written
     independently and nothing forces them to match.
     """
-    module = importlib.import_module(SERVER_MODULE)
-    server_dir = pathlib.Path(module.__file__).parent
+    # Both parameters default to this repo's live server. They exist so the
+    # regression suite can point the scan at a fabricated source: driven only
+    # against whatever the package contains, a resolution bug is invisible until
+    # some repo happens to write its handler the unhandled way — which is
+    # exactly how a whole surface went unseen once already.
+    if module is None:
+        module = importlib.import_module(SERVER_MODULE)
+    if server_dir is None:
+        server_dir = pathlib.Path(module.__file__).parent
 
     def _resolve(node) -> str:
         """Render a hint expression as text, following module-level constants.
@@ -705,7 +795,7 @@ def _error_returns_in_server():
             # payload as having none.
             if node.id in file_constants:
                 return file_constants[node.id]
-            return str(getattr(module, node.id, ""))
+            return str(getattr(module, node.id, "") if module is not None else "")
         return ""
 
     # Walk every module in the server package, not just server.py: skills split
@@ -765,10 +855,24 @@ def _error_returns_in_server():
         )
 
     for handler, file_constants in handlers:
+        # `msg = _render(...)` then `return msg` is ordinary style, and reading
+        # only the return expression made the whole surface invisible: one skill
+        # wrote its handler that way and this scan found zero error returns for
+        # a repo that has 28 tools' worth. Resolve a returned local back to what
+        # the handler assigned it.
+        locals_in_handler = {
+            t.id: n.value
+            for n in ast.walk(handler)
+            if isinstance(n, ast.Assign)
+            for t in n.targets
+            if isinstance(t, ast.Name)
+        }
         for node in ast.walk(handler):
             if not (isinstance(node, ast.Return) and node.value is not None):
                 continue
             value = node.value
+            if isinstance(value, ast.Name) and value.id in locals_in_handler:
+                value = locals_in_handler[value.id]
             folded = ""
             if isinstance(value, ast.Call):
                 fn = value.func