metergraph-cli 0.2.0-preview.4 → 0.2.0-preview.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -12
- package/assets/skill/SKILL.md +20 -1
- package/assets/skill/manifest.json +3 -3
- package/assets/skills/manifest.json +67 -0
- package/assets/skills/metergraph-analyze/SKILL.md +159 -0
- package/assets/skills/metergraph-candidates/SKILL.md +84 -0
- package/assets/skills/metergraph-evals/SKILL.md +99 -0
- package/assets/skills/metergraph-investigate/SKILL.md +105 -0
- package/assets/skills/metergraph-iterate/SKILL.md +73 -0
- package/assets/skills/metergraph-local-bundle/SKILL.md +283 -0
- package/assets/skills/metergraph-model-swap/SKILL.md +81 -0
- package/assets/skills/metergraph-onboarding/SKILL.md +156 -0
- package/assets/skills/metergraph-report/SKILL.md +91 -0
- package/assets/skills/metergraph-workloads/SKILL.md +94 -0
- package/bin/metergraph.js +4 -4
- package/package.json +3 -1
- package/src/approval-handoff.js +91 -0
- package/src/args.js +50 -11
- package/src/auth-login.js +27 -13
- package/src/cli.js +70 -7
- package/src/constants.js +9 -0
- package/src/deployment-route.js +1 -1
- package/src/output.js +138 -9
- package/src/setup-deployment.js +103 -7
- package/src/setup.js +33 -16
- package/src/skill-bundle.js +68 -1
- package/src/skill.js +175 -14
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: metergraph-investigate
|
|
3
|
+
description: Investigate a failed or anomalous LLM trace, usage spike or incident through an already connected Metergraph Agent Access server, answering only from bounded, cited evidence. Use after setup is verified; use the metergraph skill first when not connected.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Investigate with Metergraph evidence
|
|
7
|
+
|
|
8
|
+
This skill expands the bounded investigation section of the public `metergraph`
|
|
9
|
+
skill. It adds no tools, scopes or authority. When no Metergraph MCP server is
|
|
10
|
+
connected, or the user has not chosen a client, runtime and deployment, stop and
|
|
11
|
+
use the `metergraph` skill for setup routing. Never ask for, echo or store a
|
|
12
|
+
key.
|
|
13
|
+
|
|
14
|
+
## 1. Discover before reading data
|
|
15
|
+
|
|
16
|
+
1. Call `metergraph_get_workspace_context`, then `metergraph_get_capabilities`.
|
|
17
|
+
Do not call any other Metergraph tool first.
|
|
18
|
+
2. Confirm with the user that `provenance.workspace_id`, the workspace name and
|
|
19
|
+
`provenance.deployment_profile` match the installation they intend. A staging
|
|
20
|
+
or demo workspace is not their production workspace. Stop on a mismatch,
|
|
21
|
+
an authorization error, or revoked or expired access, and tell the human to
|
|
22
|
+
reconnect through their client's secure flow.
|
|
23
|
+
3. Build the tool plan only from what the server returns: the workspace
|
|
24
|
+
context's `access.scopes`, and each capability under the capabilities
|
|
25
|
+
response's `agent` object with its `available`, `privacy_class` (metadata,
|
|
26
|
+
content or replay), `required_scope`, `mutates` and `external_calls`, plus
|
|
27
|
+
`bounds` (`max_days`, `max_rows`). Treat the tools your client lists as the
|
|
28
|
+
only callable tools. A tool or capability that is absent, or has
|
|
29
|
+
`available: false`, is a blocker to report, not something to
|
|
30
|
+
guess, retry under another name or replace with a dashboard API, generic SQL
|
|
31
|
+
or a shell command. Some deployments are content-blind or have no replay,
|
|
32
|
+
incidents, ingestion health or reports.
|
|
33
|
+
|
|
34
|
+
## 2. Fix the question and the bound
|
|
35
|
+
|
|
36
|
+
Restate the question, the route or workload if known, and an explicit window in
|
|
37
|
+
whole days (tools accept `days` from 1 to 90; default to the shortest window
|
|
38
|
+
that answers the question). Then choose the smallest tool that can answer:
|
|
39
|
+
|
|
40
|
+
| Question | First tool | Bound |
|
|
41
|
+
| --- | --- | --- |
|
|
42
|
+
| How much, how many, which route costs most | `metergraph_get_usage` | `days`, small `limit` |
|
|
43
|
+
| Which routes exist or are replay eligible | `metergraph_list_routes` | none needed |
|
|
44
|
+
| Is data arriving | `metergraph_get_ingestion_health` | `days` |
|
|
45
|
+
| Known alert or anomaly | `metergraph_list_incidents` | small `limit` |
|
|
46
|
+
| One failed, slow or costly call | `metergraph_query_traces` with `status`, `route`, `trace_id` or `request_id` | `limit` 1 to 5 |
|
|
47
|
+
| A finished analysis | `metergraph_list_reports`, then `metergraph_get_report` | small `limit` |
|
|
48
|
+
|
|
49
|
+
Follow `next_cursor` only while you are still inside the agreed bound, and say
|
|
50
|
+
how many pages you read. Never request every trace, every report or unbounded
|
|
51
|
+
history.
|
|
52
|
+
|
|
53
|
+
## 3. Escalate privacy classes one step at a time
|
|
54
|
+
|
|
55
|
+
- Metadata tools answer most questions. Stay there unless the answer needs
|
|
56
|
+
captured prompts, outputs, tool calls or report evidence.
|
|
57
|
+
- `metergraph_get_trace` and `metergraph_get_report_evidence` return captured
|
|
58
|
+
content (`agent:read`). Ask the user before calling them, name the single trace
|
|
59
|
+
or workload you will read, and do not quote more captured content than the
|
|
60
|
+
answer needs. Never paste secrets you find in content.
|
|
61
|
+
- `metergraph_replay_trace` (`agent:replay`) sends a captured trace to a model
|
|
62
|
+
provider and can spend money. It needs separate, explicit approval that names
|
|
63
|
+
the trace, the overrides and the timeout. Replay is non-persistent: it does not
|
|
64
|
+
change production telemetry, prove a fix or substitute for an evaluation run.
|
|
65
|
+
|
|
66
|
+
Read access never grants evaluation writes, analysis launch, dashboard changes,
|
|
67
|
+
key rotation or production configuration changes. Refuse those requests and
|
|
68
|
+
point to the human action in the Metergraph app instead.
|
|
69
|
+
|
|
70
|
+
## 4. Answer only from returned evidence
|
|
71
|
+
|
|
72
|
+
Every answer cites, from the tool responses you actually received:
|
|
73
|
+
|
|
74
|
+
- **Workspace**: workspace name and `provenance.workspace_id`, plus
|
|
75
|
+
`deployment_profile`.
|
|
76
|
+
- **Window**: `window.since` to `window.until` (or `days`) for each read.
|
|
77
|
+
- **Provenance**: the tool name, `provenance.source` and `generated_at`.
|
|
78
|
+
- **Evidence and completeness**: `evidence.rows`, `evidence.complete`,
|
|
79
|
+
`page.truncated` and whether a `next_cursor` was left unread.
|
|
80
|
+
- **Warnings**: every returned `warnings[].code` with its message. Say "no
|
|
81
|
+
warnings returned" when the list is empty.
|
|
82
|
+
- **Identifiers and links**: trace, request, incident or analysis IDs, and
|
|
83
|
+
`metergraph_links` values exactly as returned. Never construct a link.
|
|
84
|
+
|
|
85
|
+
Keep these states distinct and name the one that applies: no data in the
|
|
86
|
+
window, incomplete or truncated evidence, capability unavailable in this
|
|
87
|
+
deployment, classification pending, ambiguous request match
|
|
88
|
+
(`request_match.ambiguous`), failed analysis, and missing report. An empty page
|
|
89
|
+
does not prove there was no traffic; check ingestion health or a longer bounded
|
|
90
|
+
window before saying so. Do not round, extrapolate or invent totals.
|
|
91
|
+
|
|
92
|
+
## 5. Worked workflow: one failed trace
|
|
93
|
+
|
|
94
|
+
1. Discover: context, then capabilities. Confirm the workspace with the user.
|
|
95
|
+
2. Find: `metergraph_query_traces` with `status` set to the failure status, the
|
|
96
|
+
agreed `days` and `limit` 1. If the user gave a request ID and the match is
|
|
97
|
+
ambiguous, list the candidates and ask which one.
|
|
98
|
+
3. Explain from metadata: route, model, provider, tokens, cost, latency, status
|
|
99
|
+
and span count, with the citations above. If metadata is enough, stop here.
|
|
100
|
+
4. Optional content: with consent, `metergraph_get_trace` for that one trace to
|
|
101
|
+
read the error, tool calls and root-cause classification.
|
|
102
|
+
5. Optional replay: only with explicit approval and when the trace is replay
|
|
103
|
+
eligible, run one bounded `metergraph_replay_trace`, report the returned
|
|
104
|
+
status and comparison, and state that nothing in production changed.
|
|
105
|
+
6. Close with what is known, what is unknown, and the next bounded check.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: metergraph-iterate
|
|
3
|
+
description: Refine and rerun a Metergraph model-swap analysis after the person has inspected the report in the app - turn their findings into a specific change (eval revision, cases, candidates, traffic or route settings), start a new approved run linked to the previous one, and compare the two honestly. Use when someone returns with "the grader is wrong on these cases", "drop that model", "rerun it", "what changed since last run", or stage 6 of the metergraph-model-swap loop.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Refine and rerun
|
|
7
|
+
|
|
8
|
+
A rerun is a new run with one deliberate change. The earlier run, report and
|
|
9
|
+
eval revision stay as they are.
|
|
10
|
+
|
|
11
|
+
## 0. Confirm the connection
|
|
12
|
+
|
|
13
|
+
If this conversation has not already confirmed the workspace, call
|
|
14
|
+
`metergraph_get_workspace_context`, then `metergraph_get_capabilities`, before
|
|
15
|
+
anything else. Say which workspace you are connected to in one line, and plan
|
|
16
|
+
only with the tools the capabilities advertise.
|
|
17
|
+
|
|
18
|
+
## 1. Recover where the loop is
|
|
19
|
+
|
|
20
|
+
1. Read the loop record (in the conversation or `.metergraph/model-swap.md`). If
|
|
21
|
+
there is none, `metergraph_list_analysis_runs` and `metergraph_get_report`
|
|
22
|
+
on the latest run for this workload, and rebuild it.
|
|
23
|
+
2. Confirm the workload's `selection_reference` still exists with
|
|
24
|
+
`metergraph_list_classified_workloads`. A newer classification may have
|
|
25
|
+
replaced it; say so and pick the matching workload by its traces, not by its
|
|
26
|
+
name alone.
|
|
27
|
+
|
|
28
|
+
## 2. Turn the finding into one change
|
|
29
|
+
|
|
30
|
+
Ask what the person saw in the app, then map it:
|
|
31
|
+
|
|
32
|
+
| Finding | Change | Skill |
|
|
33
|
+
| --- | --- | --- |
|
|
34
|
+
| A grade is wrong on some cases | Fix the check or rubric criterion; save a new eval revision; add the cases as reference cases | `metergraph-evals` |
|
|
35
|
+
| The bar is missing something important | Add a check or criterion, test it on the preview, save a new revision | `metergraph-evals` |
|
|
36
|
+
| A candidate is clearly out, or one is missing | Change candidate selection on the Opportunities page | `metergraph-candidates` |
|
|
37
|
+
| Too few samples, or `insufficient_samples` | More production traffic for this workload, then rerun | `metergraph-workloads` |
|
|
38
|
+
| Traces were skipped | Replay opt-out or residency on a route; the person changes it in the app | `metergraph-candidates` |
|
|
39
|
+
| The winner looks right | No rerun. Plan the code change and a canary instead | — |
|
|
40
|
+
|
|
41
|
+
Change one thing per rerun when you can, so the comparison means something.
|
|
42
|
+
If several must change, list them all in the confirmation.
|
|
43
|
+
|
|
44
|
+
## 3. Rerun only with a new yes
|
|
45
|
+
|
|
46
|
+
A rerun spends money and sends captured prompts to providers again. Get the
|
|
47
|
+
same explicit confirmation as the first run (see `metergraph-analyze`), plus
|
|
48
|
+
what changed. If the request already names the change and approves a budget and
|
|
49
|
+
time, that is the confirmation: restate it, make the change (for an eval fix,
|
|
50
|
+
save the new revision first, as in `metergraph-evals`), then start. Then call `metergraph_start_analysis` with:
|
|
51
|
+
|
|
52
|
+
- a **new** `request_key`, for example `rerun-<pattern_id>-<yyyymmdd>-<n>`;
|
|
53
|
+
- the same `pinned_workload`, unless the workload itself changed;
|
|
54
|
+
- `rerun_of_run_id` set to the previous run ID;
|
|
55
|
+
- `reference_binding_id` when reference cases were bound.
|
|
56
|
+
|
|
57
|
+
Poll and read the report as in `metergraph-analyze` and `metergraph-report`.
|
|
58
|
+
|
|
59
|
+
## 4. Compare honestly
|
|
60
|
+
|
|
61
|
+
Show the two runs side by side only for what is comparable:
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
run <a> (<date>) run <b> (<date>)
|
|
65
|
+
eval <revision a> <revision b>
|
|
66
|
+
samples <n> <n>
|
|
67
|
+
<candidate> <grade, cost ratio> <grade, cost ratio>
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
If the eval revision, the sample or the candidate set changed, say that a
|
|
71
|
+
different grade may come from the change, not from the model. Then give the
|
|
72
|
+
decision line from `metergraph-report` with the new report link, and update the
|
|
73
|
+
loop record.
|
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: metergraph-local-bundle
|
|
3
|
+
description: Verify and start the signed Metergraph customer-local bundle on a person's own machine, preserve existing local data and secrets, then hand over to metergraph setup. Use when customer-local setup reports that the bundle_started_verified prerequisite is incomplete or when installing the commercial local bundle from a public release.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Start the signed customer-local bundle
|
|
7
|
+
|
|
8
|
+
Stand up the released bundle only on the person's own machine, then run
|
|
9
|
+
`metergraph setup`. The CLI does not provision it, and
|
|
10
|
+
`--confirm-prerequisites` records confirmation rather than installing anything.
|
|
11
|
+
|
|
12
|
+
Keep registry, admin and database secrets out of chat, command arguments and
|
|
13
|
+
output. Never read or print `.env` values, `~/.docker/config.json`,
|
|
14
|
+
`docker compose config` output or container environment from `docker inspect`.
|
|
15
|
+
Do not configure model-provider keys.
|
|
16
|
+
|
|
17
|
+
## Stop in remote runtimes
|
|
18
|
+
|
|
19
|
+
Before changing anything, determine where the agent executes. Check only
|
|
20
|
+
whether these variables exist, never their values:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
for name in SSH_CONNECTION SSH_CLIENT SSH_TTY CODESPACES GITPOD_WORKSPACE_ID CLOUD_SHELL CI; do
|
|
24
|
+
if printenv "$name" >/dev/null 2>&1; then printf '%s is set\n' "$name"; fi
|
|
25
|
+
done
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
If any is set, or this is otherwise a hosted/cloud runtime, download, verify,
|
|
29
|
+
extract and start nothing. Prefer an installed `metergraph` whose `--help`
|
|
30
|
+
lists `setup`; otherwise state that the fallback fetches the preview CLI and
|
|
31
|
+
use `npx --yes metergraph-cli@next` as the prefix. Run only:
|
|
32
|
+
|
|
33
|
+
```sh
|
|
34
|
+
metergraph setup --runtime cloud --deployment customer-local --url http://localhost:8080 --workspace 11111111-1111-4111-8111-111111111111 --skip-skill --json
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Relay its `data.next_action.kind: "run_on_customer_machine"` handoff and tell
|
|
38
|
+
the person to run this skill on their own machine. The placeholder workspace is
|
|
39
|
+
intentional: the runtime check refuses before making a request or writing a
|
|
40
|
+
file. When an SSH, cloud or CI marker is set, even `--runtime local` is refused
|
|
41
|
+
with reason `run_on_local_machine` and the same handoff. Valid runtimes are
|
|
42
|
+
`local`, `cloud` and `cloud-no-shell`. Never expose localhost to the remote
|
|
43
|
+
session.
|
|
44
|
+
|
|
45
|
+
## Preflight and reuse
|
|
46
|
+
|
|
47
|
+
Ask for the coding client and bundle version. Default to the newest version at
|
|
48
|
+
https://www.metergraph.dev/docs/releases/ and record its `v`-prefixed value.
|
|
49
|
+
Record the application repository directory to return to for setup. Run:
|
|
50
|
+
|
|
51
|
+
```sh
|
|
52
|
+
docker info >/dev/null
|
|
53
|
+
docker compose version
|
|
54
|
+
cosign version
|
|
55
|
+
curl -fsS http://localhost:8080/v1/deployment
|
|
56
|
+
lsof -nP -iTCP:8080 -sTCP:LISTEN
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Docker with Compose must be running and `cosign` installed. If the deployment
|
|
60
|
+
endpoint returns the `local` profile, reuse that stack and download nothing.
|
|
61
|
+
Find its bundle directory using only the Compose working-directory label:
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
container_id="$(docker ps --filter publish=8080 --format '{{.ID}}' | head -n 1)"
|
|
65
|
+
bundle_dir="$(docker inspect --format '{{ index .Config.Labels "com.docker.compose.project.working_dir" }}' "$container_id")"
|
|
66
|
+
(cd "$bundle_dir" && ./bin/status)
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Do not inspect container environment. Read the workspace ID from the
|
|
70
|
+
`workspace` check and skip to **Run setup**. If the directory cannot be found,
|
|
71
|
+
ask for the workspace ID from Settings. If nothing listens on 8080, continue.
|
|
72
|
+
If anything else owns it, stop and ask; never stop or reconfigure it.
|
|
73
|
+
|
|
74
|
+
Use `~/metergraph/<version>` and set
|
|
75
|
+
`bundle_dir="$HOME/metergraph/$version/local"`. If it contains `.env`, reuse
|
|
76
|
+
that existing installation: never overwrite, delete or read the file, and
|
|
77
|
+
never extract over `local/`. Continue at **Log in and start**. Otherwise check:
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
docker volume inspect local_pgdata >/dev/null 2>&1
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Because the extracted directory and Compose project are `local`, an existing
|
|
84
|
+
volume is `local_pgdata`. If it exists without the matching `local/.env`, stop
|
|
85
|
+
and ask for the prior bundle/configuration. Postgres fixed its password when
|
|
86
|
+
the volume was created; a new `.env` can cause `password authentication failed`.
|
|
87
|
+
Never remove the volume, delete `.env` or use `down --volumes`.
|
|
88
|
+
|
|
89
|
+
## Download and verify
|
|
90
|
+
|
|
91
|
+
Require a fresh, empty version directory. Before creating it, tell the person
|
|
92
|
+
the directory and four public files that will be written. No account is needed.
|
|
93
|
+
Set `version` to the recorded version and run:
|
|
94
|
+
|
|
95
|
+
```sh
|
|
96
|
+
release_dir="$HOME/metergraph/$version"; mkdir -p "$release_dir"; cd "$release_dir"
|
|
97
|
+
archive="metergraph-customer-local-$version.tar.gz"
|
|
98
|
+
base="https://www.metergraph.dev/releases/$version"
|
|
99
|
+
curl -fsSLO "$base/$archive"
|
|
100
|
+
curl -fsSLO "$base/metergraph-byoc-release.json"
|
|
101
|
+
curl -fsSLO "$base/metergraph-byoc-release.json.sig"
|
|
102
|
+
curl -fsSLO "$base/metergraph-byoc-release.json.pem"
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Do not list, extract or run archive contents. Verify the manifest signature
|
|
106
|
+
with exactly this command. The issuer and identity are pinned here, not read
|
|
107
|
+
from the release page, so a changed download site cannot change who must have
|
|
108
|
+
signed the release:
|
|
109
|
+
|
|
110
|
+
```sh
|
|
111
|
+
cosign verify-blob \
|
|
112
|
+
--certificate metergraph-byoc-release.json.pem \
|
|
113
|
+
--signature metergraph-byoc-release.json.sig \
|
|
114
|
+
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
|
|
115
|
+
--certificate-identity-regexp '^https://github\.com/PioneerSquareLabs/metergraph-internal/\.github/workflows/byoc-release\.yml@refs/(heads/main|tags/v.+)$' \
|
|
116
|
+
metergraph-byoc-release.json
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Never loosen or drop the issuer or identity, and never add
|
|
120
|
+
`--insecure-ignore-tlog` or a similar bypass. If the release page publishes a
|
|
121
|
+
different identity, stop and tell the person rather than using it.
|
|
122
|
+
|
|
123
|
+
If cosign fails, stop: "The manifest signature did not verify. Do not extract
|
|
124
|
+
or run this bundle." After it succeeds, check signed fields:
|
|
125
|
+
|
|
126
|
+
```sh
|
|
127
|
+
python3 - "$version" "$archive" <<'EOF'
|
|
128
|
+
import json, sys
|
|
129
|
+
version, archive = sys.argv[1:]
|
|
130
|
+
m = json.load(open("metergraph-byoc-release.json"))
|
|
131
|
+
required = ("release", "local_bundle", "local_bundle_sha256",
|
|
132
|
+
"local_image_digest", "local_analysis_image_digest")
|
|
133
|
+
missing = [key for key in required if not m.get(key)]
|
|
134
|
+
if missing: sys.exit("Signed manifest is missing: " + ", ".join(missing))
|
|
135
|
+
if m["release"] != version: sys.exit("Signed manifest release mismatch. Do not run it.")
|
|
136
|
+
if m["local_bundle"] != archive: sys.exit("Signed archive name mismatch. Do not run it.")
|
|
137
|
+
print("Signed manifest release, archive, and image digests are present.")
|
|
138
|
+
EOF
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Then run this digest check, still before any `tar` command:
|
|
142
|
+
|
|
143
|
+
```sh
|
|
144
|
+
python3 - <<'EOF'
|
|
145
|
+
import hashlib, json, sys
|
|
146
|
+
manifest = json.load(open("metergraph-byoc-release.json"))
|
|
147
|
+
name = manifest["local_bundle"]
|
|
148
|
+
actual = hashlib.sha256(open(name, "rb").read()).hexdigest()
|
|
149
|
+
if actual != manifest["local_bundle_sha256"]:
|
|
150
|
+
sys.exit(f"{name} does not match the signed manifest. Do not run it.")
|
|
151
|
+
print(f"{name} matches the signed manifest.")
|
|
152
|
+
EOF
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Stop on failure. After both checks pass, list and validate archive paths:
|
|
156
|
+
|
|
157
|
+
```sh
|
|
158
|
+
tar -tzf "$archive"
|
|
159
|
+
python3 - "$archive" <<'EOF'
|
|
160
|
+
from pathlib import PurePosixPath
|
|
161
|
+
import sys, tarfile
|
|
162
|
+
bad = []
|
|
163
|
+
with tarfile.open(sys.argv[1], "r:gz") as bundle:
|
|
164
|
+
for member in bundle.getmembers():
|
|
165
|
+
path = PurePosixPath(member.name)
|
|
166
|
+
if path.is_absolute() or not path.parts or path.parts[0] != "local" or ".." in path.parts:
|
|
167
|
+
bad.append(member.name)
|
|
168
|
+
if bad: sys.exit("Unsafe archive paths; do not extract: " + ", ".join(bad[:10]))
|
|
169
|
+
print("Every archive entry is beneath local/ with a safe relative path.")
|
|
170
|
+
EOF
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Only now tell the person extraction will create `local/`, then run:
|
|
174
|
+
|
|
175
|
+
```sh
|
|
176
|
+
tar xzf "$archive"
|
|
177
|
+
mv metergraph-byoc-release.json* local/
|
|
178
|
+
cd local
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
## Configure without exposing passwords
|
|
182
|
+
|
|
183
|
+
Reuse an existing `.env` unchanged. For a new installation, state that a
|
|
184
|
+
mode-0600 file with random admin/database passwords and empty provider keys
|
|
185
|
+
will be created. Confirm `local_pgdata` did not exist, then run:
|
|
186
|
+
|
|
187
|
+
```sh
|
|
188
|
+
umask 077; cp .env.example .env; chmod 600 .env
|
|
189
|
+
python3 - <<'EOF'
|
|
190
|
+
from pathlib import Path
|
|
191
|
+
import secrets
|
|
192
|
+
path = Path(".env")
|
|
193
|
+
keys = ("METERGRAPH_LOCAL_ADMIN_PASSWORD", "METERGRAPH_DB_PASSWORD")
|
|
194
|
+
values = {key: secrets.token_urlsafe(24) for key in keys}
|
|
195
|
+
lines, found = path.read_text().splitlines(), set()
|
|
196
|
+
for i, line in enumerate(lines):
|
|
197
|
+
for key in keys:
|
|
198
|
+
if line.startswith(key + "="):
|
|
199
|
+
lines[i], found = key + "=" + values[key], found | {key}
|
|
200
|
+
if found != set(keys): raise SystemExit("Password keys missing from .env.example")
|
|
201
|
+
path.write_text("\n".join(lines) + "\n")
|
|
202
|
+
print("Set random values for: " + ", ".join(keys))
|
|
203
|
+
EOF
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Do not display, source or parse `.env`. Leave `AI_GATEWAY_API_KEY`,
|
|
207
|
+
`FIREWORKS_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` and `GOOGLE_API_KEY`
|
|
208
|
+
empty.
|
|
209
|
+
|
|
210
|
+
## Log in and start
|
|
211
|
+
|
|
212
|
+
Stop for the person to log in to the registry in their own terminal. They use
|
|
213
|
+
the pull-only invitation values, never chat or an agent-visible command:
|
|
214
|
+
|
|
215
|
+
```sh
|
|
216
|
+
echo "$METERGRAPH_PULL_TOKEN" | docker login ghcr.io -u "$METERGRAPH_PULL_USER" --password-stdin
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Wait for confirmation only; never read Docker's credential file. State that
|
|
220
|
+
`./bin/start` will re-verify the manifest and both image digests, pull by digest,
|
|
221
|
+
start containers, and on a new database send one synthetic demo trace labeled
|
|
222
|
+
`environment=local-demo`. Then run:
|
|
223
|
+
|
|
224
|
+
```sh
|
|
225
|
+
./bin/start
|
|
226
|
+
./bin/status
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
An image-pull failure usually means login is missing or expired. On other
|
|
230
|
+
failure, read only the log named by `./bin/start` and `./bin/status`; never read
|
|
231
|
+
rendered Compose configuration or container environment. On password
|
|
232
|
+
authentication failure with `local_pgdata`, stop and ask for its matching
|
|
233
|
+
`.env`. Never delete data. Read the workspace ID and suggested command from the
|
|
234
|
+
status JSON's `workspace` check. The service listens only on localhost:8080.
|
|
235
|
+
|
|
236
|
+
## Run setup
|
|
237
|
+
|
|
238
|
+
Return to the application repository. Prefer installed `metergraph` only when
|
|
239
|
+
its help lists `setup`; otherwise state that `npx --yes metergraph-cli@next`
|
|
240
|
+
fetches the preview CLI and use it as the prefix. Run with `--json`:
|
|
241
|
+
|
|
242
|
+
```sh
|
|
243
|
+
metergraph setup --runtime local --deployment customer-local --url http://localhost:8080 --workspace 11111111-1111-4111-8111-111111111111 --confirm-prerequisites --client <client> --json
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
Replace the example UUID and client only. Setup writes an ingest key to the
|
|
247
|
+
application `.env`, installs the skill and requests Metadata access only; never
|
|
248
|
+
print that file. Setup opens the deployment's browser consent page and waits;
|
|
249
|
+
the person signs in as the local admin and completes two approvals. If the
|
|
250
|
+
browser cannot open, follow the `metergraph` skill's guidance for handing the
|
|
251
|
+
person the approval URL. Continue until `data.status` is `ok`. For
|
|
252
|
+
`operator_handoff`, relay `data.next_action.kind` (`complete_prerequisite`,
|
|
253
|
+
`run_on_customer_machine`, `connection_guide`, or other); never report a
|
|
254
|
+
handoff as success.
|
|
255
|
+
|
|
256
|
+
The agent never runs a command that reads `.env`. To see the admin login without
|
|
257
|
+
printing it, the person runs these no-output pipelines themselves, one at a
|
|
258
|
+
time, in their own terminal or in Claude Code with the `!` prefix. Fill in the
|
|
259
|
+
absolute bundle directory for them; their shell does not have `bundle_dir`:
|
|
260
|
+
|
|
261
|
+
```sh
|
|
262
|
+
sed -n 's/^METERGRAPH_LOCAL_ADMIN_EMAIL=//p' ~/metergraph/<version>/local/.env | pbcopy
|
|
263
|
+
sed -n 's/^METERGRAPH_LOCAL_ADMIN_PASSWORD=//p' ~/metergraph/<version>/local/.env | pbcopy
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
They may instead use their own editor. Never use `cat`, `grep` or any other
|
|
267
|
+
command that prints `.env`, particularly with `!`, because `!` output lands in
|
|
268
|
+
the transcript. A new configuration keeps `admin@example.com` as the email.
|
|
269
|
+
|
|
270
|
+
## Recovery and stopping
|
|
271
|
+
|
|
272
|
+
| State | Action |
|
|
273
|
+
| --- | --- |
|
|
274
|
+
| Signature, manifest, digest or path check fails | Stop; extract and run nothing |
|
|
275
|
+
| Port 8080 belongs to something else | Stop and ask; do not alter it |
|
|
276
|
+
| Registry pull is denied | Person repeats private login; never request the token |
|
|
277
|
+
| Existing `.env` | Reuse unchanged; never regenerate passwords |
|
|
278
|
+
| `local_pgdata` lacks its matching `.env` | Stop and find prior configuration; never remove it |
|
|
279
|
+
| Startup fails | Use status and only the named log; preserve secrets and data |
|
|
280
|
+
| Setup returns a handoff | Relay exact status/next action; do not report success |
|
|
281
|
+
|
|
282
|
+
To stop later, follow the bundle README's `stop`. Prefer it over `down`, which
|
|
283
|
+
loses raw payloads in local object storage. Never use `down --volumes`.
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: metergraph-model-swap
|
|
3
|
+
description: Run the Metergraph model-swap loop from a coding agent - choose a workload, choose traces and candidate models, define or refine the eval, start the analysis, summarize the report with its app link, then refine and rerun. Use when someone asks whether a workload could use a cheaper or better model, asks to plan or run a model swap analysis, asks "what should my largest workload run on", or comes back after inspecting a Metergraph report. Use the metergraph skill first when Metergraph is not connected.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Model-swap loop
|
|
7
|
+
|
|
8
|
+
This skill is the map. Each stage has its own skill; load it when you reach the
|
|
9
|
+
stage. The person stays in the coding agent for choosing, defining and
|
|
10
|
+
rerunning, and goes to the Metergraph app to read the detailed report and
|
|
11
|
+
inspect evidence.
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
choose workload → choose traces and models → define or refine eval
|
|
15
|
+
→ run analysis → summary + report link → (person inspects evidence in the app)
|
|
16
|
+
→ refine and rerun ↺
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
| Stage | Skill | Ends with |
|
|
20
|
+
| --- | --- | --- |
|
|
21
|
+
| 1. Choose workload | `metergraph-workloads` | One classified workload, with its `selection_reference` and why it was picked |
|
|
22
|
+
| 2. Choose traces and models | `metergraph-candidates` | A run plan: environment, window, candidate models, route settings that must change |
|
|
23
|
+
| 3. Define or refine the eval | `metergraph-evals` | A saved eval revision bound to the workload, or a written rubric the person enters in the app |
|
|
24
|
+
| 4. Run the analysis | `metergraph-analyze` | A run ID, polled to a final state |
|
|
25
|
+
| 5. Summarize | `metergraph-report` | A short summary and the report link |
|
|
26
|
+
| 6. Refine and rerun | `metergraph-iterate` | What changed and a new, separately approved run |
|
|
27
|
+
|
|
28
|
+
## Start every session the same way
|
|
29
|
+
|
|
30
|
+
1. Call `metergraph_get_workspace_context`, then `metergraph_get_capabilities`.
|
|
31
|
+
Say which workspace and deployment you are connected to in one line. Stop if
|
|
32
|
+
it is not the one the person means.
|
|
33
|
+
2. Call `metergraph_list_analysis_runs` with a small limit. If a run already
|
|
34
|
+
exists for this workload, the person is probably on stage 6: read its report
|
|
35
|
+
first instead of starting over, and give the report link
|
|
36
|
+
(`<origin>/#analysis?report=<analysis_id>`, as in `metergraph-report`).
|
|
37
|
+
3. Write down the tools that are advertised. Plan only with those. When a stage
|
|
38
|
+
needs a tool that is not advertised, say which one, keep the selection you
|
|
39
|
+
have, and hand that step to the app page named in the stage skill. Never
|
|
40
|
+
invent a tool, call a dashboard API with an agent key, or drive the app in a
|
|
41
|
+
browser unless the person asks for that specific change.
|
|
42
|
+
|
|
43
|
+
## Rules that hold across stages
|
|
44
|
+
|
|
45
|
+
- **Workloads are not routes.** Routes are labels on traffic. An analysis runs
|
|
46
|
+
on a classified workload, identified by `source_run_id`, `pattern_id` and
|
|
47
|
+
`pattern_set_version`. Do not rank or plan on routes and then discover this at
|
|
48
|
+
the start step.
|
|
49
|
+
- **Check the data before planning.** A workspace created today, data that all
|
|
50
|
+
arrived in one batch, fewer than 10 samples, or totals that disagree between
|
|
51
|
+
tools and the app all change the plan. Say so first, with the numbers.
|
|
52
|
+
- **One decision at a time.** End each turn with the single decision you need
|
|
53
|
+
from the person, not a list of everything that could be configured.
|
|
54
|
+
- **Spending and data egress need a yes.** Starting an analysis sends copies of
|
|
55
|
+
captured prompts to other model providers and costs money. Stage 4 has the
|
|
56
|
+
exact confirmation to get. A connected key, a configured provider key or an
|
|
57
|
+
earlier yes for a different run is not approval.
|
|
58
|
+
- **Settings are the person's.** Replay opt-out, data residency, candidate
|
|
59
|
+
models, cost and latency limits and the size floor are changed in the app.
|
|
60
|
+
Explain the change and give the page; do not make it for them.
|
|
61
|
+
- **Keep answers short.** Lead with the answer or the blocker. Tables only for
|
|
62
|
+
three or more rows. Give confidence once, where it matters.
|
|
63
|
+
|
|
64
|
+
## Keep a loop record
|
|
65
|
+
|
|
66
|
+
Keep these values in the conversation, and, if you can write files in the
|
|
67
|
+
person's project and they agree, in `.metergraph/model-swap.md`. They are not
|
|
68
|
+
secret. They let a later session pick up at stage 6.
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
workspace: <name> (<workspace_id>) at <origin>
|
|
72
|
+
workload: <display_name> source_run_id=<…> pattern_id=<…> pattern_set_version=<…>
|
|
73
|
+
environment: <production|…> window: <from>–<to>
|
|
74
|
+
candidates: <model ids>
|
|
75
|
+
eval: <suite or rubric revision id, or "rubric in chat, not saved">
|
|
76
|
+
runs: <run_id> (<status>) → report <analysis_id> <link>
|
|
77
|
+
decision: <switch | no switch | more testing> because <reason>
|
|
78
|
+
next: <the one next step>
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Never put keys, `.env` values or captured prompt or response text in it.
|