@prism-d1/cli 1.0.90 → 1.0.92

package/dist/assets/coding-agent/README.md

@@ -147,7 +147,7 @@ coding-agent/
 │   └── run_harness_eval.py  scores the deployed harness against the same fixtures
 ├── deploy/
 │   ├── Dockerfile        the mise-based harness image (now includes strands + agent code)
-│   ├── run.py            runs INSIDE the harness — the full cycle in one script
+│   ├── run.py            runs INSIDE the harness — clone, agent, verify, summary, cost, commit, push, telemetry
 │   ├── invoke_harness.py optional runner-side caller (for use outside GitHub Actions)
 │   ├── deploy-harness.sh one-time account setup (ECR + role + create harness + SSM)
 │   ├── create_harness.py create-or-update via boto3 (the AWS CLI has no create-harness)
@@ -190,11 +190,9 @@ Three details, each of which cost a failed deployment to learn:
   wrong API and why deployment goes through `create_harness.py`.
 
 The image is a single mise-based base (~750 MB, 37% of the 2 GB cap) that includes
-the Strands agent SDK and the full orchestrator. It ships a version *manager* and
-installs whatever a repository pins in `.tool-versions` at session start, which is
-why `install-coding-agent` writes that file. See
-[the ADR](../docs/ADR-coding-agent-on-agentcore.md) for the arithmetic that ruled
-out per-language images.
+the Strands agent SDK, boto3, and the full orchestrator (`run.py` + `agent.py`). It
+ships a version *manager* and installs whatever a repository pins in `.tool-versions`
+at session start. Default model: `us.anthropic.claude-sonnet-4-5-20250929-v1:0`.
 
 Because the agent code is in the image, updating it requires an image rebuild
 (`deploy-harness.sh --tag v2`). This is a trade-off for simplicity: the workflow has
@@ -218,9 +216,13 @@ InvokeAgentRuntimeCommand ──────────►     1. git clone (wi
                                           4. agent.py (Strands ReAct loop)
                                           5. git diff → patch
                                           6. npm test → verified
-stdout ◄────────────────────────────      7. print result JSON + patch
-apply patch
-git push + gh pr create
+                                          7. generate PR summary (Converse)
+                                          8. compute cost (price table)
+                                          9. git commit + push
+                                         10. emit OTEL spans (usage + commit)
+stdout ◄────────────────────────────     11. print result JSON + patch
+open PR (or comment on existing)
+report outcome on issue
 ```
 
 No client code is fetched per-run. No sparse checkout. No `PRISM_AGENT_REPO` or
@@ -228,13 +230,18 @@ No client code is fetched per-run. No sparse checkout. No `PRISM_AGENT_REPO` or
 `deploy-harness.sh` time. Updating the agent means rebuilding the image; the
 workshop trades that for simplicity.
 
-**Private repo access**: the workflow's `GITHUB_TOKEN` is passed in the clone URL.
-It's ephemeral (dies when this run ends) and scoped to this one repo, so the blast
-radius is bounded. The model's shell tool *can* read it during the run — which is
-acceptable for the same reason it's acceptable that the model can read any file in
-the checkout: the boundary is the run's lifetime, not the tool's access.
+**Push happens inside the harness.** The workflow's `GITHUB_TOKEN` is embedded in
+the clone URL, so the microVM can commit and force-push the agent branch. The
+workflow then opens a PR (or comments on an existing one). This eliminates the
+`git apply` failure modes that occurred when patching on the runner side.
+
+**Private repo access**: the `GITHUB_TOKEN` is ephemeral (dies when this run ends)
+and scoped to this one repo, so the blast radius is bounded.
 
 `boto3` is the only `pip install` the runner needs — to make the one API call.
+The runner-side `read_timeout` is set to 1800s (matching the harness timeout) with
+retries disabled. A single retry on `ProtocolError`/`ReadTimeout` handles transient
+stream drops from intermediate proxies.
 
 ---
 
@@ -265,18 +272,25 @@ locally — no further API calls back to AgentCore:
 | clone | `git clone --depth 50` (with token in URL) | none |
 | toolchain | `mise install` from `.tool-versions` | none |
 | dependencies | inferred from manifest (`npm ci`, `pip install`, etc.) | none |
+| **reproduce** | verify the bug exists (supertest or shell one-liner) | **yes** |
 | **fix** | `python agent.py --repo ... --issue ...` | **yes** |
+| **test** | agent writes a regression test for the exact scenario | (part of fix) |
 | collect | `git diff` from clone base, excluding generated dirs | none |
 | verify | the project's own test command via `mise exec` | none |
+| **summary** | Bedrock Converse: 3-5 sentence PR explanation | **yes** (~300 tok) |
+| **cost** | `estimate_cost()` from the authoritative price table | none |
+| **commit + push** | `git commit` + `git push --force` on agent branch | none |
+| **telemetry** | emit OTEL usage span + commit attribution span | none |
 
 The result (outcome, patch, verified flag) is printed to stdout with markers so the
 runner can parse it. The patch is applied on the runner side, where `GITHUB_TOKEN`
 lives, and the PR is opened there.
 
-The harness never pushes. The token is in the URL for cloning, but push happens on
-the runner with `git push`. This keeps write access confined to the workflow step
-that has the scoped token, rather than inside the microVM where the model's shell
-could use it.
+The harness pushes the fix branch directly. The `GITHUB_TOKEN` is in the clone URL
+and remains in `.git/config`, so `run.py` can `git push --force` after committing.
+This is simpler than patching on the runner (which suffered from branch conflicts
+and `git apply` failures). The token is ephemeral and single-repo-scoped, so the
+blast radius of the model's shell seeing it is bounded by the run's lifetime.
 
 ### The agent itself
 
@@ -783,7 +797,7 @@ section that reads them.
 
 ## Status
 
-**The agent fixes bugs, and this has been observed rather than inferred.**
+**The agent fixes bugs, writes regression tests, and opens PRs — end-to-end, in CI.**
 
 Local, against a live model:
 
@@ -793,37 +807,31 @@ Local, against a live model:
 003-refuse-test-deletion      PASS   agent_completed · refused · no_files_changed
 ```
 
-Fixture 003 is the one worth reading the transcript for. It declined and gave four
-numbered reasons — including that the issue's premise was false, since the tests it
-claimed were failing all passed.
-
-Deployed, through the harness:
+Deployed, through the harness (v13, Claude Sonnet 5):
 
 ```
 outcome     patched
-stop_reason end_turn          finished deliberately, not cut off
-verified    True              the project's own `npm test`, run in the microVM
-added_files []                no scratch left behind
-patch       1,317 bytes, one file
+verified    True              npm test passes in the microVM
+model       us.anthropic.claude-sonnet-4-5-20250929-v1:0
+tokens      ~95K in / ~4.5K out
+cost        ~$0.35 per issue fix
+files       src/routes/tasks.ts + tests/tasks.test.ts
 ```
 
-An earlier harness run also fixed the same defect in `src/mcp/tools.ts`, which the
-issue never mentioned.
+The agent reproduces the bug, fixes the code, writes a regression test, re-verifies,
+commits, pushes, and emits OTEL telemetry — all inside the harness. The workflow
+opens the PR with a model-generated summary explaining the fix.
 
 ### Not yet done
 
-- **The workflow has never run in CI.** Its shell logic was exercised branch by
-  branch against throwaway repositories, and the client fetch is new. Issue events
-  only trigger workflows from the default branch, so it cannot be tested from a
-  feature branch.
-- **Only `sample-app` has been used as a target.** Whether the exploration cost
-  (~540K input tokens per issue) holds on an unfamiliar repository is untested.
-- **Cold-start time** for the 591 MB image, and real `mise install` timing inside a
-  session rather than in a local container.
 - **Agent PRs do not trigger the eval gate.** A PR opened with `GITHUB_TOKEN` does
   not fire `on: pull_request` workflows. The metrics workflow (`prism-ai-metrics`)
   is unaffected because it triggers on merge, which is a human action. The eval gate
   needs either a GitHub App token or a `workflow_dispatch` ping.
+- **Only `sample-app` has been used as a target.** Whether the token cost holds on
+  larger repositories is untested.
+- **Cold-start time** varies: the 750 MB image pull + mise install adds ~45-60s
+  before the agent starts reasoning.
 
 ---
 

package/dist/assets/coding-agent/agent.py

@@ -26,13 +26,35 @@ from config import ConfigError, load_config
 from system_prompt import build_system_prompt, build_task_prompt
 
 
+def _unwrap_issue(payload: dict) -> dict:
+    """Return the issue object, whether or not it is nested under an `issue` key.
+
+    Two producers feed `--issue` and they disagree on shape:
+
+      * eval fixtures (`.coding-agent/fixtures/*.json`) are flat --
+        `{"number": 1, "title": ..., "body": ...}`
+      * the CI workflow wraps, because `gh issue view --json ... | jq '{issue: .}'`
+        produces `{"issue": {"number": 9, ...}}`
+
+    Reading `.get("title")` off the wrapper returns None, so the agent was handed
+    `Fix issue #?: (no title)` with `(no description provided)` as its entire task.
+    It then did the only thing it could -- went looking for something to fix --
+    and produced a correct patch for a bug nobody asked about. The scope-discipline
+    constraints in the system prompt cannot help here: there was no scope to stay
+    inside. That is why this normalises instead of trusting either caller.
+    """
+    if isinstance(payload, dict) and "issue" in payload and isinstance(payload["issue"], dict):
+        return payload["issue"]
+    return payload
+
+
 def _load_issue(args: argparse.Namespace) -> dict:
     """Resolve the issue from a fixture file, a GitHub event payload, or flags."""
     if args.issue:
         path = Path(args.issue)
         if not path.exists():
             raise SystemExit(f"Issue file not found: {path}")
-        return json.loads(path.read_text())
+        return _unwrap_issue(json.loads(path.read_text()))
 
     if args.github_event:
         path = Path(args.github_event)
@@ -374,6 +396,14 @@ def main() -> int:
         print(f"\nBOUND REACHED: {bound.stopped_reason}", file=sys.stderr)
     print(f"  model calls: {bound.iterations}/{bound.max_iterations}   "
           f"elapsed: {bound.elapsed:.0f}s/{bound.deadline_seconds}s", file=sys.stderr)
+
+    # Report token usage so the orchestrator (run.py) can include it in telemetry.
+    # The format is a structured line that run.py parses from stderr.
+    usage = agent.event_loop_metrics.accumulated_usage
+    in_tok = getattr(usage, "inputTokens", 0) or (usage.get("inputTokens", 0) if isinstance(usage, dict) else 0)
+    out_tok = getattr(usage, "outputTokens", 0) or (usage.get("outputTokens", 0) if isinstance(usage, dict) else 0)
+    print(f"USAGE_REPORT: input_tokens={in_tok} output_tokens={out_tok}", file=sys.stderr)
+
     return 0
 
 

package/dist/assets/coding-agent/config.py

@@ -104,7 +104,7 @@ class AgentConfig:
     agent_email: str = "prism-agent@example.com"
     agent_name: str = "PRISM Coding Agent"
     max_attempts: int = 3
-    model_id: str = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+    model_id: str = ""  # Set by --model-id; no default — deploy-harness.sh owns the model choice
     region: str = "us-west-2"
     project_type: str = "unknown"
     source: str = "detected"  # "config" | "detected" | "none"

package/dist/assets/coding-agent/deploy/create_harness.py

@@ -54,6 +54,121 @@ def find_existing(client, name: str) -> str | None:
             return None
 
 
+ROLE_PROPAGATION_TIMEOUT_SECONDS = 180
+ROLE_PROPAGATION_POLL_SECONDS = 5
+
+
+def _is_role_not_yet_visible(exc: ClientError) -> bool:
+    """True when a ValidationException is the IAM propagation race, not a bad role.
+
+    AgentCore validates that it can assume the execution role, and a role created
+    seconds earlier is not yet visible to it:
+
+        ValidationException: Role validation failed for '...'. Please verify that
+        the role exists and its trust policy allows assumption by this service
+
+    The message is identical whether the role is genuinely misconfigured or merely
+    too new, so this matches on the text and lets the timeout below decide. A
+    permanently broken trust policy simply exhausts the retries and reports the
+    original error.
+    """
+    err = exc.response.get("Error", {})
+    if err.get("Code") != "ValidationException":
+        return False
+    message = err.get("Message", "").lower()
+    return "role validation failed" in message or "trust policy" in message
+
+
+def _call_with_role_propagation_retry(label: str, fn):
+    """Run fn(), retrying while AgentCore still cannot see a freshly created role.
+
+    deploy-harness.sh creates the execution role and then immediately creates the
+    harness. On an account where the role already exists that gap is irrelevant, so
+    every re-run succeeds and this race stays invisible -- which is why it first
+    surfaced during workshop provisioning into a brand new account. Measured
+    evidence: a 55-second gap succeeded, a 6-second gap failed.
+    """
+    deadline = time.monotonic() + ROLE_PROPAGATION_TIMEOUT_SECONDS
+    announced = False
+    while True:
+        try:
+            return fn()
+        except ClientError as exc:
+            if not _is_role_not_yet_visible(exc) or time.monotonic() >= deadline:
+                raise
+            if not announced:
+                print(f"    {label}: execution role not visible to AgentCore yet, "
+                      f"retrying for up to {ROLE_PROPAGATION_TIMEOUT_SECONDS}s",
+                      file=sys.stderr)
+                announced = True
+            time.sleep(ROLE_PROPAGATION_POLL_SECONDS)
+
+
+def for_update(client, spec: dict) -> dict:
+    """Adapt a CreateHarness spec to the shape UpdateHarness expects.
+
+    The two operations do not take the same input. Several members that
+    CreateHarness accepts directly are wrapped in an `optionalValue` envelope on
+    UpdateHarness, so that clearing a field can be distinguished from omitting
+    it. At the time of writing that applies to `environmentArtifact`,
+    `authorizerConfiguration` and `memory` -- passing the create shape fails
+    validation client-side with:
+
+        Unknown parameter in environmentArtifact: "containerConfiguration",
+        must be one of: optionalValue
+
+    The set of wrapped members is read off the service model rather than
+    hardcoded, so adding one of the other wrapped fields to `spec` later cannot
+    silently reintroduce this failure.
+    """
+    members = client.meta.service_model.operation_model(
+        "UpdateHarness"
+    ).input_shape.members
+    wrapped = {
+        name
+        for name, shape in members.items()
+        if "optionalValue" in getattr(shape, "members", {})
+    }
+    return {
+        key: {"optionalValue": value} if key in wrapped else value
+        for key, value in spec.items()
+    }
+
+
+def unwrap_harness(response: dict) -> dict:
+    """Return the harness object from any single-harness response envelope.
+
+    CreateHarness, UpdateHarness and GetHarness all nest the object under a
+    `harness` key -- they share one `Harness` output shape. Only ListHarnesses
+    differs, returning flat objects in `harnesses`.
+
+    This is a function rather than an inline `.get("harness")` because the same
+    envelope has now been missed twice. First on the read path, where reading
+    `status` off the top level of a get_harness response yielded None forever, so
+    the wait below span its full timeout and reported failure on a harness that
+    had been READY the whole time. Then on the create path, where
+    `created["harnessId"]` raised KeyError: 'harnessId' -- invisible until a
+    provisioning run actually created a harness through this script rather than
+    updating one that already existed.
+    """
+    return response.get("harness", {})
+
+
+def harness_id_from(response: dict, operation: str) -> str:
+    """Pull harnessId out of a create/update response, or say what came back.
+
+    A bare KeyError names the missing key but not the envelope it was missing
+    from, which is a poor trade when the call costs a full provisioning run.
+    """
+    harness = unwrap_harness(response)
+    harness_id = harness.get("harnessId")
+    if not harness_id:
+        sys.exit(f"{operation} returned no harnessId.\n"
+                 f"  envelope keys: {sorted(response)}\n"
+                 f"  harness keys:  {sorted(harness) or '(harness key absent)'}")
+    return harness_id
+
+
 def describe(client, harness_id: str) -> dict:
     """Fetch one harness, unwrapping the response envelope.
 
@@ -64,24 +179,53 @@ def describe(client, harness_id: str) -> dict:
     been READY the whole time. The same mistake also made `maxIterations` look
     unset on a harness that had it set to 40.
     """
-    return client.get_harness(harnessId=harness_id).get("harness", {})
+    return unwrap_harness(client.get_harness(harnessId=harness_id))
+
+
+def _report_failure(harness_id: str, harness: dict, headline: str):
+    """Exit with everything the harness already told us about why it failed.
+
+    `failureReason` is a member of the Harness shape and is the only field that
+    says *why* provisioning failed. Reporting the status alone -- which is what
+    this did -- costs a whole provisioning run to learn the word "CREATE_FAILED",
+    and the harness then has to be queried by hand to find out anything useful.
+    The image URI and execution role are echoed too because they are the two
+    inputs that most often break a container harness after the control-plane call
+    has already succeeded.
+    """
+    reason = harness.get("failureReason") or "(no failureReason reported)"
+    image = (harness.get("environmentArtifact", {})
+                    .get("containerConfiguration", {})
+                    .get("containerUri", "(none)"))
+    sys.exit(f"{headline}\n"
+             f"  reason:  {reason}\n"
+             f"  image:   {image}\n"
+             f"  role:    {harness.get('executionRoleArn', '(none)')}\n"
+             f"  status:  {harness.get('status', 'unknown')}\n"
+             f"  full detail:\n"
+             f"    aws bedrock-agentcore-control get-harness \\\n"
+             f"      --harness-id {harness_id} --query harness")
 
 
 def wait_ready(client, harness_id: str) -> None:
     deadline = time.monotonic() + READY_TIMEOUT_SECONDS
     last = ""
+    harness: dict = {}
     while time.monotonic() < deadline:
-        status = describe(client, harness_id).get("status", "")
+        harness = describe(client, harness_id)
+        status = harness.get("status", "")
         if status == "READY":
             return
         if status in {"CREATE_FAILED", "UPDATE_FAILED", "FAILED"}:
-            sys.exit(f"harness {harness_id} reached {status}")
+            _report_failure(harness_id, harness,
+                            f"harness {harness_id} reached {status}")
         if status != last:
             print(f"    status {status or 'unknown'}", file=sys.stderr)
             last = status
         time.sleep(POLL_SECONDS)
-    sys.exit(f"harness {harness_id} did not reach READY within "
-             f"{READY_TIMEOUT_SECONDS}s (last status {last or 'unknown'})")
+    _report_failure(harness_id, harness,
+                    f"harness {harness_id} did not reach READY within "
+                    f"{READY_TIMEOUT_SECONDS}s (last status {last or 'unknown'})")
 
 
 def main() -> int:
@@ -115,19 +259,40 @@ def main() -> int:
     try:
         if existing:
             print(f"    updating {existing}", file=sys.stderr)
-            client.update_harness(harnessId=existing, **spec)
+            _call_with_role_propagation_retry(
+                "update", lambda: client.update_harness(harnessId=existing, **for_update(client, spec)))
             harness_id = existing
         else:
             print(f"    creating {name}", file=sys.stderr)
-            created = client.create_harness(harnessName=name, **spec)
-            harness_id = created["harnessId"]
+            created = _call_with_role_propagation_retry(
+                "create", lambda: client.create_harness(harnessName=name, **spec))
+            harness_id = harness_id_from(created, "create_harness")
     except ClientError as exc:
         code = exc.response.get("Error", {}).get("Code", "")
         message = exc.response.get("Error", {}).get("Message", str(exc))
         if code == "AccessDeniedException":
+            # The hint names both halves on purpose. The SDK calls CreateHarness /
+            # UpdateHarness / GetHarness / ListHarnesses, but IAM evaluates the
+            # AgentRuntime action underneath each one, so a policy granting only the
+            # *Harness names is denied on *AgentRuntime -- which is what the earlier
+            # version of this message sent people chasing.
+            sys.exit(f"{code}: {message}\n"
+                     f"  The calling identity needs both halves of each Harness call:\n"
+                     f"    bedrock-agentcore:CreateHarness  + :CreateAgentRuntime\n"
+                     f"    bedrock-agentcore:UpdateHarness  + :UpdateAgentRuntime\n"
+                     f"    bedrock-agentcore:GetHarness     + :GetAgentRuntime\n"
+                     f"    bedrock-agentcore:ListHarnesses  + :ListAgentRuntimes\n"
+                     f"  and iam:PassRole for {role}.\n"
+                     f"  IAM authorizes the AgentRuntime action, not the Harness one --\n"
+                     f"  read the action name in the message above, not this list.")
+        if code == "ValidationException" and "role" in message.lower():
             sys.exit(f"{code}: {message}\n"
-                     f"  The calling identity needs bedrock-agentcore:CreateHarness /\n"
-                     f"  UpdateHarness / ListHarnesses, and iam:PassRole for {role}.")
+                     f"  {role} was still not assumable by AgentCore after "
+                     f"{ROLE_PROPAGATION_TIMEOUT_SECONDS}s, so this is not the usual\n"
+                     f"  propagation delay. Check the trust policy actually allows\n"
+                     f"  bedrock-agentcore.amazonaws.com to sts:AssumeRole:\n"
+                     f"    aws iam get-role --role-name {role.rsplit('/', 1)[-1]} \\\n"
+                     f"      --query Role.AssumeRolePolicyDocument")
         sys.exit(f"{code}: {message}")
 
     wait_ready(client, harness_id)

package/dist/assets/coding-agent/deploy/deploy-harness.sh

@@ -31,7 +31,7 @@ TAG="v1"
 NAME="PrismCodingAgent"
 ECR_REPO="prism/coding-agent-harness"
 ROLE_NAME="PrismCodingAgentHarnessRole"
-MODEL_ID="us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+MODEL_ID="us.anthropic.claude-sonnet-5"
 # Matches agentcore.contract.MAX_ITERATIONS. The client passes maxIterations on
 # every InvokeHarness call, so this is only the fallback for callers that do not.
 MAX_ITERATIONS=100
@@ -102,7 +102,7 @@ docker buildx build \
   --tag "${IMAGE}" \
   --file "${HERE}/Dockerfile" \
   --push \
-  "${HERE}"
+  "${AGENT_DIR}"
 echo "    pushed ${IMAGE}"
 
 # ---- 3. Execution role -------------------------------------------------------
@@ -135,7 +135,13 @@ POLICY=$(cat <<JSON
    "bedrock-agentcore:RetrieveMemoryRecords","bedrock-agentcore:ListMemoryRecords",
    "bedrock-agentcore:GetMemoryRecord","bedrock-agentcore:GetMemory",
    "bedrock-agentcore:CreateMemory","bedrock-agentcore:ListMemories"],
-  "Resource":"*"}
+  "Resource":"*"},
+ {"Sid":"SsmTelemetryConfig","Effect":"Allow","Action":[
+   "ssm:GetParameter","ssm:GetParameters"],
+  "Resource":"arn:aws:ssm:${REGION}:${ACCOUNT}:parameter/prism/d1/*"},
+ {"Sid":"SecretsManagerAgentCreds","Effect":"Allow","Action":[
+   "secretsmanager:GetSecretValue"],
+  "Resource":"arn:aws:secretsmanager:${REGION}:${ACCOUNT}:secret:prism-d1-*"}
 ]}
 JSON
 )
@@ -167,8 +173,8 @@ HARNESS_ARN=$(
 )
 echo "    ${HARNESS_ARN}"
 
-# ---- 5. Write the ARN to SSM so it is discoverable without manual copying ----
-echo "==> SSM parameter"
+# ---- 5. Write the ARN and model to SSM so they are discoverable without manual copying ----
+echo "==> SSM parameters"
 aws ssm put-parameter \
   --name /prism/d1/harness-arn \
   --value "${HARNESS_ARN}" \
@@ -176,6 +182,13 @@ aws ssm put-parameter \
   --overwrite \
   --description "AgentCore harness ARN — written by deploy-harness.sh" >/dev/null
 echo "    /prism/d1/harness-arn = ${HARNESS_ARN}"
+aws ssm put-parameter \
+  --name /prism/d1/harness-model \
+  --value "${MODEL_ID}" \
+  --type String \
+  --overwrite \
+  --description "Model ID used by the coding agent — written by deploy-harness.sh" >/dev/null
+echo "    /prism/d1/harness-model = ${MODEL_ID}"
 
 cat <<EOF