humanish 0.141.0 → 0.143.0

package/docs/contracts/study-analysis.md

@@ -5,7 +5,8 @@ It is separate from the participant's account, recorded outcome, and the run's
 deterministic review verdict. Opening an Observer never starts a provider request.
 Supported live runs request analysis on completion by default. A study that declares no analyst
 gets the Codex account analyst when its participant is a Codex `local-agent`, and the OpenAI
-analyst with a separate $10 cap on its expected cost otherwise. Set `review.analysis: false` to
+analyst with a separate $10 cap on its expected cost otherwise. A study can also select the
+[Gemini API analyst](#gemini-api-analysis). Set `review.analysis: false` to
 disable that request; see [automatic analysis](../product/automatic-analysis.md).
 
 ## Invocation
@@ -24,7 +25,8 @@ study's input and admission estimate without credentials, a provider request,
 or a new analysis artifact. It prints the expected cost, the worst case and the
 cap; a refused dry run says why, and names the `--max-cost` that admits it.
 
-The default provider is `openai`; its default model is `gpt-6-astra`, with high reasoning effort. A request sends selected retained text and
+The default provider is `openai`; its default model is `gpt-6.1-sol`, at medium reasoning effort
+([reasoning effort](#reasoning-effort)). A request sends selected retained text and
 captures to OpenAI, without tools, redirects, provider-side response storage, or
 automatic retries. `--question` adds a reviewer question; it never changes the
 participant assignment. For OpenAI API analysis, `--max-cost` is required, including for dry-run
@@ -117,6 +119,81 @@ the current merge instructions, billed $1.92, most of its input served from the
 provider's prompt cache, and each of its 466 citations resolved. Two runs do not
 calibrate the merge request's estimate.
 
+## Reasoning effort
+
+`review.analysis.reasoningEffort`, or `analyze --reasoning-effort <level>`, sets how long an API
+analyst reasons before it answers. OpenAI receives it as `reasoning.effort`, and Gemini as
+`generationConfig.thinkingConfig.thinkingLevel` in capitals. Each model takes the levels its
+provider documents for it (OpenAI's model pages and Gemini's thinking guide, read 2026-10-21):
+
+| Model                                                                   | Levels                                          |
+| ----------------------------------------------------------------------- | ----------------------------------------------- |
+| `gpt-6-astra`, `gpt-6.1-sol`                                            | `low`, `medium`, `high`, `xhigh`, `max`         |
+| `gpt-6-luna`, `gpt-5.6`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna` | `none`, `low`, `medium`, `high`, `xhigh`, `max` |
+| `gpt-5.5`                                                               | `none`, `low`, `medium`, `high`, `xhigh`        |
+| `gemini-3.8-flash`                                                      | `low`, `medium`, `high`                         |
+
+Any other level is refused before a request, with a message naming the levels the model takes.
+The Codex account analyst runs at its fixed low effort, and a declared effort with
+`provider: codex` is refused.
+
+The default analyst, an OpenAI analysis that names no model and declares no effort, is
+`gpt-6.1-sol` at `medium`, and its resolved configuration writes `reasoningEffort: medium`. A study
+or command that names a model but no effort runs that model at `high`; only the default analyst
+runs at medium. That analysis's configuration has no `reasoningEffort` field, so its configuration
+hash, and with it reuse of an earlier analysis, is what it was before the field existed. A declared effort is stored in the analysis artifact's
+`config`, `high` included, and gives the configuration its own hash. An artifact whose config has
+no `reasoningEffort` ran at `high`.
+
+Admission expects the same output at every effort, from analyses billed at `high`. No billed
+analysis at a lower effort calibrates a smaller expectation yet, so a lower effort's expected cost
+is likely over its bill: on 24 one-participant Taskly analyses on 2026-10-22, `gpt-6.1-sol` at
+medium wrote a mean 3,747 output tokens against the 13,000 admission expects. The worst case spends the whole output allowance at any effort.
+
+## Gemini API analysis
+
+`review.analysis.provider: gemini`, or `analyze --provider gemini`, sends the same request to
+Google's Gemini API:
+
+```bash
+humanish keys set gemini
+humanish analyze --run latest --provider gemini --max-cost 2 --dry-run --json
+humanish analyze --run latest --provider gemini --model gemini-3.8-flash --max-cost 2 --json
+```
+
+The request goes to `generativelanguage.googleapis.com/v1beta/models/<model>:generateContent`
+with the key in the `x-goog-api-key` header. It carries the same instructions, packet and
+captures, with each capture at `MEDIA_RESOLUTION_HIGH`, the analysis's reasoning effort as its
+thinking level (`HIGH` unless declared), one candidate,
+the standard service tier, no tools and `store: false`. The key is `GEMINI_API_KEY` when set,
+else `GOOGLE_API_KEY`. Google's client libraries prefer `GOOGLE_API_KEY`; humanish does not, so a
+`GOOGLE_API_KEY` exported for another Google API never replaces the stored Gemini key. Key
+discovery fills neither name while the other is already in the environment. Without either, the attempt ends with
+`ANALYSIS_API_KEY_MISSING` and nothing is sent; the OpenAI key is never used.
+
+The supported model is `gemini-3.8-flash`, the one with a completed live analysis. The other
+Gemini models whose pages list image input and structured output join once a live analysis on
+each completes. `--max-cost` is required, and admission, the cost cap, `--max-output-tokens`
+(at most 32,768) and the partial result for a bill above the worst case work as on OpenAI.
+Admission prices each capture at 1,120 tokens, the media-resolution guide's figure for a
+high-resolution image, and expects twice the output it expects from OpenAI: one
+`gemini-3.8-flash` analysis of one participant wrote 24,246 output tokens. Usage maps
+`promptTokenCount` to input, `candidatesTokenCount` plus `thoughtsTokenCount` to output, and
+`cachedContentTokenCount` to cached input; a usage record whose `totalTokenCount` disagrees
+with those parts is kept as unknown. Rates are Google's paid standard tier. `gemini-3.8-flash`
+has a published price change on 2027-01-01, and each request is priced at the rates of the day
+it is sent. A free-tier key bills nothing, and Google's terms
+let it use free-tier content to improve its products.
+
+Gemini refuses the analysis schemas while they carry `maxItems`, so the request leaves those
+array limits out of the schema and states them, by field path, at the end of the system
+instruction. The response is still checked against the full schema. A `STOP` candidate
+whose only non-thought part is the JSON answer completes. `MAX_TOKENS` and `CONTINUATION` end
+as `output_incomplete`. A blocked prompt and the `SAFETY`, `RECITATION`, `LANGUAGE`,
+`BLOCKLIST`, `PROHIBITED_CONTENT`, `SPII`, `ESCALATION` and image-safety finish reasons end as
+`refusal`. Any other finish reason or answer shape ends as `invalid_response`. An HTTP error
+keeps only its status code, which the attempt's warnings name.
+
 ## Codex account analysis
 
 A study whose participant is a Codex `local-agent` uses this analyst unless it declares another,
@@ -492,11 +569,15 @@ The states without findings:
   `analyze --max-cost 10`, which runs an analysis anyway. For a skip without `OPENAI_API_KEY`
   on a machine where Codex is signed in with a ChatGPT account (read when the view is built),
   `next` is `analyze --provider codex` and the message names `review.analysis.provider: codex`
-  in the run's study file; otherwise the message says to set the key. For an analysis refused for its
+  in the run's study file; otherwise the message says to set the key. A study analyzed on Gemini
+  is skipped with `AUTOMATIC_ANALYSIS_GEMINI_KEY_MISSING`; its message names `keys set gemini`,
+  and `next` is `analyze --provider gemini --model <model> --max-cost 10`.
+  For an analysis refused for its
   cost, the message gives the expected cost, the worst case and the cap, and `next` sets
   `--max-cost` to the worst case rounded up.
 - `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
-  reruns it with the same analyst, `--provider codex` for the Codex account analyst.
+  reruns it with the same analyst: `--provider codex` for the Codex account analyst, and
+  `--provider gemini --model <model>` for a Gemini one.
 - `none`: no analysis ran for this live run and nothing records why, for example a run that
   started from a library call, or one whose study file has moved or changed id.
 - `dry_run`: a dry run has no participant evidence to analyze. `next` is `run` with the run's own

package/docs/product/automatic-analysis.md

@@ -10,8 +10,12 @@ Every other participant, including a Claude Code `local-agent` participant, gets
 analyst. humanish never moves a study to the other analyst because a key is missing: the account
 that pays for analysis is the one the study names or its participant implies.
 
-The OpenAI default is `gpt-6-astra` with high reasoning effort, a separate $10 cap on its
-expected cost and a 600-second timeout. When no output limit is specified,
+The OpenAI default is `gpt-6.1-sol` at medium reasoning effort (`review.analysis.reasoningEffort`
+sets another), a separate $10 cap on its
+expected cost and a 600-second timeout. A study or command that names a model but no effort runs
+that model at high; only the default analyst, with no model named, runs at medium. On the same
+recordings it matched `gpt-6-astra` at high, the earlier default, at about one seventh of the cost
+([analysis models on fixed recordings](https://github.com/danielgwilson/humanish/blob/main/docs/evidence/benchmark/2026-10-22-analysis-models.md)). When no output limit is specified,
 humanish selects 32,768 tokens if the exact input's expected cost still fits that
 cap; otherwise it keeps the established 16,384-token allowance. This preserves
 previously admitted studies without increasing their spending limit. To customize it:
@@ -21,7 +25,9 @@ review:
   analysis:
     maxCostUsd: 10
     # Optional overrides. Omit maxOutputTokens for budget-aware selection.
-    model: gpt-6-astra
+    # A named model without reasoningEffort runs at high.
+    model: gpt-6.1-sol
+    reasoningEffort: medium
     timeoutMs: 600000
     # maxOutputTokens: 32768
     # question: Where did participants need to recover?
@@ -41,16 +47,20 @@ the cap admits no analysis at all, it says so instead. A study of more than 16 p
 analysed in cohorts of at most 16, one request each, and one more request merges their reports;
 the range covers every request and the analysis line names them (see
 [runs with more than 16 participants](../contracts/study-analysis.md#runs-with-more-than-16-participants)).
-The default $10 cap admits a single request at the evidence limits ($3.79 expected for one
-participant, $4.54 for 16) and the analysis of 17 to 40 participants from $4.60 to $7.05 with no
-evidence; a run that keeps more evidence can still be refused, and its skip names the
+The default $10 cap admits the default analyst up to the evidence limits: a single request at
+$0.76 expected for one participant and $0.91 for 16, and 17 to 40 participants at $0.92 to $3.24
+from no evidence to the limits. Admission expects 13,000 output tokens for one participant at any
+effort; on 24 Taskly analyses `gpt-6.1-sol` at medium wrote a mean 3,747 and cost a mean $0.06.
+`gpt-6-astra` at high expects $3.79 for one participant at the limits and $4.60 to $7.05 for 17 to
+40 with no evidence, so a run that keeps more evidence can be refused, and its skip names the
 `--max-cost` that admits it. Use `analyze --dry-run --max-cost <usd>` on retained
 evidence to inspect the expected cost, the worst case and the selected token
 allowance before deliberately choosing a larger budget. Explicit
 `maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
 includes reasoning as well as the report; exhausting it does not produce a usable
 report and never starts an automatic retry. Analysis
-sends selected retained text and captures to OpenAI using `OPENAI_API_KEY`.
+sends selected retained text and captures to OpenAI using `OPENAI_API_KEY`, or to Google with
+`provider: gemini`.
 Analysis runs in the humanish runner using its credentials. This setting adds no
 credential channel to the target application; each participant's route keeps
 its existing authentication boundary. Review the separate analysis budget before running a manifest live; an actor's
@@ -78,6 +88,28 @@ remain untested until a request. An omitted provider means this analyst for a Co
 `local-agent` participant on any target, and OpenAI for every other participant. This setting
 does not enable managed local desktops.
 
+To use Google's Gemini API for the separate analyst:
+
+```yaml
+review:
+  analysis:
+    provider: gemini
+    model: gemini-3.8-flash
+    maxCostUsd: 2
+```
+
+Gemini analysis sends the same evidence packet and captures to Google, at the high thinking level
+unless the study declares a `reasoningEffort`,
+with each capture at high media resolution. It reads `GEMINI_API_KEY`, which `humanish keys set
+gemini` stores, and `GOOGLE_API_KEY` only when `GEMINI_API_KEY` is unset. Key discovery treats
+the two names as one key: a project's `.humanish/local/provider.env` or the user store never adds
+one while the other is already in the environment. A missing Gemini key skips the analysis
+(`AUTOMATIC_ANALYSIS_GEMINI_KEY_MISSING`); the OpenAI key is never used in its place. Admission,
+`maxCostUsd`, `maxOutputTokens` and priced usage work as they do on OpenAI, at the paid-tier
+rates in `src/run/pricing.ts`; requests made with a free-tier key bill nothing, and Google's
+terms let it use free-tier content to improve its products. The supported model is
+`gemini-3.8-flash`, the one with a completed live analysis.
+
 The same configuration works through `humanish run <study>`,
 `watch <study>`, and TUI live starts. Direct library calls to the five recording
 producers honor it too. Supported routes are computer-use, scripted-browser,
@@ -147,7 +179,8 @@ run's own result is ok, the first line of the human output reads
 command: with Codex signed in with a ChatGPT account it is `humanish analyze --run <id>
 --provider codex`, and the message names `review.analysis.provider: codex` as the setting that
 makes that analyst the study's; otherwise it says to set the key and names `humanish analyze
---run <id> --max-cost 10`. So does a
+--run <id> --max-cost 10`. A study analyzed on Gemini names `humanish keys set gemini` and
+`humanish analyze --run <id> --provider gemini --model <model> --max-cost 10`. So does a
 default analysis whose expected cost, plus the margin, is over the default $10 cap:
 it is skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` and the run exits 0. For
 any analysis refused for its cost, the job records the expected cost, the worst
@@ -197,6 +230,7 @@ JSON also has `findings`:
 | `AUTOMATIC_ANALYSIS_SOURCE_UNAVAILABLE`       | `skipped`                 | This invocation published no run to analyze, or the run id is invalid |
 | `AUTOMATIC_ANALYSIS_SOURCE_CHANGED`           | `failed`                  | The run directory changed after the run published its bundle          |
 | `AUTOMATIC_ANALYSIS_KEY_MISSING`              | `skipped`                 | No OpenAI API key                                                     |
+| `AUTOMATIC_ANALYSIS_GEMINI_KEY_MISSING`       | `skipped`                 | No Gemini API key for a study analyzed on Gemini                      |
 | `AUTOMATIC_ANALYSIS_NO_PARTICIPANT_EVIDENCE`  | `skipped`                 | A default analysis found no participant evidence                      |
 | `AUTOMATIC_ANALYSIS_ACTOR_CANCELLED`          | `skipped`                 | The harness cancelled the participants, or a signal stopped the run   |
 | `AUTOMATIC_ANALYSIS_CLEANUP_UNCONFIRMED`      | `skipped`                 | A local VM study could not confirm its cleanup; the command exits 2   |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "humanish",
-  "version": "0.141.0",
+  "version": "0.143.0",
   "description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
   "keywords": [
     "cli",
@@ -96,7 +96,7 @@
     "comms:agentmail:contract": "node --import tsx scripts/agentmail-contract-check.mjs",
     "format": "oxfmt",
     "format:check": "oxfmt --check",
-    "lint": "node scripts/check-lint-cap.mjs --max-warnings=411",
+    "lint": "node scripts/check-lint-cap.mjs --max-warnings=410",
     "knip": "knip",
     "prose:check": "node scripts/check-code-prose.mjs",
     "vocabulary:check": "tsx scripts/check-retired-words.ts",

package/skills/humanish/SKILL.md

@@ -98,7 +98,8 @@ exact returned path, not a basename that could resolve to another manifest.
 Analysis is separate from the participant. A study that declares no analyst
 gets one from its participant: a Codex `local-agent` participant, local or
 hosted, defaults to a separate Codex account analyst; every other participant,
-Claude Code included, defaults to the OpenAI API with a $10 admission budget.
+Claude Code included, defaults to the OpenAI API (`gpt-6.1-sol` at medium
+reasoning effort) with a $10 admission budget.
 Without `OPENAI_API_KEY` that OpenAI analysis is skipped, and the run output
 names the next command: `analyze --provider codex` when Codex is signed in. To
 select the restricted Codex ChatGPT account analyst on any supported study,
@@ -108,6 +109,22 @@ not local inference or the participant's existing conversation. See
 [the analysis contract](../../docs/contracts/study-analysis.md) for the current
 CLI/model qualification and setup limits.
 
+To analyze on Gemini, the user stores a key with `npx humanish keys set gemini`
+(`GEMINI_API_KEY`; `GOOGLE_API_KEY` is read only when `GEMINI_API_KEY` is
+unset), then sets `review.analysis: {provider: gemini, maxCostUsd: <n>}` or
+passes `analyze --provider gemini --max-cost <n>`. The supported model is
+`gemini-3.8-flash`. This sends the evidence packet and
+captures to Google. Admission, the cost cap and priced usage work as on OpenAI,
+and a missing Gemini key never falls back to OpenAI. On a free-tier key, Google
+may use the content to improve its products; tell the user before a study with
+private evidence runs on one.
+
+`review.analysis.reasoningEffort` (or `analyze --reasoning-effort <level>`)
+sets an OpenAI or Gemini analyst's reasoning effort. When unset it is `medium`
+for the default analyst, which names no model, and `high` for a named model.
+Each model takes only the levels its provider documents, and `analyze` refuses
+the others before any request with the levels the model takes.
+
 When post-run analysis is refused for its cost, the run output and `humanish
 review` give its expected cost, its worst case and the command that runs it,
 `humanish analyze --run <id> --max-cost <n>`. Show the user those costs and run