tinybeeai 0.0.0-stage → 0.16.1

package/docs/CHAT-EVAL.md

@@ -0,0 +1,130 @@
+# Chat eval
+
+`npm run eval:chat -- --backend codex-cli` sends a fixed set of chat requests to a dot and checks the tool use and the answer of each one. It runs on a throwaway database, and every tool answers from a fixed stub, so a case measures what the chat did with the message and never touches a tracker, a repo, or a real instance.
+
+A case passes when all of these hold:
+
+- The chat called the expected tools. A ticket or PR link starts with `review_ticket`, `get_ticket`, or `review_pr`. A question about the ongoing work calls `open_prs`, `runs_in_progress`, and `pending_approvals`. A question about breakage calls `recent_defects` or `list_runs`.
+- No forbidden tool ran. A ticket question does not read run history, probes, or memory, and "what is the ongoing work" does not read memory or probes, so a tool card only shows when its tool was needed. A refusal ("do not review TAD-3") calls no review tool at all.
+- No tool ran twice with the same input in one turn.
+- The answer carries the block kinds the case names (KAT-136). A case lists `blocks` (every kind must show), `blocksAny` (one of these kinds must show, for a question the model may answer from either of two tools), or `noBlocks` (every lookup of the turn was context, so no block may show). A case that names none does not look at blocks. The runner reads the `block` events of the stream, so it measures what the server built from each tool result, and a missing kind fails the case with "no runs block, got none". The stubs return the same result shape as the real tools, and the eval seeds the stored review and the comment draft of TAD-3 that a block reads behind its row.
+- The answer matches the case's patterns and the plain-words rubric: it says the fact the case asks for, stays under 700 characters (link addresses not counted), and holds no tool names, field names, JSON, ids from the stub, or em dashes.
+- Every PR and ticket the stubs know (PRs 11, 12, 14, 32, 39 and TAD-3) shows as its title with a link, never as a bare "PR #12", "#11", or "TAD-3" (KAT-225). The reference renderer (`src/refs.ts`) rewrites the final answer, so a model that writes a bare number still passes; a model that writes a title with its number in parentheses passes the same way. A number the eval cannot know, such as issue 9 of another repo, is not checked.
+
+On the `claude-cli` and `codex-cli` backends the model calls the tools itself through the localdost chat MCP server (KAT-133), so the first-tool and which-tools checks measure the model's own choices, not a fixed lookup. The result tables record the time to the first streamed text and the total time of each case.
+
+| Case | Message | Tools expected | Blocks expected | Answer rubric |
+|---|---|---|---|---|
+| ticket-link | check this https://acme-test.atlassian.net/browse/TAD-3 | review_ticket first | None named (the run card and the draft card carry the review) | Reviews the ticket, names the top risk, says the comment draft waits for approval |
+| ticket-key-question | what is TAD-3 about? (app chat) | get_ticket first | No block: the stored review is not what was asked, so reading the ticket is context | Says what the ticket is about; starts no review |
+| ticket-review-question | what did the review say about TAD-3? (app chat) | get_ticket or list_tickets_reviewed | `ticket_review` | Names the top finding in one or two sentences; the review shows as a block |
+| pr-link | review https://github.com/acme/shop/pull/12 | review_pr first | None named | Names PR 12 and says where the result lands |
+| prod-up | is prod up? (app chat) | probe_status first | `probes` | Says production is up, from the probe |
+| what-broke | what broke since the last merge? | recent_defects or list_runs | None named (a call shows no block) | Names the failing checkout total test after merge 11 and the 45.00 versus 45.01 mismatch |
+| defects-question | any defects since the last merge? (app chat) | recent_defects or list_runs | `defects` or `runs` | Names the failing checkout total test after merge 11 in one or two sentences; the problem shows as a block |
+| recent-defects | list the defects since the last merge (app chat) | recent_defects first | `defects` | Names the failing checkout total test after merge 11 in one or two sentences; the problem shows as a defects block that links its run (mock only: a model asked to list defects writes the run id of each, which the plain-words check refuses) |
+| memory-question | what do you remember about this dot? (app chat) | get_memory first | `memory` | Names one or two stored notes in a sentence; the notes show as a block with Edit and Delete, and nothing is saved (the CLI backends answer it from the rule-based handler, because the word "remember" skips the model) |
+| triage-view | what broke? (app chat) | none | `triage` | One short line and the triage view, with the checkout total test under product; the rule picks the view, so no tool and no model call (KAT-247) |
+| ongoing-work | what is the ongoing work (app chat) | open_prs, runs_in_progress, and pending_approvals | `pulls`, `runs`, and `approvals` | Says in one or two sentences that two pull requests are open, a check is running now, and a draft waits for approval; the lists show the pull requests, the running check, and the draft |
+| flaky-by-hour | flaky tests by hour of day (app chat) | test_results, then compose_view | `view` | Says 14:00 is the busiest hour with 3 of the 5 flaky results; the view has a tile for the busiest hour and a chart of all 24 hours, which the check computes the way the app does (KAT-366) |
+| compare-branches | compare pass rate of two branches this week (app chat) | test_results, then compose_view | `view` | Says feat/search passed 100% of 7 tests and main 83.3% of 6; the view has tiles, a chart tab, and a table tab, and the check finds both rates in what it computes |
+| slower-suites | which suites got slower since last release (app chat) | test_results, then compose_view | `view` | Says 2 suites got slower since the last merge, and that e2e/cart.spec.ts took 159% longer; the view lists the suites that got slower, biggest first |
+| review-titled | review #39 | review_pr first | None named | Queues the review and names PR 39 by its title, "Controls keep their labels on one line", with a link, never as "PR #39" |
+| bare-refs | did #39 and #32 get checked? | none required | None named | Whatever it says about #39 and #32, each shows as its title with a link, not as a number (not run on mock) |
+| waiting-approval | what is waiting for my approval? (app chat) | pending_approvals first | `approvals` | Says one comment draft for TAD-3 waits |
+| lead-team-status | how is the team doing? (app chat, asked of the Lead QA dot) | team_status first | `table` | Says the team is failing and names Review Jira with its reason; the members show as a table with their job and status (KAT-405) |
+| lead-release | can we release 2026.11? (app chat, asked of the Lead QA dot) | none | `release_call` | Answers Not ready in one line with the counts and the blockers, from the stored rows: the rule picks the view, so no tool and no model call (KAT-405) |
+| lead-rerun | run the tests again (asked of the Lead QA dot) | rerun_tests first | None named | Sends the rerun to the member whose job it is and says who was asked: "Asked Review Jira to run the tests again" (KAT-405) |
+| do-not-review | do not review TAD-3 (app chat) | no review tool | No block: a refusal does not offer the stored review | Queues no review and says TAD-3 was left alone (not run on mock) |
+| not-connected | check https://linear.app/acme/issue/ENG-5/discount-rules | review_ticket first | None named | Says Linear is not connected and where to connect it |
+| other-repo | check https://github.com/acme/other/issues/9 | review_ticket first | None named | Says the dot does not watch that repo |
+
+A case that expects or forbids a block asks in the app chat (marked above), because a call hears the answer and the server builds no block for it.
+
+The three lead cases (KAT-405) ask a Lead QA dot that names the eval dot as its only member. Their stubs answer `team_status` and the routed `rerun_tests` the way the real tools do (a table block, `routedTo` and `ask`), and the release question is answered by the rule-based view from stored rows, so it needs no stub: the eval seeds a ticket of milestone 2026.11 that no test touched, beside the failing merge of PR 11.
+
+The three view cases (KAT-366) read one fixed set of runs (`src/cli/eval-chat-results.ts`). The stub answers `test_results` through the real function, so the rows, the filters, and the held result are the ones a live dot gives; the stubs for the other tools are unchanged. `compose_view` is never stubbed: the case runs the spec check, the binding, and the stream. A case with a `view` check reads the finished view block of the stream, runs its datasets and vars with the same code the app's Web Worker runs (`src/view-compute.ts`), and fails with a plain message when a number is not the one the runs give, for example "the busiest hour is 9 with 2, expected 14 with 3". The checks look for values, not for the names a model gives its datasets, so a model's own spec passes when its numbers are right. The runner keeps the last block of each id, so a view that streams as a shell and then a finished block counts once. Every case that carries a view also reads its facts for rows that were cut: when a source of the view holds fewer rows than the tool found (`more`), the case fails unless the answer says it covers part of the data (the newest rows, a shorter window, part of the data). The fixed runs are small, so the three cases here never trip it; `test/chat-eval-blocks.test.ts` feeds the check a view over cut rows, and `test/chat-views.test.ts` runs the rule-based chat over a busy dot. The other cases ask on the call channel. The app chat keeps history, so the runner clears the stored conversation before each case. The eval seeds two stored runs for `triage-view`, since the triage reads runs from the database and not through a tool.
+
+Options: `--only <id>` runs one case, `--gate` exits 1 when a case fails, and `--record` writes the result table for the backend into this page. Run it on every backend you deploy (`mock`, `claude-cli`, `codex-cli`, `anthropic`), and in every dogfood round.
+
+A case where the backend gave up and the rule-based reply answered counts as a fail, so a model outage cannot pass. On 2026-10-08 the first codex-cli run hit the Codex workspace spend cap and fell back on every case; the run below is from after the cap cleared.
+
+## Results
+
+The `mock`, `claude-cli`, and `codex-cli` tables below were recorded on 2026-10-13, after the block expectations and the titled-reference check existed, and carry a `Blocks` column. The three view cases (KAT-366) were added on 2026-10-14 and are recorded on `mock` only; a model backend writes its own spec for them, so the next model run is the first measure of how well a model composes a view. There is no `anthropic` table: the machine that recorded the others has no API key. The model-driven backends are the real test of the `context` flag: a model that marks a lookup that answers the question as context fails its case with the missing kind. On `claude-cli` and `codex-cli`, `ticket-review-question`, `prod-up`, `defects-question`, `ongoing-work`, and `waiting-approval` all showed their block, so neither model marked a lookup that answers the question as context, and `ticket-key-question` and `do-not-review` showed none.
+
+Two rows are not model runs. `memory-question` and `review-titled` answer from the rule-based handler on the CLI backends, because a line with the word "remember" (or a PR to review) skips the model, so their times are near zero. `recent-defects` runs on `mock` only: a model that is asked to list defects writes the run id of each into the answer, which the plain-words check refuses (claude-cli, four tries). The one failing row is `bare-refs` on `codex-cli`: the model answers that neither PR appears in the runs or the open PRs and names neither, so there is no reference to turn into a link (two more tries gave the same answer). The `claude-cli` run passed it.
+
+<!-- eval:mock:start -->
+### mock, 2026-10-22: 21 of 21 passed
+
+| Case | Result | Tools called | Blocks | First text | Total | Answer |
+|---|---|---|---|---|---|---|
+| ticket-link | pass | review_ticket | none | 0.0 s | 0.5 s | [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) has no acceptance criteria; stacking and expired codes are undefined.  |
+| ticket-key-question | pass | get_ticket | none | 0.0 s | 0.4 s | [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) (Story, High). It has no acceptance criteria. My last review: [Apply a |
+| ticket-review-question | pass | get_ticket | ticket_review | 0.0 s | 0.4 s | [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) (Story, High). It has no acceptance criteria. My last review: [Apply a |
+| pr-link | pass | review_pr | none | 0.5 s | 0.8 s | Queued a PR check for [Fix cart total rounding (#12)](https://github.com/acme/shop/pull/12). The result lands here, and on the PR when there is one. |
+| prod-up | pass | probe_status | probes | 0.0 s | 0.1 s | Yes, production is up. The probe passed the last check. |
+| what-broke | pass | recent_defects | none | 0.0 s | 0.2 s | 1 problem found: checkout total spec fails after [Round cart totals (#11)](https://github.com/acme/shop/pull/11): expected 45.00, received 45.01. |
+| defects-question | pass | recent_defects | defects | 0.0 s | 0.2 s | 1 problem found: checkout total spec fails after [Round cart totals (#11)](https://github.com/acme/shop/pull/11): expected 45.00, received 45.01. |
+| recent-defects | pass | recent_defects | defects | 0.0 s | 0.2 s | 1 problem found: checkout total spec fails after [Round cart totals (#11)](https://github.com/acme/shop/pull/11): expected 45.00, received 45.01. |
+| memory-question | pass | get_memory | memory | 0.0 s | 0.3 s | I remember 2 things. flaky spec: e2e/cart.spec.ts :: removes item (passed on retry). convention: Prices show two decimals. |
+| triage-view | pass | none | triage | 0.0 s | 0.0 s | 1 test failed or flaked in the last 2 runs of Review Jira (since 02:31 today). Suspected causes: 1 product. |
+| ongoing-work | pass | open_prs, runs_in_progress, pending_approvals | pulls, runs, approvals | 0.0 s | 0.2 s | 2 pull requests are open. 1 run is running now. 1 draft waits for your approval. |
+| flaky-by-hour | pass | test_results, compose_view | view | 0.0 s | 0.3 s | Flaky tests cluster at 14:00, with 3 of 5 flaky results in the last 14 days. The chart shows every hour. |
+| compare-branches | pass | test_results, compose_view | view | 0.0 s | 0.3 s | feat/search passed 100% of 7 tests this week and main passed 83.3% of 6, a gap of 16.7 points. |
+| slower-suites | pass | test_results, compose_view | view | 0.0 s | 0.3 s | 2 suites got slower since the last merge. e2e/cart.spec.ts went from 1.2 s to 3.1 s, up 159.4%. |
+| review-titled | pass | review_pr | none | 0.5 s | 0.9 s | Queued a PR check for [Controls keep their labels on one line (#39)](https://github.com/acme/shop/pull/39). The result lands here, and on the PR when there is o |
+| waiting-approval | pass | pending_approvals | approvals | 0.0 s | 0.1 s | 1 waiting for your approval: QA review for [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3). |
+| lead-team-status | pass | team_status | table | 0.0 s | 0.2 s | The team is failing. Review Jira: the last merge found a bug. |
+| lead-release | pass | none | release_call | 0.0 s | 0.0 s | Lead QA: Not ready. 0 of 1 requirement passing, 1 not tested. 1 thing blocks it. |
+| lead-rerun | pass | rerun_tests | none | 0.5 s | 0.9 s | Asked Review Jira to run the tests again. Queued a run of the tests for Review Jira. I will report back here and on your phone if anything fails. |
+| not-connected | pass | review_ticket | none | 0.0 s | 0.2 s | Linear is not connected. Connect it on the Setup page (API key), then ask again. |
+| other-repo | pass | review_ticket | none | 0.0 s | 0.3 s | Review Jira does not watch acme/other, so it cannot review that issue. Add the repo to the dot's GitHub Issues tracker. |
+<!-- eval:mock:end -->
+
+<!-- eval:codex-cli:start -->
+### codex-cli, 2026-10-13: 15 of 16 passed
+
+| Case | Result | Tools called | Blocks | First text | Total | Answer |
+|---|---|---|---|---|---|---|
+| ticket-link | pass | review_ticket | none | 21.8 s | 22.5 s | [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) needs attention: it has no acceptance criteria. - High risk: expired o |
+| ticket-key-question | pass | get_ticket | none | 17.7 s | 18.4 s | [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) is a High-priority Story: shoppers enter a discount code at checkout,  |
+| ticket-review-question | pass | get_ticket | ticket_review | 15.0 s | 15.7 s | The review found that [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) has no acceptance criteria. Discount stacking an |
+| pr-link | pass | review_pr | none | 15.3 s | 16.0 s | Review queued for [Fix cart total rounding (#12)](https://github.com/acme/shop/pull/12). Results are pending. |
+| prod-up | pass | probe_status | probes | 12.3 s | 13.3 s | The last probe was up: HTTP 200 with 0 consecutive failures. That check was on October 8, so it doesn’t confirm production is up now. |
+| what-broke | pass | recent_defects | none | 18.2 s | 18.9 s | One product failure since [Round cart totals (#11)](https://github.com/acme/shop/pull/11): the “checkout total” test expected 45.00 but received 45.01. The susp |
+| defects-question | pass | recent_defects | defects | 15.5 s | 16.5 s | Yes, 1 defect since [Round cart totals (#11)](https://github.com/acme/shop/pull/11): the checkout total test expected 45.00 but received 45.01. The suspected ca |
+| memory-question | pass | get_memory | memory | 0.0 s | 0.3 s | I remember 2 things. flaky spec: e2e/cart.spec.ts :: removes item (passed on retry). convention: Prices show two decimals. |
+| triage-view | pass | none | triage | 0.0 s | 0.0 s | 1 test failed or flaked in the last 2 runs of Review Jira (since 13:43 today). Suspected causes: 1 product. |
+| ongoing-work | pass | open_prs, pending_approvals, runs_in_progress | pulls, approvals, runs | 15.7 s | 16.5 s | There are 2 open PRs, with checks running for [Fix cart total rounding (#12)](https://github.com/acme/shop/pull/12) for 3 minutes and 0 jobs queued. QA review f |
+| review-titled | pass | review_pr | none | 0.5 s | 0.8 s | Queued a PR check for [Controls keep their labels on one line (#39)](https://github.com/acme/shop/pull/39). The result lands here, and on the PR when there is o |
+| bare-refs | fail: answer does not match /\(#39\)\]\(https:\/\/github\.com\/acme\/shop\/pull\/39\)/; answer does not match /\(#32\)\]\(https:\/\/github\.com\/acme\/shop\/pull\/32\)/ | list_runs, open_prs | none | 20.4 s | 21.3 s | I couldn’t confirm either was checked. Neither appears in the returned run history or the open pull requests for acme/shop. |
+| waiting-approval | pass | pending_approvals | approvals | 14.0 s | 15.1 s | 1 ticket comment is waiting for your approval: QA review for [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) ([Apply a |
+| do-not-review | pass | none | none | 5.3 s | 5.7 s | Understood. I won’t review that ticket. |
+| not-connected | pass | review_ticket | none | 18.4 s | 19.3 s | Linear is not connected, so I couldn’t review the ticket. Connect Linear with an API key on the Setup page, then ask me to check it again. |
+| other-repo | pass | review_ticket | none | 15.3 s | 16.3 s | Review Jira cannot review that issue because it does not watch acme/other. Add acme/other to the dot’s GitHub Issues tracker on the Setup page. |
+<!-- eval:codex-cli:end -->
+
+<!-- eval:claude-cli:start -->
+### claude-cli, 2026-10-13: 16 of 16 passed
+
+| Case | Result | Tools called | Blocks | First text | Total | Answer |
+|---|---|---|---|---|---|---|
+| ticket-link | pass | review_ticket | none | 8.7 s | 11.2 s | The review of [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) says it needs attention. The ticket has no acceptance cr |
+| ticket-key-question | pass | get_ticket | none | 4.9 s | 6.6 s | [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) is a High priority Story. Shoppers enter a discount code at checkout a |
+| ticket-review-question | pass | get_ticket | ticket_review | 6.2 s | 7.7 s | The review of [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3) found that the ticket has no acceptance criteria. It als |
+| pr-link | pass | review_pr | none | 6.5 s | 8.6 s | I queued a review of "Fix cart total rounding" ([Fix cart total rounding (#12)](https://github.com/acme/shop/pull/12)). Because the PR is open, it gets the PR c |
+| prod-up | pass | probe_status | probes | 5.9 s | 8.2 s | Prod looks up. The health check at https://shop.example.test/health returned 200 with 0 consecutive failures. That check is from 2026-10-08, five days ago, so i |
+| what-broke | pass | recent_defects | none | 8.3 s | 11.2 s | One thing broke since the last merge. The post-merge run after [Round cart totals (#11)](https://github.com/acme/shop/pull/11) found a product failure. - **Fail |
+| defects-question | pass | recent_defects | defects | 5.3 s | 8.0 s | Yes, one defect. The post-merge run after "Round cart totals" ([Round cart totals (#11)](https://github.com/acme/shop/pull/11)) found a product failure. - **Fai |
+| memory-question | pass | get_memory | memory | 0.0 s | 0.2 s | I remember 2 things. flaky spec: e2e/cart.spec.ts :: removes item (passed on retry). convention: Prices show two decimals. |
+| triage-view | pass | none | triage | 0.0 s | 0.0 s | 1 test failed or flaked in the last 2 runs of Review Jira (since 13:41 today). Suspected causes: 1 product. |
+| ongoing-work | pass | open_prs, runs_in_progress, pending_approvals | pulls, runs, approvals | 5.3 s | 7.6 s | Two PRs are open, one check is running, and one comment is waiting for your approval. - A check is running on "Fix cart total rounding" ([Fix cart total roundin |
+| review-titled | pass | review_pr | none | 0.5 s | 0.8 s | Queued a PR check for [Controls keep their labels on one line (#39)](https://github.com/acme/shop/pull/39). The result lands here, and on the PR when there is o |
+| bare-refs | pass | list_runs | none | 5.0 s | 7.5 s | I don't see any checks on either one. In the last 30 days there are only two runs, and both are post-merge runs for other PRs: [Round cart totals (#11)](https:/ |
+| waiting-approval | pass | pending_approvals | approvals | 6.4 s | 8.6 s | One item is waiting for your approval. It's a comment draft, "QA review for [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/T |
+| do-not-review | pass | none | none | 3.1 s | 6.4 s | Understood, I won't review [Apply a discount code at checkout (TAD-3)](https://acme-test.atlassian.net/browse/TAD-3). I haven't taken any action on it, and noth |
+| not-connected | pass | review_ticket | none | 4.6 s | 6.6 s | I can't review that ticket because Linear isn't connected. Add your Linear API key on the Setup page, then ask me again. |
+| other-repo | pass | review_ticket | none | 3.7 s | 6.2 s | I can't review that issue. This watch only covers acme/shop, not acme/other. To change that, add acme/other to the dot's GitHub Issues tracker on the Setup page |
+<!-- eval:claude-cli:end -->