@opensoftware/scorekit 0.0.0-stage → 0.4.0

package/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -1,3 +1,227 @@
-# Temporary Holding Version
+---
+covers: [modules/scorekit/src/**, modules/scorekit/scorers/**]
+---
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# scorekit
+
+Grade a team's agent sessions on concrete failure modes. A scorer is a Markdown file with a judge prompt, a pass
+criterion and a sampling rate. scorekit asks an LLM judge about each sampled run and writes one score per run, with
+pass or fail, the reason and the judge's cost. `calibrate` checks a judge against labels a person wrote before you
+trust it, and `report` shows each scorer's pass rate over time. `export` turns the
+scorers into Langfuse evaluators, so Langfuse can judge new runs as they arrive. Decisions:
+[ADR 0032](../../docs/decisions/0032-scorekit.md), [ADR 0033](../../docs/decisions/0033-scorekit-langfuse-evaluators.md).
+
+## Quick start
+
+This quick start is scenario `SCN-SCORE-001` and is tested on every change. Save two runs as runlog writes them
+([`modules/runlog`](../runlog/README.md)):
+
+```jsonl runs.jsonl
+{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"claude-code"}},{"key":"osf.run.id","value":{"stringValue":"run-01"}},{"key":"osf.harness.name","value":{"stringValue":"claude-code"}}]},"scopeSpans":[{"scope":{"name":"@opensoftware/runlog"},"spans":[{"traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"0af7651916cd43dd","name":"invoke_agent","startTimeUnixNano":"1767258000000000000","endTimeUnixNano":"1767258060000000000","attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"invoke_agent"}},{"key":"gen_ai.request.model","value":{"stringValue":"claude-sonnet-5-5"}},{"key":"osf.outcome","value":{"stringValue":"succeeded"}}]},{"traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"8448eb211c80319c","parentSpanId":"0af7651916cd43dd","name":"chat claude-sonnet-5-5","startTimeUnixNano":"1767258001000000000","endTimeUnixNano":"1767258003000000000","attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"chat"}},{"key":"gen_ai.input.messages","value":{"stringValue":"[{\"role\":\"user\",\"parts\":[{\"type\":\"text\",\"content\":\"Add a sum function with a test\"}]}]"}}]},{"traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"af7651916cd43dd8","parentSpanId":"0af7651916cd43dd","name":"execute_tool Write","startTimeUnixNano":"1767258004000000000","endTimeUnixNano":"1767258004500000000","attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"execute_tool"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Write"}},{"key":"gen_ai.tool.call.arguments","value":{"stringValue":"{\"file_path\":\"test/sum.test.ts\"}"}}]}]}]}]}
+{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"claude-code"}},{"key":"osf.run.id","value":{"stringValue":"run-02"}},{"key":"osf.harness.name","value":{"stringValue":"claude-code"}}]},"scopeSpans":[{"scope":{"name":"@opensoftware/runlog"},"spans":[{"traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"4bf92f3577b34da6","name":"invoke_agent","startTimeUnixNano":"1767258000000000000","endTimeUnixNano":"1767258060000000000","attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"invoke_agent"}},{"key":"gen_ai.request.model","value":{"stringValue":"claude-sonnet-5-5"}},{"key":"osf.outcome","value":{"stringValue":"succeeded"}}]},{"traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"a3ce929d0e0e4736","parentSpanId":"4bf92f3577b34da6","name":"chat claude-sonnet-5-5","startTimeUnixNano":"1767258001000000000","endTimeUnixNano":"1767258003000000000","attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"chat"}},{"key":"gen_ai.input.messages","value":{"stringValue":"[{\"role\":\"user\",\"parts\":[{\"type\":\"text\",\"content\":\"Fix the login redirect\"}]}]"}}]},{"traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"bf92f3577b34da6a","parentSpanId":"4bf92f3577b34da6","name":"execute_tool Write","startTimeUnixNano":"1767258004000000000","endTimeUnixNano":"1767258004500000000","attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"execute_tool"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Write"}},{"key":"gen_ai.tool.call.arguments","value":{"stringValue":"{\"file_path\":\"src/login.ts\"}"}}]}]}]}]}
+```
+
+Point scorekit at any OpenAI-compatible gateway, for example LiteLLM with a model alias `judge`, and score both runs:
+
+```sh
+export SCOREKIT_INFERENCE_URL=http://localhost:4000/v1
+npx @opensoftware/scorekit score --scorer redundant-tests --sample 1 runs.jsonl
+```
+
+It appends two lines to `$OUTPUT/scores.jsonl`: one score per run in the format of the `score` contract
+([`contracts/`](../../contracts/README.md)), with the value 1 for pass or 0 for fail and the judge's reason as
+`comment`. It prints `scored 2 of 2 runs with redundant-tests: ...`.
+
+## Commands
+
+| Command | What it does |
+| --- | --- |
+| `scorekit score --scorer <name or file> <runs>...` | Judge each sampled run that has no score of the scorer yet in `--out` (`$OUTPUT/scores.jsonl`) and append its score. `--seed <n>` (0) and `--sample <0..1>` change the sampling. |
+| `scorekit calibrate --scorer <name or file> --labels <file> <runs>...` | Judge every labeled run and write the agreement with the labels to `--out` (`$OUTPUT/calibration.json`). `--min-agreement <0..1>` exits 1 below it. |
+| `scorekit report <scores>...` | Write each scorer's runs, passes, fails, pass rate, cost, pass rate per day and scored runs to `--out` (`$OUTPUT/report.json`); `--markdown <file>` writes the same as Markdown tables. |
+| `scorekit push <scores>...` | Send the scores to Langfuse's score API (`POST /api/public/scores`). |
+| `scorekit export [--scorer <name or file>]...` | Write the scorers (default: the pack) as Langfuse LLM-as-a-judge evaluators with their evaluation rules to `--out` (`$OUTPUT/evaluators.json`). `--provider <name> --model <name>` set the evaluators' model. |
+| `scorekit export --push [--enable] [--scorer <name or file>]...` | Create or update the evaluators in Langfuse. A new evaluator gets a rule, disabled unless `--enable`. See [Langfuse evaluators](#langfuse-evaluators). |
+| `scorekit scorers` | List the scorers of the pack with their sampling rate and description. |
+
+`<runs>` are files of OTLP/JSON trace export requests, one per line, or folders: every `.jsonl` file below a folder
+is read. A run is the span `invoke_agent` with the other spans of its trace; its ID is `osf.run.id`. A wrong
+option, scorer file or input file exits 2 before any judge call. A failed judge call or an answer without a verdict
+is named on the error output; the other runs are still judged and the command exits 1.
+
+## Scorer files
+
+```markdown scorers/slow-tests.md
+---
+description: Tests wait for time to pass.
+pass: No test waits longer than one second.
+sample: 0.2
+model: judge
+price: { input: 3, output: 15 }
+---
+Look for sleep calls and fixed timeouts in the tests the agent wrote or ran.
+```
+
+| Key | Meaning |
+| --- | --- |
+| `name` | The scorer's name in each score. Default: the file name without `.md`. |
+| `description` | One line for `scorekit scorers`. |
+| `pass` | Required. The pass criterion, one sentence. |
+| `sample` | Share of runs to score, 0 to 1. Default 1. |
+| `model` | The judge model the gateway gets. Default `judge`. |
+| `price` | `input` and `output` in USD per million tokens. With it, each score records the judge cost from the token usage the gateway reports. |
+| `max_chars` | Longest transcript the judge reads. Default 200000. A longer transcript loses its middle, with a marker that says how many characters were cut. |
+
+The text after the front matter is the judge prompt. Any other key exits 2. In a factory definition,
+`scorers.<name>.judge` points at this file.
+
+## What the judge sees
+
+One chat completion per run, at temperature 0. The system message holds the prompt and the pass criterion and asks
+for one JSON object: `{"result": "pass" | "fail", "reason": "..."}`. The user message is the transcript: the run
+ID, harness, stage, model, outcome, pull requests, issues, duration and tokens, then each step in time order. A step
+is a message of a `chat` span when the run carries prompt content (runlog `--include-content`), or a tool call with
+its name and, when present, its arguments and result. Without prompt content the judge sees only the tool calls.
+
+## Sampling
+
+A run is scored when the first 32 bits of SHA-256 of `<seed>:<scorer name>:<run id>`, divided by 2^32, are below
+`sample`. So the same run is always in or out for a scorer and a seed, whatever the other runs are. With 100 runs
+and `sample: 0.2`, about 20 are scored. `score` skips a run that already has a score of the scorer in the output
+file, so a nightly job can run on all runs and only pays for new ones.
+
+## Calibration
+
+Labels are score records with the source `ANNOTATION`, as a person writes them in Langfuse or by hand: `name` is
+the scorer, `value` is 1 for pass or 0 for fail, and `metadata."osf.run.id"` names the run. `calibrate` judges
+every labeled run without sampling and writes:
+
+| Field | Meaning |
+| --- | --- |
+| `agreement` | Share of runs where the judge and the label agree. |
+| `kappa` | Cohen's kappa: agreement beyond chance; `null` when chance explains it all. |
+| `counts` | `truePass`, `falsePass` (judge pass, label fail), `trueFail`, `falseFail` (judge fail, label pass). |
+| `disagreements` | Each run where they differ, with the label, the result and the judge's reason. |
+| `missing` | Labels without a run in the runs files; they are named on the error output and not counted. |
+
+It prints `agreement 18 of 20 (90.0%), kappa 0.80`. With `--min-agreement 0.8` it exits 1 below 80 percent.
+
+## Scorer pack
+
+The package ships these scorers in [`scorers/`](scorers/). Use them by name or copy one as a start:
+
+| Scorer | Fails a run when |
+| --- | --- |
+| [`redundant-tests`](scorers/redundant-tests.md) | the agent added tests that prove nothing new |
+| [`change-requested`](scorers/change-requested.md) | a person had to ask for changes after the agent said it was done |
+| [`scope-creep`](scorers/scope-creep.md) | the agent changed more than the task asked for |
+| [`skipped-checks`](scorers/skipped-checks.md) | the agent claimed success without running the checks |
+| [`needless-human-question`](scorers/needless-human-question.md) | the agent asked a person something it could have found out itself |
+
+## Langfuse
+
+A score is a subset of Langfuse's create-score request, so `push` sends it as is to `$LANGFUSE_HOST/api/public/scores`
+with basic auth from `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY`. The score's `traceId` is the run's trace, so
+the score appears on the run in Langfuse when the run was sent there too. Langfuse keeps one score per ID, and a
+score's ID comes from the scorer and the run, so a second push changes nothing.
+
+## Langfuse evaluators
+
+`export` maps each scorer to Langfuse's public API v2 (ADR 0033):
+
+| Scorer | Evaluator in Langfuse |
+| --- | --- |
+| `name`, `description` | `name`, `description` of an `llm_as_judge` evaluator |
+| prompt and `pass` | a system message with the prompt and `Pass criterion: <pass>`, and a user message with `{{input}}` and `{{output}}`, mapped to the observation's input and output |
+| `pass` | a `BOOLEAN` output whose value instructions repeat the pass criterion |
+| `sample` | the sampling of the rule `scorekit <name>`, which matches the observations named `invoke_agent`, the root span of a run |
+| `model` | not exported: it is a gateway alias. `--provider` and `--model` set `modelConfig`; without them Langfuse uses the project's default evaluation model |
+
+```sh
+export LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-...
+npx @opensoftware/scorekit export --push --scorer redundant-tests --provider anthropic --model claude-haiku-5-5
+```
+
+`--push` lists the project's evaluators and takes the newest one with the scorer's name. When there is none, it
+creates the evaluator and its rule and prints `created <scorer>`. When the description, prompt, output and model
+are the same, it sends nothing and prints `unchanged <scorer>`. Otherwise it replaces the definition, which makes a
+new version in Langfuse with the same ID and rules, and prints `updated <scorer>`. An existing rule is never
+changed, so a sampling rate or filter a person set in Langfuse stays. The rule starts disabled, because an enabled
+rule makes Langfuse call a paid model for each new run; enable it in Langfuse or pass `--enable`.
+
+The evaluator sees only the root observation's input and output, not the transcript with every tool call that
+`score` builds. Calibrate the scores of each runtime before you trust them.
+
+## Releases
+
+release-please opens a release pull request from the Conventional Commits under `modules/scorekit/`. Merging it
+creates the tag `scorekit-v<version>`, and `.github/workflows/publish-scorekit.yml` builds `dist/` and publishes
+`@opensoftware/scorekit` to npm. The first release is `0.1.0`; until `1.0.0`, a breaking change raises the minor
+version. The one-time setup is the same as for contracts ([`contracts/README.md`](../../contracts/README.md#releases)),
+with a trusted publisher for `publish-scorekit.yml`. No credential is stored in the repository.
+
+## Configuration
+
+<!-- generated by make docs from src/config.schema.json: do not edit -->
+
+| Variable | Type | Default | Meaning |
+| --- | --- | --- | --- |
+| `OUTPUT` | string | `"."` | Folder of scores.jsonl, calibration.json, report.json and evaluators.json when --out is not given. |
+| `SCOREKIT_INFERENCE_URL` | string |  | Base URL of the OpenAI-compatible gateway of the judge, ending in /v1, for example http://litellm:4000/v1. Required by score and calibrate. |
+| `SCOREKIT_INFERENCE_KEY` | string |  | API key of the gateway, sent as a bearer token. Never write the value into a file. |
+| `LANGFUSE_HOST` | string | `"https://cloud.langfuse.com"` | Base URL of Langfuse for push and export --push. |
+| `LANGFUSE_PUBLIC_KEY` | string |  | Langfuse public key for push and export --push. |
+| `LANGFUSE_SECRET_KEY` | string |  | Langfuse secret key for push and export --push. Never write the value into a file. |
+| `OSF_NOW` | string |  | The time of new scores as an ISO 8601 timestamp. Default: the system clock. |
+
+## Scenarios
+
+<!-- generated by make docs from features/: do not edit -->
+
+| ID | Status | Scenario |
+| --- | --- | --- |
+| SCN-SCORE-001 | implemented | A new user scores the sample runs from the README |
+| SCN-SCORE-002 | implemented | The help lists every command |
+| SCN-SCORE-003 | implemented | A run the judge passes gets the score 1 with the reason |
+| SCN-SCORE-004 | implemented | A run the judge fails gets the score 0 |
+| SCN-SCORE-005 | implemented | The judge gets the scorer's prompt, its pass criterion and the run's tool calls |
+| SCN-SCORE-006 | implemented | The judge gets the run's messages when the run carries prompt content |
+| SCN-SCORE-007 | implemented | A scorer with a sampling rate of 20 percent scores the runs its seed picks |
+| SCN-SCORE-008 | implemented | The seed picks the same runs when the runs file lists them in another order |
+| SCN-SCORE-009 | implemented | The option "--sample" overrides the scorer's sampling rate |
+| SCN-SCORE-010 | implemented | The judge cost is recorded from the token usage and the scorer's price |
+| SCN-SCORE-011 | implemented | A run that already has a score of the scorer is not judged again |
+| SCN-SCORE-012 | implemented | An answer that is not a verdict gives no score and fails the command |
+| SCN-SCORE-013 | implemented | A failed judge call is named and the other runs are still scored |
+| SCN-SCORE-014 | implemented | A transcript longer than the scorer's limit loses its middle |
+| SCN-SCORE-015 | implemented | A scorer file with an unknown key is refused |
+| SCN-SCORE-016 | implemented | Without an inference URL the command names the missing variable |
+| SCN-SCORE-017 | implemented | The inference key is sent as a bearer token |
+| SCN-SCORE-018 | implemented | A folder of runs files is read |
+| SCN-SCORE-019 | implemented | Calibration reports the agreement with 20 labeled runs |
+| SCN-SCORE-020 | implemented | Calibration lists each disagreement with the label, the result and the reason |
+| SCN-SCORE-021 | implemented | Calibration below the minimum agreement fails the command |
+| SCN-SCORE-022 | implemented | A label without a run is named and not counted |
+| SCN-SCORE-023 | implemented | The report shows each scorer's pass rate, cost and scored runs |
+| SCN-SCORE-024 | implemented | The report shows the pass rate per day |
+| SCN-SCORE-025 | implemented | The report is written as Markdown |
+| SCN-SCORE-026 | implemented | Push sends every score to Langfuse with basic auth |
+| SCN-SCORE-027 | implemented | A score Langfuse refuses is named and fails the command |
+| SCN-SCORE-028 | implemented | The pack lists five scorers with their sampling rate and description |
+| SCN-SCORE-029 | implemented | An unknown scorer name is refused with the names of the pack |
+| SCN-SCORE-030 | implemented | A scorer file without a pass criterion is refused |
+| SCN-SCORE-031 | implemented | A scorer file whose front matter is not YAML is refused |
+| SCN-SCORE-032 | implemented | A runs file line that is not JSON is named with its file and line |
+| SCN-SCORE-033 | implemented | Export writes the five scorers of the pack as evaluators |
+| SCN-SCORE-034 | implemented | An exported evaluator carries the judge prompt and the pass criterion as a boolean score |
+| SCN-SCORE-035 | implemented | The evaluation rule samples the scorer's share of the runs and starts disabled |
+| SCN-SCORE-036 | implemented | The options "--provider" and "--model" set the evaluator's model |
+| SCN-SCORE-037 | implemented | Without "--model" the evaluator uses the project's default evaluation model |
+| SCN-SCORE-038 | implemented | The option "--model" without "--provider" is refused |
+| SCN-SCORE-039 | implemented | Push creates a missing evaluator and its disabled rule in Langfuse with basic auth |
+| SCN-SCORE-040 | implemented | Push with "--enable" creates the rule enabled |
+| SCN-SCORE-041 | implemented | Push leaves an evaluator alone when Langfuse already holds the same definition |
+| SCN-SCORE-042 | implemented | Push updates a changed evaluator and keeps its rule |
+| SCN-SCORE-043 | implemented | Push finds an evaluator on the second page of Langfuse's list |
+| SCN-SCORE-044 | implemented | An evaluator Langfuse refuses is named and fails the command |
+| SCN-SCORE-045 | implemented | Push without Langfuse keys names the missing variables |

package/dist/adapters/files.js

@@ -0,0 +1,53 @@
+// Files: runs files and folders, scorer files and the pack, score files.
+import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync } from "node:fs";
+import { basename, dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { readInputs, UsageError } from "@opensoftware/cli";
+import { validate } from "@opensoftware/contracts";
+import { jsonLines } from "../domain/lines.js";
+import { parseScorer } from "../domain/scorer.js";
+/** The scorers the package ships, in scorers/ next to dist/. */
+const PACK = fileURLToPath(new URL("../../scorers/", import.meta.url));
+/** Each runs file named, and every .jsonl file below each folder named, sorted by path. */
+export function readRunFiles(paths) {
+    if (!paths.length)
+        throw new UsageError("name at least one runs file or folder");
+    return readInputs(paths, /\.jsonl$/);
+}
+export function packNames() {
+    return readdirSync(PACK)
+        .filter((file) => file.endsWith(".md"))
+        .map((file) => file.slice(0, -3))
+        .sort();
+}
+/** A scorer by pack name, or by path when the value names a Markdown file or a folder. */
+export function readScorer(value) {
+    if (!value)
+        throw new UsageError("--scorer is required: a pack name or a scorer file");
+    const isPath = value.endsWith(".md") || value.includes("/");
+    const file = isPath ? value : join(PACK, `${value}.md`);
+    if (!existsSync(file))
+        throw new UsageError(isPath ? `${value}: no such scorer file` : `no scorer "${value}" in the pack: ${packNames().join(", ")}`);
+    return parseScorer(isPath ? value : `${value}.md`, basename(file, ".md"), readFileSync(file, "utf8"));
+}
+/** The score records of score files, each checked against the `score` contract. A missing file has none. */
+export function readScores(paths, { missingOk = false } = {}) {
+    return paths.flatMap((path) => {
+        if (missingOk && !existsSync(path))
+            return [];
+        return jsonLines(path, readFileSync(path, "utf8")).map(({ line, value }) => {
+            const [problem] = validate("score", value);
+            if (problem)
+                throw new UsageError(`${path}:${line}: not a score: ${problem.path} ${problem.message}`);
+            return value;
+        });
+    });
+}
+export function appendScore(path, score) {
+    const problems = validate("score", score);
+    if (problems.length)
+        throw new Error(`a score breaks the contract: ${problems[0]?.path} ${problems[0]?.message}`);
+    mkdirSync(dirname(path), { recursive: true });
+    appendFileSync(path, `${JSON.stringify(score)}\n`);
+}
+//# sourceMappingURL=files.js.map

package/dist/adapters/files.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"files.js","sourceRoot":"","sources":["../../src/adapters/files.ts"],"names":[],"mappings":"AAAA,yEAAyE;AACzE,OAAO,EAAE,cAAc,EAAE,UAAU,EAAE,SAAS,EAAE,WAAW,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AAC3F,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACpD,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAc,UAAU,EAAE,UAAU,EAAE,MAAM,mBAAmB,CAAC;AACvE,OAAO,EAAc,QAAQ,EAAE,MAAM,yBAAyB,CAAC;AAC/D,OAAO,EAAE,SAAS,EAAE,MAAM,oBAAoB,CAAC;AAC/C,OAAO,EAAE,WAAW,EAAe,MAAM,qBAAqB,CAAC;AAE/D,gEAAgE;AAChE,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,GAAG,CAAC,gBAAgB,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;AAEvE,2FAA2F;AAC3F,MAAM,UAAU,YAAY,CAAC,KAAe;IAC1C,IAAI,CAAC,KAAK,CAAC,MAAM;QAAE,MAAM,IAAI,UAAU,CAAC,uCAAuC,CAAC,CAAC;IACjF,OAAO,UAAU,CAAC,KAAK,EAAE,UAAU,CAAC,CAAC;AACvC,CAAC;AAED,MAAM,UAAU,SAAS;IACvB,OAAO,WAAW,CAAC,IAAI,CAAC;SACrB,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC;SACtC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;SAChC,IAAI,EAAE,CAAC;AACZ,CAAC;AAED,0FAA0F;AAC1F,MAAM,UAAU,UAAU,CAAC,KAAyB;IAClD,IAAI,CAAC,KAAK;QAAE,MAAM,IAAI,UAAU,CAAC,oDAAoD,CAAC,CAAC;IACvF,MAAM,MAAM,GAAG,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;IAC5D,MAAM,IAAI,GAAG,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,KAAK,KAAK,CAAC,CAAC;IACxD,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;QACnB,MAAM,IAAI,UAAU,CAClB,MAAM,CAAC,CAAC,CAAC,GAAG,KAAK,uBAAuB,CAAC,CAAC,CAAC,cAAc,KAAK,kBAAkB,SAAS,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACzG,CAAC;IACJ,OAAO,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,KAAK,KAAK,EAAE,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,EAAE,YAAY,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC;AACxG,CAAC;AAED,4GAA4G;AAC5G,MAAM,UAAU,UAAU,CAAC,KAAe,EAAE,EAAE,SAAS,GAAG,KAAK,EAAE,GAAG,EAAE;IACpE,OAAO,KAAK,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,EAAE;QAC5B,IAAI,SAAS,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;YAAE,OAAO,EAAE,CAAC;QAC9C,OAAO,SAAS,CAAC,IAAI,EAAE,YAAY,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,EAAE,EAAE;YACzE,MAAM,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;YAC3C,IAAI,OAAO;gBAAE,MAAM,IAAI,UAAU,CAAC,GAAG,IAAI,IAAI,IAAI,kBAAkB,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC,CAAC;YACtG,OAAO,KAAc,CAAC;QACxB,CAAC,CAAC,CAAC;IACL,CAAC,CAAC,CAAC;AACL,CAAC;AAED,MAAM,UAAU,WAAW,CAAC,IAAY,EAAE,KAAY;IACpD,MAAM,QAAQ,GAAG,QAAQ,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;IAC1C,IAAI,QAAQ,CAAC,MAAM;QAAE,MAAM,IAAI,KAAK,CAAC,gCAAgC,QAAQ,CAAC,CAAC,CAAC,EAAE,IAAI,IAAI,QAAQ,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC;IAClH,SAAS,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IAC9C,cAAc,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;AACrD,CAAC"}

package/dist/adapters/judge.js

@@ -0,0 +1,23 @@
+export class JudgeError extends Error {
+    name = "JudgeError";
+}
+export async function askJudge(gateway, model, messages) {
+    const response = await fetch(`${gateway.url.replace(/\/+$/, "")}/chat/completions`, {
+        method: "POST",
+        headers: {
+            "content-type": "application/json",
+            ...(gateway.key ? { authorization: `Bearer ${gateway.key}` } : {}),
+        },
+        body: JSON.stringify({ model, temperature: 0, messages }),
+    });
+    if (!response.ok)
+        throw new JudgeError(`the judge call failed with HTTP ${response.status}`);
+    const body = (await response.json().catch(() => ({})));
+    const usage = body.usage;
+    return {
+        text: body.choices?.[0]?.message?.content ?? "",
+        model: body.model ?? model,
+        usage: usage ? { input: usage.prompt_tokens ?? 0, output: usage.completion_tokens ?? 0 } : undefined,
+    };
+}
+//# sourceMappingURL=judge.js.map

package/dist/adapters/judge.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"judge.js","sourceRoot":"","sources":["../../src/adapters/judge.ts"],"names":[],"mappings":"AASA,MAAM,OAAO,UAAW,SAAQ,KAAK;IAC1B,IAAI,GAAG,YAAY,CAAC;CAC9B;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAC5B,OAAsC,EACtC,KAAa,EACb,QAAmB;IAEnB,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,mBAAmB,EAAE;QAClF,MAAM,EAAE,MAAM;QACd,OAAO,EAAE;YACP,cAAc,EAAE,kBAAkB;YAClC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,UAAU,OAAO,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SACnE;QACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,WAAW,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC;KAC1D,CAAC,CAAC;IACH,IAAI,CAAC,QAAQ,CAAC,EAAE;QAAE,MAAM,IAAI,UAAU,CAAC,mCAAmC,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;IAC7F,MAAM,IAAI,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAIpD,CAAC;IACF,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC;IACzB,OAAO;QACL,IAAI,EAAE,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,IAAI,EAAE;QAC/C,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,KAAK;QAC1B,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,KAAK,CAAC,aAAa,IAAI,CAAC,EAAE,MAAM,EAAE,KAAK,CAAC,iBAAiB,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,SAAS;KACrG,CAAC;AACJ,CAAC"}

package/dist/adapters/langfuse.js

@@ -0,0 +1,54 @@
+// Langfuse's public API with basic auth: the score API (ADR 0032) and the v2 evaluator and evaluation-rule API
+// (ADR 0033). Langfuse keeps one score per ID.
+import { UsageError } from "@opensoftware/cli";
+/** Langfuse from LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY. */
+export function langfuseFrom(env) {
+    const { LANGFUSE_PUBLIC_KEY: publicKey, LANGFUSE_SECRET_KEY: secretKey } = env;
+    if (!publicKey || !secretKey)
+        throw new UsageError("set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY");
+    return { host: env.LANGFUSE_HOST || "https://cloud.langfuse.com", publicKey, secretKey };
+}
+async function call(langfuse, method, path, body) {
+    const auth = Buffer.from(`${langfuse.publicKey}:${langfuse.secretKey}`).toString("base64");
+    const response = await fetch(`${langfuse.host.replace(/\/+$/, "")}${path}`, {
+        method,
+        headers: { "content-type": "application/json", authorization: `Basic ${auth}` },
+        body: body === undefined ? undefined : JSON.stringify(body),
+    });
+    if (!response.ok)
+        return { problem: `Langfuse answered HTTP ${response.status}` };
+    return { body: (await response.json().catch(() => ({}))) };
+}
+export async function pushScore(langfuse, score) {
+    return (await call(langfuse, "POST", "/api/public/scores", score)).problem;
+}
+/** Every evaluator of the project, newest first, following the cursor. */
+export async function listEvaluators(langfuse) {
+    const evaluators = [];
+    let cursor;
+    do {
+        const query = cursor === undefined ? "" : `&cursor=${encodeURIComponent(cursor)}`;
+        const { problem, body } = await call(langfuse, "GET", `/api/public/v2/evaluators?limit=100${query}`);
+        if (problem)
+            throw new Error(`listing evaluators: ${problem}`);
+        evaluators.push(...(body?.data ?? []));
+        const next = body?.meta?.cursor;
+        cursor = typeof next === "string" && next ? next : undefined;
+    } while (cursor !== undefined);
+    return evaluators;
+}
+/** Creates the evaluator; its ID, or the problem. */
+export async function createEvaluator(langfuse, evaluator) {
+    return call(langfuse, "POST", "/api/public/v2/evaluators", evaluator);
+}
+/** Replaces the evaluator's definition, which makes a new version. */
+export async function updateEvaluator(langfuse, id, evaluator) {
+    return call(langfuse, "PATCH", `/api/public/v2/evaluators/${encodeURIComponent(id)}`, evaluator);
+}
+export async function createRule(langfuse, rule, evaluatorId) {
+    return call(langfuse, "POST", "/api/public/v2/evaluation-rules", {
+        ...rule,
+        evaluatorAssignments: [{ evaluatorId }],
+    });
+}
+//# sourceMappingURL=langfuse.js.map

package/dist/adapters/langfuse.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"langfuse.js","sourceRoot":"","sources":["../../src/adapters/langfuse.ts"],"names":[],"mappings":"AAAA,+GAA+G;AAC/G,+CAA+C;AAE/C,OAAO,EAAE,UAAU,EAAE,MAAM,mBAAmB,CAAC;AAU/C,gFAAgF;AAChF,MAAM,UAAU,YAAY,CAAC,GAAsB;IACjD,MAAM,EAAE,mBAAmB,EAAE,SAAS,EAAE,mBAAmB,EAAE,SAAS,EAAE,GAAG,GAAG,CAAC;IAC/E,IAAI,CAAC,SAAS,IAAI,CAAC,SAAS;QAAE,MAAM,IAAI,UAAU,CAAC,iDAAiD,CAAC,CAAC;IACtG,OAAO,EAAE,IAAI,EAAE,GAAG,CAAC,aAAa,IAAI,4BAA4B,EAAE,SAAS,EAAE,SAAS,EAAE,CAAC;AAC3F,CAAC;AAID,KAAK,UAAU,IAAI,CAAC,QAAkB,EAAE,MAAc,EAAE,IAAY,EAAE,IAAc;IAClF,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,GAAG,QAAQ,CAAC,SAAS,IAAI,QAAQ,CAAC,SAAS,EAAE,CAAC,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;IAC3F,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,GAAG,IAAI,EAAE,EAAE;QAC1E,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,aAAa,EAAE,SAAS,IAAI,EAAE,EAAE;QAC/E,IAAI,EAAE,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;KAC5D,CAAC,CAAC;IACH,IAAI,CAAC,QAAQ,CAAC,EAAE;QAAE,OAAO,EAAE,OAAO,EAAE,0BAA0B,QAAQ,CAAC,MAAM,EAAE,EAAE,CAAC;IAClF,OAAO,EAAE,IAAI,EAAE,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAA4B,EAAE,CAAC;AACxF,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,SAAS,CAAC,QAAkB,EAAE,KAAY;IAC9D,OAAO,CAAC,MAAM,IAAI,CAAC,QAAQ,EAAE,MAAM,EAAE,oBAAoB,EAAE,KAAK,CAAC,CAAC,CAAC,OAAO,CAAC;AAC7E,CAAC;AAED,0EAA0E;AAC1E,MAAM,CAAC,KAAK,UAAU,cAAc,CAAC,QAAkB;IACrD,MAAM,UAAU,GAA8B,EAAE,CAAC;IACjD,IAAI,MAA0B,CAAC;IAC/B,GAAG,CAAC;QACF,MAAM,KAAK,GAAG,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,WAAW,kBAAkB,CAAC,MAAM,CAAC,EAAE,CAAC;QAClF,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,GAAG,MAAM,IAAI,CAAC,QAAQ,EAAE,KAAK,EAAE,sCAAsC,KAAK,EAAE,CAAC,CAAC;QACrG,IAAI,OAAO;YAAE,MAAM,IAAI,KAAK,CAAC,uBAAuB,OAAO,EAAE,CAAC,CAAC;QAC/D,UAAU,CAAC,IAAI,CAAC,GAAI,CAAC,IAAI,EAAE,IAAI,IAAI,EAAE,CAA+B,CAAC,CAAC;QACtE,MAAM,IAAI,GAAI,IAAI,EAAE,IAA4C,EAAE,MAAM,CAAC;QACzE,MAAM,GAAG,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;IAC/D,CAAC,QAAQ,MAAM,KAAK,SAAS,EAAE;IAC/B,OAAO,UAAU,CAAC;AACpB,CAAC;AAED,qDAAqD;AACrD,MAAM,CAAC,KAAK,UAAU,eAAe,CAAC,QAAkB,EAAE,SAAoB;IAC5E,OAAO,IAAI,CAAC,QAAQ,EAAE,MAAM,EAAE,2BAA2B,EAAE,SAAS,CAAC,CAAC;AACxE,CAAC;AAED,sEAAsE;AACtE,MAAM,CAAC,KAAK,UAAU,eAAe,CAAC,QAAkB,EAAE,EAAU,EAAE,SAAoB;IACxF,OAAO,IAAI,CAAC,QAAQ,EAAE,OAAO,EAAE,6BAA6B,kBAAkB,CAAC,EAAE,CAAC,EAAE,EAAE,SAAS,CAAC,CAAC;AACnG,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,UAAU,CAAC,QAAkB,EAAE,IAAU,EAAE,WAAmB;IAClF,OAAO,IAAI,CAAC,QAAQ,EAAE,MAAM,EAAE,iCAAiC,EAAE;QAC/D,GAAG,IAAI;QACP,oBAAoB,EAAE,CAAC,EAAE,WAAW,EAAE,CAAC;KACxC,CAAC,CAAC;AACL,CAAC"}