dokimi 0.1.0__tar.gz

dokimi-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Intel Corporation
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

dokimi-0.1.0/NOTICE

@@ -0,0 +1,6 @@
+dokimi
+Copyright 2026 Intel Corporation
+
+This product is licensed under the Apache License, Version 2.0 (the "License").
+You may obtain a copy of the License in the LICENSE file in this repository or
+at http://www.apache.org/licenses/LICENSE-2.0.

dokimi-0.1.0/PKG-INFO

@@ -0,0 +1,402 @@
+Metadata-Version: 2.4
+Name: dokimi
+Version: 0.1.0
+Summary: Measures whether a skill, MCP doc source, documentation set or agent profile actually improves AI-agent answers and task outcomes
+Author: Intel Corporation
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/intel/dokimi
+Project-URL: Documentation, https://intel.github.io/dokimi/
+Project-URL: Repository, https://github.com/intel/dokimi
+Project-URL: Issues, https://github.com/intel/dokimi/issues
+Project-URL: Changelog, https://github.com/intel/dokimi/blob/main/dokimi/CHANGELOG.md
+Keywords: documentation,benchmark,evaluation,llm,rag
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: PyYAML<7,>=6.0.1
+Requires-Dist: jsonschema<5,>=4.20
+Requires-Dist: litellm>=1.105.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: scipy>=1.11.0
+Requires-Dist: PyGithub>=2.1.0
+Requires-Dist: datasets<6,>=2.20
+Requires-Dist: openai<3,>=1.51
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
+Requires-Dist: pytest-socket>=0.7; extra == "dev"
+Requires-Dist: pytest-timeout>=2.2; extra == "dev"
+Requires-Dist: pytest-randomly>=3.15; extra == "dev"
+Requires-Dist: pytest-xdist>=3.6; extra == "dev"
+Requires-Dist: diff-cover>=10.0; extra == "dev"
+Requires-Dist: ruff==0.17.0; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: types-PyYAML; extra == "dev"
+Requires-Dist: types-jsonschema; extra == "dev"
+Requires-Dist: scriv>=1.5; extra == "dev"
+Requires-Dist: dokimi[ragas]; extra == "dev"
+Provides-Extra: ragas
+Requires-Dist: ragas<0.4,>=0.3.0; extra == "ragas"
+Requires-Dist: langchain-community<0.4.2; extra == "ragas"
+Requires-Dist: langchain-anthropic<2,>=1.0; extra == "ragas"
+Requires-Dist: langchain-openai<2,>=1.0; extra == "ragas"
+Requires-Dist: langchain-core<2,>=1.0; extra == "ragas"
+Provides-Extra: ocr
+Requires-Dist: docling>=2.0; extra == "ocr"
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0; extra == "mcp"
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=2.2; extra == "embeddings"
+Provides-Extra: typed-scorer
+Requires-Dist: laya>=0.3.24; extra == "typed-scorer"
+Requires-Dist: torch>=2.0; extra == "typed-scorer"
+Dynamic: license-file
+
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Dokimi
+
+> **This is the whole project.** It evaluates whether a given skill / MCP doc
+> source / agent profile actually improves agent answers, and produces the
+> evidence (a scorecard) that decides whether the artifact is worth shipping.
+> See the [repository overview](https://github.com/intel/dokimi/blob/main/README.md) for the short version.
+>
+> **Documentation: <https://intel.github.io/dokimi/>** (the [`docs/`](https://github.com/intel/dokimi/blob/main/dokimi/docs/README.md)
+> pages plus a generated CLI and API reference).
+
+`dokimi` measures whether a skill, MCP doc source, documentation set or
+agent profile actually improves a model's or coding agent's answers and task
+outcomes. It runs the same question or task set with and without one controlled
+context layer and reports the delta together with its significance.
+
+Context layers it can measure: local Markdown/URL documentation, Context7 and
+real MCP servers, skills (`SKILL.md` and bundles), agent profiles (system
+prompts), plugins, and whole code repositories with their own tasks and graders
+(manifest validation and pinned sources today; the task-run contract is proposed).
+
+Evidence tracks:
+
+| Track | Question it answers | Guide |
+|---|---|---|
+| Context benchmark | Do answers to a fixed question set improve with the artifact? | [user-flow.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/user-flow.md), [quickstart.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/quickstart.md) |
+| Treatment arms | Which of several docs / MCP / skill / profile arms helps, and by how much? | [evaluating-treatments.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/evaluating-treatments.md) |
+| Executable tasks | Does the agent produce code that builds and passes tests in Docker? | [coding-harnesses.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/coding-harnesses.md) |
+| Exploration (ExploreBench) | Can the agent locate the right files and line ranges? | [external-content.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/external-content.md) |
+| Content novelty | What does a skill *add* to a given model, and what can be cut? | [skill-novelty.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/skill-novelty.md) |
+| Repository subjects | How well does an agent work inside a given repository? | [repo-evaluation-contract.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/repo-evaluation-contract.md) |
+| Context window | What did the agent's context hold: start-up load, growth, compactions, tool-output weight? | [context-window.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/context-window.md) |
+| Static docs checks | Are the docs well-structured, fresh, readable, with runnable examples? | [Static benchmark metrics](#static-benchmark-metrics) |
+
+### Three questions you can ask
+
+The arm comparison is the default, and two tracks ask a different question about
+the same material. Pick by what you are about to do with the answer:
+
+| Question | Use | Command |
+|---|---|---|
+| Does this artifact help, and is the delta real? | treatment arms | `cli.py arms run` |
+| *Which parts* of it help, and which parts does the model already know? | content novelty | `cli.py novelty analyze` |
+| Does a repository's own guidance reach the agent working in it? | repository as subject | `cli.py repo reach` |
+
+The first ships or cuts an artifact; the second edits one; the third measures a
+codebase instead of a document.
+[docs/README.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/README.md) explains the three in full and
+[docs/architecture.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/architecture.md) maps them to modules.
+
+The framework is content-agnostic: question sets, tasks, skills and products are
+passed in by path or registered in `products.yaml`. The bundled registry uses
+oneTBB, oneDAL and oneMKL as worked examples.
+
+The project is pre-1.0; interfaces and artifact schemas may still change.
+
+## Requirements and setup
+
+- Python 3.11–3.14.
+- API keys for the models you run, read from the environment:
+  `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or any other
+  [LiteLLM](https://docs.litellm.ai/docs/providers) provider (for example
+  `AWS_REGION` + `AWS_BEARER_TOKEN_BEDROCK` for Bedrock). `GITHUB_TOKEN` is
+  optional and raises the persona-discovery rate limit.
+- Docker, only for the executable-task, harness and repository tracks.
+- Optional extras: `ragas` (`eval ragas`, `answers generate --ragas`), `ocr`
+  (Docling ingestion) and `mcp` (real MCP servers).
+
+```bash
+pip install dokimi          # the released package and the `dokimi` command
+```
+
+From a source checkout (to contribute, or to run the bundled tasks):
+
+```bash
+pip install -e ".[dev]"     # installs the `dokimi` command plus test tooling
+# or, without installing the package:
+pip install -r requirements.txt -r requirements-test.txt
+```
+
+The examples below use `python cli.py …` from a checkout; the installed
+`dokimi …` command is equivalent. For what a run costs in API calls,
+see the [cost estimate](https://github.com/intel/dokimi/blob/main/dokimi/docs/quickstart.md#cost-estimate).
+
+## Quick start
+
+Run the canonical context benchmark:
+
+```bash
+python cli.py benchmark preflight --target onetbb
+python cli.py benchmark run --target onetbb
+```
+
+Outputs go to `results/onetbb_final/` by default. The default comparison is one fixed semantic retrieval config (`top_k=3`) with `without_docs` vs `with_docs` arms.
+
+Optional static documentation gate for Markdown/code example quality:
+
+```bash
+python cli.py run \
+  --root . \
+  --spec benchmarks/spec.v1.yaml \
+  --out-json baselines/current.json \
+  --out-md reports/current.md
+```
+
+Run the test suite:
+
+```bash
+python -m pytest -q
+```
+
+For the LLM evaluation pipeline (answers, judging, dashboard), see [docs/quickstart.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/quickstart.md). For benchmark invocation and comparison recipes, see [docs/benchmarking-and-comparison.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/benchmarking-and-comparison.md). A high-level architecture map is in [docs/architecture.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/architecture.md), and [docs/README.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/README.md) is the documentation index.
+
+## Main CLI areas
+
+The top-level CLI is `python cli.py` (or `dokimi` once installed):
+
+```text
+Context benchmark
+  benchmark           Preflight and run the benchmark for registered targets
+  evaluate            Run the full pipeline in one command
+  personas            Discover and approve target user personas
+  questions           Generate, analyze, refine, and panel-review questions
+  answers             Generate context-arm and baseline answers
+  eval                Score answers, run RAGAS, or use a multi-judge panel
+  arms                Compare treatment arms (docs / MCP / skills / profiles)
+  subjects            Run a subject descriptor (artifact + suite) into a scorecard
+
+Executable and agentic tracks
+  tasks               Run Docker coding tasks through Codex, Claude Code or Harbor harnesses
+  explore             Run file/line localization tasks (ExploreBench)
+  repo                Validate a repository's repo-eval.yaml, fetch pinned sources, probe guidance reach
+  catalog             Check which skills/tasks in an external checkout load cleanly
+  context             Read a run's context window from its trace (no LLM call)
+
+Content analysis
+  novelty             Measure what an artifact's content adds to a model
+  run                 Run the static documentation benchmark
+  ingest              Convert PDF/Office/HTML to Markdown via Docling
+
+Results and registries
+  report              Render reports (incl. model-compare)
+  compare             Compare static benchmark snapshots
+  baseline            Save and compare evaluation baselines
+  dashboard           Generate dashboard Markdown/JSON
+  dataset             Export artifacts to JSONL/Parquet/Hugging Face
+  library             List/show registered products and intent domains (alias: product)
+```
+
+That is the curated set; `python cli.py --help` lists every group.
+
+Use `--help` on any command for detailed options, for example:
+
+```bash
+python cli.py questions --help
+python cli.py eval --help
+python cli.py dashboard generate --help
+```
+
+For end-to-end benchmark and comparison commands, including static snapshots,
+LLM context-arm vs baseline runs, fair multi-model comparisons, treatment arms,
+dashboards, and baselines, see
+[docs/benchmarking-and-comparison.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/benchmarking-and-comparison.md).
+
+## Product and intent registries
+
+Two YAML registries, bundled in the package (`dokimi/products.yaml`,
+`dokimi/intents.yaml`), describe what the benchmark can evaluate:
+
+- **`products.yaml`** — *what products exist*: identity, description, repo, and
+  doc sources for each product (oneTBB, oneMKL, VTune, dpnp, …). Used by
+  `python cli.py library …` (alias: `product`) and `python cli.py benchmark …`.
+- **`intents.yaml`** — *what user problems exist*: broad scenario domains
+  (data science, machine learning, performance optimization, debugging, …),
+  each mapped to the products that serve it, with example user intents. A
+  domain may span several products, so a domain is the unit you pick when you
+  want to know which product an agent should reach for.
+
+The domain→product mapping lives only in `intents.yaml`;
+`python -m dokimi.config_check` (run in CI) fails if it references an
+unregistered product, or if `dokimi/config/products.yaml` drifts from `products.yaml`.
+Explore the mapping with:
+
+```bash
+python cli.py library intents                       # list domains + products
+python cli.py library list --domain data-science    # products for a domain
+python cli.py library show vtune                    # includes the product's domains
+```
+
+## Static benchmark metrics
+
+The default static benchmark is configured in `benchmarks/spec.v1.yaml` and currently uses four active metrics:
+
+| Metric | Purpose |
+| --- | --- |
+| `coverage` | Checks Markdown structure: headings, code blocks, and body content. |
+| `freshness_lite` | Scores files by modification age. |
+| `readability` | Estimates text readability with a Flesch-Kincaid-style score. |
+| `example_pass_rate` | Executes supported fenced code examples in isolation. |
+
+Weights are normalized across enabled metrics. The benchmark also supports soft gates, hard gates, critical bands, and regression thresholds. Hard gates and critical bands are enforced only when `--strict` is passed.
+
+## LLM and treatment evaluation workflow
+
+The LLM workflow is designed to answer a practical question: does documentation, a skill, an MCP source, or an agent profile improve model answers and task outcomes?
+
+A typical one-command benchmark flow is:
+
+```bash
+python cli.py benchmark preflight --target onetbb
+python cli.py benchmark run --target onetbb --multi-run 3
+```
+
+The pipeline reads a golden set under `--questions-root` or a set from `--questions-from`, or else discovers (or loads) personas and generates (or reuses) questions from them; it then answers each question in context-arm and baseline conditions, scores both answers with an independent judge, and writes model metadata, token usage, `question_set_hash`, reports, and trust-gate signals. For compatibility, persisted answer/evaluation JSON still uses the legacy `with_docs` and `without_docs` keys for those two conditions.
+
+For fair multi-model comparisons, generate one question set and reuse it:
+
+```bash
+python cli.py benchmark run --target onedal --output-dir results/onedal_seed
+python cli.py benchmark run --target onedal --questions-from results/onedal_seed --model gpt-6-astra
+python cli.py benchmark run --target onedal --questions-from results/onedal_seed --model claude-opus-5-5 --provider anthropic
+```
+
+Generated artifacts — answer runs, judge scores and benchmark snapshots — go under `results/`, `reports/`, `answers/`, `eval/` or `baselines/`; those paths are git-ignored and no captured run is committed. Curated, version-controlled *inputs* live under `data/` (`data/skills/fastcontext/`, `data/agent_profiles/`, `data/explore_tasks/`) — see [`data/README.md`](https://github.com/intel/dokimi/blob/main/dokimi/data/README.md). Product skills, question sets and personas are evaluation content: they come from a separate content repository and are passed in by path (`skill:<path>`, `--questions <path>`, `--questions-root <dir>`).
+
+See [docs/methodology.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/methodology.md) for the detailed model roles, question types, difficulty levels, task workflow, and trust checks.
+
+## Executable tasks
+
+Terminal-bench-style tasks are the executable side of the benchmark: they validate not just text answers but working code and measurable behavior, so they can test whether docs, skills, or an agent profile actually improve an agent's ability to produce correct and performant code.
+
+The task definitions are evaluation content and live in a separate content repository. Run them from any checkout by naming its task root:
+
+```bash
+python cli.py tasks run --tasks-root <checkout>/evaluation/harbor/tasks \
+  --tasks <task-name> --harnesses docker-oracle --out-json results/runs.json
+```
+
+The directory layout and `task.toml` fields a task must provide are specified in [docs/contributing-terminal-bench-task.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/contributing-terminal-bench-task.md). CI runs a real external catalog through the same harness (see [CI](#ci) below) with network disabled at solve and verify time.
+
+## Repository layout
+
+```text
+cli.py                   Single entry point for all CLI commands
+benchmarks/              Static benchmark spec (its schema is dokimi/runner/spec.schema.json)
+changelog.d/             Changelog fragments, one per PR (see changelog.d/README.md)
+dokimi/                  Main Python package
+  products.yaml          Product registry: identity + doc sources per product
+  intents.yaml           Intent registry: problem/task domains mapped to products
+  config/                LLM/runtime configuration for the eval track
+  agent_profiles/        Agent persona prompt loader
+  commands/              CLI subcommand definitions
+  context/               Context window of a run, read from its harness trace
+  dashboard/             Dashboard aggregation/rendering
+  eval/                  Answer generation, judges, panels, RAGAS, arm runner
+  explore/               ExploreBench localization track
+  external/              External skill/task catalog scanning
+  gate/                  Soft/hard gates and regression classification
+  harnesses/             Coding-agent harness adapters (Codex, Claude Code, Harbor)
+  ingest/                Markdown loading and chunking
+  mcp/                   Documentation source clients (incl. MCP protocol)
+  metrics/               Static documentation metrics
+  novelty/               Content-novelty engine (atoms, probes, shortening)
+  orchestrator/          Pipeline orchestration
+  personas/              Persona generation/analysis (synthetic users)
+  questions/             Question generation, validation, refinement
+  repos/                 Repository subjects: manifest, sources, validation, harness profiles, reach probe
+  report/                JSON/Markdown report generation
+  runner/                Benchmark orchestration and the spec schema
+  schemas/               JSON Schemas for every persisted artifact
+  skills/                SKILL.md loader
+  subjects/              Subject descriptors and scorecards
+  treatments/            Treatment-arm abstraction (docs/MCP/skills/profiles)
+data/                    Curated, version-controlled benchmark inputs (see data/README.md)
+  skills/fastcontext/    The SKILL.md fixture the loader tests and examples use
+  agent_profiles/        Agent persona (system) prompt fixtures
+  explore_tasks/         Localization tasks with known-good references
+docs/                    Documentation (full index: docs/README.md)
+  README.md              Concept overview and documentation index
+  quickstart.md          End-to-end LLM evaluation pipeline
+  external-content.md    Pointing the framework at your own questions, tasks, skills
+  evaluating-treatments.md
+                         Comparing docs / MCP / skill / profile arms
+  coding-harnesses.md    Executable tasks through coding-agent harnesses
+  context-window.md      What a run's context window held, from its trace
+  skill-novelty.md       What a skill adds to a model
+  repo-evaluation-contract.md
+                         Evaluating a whole repository as a subject
+  architecture.md        Module map and data flow
+  adding-doc-source.md   Local/URL/custom doc sources
+  contributing-metric.md How to add a static documentation metric
+  contributing-terminal-bench-task.md
+                         Executable task format (tasks live in the content repo)
+  methodology.md         Model roles, question types, judging, trust checks
+  skill-novelty.md       What an artifact adds to a model, and how to shorten it
+  skill-novelty-evidence.md
+                         The study the novelty defaults come from
+  repo-evaluation-contract.md
+                         What a repository supplies to be evaluated as a subject
+tests/                   Pytest suite
+```
+
+## CI
+
+Among the checks GitHub Actions runs on pull requests (more in
+[docs/architecture.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/architecture.md#ci)):
+
+1. `test` — installs `requirements.lock.txt` and runs pytest offline; on a PR, 80% of the changed lines must be covered (coverage measured on Python 3.14 only).
+2. `benchmark` — runs the static agent-quality benchmark in strict mode (a gate or band failure fails the job) and uploads artifacts.
+3. `External catalog smoke test` — clones a pinned public content catalog, scans it, and drives one of its real tasks through the `docker-oracle` harness and the `docker-probe` isolation probe, asserting both pass rates.
+4. `Build docs (strict)` — builds the documentation site with `mkdocs build --strict`, so a broken link or a list that renders differently than on GitHub fails it; a push to `main` publishes the site to GitHub Pages.
+
+## Repository hygiene
+
+Do not commit generated logs, coverage databases, tarballs, local caches, or one-off experiment outputs. Keep generated benchmark artifacts either ignored locally or explicitly curated as fixtures. Use pull requests for all changes to `main`.
+
+Status reports, phase plans, and other development-history notes do not belong in the repository — work-in-progress should live in PR descriptions and issues.
+
+## Contributing
+
+- Run `python -m pytest -q` before opening a PR.
+- See [docs/contributing-metric.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/contributing-metric.md) and
+  [docs/contributing-terminal-bench-task.md](https://github.com/intel/dokimi/blob/main/dokimi/docs/contributing-terminal-bench-task.md)
+  for the two main extension flows.
+- Keep generated artifacts under `reports/` or `results/` (both ignored).
+  Only commit curated fixtures that are part of a reproducible benchmark.
+
+## License
+
+Licensed under the [Apache License, Version 2.0](https://github.com/intel/dokimi/blob/main/LICENSE). See [NOTICE](https://github.com/intel/dokimi/blob/main/NOTICE)
+for attribution. Contributions are accepted under the same license — see
+[CONTRIBUTING.md](https://github.com/intel/dokimi/blob/main/dokimi/CONTRIBUTING.md). To report a security issue, see
+[SECURITY.md](https://github.com/intel/dokimi/blob/main/dokimi/SECURITY.md).