paperlab-space 0.1.0__tar.gz

paperlab_space-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run mypy src tests
+      - run: uv run pytest

paperlab_space-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,26 @@
+name: Publish to PyPI
+
+# Runs when a GitHub release is published. Uses PyPI trusted publishing, so no API
+# token is stored: configure the publisher once at https://pypi.org/manage/account/publishing/
+# (project paperlab-space, workflow publish.yml, environment pypi).
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - name: Check the tag matches the package version
+        run: |
+          version=$(grep -Po '^__version__ = "\K[^"]+' src/paperlab_space/__init__.py)
+          test "v$version" = "${GITHUB_REF_NAME}" || { echo "Tag ${GITHUB_REF_NAME} != v$version"; exit 1; }
+      - run: uv sync
+      - run: uv run pytest
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

paperlab_space-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+.DS_Store

paperlab_space-0.1.0/CITATION.cff

@@ -0,0 +1,16 @@
+cff-version: 1.2.0
+message: "If you use paperlab-space in your research, please cite it as below."
+title: "paperlab-space: no-code LLM evaluation on spreadsheet data"
+type: software
+authors:
+  - name: "PaperLab.space contributors"
+repository-code: "https://github.com/paperlab-space/paperlab-core"
+url: "https://paperlab.space"
+license: Apache-2.0
+version: 0.1.0
+date-released: 2026-10-17
+keywords:
+  - large language models
+  - evaluation
+  - reproducibility
+  - multilingual

paperlab_space-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 PaperLab.space contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+

paperlab_space-0.1.0/PKG-INFO

@@ -0,0 +1,155 @@
+Metadata-Version: 2.5
+Name: paperlab-space
+Version: 0.1.0
+Summary: Open-source engine for evaluating LLMs on spreadsheet data and producing paper-ready results.
+Project-URL: Homepage, https://paperlab.space
+Project-URL: Repository, https://github.com/paperlab-space/paperlab-core
+Author: PaperLab.space contributors
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: benchmark,evaluation,latex,llm,research
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: charset-normalizer>=3.3
+Requires-Dist: litellm>=1.50
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: cli
+Requires-Dist: typer>=0.12; extra == 'cli'
+Description-Content-Type: text/markdown
+
+# PaperLab.space core
+
+```bash
+pip install 'paperlab-space[cli]'
+```
+
+Evaluate LLMs on your own spreadsheet data and get paper-ready results.
+
+Give it an Excel or CSV file, say which columns are inputs and answers, pick models and metrics, and get back an Excel workbook of outputs and scores, a LaTeX table for your paper, and a reproducibility config. `paperlab-space` is the open-source engine behind [PaperLab.space](https://paperlab.space); use it from the command line or from Python.
+
+- **Spreadsheets in:** `.xlsx` (any sheet), `.csv`, `.tsv`, with encoding detection and Unicode-safe handling for non-Latin scripts. Validation catches empty cells, duplicate IDs, and garbled text before you spend API calls.
+- **Any model:** OpenRouter (including free models), Gemini, OpenAI, Anthropic, Ollama, and everything else [LiteLLM](https://docs.litellm.ai/docs/providers) supports, run side by side. You bring your own API keys.
+- **Robust runs:** per-model rate limits tuned for free tiers, retries with backoff, and resume after interruption. A cost and time estimate is shown before anything is sent.
+- **Metrics:** exact match, accuracy (with label extraction from chatty answers), precision/recall/F1, token F1. Tokenisation handles Chinese, Japanese, Thai, and Burmese.
+- **Paper-ready output:** Excel, a booktabs LaTeX table, and a YAML config recording model versions, prompt, seed, metric settings, and a hash of your data.
+
+> **Status:** early release (0.1). See [PLAN.md](PLAN.md) for the roadmap: BLEU/chrF/ROUGE, LLM-as-judge, and significance tests are next.
+
+## Install
+
+```bash
+pip install 'paperlab-space[cli]'
+```
+
+Until the first PyPI release, install from GitHub:
+
+```bash
+pip install 'paperlab-space[cli] @ git+https://github.com/paperlab-space/paperlab-core'
+```
+
+## Quickstart (command line)
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...   # free key from https://openrouter.ai
+export GEMINI_API_KEY=...             # free key from https://aistudio.google.com
+
+paperlab-space init my_data.xlsx            # writes run.yaml, guessing the column roles
+paperlab-space validate my_data.xlsx -i question -e answer   # check the data, preview JSONL
+paperlab-space estimate run.yaml            # tokens, cost, and time; no requests sent
+paperlab-space run run.yaml                 # writes results-run/
+```
+
+`run.yaml` looks like this (see [examples/sentiment](examples/sentiment)):
+
+```yaml
+dataset:
+  path: data.csv
+mapping:
+  input: text            # column(s) available to the prompt
+  context: [language]    # extra columns available to the prompt
+  expected: label        # the answer key (never shown to the model)
+  id: id
+prompt: |-
+  Classify the sentiment of this {language} text as "positive" or "negative".
+  Text: {text}
+models:
+  - openrouter/nvidia/nemotron-3-super-120b-a12b:free
+  - id: gemini/gemini-2.5-flash
+    temperature: 0
+metrics:
+  - name: accuracy
+    labels: [positive, negative]
+  - f1
+seed: 42
+```
+
+The output folder contains:
+
+| File | What it is |
+| --- | --- |
+| `results.xlsx` | Summary sheet (scores, failures, cost), per-row outputs and scores, and the config |
+| `table.tex` | A booktabs table to `\input{}` into your paper |
+| `run.yaml` | The reproducibility config: re-run with `paperlab-space run run.yaml` |
+| `results.json` | Everything, for `paperlab-space run run.yaml --resume results.json` after an interruption |
+
+Free models change often; `paperlab-space run` stops a model early with a clear message if its ID no longer exists. Browse current free models at [openrouter.ai/models](https://openrouter.ai/models?q=free).
+
+## Quickstart (Python)
+
+```python
+from paperlab_space import ColumnMapping, RunConfig, estimate, export, load_dataset, run
+
+data = load_dataset("my_data.xlsx", sheet="test")
+config = RunConfig(
+    mapping=ColumnMapping(input="question", expected="answer"),
+    prompt="Answer briefly: {question}",
+    models=["openrouter/nvidia/nemotron-3-super-120b-a12b:free", "gemini/gemini-2.5-flash"],
+    metrics=["exact_match", "token_f1"],
+)
+print(estimate(config, data))
+results = run(config, data)
+print(results.scores["gemini-2.5-flash"]["exact_match"].value)
+export.write_all(results, "results/")
+```
+
+`run` works in scripts and Jupyter notebooks. Use `await arun(...)` inside your own async code.
+
+## Metrics
+
+Run `paperlab-space metrics` for the list. Every metric takes normalisation options in the config:
+
+```yaml
+metrics:
+  - exact_match                         # case-insensitive, trims punctuation like "Paris."
+  - name: exact_match
+    label: exact_match_strict           # same metric again with different options
+    ignore_case: false
+    trim_punctuation: false
+  - name: f1
+    average: macro                      # macro | micro | weighted
+```
+
+Custom metrics subclass `paperlab_space.metrics.Metric` and use the `@register` decorator.
+
+## Citing
+
+If paperlab-space helps your research, please cite it. See [CITATION.cff](CITATION.cff), or use "Cite this repository" on GitHub.
+
+## Development
+
+```bash
+uv sync
+uv run pytest                   # unit tests, no network
+uv run pytest -m live           # real-provider smoke tests (needs API keys)
+uv run ruff check . && uv run ruff format --check .
+uv run mypy src tests
+```
+
+## License
+
+[Apache 2.0](LICENSE)

paperlab_space-0.1.0/PLAN.md

@@ -0,0 +1,120 @@
+# PaperLab.space core plan
+
+This repo (`paperlab-core`) holds the open-source evaluation engine behind [PaperLab.space](https://paperlab.space). It takes a spreadsheet, runs one or more LLMs over it, scores the outputs, and writes paper-ready results. It is a plain Python library plus a small CLI, published to PyPI as `paperlab-space` (`import paperlab_space`, command `paperlab-space`). The bare name `paperlab` belongs to an unrelated PyPI project, so the package name carries the `.space` brand, as the PRD intends.
+
+## Scope and boundary
+
+Core is pure **input → eval → output**. It never imports web code: no database, auth, HTTP server, payments, or job-queue infrastructure. Anything stateful across sessions belongs to the platform.
+
+| In core | Not in core (platform) |
+| --- | --- |
+| Excel/CSV loading, column mapping, validation, JSONL conversion | File upload, storage, user accounts |
+| Prompt templating | Prompt editor UI |
+| Model runner: LiteLLM wrapper, per-provider rate limiting, retries, checkpoint/resume to a local file | Distributed job queue, workers, progress over websockets |
+| Metrics | Credits, billing, cost markup |
+| Excel / LaTeX / reproducibility-config export | Run history, sharing |
+| Cost and time estimates (from token counts and public prices) | Managed API keys, encrypted key storage |
+
+The platform calls core through its public API only. If the platform needs a hook (e.g. a progress callback), core exposes a generic one.
+
+## Tech choices
+
+- Python ≥ 3.10, `src/` layout, built with hatchling, managed with `uv`
+- `openpyxl` + the standard `csv` module for input (no pandas: lighter installs and full control over cell-to-text conversion), `charset-normalizer` for legacy CSV encodings
+- `pydantic` v2 for config models (serialisable, validated, clear error messages)
+- `litellm` for one interface to every provider, imported lazily so `import paperlab_space` stays fast
+- `asyncio` runner with an even-spacing rate limiter per model
+- Own `{column}` placeholder syntax (column names may contain spaces; `{{`/`}}` for literal braces)
+- `ruff` for lint/format, `pytest` for tests, `mypy --strict` for types
+- `typer` for the CLI (optional extra: `pip install 'paperlab-space[cli]'`)
+
+## Package layout
+
+```
+src/paperlab_space/
+  data/        load.py (Excel/CSV -> Dataset), mapping.py (ColumnMapping, JSONL), validate.py
+  prompt.py    PromptTemplate
+  config.py    RunConfig, ModelSpec, MetricSpec (the single object describing a run)
+  runner/      backend.py (LiteLLM + Backend protocol), ratelimit.py, runner.py, estimate.py
+  metrics/     base.py (interface + registry), builtin.py, normalize.py
+  results.py   RunResults, JSON save/load
+  export/      excel.py, latex.py, repro.py
+  _cli.py      the `paperlab` command
+```
+
+Central object: **`RunConfig`**, which holds data source, column mapping, prompt template, models (+ temperature, seed, max tokens, rate limit), and metrics. The same object drives a run and is written out as the reproducibility config, so a run is reproducible from its exported config alone.
+
+## Public API
+
+```python
+from paperlab_space import ColumnMapping, RunConfig, estimate, export, load_dataset, run
+
+data = load_dataset("data.xlsx", sheet="test")
+mapping = ColumnMapping(input="sentence", expected="gold", id="row_id")
+print(data.map(mapping).validate())  # empty cells, duplicate IDs, encoding issues
+
+config = RunConfig(
+    mapping=mapping,
+    prompt="Translate to English: {sentence}",
+    models=["openrouter/nvidia/nemotron-3-super-120b-a12b:free", "gemini/gemini-2.5-flash"],
+    metrics=["exact_match", "token_f1"],
+)
+print(estimate(config, data))  # tokens, cost, time per model
+results = run(config, data, on_progress=print)
+export.write_all(results, "results/")  # results.xlsx, table.tex, run.yaml, results.json
+```
+
+The platform plugs in through: `LiteLLMBackend(api_keys=...)` (keys never enter configs or results), the `Backend` protocol, `on_progress`, `arun` for its own event loop, and `previous=` / `RunResults.load_json` to resume.
+
+## Milestones
+
+### M0: Skeleton ✅
+- Package, CI, license, README, plan
+
+### M1: Data in ✅
+- Load `.xlsx` (multiple sheets), `.csv`, `.tsv`; UTF-8/BOM first, then encoding detection (tested with CP874 Thai, Shift-JIS, GBK)
+- Unicode NFC normalisation; fixtures in Burmese, Thai, Chinese, Japanese, Korean
+- Excel cell types converted to text predictably (`2.0` -> `2`, dates -> ISO); blank/duplicate headers fixed
+- `ColumnMapping`: input column(s), context columns, expected column, row ID, multiple answers via delimiter; the expected column can never leak into the prompt
+- Validation report: empty cells, empty/duplicate IDs, mojibake, invisible characters
+- JSONL conversion + preview
+
+### M2: Prompt + runner ✅
+- `PromptTemplate` (user + optional system prompt) checked against mapped columns
+- Async runner over LiteLLM; tested live against **OpenRouter** and **Gemini**
+- Rate limits per model: OpenRouter `:free` 20/min, Gemini Flash 10/min, Gemini Pro 5/min; overridable
+- Retries with exponential backoff + jitter, honouring `Retry-After`; a rejected API key or unknown model stops that model immediately instead of failing every row
+- Resume: rows that succeeded in an earlier run are reused (pulled forward from v1.1)
+- Pre-run estimate: tokens (script-aware heuristic), cost (LiteLLM price table), wall time, warnings (OpenRouter daily caps, Gemini free-tier data use)
+- Works in scripts and Jupyter
+
+### M3: Metrics ✅
+- Metric interface (`compute(predictions, references) -> MetricResult`) with a registry so users can add their own
+- `exact_match`, `accuracy` (optional label extraction from chatty answers), `f1` (macro/micro/weighted, per-class), `token_f1` (character tokens for Chinese, Japanese, Thai, Burmese)
+- Configurable normalisation (NFKC, case, punctuation), recorded in the reproducibility config
+- Multi-answer support; failed requests count as wrong unless `count_failed: false`
+- Metrics are checked before any request is sent
+
+### M4: Export ✅
+- Excel: Summary (scores, failures, cost, tokens), Outputs (per-row outputs and scores), Config sheets; formula injection blocked
+- LaTeX: booktabs table, best score bolded, escaped names, percent or raw
+- Reproducibility YAML: core/LiteLLM/Python versions, provider-reported model versions, dataset SHA-256, full config (dataset path reduced to file name); loads back with `RunConfig.from_yaml`
+- Full results JSON for resuming and sharing
+
+### M5: CLI + first release (in progress)
+- ✅ `paperlab-space init | validate | estimate | run [--resume] | metrics`
+- ✅ `CITATION.cff`, release workflow (PyPI trusted publishing)
+- ⬜ Configure the PyPI trusted publisher, then publish `0.1.0`
+- ⬜ Docs site (mkdocs-material)
+
+### After 0.1 (matches the product roadmap)
+- v1.1: separate answer-key file joined by row ID; BLEU, chrF, ROUGE (`sacrebleu`, `rouge-score`); Ollama and more providers documented and tested; Zawgyi detection for Burmese input
+- v1.2: LLM-as-judge (with reference, and rubric-only with user rubric); significance tests and bootstrap confidence intervals
+- v2: inter-annotator agreement metrics (Cohen's/Fleiss' kappa, Krippendorff's alpha)
+
+## Quality bar
+
+- Every public function typed (`mypy --strict`) and documented
+- No network access in unit tests; `uv run pytest -m live` for real-provider smoke tests
+- Deterministic outputs given the same config + seed where providers allow it
+- Semantic versioning; the reproducibility config records the exact core version