worthir-eval 1.2.0__tar.gz

worthir_eval-1.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 WorthIR Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

worthir_eval-1.2.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: worthir-eval
+Version: 1.2.0
+Summary: Evaluate cost-aware query-level retrieval routing
+Author: WorthIR authors
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# WorthIR
+
+[English](https://github.com/HongTianShi/worthir-artifact/tree/_en) | [简体中文](https://github.com/HongTianShi/worthir-artifact/tree/zh-cn)
+
+[![CI](https://github.com/HongTianShi/worthir-artifact/actions/workflows/validate.yml/badge.svg?branch=_en)](https://github.com/HongTianShi/worthir-artifact/actions/workflows/validate.yml)
+![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-3776AB)
+[![MIT License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+
+WorthIR compares query-level routing policies with fixed retrieval strategies
+under a declared effectiveness measure and cost profile. It reports
+effectiveness, cost, utility, regret, and the fixed-route Pareto curve.
+
+If you are an AI tool, read [`README_FOR_AI.md`](README_FOR_AI.md) before
+searching the repository.
+
+## 60-second demo
+
+Python 3.10 or newer is required. Choose one installation path.
+
+**Source archive or Git clone:** run the local launcher. It creates the local
+environment when first needed.
+
+```powershell
+.\worthir.cmd demo-custom
+```
+
+```bash
+./worthir demo-custom
+```
+
+**Installed wheel:** install the wheel, then use the global command.
+
+```bash
+python -m pip install worthir_eval-1.2.0-py3-none-any.whl
+worthir demo-custom
+```
+
+Do not run both setup paths. Open `reproduced/custom_task/comparison.md` after
+the command finishes. The published wheel uses English terminal messages; the
+Chinese source branch provides Chinese launchers and documentation.
+
+## Use your own task
+
+Prepare `task.json`, `queries.csv`, `routes.csv`, and `outcomes.csv` as shown in
+[`examples/custom_task/source/`](examples/custom_task/source/), then run:
+
+```powershell
+.\worthir.cmd build-custom my_source my_task
+.\worthir.cmd validate-task my_task
+.\worthir.cmd evaluate my_task choices.csv --policy-id my-router
+```
+
+This path accepts any named higher-is-better effectiveness measure, arbitrary
+route prerequisites, fixed or query-dependent costs, and either cumulative or
+incremental cost input. The router receives `queries.csv`, the public route
+registry, lambda, and any costs declared as known at commitment time. Evaluator
+outcomes and costs measured only after execution remain separate.
+
+For qrels and six-column TREC runs, use the shorter [`build-trec` walkthrough](examples/trec_walkthrough/README.md).
+All input formats are described in [`docs/ADAPT_TO_NEW_TASK.md`](docs/ADAPT_TO_NEW_TASK.md).
+For direct library use, see the
+[`worthir_eval` Python example](examples/python_api/README.md).
+
+## Recompute the paper results
+
+This uses released query--route ledgers and frozen route selections. It does
+not redownload corpora or rerun retrieval models.
+
+```bash
+python paper_results/run.py
+```
+
+Open [`paper_results/reproduced/INDEX.md`](paper_results/reproduced/INDEX.md).
+The index names the exact paper version, caption, output, and reproduction
+level for every main-paper and appendix figure or table.
+
+## Rebuild the original retrieval routes
+
+This is a separate, resource-intensive workflow. It checks licensed corpora
+and checkpoints, invokes a configured task adapter, and constructs new
+query--route ledgers through five explicit stages. Start with
+[`paper_results/full_replay/README.md`](paper_results/full_replay/README.md) and
+its task-specific resource estimates. Raw corpora, indexes, and model weights
+are not included in this repository.
+
+The [`v1.2.0`](https://github.com/HongTianShi/worthir-artifact/releases/tag/v1.2.0)
+release is the published artifact currently bound to the 2026-08-16 paper
+mapping. The earlier `v1.0.0-ipmc2026` release remains the artifact submitted
+with the accepted IP&MC 2026 paper.
+
+WorthIR-authored code is released under the [MIT License](LICENSE). Third-party
+data and model terms are listed in [NOTICE](NOTICE).

worthir_eval-1.2.0/README.md

@@ -0,0 +1,92 @@
+# WorthIR
+
+[English](https://github.com/HongTianShi/worthir-artifact/tree/_en) | [简体中文](https://github.com/HongTianShi/worthir-artifact/tree/zh-cn)
+
+[![CI](https://github.com/HongTianShi/worthir-artifact/actions/workflows/validate.yml/badge.svg?branch=_en)](https://github.com/HongTianShi/worthir-artifact/actions/workflows/validate.yml)
+![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-3776AB)
+[![MIT License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+
+WorthIR compares query-level routing policies with fixed retrieval strategies
+under a declared effectiveness measure and cost profile. It reports
+effectiveness, cost, utility, regret, and the fixed-route Pareto curve.
+
+If you are an AI tool, read [`README_FOR_AI.md`](README_FOR_AI.md) before
+searching the repository.
+
+## 60-second demo
+
+Python 3.10 or newer is required. Choose one installation path.
+
+**Source archive or Git clone:** run the local launcher. It creates the local
+environment when first needed.
+
+```powershell
+.\worthir.cmd demo-custom
+```
+
+```bash
+./worthir demo-custom
+```
+
+**Installed wheel:** install the wheel, then use the global command.
+
+```bash
+python -m pip install worthir_eval-1.2.0-py3-none-any.whl
+worthir demo-custom
+```
+
+Do not run both setup paths. Open `reproduced/custom_task/comparison.md` after
+the command finishes. The published wheel uses English terminal messages; the
+Chinese source branch provides Chinese launchers and documentation.
+
+## Use your own task
+
+Prepare `task.json`, `queries.csv`, `routes.csv`, and `outcomes.csv` as shown in
+[`examples/custom_task/source/`](examples/custom_task/source/), then run:
+
+```powershell
+.\worthir.cmd build-custom my_source my_task
+.\worthir.cmd validate-task my_task
+.\worthir.cmd evaluate my_task choices.csv --policy-id my-router
+```
+
+This path accepts any named higher-is-better effectiveness measure, arbitrary
+route prerequisites, fixed or query-dependent costs, and either cumulative or
+incremental cost input. The router receives `queries.csv`, the public route
+registry, lambda, and any costs declared as known at commitment time. Evaluator
+outcomes and costs measured only after execution remain separate.
+
+For qrels and six-column TREC runs, use the shorter [`build-trec` walkthrough](examples/trec_walkthrough/README.md).
+All input formats are described in [`docs/ADAPT_TO_NEW_TASK.md`](docs/ADAPT_TO_NEW_TASK.md).
+For direct library use, see the
+[`worthir_eval` Python example](examples/python_api/README.md).
+
+## Recompute the paper results
+
+This uses released query--route ledgers and frozen route selections. It does
+not redownload corpora or rerun retrieval models.
+
+```bash
+python paper_results/run.py
+```
+
+Open [`paper_results/reproduced/INDEX.md`](paper_results/reproduced/INDEX.md).
+The index names the exact paper version, caption, output, and reproduction
+level for every main-paper and appendix figure or table.
+
+## Rebuild the original retrieval routes
+
+This is a separate, resource-intensive workflow. It checks licensed corpora
+and checkpoints, invokes a configured task adapter, and constructs new
+query--route ledgers through five explicit stages. Start with
+[`paper_results/full_replay/README.md`](paper_results/full_replay/README.md) and
+its task-specific resource estimates. Raw corpora, indexes, and model weights
+are not included in this repository.
+
+The [`v1.2.0`](https://github.com/HongTianShi/worthir-artifact/releases/tag/v1.2.0)
+release is the published artifact currently bound to the 2026-08-16 paper
+mapping. The earlier `v1.0.0-ipmc2026` release remains the artifact submitted
+with the accepted IP&MC 2026 paper.
+
+WorthIR-authored code is released under the [MIT License](LICENSE). Third-party
+data and model terms are listed in [NOTICE](NOTICE).

worthir_eval-1.2.0/contracts/README.md

@@ -0,0 +1,11 @@
+# Example contracts
+
+These files define the six-query smoke test. For a new task, use
+`worthir build-custom`, or use `worthir build-trec` for qrels and TREC runs;
+both create a bound contract and route registry.
+
+The public interface fixes five invariants: higher effectiveness is better, the
+ledger contains every query--route pair, costs are nonnegative and cumulative,
+their decision-time availability is explicit, and oracle ties prefer lower cost
+then registry order. Fixed commitment-time costs live on registered routes;
+query-dependent commitment-time costs live in `participant/route_costs.csv`.

worthir_eval-1.2.0/contracts/__init__.py

@@ -0,0 +1 @@
+"""Bundled quickstart contracts."""

worthir_eval-1.2.0/contracts/quickstart_contract.json

@@ -0,0 +1,40 @@
+{
+  "schema_version": "worthir-contract-v1.0",
+  "contract_id": "worthir-quickstart-v1",
+  "task_id": "worthir-synthetic-route-choice-v1",
+  "expected_query_count": 6,
+  "route_registry": "route_registry.json",
+  "development_selected_fixed_route": "dense",
+  "action_schema": {
+    "schema_version": "worthir-action-file-v1.0",
+    "top_level_fields": [
+      "schema_version",
+      "contract_id",
+      "policy_id",
+      "decisions"
+    ],
+    "decision_fields": [
+      "query_uid",
+      "selected_route_id"
+    ]
+  },
+  "ledger_schema": {
+    "columns": [
+      "query_uid",
+      "route_id",
+      "effectiveness",
+      "cost"
+    ]
+  },
+  "metric": {
+    "name": "synthetic_ndcg_at_10",
+    "minimum": 0.0,
+    "maximum": 1.0
+  },
+  "cost_profile": {
+    "profile_id": "synthetic_declared_schedule-v1",
+    "provenance": "declared",
+    "lambda": 0.08,
+    "availability": "known_at_commitment"
+  }
+}

worthir_eval-1.2.0/contracts/route_registry.json

@@ -0,0 +1,29 @@
+{
+  "registry_id": "worthir-quickstart-routes-v1",
+  "cost_information": {
+    "availability": "known_at_commitment",
+    "mode": "fixed",
+    "route_costs_file": null
+  },
+  "routes": [
+    {
+      "label": "Stop after BM25",
+      "prerequisites": [],
+      "route_id": "bm25",
+      "cost": 0.0
+    },
+    {
+      "label": "Dense candidate fusion",
+      "prerequisites": ["bm25"],
+      "route_id": "dense",
+      "cost": 0.2
+    },
+    {
+      "label": "Cross-encoder rerank top 20",
+      "prerequisites": ["dense"],
+      "route_id": "ce20",
+      "cost": 0.8
+    }
+  ],
+  "schema_version": "worthir-route-registry-v1.2"
+}

worthir_eval-1.2.0/examples/__init__.py

@@ -0,0 +1 @@
+"""Runnable examples bundled with the WorthIR wheel."""

worthir_eval-1.2.0/examples/custom_router/README.md

@@ -0,0 +1,36 @@
+# Bring your own router
+
+Run the complete example from the repository root:
+
+```powershell
+.\worthir.cmd demo-custom
+```
+
+```bash
+./worthir demo-custom
+```
+
+The example builds a non-TREC task, runs `router.py`, binds its CSV decisions to
+the task contract, and compares the router with every fixed route. The report is
+written to `reproduced/custom_task/comparison.md`.
+
+`router.py` reads the public task contract, route registry, lambda,
+`participant/legal_state.csv`, and `participant/route_costs.csv`. It selects the
+route with the highest predicted effectiveness minus lambda times public cost.
+It never opens `evaluator/ledger.csv`, which contains information unavailable
+when a route is selected. To use your own router, replace
+`predicted_effectiveness()` and keep the two-column output format:
+
+```text
+query_uid,selected_route_id
+```
+
+For an existing task, evaluate that CSV in one command:
+
+```powershell
+.\worthir.cmd evaluate TASK choices.csv --policy-id my-router
+```
+
+```bash
+./worthir evaluate TASK choices.csv --policy-id my-router
+```

worthir_eval-1.2.0/examples/custom_router/__init__.py

@@ -0,0 +1 @@
+"""External-router walkthrough."""

worthir_eval-1.2.0/examples/custom_router/router.py

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+"""Cost-aware router using only participant-visible task files."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+from pathlib import Path
+
+
+def read_json(path: Path) -> dict:
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def public_costs(task: Path, registry: dict) -> dict[tuple[str, str], float]:
+    info = registry["cost_information"]
+    if info["availability"] != "known_at_commitment":
+        raise ValueError("this router requires costs known at commitment time")
+    if info["mode"] == "fixed":
+        return {
+            ("*", route["route_id"]): float(route["cost"])
+            for route in registry["routes"]
+        }
+    path = (task / "contracts" / info["route_costs_file"]).resolve()
+    with path.open("r", encoding="utf-8-sig", newline="") as stream:
+        return {
+            (row["query_uid"], row["route_id"]): float(row["cost"])
+            for row in csv.DictReader(stream)
+        }
+
+
+def predicted_effectiveness(row: dict[str, str], route_id: str) -> float:
+    """Replace these estimates with predictions from your own router."""
+
+    length = int(row["question_length"])
+    product_code = row["contains_product_code"].lower() == "true"
+    if route_id == "keyword_search":
+        return 0.86 if length <= 12 and not product_code else 0.58
+    if route_id == "semantic_search":
+        return 0.90 if length > 20 else 0.79
+    if route_id == "combined_review":
+        return 0.95 if product_code else 0.87
+    return 0.0
+
+
+def route(task: Path, output: Path) -> Path:
+    contract = read_json(task / "contracts" / "task_contract.json")
+    registry = read_json(task / "contracts" / contract["route_registry"])
+    costs = public_costs(task, registry)
+    route_ids = [item["route_id"] for item in registry["routes"]]
+    lam = float(contract["cost_profile"]["lambda"])
+    with (task / "participant" / "legal_state.csv").open(
+        "r", encoding="utf-8-sig", newline=""
+    ) as stream:
+        rows = list(csv.DictReader(stream))
+
+    decisions = []
+    for row in rows:
+        query_uid = row["query_uid"]
+        selected = max(
+            route_ids,
+            key=lambda route_id: (
+                predicted_effectiveness(row, route_id)
+                - lam * costs.get((query_uid, route_id), costs.get(("*", route_id))),
+                -route_ids.index(route_id),
+            ),
+        )
+        decisions.append({"query_uid": query_uid, "selected_route_id": selected})
+
+    output.parent.mkdir(parents=True, exist_ok=True)
+    with output.open("w", encoding="utf-8", newline="") as stream:
+        writer = csv.DictWriter(
+            stream, fieldnames=["query_uid", "selected_route_id"]
+        )
+        writer.writeheader()
+        writer.writerows(decisions)
+    return output
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Create cost-aware choices without reading evaluator outcomes."
+    )
+    parser.add_argument("task_dir", type=Path, help="built WorthIR task directory")
+    parser.add_argument("output", type=Path, help="choice CSV to create")
+    args = parser.parse_args()
+    print(f"WROTE: {route(args.task_dir.resolve(), args.output.resolve())}")
+
+
+if __name__ == "__main__":
+    main()

worthir_eval-1.2.0/examples/custom_router/run.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+"""Build the custom task, run the example router, and compare its utility."""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[2]
+OUTPUT_ROOT = ROOT if (ROOT / "paper_results").is_dir() else Path.cwd()
+DESTINATION = OUTPUT_ROOT / "reproduced" / "custom_task"
+CHOICES = OUTPUT_ROOT / "reproduced" / "custom_router_choices.csv"
+
+
+def run(*arguments: str) -> None:
+    launcher = (
+        [sys.executable, str(ROOT / "worthir.py")]
+        if (ROOT / "worthir.py").is_file()
+        else [sys.executable, "-m", "worthir"]
+    )
+    completed = subprocess.run([*launcher, *arguments])
+    if completed.returncode:
+        raise SystemExit(completed.returncode)
+
+
+def main() -> None:
+    if DESTINATION.exists():
+        shutil.rmtree(DESTINATION)
+    run(
+        "build-custom",
+        str(ROOT / "examples" / "custom_task" / "source"),
+        str(DESTINATION),
+    )
+    subprocess.run(
+        [
+            sys.executable,
+            str(Path(__file__).with_name("router.py")),
+            str(DESTINATION),
+            str(CHOICES),
+        ],
+        check=True,
+    )
+    run(
+        "evaluate",
+        str(DESTINATION),
+        str(CHOICES),
+        "--policy-id",
+        "example-rule-router",
+    )
+    print(f"OPEN: {DESTINATION / 'comparison.md'}")
+
+
+if __name__ == "__main__":
+    main()

worthir_eval-1.2.0/examples/custom_task/README.md

@@ -0,0 +1,32 @@
+# Custom task input
+
+This example is not a TREC ranking task. Its effectiveness measure is
+`answer_coverage`, and its execution cost varies by query.
+
+The four source files are the complete generic adapter interface:
+
+- `task.json` names the effectiveness measure and cost profile.
+- `queries.csv` contains only information a router may use.
+- `routes.csv` defines available routes, prerequisites, and route costs.
+- `outcomes.csv` contains evaluator-only effectiveness and cost outcomes for
+  every query--route pair.
+
+Build and validate it from the repository root:
+
+```powershell
+.\worthir.cmd build-custom examples/custom_task/source reproduced/custom_task
+.\worthir.cmd validate-task reproduced/custom_task
+```
+
+```bash
+./worthir build-custom examples/custom_task/source reproduced/custom_task
+./worthir validate-task reproduced/custom_task
+```
+
+Use `cost` when cumulative route costs are already available. Use
+`incremental_cost` when the tool should sum the transitive prerequisite closure.
+Either column may appear in `outcomes.csv` to provide query-dependent values.
+Set `cost_profile.availability` to `known_at_commitment` when the router may use
+the cost before choosing a route. The builder then publishes fixed costs in the
+route registry or query-dependent costs in `participant/route_costs.csv`. Use
+`measured_after_execution` when cost is evaluator-only.

worthir_eval-1.2.0/examples/custom_task/__init__.py

@@ -0,0 +1 @@
+"""Generic non-TREC task example."""

worthir_eval-1.2.0/examples/custom_task/source/outcomes.csv

@@ -0,0 +1,13 @@
+query_uid,route_id,effectiveness,incremental_cost
+faq-001,keyword_search,0.92,0.025
+faq-001,semantic_search,0.88,0.105
+faq-001,combined_review,0.94,0.070
+faq-002,keyword_search,0.61,0.030
+faq-002,semantic_search,0.84,0.115
+faq-002,combined_review,0.96,0.075
+faq-003,keyword_search,0.45,0.035
+faq-003,semantic_search,0.81,0.140
+faq-003,combined_review,0.93,0.095
+faq-004,keyword_search,0.78,0.028
+faq-004,semantic_search,0.80,0.110
+faq-004,combined_review,0.91,0.080

worthir_eval-1.2.0/examples/custom_task/source/queries.csv

@@ -0,0 +1,5 @@
+query_uid,question_length,contains_product_code
+faq-001,6,false
+faq-002,19,true
+faq-003,31,false
+faq-004,12,true

worthir_eval-1.2.0/examples/custom_task/source/routes.csv

@@ -0,0 +1,4 @@
+route_id,label,prerequisites,incremental_cost,development_selected
+keyword_search,Keyword search,,0.03,false
+semantic_search,Semantic search,,0.12,true
+combined_review,Combined evidence review,keyword_search;semantic_search,0.08,false

worthir_eval-1.2.0/examples/custom_task/source/task.json

@@ -0,0 +1,16 @@
+{
+  "task_id": "faq-answer-routing-v1",
+  "metric": {
+    "name": "answer_coverage",
+    "minimum": 0.0,
+    "maximum": 1.0,
+    "higher_is_better": true
+  },
+  "cost_profile": {
+    "profile_id": "measured-seconds-v1",
+    "provenance": "expected execution time estimated on an independent calibration set and available before route selection",
+    "lambda": 0.15,
+    "availability": "known_at_commitment"
+  },
+  "development_selected_fixed_route": "semantic_search"
+}

worthir_eval-1.2.0/examples/trec_walkthrough/README.md

@@ -0,0 +1,28 @@
+# TREC walkthrough
+
+This is a tiny retrieval task in standard TREC formats. It demonstrates the
+full human workflow without downloading data.
+
+```powershell
+.\worthir.cmd demo
+```
+
+```bash
+./worthir demo
+```
+
+The command computes NDCG@3 from `source/qrels.tsv` and the three TREC runs,
+builds a WorthIR task, scores the supplied policy, evaluates every fixed route,
+and writes `reproduced/trec_walkthrough/comparison.md`.
+
+To try another policy after the demo:
+
+```powershell
+.\worthir.cmd actions reproduced/trec_walkthrough examples/trec_walkthrough/source/alternative_choices.csv --policy-id alternative
+.\worthir.cmd compare reproduced/trec_walkthrough
+```
+
+```bash
+./worthir actions reproduced/trec_walkthrough examples/trec_walkthrough/source/alternative_choices.csv --policy-id alternative
+./worthir compare reproduced/trec_walkthrough
+```

worthir_eval-1.2.0/examples/trec_walkthrough/__init__.py

@@ -0,0 +1 @@
+"""TREC task walkthrough."""

worthir_eval-1.2.0/examples/trec_walkthrough/source/alternative_choices.csv

@@ -0,0 +1,5 @@
+query_uid,selected_route_id
+q1,prf
+q2,rerank
+q3,rerank
+q4,base

worthir_eval-1.2.0/examples/trec_walkthrough/source/policy_choices.csv

@@ -0,0 +1,5 @@
+query_uid,selected_route_id
+q1,base
+q2,rerank
+q3,prf
+q4,base

worthir_eval-1.2.0/examples/trec_walkthrough/source/qrels.tsv

@@ -0,0 +1,8 @@
+q1 0 d1 2
+q1 0 d2 1
+q2 0 d4 2
+q2 0 d5 1
+q3 0 d7 2
+q3 0 d8 1
+q4 0 d10 2
+q4 0 d11 1

worthir_eval-1.2.0/examples/trec_walkthrough/source/queries.csv

@@ -0,0 +1,5 @@
+query_uid,query_text,base_score_margin
+q1,solar panel recycling,2.8
+q2,rare disease treatment,0.4
+q3,river restoration methods,1.1
+q4,historic bridge design,3.2

worthir_eval-1.2.0/examples/trec_walkthrough/source/routes.csv

@@ -0,0 +1,4 @@
+route_id,label,prerequisites,run_file,cost,development_selected
+base,BM25,,runs/base.trec,0.00,false
+prf,Query expansion,base,runs/prf.trec,0.25,true
+rerank,Cross-encoder,prf,runs/rerank.trec,0.75,false

worthir_eval-1.2.0/examples/trec_walkthrough/source/runs/base.trec

@@ -0,0 +1,12 @@
+q1 Q0 d1 1 8.0 base
+q1 Q0 x1 2 7.0 base
+q1 Q0 d2 3 6.0 base
+q2 Q0 x2 1 8.0 base
+q2 Q0 d5 2 7.0 base
+q2 Q0 d4 3 6.0 base
+q3 Q0 d8 1 8.0 base
+q3 Q0 x3 2 7.0 base
+q3 Q0 d7 3 6.0 base
+q4 Q0 d10 1 8.0 base
+q4 Q0 d11 2 7.0 base
+q4 Q0 x4 3 6.0 base