zsol-bench 1.0.1__tar.gz

zsol_bench-1.0.1/.git

@@ -0,0 +1 @@
+gitdir: ../.git/modules/SOL-ExecBench

zsol_bench-1.0.1/.gitignore

@@ -0,0 +1,38 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+.ruff_cache
+.pytest_cache
+
+# Virtual environments
+.venv
+
+# Claude files
+.claude
+.bash_profile
+.bashrc
+.gitconfig
+.gitmodules
+.idea
+.mcp.json
+.profile
+.ripgreprc
+.vscode
+.zprofile
+.zshrc
+HEAD
+./config
+hooks
+objects
+refs
+pytest*
+
+# datasets
+/data/*
+
+# other
+/out/

zsol_bench-1.0.1/.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.9.2
+    hooks:
+      # Run the linter
+      - id: ruff
+        args: [--fix]
+      # Run the formatter
+      - id: ruff-format
+  - repo: local
+    hooks:
+      - id: dco-sign-off
+        name: DCO Sign-off Check
+        entry: "bash -c 'grep -q \"^Signed-off-by: \" \"$1\"' --"
+        language: system
+        stages: [commit-msg]

zsol_bench-1.0.1/.python-version

@@ -0,0 +1 @@
+3.12

zsol_bench-1.0.1/CLAUDE.md

@@ -0,0 +1,68 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+SOL ExecBench is NVIDIA's GPU kernel evaluation and benchmarking framework. It evaluates custom GPU kernel solutions (PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++) for correctness and performance by comparing them against reference implementations.
+
+## Build & Run Commands
+
+```bash
+# Install dependencies (uses uv package manager)
+uv sync --all-groups
+
+# Run the CLI
+uv run sol-execbench <problem_dir> --solution solution.json
+
+# Run all tests
+uv run pytest tests/
+
+# Run a single test file
+uv run pytest tests/sol_execbench/core/data/test_definition.py
+
+# Run tests matching a keyword
+uv run pytest tests/ -k "test_correctness"
+
+# Run sample integration tests (requires CUDA GPU)
+uv run pytest tests/sol_execbench/samples/
+
+# Lint and format
+uv run ruff check .
+uv run ruff format .
+```
+
+## Architecture
+
+The codebase has three layers:
+
+1. **CLI** (`cli/main.py`) — Click-based CLI entry point (`sol-execbench`). Parses args, invokes the driver, displays Rich tables with results.
+
+2. **Driver** (`driver/`) — Orchestrates evaluation. `ProblemPackager` stages problem files (definition, workloads, solution sources) into a temp directory, then runs compilation and evaluation as **subprocesses**. The eval driver template (`driver/templates/eval_driver.py`, ~800 lines) is the self-contained script that runs inside the subprocess — it imports torch/triton, loads inputs, executes kernels, computes correctness/performance, and emits JSONL traces to stdout.
+
+3. **Core** (`core/`) — Two sub-packages:
+   - `core/data/` — Pydantic v2 models for all data types: Definition (kernel specs, tensor shapes), Solution (source files, build specs), Workload (input generation), Trace (evaluation results with correctness/performance).
+   - `core/bench/` — Benchmarking utilities: CUDA event timing, numerical correctness computation, GPU clock locking, reward hack detection (detects timing manipulation, stream injection, monkey-patching).
+
+### Key execution flow
+
+CLI → ProblemPackager stages files to temp dir → subprocess compiles C++/CUDA (if needed) → subprocess runs eval_driver.py → eval_driver outputs JSONL traces to stdout → CLI parses traces and displays results.
+
+### Two calling conventions
+
+Kernel solutions use either **destination-passing style** (DPS, modifies pre-allocated outputs in-place) or **return-value style** (returns outputs directly). The eval driver handles both.
+
+## Test Markers & Fixtures
+
+- `@pytest.mark.requires_torch_cuda` — auto-skipped when no CUDA GPU available
+- `@pytest.mark.cpp` — compiles C++/CUDA extensions (slow)
+- `@requires_sm100` — skips on GPUs below sm_100 (Blackwell). Use this for cuTile tests.
+- `tmp_cache_dir` fixture — isolated build cache per test via `SOLEXECBENCH_CACHE_PATH`
+
+## Style
+
+Ruff for linting and formatting. Rule E741 is ignored. Exclude `data/` and `tests/data/` from lint.
+
+## Commits
+
+Format: `#<Issue> - <Title>` with DCO sign-off (`git commit -s`).

zsol_bench-1.0.1/CONTRIBUTING.md

@@ -0,0 +1,111 @@
+
+## SOL ExecBench OSS Contribution Rules
+
+#### Issue Tracking
+
+* All enhancement, bugfix, or change requests must begin with the creation of a [SOL ExecBench Issue Request](https://github.com/NVIDIA/sol-execbench/issues).
+  * The issue request must be reviewed by SOL ExecBench engineers and approved prior to code review.
+
+
+#### Coding Guidelines
+
+- All source code contributions must adhere to the style enforced by [Ruff](https://docs.astral.sh/ruff/).
+
+- In addition, please follow the existing conventions in the relevant file, submodule, module, and project when you add new code or when you extend/fix existing functionality.
+
+- Avoid introducing unnecessary complexity into existing code so that maintainability and readability are preserved.
+
+- Try to keep pull requests (PRs) as concise as possible:
+  - Avoid committing commented-out code.
+  - Wherever possible, each PR should address a single concern. If there are several otherwise-unrelated things that should be fixed to reach a desired endpoint, our recommendation is to open several PRs and indicate the dependencies in the description. The more complex the changes are in a single PR, the more time it will take to review those changes.
+
+- Write commit titles using imperative mood and [these rules](https://chris.beams.io/posts/git-commit/), and reference the Issue number corresponding to the PR. Following is the recommended format for commit texts:
+```
+#<Issue Number> - <Commit Title>
+
+<Commit Body>
+```
+
+- Ensure that the build log is clean, meaning no warnings or errors should be present.
+
+- Ensure that all `sample_*` tests pass prior to submitting your code.
+
+- All OSS components must contain accompanying documentation (READMEs) describing the functionality, dependencies, and known issues.
+
+  - See `README.md` for existing samples for reference.
+
+- All OSS components must have an accompanying test.
+
+  - If introducing a new component, such as a plugin, provide a test sample to verify the functionality.
+
+- Make sure that you can contribute your work to open source (no license and/or patent conflict is introduced by your code). You will need to [`sign`](#signing-your-work) your commit.
+
+- Thanks in advance for your patience as we review your contributions; we do appreciate them!
+
+
+#### Pull Requests
+Developer workflow for code contributions is as follows:
+
+1. Developers must first [fork](https://help.github.com/en/articles/fork-a-repo) the [upstream](https://github.com/nvidia/TensorRT) TensorRT OSS repository.
+
+2. Git clone the forked repository and push changes to the personal fork.
+
+  ```bash
+git clone https://github.com/YOUR_USERNAME/YOUR_FORK.git TensorRT
+# Checkout the targeted branch and commit changes
+# Push the commits to a branch on the fork (remote).
+git push -u origin <local-branch>:<remote-branch>
+  ```
+
+3. Once the code changes are staged on the fork and ready for review, a [Pull Request](https://help.github.com/en/articles/about-pull-requests) (PR) can be [requested](https://help.github.com/en/articles/creating-a-pull-request) to merge the changes from a branch of the fork into a selected branch of upstream.
+  * Exercise caution when selecting the source and target branches for the PR.
+    Note that versioned releases of TensorRT OSS are posted to `release/` branches of the upstream repo.
+  * Creation of a PR creation kicks off the code review process.
+  * Atleast one TensorRT engineer will be assigned for the review.
+  * While under review, mark your PRs as work-in-progress by prefixing the PR title with [WIP].
+
+4. Since there is no CI/CD process in place yet, the PR will be accepted and the corresponding issue closed only after adequate testing has been completed, manually, by the developer and/or TensorRT engineer reviewing the code.
+
+
+#### Signing Your Work
+
+* We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
+
+  * Any contribution which contains commits that are not Signed-Off will not be accepted.
+
+* To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes:
+  ```bash
+  $ git commit -s -m "Add cool feature."
+  ```
+  This will append the following to your commit message:
+  ```
+  Signed-off-by: Your Name <your@email.com>
+  ```
+
+* Full text of the DCO:
+
+  ```
+    Developer Certificate of Origin
+    Version 1.1
+    
+    Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+    1 Letterman Drive
+    Suite D4700
+    San Francisco, CA, 94129
+    
+    Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
+  ```
+
+  ```
+    Developer's Certificate of Origin 1.1
+    
+    By making a contribution to this project, I certify that:
+    
+    (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
+    
+    (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
+    
+    (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
+    
+    (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.
+  ```

zsol_bench-1.0.1/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

zsol_bench-1.0.1/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.4
+Name: zsol-bench
+Version: 1.0.1
+Summary: NVIDIA SOL ExecBench - GPU kernel evaluation framework
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: apache-tvm-ffi>=0.1.9
+Requires-Dist: click>=8.0
+Requires-Dist: cuda-tile==1.1.0
+Requires-Dist: datasets>=4.8.2
+Requires-Dist: ninja>=1.13.0
+Requires-Dist: nvidia-cudnn-frontend==1.18.0
+Requires-Dist: nvidia-cutlass-dsl[cu13]==4.4.1
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: rich>=13.0
+Requires-Dist: safetensors>=0.7.0
+Requires-Dist: torch-c-dlpack-ext>=0.1.5
+Requires-Dist: torch>=2.10.0
+Requires-Dist: torchvision>=0.24.1
+Description-Content-Type: text/markdown
+
+# SOL ExecBench
+
+<div align="center" id="top">
+
+[![Docs](https://img.shields.io/badge/docs-latest-brightgreen.svg)](/docs/)
+[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](LICENSE)
+
+[HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench)
+| [Technical Report](https://arxiv.org/abs/2603.19173) </div>
+
+**Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports.
+
+Kernels are:
+* Checked for various forms of reward hacking
+* Tested against a reference solution for numerical correctness
+* Timed under reproducible conditions
+
+Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)).  
+
+Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++.
+
+## Prerequisites
+
+- Docker with [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
+- [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/guides/cli) (`pip install huggingface-hub[cli]`)
+- NVIDIA driver version 580+
+
+## Setup
+
+### 1. Download benchmark data (one-time)
+
+```bash
+./scripts/download_data.sh
+```
+
+This downloads the [SOL-ExecBench](https://huggingface.co/datasets/nvidia/SOL-ExecBench) and [FlashInfer Trace](https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace) datasets into `data/`.
+
+### 2. Build and launch the Docker container
+
+```bash
+./scripts/run_docker.sh --build
+```
+
+This builds the image and drops you into an interactive shell inside the container. The repo's `src/`, `tests/`, and downloaded data are mounted automatically.
+
+## Evaluating a Solution
+
+Inside the container, use the `sol-execbench` CLI:
+
+```bash
+# Evaluate using a problem directory (contains definition.json + workload.jsonl)
+sol-execbench <problem_dir> --solution solution.json
+
+# Or specify files explicitly
+sol-execbench --definition def.json --workload wkl.jsonl --solution sol.json
+```
+
+### Example
+
+```bash
+# From the host — build, launch, and evaluate in one command:
+./scripts/run_docker.sh --build -- \
+  sol-execbench examples/cute_dsl/jamba_attn_proj \
+    --solution examples/cute_dsl/jamba_attn_proj/solution_cute_dsl.json
+
+# Or from inside the container:
+sol-execbench examples/cute_dsl/jamba_attn_proj \
+  --solution examples/cute_dsl/jamba_attn_proj/solution_cute_dsl.json
+```
+
+### CLI Options
+
+| Flag | Description |
+|---|---|
+| `--compile-timeout` | Compilation timeout in seconds (default: 120) |
+| `--timeout` | Evaluation timeout in seconds (default: 600) |
+| `-o, --output` | Write JSONL traces to file |
+| `--json` | Print traces as JSON to stdout |
+| `--lock-clocks` | Lock GPU clocks for stable benchmarks |
+| `--keep-staging` | Preserve staging directory after run |
+| `-v, --verbose` | Show subprocess output |
+
+## Running a Dataset
+
+Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
+
+```bash
+# Run all problems in the benchmark.
+# Auto builds solution.json from a single code file
+uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --solution-name solution.py
+
+# Run specific categories with multiple solution code files
+uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --category L1 L2 --solution-name solution.json
+
+# Run a single problem
+uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark/L1/my_problem
+
+# Limit number of problems and workloads
+uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --limit 5 --max-workloads 3 -o ./results
+```
+
+Results (traces and a summary JSON) are written to `out/run_dataset/` by default (override with `-o`). Problems that already passed are skipped on subsequent runs unless `--rerun` is specified.
+
+## Problem Format
+
+A problem directory contains:
+
+- **`definition.json`** — Kernel specification: function signature, tensor shapes, dtypes, reference implementation.
+- **`workload.jsonl`** — One JSON object per line, each defining input shapes, values, and tolerance thresholds.
+
+A solution is a separate JSON file referencing source files with the kernel implementation.
+
+See the full schema docs:
+
+- [Definition](docs/definition.md) — Kernel specification (function signature, tensor shapes, dtypes, reference code)
+- [Workload](docs/workload.md) — Concrete input configurations and tolerance thresholds
+- [Solution](docs/solution.md) — Source files and build specs for a kernel implementation
+- [Trace](docs/trace.md) — Evaluation output (correctness and performance results)
+
+## Citation
+
+```bibtex
+@misc{lin2026solexecbench,
+      title={SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits}, 
+      author={Edward Lin, Sahil Modi, Siva Kumar Sastry Hari, Qijing Huang, Zhifan Ye, Nestor Qin, Fengzhe Zhou, Yuan Zhang, Jingquan Wang, Sana Damani, Dheeraj Peri, Ouye Xie, Aditya Kane, Moshe Maor, Michael Behar, Triston Cao, Rishabh Mehta, Vartika Singh, Vikram Sharma Mailthody, Terry Chen, Zihao Ye, Hanfeng Chen, Tianqi Chen, Vinod Grover, Wei Chen, Wei Liu, Eric Chung, Luis Ceze, Roger Bringmann, Cyril Zeller, Michael Lightstone, Christos Kozyrakis, Humphrey Shi},
+      year={2026},
+      eprint={2603.19173},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2603.19173}, 
+}
+```
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE). Contributions require DCO sign-off — see [CONTRIBUTING.md](CONTRIBUTING.md).