deltaleague 0.1.0__tar.gz

deltaleague-0.1.0/ARTIFACTS.md

@@ -0,0 +1,25 @@
+# Delta submission format version 1
+
+A directory contains `manifest.json` and `weights.safetensors`. The immutable parent version and its canonical content hash are mandatory. All tensor payloads use safetensors; no pickle or participant code is loaded.
+
+```json
+{
+  "schema_version": "delta.submission.v1",
+  "reconstruction_version": "delta-reconstruct-v1",
+  "model_config_id": "qwen35-08b-v1",
+  "parent_version_id": "UUID",
+  "parent_content_hash": "64 lowercase hex characters",
+  "format": "sparse-diff",
+  "operation": "replace",
+  "artifacts": [{"name": "weights.safetensors", "sha256": "64 lowercase hex characters", "size_bytes": 1024}],
+  "tensors": {"model.language_model.layers.0.mlp.down_proj.weight": {"indices": "t0.indices", "values": "t0.values"}}
+}
+```
+
+Sparse indices are int64, flat, strictly increasing and in bounds. Values are one-dimensional and match index count. Replacement sets those elements; addition accumulates parent and delta in float32 before casting to the registered parent dtype. Untouched tensors inherit the parent. `dense-diff` uses a `values` tensor matching the full target shape. `lora` maps each target to `a`, `b`, and finite `alpha`; apply `W + (alpha/r) * (B @ A)` in float32, then cast. A is `[r, in]`, B is `[out, r]`, rank is 1–256. `full` uses exact parent tensor names and shapes; changed frozen tensors are rejected.
+
+Canonical identity is SHA-256 over canonical JSON identity metadata and sorted named tensors, each with a length-prefixed JSON header (name, shape, dtype) followed by contiguous raw CPU bytes. Model/config/tokenizer identity participates in the hash. Server reconstruction is authoritative. Different submissions may resolve to one model content hash without sharing provenance.
+
+Client requests use `Authorization: Bearer <delta-token>`. Request `/api/v1/upload-sessions` with the manifest, PUT bytes to returned URLs, then POST `/api/v1/submissions` with `upload_id`, `title`, and an `Idempotency-Key` header. Retrying a submission uses the same key and upload. Signed URLs are credentials; do not publish them. Single-object uploads support the first model; multipart/resumable uploads remain a launch-completion task.
+
+All metrics are JSON. Validation is the equally weighted mean of configured benchmark scores on a 0–100 scale. Private test is a single overall score. A missing score is null, never zero. Private example text and generations are not returned. Public validation can be downloaded through `/api/v1/evaluation-configs/<id>`.

deltaleague-0.1.0/LOCAL.md

@@ -0,0 +1,75 @@
+# Run and evaluate locally
+
+Local inference uses Transformers and PyTorch. Apple Silicon Macs select MPS, Linux machines with a compatible PyTorch GPU build select CUDA, and other machines use CPU. `--device cpu`, `--device mps`, or `--device cuda` overrides selection. There is no rented compute, server evaluation quota, or automatic submission involved.
+
+## Install
+
+From the CLI repository, in a Python 3.12+ virtual environment:
+
+```sh
+pip install ".[local]"
+delta competition init qwen35-08b-v1 --output experiment
+cd experiment
+```
+
+The extra installs the tested Transformers version and math scorer. For Linux CUDA acceleration, install a compatible CUDA build of PyTorch using [the official installer](https://pytorch.org/get-started/locally/). If your existing PyTorch build is CPU-only, installing the CLI extra will not replace it with a CUDA build. For IFEval configurations, additionally install the CLI's `[ifeval]` extra.
+
+The portable implementation loads full weights, without automatic quantization. Allow several GB of free RAM for Qwen3.5-0.8B and more for long contexts. CPU/MPS use float32; CUDA uses BF16 where supported. On Macs, unsupported MPS operations can fall back to CPU. If your model encounters an unsupported operator or memory error, try `--device cpu` and lower the output budget. macOS acceleration has not been verified on physical Apple hardware in this release; Linux CPU has been exercised with the actual Qwen checkpoint. CUDA selection is implemented but has not been hardware-tested for this CLI release.
+
+## Ask the model
+
+```sh
+delta run "What is 2 + 2?" --max-new-tokens 64 --no-thinking
+delta run "Explain your solution to 17 * 23." --max-new-tokens 512 --json
+```
+
+`run` defaults to 256 output tokens. Thinking behavior otherwise comes from `evaluation.json`. A short budget may end before the final answer; the command reports that. Text goes to stdout, progress to stderr. `--json` adds provenance and token statistics. `--seed` defaults to 1000. Use `--workspace PATH` to work from another directory.
+
+## Run any leaderboard entry
+
+```sh
+delta leaderboard --model qwen35-08b-v1
+# Copy the desired entry's version_id, not its submission ID.
+delta run "What is 2 + 2?" --version VERSION_ID --no-thinking
+delta evaluate --version VERSION_ID --limit 2 --max-new-tokens 256 --output entry-quick.json
+```
+
+`--version` downloads the immutable resulting checkpoint, verifies its hashes, and caches it under `.delta/models/VERSION_ID` inside the workspace. It works for the original model and submitted model versions, regardless of whether the submission used a LoRA, diff, or full weights. The version must belong to the workspace's model. Downloads need network access but not login. The inference runs locally; no private test data is downloaded. Completed cached files are reused on subsequent commands.
+
+For a fully offline workspace based on that entry, run `delta competition init MODEL_ID --parent VERSION_ID --output another-experiment` while online, then run/evaluate from that workspace without `--version`.
+
+## Try your own edit or patch
+
+```sh
+delta run "What is 2 + 2?" --weights edited.safetensors --no-thinking
+delta evaluate --weights edited.safetensors --limit 2 --max-new-tokens 256 --output edit-quick.json
+
+delta evaluate --patch submission --limit 2 --max-new-tokens 256 --output patch-quick.json
+```
+
+`--weights` takes a full modified safetensors checkpoint, using the workspace parent's tokenizer and configuration. `--patch` takes a submission directory with `manifest.json`; it checks the parent and artifact hashes, then reconstructs supported LoRA/sparse/dense/full artifacts into a temporary checkpoint. Neither changes the parent. You can combine either with `--version` when editing that leaderboard checkpoint; a patch must reference that exact version. Temporary reconstructed files need disk space roughly equal to a full checkpoint and are removed on completion.
+
+## Evaluate public validation
+
+Start small:
+
+```sh
+delta evaluate --limit 2 --max-new-tokens 256 --output baseline-quick.json
+```
+
+`--limit` selects the first N problems **per benchmark**; `--benchmark gsm8k` selects one benchmark. These are development checks, not representative estimates of the full public score.
+
+For the entire public set with its configured generation settings:
+
+```sh
+delta evaluate --output baseline-full.json
+delta evaluate --weights edited.safetensors --output edit-full.json
+```
+
+Full evaluation can be expensive locally: the starter configuration permits 32,768 generated tokens per example. The CLI prints the number of generations and upper token budget before loading the model. Use Ctrl-C to interrupt. Completed generation records remain in the JSONL file; no complete score report is written for an interrupted run. Choose a fresh output filename to rerun; automatic resume is not implemented.
+
+The JSON report contains per-benchmark scores, the equal-weight mean across selected benchmarks, per-example sample averaging, checkpoint hash, source version, validation hash, runtime versions, actual generation settings, coverage, and truncation statistics. A companion `*.generations.jsonl` contains answers and per-sample scores. Existing output files are never overwritten.
+
+The local scorer shares the platform's answer-scoring implementation: reasoning traces before `</think>` are excluded, math uses math-verify, and multiple-choice answers follow the same extraction rules. Configured samples use the same per-sample seeds. `--max-new-tokens` and `--thinking/--no-thinking` overrides are recorded explicitly.
+
+Scores remain **advisory local results**. Transformers, vLLM, device precision, and sampling implementations can differ even with the same seed. Compare candidates using the same local settings, then submit for the official validation and private test scores. Local evaluation does not upload results or assign a leaderboard rank.

deltaleague-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md ARTIFACTS.md LOCAL.md
+prune .github
+prune .delta
+prune var
+global-exclude *.safetensors *.bin *.pt *.pth .env .env.* auth.json .delta-upload.json *.generations.jsonl

deltaleague-0.1.0/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: deltaleague
+Version: 0.1.0
+Summary: CLI for Delta League model-weight competitions, submissions, and local evaluation
+Author: Delta League
+Project-URL: Homepage, https://deltaleague.ai
+Project-URL: Repository, https://github.com/ilblackdragon/deltaleague
+Project-URL: Documentation, https://github.com/ilblackdragon/deltaleague/blob/main/README.md
+Project-URL: Issues, https://github.com/ilblackdragon/deltaleague/issues
+Keywords: llm,benchmark,model-weights,leaderboard
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: httpx<1,>=0.28
+Requires-Dist: typer<1,>=0.16
+Requires-Dist: safetensors<1,>=0.5
+Requires-Dist: torch>=2.6
+Provides-Extra: local
+Requires-Dist: transformers==5.22.1; extra == "local"
+Requires-Dist: math-verify==0.9.0; extra == "local"
+Provides-Extra: ifeval
+Requires-Dist: lm-eval<0.5,>=0.4.9; extra == "ifeval"
+
+# Delta League CLI
+
+The CLI and submission tools for [Delta League](https://deltaleague.ai): the open competition to improve intelligence.
+
+Submit a patch. Prove an improvement. Advance the model.
+
+Repository: [ilblackdragon/deltaleague](https://github.com/ilblackdragon/deltaleague). Private during launch preparation; intended to become public. The platform is maintained separately. This package contains no private test data.
+
+## Quick start
+
+Requires Python 3.12+. Clone this repository, then install `pip install .` in a virtual environment. CPU PyTorch is sufficient for downloading and building patches; inference requires separate compute.
+
+```sh
+delta competition list
+delta competition init qwen35-08b-v1 --output experiment
+cd experiment
+```
+
+This selects a model competition and downloads its original checkpoint into `model/`, public examples into `validation.json`, and scoring/generation settings into `evaluation.json`. `delta-workspace.json` records the API, immutable parent version, and evaluation version. Use `--parent VERSION_ID` to start from another submission in the same competition. Downloads verify hashes, reuse completed files on retry, and avoid overwriting modified files.
+
+Have your agent edit `model/model.safetensors` into a **separate** `edited.safetensors` file. Preserve tensor names, shapes, and dtypes. Keep the parent unchanged. Then:
+
+```sh
+delta patch build --parent model --modified edited.safetensors --output submission
+delta validate submission
+delta login --no-browser
+# Open the research PR first; replace PR_URL with its actual URL.
+delta submit submission --title "My weight edit" --pr PR_URL
+delta submission status SUBMISSION_ID --watch
+delta submission results SUBMISSION_ID --json
+delta leaderboard --model qwen35-08b-v1
+```
+
+Before submitting, open a research PR in [the public repository](https://github.com/ilblackdragon/deltaleague) containing your hypothesis, research trace, edit/reproduction code, parent and evaluation IDs, seeds, and public validation findings. Attach its actual URL with `--pr`. The CLI records the link; it does not create the PR. Source-free submissions remain supported by omitting `--pr`.
+
+Open the authorization URL in a browser and sign in with GitHub. `delta login` opens the browser automatically; `--no-browser` suits remote shells and agents. Authentication is only required for submission/account actions. Credentials are stored with mode 0600 and scoped to the API origin. Put `--api URL` before a command for another deployment.
+
+## Run and evaluate on your computer
+
+Install with `pip install ".[local]"` from this repository, then run these inside an initialized competition workspace:
+
+```sh
+delta run "What is 2 + 2?" --no-thinking
+delta evaluate --limit 2 --max-new-tokens 256 --output baseline-quick.json
+delta evaluate --weights edited.safetensors --limit 2 --max-new-tokens 256 --output edit-quick.json
+# Any entry from `delta leaderboard` can be run or evaluated by version_id:
+delta run "What is 2 + 2?" --version VERSION_ID --no-thinking
+```
+
+Macs use MPS where available; Linux uses CUDA where available; CPU is supported on both. Local scores are advisory. See [LOCAL.md](LOCAL.md) for installation, full public evaluation, applying submission patches, hardware limits, and reproducibility details.
+
+## Agent starting prompt
+
+> Work on Delta League using its CLI. Run `delta competition list`, select `qwen35-08b-v1`, and initialize a fresh experiment with `delta competition init`. Read evaluation.json and validation.json. Install the CLI local extra and use `delta run` for inference and `delta evaluate` for public validation; start with a bounded --limit and --max-new-tokens budget, recording the overrides. Use --version VERSION_ID to test a leaderboard entry, --weights for a full edited checkpoint, or --patch for a submission directory. Keep the downloaded parent unchanged and save candidate weights separately. Record your hypothesis, changes, and public validation results. Build and validate a patch. Before submitting, open a research PR in ilblackdragon/deltaleague with the hypothesis, research trace, reproducible code, parent/evaluation IDs, seeds, and public validation findings. Exclude credentials, private test data, and weight artifacts. Ask me to complete browser authorization if needed, then submit with `delta submit submission --title "SHORT DESCRIPTION" --pr ACTUAL_PR_URL` and watch the result. The CLI does not create the PR automatically. If I explicitly choose a source-free submission, omit `--pr`. Update the research PR with the submission link and verified results when available. Never request private test examples or credentials. The server's official validation and private test scores are authoritative; local validation scores are advisory.
+
+See [the published-edit reproduction recipe](examples/blog-reproduction/README.md) for a concrete first experiment and [ARTIFACTS.md](ARTIFACTS.md) for LoRA, sparse/dense diff, and full-checkpoint formats. `delta validate` checks manifest/file integrity; the server also checks parent compatibility, numerical validity, and reconstruction. JSON output supports agent use. Public validation scores and private test scores share one leaderboard row, ranked by private test score.

deltaleague-0.1.0/README.md

@@ -0,0 +1,56 @@
+# Delta League CLI
+
+The CLI and submission tools for [Delta League](https://deltaleague.ai): the open competition to improve intelligence.
+
+Submit a patch. Prove an improvement. Advance the model.
+
+Repository: [ilblackdragon/deltaleague](https://github.com/ilblackdragon/deltaleague). Private during launch preparation; intended to become public. The platform is maintained separately. This package contains no private test data.
+
+## Quick start
+
+Requires Python 3.12+. Clone this repository, then install `pip install .` in a virtual environment. CPU PyTorch is sufficient for downloading and building patches; inference requires separate compute.
+
+```sh
+delta competition list
+delta competition init qwen35-08b-v1 --output experiment
+cd experiment
+```
+
+This selects a model competition and downloads its original checkpoint into `model/`, public examples into `validation.json`, and scoring/generation settings into `evaluation.json`. `delta-workspace.json` records the API, immutable parent version, and evaluation version. Use `--parent VERSION_ID` to start from another submission in the same competition. Downloads verify hashes, reuse completed files on retry, and avoid overwriting modified files.
+
+Have your agent edit `model/model.safetensors` into a **separate** `edited.safetensors` file. Preserve tensor names, shapes, and dtypes. Keep the parent unchanged. Then:
+
+```sh
+delta patch build --parent model --modified edited.safetensors --output submission
+delta validate submission
+delta login --no-browser
+# Open the research PR first; replace PR_URL with its actual URL.
+delta submit submission --title "My weight edit" --pr PR_URL
+delta submission status SUBMISSION_ID --watch
+delta submission results SUBMISSION_ID --json
+delta leaderboard --model qwen35-08b-v1
+```
+
+Before submitting, open a research PR in [the public repository](https://github.com/ilblackdragon/deltaleague) containing your hypothesis, research trace, edit/reproduction code, parent and evaluation IDs, seeds, and public validation findings. Attach its actual URL with `--pr`. The CLI records the link; it does not create the PR. Source-free submissions remain supported by omitting `--pr`.
+
+Open the authorization URL in a browser and sign in with GitHub. `delta login` opens the browser automatically; `--no-browser` suits remote shells and agents. Authentication is only required for submission/account actions. Credentials are stored with mode 0600 and scoped to the API origin. Put `--api URL` before a command for another deployment.
+
+## Run and evaluate on your computer
+
+Install with `pip install ".[local]"` from this repository, then run these inside an initialized competition workspace:
+
+```sh
+delta run "What is 2 + 2?" --no-thinking
+delta evaluate --limit 2 --max-new-tokens 256 --output baseline-quick.json
+delta evaluate --weights edited.safetensors --limit 2 --max-new-tokens 256 --output edit-quick.json
+# Any entry from `delta leaderboard` can be run or evaluated by version_id:
+delta run "What is 2 + 2?" --version VERSION_ID --no-thinking
+```
+
+Macs use MPS where available; Linux uses CUDA where available; CPU is supported on both. Local scores are advisory. See [LOCAL.md](LOCAL.md) for installation, full public evaluation, applying submission patches, hardware limits, and reproducibility details.
+
+## Agent starting prompt
+
+> Work on Delta League using its CLI. Run `delta competition list`, select `qwen35-08b-v1`, and initialize a fresh experiment with `delta competition init`. Read evaluation.json and validation.json. Install the CLI local extra and use `delta run` for inference and `delta evaluate` for public validation; start with a bounded --limit and --max-new-tokens budget, recording the overrides. Use --version VERSION_ID to test a leaderboard entry, --weights for a full edited checkpoint, or --patch for a submission directory. Keep the downloaded parent unchanged and save candidate weights separately. Record your hypothesis, changes, and public validation results. Build and validate a patch. Before submitting, open a research PR in ilblackdragon/deltaleague with the hypothesis, research trace, reproducible code, parent/evaluation IDs, seeds, and public validation findings. Exclude credentials, private test data, and weight artifacts. Ask me to complete browser authorization if needed, then submit with `delta submit submission --title "SHORT DESCRIPTION" --pr ACTUAL_PR_URL` and watch the result. The CLI does not create the PR automatically. If I explicitly choose a source-free submission, omit `--pr`. Update the research PR with the submission link and verified results when available. Never request private test examples or credentials. The server's official validation and private test scores are authoritative; local validation scores are advisory.
+
+See [the published-edit reproduction recipe](examples/blog-reproduction/README.md) for a concrete first experiment and [ARTIFACTS.md](ARTIFACTS.md) for LoRA, sparse/dense diff, and full-checkpoint formats. `delta validate` checks manifest/file integrity; the server also checks parent compatibility, numerical validity, and reconstruction. JSON output supports agent use. Public validation scores and private test scores share one leaderboard row, ranked by private test score.

deltaleague-0.1.0/delta_cli/artifacts.py

@@ -0,0 +1,163 @@
+"""Portable weight artifact contract. This module never imports the platform."""
+import hashlib
+import json
+import math
+import re
+import uuid
+from pathlib import Path
+
+
+def canonical_json(value):
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
+
+
+def file_hash(path):
+    with open(path, "rb") as f:
+        return hashlib.file_digest(f, "sha256").hexdigest()
+
+
+def validate_manifest(manifest):
+    if not isinstance(manifest, dict):
+        raise ValueError("invalid_manifest")
+    if manifest.get("schema_version") != "delta.submission.v1":
+        raise ValueError("unsupported_schema")
+    if manifest.get("format") not in {"sparse-diff", "dense-diff", "lora", "full"}:
+        raise ValueError("unsupported_format")
+    if manifest.get("reconstruction_version") != "delta-reconstruct-v1":
+        raise ValueError("unsupported_reconstruction")
+    try:
+        uuid.UUID(str(manifest["parent_version_id"]))
+    except (ValueError, KeyError):
+        raise ValueError("invalid_parent_version")
+    if not re.fullmatch(r"[0-9a-f]{64}", str(manifest.get("parent_content_hash", ""))):
+        raise ValueError("invalid_parent_hash")
+    if not re.fullmatch(r"[-a-zA-Z0-9_]{1,100}", str(manifest.get("model_config_id", ""))):
+        raise ValueError("invalid_model_config")
+    files = manifest.get("artifacts", [])
+    if not isinstance(files, list) or not all(isinstance(f, dict) for f in files):
+        raise ValueError("invalid_artifact_inventory")
+    if len(files) != 1 or files[0].get("name") != "weights.safetensors":
+        raise ValueError("expected_weights_safetensors")
+    for f in files:
+        if not re.fullmatch(r"[0-9a-f]{64}", f.get("sha256", "")):
+            raise ValueError("invalid_hash")
+        if type(f.get("size_bytes")) is not int or f["size_bytes"] <= 0:
+            raise ValueError("invalid_size")
+    if manifest.get("operation", "replace") not in {"replace", "add"}:
+        raise ValueError("unsupported_operation")
+    mapping = manifest.get("tensors", {})
+    if not isinstance(mapping, dict) or not all(isinstance(k, str) and isinstance(v, dict) for k, v in mapping.items()):
+        raise ValueError("invalid_tensor_mapping")
+    for target, spec in mapping.items():
+        if len(target) > 1024:
+            raise ValueError("invalid_tensor_name")
+        required = {"a", "b", "alpha"} if manifest["format"] == "lora" else ({"indices", "values"} if manifest["format"] == "sparse-diff" else {"values"})
+        if set(spec) != required or any(not isinstance(v, str) for k, v in spec.items() if k != "alpha"):
+            raise ValueError("invalid_tensor_mapping")
+    if len(mapping) > 10000:
+        raise ValueError("too_many_tensors")
+
+
+def content_hash(tensors, identity_hash):
+    import torch
+    h = hashlib.sha256(canonical_json({"schema": "delta-content-v1", "identity": identity_hash}))
+    for name, tensor in sorted(tensors.items()):
+        t = tensor.detach().cpu().contiguous()
+        header = canonical_json([name, list(t.shape), str(t.dtype)])
+        h.update(len(header).to_bytes(8, "little"))
+        h.update(header)
+        h.update(t.reshape(-1).view(torch.uint8).numpy().tobytes())
+    return h.hexdigest()
+
+
+def reconstruct(parent_path, artifact_path, manifest, allowed_prefix=""):
+    import torch
+    from safetensors.torch import load_file
+    validate_manifest(manifest)
+    base = load_file(str(parent_path))
+    patch = load_file(str(artifact_path))
+    if any(not torch.isfinite(t).all() for t in patch.values()):
+        raise ValueError("nonfinite_patch")
+    result = {k: v.clone() for k, v in base.items()}
+    used = set()
+    if manifest["format"] == "full":
+        if set(patch) != set(base):
+            raise ValueError("tensor_inventory")
+        mapping = {k: {"values": k} for k in patch}
+    else:
+        mapping = manifest.get("tensors", {})
+        if not mapping:
+            raise ValueError("empty_patch")
+    for name, spec in mapping.items():
+        if name not in base:
+            raise ValueError("unknown_tensor")
+        original = base[name]
+        if not name.startswith(allowed_prefix):
+            # Full checkpoints may carry unchanged frozen tensors.
+            if manifest["format"] == "full" and torch.equal(patch[name], original):
+                used.add(name)
+                continue
+            raise ValueError("frozen_tensor")
+        kind = manifest["format"]
+        if kind == "lora":
+            a, b = patch[spec["a"]], patch[spec["b"]]
+            used.update([spec["a"], spec["b"]])
+            if a.ndim != 2 or b.ndim != 2 or original.ndim != 2 or a.shape[0] != b.shape[1] or not 1 <= a.shape[0] <= 256:
+                raise ValueError("invalid_lora_rank")
+            alpha = float(spec["alpha"])
+            if not math.isfinite(alpha) or (b.shape[0], a.shape[1]) != tuple(original.shape):
+                raise ValueError("invalid_lora_shape")
+            updated = original.float() + (b.float() @ a.float()) * (alpha / a.shape[0])
+        else:
+            values = patch[spec["values"]]
+            used.add(spec["values"])
+            if kind == "sparse-diff":
+                indices = patch[spec["indices"]]
+                used.add(spec["indices"])
+                if indices.dtype != torch.int64 or indices.ndim != 1 or values.ndim != 1 or indices.numel() != values.numel():
+                    raise ValueError("invalid_sparse_shape")
+                if indices.numel() and (indices[0] < 0 or indices[-1] >= original.numel() or not (indices[1:] > indices[:-1]).all()):
+                    raise ValueError("invalid_sparse_indices")
+                updated = original.float().clone().reshape(-1)
+                updated[indices] = values.float() if manifest.get("operation", "replace") == "replace" else updated[indices] + values.float()
+                updated = updated.reshape(original.shape)
+            else:
+                if values.shape != original.shape:
+                    raise ValueError("tensor_shape")
+                updated = values if kind == "full" or manifest.get("operation", "replace") == "replace" else original.float() + values.float()
+        result[name] = updated.to(original.dtype).contiguous()
+        if not torch.isfinite(result[name]).all():
+            raise ValueError("nonfinite_result")
+    if used != set(patch):
+        raise ValueError("unlisted_tensors")
+    return result
+
+
+def build_sparse(parent, modified, output, parent_id, parent_hash, model_id):
+    import torch
+    from safetensors.torch import load_file, save_file
+    output = Path(output)
+    output.mkdir(parents=True, exist_ok=True)
+    before, after = load_file(str(parent)), load_file(str(modified))
+    if set(before) != set(after):
+        raise ValueError("tensor_inventory")
+    tensors, mapping = {}, {}
+    for number, name in enumerate(sorted(before)):
+        a, b = before[name], after[name]
+        if a.shape != b.shape or a.dtype != b.dtype:
+            raise ValueError("tensor_shape_or_dtype")
+        idx = torch.nonzero(a.reshape(-1) != b.reshape(-1)).flatten()
+        if idx.numel():
+            ik, vk = f"t{number}.indices", f"t{number}.values"
+            tensors[ik], tensors[vk] = idx, b.reshape(-1)[idx]
+            mapping[name] = {"indices": ik, "values": vk}
+    if not tensors:
+        raise ValueError("empty_patch")
+    path = output / "weights.safetensors"
+    save_file(tensors, str(path))
+    manifest = {"schema_version": "delta.submission.v1", "format": "sparse-diff", "operation": "replace",
+                "reconstruction_version": "delta-reconstruct-v1", "parent_version_id": str(parent_id),
+                "parent_content_hash": parent_hash, "model_config_id": model_id, "tensors": mapping,
+                "artifacts": [{"name": path.name, "sha256": file_hash(path), "size_bytes": path.stat().st_size}]}
+    (output / "manifest.json").write_text(json.dumps(manifest, indent=2))
+    return manifest

deltaleague-0.1.0/delta_cli/local.py

@@ -0,0 +1,193 @@
+"""Offline text inference and public validation; no platform or private-data imports."""
+from contextlib import contextmanager
+from importlib.metadata import version
+import json
+import os
+from pathlib import Path
+import statistics
+import tempfile
+import time
+
+from .artifacts import file_hash, reconstruct, validate_manifest
+from .scoring import score_answer
+
+
+def read_workspace(workspace):
+    workspace = Path(workspace).resolve()
+    identity = json.loads((workspace / 'delta-workspace.json').read_text())
+    config = json.loads((workspace / 'evaluation.json').read_text())
+    metadata = json.loads((workspace / 'model/delta-model.json').read_text())
+    if (identity['config_id'] != config['id'] or identity['model_id'] != config['model_id']
+            or identity['parent_version_id'] != metadata['id'] or identity['model_id'] != metadata['model_id']):
+        raise ValueError('Workspace model, parent, or evaluation configuration mismatch. Initialize a fresh workspace.')
+    return workspace, identity, config, metadata
+
+
+@contextmanager
+def model_directory(workspace, metadata, weights=None, patch=None, parent=None):
+    """Overlay a candidate without mutating the downloaded parent or tokenizer."""
+    if weights and patch:
+        raise ValueError('Choose either --weights or --patch.')
+    parent = parent or workspace / 'model'
+    for artifact in metadata['files']:
+        name = artifact['name']
+        if Path(name).name != name:
+            raise ValueError('Invalid model artifact filename.')
+        if name == 'model.safetensors' and weights:
+            continue
+        path = parent / name
+        if path.stat().st_size != artifact['size_bytes'] or file_hash(path) != artifact['sha256']:
+            raise ValueError(f'Parent artifact changed: {name}. Keep the parent untouched and use --weights or --patch.')
+    if not weights and not patch:
+        yield parent, {'kind': 'parent', 'weights_sha256': file_hash(parent / 'model.safetensors')}
+        return
+    with tempfile.TemporaryDirectory(prefix='delta-local-') as temp:
+        target = Path(temp)
+        for artifact in metadata['files']:
+            if artifact['name'] != 'model.safetensors':
+                (target / artifact['name']).symlink_to((parent / artifact['name']).resolve())
+        if patch:
+            from safetensors.torch import save_file
+            patch = Path(patch).resolve()
+            manifest = json.loads((patch / 'manifest.json').read_text())
+            validate_manifest(manifest)
+            if (manifest['parent_version_id'] != metadata['id'] or manifest['parent_content_hash'] != metadata['content_hash']
+                    or manifest['model_config_id'] != metadata['model_id']):
+                raise ValueError('Patch does not reference this workspace parent.')
+            for artifact in manifest['artifacts']:
+                path = patch / artifact['name']
+                if path.stat().st_size != artifact['size_bytes'] or file_hash(path) != artifact['sha256']:
+                    raise ValueError('Patch artifact integrity failure.')
+            tensors = reconstruct(parent / 'model.safetensors', patch / manifest['artifacts'][0]['name'], manifest)
+            save_file(tensors, str(target / 'model.safetensors'))
+            del tensors
+        else:
+            weights = Path(weights).resolve()
+            if not weights.is_file() or weights.suffix != '.safetensors':
+                raise ValueError('--weights must be a full .safetensors checkpoint.')
+            (target / 'model.safetensors').symlink_to(weights)
+        yield target, {'kind': 'patch' if patch else 'weights', 'weights_sha256': file_hash(target / 'model.safetensors')}
+
+
+def select_device(torch, requested):
+    if requested not in {'auto', 'cpu', 'cuda', 'mps'}:
+        raise ValueError('--device must be auto, cpu, cuda, or mps.')
+    if requested == 'auto':
+        return 'cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
+    if requested == 'cuda' and not torch.cuda.is_available():
+        raise ValueError('CUDA is unavailable. Install a CUDA-enabled PyTorch build or use --device cpu.')
+    if requested == 'mps' and not torch.backends.mps.is_available():
+        raise ValueError('MPS is unavailable. Use a supported Apple Silicon Mac or --device cpu.')
+    return requested
+
+
+class Runner:
+    def __init__(self, model_path, device='auto'):
+        import torch
+        from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM, AutoModelForImageTextToText
+        self.torch = torch
+        self.device = select_device(torch, device)
+        # Float32 is the portable CPU/MPS path; CUDA uses BF16 when supported.
+        dtype = torch.bfloat16 if self.device == 'cuda' and torch.cuda.is_bf16_supported() else torch.float32
+        self.tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True, trust_remote_code=False)
+        config = AutoConfig.from_pretrained(model_path, local_files_only=True, trust_remote_code=False)
+        auto = AutoModelForImageTextToText if type(config) in AutoModelForImageTextToText._model_mapping else AutoModelForCausalLM
+        self.model, info = auto.from_pretrained(model_path, config=config, local_files_only=True, trust_remote_code=False,
+                                                use_safetensors=True, dtype=dtype, output_loading_info=True)
+        if info.get('missing_keys') or info.get('unexpected_keys') or info.get('mismatched_keys') or info.get('error_msgs'):
+            raise ValueError('Checkpoint tensors do not match the model configuration; refusing partially initialized weights.')
+        self.model.to(self.device).eval()
+        self.details = {'backend': 'transformers', 'transformers': version('transformers'), 'torch': torch.__version__,
+                        'device': self.device, 'dtype': str(dtype),
+                        'mps_cpu_fallback': self.device == 'mps' and os.getenv('PYTORCH_ENABLE_MPS_FALLBACK') == '1'}
+
+    def generate(self, prompt, generation, seed):
+        from transformers import LogitsProcessor, LogitsProcessorList, set_seed
+        torch = self.torch
+        rendered = self.tokenizer.apply_chat_template([{'role': 'user', 'content': prompt}], tokenize=False,
+                    add_generation_prompt=True, enable_thinking=generation.get('enable_thinking', True))
+        inputs = self.tokenizer(rendered, return_tensors='pt', add_special_tokens=False).to(self.device)
+        prompt_length = inputs['input_ids'].shape[1]
+        max_tokens = generation.get('max_tokens', 32768)
+        if prompt_length + max_tokens > generation.get('max_model_len', 34816):
+            raise ValueError('Prompt plus output budget exceeds max_model_len; reduce --max-new-tokens.')
+        penalty = generation.get('presence_penalty', 1.5)
+
+        class PresencePenalty(LogitsProcessor):
+            def __call__(self, input_ids, scores):
+                # Match vLLM: presence penalty applies to generated tokens, not the prompt.
+                for row in range(input_ids.shape[0]):
+                    ids = input_ids[row, prompt_length:].unique()
+                    scores[row, ids] -= penalty
+                return scores
+
+        temperature = generation.get('temperature', .6)
+        kwargs = {'max_new_tokens': max_tokens, 'do_sample': temperature > 0,
+                  'logits_processor': LogitsProcessorList([PresencePenalty()])}
+        if temperature > 0:
+            kwargs.update(temperature=temperature, top_p=generation.get('top_p', .95), top_k=generation.get('top_k', 20))
+        set_seed(seed)
+        start = time.monotonic()
+        with torch.inference_mode():
+            tokens = self.model.generate(**inputs, **kwargs)[0, prompt_length:]
+        count = tokens.numel()
+        # Preserve </think> so the same scorer can remove reasoning traces as on the server.
+        text = self.tokenizer.decode(tokens, skip_special_tokens=False)
+        for token in [self.tokenizer.eos_token, self.tokenizer.pad_token]:
+            if token:
+                text = text.replace(token, '')
+        return {'text': text, 'tokens': count, 'truncated': count == max_tokens,
+                'generation_seconds': time.monotonic() - start}
+
+
+def public_examples(workspace, config, benchmark=None, limit=None):
+    path = workspace / 'validation.json'
+    if file_hash(path) != config['validation_sha256']:
+        raise ValueError('Public validation data changed; initialize a fresh workspace.')
+    data = json.loads(path.read_text())
+    names = config['validation_names']
+    if set(data) != set(names) or not names:
+        raise ValueError('Public validation benchmark inventory mismatch.')
+    if benchmark and benchmark not in names:
+        raise ValueError('Unknown benchmark; choose from: ' + ', '.join(names))
+    selected = [benchmark] if benchmark else names
+    rows = []
+    for name in selected:
+        if not data[name]:
+            raise ValueError(f'Empty benchmark: {name}')
+        for index, example in enumerate(data[name][:limit] if limit else data[name]):
+            scorer = example.get('scorer', 'exact')
+            if scorer not in {'exact', 'choice', 'math', 'ifeval'}:
+                raise ValueError(f'Unsupported scorer: {scorer}')
+            # Fail on missing scoring dependencies before spending time on inference.
+            if scorer == 'math':
+                __import__('math_verify')
+            elif scorer == 'ifeval':
+                __import__('lm_eval.tasks.ifeval.instructions_registry')
+            samples = example.get('samples', 1)
+            if type(samples) is not int or not 1 <= samples <= 16:
+                raise ValueError('Invalid sample count.')
+            for sample in range(samples):
+                rows.append({'benchmark': name, 'example_index': index, 'sample': sample,
+                             'seed': 1000 + sample, 'example': example})
+    return rows, selected, not benchmark and all(limit is None or limit >= len(v) for v in data.values())
+
+
+def evaluate(runner, tasks, generation, stream, progress):
+    scores, tokens, seconds, truncated = {}, 0, 0, 0
+    for index, task in enumerate(tasks, 1):
+        result = runner.generate(task['example']['prompt'], generation, task['seed'])
+        score = score_answer(task['example'], result['text'])
+        scores.setdefault(task['benchmark'], {}).setdefault(task['example_index'], []).append(score)
+        row = {k: v for k, v in task.items() if k != 'example'}
+        row.update(example_id=task['example']['id'], score=score, **result)
+        stream.write(json.dumps(row) + '\n')
+        stream.flush()
+        tokens += result['tokens']
+        seconds += result['generation_seconds']
+        truncated += result['truncated']
+        progress(index, len(tasks), task['benchmark'])
+    components = {name: statistics.mean(statistics.mean(samples) for samples in examples.values()) * 100
+                  for name, examples in scores.items()}
+    return {'validation_score': statistics.mean(components.values()), 'validation_benchmarks': components,
+            'output_tokens': tokens, 'generation_seconds': seconds, 'truncation_rate': truncated / len(tasks)}