cosmosforge 0.1.0__tar.gz

cosmosforge-0.1.0/.gitignore

@@ -0,0 +1,23 @@
+*.whl
+*.tar.gz
+
+reference/
+runs/
+workspace/
+experiences/
+
+__pycache__/
+*.egg-info/
+*.csv
+*.png
+*.json
+
+.venv/
+.env
+outputs/
+output/
+.pytest_cache/
+.ruff_cache/
+.DS_Store
+
+jobs/

cosmosforge-0.1.0/AGENTS.md

@@ -0,0 +1,34 @@
+你的目标是在这个文件夹下支持和完成一个针对 Harbor 格式的 terminal 任务的评估/rollout服务。
+
+## 总体思路
+- 评估包括
+    - 基本结构正确
+    - oracle 必须获得 1.0 reward，nop 必须获得 0.0 reward
+    - 13-grams 无重合
+    - rubrics 校验
+- rollout 服务包括
+    - 使用不同的 (harness, model) 组合来完成任务
+    - 报告 trace 
+    - 报告 best@N, mean@N, pass@N 等指标
+
+## 交付目标
+- 云端 service
+    - 能够支持解析用户提交上来的任务，完成评估/rollout，然后返回结果
+    - 支持大体量
+
+- 一个 cli 包
+    - 允许用户在本地通过 cli 提交任务到云端 service，获取评估/rollout 结果
+
+
+## doc 规范
+- 所有的文档必须使用英文。
+- 不要把过多细节的东西写入 README.md， README.md 主要用于介绍项目的功能和使用方法，过多的细节应该放在 docs/ 下面。
+- 除了 README.md 之外，每一个文档长度不能超过 100 行
+
+
+## 行为规范
+- 默认使用 harbor 这个 conda 环境，除非用户要求使用其他 conda 环境。
+- 非必要不用进行 HASH/SHA 校验（通常情况都不需要 HASH/SHA 校验）。
+- 要求你做可视化时，默认产出 pdf 而不是 html/svg。
+- 当我要求你 git add commit 的时候 请你将更新的内容组织成恰当个数的 commit，尽量不要把所有更新都放在一个 commit 里。每一个 commit message 用 `feat: ` / `fix: ` / `docs: ` / `test: ` / `refactor: ` / `chore: ` 开头，后面跟上简短的描述。
+- 所有的 rubrics review 和 rollout 都要基于 harbor 框架

cosmosforge-0.1.0/CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

cosmosforge-0.1.0/PKG-INFO

@@ -0,0 +1,11 @@
+Metadata-Version: 2.5
+Name: cosmosforge
+Version: 0.1.0
+Summary: CLI and evaluation service for Harbor-format terminal tasks.
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.12
+Description-Content-Type: text/markdown
+
+# CosmosForge

cosmosforge-0.1.0/README.md

@@ -0,0 +1 @@
+# CosmosForge

cosmosforge-0.1.0/docs/design/architecture.md

@@ -0,0 +1,91 @@
+# Architecture
+
+## Overview
+
+CosmosForge is a Harbor-format task evaluation and rollout service:
+
+- **Cloud service**: users submit Harbor task bundles, the service runs
+  evaluations and rollouts at scale, and returns results/traces.
+- **CLI**: local tool to submit tasks, poll status, fetch results, and run
+  rollouts with different (harness, model) combinations.
+
+```
+CLI --HTTP--> API --enqueue--> Redis <-- workers --> Docker (harbor task containers)
+                 |                                   |
+                 +--> PostgreSQL (metadata)          +--> /data volume (bundles, traces)
+```
+
+## Components
+
+### API (FastAPI + Uvicorn)
+- `POST /tasks` — upload a task bundle (tar/zip), validate structure, store it.
+- `POST /evaluations` — request an evaluation for a stored task (oracle/nop, 13-grams, rubrics).
+- `POST /rollouts` — request N rollouts with a given (harness, model) combo.
+- `GET /jobs/{id}`, `GET /results/{id}` — status and results.
+- `GET /healthz`, `GET /metrics` — liveness, queue depth, job counters.
+- Auth: static bearer token.
+
+### Queue (Redis + RQ)
+- One queue per job kind: `eval`, `rollout`.
+- Jobs carry: bundle path, config, callback/status pointer.
+- Redis bound to `127.0.0.1:6380` only.
+
+### Workers (RQ workers, N replicas)
+- Each worker processes one job at a time; concurrency = number of replicas.
+- Each job runs Harbor via its Python API inside a Docker task container
+  (workers mount the Docker socket).
+- Writes trace and result artifacts to the shared data volume, updates job
+  status in PostgreSQL.
+
+### PostgreSQL
+- Tables: `task_bundles`, `jobs`, `results`.
+- Single source of truth for job state; Redis only carries in-flight jobs.
+
+### Data volume (`/data/cosmosforge/`)
+- `bundles/` — uploaded task bundles.
+- `traces/` — rollout traces and logs.
+- `pg/` — postgres data (on the independent data disk).
+- `backups/` — nightly `pg_dump`, 7-day retention.
+
+## Evaluation core (shared library)
+
+A pure-Python package used by both workers and local CLI mode:
+
+1. **Structure check** — task.yaml, Dockerfile, tests present; Harbor-compatible layout.
+2. **Oracle/nop run** — oracle must score 1.0, nop must score 0.0.
+3. **13-grams check** — no 13-gram overlap between oracle and nop artifacts.
+4. **Rubric validation** — run Harbor rubrics and verify expected verdicts.
+5. **Metrics** — best@N, mean@N, pass@N aggregation over rollout results.
+
+## Deployment (single machine)
+
+Target host: 32 cores / 62 GB RAM / ~965 GB free on `/data`. Docker Compose:
+
+| Service   | Replicas | Limits        | Ports (host)   |
+|-----------|----------|---------------|----------------|
+| api       | 1        | 2 CPU / 2 GB  | 127.0.0.1:8100 |
+| redis     | 1        | 1 CPU / 1 GB  | 127.0.0.1:6380 |
+| postgres  | 1        | 1 CPU / 1 GB  | 127.0.0.1:5433 |
+| worker    | 8 (scale)| 2 CPU / 4 GB ea. | none        |
+
+Reliability measures:
+- `restart: unless-stopped` on all services.
+- Healthchecks; workers wait for healthy redis/postgres.
+- Log rotation: `max-size 50m`, `max-file 3`.
+- Memory limits per worker contain runaway rollouts.
+- API bound to loopback; external access via token + firewall/tunnel.
+- Conda env `harbor` is used for all local development and testing.
+
+## CLI
+
+Python package `cosmosforge` (Typer + Rich):
+- `cf submit <bundle>` — upload task bundle.
+- `cf eval <task-id>` — request evaluation, poll until done.
+- `cf rollout <task-id> --harness H --model M -n N` — request rollouts, report metrics.
+- `cf status` / `cf results <id>` — queue status and results.
+- `cf local eval <bundle>` — run evaluation fully locally (same core library), for debugging.
+
+## Scaling path
+
+Single compose host first. The queue abstraction (Redis/RQ) and stateless
+workers allow moving to multiple hosts or K8s later without API/CLI changes.

cosmosforge-0.1.0/docs/design/mvp-acceptance.md

@@ -0,0 +1,47 @@
+# MVP Acceptance Criteria (v0.1)
+
+The MVP proves the end-to-end loop on this single machine:
+submit -> evaluate -> get result, both via CLI and via the running service.
+
+## Scope
+
+Included: package skeleton, evaluation core, API, one worker, CLI local mode.
+Excluded: multi-worker scaling metrics, auth hardening, backups, dashboards.
+
+## Acceptance tests
+
+1. **Package installs**: `pip install -e .` in the `harbor` conda env succeeds;
+   `cf --help` works.
+
+2. **Structure check**: given a valid Harbor task bundle, the checker passes;
+   given a bundle missing `task.yaml` or `tests/`, it fails with a clear error.
+
+3. **Oracle/nop gate**: on a sample Harbor task,
+   - oracle run scores exactly 1.0,
+   - nop run scores exactly 0.0,
+   - the 13-gram check finds no overlap between oracle and nop artifacts.
+   A deliberately broken task (nop passes or oracle fails) is rejected.
+
+4. **Rubric validation**: rubrics on the sample task produce expected verdicts;
+   a task with a failing rubric is rejected.
+
+5. **Local evaluation**: `cf local eval <bundle>` runs the full evaluation
+   offline on this machine and prints a pass/fail report with scores.
+
+6. **Service up**: `docker compose up` starts api/redis/postgres/worker
+   (1 worker); all healthchecks pass; `/healthz` returns 200; services
+   restart automatically after `docker compose restart`.
+
+7. **Remote evaluation**: `cf submit <bundle>` uploads to the service;
+   `cf eval <task-id>` queues the job, the worker runs it, and the CLI
+   prints the same pass/fail report as local mode.
+
+8. **Rollout smoke test**: `cf rollout <task-id> --harness <h> --model <m> -n 2`
+   produces 2 traces on the server and a metrics report
+   (best@2 / mean@2 / pass@2) in the CLI. (A stub harness may stand in for a
+   real model in the MVP, as long as the loop and metrics are real.)
+
+## Done when
+
+All 8 items pass on this machine, with commands documented in the README,
+and `pytest` covers items 2–5 in CI-runnable unit/integration tests.

cosmosforge-0.1.0/examples/hello-task/environment/Dockerfile

@@ -0,0 +1,2 @@
+FROM ubuntu:24.04
+RUN apt-get update && apt-get install -y python3 && rm -rf /var/lib/apt/lists/*

cosmosforge-0.1.0/examples/hello-task/instruction.md

@@ -0,0 +1 @@
+Create a file at /home/user/hello.txt containing exactly the text "hello".

cosmosforge-0.1.0/examples/hello-task/solution/solve.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+echo "hello" > /home/user/hello.txt

cosmosforge-0.1.0/examples/hello-task/task.toml

@@ -0,0 +1,14 @@
+schema_version = "1.4"
+
+[metadata]
+name = "hello-task"
+description = "Create a hello.txt file with the content 'hello'."
+
+[verifier]
+timeout_sec = 300.0
+
+[agent]
+timeout_sec = 300.0
+
+[environment]
+build_timeout_sec = 600.0

cosmosforge-0.1.0/examples/hello-task/tests/test.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+apt-get update && apt-get install -y curl jq
+mkdir -p /logs/verifier
+if grep -qx "hello" /home/user/hello.txt; then
+  echo 1 > /logs/verifier/reward.txt
+else
+  echo 0 > /logs/verifier/reward.txt
+fi

cosmosforge-0.1.0/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "cosmosforge"
+version = "0.1.0"
+description = "CLI and evaluation service for Harbor-format terminal tasks."
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "typer>=0.12",
+    "rich>=13.0",
+    "pyyaml>=6.0",
+]
+
+[project.scripts]
+cf = "cosmosforge.client.cli:app"
+cf-server = "cosmosforge.server.cli:app"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/cosmosforge"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]