lastlight 0.6.1 → 0.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"$schema": "https://json.schemastore.org/claude-code-plugin.json",
|
|
3
3
|
"name": "lastlight",
|
|
4
4
|
"displayName": "Last Light",
|
|
5
|
-
"version": "0.6.
|
|
5
|
+
"version": "0.6.2",
|
|
6
6
|
"description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
|
|
7
7
|
"author": { "name": "Clifton Cunningham" },
|
|
8
8
|
"homepage": "https://github.com/cliftonc/lastlight",
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: lastlight-evals
|
|
3
3
|
description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases (triage / code-fix instances). GitHub is mocked, so no real GitHub token is needed — only a model provider API key.
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.1.0
|
|
5
5
|
tags: [lastlight, evals, benchmark, models, swe-bench]
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -23,16 +23,42 @@ command -v lastlight-evals >/dev/null && echo "installed" || npm i -g lastlight-
|
|
|
23
23
|
|
|
24
24
|
## 2. Scaffold a workspace
|
|
25
25
|
|
|
26
|
+
`init` is **non-interactive** when there's no TTY (piped/agent/CI) — it never
|
|
27
|
+
blocks on a prompt. Two layouts:
|
|
28
|
+
|
|
29
|
+
### Recommended — Separate (point evals at your existing deployment overlay)
|
|
30
|
+
|
|
31
|
+
If you already have a deployment overlay repo (e.g. `your-org/lastlight-instance`),
|
|
32
|
+
this is the default. It runs your **real** overlay (its config, workflows, skills,
|
|
33
|
+
persona) and keeps the eval datasets out of your deployment repo:
|
|
34
|
+
|
|
26
35
|
```bash
|
|
27
|
-
lastlight-evals init my-evals
|
|
36
|
+
lastlight-evals init my-evals --clone your-org/lastlight-instance
|
|
37
|
+
cd my-evals && lastlight-evals run # bare run — overlay + datasets auto-detected
|
|
28
38
|
```
|
|
29
39
|
|
|
30
|
-
This
|
|
31
|
-
- `
|
|
32
|
-
|
|
40
|
+
This produces:
|
|
41
|
+
- `instance/` — your overlay, cloned as its **own git checkout** (git-ignored
|
|
42
|
+
here; `cd instance && git pull` to update it)
|
|
43
|
+
- `evals/datasets/` — seeded from the built-in `triage` + `code-fix` samples (this
|
|
44
|
+
is what you edit; **always created from defaults at init**)
|
|
33
45
|
- `evals/models.json` — a copy of the built-in model registry
|
|
34
|
-
-
|
|
35
|
-
|
|
46
|
+
- `.gitignore` (ignores `instance/` + `eval-results/`), `README.md`
|
|
47
|
+
|
|
48
|
+
The runner **auto-detects** `./instance` as the overlay and `./evals/datasets` as
|
|
49
|
+
the dataset root, so a bare `lastlight-evals run` "just works" — no `--overlay` flag.
|
|
50
|
+
Re-running `init --clone` is idempotent: it won't re-clone an existing `instance/`.
|
|
51
|
+
|
|
52
|
+
### Plain — self-contained overlay + evals (no existing overlay repo)
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
lastlight-evals init my-evals # or: lastlight-evals init (→ ./lastlight-evals-workspace)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The workspace **is** its own overlay: it gets `workflows/`, `skills/`,
|
|
59
|
+
`agent-context/` placeholders + its own `config.yaml` alongside `evals/`. Run it
|
|
60
|
+
with `--overlay .`. In a TTY it offers `git init` + a private `gh repo create`
|
|
61
|
+
(`--no-git` to skip, `--yes` to take the non-interactive path).
|
|
36
62
|
|
|
37
63
|
## 3. Configure providers (`.env`)
|
|
38
64
|
|
|
@@ -57,16 +83,21 @@ how `--compare` is key-gated.
|
|
|
57
83
|
|
|
58
84
|
## 4. Run
|
|
59
85
|
|
|
86
|
+
From inside the workspace. For the **Separate** layout the overlay (`./instance`)
|
|
87
|
+
and datasets (`./evals/datasets`) are auto-detected — no `--overlay` needed. For
|
|
88
|
+
the **Plain** layout pass `--overlay .`.
|
|
89
|
+
|
|
60
90
|
```bash
|
|
61
91
|
cd my-evals
|
|
62
|
-
lastlight-evals run
|
|
63
|
-
lastlight-evals run triage
|
|
64
|
-
lastlight-evals run triage code-fix
|
|
92
|
+
lastlight-evals run # default model, triage tier (auto overlay+datasets)
|
|
93
|
+
lastlight-evals run triage # one tier
|
|
94
|
+
lastlight-evals run triage code-fix # multiple tiers
|
|
65
95
|
lastlight-evals run triage --model haiku # fuzzy match in models.json
|
|
66
96
|
lastlight-evals run triage --model openai/gpt-5.5,anthropic/claude-opus-4-8
|
|
67
97
|
lastlight-evals run --compare # cross-vendor set (only models whose envKey is present)
|
|
68
98
|
lastlight-evals run triage --runs 3 # repeat each case 3× (worst-case verdict, mean metrics)
|
|
69
99
|
lastlight-evals run triage --no-open # don't open the report
|
|
100
|
+
# Plain layout: add --overlay . (e.g. lastlight-evals run triage --overlay .)
|
|
70
101
|
```
|
|
71
102
|
|
|
72
103
|
Output lands in `./eval-results/<tiers>/`: `index.html` (scorecard),
|
|
@@ -80,18 +111,20 @@ workflow with held-out tests). To add cases or a custom tier, read
|
|
|
80
111
|
**`references/instance-schema.md`** — it has the `SweBenchInstance` schema, the
|
|
81
112
|
exact files to create for each tier, and worked examples.
|
|
82
113
|
|
|
83
|
-
Quick shape:
|
|
84
|
-
- **Triage case:** append a `SweBenchInstance` to `datasets/triage/instances.json`
|
|
114
|
+
Quick shape (paths are relative to the workspace's `evals/` dir):
|
|
115
|
+
- **Triage case:** append a `SweBenchInstance` to `evals/datasets/triage/instances.json`
|
|
85
116
|
with `issue`, `triage_gold`, and `expect_github`.
|
|
86
|
-
- **Code-fix case:** add the instance to `datasets/code-fix/instances.json` **and**
|
|
87
|
-
create `datasets/code-fix/repos/<instance_id>/` (fixture repo at base) +
|
|
88
|
-
`datasets/code-fix/tests/<instance_id>/` (held-out tests applied at grade time).
|
|
89
|
-
- **Custom tier:** a new `datasets/<tier>/` with `tier.json` +
|
|
117
|
+
- **Code-fix case:** add the instance to `evals/datasets/code-fix/instances.json` **and**
|
|
118
|
+
create `evals/datasets/code-fix/repos/<instance_id>/` (fixture repo at base) +
|
|
119
|
+
`evals/datasets/code-fix/tests/<instance_id>/` (held-out tests applied at grade time).
|
|
120
|
+
- **Custom tier:** a new `evals/datasets/<tier>/` with `tier.json` +
|
|
90
121
|
`instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
|
|
91
122
|
automatic — no code change.
|
|
92
123
|
|
|
93
124
|
## Done when
|
|
94
125
|
|
|
95
|
-
The workspace is scaffolded
|
|
96
|
-
|
|
97
|
-
|
|
126
|
+
The workspace is scaffolded (Separate: overlay in `instance/`, evals at root;
|
|
127
|
+
Plain: self-overlay), `.env` has a working provider key, and a bare
|
|
128
|
+
`lastlight-evals run` produces a scorecard under `eval-results/`. Report the
|
|
129
|
+
workspace path, the layout used, the provider(s) configured, and the run/compare
|
|
130
|
+
command.
|