lastlight 0.4.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +14 -0
- package/README.md +56 -6
- package/agent-context/rules.md +0 -38
- package/agent-context/soul.md +0 -11
- package/deploy/entrypoint.sh +21 -6
- package/deploy/sandbox-entrypoint.sh +1 -1
- package/dist/cli.js +24 -0
- package/dist/cli.js.map +1 -1
- package/dist/config.d.ts +13 -2
- package/dist/config.js +29 -9
- package/dist/config.js.map +1 -1
- package/dist/connectors/messaging/base.js +22 -9
- package/dist/connectors/messaging/base.js.map +1 -1
- package/dist/connectors/slack/connector.d.ts +62 -9
- package/dist/connectors/slack/connector.js +216 -63
- package/dist/connectors/slack/connector.js.map +1 -1
- package/dist/connectors/slack/connector.test.d.ts +1 -0
- package/dist/connectors/slack/connector.test.js +135 -0
- package/dist/connectors/slack/connector.test.js.map +1 -0
- package/dist/connectors/types.d.ts +7 -0
- package/dist/engine/agent-executor.d.ts +29 -144
- package/dist/engine/agent-executor.js +280 -962
- package/dist/engine/agent-executor.js.map +1 -1
- package/dist/engine/agent-executor.seam.test.d.ts +1 -0
- package/dist/engine/agent-executor.seam.test.js +57 -0
- package/dist/engine/agent-executor.seam.test.js.map +1 -0
- package/dist/engine/dispatcher.d.ts +13 -0
- package/dist/engine/dispatcher.js +26 -7
- package/dist/engine/dispatcher.js.map +1 -1
- package/dist/engine/dispatcher.test.js.map +1 -1
- package/dist/engine/executors/backends.d.ts +41 -0
- package/dist/engine/executors/backends.js +541 -0
- package/dist/engine/executors/backends.js.map +1 -0
- package/dist/engine/executors/shared.d.ts +189 -0
- package/dist/engine/executors/shared.js +612 -0
- package/dist/engine/executors/shared.js.map +1 -0
- package/dist/engine/message-batcher.d.ts +32 -0
- package/dist/engine/message-batcher.js +109 -0
- package/dist/engine/message-batcher.js.map +1 -0
- package/dist/engine/message-batcher.test.d.ts +1 -0
- package/dist/engine/message-batcher.test.js +109 -0
- package/dist/engine/message-batcher.test.js.map +1 -0
- package/dist/engine/profiles.d.ts +9 -0
- package/dist/engine/profiles.js.map +1 -1
- package/dist/evals-api.d.ts +24 -0
- package/dist/evals-api.js +22 -0
- package/dist/evals-api.js.map +1 -0
- package/dist/index.js +41 -4
- package/dist/index.js.map +1 -1
- package/dist/sandbox/command-exec.integration.test.d.ts +1 -0
- package/dist/sandbox/command-exec.integration.test.js +183 -0
- package/dist/sandbox/command-exec.integration.test.js.map +1 -0
- package/dist/sandbox/docker.d.ts +32 -0
- package/dist/sandbox/docker.js +109 -4
- package/dist/sandbox/docker.js.map +1 -1
- package/dist/sandbox/index.d.ts +1 -1
- package/dist/sandbox/index.js +1 -1
- package/dist/sandbox/index.js.map +1 -1
- package/dist/sandbox/smol.d.ts +162 -0
- package/dist/sandbox/smol.integration.test.d.ts +1 -0
- package/dist/sandbox/smol.integration.test.js +130 -0
- package/dist/sandbox/smol.integration.test.js.map +1 -0
- package/dist/sandbox/smol.js +485 -0
- package/dist/sandbox/smol.js.map +1 -0
- package/dist/skills-install.d.ts +12 -0
- package/dist/skills-install.js +179 -0
- package/dist/skills-install.js.map +1 -0
- package/dist/workflows/loader.test.js +83 -0
- package/dist/workflows/loader.test.js.map +1 -1
- package/dist/workflows/phase-executor.d.ts +28 -3
- package/dist/workflows/phase-executor.js +170 -45
- package/dist/workflows/phase-executor.js.map +1 -1
- package/dist/workflows/phase-executor.test.js +64 -1
- package/dist/workflows/phase-executor.test.js.map +1 -1
- package/dist/workflows/runner.test.js +32 -27
- package/dist/workflows/runner.test.js.map +1 -1
- package/dist/workflows/schema.d.ts +20 -0
- package/dist/workflows/schema.js +72 -2
- package/dist/workflows/schema.js.map +1 -1
- package/package.json +15 -3
- package/plugins/lastlight/.claude-plugin/plugin.json +12 -0
- package/plugins/lastlight/README.md +44 -0
- package/plugins/lastlight/skills/lastlight-client/SKILL.md +82 -0
- package/plugins/lastlight/skills/lastlight-evals/SKILL.md +97 -0
- package/plugins/lastlight/skills/lastlight-evals/references/instance-schema.md +84 -0
- package/plugins/lastlight/skills/lastlight-evals/references/models-json.md +42 -0
- package/plugins/lastlight/skills/lastlight-overlay/SKILL.md +72 -0
- package/plugins/lastlight/skills/lastlight-overlay/references/forking.md +55 -0
- package/plugins/lastlight/skills/lastlight-overlay/references/overlay-layout.md +52 -0
- package/plugins/lastlight/skills/lastlight-server/SKILL.md +124 -0
- package/plugins/lastlight/skills/lastlight-server/references/env-schema.md +103 -0
- package/plugins/lastlight/skills/lastlight-server/references/operations.md +60 -0
- package/sandbox.Dockerfile +7 -0
- package/skills/browser-qa/scripts/agent-browser.mjs +15 -0
- package/skills/demo/SKILL.md +45 -19
- package/skills/demo/scripts/compose-demo.sh +77 -39
- package/workflows/examples/bash-smoke.yaml +41 -0
- package/workflows/prompts/demo.md +27 -5
- package/workflows/qa-test.yaml +1 -1
- package/workflows/verify.yaml +1 -1
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lastlight-client
|
|
3
|
+
description: Install the `lastlight` CLI and connect it as a CLIENT to an existing Last Light server — log in, save the token, and verify the connection. Use when the user wants to "connect / point my lastlight CLI at a server", "log in to Last Light", "set up the lastlight client", or run lastlight commands against a remote instance. For standing up the server itself use lastlight-server; for editing a deployment's config use lastlight-overlay.
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
tags: [lastlight, client, cli, login, auth]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Install & configure the Last Light CLI client
|
|
9
|
+
|
|
10
|
+
The `lastlight` CLI is a thin HTTP client: it POSTs triggers and reads a running
|
|
11
|
+
instance's admin API. As a *client* it needs only the instance URL and an auth
|
|
12
|
+
token, saved to `~/.lastlight/config.json` (mode 0600).
|
|
13
|
+
|
|
14
|
+
## 1. Install the CLI
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
command -v lastlight >/dev/null && echo "installed" || npm i -g lastlight
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## 2. Get the server URL
|
|
21
|
+
|
|
22
|
+
Ask the user for the instance URL (e.g. `https://lastlight.example.com`). If they
|
|
23
|
+
don't know it, they need it from whoever runs the server.
|
|
24
|
+
|
|
25
|
+
## 3. Log in
|
|
26
|
+
|
|
27
|
+
Two paths — pick based on environment:
|
|
28
|
+
|
|
29
|
+
- **Browser handoff (default, interactive desktop):**
|
|
30
|
+
```bash
|
|
31
|
+
lastlight login https://lastlight.example.com
|
|
32
|
+
```
|
|
33
|
+
Opens the dashboard; the user authenticates with whatever method the server
|
|
34
|
+
has (password / Slack / GitHub OAuth) and the token is captured automatically.
|
|
35
|
+
|
|
36
|
+
- **Headless / no browser (SSH, CI, server box):**
|
|
37
|
+
```bash
|
|
38
|
+
lastlight login https://lastlight.example.com --password
|
|
39
|
+
```
|
|
40
|
+
Prompts for the admin password and POSTs it to `/admin/api/login`. Requires the
|
|
41
|
+
server to have `ADMIN_PASSWORD` set.
|
|
42
|
+
|
|
43
|
+
The token (≈7-day TTL) is saved to `~/.lastlight/config.json`.
|
|
44
|
+
|
|
45
|
+
## 4. Verify
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
lastlight status
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Confirm: **Server healthy**, **Token valid**. If the token shows
|
|
52
|
+
`expired/invalid`, re-run `lastlight login`.
|
|
53
|
+
|
|
54
|
+
## Overrides & alternatives
|
|
55
|
+
|
|
56
|
+
- Skip the saved file with env vars or flags (precedence: flags → env → saved
|
|
57
|
+
file → default `http://localhost:8644`):
|
|
58
|
+
```bash
|
|
59
|
+
LASTLIGHT_URL=https://ll.example.com LASTLIGHT_TOKEN=... lastlight status
|
|
60
|
+
lastlight status --url https://ll.example.com --token ...
|
|
61
|
+
```
|
|
62
|
+
- `lastlight logout` clears the saved config.
|
|
63
|
+
- One-shot wizard: `lastlight setup --client` is equivalent to `lastlight login`.
|
|
64
|
+
|
|
65
|
+
## What you can do once connected
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
lastlight chat "hello" # chat with the bot (REPL if no message)
|
|
69
|
+
lastlight owner/repo#123 # triage an issue (default, cheap)
|
|
70
|
+
lastlight build owner/repo#123 # full build cycle
|
|
71
|
+
lastlight triage|review owner/repo[#N] # repo scan or single issue/PR
|
|
72
|
+
lastlight health|security owner/repo # repo-level report
|
|
73
|
+
lastlight workflow list|log <id> # inspect runs
|
|
74
|
+
lastlight session list|log <id> -f # tail a sandbox session
|
|
75
|
+
lastlight logs search "<text>" # search errors / transcripts
|
|
76
|
+
lastlight approvals list|approve|reject
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Done when
|
|
80
|
+
|
|
81
|
+
`lastlight status` reports the server healthy and the token valid. Report the
|
|
82
|
+
connected instance URL and a couple of commands the user can run next.
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lastlight-evals
|
|
3
|
+
description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases (triage / code-fix instances). GitHub is mocked, so no real GitHub token is needed — only a model provider API key.
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
tags: [lastlight, evals, benchmark, models, swe-bench]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Set up & run Last Light Evals
|
|
9
|
+
|
|
10
|
+
`lastlight-evals` runs Last Light's **real** production workflows (issue-triage,
|
|
11
|
+
build, …) end-to-end against a mocked GitHub, grades the results deterministically
|
|
12
|
+
(no LLM-as-judge), and compares models on pass rate, cost, and latency. It's a
|
|
13
|
+
thin CLI on top of the `lastlight` core package (via the `lastlight/evals`
|
|
14
|
+
barrel), so it exercises the same workflows/skills production does. SWE-bench
|
|
15
|
+
compatible. **Node 24+.**
|
|
16
|
+
|
|
17
|
+
## 1. Check prerequisites
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
node --version # need >= 24
|
|
21
|
+
command -v lastlight-evals >/dev/null && echo "installed" || npm i -g lastlight-evals
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## 2. Scaffold a workspace
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
lastlight-evals init my-evals # or: lastlight-evals init (→ ./lastlight-evals-workspace)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
This scaffolds an **overlay + evals** workspace:
|
|
31
|
+
- `workflows/`, `skills/`, `agent-context/` — empty dirs for override assets
|
|
32
|
+
- `evals/datasets/` — seeded from the built-in `triage` + `code-fix` samples
|
|
33
|
+
- `evals/models.json` — a copy of the built-in model registry
|
|
34
|
+
- `config.yaml`, `.gitignore`, `README.md`
|
|
35
|
+
- optionally `git init` + a private `gh repo create`
|
|
36
|
+
|
|
37
|
+
## 3. Configure providers (`.env`)
|
|
38
|
+
|
|
39
|
+
Create `.env` in the workspace with **at least one** provider key. GitHub is
|
|
40
|
+
mocked end-to-end, so **no real GitHub token is needed** (the harness sets a
|
|
41
|
+
dummy one internally).
|
|
42
|
+
|
|
43
|
+
```dotenv
|
|
44
|
+
# any one (or more) of:
|
|
45
|
+
OPENAI_API_KEY=sk-...
|
|
46
|
+
ANTHROPIC_API_KEY=sk-ant-...
|
|
47
|
+
FIREWORKS_API_KEY=fw-...
|
|
48
|
+
OPENROUTER_API_KEY=sk-or-...
|
|
49
|
+
DEEPSEEK_API_KEY=...
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Useful optional vars: `EVAL_MODELS` (comma-list override), `LASTLIGHT_EVALS_OUT`
|
|
53
|
+
(scorecard dir, default `./eval-results/`), `LASTLIGHT_CORE_DIR` (point at a local
|
|
54
|
+
lastlight checkout to eval un-published asset edits), `CI=1` (don't open a
|
|
55
|
+
browser). See **`references/models-json.md`** for the model registry format and
|
|
56
|
+
how `--compare` is key-gated.
|
|
57
|
+
|
|
58
|
+
## 4. Run
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
cd my-evals
|
|
62
|
+
lastlight-evals run --overlay . # default model, triage tier
|
|
63
|
+
lastlight-evals run triage --overlay . # one tier
|
|
64
|
+
lastlight-evals run triage code-fix --overlay . # multiple tiers
|
|
65
|
+
lastlight-evals run triage --model haiku # fuzzy match in models.json
|
|
66
|
+
lastlight-evals run triage --model openai/gpt-5.5,anthropic/claude-opus-4-8
|
|
67
|
+
lastlight-evals run --compare # cross-vendor set (only models whose envKey is present)
|
|
68
|
+
lastlight-evals run triage --runs 3 # repeat each case 3× (worst-case verdict, mean metrics)
|
|
69
|
+
lastlight-evals run triage --no-open # don't open the report
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Output lands in `./eval-results/<tiers>/`: `index.html` (scorecard),
|
|
73
|
+
`scorecard.json`, `predictions.jsonl` (SWE-bench format). Re-render a report
|
|
74
|
+
without re-running: `lastlight-evals report ./eval-results/triage`.
|
|
75
|
+
|
|
76
|
+
## 5. Author eval cases (optional)
|
|
77
|
+
|
|
78
|
+
Two tiers ship: **triage** (cheap, issue-triage) and **code-fix** (heavy, build
|
|
79
|
+
workflow with held-out tests). To add cases or a custom tier, read
|
|
80
|
+
**`references/instance-schema.md`** — it has the `SweBenchInstance` schema, the
|
|
81
|
+
exact files to create for each tier, and worked examples.
|
|
82
|
+
|
|
83
|
+
Quick shape:
|
|
84
|
+
- **Triage case:** append a `SweBenchInstance` to `datasets/triage/instances.json`
|
|
85
|
+
with `issue`, `triage_gold`, and `expect_github`.
|
|
86
|
+
- **Code-fix case:** add the instance to `datasets/code-fix/instances.json` **and**
|
|
87
|
+
create `datasets/code-fix/repos/<instance_id>/` (fixture repo at base) +
|
|
88
|
+
`datasets/code-fix/tests/<instance_id>/` (held-out tests applied at grade time).
|
|
89
|
+
- **Custom tier:** a new `datasets/<tier>/` with `tier.json` +
|
|
90
|
+
`instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
|
|
91
|
+
automatic — no code change.
|
|
92
|
+
|
|
93
|
+
## Done when
|
|
94
|
+
|
|
95
|
+
The workspace is scaffolded, `.env` has a working provider key, and a run
|
|
96
|
+
produces a scorecard under `eval-results/`. Report the workspace path, the
|
|
97
|
+
provider(s) configured, and the command to run/compare.
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Eval instance schema & authoring cases
|
|
2
|
+
|
|
3
|
+
An eval case is a **`SweBenchInstance`** — SWE-bench-compatible core fields plus
|
|
4
|
+
Last Light extensions (the GitHub fixtures + behavioral expectations that let the
|
|
5
|
+
harness drive and grade the real workflow against a mocked GitHub).
|
|
6
|
+
|
|
7
|
+
Datasets are discovered from (overlay > user > built-in): `<overlay>/evals/datasets/`,
|
|
8
|
+
`--datasets <dir>` / `LASTLIGHT_EVALS_DATASETS`, and the package's built-in
|
|
9
|
+
`datasets/`. A tier is a directory with a `tier.json` and an `instances.json`.
|
|
10
|
+
|
|
11
|
+
## `tier.json`
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{ "name": "triage", "defaultWorkflow": "issue-triage", "description": "..." }
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## `SweBenchInstance` fields
|
|
18
|
+
|
|
19
|
+
```jsonc
|
|
20
|
+
{
|
|
21
|
+
// ── SWE-bench core ──
|
|
22
|
+
"instance_id": "triage__my-case", // unique id
|
|
23
|
+
"repo": "owner/repo", // logical; fixture origin is a local bare repo
|
|
24
|
+
"base_commit": "0000000...", // code-fix only
|
|
25
|
+
"problem_statement": "short issue text",
|
|
26
|
+
"patch": "...", // gold patch — reference only, NOT graded
|
|
27
|
+
"test_patch": "...", // held-out tests (code-fix)
|
|
28
|
+
"FAIL_TO_PASS": ["test id 1"], // must go red→green (code-fix)
|
|
29
|
+
"PASS_TO_PASS": ["test id 2"], // must stay green (code-fix)
|
|
30
|
+
|
|
31
|
+
// ── Last Light extensions ──
|
|
32
|
+
"workflow": "issue-triage", // optional; defaults to the tier's defaultWorkflow
|
|
33
|
+
"issue": { // seed state for the fake GitHub
|
|
34
|
+
"number": 110, "title": "...", "body": "...",
|
|
35
|
+
"labels": [], "user": "alice",
|
|
36
|
+
"comments": [{ "user": "bob", "body": "..." }],
|
|
37
|
+
"state": "open"
|
|
38
|
+
},
|
|
39
|
+
"triage_gold": { "category": "bug", "state": "ready-for-agent" }, // triage grading
|
|
40
|
+
"expect_github": { // behavioral assertions on recorded GitHub calls
|
|
41
|
+
"labels_added": ["bug"],
|
|
42
|
+
"labels_absent": ["wontfix"],
|
|
43
|
+
"issue_closed": false,
|
|
44
|
+
"comment_matches": "(?i)thanks",
|
|
45
|
+
"pr_opened": { "base": "main", "head_is_branch": true, "title_matches": "(?i)fix" }
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Every `expect_github` field is optional — only the present ones are checked.
|
|
51
|
+
|
|
52
|
+
## Add a triage case
|
|
53
|
+
|
|
54
|
+
Append a `SweBenchInstance` to `datasets/triage/instances.json` with `issue`,
|
|
55
|
+
`triage_gold`, and the `expect_github` assertions (e.g. `labels_added`). That's
|
|
56
|
+
it — triage is graded on the triage decision + GitHub mutations.
|
|
57
|
+
|
|
58
|
+
## Add a code-fix case (three things, keyed by `instance_id`)
|
|
59
|
+
|
|
60
|
+
1. **`datasets/code-fix/instances.json`** — append the instance with
|
|
61
|
+
`base_commit`, `FAIL_TO_PASS`, `PASS_TO_PASS`, `issue`, and `expect_github`
|
|
62
|
+
(e.g. `pr_opened`).
|
|
63
|
+
2. **`datasets/code-fix/repos/<instance_id>/`** — the fixture repo at the base
|
|
64
|
+
commit (the buggy code *before* the fix; no held-out tests here).
|
|
65
|
+
3. **`datasets/code-fix/tests/<instance_id>/`** — the held-out test files, copied
|
|
66
|
+
into the repo at grade time and run to compute `FAIL_TO_PASS` / `PASS_TO_PASS`.
|
|
67
|
+
|
|
68
|
+
## Add a custom tier
|
|
69
|
+
|
|
70
|
+
Create `datasets/<tier-name>/` with `tier.json` (`name`, `defaultWorkflow`,
|
|
71
|
+
`description`) + `instances.json`. For code-fix-style tiers also add `repos/<id>/`
|
|
72
|
+
and `tests/<id>/`. Discovery auto-finds it — no code change. Run it with
|
|
73
|
+
`lastlight-evals run <tier-name> --overlay .`.
|
|
74
|
+
|
|
75
|
+
## Grading (how a case passes)
|
|
76
|
+
|
|
77
|
+
- **Behavioral:** did the workflow take the expected GitHub actions
|
|
78
|
+
(`expect_github`)?
|
|
79
|
+
- **Triage:** did the decision match `triage_gold`?
|
|
80
|
+
- **Execution (code-fix):** all `FAIL_TO_PASS` green AND all `PASS_TO_PASS` still
|
|
81
|
+
green after applying held-out tests.
|
|
82
|
+
- With `--runs N` (N>1) the binary verdict is **worst-case** (passes only if every
|
|
83
|
+
trial passed); the scorecard also shows per-verdict pass counts to expose
|
|
84
|
+
variance.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Model registry — `models.json`
|
|
2
|
+
|
|
3
|
+
The eval harness picks models from a `models.json`. Resolution: the scaffolded
|
|
4
|
+
`evals/models.json` is picked up automatically (or pass `--models-file <path>`).
|
|
5
|
+
|
|
6
|
+
```json
|
|
7
|
+
{
|
|
8
|
+
"default": "openai/gpt-5.4-mini",
|
|
9
|
+
"compare": [
|
|
10
|
+
{ "id": "openai/gpt-5.5", "label": "GPT-5.5", "provider": "OpenAI", "envKey": "OPENAI_API_KEY" },
|
|
11
|
+
{ "id": "anthropic/claude-opus-4-8", "label": "Claude Opus 4.8","provider": "Anthropic", "envKey": "ANTHROPIC_API_KEY" },
|
|
12
|
+
{ "id": "fireworks/accounts/fireworks/models/glm-5p2", "label": "GLM-5.2", "provider": "Fireworks (open)", "envKey": "FIREWORKS_API_KEY" }
|
|
13
|
+
]
|
|
14
|
+
}
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Fields:
|
|
18
|
+
- **`default`** — the single model `run` uses when not `--compare`.
|
|
19
|
+
- **`compare`** — the cross-vendor set `--compare` runs. **Each entry runs only if
|
|
20
|
+
its `envKey` is present** in the environment, so you can list many and only the
|
|
21
|
+
ones you have keys for execute. Add/remove freely.
|
|
22
|
+
- Per entry: `id` is the agentic-pi/pi-ai `provider/model` spec; `label` is the
|
|
23
|
+
scorecard display name; `provider` is a grouping label; `envKey` is the env var
|
|
24
|
+
that gates it.
|
|
25
|
+
|
|
26
|
+
## Selecting models on the CLI
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
lastlight-evals run triage --model haiku # fuzzy substring match against ids/labels
|
|
30
|
+
lastlight-evals run triage --model openai/gpt-5.5 # exact provider/model id
|
|
31
|
+
lastlight-evals run triage --model glm,deepseek # comma-list
|
|
32
|
+
lastlight-evals run --compare # the whole compare set (key-gated)
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
You can also override the set with `EVAL_MODELS=<comma-list>` in the environment.
|
|
36
|
+
|
|
37
|
+
## Provider keys
|
|
38
|
+
|
|
39
|
+
Set whichever provider keys you have in the workspace `.env`:
|
|
40
|
+
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `FIREWORKS_API_KEY` (GLM / DeepSeek /
|
|
41
|
+
GPT-OSS), `OPENROUTER_API_KEY`, `DEEPSEEK_API_KEY`. No GitHub credentials needed —
|
|
42
|
+
GitHub is mocked.
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lastlight-overlay
|
|
3
|
+
description: Create or customize a Last Light deployment OVERLAY (the private instance/ repo) — scaffold it, then fork built-in workflows, prompts, skills, or the agent persona (agent-context) so a deployment can override them. Use when the user wants to "create a Last Light overlay / instance repo", "customize / fork a workflow", "override a prompt or skill", "change the agent's persona/soul/rules", or tune a deployment's config without forking the whole codebase. For first-time server install use lastlight-server.
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
tags: [lastlight, overlay, instance, fork, workflows, customization]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Create & customize a Last Light overlay
|
|
9
|
+
|
|
10
|
+
Last Light keeps per-deployment customization in a private **overlay** at
|
|
11
|
+
`instance/`, layered over the packaged defaults at startup. The overlay can hold
|
|
12
|
+
`config.yaml`, `secrets/`, and overrides of any asset — `workflows/`,
|
|
13
|
+
`workflows/prompts/`, `skills/`, `agent-context/`. An overlay copy **shadows the
|
|
14
|
+
built-in by logical name**, so you only fork what you want to change.
|
|
15
|
+
|
|
16
|
+
These are host-local file operations on the working directory (resolved from
|
|
17
|
+
`--home` → `LASTLIGHT_HOME` → saved `serverHome` → `~/lastlight`).
|
|
18
|
+
|
|
19
|
+
## 1. Scaffold or adopt the overlay
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
lastlight server setup
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
This scaffolds/adopts the working directory: clones the core repo if needed, then
|
|
26
|
+
**creates a fresh overlay** (scaffold + optional private `gh repo create`) or
|
|
27
|
+
**clones an existing** overlay repo — your choice at the prompt. It also symlinks
|
|
28
|
+
`instance/docker-compose.override.yml` and saves the working dir for later
|
|
29
|
+
`lastlight server …` commands.
|
|
30
|
+
|
|
31
|
+
Read **`references/overlay-layout.md`** for the full directory shape and how the
|
|
32
|
+
layering/merge rules work (arrays replace, maps deep-merge, overlay wins by
|
|
33
|
+
name).
|
|
34
|
+
|
|
35
|
+
## 2. Tune non-secret config
|
|
36
|
+
|
|
37
|
+
Edit `instance/config.yaml` to set `managedRepos`, and optionally override
|
|
38
|
+
`models`, `variants`, `routes`, `approvals`, `disabled.*`. Secrets stay in
|
|
39
|
+
`instance/secrets/.env` (never in config.yaml).
|
|
40
|
+
|
|
41
|
+
## 3. Fork built-in assets to customize them
|
|
42
|
+
|
|
43
|
+
Use `lastlight fork` — it copies a built-in into `instance/` so your edited copy
|
|
44
|
+
shadows the default. Read **`references/forking.md`** for exactly what each fork
|
|
45
|
+
copies and the editing workflow.
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
lastlight fork # list forkable workflows + agent-context (marks what's already forked)
|
|
49
|
+
lastlight fork <workflow> # copy a workflow YAML + every prompt & skill its phases reference
|
|
50
|
+
lastlight fork agent-context # copy soul.md / rules.md / security.md (the persona)
|
|
51
|
+
lastlight fork agent-context soul.md # just one persona file
|
|
52
|
+
# add --force to overwrite an existing overlay copy; --home <dir> to target a specific working dir
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Then edit the copied files under `instance/…`.
|
|
56
|
+
|
|
57
|
+
## 4. Apply
|
|
58
|
+
|
|
59
|
+
- **Config-only or overlay-asset edits** (committed to the overlay): take effect
|
|
60
|
+
with `lastlight server restart agent` — no image rebuild.
|
|
61
|
+
- If the overlay is a separate git repo, commit + push there first, then restart.
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
lastlight server status # shows forked/overridden assets + version drift
|
|
65
|
+
lastlight server restart agent
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## Done when
|
|
69
|
+
|
|
70
|
+
The overlay exists with the intended `config.yaml` and any forked assets under
|
|
71
|
+
`instance/`, `lastlight server status` lists the overrides, and the agent has
|
|
72
|
+
been restarted to apply them. Report which assets were forked and where they live.
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Forking built-in assets with `lastlight fork`
|
|
2
|
+
|
|
3
|
+
`lastlight fork` copies a built-in asset into the overlay (`instance/`) so your
|
|
4
|
+
edited copy shadows the default by logical name. It's host-local — it operates on
|
|
5
|
+
the working directory's files, not over HTTP.
|
|
6
|
+
|
|
7
|
+
Path resolution: an explicit `--home <dir>` wins; otherwise if you're standing in
|
|
8
|
+
an overlay it writes there (reading built-ins from the parent checkout); if
|
|
9
|
+
you're in a core checkout it writes to `<checkout>/instance`; else it falls back
|
|
10
|
+
to `LASTLIGHT_HOME` / the saved server home / `~/lastlight`.
|
|
11
|
+
|
|
12
|
+
## Commands
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
lastlight fork # list forkable workflows + agent-context files; marks what's already forked
|
|
16
|
+
lastlight fork <workflow> # e.g. `lastlight fork build`
|
|
17
|
+
lastlight fork agent-context # all of soul.md / rules.md / security.md
|
|
18
|
+
lastlight fork agent-context <file> # a single persona file, e.g. soul.md
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Flags: `--force` overwrites an existing overlay copy (otherwise existing files
|
|
22
|
+
are skipped); `--home <dir>` targets a specific working directory.
|
|
23
|
+
|
|
24
|
+
## What each fork copies
|
|
25
|
+
|
|
26
|
+
- **`lastlight fork <workflow>`** copies the workflow YAML
|
|
27
|
+
(`workflows/<name>.yaml`) **plus every prompt and skill its phases reference**
|
|
28
|
+
(`workflows/prompts/*.md`, `skills/<name>/`). So the forked workflow is
|
|
29
|
+
self-contained and editable in the overlay. Existing destinations are skipped
|
|
30
|
+
unless `--force`.
|
|
31
|
+
- **`lastlight fork agent-context [file]`** copies the agent "personality" files
|
|
32
|
+
(`agent-context/*.md` — typically `soul.md`, `rules.md`, `security.md`) into
|
|
33
|
+
`instance/agent-context/`. With no file argument it forks all of them; pass a
|
|
34
|
+
filename for just one.
|
|
35
|
+
|
|
36
|
+
`agent-context` is forked only via the literal `agent-context` target — a bare
|
|
37
|
+
name is always treated as a workflow.
|
|
38
|
+
|
|
39
|
+
## Editing workflow
|
|
40
|
+
|
|
41
|
+
1. Fork the asset.
|
|
42
|
+
2. Edit the copy under `instance/…`.
|
|
43
|
+
3. If the overlay is its own git repo, commit + push there.
|
|
44
|
+
4. Apply: `lastlight server restart agent` (config/asset overlay change — no
|
|
45
|
+
image rebuild).
|
|
46
|
+
5. Confirm with `lastlight server status` (lists forked/overridden assets) and
|
|
47
|
+
the dashboard `/config` view.
|
|
48
|
+
|
|
49
|
+
## Common targets
|
|
50
|
+
|
|
51
|
+
- Workflows: `build`, `pr-fix`, `pr-review`, `issue-triage`, `issue-comment`,
|
|
52
|
+
`repo-health`, and the `cron-*` workflows. Run `lastlight fork` to see the live
|
|
53
|
+
list.
|
|
54
|
+
- Persona: `soul.md` (voice/identity), `rules.md` (hard rules), `security.md`
|
|
55
|
+
(security posture).
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Overlay layout & layering rules
|
|
2
|
+
|
|
3
|
+
The overlay is a private directory (often its own git repo) mounted read-only at
|
|
4
|
+
`/app/instance` and pointed to by `LASTLIGHT_OVERLAY_DIR`. It is layered over the
|
|
5
|
+
packaged defaults at startup.
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
instance/
|
|
9
|
+
config.yaml # merged over config/default.yaml
|
|
10
|
+
.gitignore # ignores secrets/ + *.pem
|
|
11
|
+
docker-compose.override.yml # optional (e.g. Caddy disabled); symlinked into the project root
|
|
12
|
+
secrets/ # host-only, gitignored
|
|
13
|
+
.env # all env vars (mode 0600)
|
|
14
|
+
app.pem # GitHub App private key (mode 0600)
|
|
15
|
+
agent-context/ # optional persona overrides
|
|
16
|
+
soul.md
|
|
17
|
+
rules.md
|
|
18
|
+
security.md
|
|
19
|
+
workflows/ # optional workflow overrides
|
|
20
|
+
<name>.yaml
|
|
21
|
+
prompts/
|
|
22
|
+
<name>.md
|
|
23
|
+
skills/ # optional skill overrides
|
|
24
|
+
<name>/
|
|
25
|
+
SKILL.md
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Layering / merge rules
|
|
29
|
+
|
|
30
|
+
- **config.yaml** is deep-merged over the packaged `config/default.yaml`:
|
|
31
|
+
- **maps** deep-merge,
|
|
32
|
+
- **arrays** replace wholesale (e.g. `managedRepos`, `disabled.*`),
|
|
33
|
+
- environment variables override both.
|
|
34
|
+
- Secrets are env-only — never put them in config.yaml.
|
|
35
|
+
- **Assets** (`workflows/`, `workflows/prompts/`, `skills/`, `agent-context/`)
|
|
36
|
+
resolve **layer-aware by logical name**: an overlay file with the same logical
|
|
37
|
+
name as a built-in *wins*; built-ins are the fallback. So you fork only what
|
|
38
|
+
you want to change.
|
|
39
|
+
|
|
40
|
+
## What `config.yaml` can override
|
|
41
|
+
|
|
42
|
+
`managedRepos`, `models` / per-task model overrides, `variants` (reasoning
|
|
43
|
+
effort), `routes`, `approvals`, and `disabled.*`. See the repo's
|
|
44
|
+
`config/default.yaml` for the authoritative shape and keys.
|
|
45
|
+
|
|
46
|
+
## Applying changes
|
|
47
|
+
|
|
48
|
+
- config.yaml change, or an added/changed `.env` value → `lastlight server
|
|
49
|
+
restart agent`.
|
|
50
|
+
- **Removing** an `.env` value → `lastlight server start agent` (recreate).
|
|
51
|
+
- The dashboard's `/config` view shows Default / Overlay / Merged (non-secret) so
|
|
52
|
+
you can confirm what took effect.
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lastlight-server
|
|
3
|
+
description: Install and configure a Last Light SERVER — the GitHub maintenance agent plus its docker-compose stack — on a host. Use when the user wants to "set up / install / deploy / stand up a Last Light server or instance", configure its GitHub App, models, managed repos, or domain, or get the agent running for the first time. Drives the `lastlight` CLI; for connecting an existing CLI to a server use lastlight-client instead, and for editing a running deployment's config use lastlight-overlay.
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
tags: [lastlight, server, deploy, docker, setup]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Install & configure a Last Light server
|
|
9
|
+
|
|
10
|
+
A Last Light **server** runs the agent and a docker-compose stack on a host. It
|
|
11
|
+
needs: Docker, a GitHub App (so it can act on repos), a model provider API key,
|
|
12
|
+
and a list of repos it's allowed to manage. Configuration lives in a private
|
|
13
|
+
**overlay** at `instance/` (a `config.yaml` plus `secrets/.env` + `secrets/app.pem`).
|
|
14
|
+
|
|
15
|
+
Your job: gather the inputs, get the working directory in place, write the
|
|
16
|
+
config, then build and launch. Prefer the deterministic file-writing path below
|
|
17
|
+
(fully automatable) over the interactive wizard.
|
|
18
|
+
|
|
19
|
+
## 1. Check prerequisites
|
|
20
|
+
|
|
21
|
+
Run these and report anything missing before continuing:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
docker info >/dev/null 2>&1 && echo "docker: ok" || echo "docker: NOT running"
|
|
25
|
+
git --version; node --version
|
|
26
|
+
command -v lastlight >/dev/null && lastlight --help >/dev/null 2>&1 && echo "lastlight: installed" || echo "lastlight: missing"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
- Docker must be running.
|
|
30
|
+
- If `lastlight` is missing: `npm i -g lastlight`.
|
|
31
|
+
|
|
32
|
+
## 2. Gather inputs from the user
|
|
33
|
+
|
|
34
|
+
Ask for each of these (don't guess). Group the questions; explain what each is for.
|
|
35
|
+
|
|
36
|
+
**Required**
|
|
37
|
+
- **GitHub App** — App ID, Installation ID, and the path to the App's private
|
|
38
|
+
key `.pem` file. (If they don't have a GitHub App yet, point them at GitHub →
|
|
39
|
+
Settings → Developer settings → GitHub Apps; it needs webhook + repo
|
|
40
|
+
contents/issues/pull-requests permissions. They install it on their repos to
|
|
41
|
+
get the Installation ID.)
|
|
42
|
+
- **Domain** — the public hostname for webhooks/dashboard, e.g.
|
|
43
|
+
`lastlight.example.com`. Ask whether to use the bundled **Caddy** for
|
|
44
|
+
automatic TLS (default yes). The GitHub App webhook URL will be
|
|
45
|
+
`https://<domain>/webhook`.
|
|
46
|
+
- **Managed repos** — one or more `owner/repo` the bot is allowed to act on. The
|
|
47
|
+
bot ignores any repo not listed.
|
|
48
|
+
- **Model** — a `provider/model` string (default `anthropic/claude-sonnet-4-6`),
|
|
49
|
+
plus the **matching** provider API key. Key→provider mapping:
|
|
50
|
+
`sk-ant-…` → `ANTHROPIC_API_KEY`, `sk-or-…` → `OPENROUTER_API_KEY`, other
|
|
51
|
+
`sk-…` → `OPENAI_API_KEY`.
|
|
52
|
+
|
|
53
|
+
**Optional**
|
|
54
|
+
- Admin dashboard password (≥8 chars) to protect `/admin`.
|
|
55
|
+
- Slack: bot token (`xoxb-…`), app token (`xapp-…`), delivery channel id,
|
|
56
|
+
allowed user ids.
|
|
57
|
+
|
|
58
|
+
## 3. Working directory + overlay
|
|
59
|
+
|
|
60
|
+
The server runs out of a working directory: a checkout of the lastlight repo
|
|
61
|
+
plus an `instance/` overlay. Scaffold it (this clones the core repo if needed
|
|
62
|
+
and creates/clones the overlay):
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
lastlight server setup # interactive: confirms working dir + overlay
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
If you need to do this non-interactively / the user already has a checkout, work
|
|
69
|
+
in that directory directly (it has `docker-compose.yml`, `workflows/`, `skills/`,
|
|
70
|
+
`config/`). The overlay you write to is `<workdir>/instance/`.
|
|
71
|
+
|
|
72
|
+
## 4. Write the configuration
|
|
73
|
+
|
|
74
|
+
Create the overlay files directly (this is the automatable path — it avoids the
|
|
75
|
+
TTY wizard). From the working directory:
|
|
76
|
+
|
|
77
|
+
1. `mkdir -p instance/secrets && chmod 700 instance/secrets`
|
|
78
|
+
2. Copy the user's PEM to `instance/secrets/app.pem` and `chmod 600` it.
|
|
79
|
+
3. Generate two random secrets: `openssl rand -hex 32` for `WEBHOOK_SECRET` and
|
|
80
|
+
again for `ADMIN_SECRET`.
|
|
81
|
+
4. Write `instance/secrets/.env` (`chmod 600`) and `instance/config.yaml`
|
|
82
|
+
following **`references/env-schema.md`** — it has the exact keys, value
|
|
83
|
+
formats, and ready-to-fill templates for both files.
|
|
84
|
+
5. If the user declined Caddy, write `instance/docker-compose.override.yml` to
|
|
85
|
+
disable it (template in `references/env-schema.md`).
|
|
86
|
+
|
|
87
|
+
Read `references/env-schema.md` now before writing the files — it has the precise
|
|
88
|
+
key list and value formats.
|
|
89
|
+
|
|
90
|
+
> Alternative (let the user drive): if they'd rather answer prompts themselves,
|
|
91
|
+
> have them run `lastlight setup --server` in the working directory — the same
|
|
92
|
+
> wizard, interactive. It is a TTY wizard, so you can't fill it in from a script.
|
|
93
|
+
|
|
94
|
+
## 5. Build and launch
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
lastlight server update # build images, bring stack up, restart sidecars, health-check
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Then verify:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
curl -fsS http://127.0.0.1:8644/health && echo " ← healthy"
|
|
104
|
+
lastlight server status # compose state + version drift
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
## 6. Hand-off checklist
|
|
108
|
+
|
|
109
|
+
Tell the user to:
|
|
110
|
+
- Paste the generated `WEBHOOK_SECRET` and the webhook URL
|
|
111
|
+
(`https://<domain>/webhook`) into their **GitHub App** settings, and confirm a
|
|
112
|
+
test delivery succeeds.
|
|
113
|
+
- Visit `https://<domain>/admin` (password is `ADMIN_PASSWORD` if they set one).
|
|
114
|
+
- Make later config edits in the overlay, then `lastlight server restart agent`
|
|
115
|
+
(no rebuild). See the **lastlight-overlay** skill for forking workflows/assets.
|
|
116
|
+
|
|
117
|
+
For day-2 operations (start/stop/restart/update, logs, redeploy after a code
|
|
118
|
+
change), read **`references/operations.md`**.
|
|
119
|
+
|
|
120
|
+
## Done when
|
|
121
|
+
|
|
122
|
+
The stack is up, `GET /health` returns ok, `lastlight server status` shows the
|
|
123
|
+
agent running, and the user has the webhook secret + URL to finish GitHub App
|
|
124
|
+
setup. Report the webhook URL, dashboard URL, and which config you wrote.
|