@zenera/cli 1.1.22 → 1.1.24
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +72 -2
- package/dist/archive.d.ts +139 -0
- package/dist/archive.js +376 -0
- package/dist/ask.d.ts +51 -0
- package/dist/ask.js +143 -0
- package/dist/batch.d.ts +77 -0
- package/dist/batch.js +338 -0
- package/dist/command.d.ts +6 -0
- package/dist/commands/export.d.ts +3 -0
- package/dist/commands/export.js +255 -0
- package/dist/commands/import.d.ts +3 -0
- package/dist/commands/import.js +173 -0
- package/dist/commands/index.d.ts +5 -1
- package/dist/commands/index.js +13 -1
- package/dist/commands/inspect.d.ts +11 -0
- package/dist/commands/inspect.js +388 -89
- package/dist/commands/list.js +128 -19
- package/dist/commands/memory.js +453 -16
- package/dist/commands/models.js +14 -2
- package/dist/commands/run.js +139 -34
- package/dist/commands/sandbox.js +72 -1
- package/dist/engine.d.ts +19 -0
- package/dist/engine.js +60 -8
- package/dist/liveness.js +6 -3
- package/dist/main.js +5 -2
- package/dist/podman.d.ts +16 -0
- package/dist/podman.js +115 -2
- package/dist/progress.d.ts +33 -0
- package/dist/progress.js +402 -0
- package/dist/request.d.ts +45 -0
- package/dist/request.js +300 -0
- package/dist/sandbox.js +10 -0
- package/dist/scaffold.d.ts +8 -0
- package/dist/scaffold.js +4 -1
- package/dist/session.d.ts +26 -1
- package/dist/session.js +42 -3
- package/dist/term.d.ts +3 -0
- package/dist/term.js +17 -1
- package/dist/trace.d.ts +84 -0
- package/dist/trace.js +686 -0
- package/dist/validate.js +93 -4
- package/package.json +12 -6
- package/templates/editor/.github/copilot-instructions.md +11 -6
- package/templates/editor/.github/prompts/finetune.prompt.md +251 -0
- package/templates/editor/.github/prompts/spec-sync-project.prompt.md +25 -2
- package/templates/editor/.github/skills/zen-cli/SKILL.md +14 -3
- package/templates/editor/.github/skills/zen-cli/references/check.md +13 -5
- package/templates/editor/.github/skills/zen-cli/references/export.md +107 -0
- package/templates/editor/.github/skills/zen-cli/references/inspect.md +262 -21
- package/templates/editor/.github/skills/zen-cli/references/memory.md +123 -15
- package/templates/editor/.github/skills/zen-cli/references/models.md +1 -1
- package/templates/editor/.github/skills/zen-cli/references/projects.md +12 -5
- package/templates/editor/.github/skills/zen-cli/references/rag.md +6 -0
- package/templates/editor/.github/skills/zen-cli/references/run.md +201 -4
- package/templates/editor/.github/skills/zen-finetune/SKILL.md +1000 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/collect.sh +432 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/sample.sh +199 -0
- package/templates/editor/.github/skills/zen-inspect/SKILL.md +396 -0
- package/templates/editor/.github/skills/zen-memory/SKILL.md +74 -28
- package/templates/editor/.github/skills/zen-memory-warmup/SKILL.md +370 -408
- package/templates/editor/.github/skills/zen-sandbox/SKILL.md +19 -0
- package/templates/editor/.github/skills/zen-sandbox-capacity/SKILL.md +334 -0
- package/templates/editor/.github/skills/zen-sandbox-capacity/references/loadtest.md +181 -0
- package/templates/editor/.github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh +457 -0
- package/templates/editor/.github/skills/zen-spec-sync/SKILL.md +6 -5
- package/templates/editor/.vscode/agents.schema.json +1 -1
- package/templates/editor/.vscode/settings.json +1 -0
- package/templates/project/SPECIFICATION.md.tmpl +2 -0
- package/templates/project/agents/files-instructions.md +13 -0
- package/templates/project/agents/fork-instructions.md +27 -0
- package/templates/project/agents/memory-instructions.md +80 -7
- package/templates/project/gitignore +3 -0
- package/templates/project/scripts/_setup.sh +49 -7
|
@@ -0,0 +1,457 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
#
|
|
3
|
+
# Whether this machine can run the batch you are about to start.
|
|
4
|
+
#
|
|
5
|
+
# preflight_sandbox.sh --concurrency 8 the gate: exit 0 fits, 1 does not
|
|
6
|
+
# preflight_sandbox.sh --skip-probe the free half, in about a second
|
|
7
|
+
# preflight_sandbox.sh --peak-rss 912 use a figure you measured yourself
|
|
8
|
+
#
|
|
9
|
+
# Containers do not run on the host. On macOS they run inside one podman virtual
|
|
10
|
+
# machine whose memory is fixed, is not the host's, and is 2 GiB by default — so
|
|
11
|
+
# eight concurrent containers over a multi-gigabyte index thrash and are killed
|
|
12
|
+
# while the host sits nearly empty. The symptom is exit 124, which everybody
|
|
13
|
+
# already has a theory about, so the VM is the last thing anyone looks at.
|
|
14
|
+
#
|
|
15
|
+
# Two bounds are reported, because they fail for different reasons:
|
|
16
|
+
#
|
|
17
|
+
# admission concurrency x sandbox.memory <= vm_memory
|
|
18
|
+
# what podman will permit to be allocated. Static and free.
|
|
19
|
+
# working set largest_index + concurrency x peak_rss + 1 GiB <= vm_memory
|
|
20
|
+
# what the work actually wants. The index is counted once: it
|
|
21
|
+
# is page cache in the VM and every container shares it.
|
|
22
|
+
#
|
|
23
|
+
# It measures; it never changes anything. `podman machine set` needs the machine
|
|
24
|
+
# stopped, which kills every running container, so that command is printed and
|
|
25
|
+
# handed over rather than run.
|
|
26
|
+
#
|
|
27
|
+
# Exit 0 sufficient, 1 insufficient, 3 not applicable. See
|
|
28
|
+
# .github/skills/zen-sandbox-capacity/SKILL.md.
|
|
29
|
+
|
|
30
|
+
set -eu
|
|
31
|
+
|
|
32
|
+
# Taken before the cd, or a relative $0 stops naming this file.
|
|
33
|
+
SELF="$(cd "$(dirname "$0")" && pwd)/$(basename "$0")"
|
|
34
|
+
cd "$(dirname "$0")/../../../.."
|
|
35
|
+
|
|
36
|
+
CONCURRENCY=1
|
|
37
|
+
ASSETS=assets
|
|
38
|
+
SKIP_PROBE=0
|
|
39
|
+
PEAK_RSS=0
|
|
40
|
+
AS_JSON=0
|
|
41
|
+
PROBE_TIMEOUT=25
|
|
42
|
+
|
|
43
|
+
# A node-based agent container costs this much before it has done anything, so
|
|
44
|
+
# a generic probe that measures less than it has measured the probe, not the
|
|
45
|
+
# agent. Override with a real figure using --peak-rss.
|
|
46
|
+
# A gate that guesses should guess high: guessing low returns a cheerful "ok"
|
|
47
|
+
# and the batch dies anyway. 1024 is a round number just above the only figure
|
|
48
|
+
# anyone has actually measured here (912 MiB), so an unprobed answer errs
|
|
49
|
+
# toward refusing. Pass --peak-rss with a real number and this is unused.
|
|
50
|
+
PEAK_FLOOR_MIB=1024
|
|
51
|
+
HEADROOM_MIB=1024
|
|
52
|
+
|
|
53
|
+
# What zen scaffolds when `sandbox:` names neither, and what `strict` uses.
|
|
54
|
+
DEFAULT_CONTAINER_MIB=4096
|
|
55
|
+
STRICT_CONTAINER_MIB=2048
|
|
56
|
+
|
|
57
|
+
usage() {
|
|
58
|
+
sed -n '3,7p' "$SELF" | sed 's/^# \{0,1\}//'
|
|
59
|
+
echo
|
|
60
|
+
echo " --concurrency <n> batch width to answer for; default $CONCURRENCY"
|
|
61
|
+
echo " --assets <dir> where the indexes are; default $ASSETS"
|
|
62
|
+
echo " --skip-probe admission bound only; starts no container"
|
|
63
|
+
echo " --peak-rss <MiB> a measured per-container figure, instead of sampling"
|
|
64
|
+
echo " --json the same answer as data"
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
while [ $# -gt 0 ]; do
|
|
68
|
+
case "$1" in
|
|
69
|
+
--concurrency | -c) CONCURRENCY="${2:?--concurrency needs a number}"; shift 2 ;;
|
|
70
|
+
--assets) ASSETS="${2:?--assets needs a directory}"; shift 2 ;;
|
|
71
|
+
--peak-rss) PEAK_RSS="${2:?--peak-rss needs a number of MiB}"; shift 2 ;;
|
|
72
|
+
--skip-probe) SKIP_PROBE=1; shift ;;
|
|
73
|
+
--json) AS_JSON=1; shift ;;
|
|
74
|
+
-h | --help) usage; exit 0 ;;
|
|
75
|
+
*)
|
|
76
|
+
echo "preflight_sandbox.sh: unknown argument $1" >&2
|
|
77
|
+
usage >&2
|
|
78
|
+
exit 2
|
|
79
|
+
;;
|
|
80
|
+
esac
|
|
81
|
+
done
|
|
82
|
+
|
|
83
|
+
case "$CONCURRENCY$PEAK_RSS" in
|
|
84
|
+
*[!0-9]*)
|
|
85
|
+
echo "preflight_sandbox.sh: --concurrency and --peak-rss take whole numbers" >&2
|
|
86
|
+
exit 2
|
|
87
|
+
;;
|
|
88
|
+
esac
|
|
89
|
+
if [ "$CONCURRENCY" -lt 1 ]; then
|
|
90
|
+
echo "preflight_sandbox.sh: --concurrency must be at least 1" >&2
|
|
91
|
+
exit 2
|
|
92
|
+
fi
|
|
93
|
+
|
|
94
|
+
NOTES=""
|
|
95
|
+
note() {
|
|
96
|
+
NOTES="$NOTES$1
|
|
97
|
+
"
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
mib() { awk -v b="$1" 'BEGIN { printf "%d", b / 1048576 }'; }
|
|
101
|
+
gib() { awk -v m="$1" 'BEGIN { printf "%.1f", m / 1024 }'; }
|
|
102
|
+
|
|
103
|
+
# Everything below prints through here, so --json can suppress the prose without
|
|
104
|
+
# every site having to know about it.
|
|
105
|
+
say() {
|
|
106
|
+
if [ "$AS_JSON" -eq 0 ]; then printf '%s\n' "$1"; fi
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
# ---------------------------------------------------------------------------
|
|
110
|
+
# 1. Does this apply at all
|
|
111
|
+
#
|
|
112
|
+
# Three ways not to. Each is a real answer and none of them is a failure, so
|
|
113
|
+
# they share exit 3 and each says what could not be checked rather than what
|
|
114
|
+
# went wrong.
|
|
115
|
+
# ---------------------------------------------------------------------------
|
|
116
|
+
|
|
117
|
+
not_applicable() {
|
|
118
|
+
if [ "$AS_JSON" -eq 1 ]; then
|
|
119
|
+
printf '{"applicable":false,"reason":%s}\n' "\"$1\""
|
|
120
|
+
else
|
|
121
|
+
echo "not applicable: $1"
|
|
122
|
+
if [ -n "$2" ]; then echo " $2"; fi
|
|
123
|
+
fi
|
|
124
|
+
exit 3
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
if ! command -v podman > /dev/null 2>&1; then
|
|
128
|
+
not_applicable "podman is not on PATH" \
|
|
129
|
+
"nothing here is measurable; install podman, or run \`zen sandbox up\`"
|
|
130
|
+
fi
|
|
131
|
+
|
|
132
|
+
if [ "$(uname -s)" = "Linux" ]; then
|
|
133
|
+
not_applicable "podman is native on Linux — there is no VM to be the ceiling" \
|
|
134
|
+
"containers here are bounded by the host and by cgroups, not by a machine"
|
|
135
|
+
fi
|
|
136
|
+
|
|
137
|
+
CONF=""
|
|
138
|
+
for candidate in agents.yaml agents.yml agents/agents.yaml agents/agents.yml; do
|
|
139
|
+
if [ -f "$candidate" ]; then CONF="$candidate"; break; fi
|
|
140
|
+
done
|
|
141
|
+
if [ -z "$CONF" ]; then
|
|
142
|
+
not_applicable "no agents.yaml in $PWD" \
|
|
143
|
+
"run this from a project, or from anywhere inside one"
|
|
144
|
+
fi
|
|
145
|
+
if ! grep -Eq 'sandbox:(\*|run_command|run_command_background|read_command_output|stop_command)' "$CONF"; then
|
|
146
|
+
not_applicable "no agent in $CONF holds a sandbox tool" \
|
|
147
|
+
"no container is ever started, so capacity cannot be the fault"
|
|
148
|
+
fi
|
|
149
|
+
|
|
150
|
+
# ---------------------------------------------------------------------------
|
|
151
|
+
# 2. The host
|
|
152
|
+
#
|
|
153
|
+
# It bounds what the VM may be given and it is where `zen run batch` itself
|
|
154
|
+
# runs, which has been OOM-killed in its own right. Free pages are the ones
|
|
155
|
+
# that could be handed to a resize today.
|
|
156
|
+
# ---------------------------------------------------------------------------
|
|
157
|
+
|
|
158
|
+
HOST_MIB="$(mib "$(sysctl -n hw.memsize 2> /dev/null || echo 0)")"
|
|
159
|
+
HOST_FREE_MIB=0
|
|
160
|
+
if command -v vm_stat > /dev/null 2>&1; then
|
|
161
|
+
HOST_FREE_MIB="$(
|
|
162
|
+
vm_stat 2> /dev/null | awk '
|
|
163
|
+
/page size of/ { gsub(/[^0-9]/, "", $8); ps = $8 }
|
|
164
|
+
/Pages free/ || /Pages inactive/ { gsub(/[^0-9]/, "", $NF); p += $NF }
|
|
165
|
+
END { printf "%d", (ps ? ps : 4096) * p / 1048576 }' || true
|
|
166
|
+
)"
|
|
167
|
+
fi
|
|
168
|
+
|
|
169
|
+
# ---------------------------------------------------------------------------
|
|
170
|
+
# 3. The VM — the ceiling
|
|
171
|
+
#
|
|
172
|
+
# `podman info` is the authority because it reports the machine as it is
|
|
173
|
+
# running, swap included. A machine that is stopped cannot answer, so the
|
|
174
|
+
# configured size from `machine list` stands in and is labelled as such.
|
|
175
|
+
# ---------------------------------------------------------------------------
|
|
176
|
+
|
|
177
|
+
VM_MIB=0
|
|
178
|
+
VM_SWAP_MIB=0
|
|
179
|
+
VM_CPUS=0
|
|
180
|
+
VM_NAME=""
|
|
181
|
+
VM_STATE=running
|
|
182
|
+
|
|
183
|
+
INFO="$(podman info --format '{{.Host.MemTotal}} {{.Host.SwapTotal}} {{.Host.CPUs}}' 2> /dev/null || true)"
|
|
184
|
+
case "$INFO" in
|
|
185
|
+
[0-9]*)
|
|
186
|
+
VM_MIB="$(mib "$(echo "$INFO" | awk '{print $1}')")"
|
|
187
|
+
VM_SWAP_MIB="$(mib "$(echo "$INFO" | awk '{print $2}')")"
|
|
188
|
+
VM_CPUS="$(echo "$INFO" | awk '{print $3}')"
|
|
189
|
+
;;
|
|
190
|
+
*) VM_STATE=stopped ;;
|
|
191
|
+
esac
|
|
192
|
+
|
|
193
|
+
# `Memory` is bytes on podman 4 and a string like `2GiB` on podman 5, so it is
|
|
194
|
+
# normalised rather than trusted. Read whatever the state, for the name.
|
|
195
|
+
MACHINE="$(podman machine list --format '{{.Name}} {{.Memory}} {{.CPUs}} {{.Running}}' 2> /dev/null | head -1 || true)"
|
|
196
|
+
if [ -n "$MACHINE" ]; then
|
|
197
|
+
# `list` marks the default machine with a trailing asterisk.
|
|
198
|
+
VM_NAME="$(echo "$MACHINE" | awk '{print $1}' | sed 's/\*$//')"
|
|
199
|
+
if [ "$VM_STATE" = stopped ]; then
|
|
200
|
+
VM_MIB="$(
|
|
201
|
+
echo "$MACHINE" | awk '{
|
|
202
|
+
v = $2
|
|
203
|
+
n = v + 0
|
|
204
|
+
if (v ~ /GiB|GB|[Gg]$/) n = n * 1024
|
|
205
|
+
else if (v ~ /MiB|MB|[Mm]$/) n = n
|
|
206
|
+
else if (n > 1073741823) n = n / 1048576
|
|
207
|
+
printf "%d", n
|
|
208
|
+
}' || true
|
|
209
|
+
)"
|
|
210
|
+
VM_CPUS="$(echo "$MACHINE" | awk '{print $3}')"
|
|
211
|
+
fi
|
|
212
|
+
fi
|
|
213
|
+
|
|
214
|
+
if [ "$VM_STATE" = stopped ]; then
|
|
215
|
+
note "the podman machine is not running, so these are its configured sizes, not live ones"
|
|
216
|
+
note " start it to measure: podman machine start"
|
|
217
|
+
fi
|
|
218
|
+
if [ "$VM_MIB" -eq 0 ]; then
|
|
219
|
+
not_applicable "podman could not report the machine's memory" \
|
|
220
|
+
"try \`podman machine start\`, then run this again"
|
|
221
|
+
fi
|
|
222
|
+
|
|
223
|
+
# ---------------------------------------------------------------------------
|
|
224
|
+
# 4. The container cap, out of agents.yaml
|
|
225
|
+
#
|
|
226
|
+
# A numeric `memory:` can only be the sandbox's: the memory *graph* block takes
|
|
227
|
+
# a directory and an embedding, and an agent's binding takes a boolean or a map.
|
|
228
|
+
# Per-agent overrides are read too, and the largest wins — it is the one that
|
|
229
|
+
# decides whether the sum fits.
|
|
230
|
+
# ---------------------------------------------------------------------------
|
|
231
|
+
|
|
232
|
+
CONTAINER_MIB="$(
|
|
233
|
+
grep -E '^[[:space:]]+memory:[[:space:]]*[0-9]+[[:space:]]*(#.*)?$' "$CONF" 2> /dev/null \
|
|
234
|
+
| sed -E 's/.*memory:[[:space:]]*([0-9]+).*/\1/' | sort -n | tail -1 || true
|
|
235
|
+
)"
|
|
236
|
+
CONTAINER_SOURCE="$CONF"
|
|
237
|
+
if [ -z "$CONTAINER_MIB" ]; then
|
|
238
|
+
if grep -Eq '^[[:space:]]+hardening:[[:space:]]*strict' "$CONF"; then
|
|
239
|
+
CONTAINER_MIB="$STRICT_CONTAINER_MIB"
|
|
240
|
+
CONTAINER_SOURCE="default (strict)"
|
|
241
|
+
else
|
|
242
|
+
CONTAINER_MIB="$DEFAULT_CONTAINER_MIB"
|
|
243
|
+
CONTAINER_SOURCE="default"
|
|
244
|
+
fi
|
|
245
|
+
fi
|
|
246
|
+
|
|
247
|
+
CONTAINER_CPUS="$(
|
|
248
|
+
grep -E '^[[:space:]]+cpus:[[:space:]]*[0-9]+' "$CONF" 2> /dev/null \
|
|
249
|
+
| sed -E 's/.*cpus:[[:space:]]*([0-9]+).*/\1/' | sort -n | tail -1 || true
|
|
250
|
+
)"
|
|
251
|
+
CONTAINER_CPUS="${CONTAINER_CPUS:-4}"
|
|
252
|
+
|
|
253
|
+
if [ "$CONTAINER_MIB" -eq 0 ]; then
|
|
254
|
+
note "sandbox.memory is 0 — unconstrained, so one container may take the whole VM"
|
|
255
|
+
CONTAINER_MIB="$VM_MIB"
|
|
256
|
+
fi
|
|
257
|
+
|
|
258
|
+
# ---------------------------------------------------------------------------
|
|
259
|
+
# 5. The indexes
|
|
260
|
+
#
|
|
261
|
+
# An index is read through the page cache and the cache lives in the VM, so an
|
|
262
|
+
# index larger than the VM cannot be cached at all — which is the difference
|
|
263
|
+
# between a five-second grep and a two-minute one. Counted once: the mount is
|
|
264
|
+
# read-only and shared by every container.
|
|
265
|
+
# ---------------------------------------------------------------------------
|
|
266
|
+
|
|
267
|
+
INDEX_MIB=0
|
|
268
|
+
INDEX_NAME=""
|
|
269
|
+
INDEX_COUNT=0
|
|
270
|
+
if [ -d "$ASSETS" ]; then
|
|
271
|
+
while IFS= read -r manifest; do
|
|
272
|
+
[ -n "$manifest" ] || continue
|
|
273
|
+
dir="$(dirname "$manifest")"
|
|
274
|
+
size="$(du -sk "$dir" 2> /dev/null | awk '{printf "%d", $1 / 1024}' || echo 0)"
|
|
275
|
+
INDEX_COUNT=$((INDEX_COUNT + 1))
|
|
276
|
+
if [ "${size:-0}" -gt "$INDEX_MIB" ]; then
|
|
277
|
+
INDEX_MIB="$size"
|
|
278
|
+
INDEX_NAME="$dir"
|
|
279
|
+
fi
|
|
280
|
+
done <<EOF
|
|
281
|
+
$(find "$ASSETS" -maxdepth 3 -name manifest.json -type f 2> /dev/null || true)
|
|
282
|
+
EOF
|
|
283
|
+
fi
|
|
284
|
+
|
|
285
|
+
# ---------------------------------------------------------------------------
|
|
286
|
+
# 6. The probe — one container, one real read, peak RSS while it runs
|
|
287
|
+
#
|
|
288
|
+
# Process RSS, deliberately, not the cgroup's `memory.current`: that includes
|
|
289
|
+
# page cache, and the index is already a term of its own in the budget. Summing
|
|
290
|
+
# VmRSS across /proc inside the container is exactly the container's processes,
|
|
291
|
+
# because it has its own pid namespace.
|
|
292
|
+
#
|
|
293
|
+
# The command is `local` — it makes no network call. A gate that needed a
|
|
294
|
+
# credential would fail for reasons that are not about this machine.
|
|
295
|
+
# ---------------------------------------------------------------------------
|
|
296
|
+
|
|
297
|
+
PROBE_NAME=none
|
|
298
|
+
PROBE_SECS=0
|
|
299
|
+
SAMPLED_MIB=0
|
|
300
|
+
|
|
301
|
+
probe() {
|
|
302
|
+
if [ -z "$INDEX_NAME" ]; then
|
|
303
|
+
note "no index under ${ASSETS}/ to read — nothing representative to probe"
|
|
304
|
+
return 0
|
|
305
|
+
fi
|
|
306
|
+
image="${ZEN_IMAGE:-$(zen sandbox status 2> /dev/null | awk '$1 == "image" { print $2 }' || true)}"
|
|
307
|
+
if [ -z "$image" ]; then
|
|
308
|
+
note "no sandbox image to probe with — run \`zen sandbox up\`, or pass --peak-rss"
|
|
309
|
+
return 0
|
|
310
|
+
fi
|
|
311
|
+
PROBE_NAME="read $INDEX_NAME"
|
|
312
|
+
|
|
313
|
+
# Passed as one argv element, so no second shell re-parses it. The loop
|
|
314
|
+
# bounds itself rather than trusting `timeout` to be in the project's image.
|
|
315
|
+
script="$(
|
|
316
|
+
cat << EOS
|
|
317
|
+
( zen --version > /dev/null 2>&1; grep -rl zzzz-not-present /assets > /dev/null 2>&1 ) &
|
|
318
|
+
p=\$!
|
|
319
|
+
peak=0
|
|
320
|
+
ticks=0
|
|
321
|
+
while kill -0 \$p 2> /dev/null; do
|
|
322
|
+
cur=\$(awk '/^VmRSS:/ { s += \$2 } END { print s + 0 }' /proc/[0-9]*/status 2> /dev/null)
|
|
323
|
+
if [ "\${cur:-0}" -gt "\$peak" ]; then peak=\$cur; fi
|
|
324
|
+
ticks=\$((ticks + 1))
|
|
325
|
+
if [ \$ticks -gt $((PROBE_TIMEOUT * 5)) ]; then kill \$p 2> /dev/null; break; fi
|
|
326
|
+
sleep 0.2
|
|
327
|
+
done
|
|
328
|
+
wait \$p 2> /dev/null
|
|
329
|
+
echo "PEAK_KB=\$peak"
|
|
330
|
+
EOS
|
|
331
|
+
)"
|
|
332
|
+
|
|
333
|
+
started="$(date +%s)"
|
|
334
|
+
out="$(
|
|
335
|
+
podman run --rm -v "$PWD/$ASSETS:/assets:ro" \
|
|
336
|
+
--memory "${CONTAINER_MIB}m" --entrypoint="" \
|
|
337
|
+
"$image" sh -c "$script" 2> /dev/null || true
|
|
338
|
+
)"
|
|
339
|
+
PROBE_SECS=$(($(date +%s) - started))
|
|
340
|
+
|
|
341
|
+
kb="$(printf '%s' "$out" | sed -n 's/^PEAK_KB=\([0-9]*\)$/\1/p' | tail -1 || true)"
|
|
342
|
+
if [ -n "${kb:-}" ] && [ "${kb:-0}" -gt 0 ]; then
|
|
343
|
+
SAMPLED_MIB=$((kb / 1024))
|
|
344
|
+
else
|
|
345
|
+
note "the probe reported nothing — using the floor of ${PEAK_FLOOR_MIB} MiB"
|
|
346
|
+
fi
|
|
347
|
+
}
|
|
348
|
+
|
|
349
|
+
if [ "$PEAK_RSS" -gt 0 ]; then
|
|
350
|
+
PEAK_MIB="$PEAK_RSS"
|
|
351
|
+
PEAK_SOURCE="--peak-rss"
|
|
352
|
+
elif [ "$SKIP_PROBE" -eq 1 ]; then
|
|
353
|
+
PEAK_MIB="$PEAK_FLOOR_MIB"
|
|
354
|
+
PEAK_SOURCE="floor (--skip-probe)"
|
|
355
|
+
else
|
|
356
|
+
probe
|
|
357
|
+
if [ "$SAMPLED_MIB" -gt "$PEAK_FLOOR_MIB" ]; then
|
|
358
|
+
PEAK_MIB="$SAMPLED_MIB"
|
|
359
|
+
PEAK_SOURCE="sampled at concurrency 1"
|
|
360
|
+
elif [ "$SAMPLED_MIB" -gt 0 ]; then
|
|
361
|
+
PEAK_MIB="$PEAK_FLOOR_MIB"
|
|
362
|
+
PEAK_SOURCE="floor — the probe measured ${SAMPLED_MIB} MiB, below it"
|
|
363
|
+
else
|
|
364
|
+
# The probe did not run, or reported nothing. `note` has already said
|
|
365
|
+
# which; do not claim a measurement that was never taken.
|
|
366
|
+
PEAK_MIB="$PEAK_FLOOR_MIB"
|
|
367
|
+
PEAK_SOURCE="floor (unmeasured)"
|
|
368
|
+
fi
|
|
369
|
+
fi
|
|
370
|
+
|
|
371
|
+
# ---------------------------------------------------------------------------
|
|
372
|
+
# 7. The verdict
|
|
373
|
+
# ---------------------------------------------------------------------------
|
|
374
|
+
|
|
375
|
+
ADMITTED_MIB=$((CONCURRENCY * CONTAINER_MIB))
|
|
376
|
+
REQUIRED_MIB=$((INDEX_MIB + CONCURRENCY * PEAK_MIB + HEADROOM_MIB))
|
|
377
|
+
|
|
378
|
+
OVERSUBSCRIBED=0
|
|
379
|
+
if [ "$ADMITTED_MIB" -gt "$VM_MIB" ]; then OVERSUBSCRIBED=1; fi
|
|
380
|
+
|
|
381
|
+
SUFFICIENT=1
|
|
382
|
+
if [ "$REQUIRED_MIB" -gt "$VM_MIB" ]; then SUFFICIENT=0; fi
|
|
383
|
+
|
|
384
|
+
# The width that does fit today, which is the remedy available on every machine.
|
|
385
|
+
FITS=0
|
|
386
|
+
budget=$((VM_MIB - INDEX_MIB - HEADROOM_MIB))
|
|
387
|
+
if [ "$budget" -gt 0 ]; then FITS=$((budget / PEAK_MIB)); fi
|
|
388
|
+
|
|
389
|
+
# Rounded up to the next whole GiB, because `machine set` takes MiB and nobody
|
|
390
|
+
# wants a ceiling that is exact.
|
|
391
|
+
SUGGEST_MIB=$(((REQUIRED_MIB + 1023) / 1024 * 1024))
|
|
392
|
+
|
|
393
|
+
if [ "$AS_JSON" -eq 1 ]; then
|
|
394
|
+
printf '{"applicable":true,"sufficient":%s,"oversubscribed":%s,' \
|
|
395
|
+
"$([ "$SUFFICIENT" -eq 1 ] && echo true || echo false)" \
|
|
396
|
+
"$([ "$OVERSUBSCRIBED" -eq 1 ] && echo true || echo false)"
|
|
397
|
+
printf '"concurrency":%s,"hostMib":%s,"hostFreeMib":%s,' \
|
|
398
|
+
"$CONCURRENCY" "$HOST_MIB" "$HOST_FREE_MIB"
|
|
399
|
+
printf '"vm":{"name":"%s","state":"%s","memoryMib":%s,"swapMib":%s,"cpus":%s},' \
|
|
400
|
+
"$VM_NAME" "$VM_STATE" "$VM_MIB" "$VM_SWAP_MIB" "$VM_CPUS"
|
|
401
|
+
printf '"container":{"memoryMib":%s,"cpus":%s,"source":"%s"},' \
|
|
402
|
+
"$CONTAINER_MIB" "$CONTAINER_CPUS" "$CONTAINER_SOURCE"
|
|
403
|
+
printf '"index":{"largestMib":%s,"name":"%s","count":%s},' \
|
|
404
|
+
"$INDEX_MIB" "$INDEX_NAME" "$INDEX_COUNT"
|
|
405
|
+
printf '"peakRssMib":%s,"peakSource":"%s","probe":"%s","probeSeconds":%s,' \
|
|
406
|
+
"$PEAK_MIB" "$PEAK_SOURCE" "$PROBE_NAME" "$PROBE_SECS"
|
|
407
|
+
printf '"admittedMib":%s,"requiredMib":%s,"suggestMib":%s,"fitsAtConcurrency":%s}\n' \
|
|
408
|
+
"$ADMITTED_MIB" "$REQUIRED_MIB" "$SUGGEST_MIB" "$FITS"
|
|
409
|
+
else
|
|
410
|
+
say "host $(gib "$HOST_MIB") GiB total · $(gib "$HOST_FREE_MIB") GiB free"
|
|
411
|
+
swap="$VM_SWAP_MIB MiB swap"
|
|
412
|
+
if [ "$VM_SWAP_MIB" -eq 0 ]; then swap="no swap — the kill is immediate, not gradual"; fi
|
|
413
|
+
say "vm $(gib "$VM_MIB") GiB total · $swap · ${VM_CPUS} cpus ${VM_NAME:-?} ($VM_STATE)"
|
|
414
|
+
say "container ${CONTAINER_MIB} MiB cap · ${CONTAINER_CPUS} cpus ${CONTAINER_SOURCE}"
|
|
415
|
+
if [ "$INDEX_COUNT" -gt 0 ]; then
|
|
416
|
+
say "index $(gib "$INDEX_MIB") GiB ${INDEX_NAME} (largest of ${INDEX_COUNT})"
|
|
417
|
+
else
|
|
418
|
+
say "index none found under ${ASSETS}/"
|
|
419
|
+
fi
|
|
420
|
+
timing=""
|
|
421
|
+
if [ "$PROBE_SECS" -gt 0 ]; then timing=" · probe ${PROBE_SECS}s"; fi
|
|
422
|
+
say "peak rss ${PEAK_MIB} MiB ${PEAK_SOURCE}${timing}"
|
|
423
|
+
say ""
|
|
424
|
+
|
|
425
|
+
if [ "$OVERSUBSCRIBED" -eq 1 ]; then
|
|
426
|
+
say "admission ${CONCURRENCY} x ${CONTAINER_MIB} MiB = $(gib "$ADMITTED_MIB") GiB permitted against a $(gib "$VM_MIB") GiB vm OVERSUBSCRIBED"
|
|
427
|
+
if [ "$CONTAINER_MIB" -gt "$VM_MIB" ]; then
|
|
428
|
+
say " one container alone is permitted more than the whole machine"
|
|
429
|
+
else
|
|
430
|
+
say " nothing will refuse the sum; only the containers not all"
|
|
431
|
+
say " peaking together is keeping this alive"
|
|
432
|
+
fi
|
|
433
|
+
fi
|
|
434
|
+
|
|
435
|
+
if [ "$SUFFICIENT" -eq 1 ]; then
|
|
436
|
+
say "concurrency ${CONCURRENCY} needs ~$(gib "$REQUIRED_MIB") GiB vm has $(gib "$VM_MIB") GiB ok"
|
|
437
|
+
else
|
|
438
|
+
say "concurrency ${CONCURRENCY} needs ~$(gib "$REQUIRED_MIB") GiB vm has $(gib "$VM_MIB") GiB INSUFFICIENT"
|
|
439
|
+
say " fix: podman machine stop && podman machine set --memory ${SUGGEST_MIB} && podman machine start"
|
|
440
|
+
if [ "$FITS" -ge 1 ]; then
|
|
441
|
+
say " or: run the batch at --concurrency ${FITS}"
|
|
442
|
+
else
|
|
443
|
+
say " or: nothing fits — the index alone exceeds this vm; resizing is the only way"
|
|
444
|
+
fi
|
|
445
|
+
if [ "$SUGGEST_MIB" -gt "$HOST_MIB" ]; then
|
|
446
|
+
say " note: that is more than the host has. Lower the concurrency instead"
|
|
447
|
+
fi
|
|
448
|
+
fi
|
|
449
|
+
|
|
450
|
+
if [ -n "$NOTES" ]; then
|
|
451
|
+
say ""
|
|
452
|
+
printf '%s' "$NOTES" | sed 's/^/note: /'
|
|
453
|
+
fi
|
|
454
|
+
fi
|
|
455
|
+
|
|
456
|
+
if [ "$SUFFICIENT" -eq 1 ]; then exit 0; fi
|
|
457
|
+
exit 1
|
|
@@ -111,11 +111,12 @@ However small the diff:
|
|
|
111
111
|
last.
|
|
112
112
|
2. **`scripts/_setup.sh` has run to completion in this session, and re-runs
|
|
113
113
|
clean.** A one-line specification change can still break a setup step, and a
|
|
114
|
-
pass that did not run it does not know. `
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
committed `manifest.json` skips on a
|
|
118
|
-
vectors are missing, and the project
|
|
114
|
+
pass that did not run it does not know. Then `scripts/_setup.sh --check` must
|
|
115
|
+
report every step `skipped`: it makes each step's own readiness test without
|
|
116
|
+
doing the work, which is what tells a real `skipped` from a step that only
|
|
117
|
+
looked done. A step that tests for a committed `manifest.json` skips on a
|
|
118
|
+
fresh clone whose git-ignored `lance/` vectors are missing, and the project
|
|
119
|
+
cannot search.
|
|
119
120
|
3. **§9 of the copilot instructions, worked as a checklist**, group by group.
|
|
120
121
|
4. **The capability obligations in §0.1.** For every capability the project has
|
|
121
122
|
turned on, load that capability's skill and re-check what it requires - in
|
|
@@ -28,6 +28,8 @@ something in it — and report what was done in enough detail to be checked.
|
|
|
28
28
|
are the steps it runs in order. A step is safe to run twice and reports
|
|
29
29
|
`skipped` the second time — testing for something that survives a clone, not
|
|
30
30
|
for a file `.gitignore` excludes.
|
|
31
|
+
- `scripts/_setup.sh --check` asks the same questions and does none of the work,
|
|
32
|
+
so what is missing can be read without paying to find out.
|
|
31
33
|
- Currently: no steps. Nothing has to be prepared before a run.
|
|
32
34
|
|
|
33
35
|
## Agents
|
|
@@ -69,6 +69,19 @@ Example patch:
|
|
|
69
69
|
Use `write_file` only to create a **new** file or to overwrite a small file completely.
|
|
70
70
|
Never use `write_file` to update an existing file when only a few lines need to change.
|
|
71
71
|
|
|
72
|
+
### Copying: `copy_file`
|
|
73
|
+
|
|
74
|
+
Use `copy_file` to duplicate a file rather than reading it and writing it back out.
|
|
75
|
+
The bytes go straight from one path to the other without passing through the
|
|
76
|
+
conversation, so the size cap on `read_file` does not apply and a binary arrives
|
|
77
|
+
intact.
|
|
78
|
+
|
|
79
|
+
- The source may be in a read-only mount; only the destination has to be writable.
|
|
80
|
+
This is how something under `/assets`, `/skills` or `/memory` is brought into
|
|
81
|
+
`/workspace`.
|
|
82
|
+
- Set `overwrite: true` only when intentionally replacing an existing destination.
|
|
83
|
+
- Copying a directory and its contents requires `recursive: true`.
|
|
84
|
+
|
|
72
85
|
### Moving and deleting
|
|
73
86
|
|
|
74
87
|
- Use `move_file` to rename or relocate a file or directory. It creates parent directories
|
|
@@ -48,6 +48,31 @@ the tool call.
|
|
|
48
48
|
Do not fork a sequence. Branches run at once and can exchange nothing, so a step
|
|
49
49
|
that needs the step before it has to stay in this conversation.
|
|
50
50
|
|
|
51
|
+
## When the assignment is yours
|
|
52
|
+
|
|
53
|
+
These rules reach a branch as well as the agent that forked it, and a branch
|
|
54
|
+
arrives in the middle of someone else's sentence. Your assignment comes to you
|
|
55
|
+
as the result of a `fork` call, because that is how the runtime delivers it —
|
|
56
|
+
and that is indistinguishable, in a transcript, from a fork you made yourself
|
|
57
|
+
coming back with an answer.
|
|
58
|
+
|
|
59
|
+
It is not one. When a message begins "You are branch ... of the fork above", you
|
|
60
|
+
are that branch and that text is your assignment:
|
|
61
|
+
|
|
62
|
+
- **Nothing has run yet.** There is no result in it and nothing to verify. The
|
|
63
|
+
fork it names is above in the conversation you were forked out of, which is
|
|
64
|
+
not the one you are in — yours begins with that message.
|
|
65
|
+
- **Do the work yourself.** Never fork a branch to carry out the assignment you
|
|
66
|
+
were handed: that branch is you one level down holding the same instructions,
|
|
67
|
+
it will read them the same way, and each level spends a whole system prompt to
|
|
68
|
+
arrive back where it started. At the depth cap the tool is withdrawn and the
|
|
69
|
+
work comes back undone.
|
|
70
|
+
- **Do not go looking for proof it was already done.** An empty workspace is
|
|
71
|
+
what an unstarted job looks like, not a sign that something failed upstream.
|
|
72
|
+
|
|
73
|
+
Fork from inside a branch only for work that is genuinely divisible further and
|
|
74
|
+
that you were not handed already divided.
|
|
75
|
+
|
|
51
76
|
## When it is worth it
|
|
52
77
|
|
|
53
78
|
One question decides it: **could this be finished by someone who cannot ask you
|
|
@@ -63,6 +88,8 @@ no, the work stays here.
|
|
|
63
88
|
| Find the config, then change it, then test it | Not a fork. Each step needs the one before it |
|
|
64
89
|
| A change you would want to steer while it happens | Not a fork. A branch cannot be corrected once it starts |
|
|
65
90
|
| One read, one search, one command | Not a fork. A whole conversation to save yourself a line |
|
|
91
|
+
| Twenty items one call already takes as a list | Not a fork. Pass the list |
|
|
92
|
+
| The assignment you were handed, whole | Not a fork. It is yours — see above |
|
|
66
93
|
| Two jobs that would write the same file | Not a fork until they write different files — see below |
|
|
67
94
|
|
|
68
95
|
## Fanning out
|