@zibby/skills 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/datasetStore.js +6 -4
- package/dist/index.js +75 -73
- package/dist/package.json +1 -1
- package/docs/concepts/designing-agents.md +46 -0
- package/package.json +1 -1
package/dist/package.json
CHANGED
|
@@ -164,6 +164,52 @@ graph.addNode('build_report', {
|
|
|
164
164
|
every agent, not a registry Store).
|
|
165
165
|
:::
|
|
166
166
|
|
|
167
|
+
### File-store size limit — and how to handle big files
|
|
168
|
+
|
|
169
|
+
A `file` store's per-file cap is **4 MiB of raw bytes** (the content travels
|
|
170
|
+
base64-inside-JSON through the API). Two ways past it:
|
|
171
|
+
|
|
172
|
+
- **A relational/append-only shape? Use `sqlite`/`dataset`, not `file`.** Those
|
|
173
|
+
aren't whole-file uploads — they take rows/records, so the 4 MiB blob cap
|
|
174
|
+
doesn't apply. This is the right move for raw event data (commits, worklogs).
|
|
175
|
+
- **A genuinely large blob** (a multi-MB `.jsonl` a script must read whole)?
|
|
176
|
+
**gzip it, transparently.** Text compresses well; store the compressed bytes
|
|
177
|
+
and decompress on read, so the generating script still sees the original file:
|
|
178
|
+
|
|
179
|
+
```js
|
|
180
|
+
// write node: gzip + base64 → store as <name>.gz.b64
|
|
181
|
+
import { gzipSync } from 'node:zlib';
|
|
182
|
+
const packed = gzipSync(Buffer.from(rawText)).toString('base64'); // 5.6MB → ~0.8MB
|
|
183
|
+
await ds(fileStore, 'put', { path: 'big.jsonl.gz.b64', content: packed });
|
|
184
|
+
|
|
185
|
+
// read node: fetch → base64-decode → gunzip → original file on disk
|
|
186
|
+
import { gunzipSync } from 'node:zlib';
|
|
187
|
+
const got = await ds(fileStore, 'get', { path: 'big.jsonl.gz.b64' });
|
|
188
|
+
const raw = gunzipSync(Buffer.from(got.content, 'base64')).toString('utf8');
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Keep the list of "which files are compressed" as data (one array), so adding a
|
|
192
|
+
big file later is a one-line change — the read/write nodes stay generic.
|
|
193
|
+
|
|
194
|
+
:::caution Check every dependency's SIZE up front
|
|
195
|
+
When you list a store's inputs, don't glob one extension and assume — a single
|
|
196
|
+
oversized dependency (e.g. a 5.6 MB `.jsonl` hiding among small `.json`s) is what
|
|
197
|
+
trips the cap. Enumerate the real files + their bytes before wiring the store.
|
|
198
|
+
File paths accept Unicode (`报告.html` is fine); only `/ \ : ? # [ ] " < > | *`
|
|
199
|
+
and control characters are rejected.
|
|
200
|
+
:::
|
|
201
|
+
|
|
202
|
+
### Debugging a self-host run when logs look truncated
|
|
203
|
+
|
|
204
|
+
Self-host truncates long per-node step logs, so a node can report `success`
|
|
205
|
+
while a downstream step silently failed (you see all-green but empty output).
|
|
206
|
+
The robust pattern: **have each node persist its own full result JSON into an
|
|
207
|
+
output store** (e.g. `diag/<node>.json`) — best-effort, never throwing, never
|
|
208
|
+
changing the node's return value. Then read the diagnostics back with
|
|
209
|
+
`zibby_store_peek` to see exactly what each node saw (staged files + sizes,
|
|
210
|
+
stderr, the real error). This turns a silent all-green failure into a precise
|
|
211
|
+
root cause.
|
|
212
|
+
|
|
167
213
|
---
|
|
168
214
|
|
|
169
215
|
## Pattern 3 — event-driven agents
|