opensteer 0.8.5 → 0.8.6

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opensteer",
-  "version": "0.8.5",
+  "version": "0.8.6",
   "description": "Opensteer browser automation, replay, and reverse-engineering toolkit.",
   "license": "MIT",
   "type": "module",
@@ -56,14 +56,14 @@
     "sharp": "^0.34.5",
     "skills": "^1.4.6",
     "ws": "^8.18.0",
-    "@opensteer/engine-playwright": "0.8.0",
-    "@opensteer/runtime-core": "0.1.0"
+    "@opensteer/engine-playwright": "0.8.1",
+    "@opensteer/runtime-core": "0.1.1"
   },
   "optionalDependencies": {
     "webcrack": "^2.15.1"
   },
   "peerDependencies": {
-    "@opensteer/engine-abp": "0.8.1"
+    "@opensteer/engine-abp": "0.8.2"
   },
   "peerDependenciesMeta": {
     "@opensteer/engine-abp": {
@@ -71,9 +71,9 @@
     }
   },
   "devDependencies": {
-    "@opensteer/browser-core": "0.7.0",
-    "@opensteer/engine-abp": "0.8.1",
-    "@opensteer/protocol": "0.7.0"
+    "@opensteer/browser-core": "0.7.1",
+    "@opensteer/engine-abp": "0.8.2",
+    "@opensteer/protocol": "0.7.1"
   },
   "scripts": {
     "build": "tsup && node ../../scripts/sync-package-skills.mjs",

package/skills/opensteer/SKILL.md

@@ -10,9 +10,9 @@ argument-hint: "[goal]"
 
 1. Run `snapshot action` or `snapshot extraction` first. The output is JSON with `{url, title, mode, html, counters}`. **Read the `html` field** — it is a clean filtered DOM with inline `c="N"` attributes marking every element. Do NOT parse the `counters` array for element discovery — it is verbose metadata.
 2. Use `element` + `persistAsDescription` to act on elements. Use `extract()` with `description` + `schema` to extract data. Do NOT use `page.evaluate()`, CSS selectors, or raw DOM parsing when `extract()` can express the output.
-3. Array extraction auto-generalizes: provide 1-2 representative rows as templates, and the extractor finds ALL matching rows on the page.
+3. Extraction schemas are **literal**. If you provide 2 template rows, you get exactly 2 rows back. The framework consolidates those templates into a generalized selector behind the scenes and saves it as a descriptor. Replaying with `description` alone (no schema) uses that generalized selector to return **ALL** matching rows.
 4. `persistAsDescription` requires the verbose `opensteer run dom.*` syntax. The short CLI commands (`click`, `input`, etc.) do NOT support it.
-5. Phase 1 = CLI exploration (snapshot, act, extract). Phase 2 = deterministic scripts using `description` alone. No snapshots in Phase 2.
+5. Phase 1 = CLI exploration (snapshot, act, extract with schema). Phase 2 = deterministic replay using `description` alone returns all matching data. No snapshots in Phase 2.
 
 If invoked directly, treat `$ARGUMENTS` as the concrete browser or replay goal. First decide whether the task is primarily DOM automation, request capture/replay, or workspace browser administration.
 
@@ -73,23 +73,23 @@ opensteer snapshot extraction --workspace demo
 # → Read html field: <div c="13">Apple AirPods</div> <span c="14">$189.99</span> ...
 
 opensteer extract --workspace demo --description "search results" \
-  --schema-json '{"items":[{"name":{"element":13},"price":{"element":14}}]}'
-# → Returns ALL matching rows on the page, not just the 1 template
+  --schema-json '{"items":[{"name":{"element":13},"price":{"element":14}},{"name":{"element":22},"price":{"element":23}}]}'
+# → Returns exactly 2 rows (the literal template values)
+# → Behind the scenes: consolidates templates into a generalized selector and saves it as a descriptor
 
 opensteer close --workspace demo
 ```
 
-**Phase 2 — Deterministic script (reusable):**
+**Phase 2 — Deterministic replay (reusable):**
 
 1. Use `description` alone for all interactions — resolves from cached descriptors.
-2. Use `description + schema` for extraction — caches the extraction descriptor.
-3. Use bare `description` for extraction replay.
-4. No snapshot calls needed in scripts. Just descriptions.
+2. Use `description` alone for extraction replay — uses the generalized selector to return **ALL** matching rows.
+3. No snapshot calls needed. Just descriptions.
 
 ## Shared Rules
 
 - The short CLI commands (`click`, `input`, etc.) accept exactly one of `--element`, `--selector`, or `--description`. Use `opensteer run dom.*` with `--input-json` when you need `persistAsDescription`.
-- For extraction, `description + schema` authors or updates a persisted extraction descriptor. `description` alone replays the stored extraction payload.
+- For extraction, `description + schema` returns literal template values and saves a generalized extraction descriptor. `description` alone replays the descriptor and returns ALL matching rows.
 - Extraction schemas are explicit JSON objects and arrays. Each leaf must be `{ element: N }`, `{ selector: "..." }`, optional `attribute`, or `{ source: "current_url" }`.
 - Persisted extraction replay is deterministic and snapshot-backed. Do not replace `extract()` with `evaluate()` or custom DOM parsing when the desired output fits the extraction schema.
 - Use recipes for deterministic setup work. Use auth recipes for auth refresh/setup specifically. They live in separate registries.
@@ -102,3 +102,4 @@ opensteer close --workspace demo
 - Using `page.evaluate()` or CSS selectors instead of `extract()`. Use extract with element-based schemas.
 - Forgetting to re-snapshot after navigation. Always re-snapshot before targeting new elements.
 - Using short CLI (`click`, `input`) when `persistAsDescription` is needed. Use `opensteer run dom.*`.
+- Expecting `extract --schema-json` with array templates to return all rows. The schema is literal — you get back exactly the rows you specified. Use description-only replay (`extract --description`) to get all matching rows.

package/skills/opensteer/references/cli-reference.md

@@ -45,6 +45,8 @@ opensteer close --workspace demo
 - Replay cached actions with `--description` alone — no snapshot needed.
 - `extract --description --schema-json` writes a persisted extraction descriptor.
 - `extract --description` replays the stored extraction.
+- Saved-network persistence (`network.save`, `network.query` with `source: "saved"`, and `network.clear`) is SQLite-backed and initializes on first use.
+- Generic workspace and browser commands do not require SQLite capability unless they touch saved-network persistence.
 
 ## Snapshot Output — What To Read
 
@@ -174,10 +176,12 @@ opensteer run request.execute --workspace demo \
 - Use `run page.goto` when you need `networkTag` on navigation. The short `goto` form only parses the URL positional.
 - Use `run dom.click` / `run dom.input` / `run dom.hover` / `run dom.scroll` when you need `persistAsDescription`.
 
-## Extraction Schema And Array Auto-Generalization
+## Extraction Schema
 
 Always run `snapshot extraction` before building a schema — you need the `c="N"` counter values from the HTML.
 
+Schemas are **literal**: each `element` reference points to one specific DOM element, and you get back exactly the values you asked for. The schema is not a prompt or pattern — it is a precise specification.
+
 Flat field bindings:
 
 ```bash
@@ -190,7 +194,13 @@ opensteer extract --workspace demo \
   --schema-json '{"url":{"selector":"a.primary","attribute":"href"},"pageUrl":{"source":"current_url"}}'
 ```
 
-Array extraction with auto-generalization:
+## Array Extraction (Two-Step Process)
+
+Extracting lists of items (product rows, search results, etc.) is a two-step process:
+
+**Step 1 — Teach the pattern (schema + description):**
+
+Provide 2 structurally similar rows as templates. The extractor returns exactly those rows (literal), but behind the scenes consolidates the templates into a generalized selector and saves it as a descriptor.
 
 ```bash
 opensteer extract --workspace demo \
@@ -198,31 +208,52 @@ opensteer extract --workspace demo \
   --schema-json '{"items":[{"name":{"element":13},"price":{"element":14}},{"name":{"element":22},"price":{"element":23}}]}'
 ```
 
-The extractor uses the 1-2 representative rows as **templates**, then automatically finds **ALL matching rows** on the page:
+Returns exactly 2 rows (the literal template values):
+```json
+{
+  "items": [
+    {"name": "Apple AirPods Max", "price": "$549.99"},
+    {"name": "Apple AirPods Pro", "price": "$249.99"}
+  ]
+}
+```
+
+**Step 2 — Replay to get all rows (description only, no schema):**
+
+Replay the saved descriptor. The generalized selector finds ALL matching rows on the page:
 
+```bash
+opensteer extract --workspace demo --description "product list"
+```
+
+Returns all matching rows:
 ```json
 {
   "items": [
     {"name": "Apple AirPods Max", "price": "$549.99"},
     {"name": "Apple AirPods Pro", "price": "$249.99"},
     {"name": "Apple AirPods 4", "price": "$129.99"},
-    ...
+    {"name": "Apple AirPods 4 (ANC)", "price": "$179.99"}
   ]
 }
 ```
 
+This replay is deterministic and works across page updates, pagination, and different page states.
+
 Rules:
 
 - Build the exact JSON shape you want. The extractor does not accept `"string"` or prompt-style schemas.
 - Each leaf must be `{ element: N }`, `{ selector: "..." }`, optional `attribute`, or `{ source: "current_url" }`.
 - Use `element` fields during CLI exploration with a fresh snapshot. Deterministic scripts use `description`.
-- For arrays, provide 1-2 representative objects. Add 2 when repeated rows have structural variants.
+- For arrays, provide 2 representative objects from different positions in the list. This gives the consolidator enough structural signal to generalize. Add more when rows have structural variants.
 - Nested arrays are not supported.
+- Do NOT expect the initial `extract --schema-json` call to return all rows. It returns exactly the template rows. Use description-only replay for the full list.
 
 ## What NOT To Do
 
 - Do NOT use `page.evaluate()` to scrape DOM data. Use `extract()` with element-based schemas.
 - Do NOT parse the `counters` array to find elements. Read the `html` string and find `c="N"` values.
 - Do NOT use CSS selectors in reusable scripts. Use `description` from cached descriptors.
-- Do NOT write loops to enumerate list items. Use array extraction with 1-2 template rows.
+- Do NOT write loops to enumerate list items. Use array extraction: provide 2 template rows in the schema, then replay with description only to get all rows.
+- Do NOT expect `extract --schema-json` with array templates to return all rows. It returns exactly the templates. Replay with `--description` alone for the full list.
 - Do NOT skip re-snapshot after navigation. Always re-snapshot before targeting new elements.

package/skills/opensteer/references/request-workflow.md

@@ -96,6 +96,8 @@ await opensteer.saveNetwork({
 });
 ```
 
+Saved-network persistence is SQLite-backed and initializes on first use. Generic workspace and browser flows do not require SQLite capability unless they touch saved-network persistence.
+
 6. Replay the plan from code.
 
 ```ts

package/skills/opensteer/references/sdk-reference.md

@@ -35,6 +35,8 @@ const opensteer = new Opensteer({
 - `opensteer.browser.status()`, `clone()`, `reset()`, and `delete()` manage the persistent workspace browser.
 - `close()` shuts the current session and, for persistent workspaces, closes the live browser process.
 - `disconnect()` detaches local runtime handles and leaves the workspace/browser files intact.
+- Saved-network persistence is SQLite-backed and initializes on first `saveNetwork()`, saved-source `queryNetwork()`, or `clearNetwork()` use.
+- Generic workspace and browser helpers do not require SQLite capability unless they touch saved-network persistence.
 - The current public SDK does not expose `Opensteer.attach()`, cloud session helpers, or the ABP engine.
 
 ## DOM Automation And Extraction