portage-cli 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c506344f3b17a2ea4199f84efaf70d658b0debec2fe03d167b847c58617e800b
-  data.tar.gz: 20c00c246e59fa9059611f7bb54380140295eada2c5deb457a11ec94309c1b04
+  metadata.gz: f4200c871324b9f35de00e58c32177ec087c9922b3adda2ae45aa1b4673478f1
+  data.tar.gz: e719be3b212a547be84ff5fa1232b08303c96288b7acf60f0fd9c2f1982fa0ef
 SHA512:
-  metadata.gz: 9a3eff64f57ab81774a2a52034232a4667227aaff13aa7279d26dd41cf52faeafeea691711755fb41fd2f3a0a67f77d144d248d48c2631779a49c10bec9f2882
-  data.tar.gz: ade0a2ba64ba928ec1272a8902858bad1f79897f5059d428528b8978e4045047ee1a0e6e2579dc2f8a468fa73b0c3491cc66c48f2bc1a18951600e05bd2a25e1
+  metadata.gz: 3bb580b0b6ac745462ecc241b4e151fa16c2a8651ac32aa6d53c84aca0b910da6095182680b6a832b3431539904a697332279de9b9340fb491581821eccf6519
+  data.tar.gz: 7a75983b229489dc8519caa7a176122b00c8f41f56922f4b373c551a9ab044dace860f7ecdd6ba93d63feddf84177391d3ae92f13a6280acd138e704590ea4f3

data/CHANGELOG.md

@@ -6,6 +6,58 @@ pre-1.0, so APIs may still shift between minor versions.
 
 ## [Unreleased]
 
+## [0.11.0] - 2026-10-01
+
+- **Category classification uses the whole taxonomy.** `known-stores/categories.yml` is now generated
+  by `script/categories` (stdlib only) from Google's product taxonomy, edition 2021-09-21, instead of
+  coming from a script that was never committed. It keeps the same ids and the same top-two-level keys
+  (so `~/.portage/categories.yml` overrides still work), but a level-2 node's `keywords` now include
+  its descendants' names ("Chandeliers" counts for Lighting), and its parent's words sit in a separate
+  `parent_keywords` list. The file grows from 20KB to 101KB. A shipped stoplist
+  (`known-stores/category-stoplist.yml`, every word with its reason) removes merchandising and url words
+  ("new", "collection", "sale", "gift", "accessories", "products", ...) from both the data and the input,
+  and a small synonyms file (`category-synonyms.yml`, each word tied to a golden case) adds the few
+  words the taxonomy lacks ("pendant", "sconce"). `Classifier.categories_for` keeps its signature (plus
+  an optional `stoplist_path:`) and returns at most three ids, best first. Scoring: a keyword counts
+  2 and a `parent_keyword` 1, times how often the input repeats the word (1 + ln count) and, for the
+  cut, how rare it is. A word counts once per node, and an id scoring under half the strongest is
+  dropped. Ties go to the node whose own name the input covers most, then to one whose name has no
+  stoplisted word, then to the smaller node. On a 100-case golden set (Light Yard, JB Hi-Fi, shopper
+  queries, browser-import titles and urls) top-1 accuracy goes from 12% to 70%. "Pendant Light" is
+  Lighting, "New Collection" is no longer Toll Collection Devices, and 160 of Light Yard's 164 products
+  classify as Lighting (1 before). Products already in the index keep the category they were crawled
+  with: crawl the store again (`portage index build --sources storefront_products`) to re-classify them.
+- **The local index is now a SQLite file.** `~/.portage/index/index.sqlite3` (mode 0600, WAL)
+  replaces `stores.json` and `products.json`, through the new `sqlite3` gem dependency (`~> 2.9`,
+  precompiled for macOS and Linux). An existing `stores.json`/`products.json` is imported once on
+  first use and renamed to `*.json.migrated`; it is never deleted. `Index::Store` and
+  `Index::ProductStore` keep their APIs, and `ProductStore#upsert_many` writes a batch in one
+  transaction. Index write failures now raise instead of being dropped. `portage doctor` reports the
+  database path, row counts and whether FTS5 is available. The known-stores cache and `index export`
+  output stay JSON.
+- **Storefront catalogue crawl and `portage index search`.** A new opt-in index source,
+  `storefront_products`, reads a Shopify store's public `/products.json` into the local index. Run it
+  with `portage index add URL --crawl` for one store, or `portage index build --sources
+  storefront_products` for stores already indexed (25 a run, least recently crawled first). Each
+  product is mapped through the UCP `Product` shape and stored with its handle, URL, first image,
+  options and variant ids. Price and availability are dropped before anything is written. A crawl
+  reads at most 20 pages of 250 a store, 1s apart. It waits out one 429's `Retry-After` (capped at 60s)
+  and stops on a second, and it obeys `robots.txt` for every page URL. It never contacts a hand-off-only
+  host, and it skips a 404, a redirect or a non-JSON answer (a bot wall). What happened is kept on the
+  store entry as `crawl`. `portage index search QUERY [--category ID] [--store HOST] [--limit N]
+  [--json]` searches the index locally with SQLite FTS5, ranked by bm25, and sends no request. Without
+  FTS5 it falls back to an unranked text match and reports `engine: "like"`. `index show --products` now
+  pages (`--page N`, `--per-page N`, default 50). `portage check` suggests the `index add ... --crawl`
+  command for a Shopify or native-UCP store (`index_hint`) but never runs it. `find` and `buy` never
+  crawl.
+- **Product cards for agents.** `find` offers (and `shopify_catalog` offers) gain a `product` field:
+  the store's UCP `Product` wire hash as served, with `media` cut to the first image, so an agent or
+  UI can draw a card without another request. The flat fields (`title`, `amount`, `currency`, `url`,
+  `product_id`, `store`) are unchanged, the retailer API sources leave `product` out, and `history`
+  does not save it. `portage index search` results are marked `live: false` and each hit gains
+  `product`, built from the fields the index keeps (title, handle, URL, first image, options, variant
+  ids, category), never a price. Text output says the hits are not live.
+
 ## [0.10.0] - 2026-09-29
 
 - **`portage check <url> [--json]`.** Reports whether Portage can buy from a store and

data/README.md

@@ -155,8 +155,9 @@ portage policy set [--per-transaction-cap N --currency CUR]
 portage orders reconcile [--checkout ID] [--json]
 portage index build [--sources a,b] [--queries FILE] [--dry-run] [--export DIR] [--json]
 portage index refresh [--sources a,b] [--queries FILE] [--dry-run] [--export DIR] [--json]
-portage index show [--stores|--products] [--json]
-portage index add <url> [--json]
+portage index show [--stores|--products [--page N] [--per-page N]] [--json]
+portage index search QUERY [--category ID] [--store HOST] [--limit N] [--json]
+portage index add <url> [--crawl] [--json]
 portage index remove <host> [--json]
 portage index sources [--json]
 portage browser import [--browser chrome|edge|brave|arc|firefox|safari] [--profile-root DIR]
@@ -478,13 +479,16 @@ portage index build --sources shopify_catalog,stores_file
 portage index build --queries queries.txt    # one query per line, instead of the built-in taxonomy sweep
 portage index refresh                        # re-verify entries older than 7 days, add new ones
 portage index show --stores --json
-portage index show --products --json
+portage index show --products --page 2 --json   # 50 a page; --per-page N
+portage index search "wall light" --store some-shop.example --json
 portage index add https://some-shop.example
+portage index add https://some-shop.example --crawl   # also read its /products.json catalogue
+portage index build --sources storefront_products     # crawl the catalogues of stores already indexed
 portage index remove some-shop.example
 portage index sources                        # name, what each fetches, source file path
 ```
 
-Stored at `~/.portage/index/{stores,products}.json`, **never in git** and
+Stored in `~/.portage/index/index.sqlite3` (mode 0600), **never in git** and
 **never containing a price or stock field** — those are always fetched live.
 Each new origin gets exactly one `/.well-known/ucp` probe (capped at 500 new
 probes per run), throttled, with progress output. Sources:
@@ -496,6 +500,18 @@ probes per run), throttled, with progress output. Sources:
 | `browser` | Whatever `portage browser import` (below) already saved — this source itself never reads a browser | on, but yields nothing unless you've run `browser import` |
 | `wikidata` | Retailers'/brands' official sites via a public SPARQL query | opt-in (`--sources wikidata`) |
 | `webmcp_sweep` | Which WebMCP preset an origin matches, when a bridge is attached | opt-in, needs a bridge |
+| `storefront_products` | Each indexed Shopify store's own `/products.json`: title, brand, handle, URL, first image, options and variant ids, mapped through the UCP `Product` shape with price and availability dropped | opt-in (`--sources storefront_products`, or `index add URL --crawl`) |
+
+**Catalogue crawls are polite and opt-in.** `storefront_products` never runs
+from `find` or `buy` (`portage check` only prints the `index add ... --crawl`
+command). It crawls at most 20 pages of 250 products a store and 25 stores a
+run, least recently crawled first, 1s apart. It waits out one 429's
+`Retry-After` (capped at 60s) and stops that store on a second. It obeys
+`robots.txt` for every page URL, never contacts a hand-off-only host, and
+skips a store that answers 404, a redirect or anything but products JSON (a
+bot wall). What happened is kept on the store entry as `crawl`.
+`portage index search` then searches those products locally (SQLite FTS5,
+or a plain text match if FTS5 is missing), with no request.
 
 **The index is untrusted data, on the same footing as any other `find`
 candidate.** It never feeds `Policy#merchant_allowlist` and never counts as