portage-cli 0.9.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3d34c9ba25977baefa9f1f802e38ae6b2175e455d246f7a0dd3a2833136c46e8
-  data.tar.gz: 2c1d701bafd9c914115bdb02ab661906eaddba139e0627a56bfa0cf094e71d06
+  metadata.gz: f4200c871324b9f35de00e58c32177ec087c9922b3adda2ae45aa1b4673478f1
+  data.tar.gz: e719be3b212a547be84ff5fa1232b08303c96288b7acf60f0fd9c2f1982fa0ef
 SHA512:
-  metadata.gz: ec3b6bf08d832abbe197591050b229f38a3650603feb6e94b47a6e84fbb9d3b9ddf531c51e84d71c8b708131b655e6b501e6a55662908eb6edb60981da4f14a2
-  data.tar.gz: 137b718fd1283383aa70879a6f2290a35dc1bc3f3a898ceff6965019b110329caf943dcf3ef39c04c29d1661beee3da6edd3a056877b7a6c3a932debce90c99f
+  metadata.gz: 3bb580b0b6ac745462ecc241b4e151fa16c2a8651ac32aa6d53c84aca0b910da6095182680b6a832b3431539904a697332279de9b9340fb491581821eccf6519
+  data.tar.gz: 7a75983b229489dc8519caa7a176122b00c8f41f56922f4b373c551a9ab044dace860f7ecdd6ba93d63feddf84177391d3ae92f13a6280acd138e704590ea4f3

data/CHANGELOG.md

@@ -6,6 +6,70 @@ pre-1.0, so APIs may still shift between minor versions.
 
 ## [Unreleased]
 
+## [0.11.0] - 2026-10-01
+
+- **Category classification uses the whole taxonomy.** `known-stores/categories.yml` is now generated
+  by `script/categories` (stdlib only) from Google's product taxonomy, edition 2021-09-21, instead of
+  coming from a script that was never committed. It keeps the same ids and the same top-two-level keys
+  (so `~/.portage/categories.yml` overrides still work), but a level-2 node's `keywords` now include
+  its descendants' names ("Chandeliers" counts for Lighting), and its parent's words sit in a separate
+  `parent_keywords` list. The file grows from 20KB to 101KB. A shipped stoplist
+  (`known-stores/category-stoplist.yml`, every word with its reason) removes merchandising and url words
+  ("new", "collection", "sale", "gift", "accessories", "products", ...) from both the data and the input,
+  and a small synonyms file (`category-synonyms.yml`, each word tied to a golden case) adds the few
+  words the taxonomy lacks ("pendant", "sconce"). `Classifier.categories_for` keeps its signature (plus
+  an optional `stoplist_path:`) and returns at most three ids, best first. Scoring: a keyword counts
+  2 and a `parent_keyword` 1, times how often the input repeats the word (1 + ln count) and, for the
+  cut, how rare it is. A word counts once per node, and an id scoring under half the strongest is
+  dropped. Ties go to the node whose own name the input covers most, then to one whose name has no
+  stoplisted word, then to the smaller node. On a 100-case golden set (Light Yard, JB Hi-Fi, shopper
+  queries, browser-import titles and urls) top-1 accuracy goes from 12% to 70%. "Pendant Light" is
+  Lighting, "New Collection" is no longer Toll Collection Devices, and 160 of Light Yard's 164 products
+  classify as Lighting (1 before). Products already in the index keep the category they were crawled
+  with: crawl the store again (`portage index build --sources storefront_products`) to re-classify them.
+- **The local index is now a SQLite file.** `~/.portage/index/index.sqlite3` (mode 0600, WAL)
+  replaces `stores.json` and `products.json`, through the new `sqlite3` gem dependency (`~> 2.9`,
+  precompiled for macOS and Linux). An existing `stores.json`/`products.json` is imported once on
+  first use and renamed to `*.json.migrated`; it is never deleted. `Index::Store` and
+  `Index::ProductStore` keep their APIs, and `ProductStore#upsert_many` writes a batch in one
+  transaction. Index write failures now raise instead of being dropped. `portage doctor` reports the
+  database path, row counts and whether FTS5 is available. The known-stores cache and `index export`
+  output stay JSON.
+- **Storefront catalogue crawl and `portage index search`.** A new opt-in index source,
+  `storefront_products`, reads a Shopify store's public `/products.json` into the local index. Run it
+  with `portage index add URL --crawl` for one store, or `portage index build --sources
+  storefront_products` for stores already indexed (25 a run, least recently crawled first). Each
+  product is mapped through the UCP `Product` shape and stored with its handle, URL, first image,
+  options and variant ids. Price and availability are dropped before anything is written. A crawl
+  reads at most 20 pages of 250 a store, 1s apart. It waits out one 429's `Retry-After` (capped at 60s)
+  and stops on a second, and it obeys `robots.txt` for every page URL. It never contacts a hand-off-only
+  host, and it skips a 404, a redirect or a non-JSON answer (a bot wall). What happened is kept on the
+  store entry as `crawl`. `portage index search QUERY [--category ID] [--store HOST] [--limit N]
+  [--json]` searches the index locally with SQLite FTS5, ranked by bm25, and sends no request. Without
+  FTS5 it falls back to an unranked text match and reports `engine: "like"`. `index show --products` now
+  pages (`--page N`, `--per-page N`, default 50). `portage check` suggests the `index add ... --crawl`
+  command for a Shopify or native-UCP store (`index_hint`) but never runs it. `find` and `buy` never
+  crawl.
+- **Product cards for agents.** `find` offers (and `shopify_catalog` offers) gain a `product` field:
+  the store's UCP `Product` wire hash as served, with `media` cut to the first image, so an agent or
+  UI can draw a card without another request. The flat fields (`title`, `amount`, `currency`, `url`,
+  `product_id`, `store`) are unchanged, the retailer API sources leave `product` out, and `history`
+  does not save it. `portage index search` results are marked `live: false` and each hit gains
+  `product`, built from the fields the index keeps (title, handle, URL, first image, options, variant
+  ids, category), never a price. Text output says the hits are not live.
+
+## [0.10.0] - 2026-09-29
+
+- **`portage check <url> [--json]`.** Reports whether Portage can buy from a store and
+  how: `verdict` (`automated`, `webmcp`, `handoff`, `unsupported`), `next_step`, and the
+  detail behind them (native UCP, platform, adapter install and missing env, hand-off-only,
+  WebMCP status). Wraps `Portage::Ucp::Check` and follows the precedence `buy` uses.
+  Plain GETs only, hand-off-only hosts are not contacted, and WebMCP is read only from an
+  already-open Portage profile tab. Exits `0` for `automated` and `webmcp`. The hand-off-only
+  test moved into a shared `HandoffHost` so `buy` and `check` can't disagree.
+  Follows a homepage `<link rel="ucp">` manifest pointer through `portage-ucp` 0.11.0,
+  which this release now requires (`~> 0.11`).
+
 ## [0.9.0] - 2026-09-29
 
 - **Human pick and approve** (`docs/plans/human-pick-and-approve.md`, Phases 1-3).

data/README.md

@@ -133,6 +133,7 @@ portage buy --query "..." [--store URL] [--max-price N] [--limit N] ...
 portage find --query "..." [--max-price N] [--limit N] [--json]
 portage compare <url> --product-id ID [--id VALUE ...] [--results N]
                        [--max-price N] [--json]
+portage check <url> [--json]
 portage pick [--search LAST|SEARCH_ID] [--via auto|tty|agent] [--json]
              [--choose REF | --compare REF | --view REF]
 portage approve QUOTE_ID [--via auto|tty|agent] [--relayed-yes | --view] [--json]
@@ -154,8 +155,9 @@ portage policy set [--per-transaction-cap N --currency CUR]
 portage orders reconcile [--checkout ID] [--json]
 portage index build [--sources a,b] [--queries FILE] [--dry-run] [--export DIR] [--json]
 portage index refresh [--sources a,b] [--queries FILE] [--dry-run] [--export DIR] [--json]
-portage index show [--stores|--products] [--json]
-portage index add <url> [--json]
+portage index show [--stores|--products [--page N] [--per-page N]] [--json]
+portage index search QUERY [--category ID] [--store HOST] [--limit N] [--json]
+portage index add <url> [--crawl] [--json]
 portage index remove <host> [--json]
 portage index sources [--json]
 portage browser import [--browser chrome|edge|brave|arc|firefox|safari] [--profile-root DIR]
@@ -218,6 +220,18 @@ used (an unreadable value, an out-of-range threshold) stops the buy before
 anything runs: on stderr normally, or as a JSON report with `outcome:
 "invalid_option"` under `--json`.
 
+### Check
+
+`portage check <url> [--json]` answers "can Portage buy from this store, and how?".
+It checks for a native `/.well-known/ucp` manifest, detects the platform, notes
+whether its adapter gem is installed and which env vars are missing, and looks for
+WebMCP tools, using the same hand-off-only rules as `buy`. `verdict` is
+`automated`, `webmcp`, `handoff` or `unsupported`, with a plain-English
+`next_step`. Exits `0` for `automated` and `webmcp`. It sends plain GETs only, and
+never contacts a hand-off-only host. WebMCP is read only from a tab your Portage
+browser profile already has open on the store; it never launches a browser.
+Takes the `--proxy*` flags.
+
 ### Compare
 
 `portage compare <url> --product-id ID` finds other stores selling the same
@@ -465,13 +479,16 @@ portage index build --sources shopify_catalog,stores_file
 portage index build --queries queries.txt    # one query per line, instead of the built-in taxonomy sweep
 portage index refresh                        # re-verify entries older than 7 days, add new ones
 portage index show --stores --json
-portage index show --products --json
+portage index show --products --page 2 --json   # 50 a page; --per-page N
+portage index search "wall light" --store some-shop.example --json
 portage index add https://some-shop.example
+portage index add https://some-shop.example --crawl   # also read its /products.json catalogue
+portage index build --sources storefront_products     # crawl the catalogues of stores already indexed
 portage index remove some-shop.example
 portage index sources                        # name, what each fetches, source file path
 ```
 
-Stored at `~/.portage/index/{stores,products}.json`, **never in git** and
+Stored in `~/.portage/index/index.sqlite3` (mode 0600), **never in git** and
 **never containing a price or stock field** — those are always fetched live.
 Each new origin gets exactly one `/.well-known/ucp` probe (capped at 500 new
 probes per run), throttled, with progress output. Sources:
@@ -483,6 +500,18 @@ probes per run), throttled, with progress output. Sources:
 | `browser` | Whatever `portage browser import` (below) already saved — this source itself never reads a browser | on, but yields nothing unless you've run `browser import` |
 | `wikidata` | Retailers'/brands' official sites via a public SPARQL query | opt-in (`--sources wikidata`) |
 | `webmcp_sweep` | Which WebMCP preset an origin matches, when a bridge is attached | opt-in, needs a bridge |
+| `storefront_products` | Each indexed Shopify store's own `/products.json`: title, brand, handle, URL, first image, options and variant ids, mapped through the UCP `Product` shape with price and availability dropped | opt-in (`--sources storefront_products`, or `index add URL --crawl`) |
+
+**Catalogue crawls are polite and opt-in.** `storefront_products` never runs
+from `find` or `buy` (`portage check` only prints the `index add ... --crawl`
+command). It crawls at most 20 pages of 250 products a store and 25 stores a
+run, least recently crawled first, 1s apart. It waits out one 429's
+`Retry-After` (capped at 60s) and stops that store on a second. It obeys
+`robots.txt` for every page URL, never contacts a hand-off-only host, and
+skips a store that answers 404, a redirect or anything but products JSON (a
+bot wall). What happened is kept on the store entry as `crawl`.
+`portage index search` then searches those products locally (SQLite FTS5,
+or a plain text match if FTS5 is missing), with no request.
 
 **The index is untrusted data, on the same footing as any other `find`
 candidate.** It never feeds `Policy#merchant_allowlist` and never counts as