amaprice 1.0.5 → 1.0.6

package/README.md

@@ -1,6 +1,11 @@
 # amaprice
 
-CLI tool to look up and track Amazon product prices.
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Node.js >=20](https://img.shields.io/badge/node-%3E%3D20-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
+[![Playwright](https://img.shields.io/badge/Playwright-Chromium-2EAD33?logo=playwright&logoColor=white)](https://playwright.dev/)
+[![Supabase](https://img.shields.io/badge/Supabase-Postgres-3ECF8E?logo=supabase&logoColor=white)](https://supabase.com/)
+
+`amaprice` is a terminal-first CLI to check Amazon prices, track products, and build shared price history automatically.
 
 ## Install
 
@@ -8,30 +13,19 @@ CLI tool to look up and track Amazon product prices.
 npm install -g amaprice
 ```
 
-## Usage
+## Quickstart
 
 ```bash
-# One-shot price lookup
-amaprice "https://www.amazon.de/dp/B0DZ5P7JD6"
-amaprice B0DZ5P7JD6
+# one-shot lookup
 amaprice price "https://www.amazon.de/dp/B0DZ5P7JD6"
-amaprice price B0DZ5P7JD6
-amaprice price
-# then paste full Amazon URL or ASIN when prompted
-
-# JSON output (for scripts / AI agents)
-amaprice price "https://www.amazon.de/dp/B0DZ5P7JD6" --json
 
-# Track a product's price over time
-amaprice track "https://www.amazon.de/dp/B0DZ5P7JD6"
-amaprice track B0DZ5P7JD6
-amaprice track
-# then paste full Amazon URL or ASIN when prompted
+# start tracking with a tier
+amaprice track B0DZ5P7JD6 --tier daily
 
-# View price history
-amaprice history B0DZ5P7JD6
+# show history
+amaprice history B0DZ5P7JD6 --limit 30
 
-# List all tracked products
+# list tracked products
 amaprice list
 ```
 
@@ -40,23 +34,154 @@ amaprice list
 | Command | Description |
 |---|---|
 | `amaprice [url\|asin]` | Shortcut for `amaprice price [url\|asin]` |
-| `amaprice price [url\|asin]` | One-shot price lookup (or prompt if omitted) |
-| `amaprice track [url\|asin]` | Track a product's price (or prompt if omitted) |
-| `amaprice history <url\|asin>` | Show price history (`--limit N`, default 30) |
-| `amaprice list` | Show all tracked products with latest price |
+| `amaprice price [url\|asin]` | One-shot lookup and silent history insert |
+| `amaprice track [url\|asin]` | Track product + current price |
+| `amaprice history <url\|asin>` | Show history (`--limit N`) |
+| `amaprice list` | List tracked products + latest price |
+| `amaprice sync --limit <n>` | Run background sync for due products |
+| `amaprice tier <url\|asin> <hourly\|daily\|weekly>` | Set tier for tracked product |
+
+All commands support `--json`.
+
+## Testing
+
+Run regression and parser tests:
+
+```bash
+npm test
+```
+
+## Tiered Background Model
+
+Each product has:
+- `tier`: `hourly`, `daily`, or `weekly`
+- `tier_mode`: `auto` or `manual`
+- `next_scrape_at`: when the worker should scrape next
+
+How tiers are determined in `auto` mode:
+- `hourly`: 2+ price changes in 48h, or >=5% change across 7 days
+- `daily`: normal active products
+- `weekly`: no observed change in 30 days
+
+Worker behavior:
+- claims due products
+- scrapes with Playwright
+- writes `price_history`
+- writes `scrape_attempts` telemetry for block/error monitoring
+- resets/backs off on failures
+- updates next run with jitter
+
+## Database Migration (Supabase)
+
+Run this SQL in Supabase SQL Editor:
+
+`supabase/migrations/20260220_add_tier_scheduler.sql`
+
+`supabase/migrations/20260220_add_scrape_attempts.sql`
+
+`supabase/migrations/20260220_add_worker_health_view.sql`
+
+`supabase/migrations/20260220_grant_worker_health_select.sql`
+
+`supabase/migrations/20260220_add_price_history_currency.sql`
+
+These migrations add tier fields, indexes, telemetry, worker health rollups, and `price_history.currency`.
+
+## Block Detection Queries
+
+Products currently failing or likely blocked:
+
+```sql
+select asin, tier, consecutive_failures, last_error, last_scraped_at, next_scrape_at
+from products
+where consecutive_failures >= 3
+   or last_error ilike '%captcha%'
+   or last_error ilike '%robot%'
+   or last_error ilike '%503%'
+order by consecutive_failures desc, next_scrape_at asc;
+```
+
+Hourly block-rate from telemetry:
+
+```sql
+select
+  date_trunc('hour', scraped_at) as hour,
+  count(*) as total,
+  sum(case when blocked_signal then 1 else 0 end) as blocked,
+  round(100.0 * sum(case when blocked_signal then 1 else 0 end) / nullif(count(*), 0), 2) as blocked_pct
+from scrape_attempts
+where scraped_at >= now() - interval '24 hours'
+group by 1
+order by 1 desc;
+```
+
+Single-row worker health view:
+
+```sql
+select * from worker_health;
+```
+
+## Local/Worker Environment
+
+Use env vars (recommended):
+
+```bash
+export SUPABASE_URL="https://<project-ref>.supabase.co"
+export SUPABASE_KEY="<anon-or-service-role-key>"
+```
+
+For production background workers, prefer the Supabase **service role key**.
 
-All commands support `--json` for machine-readable output.
+## Railway Worker Deployment
 
-If your URL contains query parameters (`?` / `&`), either wrap it in quotes or run the command without an argument and paste the full URL into the prompt.
+This repo includes:
+- `src/worker.js` (long-running loop worker)
+- `railway.json` + `Dockerfile` (Playwright-ready runtime)
 
-## Community Price Database
+Steps:
+1. Create a Railway project from this repo.
+2. Add env vars: `SUPABASE_URL`, `SUPABASE_KEY`.
+3. Optional env vars:
+   - `SYNC_INTERVAL_MINUTES=5`
+   - `SYNC_LIMIT=20`
+4. Ensure builder is Dockerfile (root `Dockerfile`).
+5. Deploy.
+6. Confirm logs show `[worker] processed=...`.
 
-amaprice contributes anonymized price data (product title, ASIN, price, and timestamp) to a shared database. This means every lookup helps build a broader price history that benefits all users — the more people use amaprice, the richer the tracking data becomes for everyone. No personal or device information is collected.
+If Railway still uses Railpack instead of Dockerfile, set builder to Dockerfile manually in Railway service settings and redeploy.
+
+One-shot run for testing:
+
+```bash
+npm run worker:once
+```
+
+## Vercel Website Deployment (`amaprice.sh`)
+
+Lean marketing site is a Next.js app in `website/`.
+
+Steps:
+1. Import the repo in Vercel.
+2. Leave the project at repo root (deployment is controlled by root `vercel.json`).
+3. Set website env vars:
+   - `NEXT_PUBLIC_SUPABASE_URL`
+   - `NEXT_PUBLIC_SUPABASE_ANON_KEY`
+4. Deploy.
+5. Add domain `amaprice.sh` in Vercel Domains and assign to this project.
+6. Set `www.amaprice.sh` redirect to `amaprice.sh`.
+
+Local website development:
+
+```bash
+cd website
+npm install
+npm run dev
+```
 
-## Requirements
+## Community Price Data
 
-- Node.js >= 18
-- Chromium is installed automatically via Playwright
+`amaprice` contributes anonymized price snapshots (title, ASIN, price, timestamp) to a shared dataset.
+No personal/device data is stored.
 
 ## License
 

package/bin/cli.js

@@ -2,7 +2,7 @@
 
 const { program } = require('commander');
 const pkg = require('../package.json');
-const KNOWN_COMMANDS = new Set(['price', 'track', 'history', 'list', 'help']);
+const KNOWN_COMMANDS = new Set(['price', 'track', 'history', 'list', 'sync', 'tier', 'help']);
 
 const userArgs = process.argv.slice(2);
 if (userArgs.length > 0) {
@@ -23,5 +23,7 @@ require('../src/commands/price')(program);
 require('../src/commands/track')(program);
 require('../src/commands/history')(program);
 require('../src/commands/list')(program);
+require('../src/commands/sync')(program);
+require('../src/commands/tier')(program);
 
 program.parse();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "amaprice",
-  "version": "1.0.5",
+  "version": "1.0.6",
   "description": "CLI tool to scrape and track Amazon product prices",
   "main": "src/scraper.js",
   "type": "commonjs",
@@ -8,6 +8,10 @@
     "amaprice": "bin/cli.js"
   },
   "scripts": {
+    "sync": "node bin/cli.js sync",
+    "worker": "node src/worker.js",
+    "worker:once": "SYNC_RUN_ONCE=1 node src/worker.js",
+    "test": "node --test",
     "postinstall": "npx --yes playwright install chromium"
   },
   "files": [
@@ -17,7 +21,7 @@
     "LICENSE"
   ],
   "engines": {
-    "node": ">=18"
+    "node": ">=20"
   },
   "keywords": [
     "amazon",

package/src/commands/list.js

@@ -16,6 +16,10 @@ module.exports = function (program) {
             title: p.title,
             url: p.url,
             domain: p.domain,
+            tier: p.tier ?? 'daily',
+            tierMode: p.tier_mode ?? 'auto',
+            active: p.is_active ?? true,
+            nextScrapeAt: p.next_scrape_at ?? null,
             latestPrice: p.latestPrice ? parseFloat(p.latestPrice.price) : null,
             currency: p.latestPrice?.currency ?? null,
             lastScraped: p.latestPrice?.scraped_at ?? null,
@@ -30,7 +34,9 @@ module.exports = function (program) {
             const price = p.latestPrice
               ? formatPrice(parseFloat(p.latestPrice.price), p.latestPrice.currency)
               : 'N/A';
-            console.log(`  ${p.asin}  ${price}  ${p.title}`);
+            const tier = p.tier || 'daily';
+            const status = p.is_active === false ? 'paused' : tier;
+            console.log(`  ${p.asin}  ${price}  [${status}]  ${p.title}`);
           }
         }
       } catch (err) {

package/src/commands/price.js

@@ -1,7 +1,8 @@
 const { normalizeAmazonInput } = require('../url');
 const { resolveCliInput } = require('../input');
 const { scrapePrice } = require('../scraper');
-const { upsertProduct, insertPrice } = require('../db');
+const { upsertProduct, insertPrice, updateProductById } = require('../db');
+const { normalizeTier, computeNextScrapeAt } = require('../tiering');
 
 module.exports = function (program) {
   program
@@ -49,6 +50,15 @@ module.exports = function (program) {
               price: result.price.numeric,
               currency: result.price.currency,
             });
+            const tier = normalizeTier(product.tier, 'daily');
+            await updateProductById(product.id, {
+              last_price: result.price.numeric,
+              last_scraped_at: new Date().toISOString(),
+              consecutive_failures: 0,
+              last_error: null,
+              next_scrape_at: computeNextScrapeAt(tier),
+              last_price_change_at: new Date().toISOString(),
+            });
           } catch {
             // Silent — don't disrupt the user experience
           }

package/src/commands/sync.js

@@ -0,0 +1,38 @@
+const { runDueSync } = require('../sync-runner');
+
+module.exports = function (program) {
+  program
+    .command('sync')
+    .description('Run background sync for due products (for cron/worker usage)')
+    .option('--limit <n>', 'Max products to process in one run', '20')
+    .option('--json', 'Output as JSON')
+    .action(async (opts) => {
+      const limit = Math.max(1, parseInt(opts.limit, 10) || 20);
+
+      try {
+        const report = await runDueSync({ limit });
+        if (opts.json) {
+          console.log(JSON.stringify(report));
+        } else {
+          if (report.processed === 0) {
+            console.log('No due products.');
+            return;
+          }
+
+          console.log(`Processed: ${report.processed}`);
+          console.log(`Success:   ${report.success}`);
+          console.log(`Failed:    ${report.failed}`);
+          for (const item of report.items) {
+            if (item.status === 'ok') {
+              console.log(`  OK    ${item.asin}  ${item.price} ${item.currency}  tier=${item.tier}`);
+            } else {
+              console.log(`  FAIL  ${item.asin}  tier=${item.tier}  ${item.error}`);
+            }
+          }
+        }
+      } catch (err) {
+        console.error(`Error: ${err.message}`);
+        process.exit(1);
+      }
+    });
+};

package/src/commands/tier.js

@@ -0,0 +1,75 @@
+const { extractAsin } = require('../url');
+const { getProductByAsin, updateProductByAsin } = require('../db');
+const { normalizeTier, computeNextScrapeAt } = require('../tiering');
+
+module.exports = function (program) {
+  program
+    .command('tier <url-or-asin> <tier>')
+    .description('Set polling tier for a tracked product (hourly|daily|weekly)')
+    .option('--auto', 'Enable automatic re-tiering based on price behavior')
+    .option('--manual', 'Keep this tier fixed')
+    .option('--activate', 'Enable background sync for this product')
+    .option('--deactivate', 'Disable background sync for this product')
+    .option('--json', 'Output as JSON')
+    .action(async (urlOrAsin, tierArg, opts) => {
+      const asin = extractAsin(urlOrAsin);
+      if (!asin) {
+        console.error('Error: Could not extract ASIN from input.');
+        process.exit(1);
+      }
+
+      const tier = normalizeTier(tierArg);
+      if (!tier) {
+        console.error('Error: Tier must be one of: hourly, daily, weekly.');
+        process.exit(1);
+      }
+
+      if (opts.activate && opts.deactivate) {
+        console.error('Error: Use either --activate or --deactivate, not both.');
+        process.exit(1);
+      }
+
+      try {
+        const product = await getProductByAsin(asin);
+        if (!product) {
+          console.error(`Error: Product with ASIN ${asin} is not tracked yet.`);
+          process.exit(1);
+        }
+
+        let tierMode = product.tier_mode || 'auto';
+        if (opts.auto) tierMode = 'auto';
+        if (opts.manual) tierMode = 'manual';
+        if (!opts.auto && !opts.manual) tierMode = 'manual';
+
+        const patch = {
+          tier,
+          tier_mode: tierMode,
+          next_scrape_at: computeNextScrapeAt(tier),
+        };
+        if (opts.activate) patch.is_active = true;
+        if (opts.deactivate) patch.is_active = false;
+
+        const updated = await updateProductByAsin(asin, patch);
+
+        if (opts.json) {
+          console.log(JSON.stringify({
+            asin: updated.asin,
+            tier: updated.tier,
+            tierMode: updated.tier_mode,
+            active: updated.is_active,
+            nextScrapeAt: updated.next_scrape_at,
+          }));
+        } else {
+          console.log(`ASIN:        ${updated.asin}`);
+          console.log(`Tier:        ${updated.tier}`);
+          console.log(`Tier mode:   ${updated.tier_mode}`);
+          console.log(`Active:      ${updated.is_active ? 'yes' : 'no'}`);
+          console.log(`Next scrape: ${updated.next_scrape_at}`);
+        }
+      } catch (err) {
+        console.error(`Error: ${err.message}`);
+        process.exit(1);
+      }
+    });
+};
+

package/src/commands/track.js

@@ -1,12 +1,17 @@
 const { normalizeAmazonInput } = require('../url');
 const { resolveCliInput } = require('../input');
 const { scrapePrice } = require('../scraper');
-const { upsertProduct, insertPrice } = require('../db');
+const { upsertProduct, insertPrice, updateProductById } = require('../db');
+const { normalizeTier, computeNextScrapeAt } = require('../tiering');
 
 module.exports = function (program) {
   program
     .command('track [input...]')
     .description('Save product + current price to Supabase')
+    .option('--tier <tier>', 'Set polling tier: hourly|daily|weekly')
+    .option('--manual-tier', 'Pin this product to its current tier (disable auto-tier)')
+    .option('--auto-tier', 'Enable automatic tiering for this product')
+    .option('--inactive', 'Track product but do not include it in background sync')
     .option('--json', 'Output as JSON')
     .action(async (inputParts, opts) => {
       const input = await resolveCliInput(inputParts);
@@ -16,6 +21,16 @@ module.exports = function (program) {
         process.exit(1);
       }
 
+      const selectedTier = opts.tier ? normalizeTier(opts.tier) : undefined;
+      if (opts.tier && !selectedTier) {
+        console.error('Error: Tier must be one of: hourly, daily, weekly.');
+        process.exit(1);
+      }
+      if (opts.manualTier && opts.autoTier) {
+        console.error('Error: Use either --manual-tier or --auto-tier, not both.');
+        process.exit(1);
+      }
+
       try {
         const result = await scrapePrice(normalized.url);
 
@@ -29,6 +44,10 @@ module.exports = function (program) {
           title: result.title,
           url: result.url,
           domain: result.domain,
+          tier: selectedTier,
+          tierMode: opts.manualTier ? 'manual' : (opts.autoTier ? 'auto' : undefined),
+          isActive: opts.inactive ? false : undefined,
+          nextScrapeAt: selectedTier ? computeNextScrapeAt(selectedTier) : undefined,
         });
 
         const priceRecord = await insertPrice({
@@ -37,6 +56,20 @@ module.exports = function (program) {
           currency: result.price.currency,
         });
 
+        const nextTier = normalizeTier(product.tier, selectedTier || 'daily');
+        try {
+          await updateProductById(product.id, {
+            last_price: result.price.numeric,
+            last_scraped_at: priceRecord.scraped_at,
+            consecutive_failures: 0,
+            last_error: null,
+            next_scrape_at: computeNextScrapeAt(nextTier),
+            last_price_change_at: priceRecord.scraped_at,
+          });
+        } catch {
+          // Background scheduling fields may not exist before migration.
+        }
+
         if (opts.json) {
           console.log(JSON.stringify({
             product: result.title,
@@ -47,11 +80,15 @@ module.exports = function (program) {
             productId: product.id,
             priceRecordId: priceRecord.id,
             trackedAt: priceRecord.scraped_at,
+            tier: nextTier,
+            tierMode: opts.manualTier ? 'manual' : (opts.autoTier ? 'auto' : (product.tier_mode || 'auto')),
+            active: opts.inactive ? false : (product.is_active ?? true),
           }));
         } else {
           console.log(`Tracking: ${result.title}`);
           console.log(`ASIN:     ${result.asin}`);
           console.log(`Price:    ${result.priceRaw}`);
+          console.log(`Tier:     ${nextTier}`);
           console.log(`Saved to Supabase.`);
         }
       } catch (err) {

package/src/config.js

@@ -1,4 +1,7 @@
-const SUPABASE_URL = 'https://fetgmcukbeetwdahrkhe.supabase.co';
-const SUPABASE_KEY = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImZldGdtY3VrYmVldHdkYWhya2hlIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzE2MTQ2MTQsImV4cCI6MjA4NzE5MDYxNH0.KOymOB5I05eO_MMXyVHkQ2PukXkDIbFVKmukOI71r4Y';
+const DEFAULT_SUPABASE_URL = 'https://fetgmcukbeetwdahrkhe.supabase.co';
+const DEFAULT_SUPABASE_KEY = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImZldGdtY3VrYmVldHdkYWhya2hlIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzE2MTQ2MTQsImV4cCI6MjA4NzE5MDYxNH0.KOymOB5I05eO_MMXyVHkQ2PukXkDIbFVKmukOI71r4Y';
+
+const SUPABASE_URL = process.env.SUPABASE_URL || DEFAULT_SUPABASE_URL;
+const SUPABASE_KEY = process.env.SUPABASE_KEY || process.env.SUPABASE_ANON_KEY || DEFAULT_SUPABASE_KEY;
 
 module.exports = { SUPABASE_URL, SUPABASE_KEY };