codymaster 4.1.4 → 4.4.1

package/skills/marketplace-report-crawler/SKILL.md

@@ -0,0 +1,176 @@
+---
+name: marketplace-report-crawler
+description: |
+  Automated marketplace report crawler for Shopee, Lazada, TikTok seller portals.
+  Uses CSV-based account config with credential security. Includes dashboard,
+  activity/bug logging, and progress reporting.
+
+  SECURITY: AI MUST NEVER read username/password columns from accounts.csv.
+  Use get-credential.js to retrieve credentials for browser-only use.
+
+  ALWAYS trigger for: marketplace report, crawl data, download marketplace report, export shopee,
+  export lazada, export tiktok, GBS data, seller report, download seller report,
+  "aggregate reports", "crawl marketplace", "data crawling", "seller data",
+  "marketplace income", "ads report", "affiliate report", "advertising report",
+  "wallet statement", "shipping fee", "shipping cost"
+---
+
+# Marketplace Report Crawler v2.0
+
+## Key Principle: Agent-Native + Session Persistence
+
+> **Login happens ONCE. OTP is entered ONCE. Sessions are saved FOREVER.**
+
+Marketplaces require OTP/CAPTCHA to login. OTP codes are held by customers/accountants
+and can't be requested repeatedly. Therefore:
+
+1. **First login**: AI agent uses its OWN browser to login (bypasses CAPTCHA naturally)
+2. **Export cookies**: After login, cookies are saved to disk
+3. **Future runs**: Inject saved cookies → skip login entirely
+4. **Only re-login**: When session actually expires (rare)
+
+## 🔒 Security Rules
+
+> **CRITICAL**: These rules MUST be followed at all times.
+
+1. **NEVER** read/log/print `username`/`password` columns from `accounts.csv`
+2. **ONLY** use helper scripts to access credentials:
+   - `node scripts/get-credential.js <id> username` → raw username
+   - `node scripts/get-credential.js <id> password` → raw password
+3. Credentials go **directly** into browser form fields — never stored in context
+
+## Architecture
+
+```
+scripts/
+├── get-login-info.js    # Platform config (URL, selectors) — NO credentials
+├── get-credential.js    # Raw credential value for browser fill
+├── session-store.js     # Cookie persistence (save/check/inject/list)
+├── auto-login.js        # Automated login (agent-browser) + agent-delegate
+├── pipeline.js          # Sequential pipeline orchestrator
+├── csv-reader.js        # CSV parser with credential masking
+├── logger.js            # Activity + Bug logging
+└── run-report.js        # Progress report generator
+
+config/
+├── accounts.csv         # 🔐 Credentials (gitignored)
+├── pipeline-config.json # Platform selectors, timeouts
+└── report-types.json    # 25 report type definitions
+
+~/.marketplace-crawler/sessions/  # Persistent cookie storage
+└── <account_id>/
+    ├── cookies.json
+    └── session-meta.json
+```
+
+## Workflow: First-Time Login (One-Time)
+
+### Step 1: Get Login Info
+```bash
+node scripts/get-login-info.js zott-lazada
+# Returns: { loginUrl, platform, dashboardUrl, loginSelectors }
+```
+
+### Step 2: Login via Agent Browser
+
+Use YOUR browser tool (browser_subagent / Claude browser / Manus Chrome):
+
+1. Navigate to the `loginUrl` from step 1
+2. Get credentials:
+   ```bash
+   node scripts/get-credential.js zott-lazada username  # → nguy...@zott.vn
+   node scripts/get-credential.js zott-lazada password  # → raw password
+   ```
+3. Fill username into the login form
+4. Fill password
+5. Click login button
+6. **If CAPTCHA**: Solve it visually (your browser looks like a real user)
+7. **If OTP**: Ask user for the 6-digit code, enter it
+8. Verify you reached the dashboard
+
+### Step 3: Export & Save Cookies
+
+After successful login, run this in the browser console:
+```javascript
+// Get all cookies for the current domain
+const cookies = document.cookie.split(';').map(c => {
+  const [name, ...rest] = c.trim().split('=');
+  return { name, value: rest.join('='), domain: location.hostname, path: '/' };
+});
+JSON.stringify(cookies);
+```
+
+Save the output:
+```bash
+echo '<cookies_json>' | node scripts/session-store.js save zott-lazada agent-browser
+```
+
+### Step 4: Verify Session Saved
+```bash
+node scripts/session-store.js list
+node scripts/session-store.js check zott-lazada
+```
+
+## Workflow: Subsequent Runs (Automatic)
+
+```bash
+# 1. Check if session is still valid
+node scripts/session-store.js check zott-lazada
+
+# 2. Inject cookies and navigate to dashboard
+node scripts/session-store.js inject zott-lazada
+# → Outputs JS to set cookies in browser
+
+# 3. Run pipeline for reports
+node scripts/pipeline.js --period 202603 --platform lazada --dry-run
+node scripts/pipeline.js --period 202603 --platform lazada
+```
+
+## Auto-Login Fallback
+
+For accounts without CAPTCHA/OTP challenges, `auto-login.js` can automate login:
+
+```bash
+# Automated login via agent-browser CLI
+node scripts/auto-login.js --account zott-lazada --headed
+
+# If CAPTCHA detected → exits with code 2 + JSON with screenshot
+# Agent then takes over with its own browser
+
+# After agent solves challenge:
+node scripts/auto-login.js --account zott-lazada --verify
+```
+
+## Pipeline Commands
+
+```bash
+# Dry run — see what will happen
+node scripts/pipeline.js --period 202603 --dry-run
+
+# Run specific platform
+node scripts/pipeline.js --period 202603 --platform lazada
+
+# Run single account
+node scripts/pipeline.js --period 202603 --account zott-lazada --headed
+
+# Resume after interruption
+node scripts/pipeline.js --period 202603 --resume
+```
+
+## Report Types (25 total)
+
+| Platform | Reports | Format |
+|----------|---------|--------|
+| Shopee (7) | Income, PDF Report, Wallet, Affiliate Invoice/Product, Ads, Ads Wallet | Excel/PDF |
+| Lazada (7) | Income, PDF Report, Wallet, Ads Onsite/Product, Ads Wallet, Shipping Fee | Excel/PDF |
+| TikTok (7) | Income, Wallet, Affiliate Orders/Invoice, Ads Dashboard/Cost/Revenue | Excel |
+
+## Session Store Commands
+
+```bash
+node scripts/session-store.js save <id>           # Save cookies from stdin
+node scripts/session-store.js check <id>           # Check validity
+node scripts/session-store.js inject <id>          # Output injection JS
+node scripts/session-store.js list                 # List all sessions
+node scripts/session-store.js export-script <id>   # Browser export helper
+```

package/skills/marketplace-report-crawler/config/accounts.json

@@ -0,0 +1,41 @@
+{
+  "defaultOutputRoot": "/Users/todyle/Builder/Boxme/AutomateReport/data",
+  "accounts": [
+    {
+      "id": "vita-dairy",
+      "brand": "Vita Dairy",
+      "brandSlug": "Vita_Dairy",
+      "platforms": ["shopee", "lazada", "tiktok"],
+      "chromeProfile": "Vita Dairy",
+      "active": true,
+      "notes": "Primary brand across all platforms"
+    },
+    {
+      "id": "dutch-lady",
+      "brand": "Dutch Lady",
+      "brandSlug": "Dutch_Lady",
+      "platforms": ["shopee", "lazada", "tiktok"],
+      "chromeProfile": "Dutch Lady",
+      "active": true,
+      "notes": "Active on all 3 platforms"
+    },
+    {
+      "id": "friso",
+      "brand": "Friso",
+      "brandSlug": "Friso",
+      "platforms": ["tiktok"],
+      "chromeProfile": "Friso",
+      "active": true,
+      "notes": "TikTok only (from flow analysis)"
+    },
+    {
+      "id": "wilmar",
+      "brand": "Wilmar",
+      "brandSlug": "Wilmar",
+      "platforms": ["tiktok"],
+      "chromeProfile": "Wilmar",
+      "active": true,
+      "notes": "TikTok only (from flow analysis)"
+    }
+  ]
+}

package/skills/marketplace-report-crawler/config/report-types.json

@@ -0,0 +1,422 @@
+{
+  "reportTypes": [
+    {
+      "id": "platform_income",
+      "name": "Income Sàn",
+      "nameVi": "Thu nhập sàn",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/finance/income?type=2&dateRange=THIS_WEEK",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/K1mqjTIc1S6t",
+      "status": "active",
+      "dateSelectionType": "calendar_range",
+      "exportSteps": [
+        "Click date range dropdown → 'Chọn ngày'",
+        "Navigate calendar to target month",
+        "Select first day → last day",
+        "Click 'Xuất' button"
+      ]
+    },
+    {
+      "id": "platform_report",
+      "name": "Report Sàn (PDF)",
+      "nameVi": "Báo cáo tổng hợp thu nhập (PDF)",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/finance/income/statement",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "pdf",
+      "fileExtension": "pdf",
+      "screenshotRef": "https://prnt.sc/TvOCUH0QMxEZ",
+      "status": "active",
+      "dateSelectionType": "month_picker",
+      "exportSteps": [
+        "Select target month",
+        "Click download PDF"
+      ]
+    },
+    {
+      "id": "platform_wallet",
+      "name": "Sao kê số dư ví",
+      "nameVi": "Sao kê ví ShopeePay",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/finance/wallet/shopeepay",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/cLm_o9lJ0zFj",
+      "status": "active",
+      "dateSelectionType": "calendar_range",
+      "exportSteps": [
+        "Open date range picker",
+        "Navigate to target month",
+        "Select first day → last day",
+        "Click export"
+      ]
+    },
+    {
+      "id": "affiliate_bill",
+      "name": "Data tiếp thị liên kết theo hóa đơn",
+      "nameVi": "Dữ liệu affiliate theo hoá đơn",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/web-seller-affiliate/validation_bill",
+      "frequency": "monthly",
+      "priority": false,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/H3M6TCwnUtbY",
+      "status": "active",
+      "dateSelectionType": "date_picker",
+      "exportSteps": [
+        "Select date range",
+        "Click export"
+      ]
+    },
+    {
+      "id": "paid_ads_onsite_report",
+      "name": "Báo cáo paid ads onsite",
+      "nameVi": "Quảng cáo trả phí onsite",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/marketing/pas/index",
+      "frequency": "daily_overwrite",
+      "priority": false,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/G37bF0Qa9DB7",
+      "status": "active",
+      "dateSelectionType": "date_input",
+      "exportSteps": [
+        "Set start date input",
+        "Set end date input",
+        "Click download"
+      ]
+    },
+    {
+      "id": "affiliate_product",
+      "name": "Báo cáo tiếp thị liên kết theo Product",
+      "nameVi": "Affiliate theo sản phẩm",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/web-seller-affiliate/product_analytics",
+      "frequency": "daily_overwrite",
+      "priority": false,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/VOUR10z1kJMT",
+      "status": "active",
+      "dateSelectionType": "date_picker",
+      "exportSteps": [
+        "Select date range",
+        "Click export"
+      ]
+    },
+    {
+      "id": "ads_wallet_historical",
+      "name": "Paid ads onsite historical wallet",
+      "nameVi": "Lịch sử ví quảng cáo",
+      "platform": "shopee",
+      "url": "https://banhang.shopee.vn/portal/marketing/pas/wallet",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/R-QW1TK3_aZ2",
+      "status": "active",
+      "dateSelectionType": "calendar_range",
+      "exportSteps": [
+        "Open date range picker",
+        "Select month range",
+        "Click export"
+      ]
+    },
+    {
+      "id": "platform_income",
+      "name": "Income Sàn",
+      "nameVi": "Thu nhập sàn Lazada",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/portal/apps/finance/myIncome/index",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/uGGgnTJ4QTjx",
+      "status": "active",
+      "dateSelectionType": "next_calendar_range",
+      "exportSteps": [
+        "Click 'Chi tiết Thu nhập'",
+        "Open date picker",
+        "Navigate calendar to target month (using next-calendar-btn)",
+        "Select first day → last day",
+        "Click 'Tải xuống' → 'Chi tiết đơn hàng (Excel)'",
+        "Wait for processing → Download Result File"
+      ]
+    },
+    {
+      "id": "platform_report",
+      "name": "Report Sàn (PDF)",
+      "nameVi": "Sao kê tháng Lazada",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/portal/apps/finance/myIncome/index",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "pdf",
+      "fileExtension": "pdf",
+      "screenshotRef": "https://prnt.sc/5la14Lqd2eYJ",
+      "status": "active",
+      "dateSelectionType": "month_tab",
+      "exportSteps": [
+        "Click 'Sao kê tháng' tab",
+        "Click 'Tải xuống' → 'Tổng quan giao dịch (pdf)'",
+        "Wait → Download"
+      ]
+    },
+    {
+      "id": "platform_wallet",
+      "name": "Sao kê số dư ví",
+      "nameVi": "Sao kê ví Lazada",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/apps/balance",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/rl7NgvOjQntp",
+      "status": "active",
+      "dateSelectionType": "next_calendar_range",
+      "exportSteps": [
+        "Open date range picker",
+        "Navigate calendar to target month",
+        "Select first day → last day",
+        "Click 'Xuất dữ liệu' → 'Chi tiết giao dịch (excel)'",
+        "Wait → Download Result File"
+      ]
+    },
+    {
+      "id": "paid_ads_onsite_report",
+      "name": "Báo cáo paid ads onsite",
+      "nameVi": "Quảng cáo trả phí onsite Lazada",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/sponsor/solutions/ads/report/",
+      "frequency": "daily_overwrite",
+      "priority": false,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/2e6e9sOfHAq2",
+      "status": "active",
+      "dateSelectionType": "date_input_lazada",
+      "exportSteps": [
+        "Check 'product' checkbox",
+        "Set start date DD/MM/YYYY",
+        "Set end date DD/MM/YYYY",
+        "Click 'Tải xuống'"
+      ]
+    },
+    {
+      "id": "paid_ads_product",
+      "name": "Báo cáo paid ads onsite product",
+      "nameVi": "Quảng cáo theo sản phẩm Lazada",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/sponsor/solutions/ads/report/",
+      "frequency": "daily_overwrite",
+      "priority": false,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/2LSm7ALMhy32",
+      "status": "active",
+      "dateSelectionType": "date_input_lazada",
+      "exportSteps": [
+        "Same as paid_ads_onsite but with product filter"
+      ]
+    },
+    {
+      "id": "ads_wallet_historical",
+      "name": "Paid ads onsite historical wallet",
+      "nameVi": "Lịch sử giao dịch ví quảng cáo Lazada",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/sponsor/solutions/ads/sd-account/#!/ads/1",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/jAuuSj7gcMvX",
+      "status": "active",
+      "dateSelectionType": "rc_calendar_range",
+      "exportSteps": [
+        "Scroll down to 'Lịch Sử Giao Dịch'",
+        "Open date range picker",
+        "Navigate rc-calendar to target month",
+        "Select first → last day",
+        "Click 'Tải Xuống'"
+      ]
+    },
+    {
+      "id": "shipping_fee_details",
+      "name": "Chi tiết phí vận chuyển",
+      "nameVi": "Phí vận chuyển theo đơn hàng",
+      "platform": "lazada",
+      "url": "https://sellercenter.lazada.vn/apps/finance/logisticsfee/index?curtab=order-fee",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/u5BX1bsaaZ5S",
+      "status": "active",
+      "dateSelectionType": "multi_select_menu",
+      "exportSteps": [
+        "Open statement period dropdown",
+        "Select all dates matching target YYYY-MM",
+        "Click 'Xuất dữ liệu tổng quan đơn hàng'",
+        "Click 'Xuất dữ liệu với bộ lọc hiện tại'",
+        "Wait (long) → Download"
+      ]
+    },
+    {
+      "id": "platform_income",
+      "name": "Income Sàn",
+      "nameVi": "Thu nhập sàn TikTok",
+      "platform": "tiktok",
+      "url": "https://seller-vn.tiktok.com/finance/transactions?shop_region=VN&tab=settled_tab",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/oj8apfNx6mCH",
+      "status": "active",
+      "dateSelectionType": "core_picker",
+      "exportSteps": [
+        "Wait for 'Lịch sử xuất dữ liệu' to appear",
+        "Click 'Xuất'",
+        "Open date picker (SVG icon)",
+        "Clear existing range (SVG click)",
+        "Select first day → last day using core-picker-cell-in-view",
+        "Click OK",
+        "Click 'Xuất'",
+        "Wait for 'Đang xuất' to complete",
+        "Click 'Tải xuống'"
+      ]
+    },
+    {
+      "id": "platform_wallet",
+      "name": "Sao kê số dư ví",
+      "nameVi": "Sao kê ví TikTok",
+      "platform": "tiktok",
+      "url": "https://seller-vn.tiktok.com/finance/withdraw-new?is_new_connect=0&register_libra=&shop_region=VN",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/ESRF6bEhlbbu",
+      "status": "active",
+      "dateSelectionType": "core_picker",
+      "exportSteps": [
+        "Wait for 'Lịch sử xuất dữ liệu'",
+        "Scroll down 500px",
+        "Click 'Xuất'",
+        "Open date picker → clear → select range",
+        "Click OK → Export",
+        "Wait for completion → Download"
+      ]
+    },
+    {
+      "id": "affiliate_orders",
+      "name": "Báo cáo đơn hàng Affiliate",
+      "nameVi": "Đơn hàng affiliate TikTok",
+      "platform": "tiktok",
+      "url": "https://affiliate.tiktok.com/product/order?shop_region=VN&order_type=1",
+      "frequency": "daily_overwrite",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/4bmmSLHdm_wl",
+      "status": "active",
+      "dateSelectionType": "date_picker",
+      "exportSteps": [
+        "Select date range",
+        "Click export"
+      ]
+    },
+    {
+      "id": "ads_dashboard",
+      "name": "Ads TikTok Dashboard",
+      "nameVi": "Bảng điều khiển quảng cáo TikTok",
+      "platform": "tiktok",
+      "url": "https://seller-vn.tiktok.com/ads-creation/dashboard",
+      "frequency": "daily",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/eXGtbkNrlUQf",
+      "status": "active",
+      "dateSelectionType": "dashboard_filter",
+      "exportSteps": [
+        "Set date filter",
+        "Download dashboard data"
+      ]
+    },
+    {
+      "id": "ads_cost",
+      "name": "Báo cáo paid ads onsite - chi phí",
+      "nameVi": "Chi phí quảng cáo TikTok",
+      "platform": "tiktok",
+      "url": "https://business.tiktok.com/manage/transaction?org_id=7384265942869524481",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/AcnxB06ZZKHV",
+      "status": "in_progress",
+      "dateSelectionType": "business_center",
+      "exportSteps": [
+        "Navigate to transaction history",
+        "Set date range",
+        "Export data"
+      ]
+    },
+    {
+      "id": "ads_revenue",
+      "name": "Báo cáo paid ads onsite - doanh thu",
+      "nameVi": "Doanh thu quảng cáo TikTok",
+      "platform": "tiktok",
+      "url": "https://business.tiktok.com/manage/transaction?org_id=7384265942869524481",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/rcC_Gy47r0XT",
+      "status": "in_progress",
+      "dateSelectionType": "business_center",
+      "exportSteps": [
+        "Navigate to transaction history (revenue tab)",
+        "Set date range",
+        "Export data"
+      ]
+    },
+    {
+      "id": "affiliate_invoice",
+      "name": "Báo cáo hoá đơn affiliate",
+      "nameVi": "Hoá đơn hoa hồng liên kết TikTok",
+      "platform": "tiktok",
+      "url": "https://seller-vn.tiktok.com/finance/invoice?shop_region=VN",
+      "frequency": "monthly",
+      "priority": true,
+      "exportFormat": "excel",
+      "fileExtension": "xlsx",
+      "screenshotRef": "https://prnt.sc/W8kOcqiA6mmz",
+      "status": "active",
+      "dateSelectionType": "core_month_picker",
+      "exportSteps": [
+        "Wait for page load",
+        "Click 'Biên nhận hoa hồng liên kết'",
+        "Open month picker",
+        "Select target month (Jan-Dec)",
+        "Click OK",
+        "Check if data exists",
+        "If data: set 50/Page, get max pages, extract table per page",
+        "Save each part as Excel"
+      ]
+    }
+  ]
+}

package/skills/marketplace-report-crawler/config/sessions.json

@@ -0,0 +1,3 @@
+{
+  "sessions": []
+}