codymaster 4.4.4 → 4.5.1

package/skills/marketplace-report-crawler/SKILL.md

@@ -1,176 +0,0 @@
----
-name: marketplace-report-crawler
-description: |
-  Automated marketplace report crawler for Shopee, Lazada, TikTok seller portals.
-  Uses CSV-based account config with credential security. Includes dashboard,
-  activity/bug logging, and progress reporting.
-
-  SECURITY: AI MUST NEVER read username/password columns from accounts.csv.
-  Use get-credential.js to retrieve credentials for browser-only use.
-
-  ALWAYS trigger for: marketplace report, crawl data, download marketplace report, export shopee,
-  export lazada, export tiktok, GBS data, seller report, download seller report,
-  "aggregate reports", "crawl marketplace", "data crawling", "seller data",
-  "marketplace income", "ads report", "affiliate report", "advertising report",
-  "wallet statement", "shipping fee", "shipping cost"
----
-
-# Marketplace Report Crawler v2.0
-
-## Key Principle: Agent-Native + Session Persistence
-
-> **Login happens ONCE. OTP is entered ONCE. Sessions are saved FOREVER.**
-
-Marketplaces require OTP/CAPTCHA to login. OTP codes are held by customers/accountants
-and can't be requested repeatedly. Therefore:
-
-1. **First login**: AI agent uses its OWN browser to login (bypasses CAPTCHA naturally)
-2. **Export cookies**: After login, cookies are saved to disk
-3. **Future runs**: Inject saved cookies → skip login entirely
-4. **Only re-login**: When session actually expires (rare)
-
-## 🔒 Security Rules
-
-> **CRITICAL**: These rules MUST be followed at all times.
-
-1. **NEVER** read/log/print `username`/`password` columns from `accounts.csv`
-2. **ONLY** use helper scripts to access credentials:
-   - `node scripts/get-credential.js <id> username` → raw username
-   - `node scripts/get-credential.js <id> password` → raw password
-3. Credentials go **directly** into browser form fields — never stored in context
-
-## Architecture
-
-```
-scripts/
-├── get-login-info.js    # Platform config (URL, selectors) — NO credentials
-├── get-credential.js    # Raw credential value for browser fill
-├── session-store.js     # Cookie persistence (save/check/inject/list)
-├── auto-login.js        # Automated login (agent-browser) + agent-delegate
-├── pipeline.js          # Sequential pipeline orchestrator
-├── csv-reader.js        # CSV parser with credential masking
-├── logger.js            # Activity + Bug logging
-└── run-report.js        # Progress report generator
-
-config/
-├── accounts.csv         # 🔐 Credentials (gitignored)
-├── pipeline-config.json # Platform selectors, timeouts
-└── report-types.json    # 25 report type definitions
-
-~/.marketplace-crawler/sessions/  # Persistent cookie storage
-└── <account_id>/
-    ├── cookies.json
-    └── session-meta.json
-```
-
-## Workflow: First-Time Login (One-Time)
-
-### Step 1: Get Login Info
-```bash
-node scripts/get-login-info.js zott-lazada
-# Returns: { loginUrl, platform, dashboardUrl, loginSelectors }
-```
-
-### Step 2: Login via Agent Browser
-
-Use YOUR browser tool (browser_subagent / Claude browser / Manus Chrome):
-
-1. Navigate to the `loginUrl` from step 1
-2. Get credentials:
-   ```bash
-   node scripts/get-credential.js zott-lazada username  # → nguy...@zott.vn
-   node scripts/get-credential.js zott-lazada password  # → raw password
-   ```
-3. Fill username into the login form
-4. Fill password
-5. Click login button
-6. **If CAPTCHA**: Solve it visually (your browser looks like a real user)
-7. **If OTP**: Ask user for the 6-digit code, enter it
-8. Verify you reached the dashboard
-
-### Step 3: Export & Save Cookies
-
-After successful login, run this in the browser console:
-```javascript
-// Get all cookies for the current domain
-const cookies = document.cookie.split(';').map(c => {
-  const [name, ...rest] = c.trim().split('=');
-  return { name, value: rest.join('='), domain: location.hostname, path: '/' };
-});
-JSON.stringify(cookies);
-```
-
-Save the output:
-```bash
-echo '<cookies_json>' | node scripts/session-store.js save zott-lazada agent-browser
-```
-
-### Step 4: Verify Session Saved
-```bash
-node scripts/session-store.js list
-node scripts/session-store.js check zott-lazada
-```
-
-## Workflow: Subsequent Runs (Automatic)
-
-```bash
-# 1. Check if session is still valid
-node scripts/session-store.js check zott-lazada
-
-# 2. Inject cookies and navigate to dashboard
-node scripts/session-store.js inject zott-lazada
-# → Outputs JS to set cookies in browser
-
-# 3. Run pipeline for reports
-node scripts/pipeline.js --period 202603 --platform lazada --dry-run
-node scripts/pipeline.js --period 202603 --platform lazada
-```
-
-## Auto-Login Fallback
-
-For accounts without CAPTCHA/OTP challenges, `auto-login.js` can automate login:
-
-```bash
-# Automated login via agent-browser CLI
-node scripts/auto-login.js --account zott-lazada --headed
-
-# If CAPTCHA detected → exits with code 2 + JSON with screenshot
-# Agent then takes over with its own browser
-
-# After agent solves challenge:
-node scripts/auto-login.js --account zott-lazada --verify
-```
-
-## Pipeline Commands
-
-```bash
-# Dry run — see what will happen
-node scripts/pipeline.js --period 202603 --dry-run
-
-# Run specific platform
-node scripts/pipeline.js --period 202603 --platform lazada
-
-# Run single account
-node scripts/pipeline.js --period 202603 --account zott-lazada --headed
-
-# Resume after interruption
-node scripts/pipeline.js --period 202603 --resume
-```
-
-## Report Types (25 total)
-
-| Platform | Reports | Format |
-|----------|---------|--------|
-| Shopee (7) | Income, PDF Report, Wallet, Affiliate Invoice/Product, Ads, Ads Wallet | Excel/PDF |
-| Lazada (7) | Income, PDF Report, Wallet, Ads Onsite/Product, Ads Wallet, Shipping Fee | Excel/PDF |
-| TikTok (7) | Income, Wallet, Affiliate Orders/Invoice, Ads Dashboard/Cost/Revenue | Excel |
-
-## Session Store Commands
-
-```bash
-node scripts/session-store.js save <id>           # Save cookies from stdin
-node scripts/session-store.js check <id>           # Check validity
-node scripts/session-store.js inject <id>          # Output injection JS
-node scripts/session-store.js list                 # List all sessions
-node scripts/session-store.js export-script <id>   # Browser export helper
-```

package/skills/marketplace-report-crawler/config/accounts.json

@@ -1,41 +0,0 @@
-{
-  "defaultOutputRoot": "/Users/todyle/Builder/Boxme/AutomateReport/data",
-  "accounts": [
-    {
-      "id": "vita-dairy",
-      "brand": "Vita Dairy",
-      "brandSlug": "Vita_Dairy",
-      "platforms": ["shopee", "lazada", "tiktok"],
-      "chromeProfile": "Vita Dairy",
-      "active": true,
-      "notes": "Primary brand across all platforms"
-    },
-    {
-      "id": "dutch-lady",
-      "brand": "Dutch Lady",
-      "brandSlug": "Dutch_Lady",
-      "platforms": ["shopee", "lazada", "tiktok"],
-      "chromeProfile": "Dutch Lady",
-      "active": true,
-      "notes": "Active on all 3 platforms"
-    },
-    {
-      "id": "friso",
-      "brand": "Friso",
-      "brandSlug": "Friso",
-      "platforms": ["tiktok"],
-      "chromeProfile": "Friso",
-      "active": true,
-      "notes": "TikTok only (from flow analysis)"
-    },
-    {
-      "id": "wilmar",
-      "brand": "Wilmar",
-      "brandSlug": "Wilmar",
-      "platforms": ["tiktok"],
-      "chromeProfile": "Wilmar",
-      "active": true,
-      "notes": "TikTok only (from flow analysis)"
-    }
-  ]
-}

package/skills/marketplace-report-crawler/config/report-types.json

@@ -1,422 +0,0 @@
-{
-  "reportTypes": [
-    {
-      "id": "platform_income",
-      "name": "Income Sàn",
-      "nameVi": "Thu nhập sàn",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/finance/income?type=2&dateRange=THIS_WEEK",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/K1mqjTIc1S6t",
-      "status": "active",
-      "dateSelectionType": "calendar_range",
-      "exportSteps": [
-        "Click date range dropdown → 'Chọn ngày'",
-        "Navigate calendar to target month",
-        "Select first day → last day",
-        "Click 'Xuất' button"
-      ]
-    },
-    {
-      "id": "platform_report",
-      "name": "Report Sàn (PDF)",
-      "nameVi": "Báo cáo tổng hợp thu nhập (PDF)",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/finance/income/statement",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "pdf",
-      "fileExtension": "pdf",
-      "screenshotRef": "https://prnt.sc/TvOCUH0QMxEZ",
-      "status": "active",
-      "dateSelectionType": "month_picker",
-      "exportSteps": [
-        "Select target month",
-        "Click download PDF"
-      ]
-    },
-    {
-      "id": "platform_wallet",
-      "name": "Sao kê số dư ví",
-      "nameVi": "Sao kê ví ShopeePay",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/finance/wallet/shopeepay",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/cLm_o9lJ0zFj",
-      "status": "active",
-      "dateSelectionType": "calendar_range",
-      "exportSteps": [
-        "Open date range picker",
-        "Navigate to target month",
-        "Select first day → last day",
-        "Click export"
-      ]
-    },
-    {
-      "id": "affiliate_bill",
-      "name": "Data tiếp thị liên kết theo hóa đơn",
-      "nameVi": "Dữ liệu affiliate theo hoá đơn",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/web-seller-affiliate/validation_bill",
-      "frequency": "monthly",
-      "priority": false,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/H3M6TCwnUtbY",
-      "status": "active",
-      "dateSelectionType": "date_picker",
-      "exportSteps": [
-        "Select date range",
-        "Click export"
-      ]
-    },
-    {
-      "id": "paid_ads_onsite_report",
-      "name": "Báo cáo paid ads onsite",
-      "nameVi": "Quảng cáo trả phí onsite",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/marketing/pas/index",
-      "frequency": "daily_overwrite",
-      "priority": false,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/G37bF0Qa9DB7",
-      "status": "active",
-      "dateSelectionType": "date_input",
-      "exportSteps": [
-        "Set start date input",
-        "Set end date input",
-        "Click download"
-      ]
-    },
-    {
-      "id": "affiliate_product",
-      "name": "Báo cáo tiếp thị liên kết theo Product",
-      "nameVi": "Affiliate theo sản phẩm",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/web-seller-affiliate/product_analytics",
-      "frequency": "daily_overwrite",
-      "priority": false,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/VOUR10z1kJMT",
-      "status": "active",
-      "dateSelectionType": "date_picker",
-      "exportSteps": [
-        "Select date range",
-        "Click export"
-      ]
-    },
-    {
-      "id": "ads_wallet_historical",
-      "name": "Paid ads onsite historical wallet",
-      "nameVi": "Lịch sử ví quảng cáo",
-      "platform": "shopee",
-      "url": "https://banhang.shopee.vn/portal/marketing/pas/wallet",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/R-QW1TK3_aZ2",
-      "status": "active",
-      "dateSelectionType": "calendar_range",
-      "exportSteps": [
-        "Open date range picker",
-        "Select month range",
-        "Click export"
-      ]
-    },
-    {
-      "id": "platform_income",
-      "name": "Income Sàn",
-      "nameVi": "Thu nhập sàn Lazada",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/portal/apps/finance/myIncome/index",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/uGGgnTJ4QTjx",
-      "status": "active",
-      "dateSelectionType": "next_calendar_range",
-      "exportSteps": [
-        "Click 'Chi tiết Thu nhập'",
-        "Open date picker",
-        "Navigate calendar to target month (using next-calendar-btn)",
-        "Select first day → last day",
-        "Click 'Tải xuống' → 'Chi tiết đơn hàng (Excel)'",
-        "Wait for processing → Download Result File"
-      ]
-    },
-    {
-      "id": "platform_report",
-      "name": "Report Sàn (PDF)",
-      "nameVi": "Sao kê tháng Lazada",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/portal/apps/finance/myIncome/index",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "pdf",
-      "fileExtension": "pdf",
-      "screenshotRef": "https://prnt.sc/5la14Lqd2eYJ",
-      "status": "active",
-      "dateSelectionType": "month_tab",
-      "exportSteps": [
-        "Click 'Sao kê tháng' tab",
-        "Click 'Tải xuống' → 'Tổng quan giao dịch (pdf)'",
-        "Wait → Download"
-      ]
-    },
-    {
-      "id": "platform_wallet",
-      "name": "Sao kê số dư ví",
-      "nameVi": "Sao kê ví Lazada",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/apps/balance",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/rl7NgvOjQntp",
-      "status": "active",
-      "dateSelectionType": "next_calendar_range",
-      "exportSteps": [
-        "Open date range picker",
-        "Navigate calendar to target month",
-        "Select first day → last day",
-        "Click 'Xuất dữ liệu' → 'Chi tiết giao dịch (excel)'",
-        "Wait → Download Result File"
-      ]
-    },
-    {
-      "id": "paid_ads_onsite_report",
-      "name": "Báo cáo paid ads onsite",
-      "nameVi": "Quảng cáo trả phí onsite Lazada",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/sponsor/solutions/ads/report/",
-      "frequency": "daily_overwrite",
-      "priority": false,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/2e6e9sOfHAq2",
-      "status": "active",
-      "dateSelectionType": "date_input_lazada",
-      "exportSteps": [
-        "Check 'product' checkbox",
-        "Set start date DD/MM/YYYY",
-        "Set end date DD/MM/YYYY",
-        "Click 'Tải xuống'"
-      ]
-    },
-    {
-      "id": "paid_ads_product",
-      "name": "Báo cáo paid ads onsite product",
-      "nameVi": "Quảng cáo theo sản phẩm Lazada",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/sponsor/solutions/ads/report/",
-      "frequency": "daily_overwrite",
-      "priority": false,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/2LSm7ALMhy32",
-      "status": "active",
-      "dateSelectionType": "date_input_lazada",
-      "exportSteps": [
-        "Same as paid_ads_onsite but with product filter"
-      ]
-    },
-    {
-      "id": "ads_wallet_historical",
-      "name": "Paid ads onsite historical wallet",
-      "nameVi": "Lịch sử giao dịch ví quảng cáo Lazada",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/sponsor/solutions/ads/sd-account/#!/ads/1",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/jAuuSj7gcMvX",
-      "status": "active",
-      "dateSelectionType": "rc_calendar_range",
-      "exportSteps": [
-        "Scroll down to 'Lịch Sử Giao Dịch'",
-        "Open date range picker",
-        "Navigate rc-calendar to target month",
-        "Select first → last day",
-        "Click 'Tải Xuống'"
-      ]
-    },
-    {
-      "id": "shipping_fee_details",
-      "name": "Chi tiết phí vận chuyển",
-      "nameVi": "Phí vận chuyển theo đơn hàng",
-      "platform": "lazada",
-      "url": "https://sellercenter.lazada.vn/apps/finance/logisticsfee/index?curtab=order-fee",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/u5BX1bsaaZ5S",
-      "status": "active",
-      "dateSelectionType": "multi_select_menu",
-      "exportSteps": [
-        "Open statement period dropdown",
-        "Select all dates matching target YYYY-MM",
-        "Click 'Xuất dữ liệu tổng quan đơn hàng'",
-        "Click 'Xuất dữ liệu với bộ lọc hiện tại'",
-        "Wait (long) → Download"
-      ]
-    },
-    {
-      "id": "platform_income",
-      "name": "Income Sàn",
-      "nameVi": "Thu nhập sàn TikTok",
-      "platform": "tiktok",
-      "url": "https://seller-vn.tiktok.com/finance/transactions?shop_region=VN&tab=settled_tab",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/oj8apfNx6mCH",
-      "status": "active",
-      "dateSelectionType": "core_picker",
-      "exportSteps": [
-        "Wait for 'Lịch sử xuất dữ liệu' to appear",
-        "Click 'Xuất'",
-        "Open date picker (SVG icon)",
-        "Clear existing range (SVG click)",
-        "Select first day → last day using core-picker-cell-in-view",
-        "Click OK",
-        "Click 'Xuất'",
-        "Wait for 'Đang xuất' to complete",
-        "Click 'Tải xuống'"
-      ]
-    },
-    {
-      "id": "platform_wallet",
-      "name": "Sao kê số dư ví",
-      "nameVi": "Sao kê ví TikTok",
-      "platform": "tiktok",
-      "url": "https://seller-vn.tiktok.com/finance/withdraw-new?is_new_connect=0&register_libra=&shop_region=VN",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/ESRF6bEhlbbu",
-      "status": "active",
-      "dateSelectionType": "core_picker",
-      "exportSteps": [
-        "Wait for 'Lịch sử xuất dữ liệu'",
-        "Scroll down 500px",
-        "Click 'Xuất'",
-        "Open date picker → clear → select range",
-        "Click OK → Export",
-        "Wait for completion → Download"
-      ]
-    },
-    {
-      "id": "affiliate_orders",
-      "name": "Báo cáo đơn hàng Affiliate",
-      "nameVi": "Đơn hàng affiliate TikTok",
-      "platform": "tiktok",
-      "url": "https://affiliate.tiktok.com/product/order?shop_region=VN&order_type=1",
-      "frequency": "daily_overwrite",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/4bmmSLHdm_wl",
-      "status": "active",
-      "dateSelectionType": "date_picker",
-      "exportSteps": [
-        "Select date range",
-        "Click export"
-      ]
-    },
-    {
-      "id": "ads_dashboard",
-      "name": "Ads TikTok Dashboard",
-      "nameVi": "Bảng điều khiển quảng cáo TikTok",
-      "platform": "tiktok",
-      "url": "https://seller-vn.tiktok.com/ads-creation/dashboard",
-      "frequency": "daily",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/eXGtbkNrlUQf",
-      "status": "active",
-      "dateSelectionType": "dashboard_filter",
-      "exportSteps": [
-        "Set date filter",
-        "Download dashboard data"
-      ]
-    },
-    {
-      "id": "ads_cost",
-      "name": "Báo cáo paid ads onsite - chi phí",
-      "nameVi": "Chi phí quảng cáo TikTok",
-      "platform": "tiktok",
-      "url": "https://business.tiktok.com/manage/transaction?org_id=7384265942869524481",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/AcnxB06ZZKHV",
-      "status": "in_progress",
-      "dateSelectionType": "business_center",
-      "exportSteps": [
-        "Navigate to transaction history",
-        "Set date range",
-        "Export data"
-      ]
-    },
-    {
-      "id": "ads_revenue",
-      "name": "Báo cáo paid ads onsite - doanh thu",
-      "nameVi": "Doanh thu quảng cáo TikTok",
-      "platform": "tiktok",
-      "url": "https://business.tiktok.com/manage/transaction?org_id=7384265942869524481",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/rcC_Gy47r0XT",
-      "status": "in_progress",
-      "dateSelectionType": "business_center",
-      "exportSteps": [
-        "Navigate to transaction history (revenue tab)",
-        "Set date range",
-        "Export data"
-      ]
-    },
-    {
-      "id": "affiliate_invoice",
-      "name": "Báo cáo hoá đơn affiliate",
-      "nameVi": "Hoá đơn hoa hồng liên kết TikTok",
-      "platform": "tiktok",
-      "url": "https://seller-vn.tiktok.com/finance/invoice?shop_region=VN",
-      "frequency": "monthly",
-      "priority": true,
-      "exportFormat": "excel",
-      "fileExtension": "xlsx",
-      "screenshotRef": "https://prnt.sc/W8kOcqiA6mmz",
-      "status": "active",
-      "dateSelectionType": "core_month_picker",
-      "exportSteps": [
-        "Wait for page load",
-        "Click 'Biên nhận hoa hồng liên kết'",
-        "Open month picker",
-        "Select target month (Jan-Dec)",
-        "Click OK",
-        "Check if data exists",
-        "If data: set 50/Page, get max pages, extract table per page",
-        "Save each part as Excel"
-      ]
-    }
-  ]
-}

package/skills/marketplace-report-crawler/config/sessions.json

@@ -1,3 +0,0 @@
-{
-  "sessions": []
-}