@krawlup/cli 0.0.0-stage → 1.2.1

package/CHANGELOG.md

@@ -0,0 +1,163 @@
+# Changelog
+
+All notable changes to `@krawlup/cli` are documented here. The format
+follows [Keep a Changelog](https://keepachangelog.com/) and the package
+uses [semantic versioning](https://semver.org/).
+
+## 1.2.1 - 2026-11-03
+
+### Added
+
+- **`--version` / `-v`** prints the CLI version and exits — read from
+  `package.json` at runtime, and the battery's User-Agent string
+  (`krawlup-cli/x.y.z`) now derives from the same source so the two can
+  never drift.
+
+## 1.2.0 - 2026-11-03
+
+### Added
+
+- **The local battery — ten zero-cost commands that run on your machine.**
+  `krawlup check` composes the whole audit (TLS, security + cache headers,
+  SPF/DKIM/DMARC/MX with the exact record to add, AI-crawler access from
+  robots.txt, JSON-LD structured-data lint, the social card with the text
+  X/Slack will show, depth-1 broken links with redirect chains and mixed
+  content, and the extractable-text lens with `--as-bot` fetching as
+  GPTBot); the spot commands (`robots`, `mail`, `ssl`, `headers`, `text`,
+  `schema`, `links`, `og`) each answer one question in seconds, and
+  `krawlup diff` snapshots a page and reports the exact line drift on every
+  later run. No account, no key, no quota — nothing touches the API. Every
+  command takes `--json`/`--md` and exits 0 clean / 1 findings / 2 usage, so
+  agents and CI can gate on them. The robots parser, the strip-to-text rule
+  and the broken-link probe are ports of the scanner's own logic, so a
+  `krawlup` verdict matches a krawlup scan verdict on the same input, and
+  every verdict is tri-state — unmeasurable reports as unknown, never as a
+  false pass or fail.
+
+- **The terminal report is now the whole report.** The free report and the
+  founder insights render every stored block through one set of shared
+  section components: AI readiness (llms.txt verdict, the per-agent robots
+  matrix, structured-data types, FAQ, Open Graph), the conversion checks,
+  the real-browser mobile layout test with capped evidence, Core Web
+  Vitals + the LCP element + speed opportunities, the named trust rows
+  (HTTPS redirect, TLS expiry, HSTS/CSP/X-Frame-Options, Safe Browsing,
+  OG tags, the X/Slack unfurl), CrUX field data, DR/OpenPageRank, the SEO
+  crawl, the 30-day screenshots as links, the cached AI First Edit card
+  and bottleneck, the site's category, and a per-check status block where
+  any check that did not succeed names itself. The founder `insights`
+  view gained the same sections (its scorecard planes never rendered —
+  the old code read a field the server never sent), and `analytics`
+  gained session duration, pages/session, exit pages, acquisition
+  channels, UTM sources and conversion goals (plan-gated server-side as
+  the dashboard gates them).
+
+- **Unlimited free scans for beta testers.** `krawlup report` now rides
+  the CLI's session bearer on the public scan endpoints. The server
+  reads it optionally: anonymous callers keep the 1-scan/month device
+  pool, while a signed-in founder on the beta allowlist scans without a
+  quota. A token is only ever sent to the API base it was minted for,
+  and a dead session degrades to the anonymous device pool.
+
+- **The free report renders everything the server actually sent.** The
+  axis rows use the site's own naming (Speed, Search & AI, Conversion)
+  with the primary bottleneck marked, and the block gains the rows the
+  web hero and Search & AI tab show: Ahrefs Authority (DR, integer
+  convention, ▲/▼ only from a real previous value), the Trust &
+  security pass counts, OpenPageRank (score · global rank · referring
+  domains), Lighthouse accessibility/best-practices/SEO, CrUX field
+  factors per device side, the five-plane scorecard breakdown, and the
+  cross-provider open findings (status-chipped, URL or provider
+  evidence as the detail line). A number the server never sent never
+  renders — rows drop out or say "Not checked" instead.
+
+### Changed
+
+- **The free-scan allowance is read from the server, not baked in.** The
+  per-device monthly pool is now an admin setting
+  (*Settings ▸ Free Site Scanning ▸ Free scans per device / month*), so
+  the CLI fetches it from `GET /api/tools/public-insights/scan-config`
+  at startup and renders that number in the help text, the menu hint,
+  the report footer and its quota refusals. A failed read degrades to
+  "checking…" instead of inventing a figure. The pool ships at `0` (off)
+  unless an admin grants one; signed-in beta testers are still told
+  "unlimited", and a device whose pool is spent is not misreported as
+  having never been granted one.
+- **Consistent vertical rhythm across the TUI.** Rendered blocks are
+  separated by one blank line, block and section headings sit on the
+  same two-space gutter as their rows, the live scan screen groups its
+  checklist, pacing bar and caption with real gaps, and the duplicate
+  spinner/esc hint above a running scan is gone.
+- **Tables size to their content** instead of stretching to fixed caps,
+  and numeric columns right-align — narrow terminals stay readable
+  instead of truncating values mid-number.
+- **The signed-out menu steps aside while a report is on screen.** The
+  free site report (or its refusal/error) renders alone with a single
+  `back` pick — or esc — to return to the menu, instead of the four
+  options crowding beneath the result.
+- **Bar lists and score stacks share one column.** Labels pad to the
+  widest label in the block (long ones ellipsize at 28 cells) so the
+  bars and numbers start at the same offset instead of staggering down
+  the screen.
+
+### Fixed
+
+- Help and the command list claimed the CLI gets 2 free scans per device
+  per month; the device cap is 1 (browsers get 2).
+- The TUI command list kept promising a "free full site scan" right next
+  to its own "(no free CLI scans right now)". The lead is now one shared
+  phrase (`reportLeadPhrase`) that the help screen and the command list
+  both build from, so they can never disagree again.
+- Both domain doors — the argv `report` command and the TUI domain input
+  — share one normalizer: `http://` is stripped like `https://`, trailing
+  slashes go, and anything that still cannot be a bare host (a path,
+  whitespace) is refused instead of sent to the server as a key.
+- A malformed TLS expiry date is excluded from the Trust & security pass
+  count instead of being scored as a failing check, and the pacing bar
+  reads its stage spans from the pacing it was handed.
+
+The first npm release: an interactive terminal client for your krawlup
+account — and it runs the free site report before any sign-in.
+
+- **Browser login (OAuth 2.1 + PKCE)** — no API keys. One consent in the
+  browser grants read / scans / listing management; refresh tokens
+  rotate silently; revoke from the dashboard or `krawlup logout`.
+- **Interactive mode** — a streaming prompt with slash commands, tab
+  completion and input history; results render into your scrollback.
+- **Live product pane** — `/open` opens a redraw-in-place tabbed view:
+  overview, insights, analytics, keywords, scan history.
+- **Everything your dashboard has, over the same data path**: portfolio
+  table, website analytics (realtime visitors, AI crawls, conversions),
+  keyword rank boards + research, rescans with budget readouts,
+  publishing, analytics snippets, excluded pages.
+- **One-shot mode** — every command works non-interactively, `--json`
+  included, so scripts and CI get the same surface.
+- **`krawlup report <domain>`** runs the exact flow the public
+  /site-report page runs — refresh → live stage polling → stored report
+  — keyed by the CLI's own device id (`~/.krawlup/device-id`), with no
+  account: the per-device pool is a server setting the CLI reads and
+  reports at run time (the web page's browser-cookie pool is its own
+  number; a shared per-IP backstop bounds every run); every report read
+  afterwards is free and unlimited. No login, no OAuth.
+- **A live scan screen** mirrors the web panel's own checklist: the six
+  real stages (queue wait → capture → Crux field data → Lighthouse →
+  scoring → history), a pacing bar that holds under 100% until the
+  report actually lands, queue position while waiting for a scan slot,
+  and `esc` to stop waiting without stopping the scan — it keeps
+  running on krawlup.com and any later `krawlup report <domain>` call
+  reads the stored result for free.
+- **The full interactive report block**: the overall score, band and
+  verdict, scorecards, the issue list with severities, broken-link
+  results, Lighthouse and Crux context; narrow terminals shrink to fit.
+  `--md` downloads the site's markdown export; `--json` prints the raw
+  payload. Every refusal (quota exhausted, listed domain, rate limit)
+  surfaces as the server's own sentence, unedited.
+- **Hardened from a full adversarial audit** — the browser actually opens
+  at login (printed-URL fallback when it cannot) and consents under the
+  server's first-party `krawlup-cli` client, forged loopback
+  callbacks are refused via the OAuth state check, scan refusals surface
+  instantly instead of polling, esc stops waiting on a running scan,
+  tables shrink to fit narrow terminals, session writes are atomic, and
+  `--json` errors print as JSON.
+- Removed: the vendored local audit engine and its CI action. All
+  scores now come from krawlup.com itself — identical to the website,
+  by construction.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nowshad Hossain Rahat
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,200 @@
-# Temporary Holding Version
+# Official CLI for krawlup — manage products, insights and analytics from the terminal, with a free no-account site battery.
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+An interactive, streaming CLI for [krawlup.com](https://krawlup.com):
+your products, scan scores, website analytics, keyword boards and
+rescans — without leaving the terminal. Sign in once with your browser
+(OAuth); everything after that is plain text at typing speed.
+
+Runs on **Node.js** (≥ 18) or **Bun** (≥ 1.2).
+
+## Quick Start (Zero Install)
+
+Run directly with your preferred package runner:
+
+```bash
+# npm / npx
+npx @krawlup/cli
+
+# Bun
+bunx @krawlup/cli
+
+# pnpm
+pnpm dlx @krawlup/cli
+
+# Yarn
+yarn dlx @krawlup/cli
+```
+
+## Global Installation
+
+Install globally to make the `krawlup` binary available everywhere:
+
+```bash
+# npm
+npm install -g @krawlup/cli
+
+# Bun
+bun install -g @krawlup/cli
+
+# pnpm
+pnpm add -g @krawlup/cli
+
+# Yarn
+yarn global add @krawlup/cli
+```
+
+Once installed, just type:
+
+```bash
+krawlup
+```
+
+## What it does
+
+`krawlup` boots to a welcome screen — the krawlup wordmark drawn as
+ASCII art from the actual brand logo — with a select menu (portfolio,
+products, open/scan a product, subscription…) that drops you straight
+into the result. From there you're at a prompt with slash commands
+(tab completes, ↑↓ recalls). Results stream into your normal
+scrollback — nothing takes over the screen.
+
+| Command | What you get |
+|---|---|
+| `/overview [range] [sort]` | portfolio table: visitors, views, revenue, scores — sort and range cycling |
+| `/report <domain>` | **FREE full site scan + report — no login needed** (see below) |
+| `/check <domain>` | **local battery — TLS, headers, mail, AI access, schema, links, OG, text (no account)** |
+| `/robots <domain>` / `/mail <domain>` | AI-crawler verdicts / email records — local, no account |
+| `/diff <url>` | page drift snapshots — local, no account |
+| `/products` | every product, public and private |
+| `/open <product>` | the live pane: redraw-in-place tabs — overview, insights, analytics, keywords, history |
+| `/insights <p>` | overall score, issues, First Edit, fix cards |
+| `/analytics <p> [range]` | totals + deltas, daily chart, top pages/referrers/countries/devices, realtime visitors, AI crawls |
+| `/keywords <p>` | rank board with trends, stored ideas with verdicts |
+| `/research <p> [seed]` | run keyword research (spends a monthly run — asks first) |
+| `/scan <p>` | trigger a full re-scan and wait, streaming progress |
+| `/add <url>` | readiness check + add a product |
+| `/publish <p>` / `/unpublish <p>` | directory visibility |
+| `/snippet <p>` | the analytics tag + excluded paths |
+| `/exclude <p> /a /b` | replace the excluded pages board (`none` clears) |
+| `/subscription` | plan, status, usage meters |
+
+Everything the terminal shows comes straight from your krawlup
+account's own reports — same numbers, no re-scoring, no drift.
+
+## The free site report (no login)
+
+```bash
+krawlup report example.com       # full scan + interactive report, nothing to sign up for
+krawlup report example.com --md  # same report as the site's markdown export
+krawlup report example.com --json
+```
+
+This is the same flow the public [/site-report](https://krawlup.com/site-report)
+page runs, keyed by the CLI's own device id (`~/.krawlup/device-id`)
+instead of a browser cookie:
+
+- **The per-device pool is a server setting, not a CLI constant.** The
+  CLI reads its allowance from the API at startup and prints whatever is
+  in force — the admin panel's
+  *Settings ▸ Free Site Scanning ▸ Free scans per device / month*, which
+  ships at `0` (no free CLI pool). The browser page's cookie pool is its
+  own number, and the shared per-IP backstop still bounds every run. No
+  account required to try; when the pool is off the refusal says so.
+- **Beta testers scan without a limit.** Sign in with `krawlup login`
+  before `krawlup report`; when your account is on the beta allowlist,
+  the CLI rides your session on the public endpoints and the server
+  waives the device quota (reads were already unlimited) — including
+  while the device pool is switched off for everyone else.
+- **Every report read is free and unlimited** — once a scan has run, the
+  stored report (and its markdown export) never costs anything again.
+- A **live scan screen** walks the real stages — queue wait → page
+  capture → real-user field data → Lighthouse → scoring → saving — with
+  a pacing bar that never claims 100% before the report exists, and
+  tesc to stop waiting while the scan itself keeps running server-side.
+- The report block is the site's own numbers: overall score, band,
+  verdict, scorecards, issues with severities, broken links, Lighthouse
+  and Crux context. Every server refusal (quota, listed domain, rate
+  limit) prints as that server's exact sentence.
+
+## Local checks (no account, no API)
+
+Ten commands that run entirely on **your machine** — no login, no key, no
+quota, nothing touches krawlup's servers. Each takes `--json` (and `--md`
+for markdown), and exits `0` when clean, `1` when a concrete finding was
+measured, `2` on a usage mistake — so scripts and agents can gate on them.
+
+```bash
+krawlup check example.com        # the full battery: TLS, headers, mail records,
+                                 # AI access, schema, social card, links, text
+krawlup check example.com --md   # same report as shareable markdown
+krawlup robots example.com --agent GPTBot --path /blog/x   # "may GPTBot read this?"
+krawlup mail example.com         # SPF / DKIM / DMARC / MX with the fix for each gap
+krawlup ssl example.com          # certificate expiry, issuer, protocol
+krawlup headers example.com      # HSTS / CSP / framing / caching with exact fixes
+krawlup text example.com         # what an LLM sees (--as-bot fetches as GPTBot)
+krawlup schema example.com       # JSON-LD lint: types + required properties
+krawlup links example.com        # broken links, redirect chains, mixed content
+krawlup og example.com/blog/post # social-card lint + the text X/Slack will show
+krawlup diff example.com         # snapshot a page; every later run reports the drift
+```
+
+What each one is for:
+
+- **check** — one command before a launch, a redesign, or a client call.
+  Everything the spot commands measure, composed, shareable.
+- **robots** — nobody else's CLI answers "is GPTBot allowed here?" before
+  an agent crawls. Same parser as the krawlup scanner, so the verdicts match.
+- **mail** — missing DMARC and SPF are the two most common real
+  deliverability failures; this names them in five seconds with the exact
+  DNS record to add.
+- **diff** — watch your own (or a competitor's) page between monthly
+  scans: first run stores a baseline in `~/.krawlup/snapshots/`, later runs
+  print added/removed lines, title changes and word deltas, exit 1 on drift.
+
+`check` needs a minute (it probes up to 40 links); the spot commands are
+seconds. Every verdict is tri-state: something that could not be measured
+(blocked fetch, DNS failure) reports as *unknown*, never as a pass or a fail.
+
+## Login
+
+`krawlup login` (or `npx @krawlup/cli login`) opens your browser: you approve the **Krawlup TUI**
+connection once (read your data, start scans, manage listings — the
+consent screen lists exactly that), and the terminal picks the tokens
+up on a localhost callback. Tokens rotate automatically and sit at
+`~/.krawlup/session.json` (mode 0600). Revoke any time from the
+dashboard's **Agent access** page, or `krawlup logout`.
+
+In CI (or anywhere a browser is awkward), `KRAWLUP_KEY` with an agent
+key from the same page works as a static escape hatch, and
+`KRAWLUP_API` moves the API base (a local dev API is exempt from the
+https-only rule).
+
+## Scriptable too
+
+Every command runs one-shot outside interactive mode. Perfect for scripts, CI pipelines, or AI agents:
+
+```bash
+# With global install:
+krawlup insights myapp.com
+krawlup analytics myapp.com 30d --json
+krawlup scan myapp.com
+krawlup report example.com        # the free one — no login needed
+
+# Or directly with npx / bunx:
+npx @krawlup/cli overview
+npx @krawlup/cli insights myapp.com
+npx @krawlup/cli analytics myapp.com 30d --json
+bunx @krawlup/cli scan myapp.com
+```
+
+Same renderers, `--json` for machines, honest exit codes (2 = not
+logged in, 1 = the server refused — its sentence is printed verbatim).
+
+## Agents
+
+Point any MCP client at krawlup's own MCP server (`https://api.krawlup.com/mcp`)
+— the terminal and your agent read the same tools.
+
+## License
+
+[MIT](./LICENSE)