crawlemon 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Crawlemon Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,375 @@
+# Crawlemon
+
+**Deterministic SEO checks for Next.js projects.**
+
+Crawlemon is a developer-first SEO audit and safe-fix CLI. It works like a lightweight ESLint for SEO: run it inside a Next.js repository, find issues before they reach production, and automatically repair changes that are deterministic and safe.
+
+No account, cloud connection, browser, AI provider, or API key is required.
+
+## Quick start
+
+Run an audit from the root of a Next.js project:
+
+```bash
+npx crawlemon audit
+```
+
+Preview safe automatic fixes without changing files:
+
+```bash
+npx crawlemon fix --dry-run
+```
+
+Apply the available safe fixes:
+
+```bash
+npx crawlemon fix
+```
+
+## Requirements
+
+- Node.js 20 or newer
+- A Next.js project using the App Router or Pages Router
+- Run the command from the project root
+
+Crawlemon uses static source analysis. It does not run your application, install its dependencies, or execute repository scripts.
+
+## Commands
+
+### `audit`
+
+Discover routes and run deterministic SEO checks:
+
+```bash
+npx crawlemon audit
+```
+
+The report includes:
+
+- Overall SEO score from 0 to 100
+- Technical, metadata, content, and internal-link scores
+- Errors, warnings, and informational findings
+- Route and source-file locations
+- Safe-fix availability
+
+Example:
+
+```text
+Crawlemon Audit (8 routes scanned)
+
+SEO Score: 69/100
+Technical:      51
+Metadata:       79
+Content:        85
+Internal Links: 70
+
+13 errors, 2 warnings
+11 safe fixes available
+```
+
+When the `CI` environment variable is set, `audit` exits with status code `1` if an error-level finding is present. This allows the command to act as a CI quality gate.
+
+```bash
+CI=true npx crawlemon audit
+```
+
+### `fix`
+
+Apply deterministic fixes:
+
+```bash
+npx crawlemon fix
+```
+
+Every changed file is shown as a diff. Crawlemon may:
+
+- Create `robots.txt` or `app/robots.ts`
+- Create or repair a sitemap when `siteUrl` is known
+- Add canonical metadata when the site URL and route are unambiguous
+- Repair internal-link casing when exactly one existing route matches
+
+Crawlemon will not automatically:
+
+- Rewrite marketing copy
+- Invent titles, descriptions, keywords, or image alt text
+- Change the meaning of page content
+- Delete user content
+- Execute project code
+
+### `fix --dry-run`
+
+Generate the same proposed changes and diffs without writing to disk:
+
+```bash
+npx crawlemon fix --dry-run
+```
+
+Use this command before applying fixes in an unfamiliar repository or as part of a review workflow.
+
+### `links`
+
+Inspect the internal-link graph:
+
+```bash
+npx crawlemon links
+```
+
+The report includes:
+
+- Discovered routes and internal edges
+- Broken internal links
+- Orphan pages with no incoming links
+- Dead-end and low-connectivity pages
+- Explainable internal-link recommendations
+
+Recommendations use deterministic signals such as route slugs, titles, headings, and token overlap. They do not require embeddings or an AI model.
+
+### `opportunities`
+
+Find potential content expansions from existing route patterns:
+
+```bash
+npx crawlemon opportunities
+```
+
+For example, these routes:
+
+```text
+/invoice/freelancer
+/invoice/agency
+/invoice/consultant
+```
+
+produce an `/invoice/{topic}` cluster with a small, bounded list of possible additions. Crawlemon only recommends opportunities; it never creates or publishes pages automatically.
+
+### Help and version
+
+```bash
+npx crawlemon --help
+npx crawlemon --version
+```
+
+## Zero-configuration detection
+
+Crawlemon automatically detects:
+
+- Next.js projects
+- App Router and Pages Router
+- Root-level and `src/` router directories
+- Static route structure
+- Route groups
+- Page metadata declarations
+- Titles and meta descriptions
+- Canonical URLs
+- Headings, images, and internal links
+- `robots.txt`, `app/robots.ts`, sitemap XML, and `app/sitemap.ts`
+- Static redirects declared in common `next.config` formats
+- Common JSON-LD structures
+
+Dynamic routes are discovered and audited, but they are excluded from generated sitemap entries because their final URLs cannot be inferred safely from source alone.
+
+## Configuration
+
+Configuration is optional. Create `seo.config.ts` in the project root when you need to provide a site URL, ignore routes, or change rule behavior.
+
+```ts
+export default {
+  siteUrl: "https://example.com",
+  ignore: ["/admin/**", "/api/**"],
+  rules: {
+    requireCanonical: true,
+    requireDescription: true,
+  },
+};
+```
+
+JavaScript and JSON configuration files are also supported:
+
+```text
+seo.config.ts
+seo.config.js
+seo.config.json
+```
+
+### `siteUrl`
+
+The absolute production origin for canonical and sitemap generation:
+
+```ts
+siteUrl: "https://example.com"
+```
+
+Crawlemon does not generate placeholder production URLs. Sitemap and canonical fixes that require an origin remain unavailable until a valid HTTP or HTTPS `siteUrl` is configured.
+
+### `ignore`
+
+Exclude private or non-indexable route patterns:
+
+```ts
+ignore: ["/admin/**", "/api/**", "/preview/*"]
+```
+
+### `rules`
+
+Disable an individual rule:
+
+```ts
+rules: {
+  images: false,
+  requireCanonical: false,
+}
+```
+
+Supported rule identifiers include:
+
+```text
+metadata-title
+metadata-description
+metadata-conflicts
+canonical
+headings
+images
+links
+crawlability
+structured-data
+routes
+redirects
+```
+
+## What Crawlemon checks
+
+### Metadata
+
+- Missing or empty titles
+- Suspiciously short or long titles
+- Duplicate titles
+- Missing and duplicate meta descriptions
+- Missing, malformed, duplicated, or unexpected canonical URLs
+- Conflicting metadata declaration mechanisms
+
+### Page structure and images
+
+- Missing or multiple H1 headings
+- Skipped heading levels
+- Very thin page content
+- Standard and Next.js images without `alt`
+- Empty `alt` values that may require review
+
+Crawlemon never invents image descriptions.
+
+### Links and routes
+
+- Broken internal links
+- Placeholder or malformed internal links
+- Links that create avoidable redirect hops
+- Duplicate routes
+- Redirect self-loops and redirect cycles
+- Orphan and dead-end routes
+
+### Crawlability and structured data
+
+- Missing robots and sitemap files
+- Malformed or empty XML sitemaps
+- Sitemap URLs that do not match discovered routes
+- Public routes blocked by obvious robots directives
+- Homepage `noindex`
+- Noindex pages listed in a sitemap
+- Malformed JSON-LD
+- Required fields for common supported schema types
+
+## Deterministic scoring
+
+AI never affects the SEO score.
+
+Each severity has a fixed penalty:
+
+| Severity | Penalty |
+| --- | ---: |
+| Error | 15 |
+| Warning | 6 |
+| Info | 2 |
+
+The overall score uses these category weights:
+
+| Category | Weight |
+| --- | ---: |
+| Technical | 30% |
+| Metadata | 30% |
+| Content | 20% |
+| Internal links | 20% |
+
+Repeated instances of the same rule are capped within a category. A single systemic mistake therefore remains comparable across small and large repositories.
+
+## CI examples
+
+### GitHub Actions
+
+```yaml
+name: SEO
+
+on:
+  pull_request:
+
+jobs:
+  audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npx --yes crawlemon audit
+        env:
+          CI: true
+```
+
+No application build or dependency installation is required for the static audit.
+
+## Security and privacy
+
+- Local CLI scans remain on your machine or CI runner.
+- No Crawlemon account or telemetry endpoint is required.
+- Project code is analyzed as text and is never imported or executed.
+- API keys are not required for deterministic functionality.
+- Crawlemon warns when an environment file may be tracked or missing from `.gitignore`.
+- Secret values are never printed by the secret-risk check.
+- Fixes are restricted to files inside the detected project root.
+
+For the hosted GitHub workflow and full threat model, see the project `SECURITY.md`.
+
+## Troubleshooting
+
+### No Next.js application detected
+
+Run Crawlemon from the directory containing your Next.js `package.json`, `app/`, `pages/`, or `next.config` file.
+
+```bash
+cd path/to/your-next-app
+npx crawlemon audit
+```
+
+### Canonical or sitemap fix is unavailable
+
+Add a valid production `siteUrl` to `seo.config.ts`. Crawlemon intentionally refuses to invent a domain.
+
+### A route is reported as an orphan
+
+Add a crawlable internal link from an existing page. Links rendered only after client-side runtime behavior may not be visible to static analysis.
+
+### Static analysis does not detect generated metadata
+
+Values created dynamically by `generateMetadata`, remote APIs, runtime environment variables, CMS data, or React component abstractions may require manual review. Crawlemon favors transparent static evidence and does not execute the application to guess runtime output.
+
+## GitHub pull-request automation
+
+The open-source CLI works independently. The optional Crawlemon GitHub App adds automatic pull-request regression checks, scan history, and safe fix pull requests without requiring personal access tokens.
+
+The GitHub integration is configured separately from this npm package.
+
+## Scope
+
+The current release supports Next.js only. Crawlemon is intentionally not a backlink database, rank tracker, competitor-analysis platform, or automatic content publisher.
+
+## License
+
+MIT

package/bin/autoseo.js

@@ -0,0 +1,10 @@
+#!/usr/bin/env node
+
+import { runCli } from "../dist/index.js";
+
+const args = process.argv.slice(2);
+
+runCli(args).catch((err) => {
+  console.error("\x1b[31mCrawlemon CLI Error:\x1b[0m", err?.message || err);
+  process.exit(1);
+});

package/bin/crawlemon.js

@@ -0,0 +1,10 @@
+#!/usr/bin/env node
+
+import { runCli } from "../dist/index.js";
+
+const args = process.argv.slice(2);
+
+runCli(args).catch((err) => {
+  console.error("\x1b[31mCrawlemon CLI Error:\x1b[0m", err?.message || err);
+  process.exit(1);
+});

package/dist/index.d.ts

@@ -0,0 +1 @@
+export declare function runCli(args: string[]): Promise<void>;