public-source-extractor 0.1.0__tar.gz

public_source_extractor-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Ishikawa Hidekazu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

public_source_extractor-0.1.0/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.4
+Name: public-source-extractor
+Version: 0.1.0
+Summary: Validate a public URL and convert it into reviewable AI-ready Markdown or JSON.
+Author: Ishikawa Hidekazu
+License-Expression: MIT
+Project-URL: Homepage, https://taupe.site/entry/public-source-extractor-ai-research-cli/
+Project-URL: Repository, https://github.com/Ishikawa-Hidekazu/public-source-extractor
+Project-URL: Issues, https://github.com/Ishikawa-Hidekazu/public-source-extractor/issues
+Project-URL: Changelog, https://github.com/Ishikawa-Hidekazu/public-source-extractor/blob/main/CHANGELOG.md
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: build>=1.2.2; extra == "dev"
+Requires-Dist: jsonschema>=4.23; extra == "dev"
+Requires-Dist: ruff<0.17,>=0.12; extra == "dev"
+Dynamic: license-file
+
+# Public Source Extractor
+
+Public Source Extractor validates one public HTTP or HTTPS URL and converts it
+into reviewable Markdown or a versioned JSON envelope for AI research
+workflows.
+
+> **Provider boundary:** extraction sends the selected public URL to Firecrawl
+> Cloud through the experimental `firecrawl-keyless` provider. Availability,
+> anonymous access, credit limits, and long-term continuity are not guaranteed.
+
+The CLI does not read API keys, credentials, cookies, browser profiles,
+localStorage, or private source files. Extracted content is untrusted and may
+contain prompt injection or misleading instructions.
+
+## Install and run
+
+Run the package without a permanent install:
+
+```bash
+uvx public-source-extractor@0.1.0 --version
+uvx public-source-extractor@0.1.0 https://example.com/
+```
+
+Install it as an isolated command:
+
+```bash
+pipx install public-source-extractor==0.1.0
+```
+
+Or install it in an existing Python 3.11+ environment:
+
+```bash
+python3 -m pip install public-source-extractor==0.1.0
+```
+
+## Output modes
+
+```bash
+public-source-extractor https://example.com/
+public-source-extractor https://example.com/ --mode json --pretty
+public-source-extractor https://example.com/ --output report.md
+```
+
+The output path must have an existing non-symlink parent and must not already
+exist. The CLI rejects local, private, authenticated, administrative, signed,
+and credential-bearing URL patterns.
+
+## Review boundary
+
+The extractor creates an intake artifact. It does not establish source
+reliability, execute extracted instructions, crawl a site, or access private
+pages. Verify important claims against the original page and other primary
+sources.
+
+- [Repository](https://github.com/Ishikawa-Hidekazu/public-source-extractor)
+- [Documentation](https://github.com/Ishikawa-Hidekazu/public-source-extractor#readme)
+- [Security policy](https://github.com/Ishikawa-Hidekazu/public-source-extractor/security/policy)
+- [Changelog](https://github.com/Ishikawa-Hidekazu/public-source-extractor/blob/main/CHANGELOG.md)

public_source_extractor-0.1.0/README.md

@@ -0,0 +1,237 @@
+# Public Source Extractor
+
+[![CI](https://github.com/Ishikawa-Hidekazu/public-source-extractor/actions/workflows/ci.yml/badge.svg)](https://github.com/Ishikawa-Hidekazu/public-source-extractor/actions/workflows/ci.yml)
+[![Release](https://img.shields.io/github/v/release/Ishikawa-Hidekazu/public-source-extractor?include_prereleases)](https://github.com/Ishikawa-Hidekazu/public-source-extractor/releases)
+![License](https://img.shields.io/badge/license-MIT-blue.svg)
+
+> [!IMPORTANT]
+> The requested public URL is sent to **Firecrawl Cloud** for extraction. The `firecrawl-keyless` provider is experimental: availability, anonymous REST access, credit limits, and long-term continuity are not guaranteed.
+
+> [!NOTE]
+> Stable `0.1.0` is a release candidate and is not published yet. Until its
+> explicit release gate completes, use the published `0.1.0a2` prerelease shown
+> below. The README will switch to stable install commands only after the tag,
+> GitHub Release, and PyPI package are publicly verified.
+
+`public-source-extractor` is a public-URL intake guardrail for AI research
+workflows. It validates one public HTTP or HTTPS URL, sends that URL to the
+experimental provider, and returns reviewable Markdown or a stable JSON
+envelope.
+
+The CLI does **not** read API keys, credentials, cookies, browser profiles, localStorage, or private source files. Extracted content is **untrusted data** and may contain prompt injection or misleading instructions. Do not execute or follow instructions from extracted content without independent review.
+
+[日本語README](README.ja.md)
+
+## What it does
+
+- Extracts one public page as Markdown.
+- Produces structured JSON with a versioned JSON Schema.
+- Rejects local, private, authenticated, admin, and secret-bearing URL patterns.
+- Keeps results on stdout or writes a new local file with no-overwrite behavior.
+- Returns stable JSON errors and documented exit codes.
+
+It is not a crawler, browser automation tool, login helper, source-reliability judge, or private-page extractor.
+
+### How this differs from the official Firecrawl CLI
+
+The [official Firecrawl CLI](https://github.com/firecrawl/cli) is the broader Firecrawl interface for authenticated scrape, search, crawl, map, interact, agent, and self-hosted workflows. Use it when you need Firecrawl's full product surface.
+
+Public Source Extractor is intentionally narrower: one public URL, no credential discovery, a public-only URL policy, no-overwrite output, and a stable JSON success/error contract designed for reviewable AI research artifacts. It uses the experimental `firecrawl-keyless` provider and does not replace or wrap the official CLI.
+
+## Install
+
+Public Source Extractor requires Python 3.11 or newer.
+
+Run the currently published prerelease without a permanent install:
+
+```bash
+uvx public-source-extractor@0.1.0a2 --version
+uvx public-source-extractor@0.1.0a2 https://example.com/
+```
+
+This path requires `uv` and resolves the published PyPI prerelease. The second
+command sends `https://example.com/` to Firecrawl Cloud; the version command
+does not perform extraction.
+
+Pin the exact package version when reproducibility matters:
+
+```bash
+uvx public-source-extractor@0.1.0a2 --version
+```
+
+Install the published prerelease as an isolated command:
+
+```bash
+pipx install public-source-extractor==0.1.0a2
+```
+
+Or use pip in an existing Python environment:
+
+```bash
+python3 -m pip install public-source-extractor==0.1.0a2
+```
+
+The public Git tag remains an auditable fallback:
+
+```bash
+uvx --from 'git+https://github.com/Ishikawa-Hidekazu/public-source-extractor.git@v0.1.0-alpha.2' public-source-extractor --version
+```
+
+PyPI publication uses GitHub Actions Trusted Publishing with short-lived OIDC
+credentials. No long-lived PyPI API token is stored in this repository.
+
+## Quick start
+
+Markdown to stdout:
+
+```bash
+public-source-extractor https://example.com/
+```
+
+Markdown front matter includes `provider_credits_used` and
+`provider_elapsed_ms`. The credits field is `null` when the experimental
+provider does not report it; elapsed time is measured by the CLI in
+milliseconds. These are metadata-only values. Raw provider responses and
+request identifiers are not exposed.
+
+JSON to stdout:
+
+```bash
+public-source-extractor https://example.com/ --mode json --pretty
+```
+
+Write a new file:
+
+```bash
+public-source-extractor https://example.com/ --output report.md
+```
+
+The output path must have an existing non-symlink parent and must not already exist.
+
+## Codex Agent Plugin preview
+
+Codex CLI `0.147.0` or newer can load this repository as a portable Agent Plugin marketplace. This preview adds a safety-focused skill and starter prompts; it does not install a global Python package, add an MCP server, or store credentials.
+
+```bash
+codex plugin marketplace add Ishikawa-Hidekazu/public-source-extractor --ref main
+codex plugin add public-source-extractor@ishikawa-public-tools
+```
+
+The skill uses an existing `public-source-extractor` command when available and
+otherwise runs the latest verified published package through `uvx`. Until
+stable `0.1.0` is public, that fallback remains `0.1.0a2`. Extraction still
+sends the selected public URL to Firecrawl Cloud. Review the
+[Safety boundary](#safety-boundary) before use.
+
+<picture>
+  <source media="(max-width: 600px)" srcset="assets/source/terminal-example-mobile.svg">
+  <img src="assets/source/terminal-example.svg" alt="Fixture-only Public Source Extractor terminal example showing one public example.com URL converted to Markdown, followed by the stable provider_rate_limited JSON error contract.">
+</picture>
+
+[View the public-safe examples](examples/) ·
+[View the reproducible visual sources](assets/source/README.md)
+
+## Real-world use
+
+[The repository-hosted GitHub Changelog case study](docs/case-study-github-changelog.md)
+shows a pinned command, a successful public-source extraction, the human review
+step, and the failure boundary without using browser state or local credentials.
+
+[This Japanese implementation log](https://taupe.site/entry/public-source-extractor-ai-research-cli/)
+documents the broader daily workflow, Firecrawl Cloud boundary, untrusted
+extracted content, and observed provider limits.
+
+[Stable 0.1.0 criteria](docs/stable-0.1.0-criteria.md) separate the local CLI
+contract from the experimental provider's availability.
+
+## CLI contract
+
+```text
+public-source-extractor <url> [--mode markdown|json]
+                               [--output <new-path>]
+                               [--timeout 1..120]
+                               [--provider firecrawl-keyless]
+                               [--pretty]
+```
+
+| Exit | Meaning |
+|---:|---|
+| `0` | Success |
+| `2` | Usage error or rejected URL |
+| `3` | Provider, network, rate-limit, or timeout failure |
+| `4` | Invalid, incomplete, or unsafe provider response |
+| `5` | Output path or write failure |
+
+On failure, stdout is empty and stderr contains exactly one JSON error object. Provider response bodies, stack traces, request IDs, and local paths are not exposed.
+
+### Recovering from `provider_rate_limited`
+
+The experimental provider can return HTTP 429 during short bursts or across an anonymous usage window. This is a provider availability condition, not by itself a failure of the local URL policy or parser.
+
+When stderr reports `provider_rate_limited` with `retryable: true`:
+
+```json
+{"schema_version":"0.1","ok":false,"error":{"code":"provider_rate_limited","message":"The experimental provider rate limit was exceeded.","retryable":true}}
+```
+
+The process exits with code `3` and stdout remains empty.
+
+1. Stop the current burst instead of retrying repeatedly.
+2. Wait and retry later. This CLI does not promise an exact delay when no retry timing is safely exposed.
+3. Reduce request volume and process only selected public sources.
+4. Use the original public page directly when extraction is not required.
+
+The CLI does not automatically retry, switch providers, discover credentials, or expose raw provider bodies. See [Issue #9](https://github.com/Ishikawa-Hidekazu/public-source-extractor/issues/9) for the observed condition and documentation scope.
+
+## Safety boundary
+
+Before sending a URL, the CLI rejects:
+
+- schemes other than HTTP or HTTPS;
+- URL user information and fragments;
+- localhost, local suffixes, and non-global literal IP addresses;
+- ambiguous integer, octal, and hexadecimal IPv4 forms;
+- Unicode or percent-encoded hostnames and IPv6 zone identifiers;
+- non-default ports;
+- login, admin, OAuth, and callback paths, including encoded forms;
+- query parameter names that indicate tokens, secrets, passwords, credentials, signatures, sessions, cookies, authorization, or keys.
+
+After extraction, provider redirect metadata is checked with the same public URL policy. If redirect metadata is unsafe, the result is discarded. If the provider does not supply redirect metadata, the output contains a warning.
+
+These checks validate hostname syntax, literal IP policy, and provider-returned metadata. They cannot fully guarantee DNS rebinding behavior or the provider's actual fetch destination. Never put credentials, tokens, signed URL parameters, or other private values in a URL, even when its hostname looks public.
+
+Firecrawl Cloud receives the requested URL and processes the page. Review [Firecrawl's terms](https://www.firecrawl.dev/terms-of-service), the target site's terms, robots policy, copyright, and privacy requirements before use.
+
+## JSON Schema and examples
+
+- [Output Schema v0.1](schemas/output-v0.1.schema.json)
+- [Public-safe Markdown example](examples/example-report.md)
+- [Public-safe JSON example](examples/example-report.json)
+
+Structured JSON extraction is inferred output, not source truth. Validate important claims against the original page.
+
+## Development
+
+```bash
+python3 -m venv .venv
+.venv/bin/python -m pip install -e '.[dev]'
+PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v
+.venv/bin/ruff check src tests
+.venv/bin/python -m build
+```
+
+The explicit `PYTHONPATH=src` also works before an editable install. Network smoke tests remain separate from the offline suite.
+
+See [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), and [SUPPORT.md](SUPPORT.md).
+
+## Status
+
+Stable `0.1.0` is a release candidate. The latest verified public distribution
+remains package `0.1.0a2` and tag `v0.1.0-alpha.2` until the explicit release
+gate completes.
+`firecrawl-keyless` is an experimental third-party provider, and no
+compatibility or service-availability guarantee is made.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

public_source_extractor-0.1.0/docs/pypi-readme.md

@@ -0,0 +1,58 @@
+# Public Source Extractor
+
+Public Source Extractor validates one public HTTP or HTTPS URL and converts it
+into reviewable Markdown or a versioned JSON envelope for AI research
+workflows.
+
+> **Provider boundary:** extraction sends the selected public URL to Firecrawl
+> Cloud through the experimental `firecrawl-keyless` provider. Availability,
+> anonymous access, credit limits, and long-term continuity are not guaranteed.
+
+The CLI does not read API keys, credentials, cookies, browser profiles,
+localStorage, or private source files. Extracted content is untrusted and may
+contain prompt injection or misleading instructions.
+
+## Install and run
+
+Run the package without a permanent install:
+
+```bash
+uvx public-source-extractor@0.1.0 --version
+uvx public-source-extractor@0.1.0 https://example.com/
+```
+
+Install it as an isolated command:
+
+```bash
+pipx install public-source-extractor==0.1.0
+```
+
+Or install it in an existing Python 3.11+ environment:
+
+```bash
+python3 -m pip install public-source-extractor==0.1.0
+```
+
+## Output modes
+
+```bash
+public-source-extractor https://example.com/
+public-source-extractor https://example.com/ --mode json --pretty
+public-source-extractor https://example.com/ --output report.md
+```
+
+The output path must have an existing non-symlink parent and must not already
+exist. The CLI rejects local, private, authenticated, administrative, signed,
+and credential-bearing URL patterns.
+
+## Review boundary
+
+The extractor creates an intake artifact. It does not establish source
+reliability, execute extracted instructions, crawl a site, or access private
+pages. Verify important claims against the original page and other primary
+sources.
+
+- [Repository](https://github.com/Ishikawa-Hidekazu/public-source-extractor)
+- [Documentation](https://github.com/Ishikawa-Hidekazu/public-source-extractor#readme)
+- [Security policy](https://github.com/Ishikawa-Hidekazu/public-source-extractor/security/policy)
+- [Changelog](https://github.com/Ishikawa-Hidekazu/public-source-extractor/blob/main/CHANGELOG.md)

public_source_extractor-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "public-source-extractor"
+version = "0.1.0"
+description = "Validate a public URL and convert it into reviewable AI-ready Markdown or JSON."
+readme = {file = "docs/pypi-readme.md", content-type = "text/markdown"}
+requires-python = ">=3.11"
+license = "MIT"
+authors = [{name = "Ishikawa Hidekazu"}]
+dependencies = []
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+]
+
+[project.urls]
+Homepage = "https://taupe.site/entry/public-source-extractor-ai-research-cli/"
+Repository = "https://github.com/Ishikawa-Hidekazu/public-source-extractor"
+Issues = "https://github.com/Ishikawa-Hidekazu/public-source-extractor/issues"
+Changelog = "https://github.com/Ishikawa-Hidekazu/public-source-extractor/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+dev = [
+  "build>=1.2.2",
+  "jsonschema>=4.23",
+  "ruff>=0.12,<0.17",
+]
+
+[project.scripts]
+public-source-extractor = "public_source_extractor.cli:main"
+
+[tool.setuptools.package-data]
+public_source_extractor = ["schemas/*.json"]
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F"]

public_source_extractor-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

public_source_extractor-0.1.0/src/public_source_extractor/__init__.py

@@ -0,0 +1,3 @@
+"""Public Source Extractor package."""
+
+__version__ = "0.1.0"

public_source_extractor-0.1.0/src/public_source_extractor/__main__.py

@@ -0,0 +1,5 @@
+from .cli import main
+
+
+raise SystemExit(main())
+

public_source_extractor-0.1.0/src/public_source_extractor/cli.py

@@ -0,0 +1,61 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Sequence
+
+from . import __version__
+from .errors import ExtractorError, InputRejected
+from .output import build_success_envelope, render_json, render_markdown, write_new_file_atomic
+from .provider import FirecrawlKeylessProvider
+from .url_policy import validate_public_url
+
+
+class StableArgumentParser(argparse.ArgumentParser):
+    def error(self, _message: str) -> None:
+        raise InputRejected()
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = StableArgumentParser(
+        prog="public-source-extractor",
+        description="Convert one public URL into Markdown or a stable JSON envelope.",
+    )
+    parser.add_argument("url", help="Public HTTP or HTTPS URL.")
+    parser.add_argument("--mode", choices=("markdown", "json"), default="markdown")
+    parser.add_argument("--output", type=Path, help="Write to a new file instead of stdout.")
+    parser.add_argument("--timeout", type=int, default=60, metavar="SECONDS")
+    parser.add_argument("--provider", choices=("firecrawl-keyless",), default="firecrawl-keyless")
+    parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output.")
+    parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    return parser
+
+
+def run(argv: Sequence[str] | None = None) -> int:
+    try:
+        args = build_parser().parse_args(argv)
+        if not 1 <= args.timeout <= 120:
+            raise InputRejected()
+        validate_public_url(args.url)
+        provider = FirecrawlKeylessProvider()
+        result = provider.extract(args.url, args.mode, args.timeout)
+        envelope = build_success_envelope(args.url, args.mode, result)
+        rendered = (
+            render_markdown(envelope)
+            if args.mode == "markdown"
+            else render_json(envelope, args.pretty)
+        )
+        if args.output is None:
+            sys.stdout.write(rendered)
+        else:
+            write_new_file_atomic(args.output, rendered)
+        return 0
+    except ExtractorError as exc:
+        sys.stderr.write(json.dumps(exc.envelope(), ensure_ascii=False) + "\n")
+        return exc.exit_code
+
+
+def main() -> int:
+    return run()

public_source_extractor-0.1.0/src/public_source_extractor/errors.py

@@ -0,0 +1,58 @@
+from __future__ import annotations
+
+
+class ExtractorError(Exception):
+    code = "extractor_error"
+    exit_code = 1
+    retryable = False
+    public_message = "Extraction failed."
+
+    def __init__(self, message: str | None = None) -> None:
+        super().__init__(message or self.public_message)
+
+    def envelope(self) -> dict[str, object]:
+        return {
+            "schema_version": "0.1",
+            "ok": False,
+            "error": {
+                "code": self.code,
+                "message": self.public_message,
+                "retryable": self.retryable,
+            },
+        }
+
+
+class InputRejected(ExtractorError):
+    code = "input_rejected"
+    exit_code = 2
+    public_message = "The URL was rejected by the public-source safety policy."
+
+
+class ProviderFailure(ExtractorError):
+    code = "provider_failure"
+    exit_code = 3
+    retryable = True
+    public_message = "The extraction provider is unavailable or rejected the request."
+
+
+class ProviderRateLimited(ProviderFailure):
+    code = "provider_rate_limited"
+    public_message = "The experimental provider rate limit was exceeded."
+
+
+class ProviderTimeout(ProviderFailure):
+    code = "provider_timeout"
+    public_message = "The extraction provider timed out."
+
+
+class InvalidProviderResponse(ExtractorError):
+    code = "invalid_provider_response"
+    exit_code = 4
+    public_message = "The extraction provider returned an invalid or unsafe response."
+
+
+class OutputFailure(ExtractorError):
+    code = "output_failure"
+    exit_code = 5
+    public_message = "The output file could not be written safely."
+