nscraper 0.1.0__tar.gz

nscraper-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 mikerr1@github.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nscraper-0.1.0/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.1
+Name: nscraper
+Version: 0.1.0
+Summary: A small importable Python module.
+License: MIT
+Requires-Dist: niquests==3.18.4
+Requires-Dist: justhtml==1.14.0
+Description-Content-Type: text/markdown
+
+# nscraper
+
+`nscraper` is a small Python package scaffolded for two use cases:
+
+- import it from other projects
+- run it directly with `python -m nscraper`
+
+## License
+
+MIT. You can fork, modify, and reuse it with minimal restrictions as long as
+the license notice is kept with the software.
+
+## Install
+
+```bash
+pip install nscraper
+```
+
+For development:
+
+```bash
+uv sync --dev
+```
+
+## Use as a module
+
+```python
+from nscraper import HttpScraper, ScrapeOptions
+
+options = ScrapeOptions(
+    url="https://example.com",
+    headers={"Accept": "text/html"},
+)
+
+content = HttpScraper(options).scrape()
+print(content)
+```
+
+## Run the Module
+
+```bash
+python -m nscraper -u https://example.com -H default
+```
+
+Fetch a URL:
+
+```bash
+python -m nscraper -u https://example.com -H default
+python -m nscraper -u https://example.com -H '{"Accept": "text/html"}'
+python -m nscraper -u https://example.com -H default -c cookies.json
+```
+
+## Current API
+
+- `nscraper.ScrapeOptions`
+- `nscraper.BaseScraper`
+- `nscraper.HttpScraper`
+- `nscraper.SeleniumBaseScraper`
+- `nscraper.get_scraper(options: ScrapeOptions) -> BaseScraper`
+- `nscraper.validate_url(url: str) -> str`
+- `nscraper.parse_headers(raw_headers: str | None) -> dict[str, str]`
+- `nscraper.load_cookies_file(path: Path | str | None) -> dict[str, str] | None`
+- `nscraper.basic_html_transform(content: str) -> str`
+- runtime dependency: `niquests==3.18.4`
+- runtime dependency: `justhtml==1.14.0`
+- development dependency: `pytest`
+
+## Module Flags
+
+- `-u` / `--url` required
+- `-H` / `--headers` required, or `default`
+- `-e` / `--engine` with `http` or `seleniumbase`
+- `-p` / `--proxy`
+- `--timeout` default `3`
+- `-o` / `--output`
+- `-c` / `--cookies-file` optional JSON file
+- `-t` / `--transform` default `raw`
+
+Behavior:
+
+- invalid or malformed URLs raise `InvalidUrlError`
+- missing or malformed headers raise `InvalidHeadersError`
+- missing or malformed cookie files raise `InvalidCookiesError`
+- use `-H default` to apply the built-in `Accept` and `User-Agent` header dict
+- use `-c` only when you want to send cookies; omit it to keep current behavior
+- output files are always overwritten
+- `basic_html` removes non-content elements and writes cleaned HTML output
+
+Default `User-Agent`:
+
+```text
+Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36
+```
+
+The package is intentionally minimal so you can extend it into a reusable library
+and publish it to PyPI.
+
+## GitHub And PyPI Release Flow
+
+- pull requests to `master` run tests in GitHub Actions
+- published GitHub releases run tests, build `sdist` and `wheel`, then publish to PyPI
+- the publish workflow is in [.github/workflows/release.yml](/home/ubuntu/projects/nscraper/.github/workflows/release.yml)
+
+Before the release workflow can publish, configure Trusted Publishing in PyPI:
+
+1. create the project on PyPI if it does not exist yet
+2. in PyPI, open the project publishing settings
+3. add a trusted publisher for this GitHub repository
+4. use the `release` workflow on the `master` branch
+
+After that, the normal flow is:
+
+1. push code to GitHub
+2. merge to `master`
+3. create a GitHub release for the version tag
+4. let GitHub Actions test, build, and publish the package

nscraper-0.1.0/README.md

@@ -0,0 +1,116 @@
+# nscraper
+
+`nscraper` is a small Python package scaffolded for two use cases:
+
+- import it from other projects
+- run it directly with `python -m nscraper`
+
+## License
+
+MIT. You can fork, modify, and reuse it with minimal restrictions as long as
+the license notice is kept with the software.
+
+## Install
+
+```bash
+pip install nscraper
+```
+
+For development:
+
+```bash
+uv sync --dev
+```
+
+## Use as a module
+
+```python
+from nscraper import HttpScraper, ScrapeOptions
+
+options = ScrapeOptions(
+    url="https://example.com",
+    headers={"Accept": "text/html"},
+)
+
+content = HttpScraper(options).scrape()
+print(content)
+```
+
+## Run the Module
+
+```bash
+python -m nscraper -u https://example.com -H default
+```
+
+Fetch a URL:
+
+```bash
+python -m nscraper -u https://example.com -H default
+python -m nscraper -u https://example.com -H '{"Accept": "text/html"}'
+python -m nscraper -u https://example.com -H default -c cookies.json
+```
+
+## Current API
+
+- `nscraper.ScrapeOptions`
+- `nscraper.BaseScraper`
+- `nscraper.HttpScraper`
+- `nscraper.SeleniumBaseScraper`
+- `nscraper.get_scraper(options: ScrapeOptions) -> BaseScraper`
+- `nscraper.validate_url(url: str) -> str`
+- `nscraper.parse_headers(raw_headers: str | None) -> dict[str, str]`
+- `nscraper.load_cookies_file(path: Path | str | None) -> dict[str, str] | None`
+- `nscraper.basic_html_transform(content: str) -> str`
+- runtime dependency: `niquests==3.18.4`
+- runtime dependency: `justhtml==1.14.0`
+- development dependency: `pytest`
+
+## Module Flags
+
+- `-u` / `--url` required
+- `-H` / `--headers` required, or `default`
+- `-e` / `--engine` with `http` or `seleniumbase`
+- `-p` / `--proxy`
+- `--timeout` default `3`
+- `-o` / `--output`
+- `-c` / `--cookies-file` optional JSON file
+- `-t` / `--transform` default `raw`
+
+Behavior:
+
+- invalid or malformed URLs raise `InvalidUrlError`
+- missing or malformed headers raise `InvalidHeadersError`
+- missing or malformed cookie files raise `InvalidCookiesError`
+- use `-H default` to apply the built-in `Accept` and `User-Agent` header dict
+- use `-c` only when you want to send cookies; omit it to keep current behavior
+- output files are always overwritten
+- `basic_html` removes non-content elements and writes cleaned HTML output
+
+Default `User-Agent`:
+
+```text
+Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36
+```
+
+The package is intentionally minimal so you can extend it into a reusable library
+and publish it to PyPI.
+
+## GitHub And PyPI Release Flow
+
+- pull requests to `master` run tests in GitHub Actions
+- published GitHub releases run tests, build `sdist` and `wheel`, then publish to PyPI
+- the publish workflow is in [.github/workflows/release.yml](/home/ubuntu/projects/nscraper/.github/workflows/release.yml)
+
+Before the release workflow can publish, configure Trusted Publishing in PyPI:
+
+1. create the project on PyPI if it does not exist yet
+2. in PyPI, open the project publishing settings
+3. add a trusted publisher for this GitHub repository
+4. use the `release` workflow on the `master` branch
+
+After that, the normal flow is:
+
+1. push code to GitHub
+2. merge to `master`
+3. create a GitHub release for the version tag
+4. let GitHub Actions test, build, and publish the package

nscraper-0.1.0/build_backend.py

@@ -0,0 +1,160 @@
+"""Minimal PEP 517 backend for offline editable and wheel installs."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+import base64
+import csv
+import hashlib
+import io
+import os
+import tarfile
+import tempfile
+import zipfile
+
+NAME = "nscraper"
+VERSION = "0.1.0"
+DIST_INFO = f"{NAME}-{VERSION}.dist-info"
+SUMMARY = "A small importable Python module."
+README_PATH = Path(__file__).resolve().parent / "README.md"
+
+
+@dataclass(frozen=True)
+class FileRecord:
+    path: str
+    data: bytes
+
+
+def _normalize(path: str) -> str:
+    return path.replace(os.sep, "/")
+
+
+def _metadata_text() -> str:
+    description = README_PATH.read_text(encoding="utf-8")
+    return (
+        "Metadata-Version: 2.1\n"
+        f"Name: {NAME}\n"
+        f"Version: {VERSION}\n"
+        f"Summary: {SUMMARY}\n"
+        "License: MIT\n"
+        "Requires-Dist: niquests==3.18.4\n"
+        "Requires-Dist: justhtml==1.14.0\n"
+        "Description-Content-Type: text/markdown\n"
+        "\n"
+        f"{description}"
+    )
+
+
+def _wheel_metadata() -> list[FileRecord]:
+    wheel = (
+        "Wheel-Version: 1.0\n"
+        "Generator: nscraper.build_backend\n"
+        "Root-Is-Purelib: true\n"
+        "Tag: py3-none-any\n"
+    )
+    meta = _metadata_text()
+    entry_points = "[console_scripts]\nnscraper = nscraper.__main__:main\n"
+    return [
+        FileRecord(f"{DIST_INFO}/WHEEL", wheel.encode()),
+        FileRecord(f"{DIST_INFO}/METADATA", meta.encode()),
+        FileRecord(f"{DIST_INFO}/entry_points.txt", entry_points.encode()),
+    ]
+
+
+def _package_files() -> list[FileRecord]:
+    root = Path(__file__).resolve().parent / "src" / NAME
+    records: list[FileRecord] = []
+    for path in root.rglob("*.py"):
+        rel = _normalize(str(path.relative_to(root.parent)))
+        records.append(FileRecord(rel, path.read_bytes()))
+    return records
+
+
+def _sdist_files(root: Path) -> list[tuple[Path, str]]:
+    members = ["build_backend.py", "pyproject.toml", "README.md", "LICENSE"]
+    files = [(root / member, f"{NAME}-{VERSION}/{member}") for member in members]
+    for base in ("src", "tests"):
+        for path in (root / base).rglob("*"):
+            if path.is_dir() or "__pycache__" in path.parts or path.suffix == ".pyc":
+                continue
+            arcname = f"{NAME}-{VERSION}/{path.relative_to(root)}"
+            files.append((path, arcname))
+    return files
+
+
+def _build_wheel_dir(wheel_directory: str, editable: bool = False) -> str:
+    out = Path(wheel_directory)
+    out.mkdir(parents=True, exist_ok=True)
+    wheel_name = f"{NAME}-{VERSION}-py3-none-any.whl"
+    wheel_path = out / wheel_name
+    records = _wheel_metadata()
+    if editable:
+        pth = f"{NAME}.pth"
+        src_dir = (Path(__file__).resolve().parent / "src").as_posix()
+        records.append(FileRecord(pth, f"{src_dir}\n".encode()))
+    else:
+        records.extend(_package_files())
+    with tempfile.TemporaryDirectory() as td:
+        tmp = Path(td)
+        with zipfile.ZipFile(wheel_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
+            for record in records:
+                zf.writestr(record.path, record.data)
+            dist_info = tmp / DIST_INFO
+            dist_info.mkdir()
+            record_file = dist_info / "RECORD"
+            rows = []
+            for record in records:
+                digest = base64.urlsafe_b64encode(hashlib.sha256(record.data).digest()).rstrip(b"=").decode()
+                rows.append((record.path, f"sha256={digest}", str(len(record.data))))
+            rows.append((f"{DIST_INFO}/RECORD", "", ""))
+            with record_file.open("w", newline="", encoding="utf-8") as f:
+                writer = csv.writer(f)
+                writer.writerows(rows)
+            zf.write(record_file, f"{DIST_INFO}/RECORD")
+    return wheel_name
+
+
+def get_requires_for_build_wheel(config_settings=None):
+    return []
+
+
+def get_requires_for_build_editable(config_settings=None):
+    return []
+
+
+def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
+    dist = Path(metadata_directory) / DIST_INFO
+    dist.mkdir(parents=True, exist_ok=True)
+    for record in _wheel_metadata():
+        if record.path.endswith(("WHEEL", "METADATA", "entry_points.txt")):
+            (dist / Path(record.path).name).write_bytes(record.data)
+    return DIST_INFO
+
+
+def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
+    return _build_wheel_dir(wheel_directory, editable=False)
+
+
+def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
+    return _build_wheel_dir(wheel_directory, editable=True)
+
+
+def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
+    return prepare_metadata_for_build_wheel(metadata_directory, config_settings)
+
+
+def build_sdist(sdist_directory, config_settings=None):
+    out = Path(sdist_directory)
+    out.mkdir(parents=True, exist_ok=True)
+    sdist_name = f"{NAME}-{VERSION}.tar.gz"
+    sdist_path = out / sdist_name
+    root = Path(__file__).resolve().parent
+    pkg_info = _metadata_text().encode("utf-8")
+    with tarfile.open(sdist_path, "w:gz") as tf:
+        for path, arcname in _sdist_files(root):
+            tf.add(path, arcname=arcname)
+        info = tarfile.TarInfo(name=f"{NAME}-{VERSION}/PKG-INFO")
+        info.size = len(pkg_info)
+        tf.addfile(info, fileobj=io.BytesIO(pkg_info))
+    return sdist_name

nscraper-0.1.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = []
+build-backend = "build_backend"
+backend-path = ["."]
+
+[project]
+name = "nscraper"
+version = "0.1.0"
+description = "A small importable Python module."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+  { name = "mikerr1@github.com" }
+]
+dependencies = ["niquests==3.18.4", "justhtml==1.14.0"]
+
+[project.scripts]
+nscraper = "nscraper.__main__:main"
+
+[dependency-groups]
+dev = ["pytest"]

nscraper-0.1.0/src/nscraper/__init__.py

@@ -0,0 +1,42 @@
+"""Public package interface for nscraper."""
+
+from .core import get_scraper, hello
+from .errors import (
+    InvalidHeadersError,
+    InvalidCookiesError,
+    InvalidUrlError,
+    NetworkError,
+    NscraperError,
+    RequestError,
+)
+from .models import ScrapeOptions
+from .scraper import BaseScraper, HttpScraper, SeleniumBaseScraper
+from .utils import (
+    DEFAULT_HEADERS,
+    basic_html_transform,
+    load_cookies_file,
+    parse_headers,
+    validate_url,
+    write_output,
+)
+
+__all__ = [
+    "BaseScraper",
+    "DEFAULT_HEADERS",
+    "HttpScraper",
+    "InvalidHeadersError",
+    "InvalidCookiesError",
+    "InvalidUrlError",
+    "NetworkError",
+    "NscraperError",
+    "RequestError",
+    "ScrapeOptions",
+    "SeleniumBaseScraper",
+    "basic_html_transform",
+    "load_cookies_file",
+    "hello",
+    "get_scraper",
+    "parse_headers",
+    "validate_url",
+    "write_output",
+]

nscraper-0.1.0/src/nscraper/__main__.py

@@ -0,0 +1,66 @@
+"""Module entry point for nscraper."""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+from .core import get_scraper
+from .errors import NscraperError
+from .models import ScrapeOptions
+from .utils import DEFAULT_HEADERS, load_cookies_file, parse_headers
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="nscraper")
+    parser.add_argument("-u", "--url", required=True, help="Target URL")
+    parser.add_argument(
+        "-e",
+        "--engine",
+        choices=("http", "seleniumbase"),
+        default="http",
+        help="Request engine",
+    )
+    parser.add_argument("-p", "--proxy", help="Proxy URL")
+    parser.add_argument("-H", "--headers", required=True, help='Headers as JSON string or "default"')
+    parser.add_argument("-c", "--cookies-file", help="Path to a JSON cookies file")
+    parser.add_argument("--timeout", type=float, default=3.0, help="Timeout in seconds")
+    parser.add_argument("-o", "--output", help="Output path for HTML")
+    parser.add_argument(
+        "-t",
+        "--transform",
+        choices=("raw", "basic_html"),
+        default="raw",
+        help="Transform mode",
+    )
+    return parser
+
+
+def _build_options(args: argparse.Namespace) -> ScrapeOptions:
+    headers = DEFAULT_HEADERS if args.headers == "default" else parse_headers(args.headers)
+    return ScrapeOptions(
+        url=args.url,
+        engine=args.engine,
+        proxy=args.proxy,
+        headers=headers,
+        cookies=load_cookies_file(args.cookies_file),
+        timeout=args.timeout,
+        output_path=Path(args.output) if args.output else None,
+        transform=args.transform,
+    )
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    try:
+        options = _build_options(args)
+        content = get_scraper(options).scrape()
+    except NscraperError as exc:
+        raise SystemExit(str(exc)) from exc
+    if not options.output_path:
+        print(content)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

nscraper-0.1.0/src/nscraper/core.py

@@ -0,0 +1,18 @@
+"""Core library functions."""
+
+from __future__ import annotations
+
+from .models import ScrapeOptions
+from .scraper import BaseScraper, HttpScraper, SeleniumBaseScraper
+
+
+def hello(name: str = "world") -> str:
+    """Return a friendly greeting."""
+    cleaned = name.strip() or "world"
+    return f"Hello, {cleaned}!"
+
+
+def get_scraper(options: ScrapeOptions) -> BaseScraper:
+    if options.engine == "seleniumbase":
+        return SeleniumBaseScraper(options)
+    return HttpScraper(options)

nscraper-0.1.0/src/nscraper/errors.py

@@ -0,0 +1,25 @@
+"""Project-specific exceptions for nscraper."""
+
+
+class NscraperError(Exception):
+    """Base exception for nscraper failures."""
+
+
+class InvalidUrlError(NscraperError):
+    """Raised when a URL argument is missing or invalid."""
+
+
+class InvalidHeadersError(NscraperError):
+    """Raised when headers are missing or malformed."""
+
+
+class InvalidCookiesError(NscraperError):
+    """Raised when cookies are missing or malformed."""
+
+
+class RequestError(NscraperError):
+    """Raised when an HTTP request fails with a non-success response."""
+
+
+class NetworkError(NscraperError):
+    """Raised when the request cannot be completed due to network issues."""

nscraper-0.1.0/src/nscraper/models.py

@@ -0,0 +1,24 @@
+"""Data models for nscraper operations."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal
+
+Engine = Literal["http", "seleniumbase"]
+Transform = Literal["raw", "basic_html"]
+
+
+@dataclass(frozen=True, slots=True)
+class ScrapeOptions:
+    """Normalized scrape configuration."""
+
+    url: str
+    engine: Engine = "http"
+    proxy: str | None = None
+    headers: dict[str, str] | None = None
+    cookies: dict[str, str] | None = None
+    timeout: float = 3.0
+    output_path: Path | None = None
+    transform: Transform = "raw"

nscraper-0.1.0/src/nscraper/scraper/__init__.py

@@ -0,0 +1,7 @@
+"""Scraper implementations."""
+
+from .base import BaseScraper
+from .http import HttpScraper
+from .seleniumbase import SeleniumBaseScraper
+
+__all__ = ["BaseScraper", "HttpScraper", "SeleniumBaseScraper"]

nscraper-0.1.0/src/nscraper/scraper/base.py

@@ -0,0 +1,34 @@
+"""Base scraper contract."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+
+from ..models import ScrapeOptions
+from ..utils import basic_html_transform, write_output
+
+
+class BaseScraper(ABC):
+    """Base scraper contract."""
+
+    def __init__(self, options: ScrapeOptions) -> None:
+        self.options = options
+
+    @abstractmethod
+    def send_request(self) -> str:
+        """Send the underlying request and return raw content."""
+
+    def transform(self, content: str) -> str:
+        if self.options.transform == "basic_html":
+            return basic_html_transform(content)
+        return content
+
+    def store(self, content: str) -> None:
+        if self.options.output_path:
+            write_output(self.options.output_path, content)
+
+    def scrape(self) -> str:
+        content = self.send_request()
+        content = self.transform(content)
+        self.store(content)
+        return content

nscraper-0.1.0/src/nscraper/scraper/http.py

@@ -0,0 +1,32 @@
+"""HTTP scraper implementation."""
+
+from __future__ import annotations
+
+import niquests
+
+from ..errors import InvalidHeadersError, NetworkError, RequestError
+from ..utils import validate_url
+from .base import BaseScraper
+
+
+class HttpScraper(BaseScraper):
+    """Scraper implementation backed by niquests."""
+
+    def send_request(self) -> str:
+        url = validate_url(self.options.url)
+        headers = self.options.headers or {}
+        if not headers:
+            raise InvalidHeadersError("headers are required")
+        kwargs: dict[str, object] = {"headers": headers, "timeout": self.options.timeout}
+        if self.options.proxy:
+            kwargs["proxies"] = {"http": self.options.proxy, "https": self.options.proxy}
+        if self.options.cookies:
+            kwargs["cookies"] = self.options.cookies
+        try:
+            response = niquests.get(url, **kwargs)
+            response.raise_for_status()
+        except niquests.exceptions.HTTPError as exc:
+            raise RequestError(f"request failed for {url}") from exc
+        except niquests.exceptions.RequestException as exc:
+            raise NetworkError(f"network failure while fetching {url}") from exc
+        return response.text

nscraper-0.1.0/src/nscraper/scraper/seleniumbase.py

@@ -0,0 +1,12 @@
+"""SeleniumBase scraper implementation placeholder."""
+
+from __future__ import annotations
+
+from .base import BaseScraper
+
+
+class SeleniumBaseScraper(BaseScraper):
+    """Scraper placeholder for SeleniumBase-driven scraping."""
+
+    def send_request(self) -> str:
+        raise NotImplementedError("seleniumbase scraper is not implemented yet")

nscraper-0.1.0/src/nscraper/utils.py

@@ -0,0 +1,116 @@
+"""Reusable utility functions for nscraper."""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+from urllib.parse import urlparse
+
+from justhtml import JustHTML
+
+from .errors import InvalidCookiesError, InvalidHeadersError, InvalidUrlError
+
+DEFAULT_HEADERS = {
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36",
+}
+
+BASIC_HTML_CLEANUP_SELECTORS = (
+    "script",
+    "style",
+    "noscript",
+    "iframe",
+    "source",
+    "svg",
+    "template",
+    "[aria-hidden='true']",
+    "[hidden]",
+    ".ads",
+    ".advertisement",
+    ".banner",
+    ".social-share",
+    ".newsletter",
+)
+
+_ARIA_HIDDEN_RE = re.compile(
+    r"<(?P<tag>[a-zA-Z][\w:-]*)(?=[^>]*\baria-hidden=(['\"])true\2)[^>]*>.*?</(?P=tag)>",
+    re.IGNORECASE | re.DOTALL,
+)
+_HIDDEN_RE = re.compile(
+    r"<(?P<tag>[a-zA-Z][\w:-]*)(?=[^>]*\bhidden(?:\s|>|=))[^>]*>.*?</(?P=tag)>",
+    re.IGNORECASE | re.DOTALL,
+)
+
+
+def validate_url(url: str) -> str:
+    cleaned = url.strip()
+    parsed = urlparse(cleaned)
+    if (
+        not cleaned
+        or parsed.scheme not in {"http", "https"}
+        or not parsed.netloc
+        or " " in cleaned
+        or not parsed.hostname
+    ):
+        raise InvalidUrlError("invalid url")
+    return cleaned
+
+
+def parse_headers(raw_headers: str | None) -> dict[str, str]:
+    if raw_headers is None or not raw_headers.strip():
+        raise InvalidHeadersError("headers are required")
+    try:
+        value = json.loads(raw_headers)
+    except json.JSONDecodeError as exc:
+        raise InvalidHeadersError("headers must be valid JSON") from exc
+    if not isinstance(value, dict) or not value:
+        raise InvalidHeadersError("headers must be a non-empty object")
+    headers: dict[str, str] = {}
+    for key, item in value.items():
+        if not isinstance(key, str) or not key.strip():
+            raise InvalidHeadersError("header names must be non-empty strings")
+        if not isinstance(item, str):
+            raise InvalidHeadersError("header values must be strings")
+        headers[key.strip()] = item
+    return headers
+
+
+def load_cookies_file(cookies_file: Path | str | None) -> dict[str, str] | None:
+    if cookies_file is None:
+        return None
+    path = Path(cookies_file)
+    if not path.exists():
+        raise InvalidCookiesError(f"cookies file not found: {path}")
+    try:
+        payload = json.loads(path.read_text(encoding="utf-8"))
+    except json.JSONDecodeError as exc:
+        raise InvalidCookiesError("cookies file must contain valid JSON") from exc
+    if not isinstance(payload, dict) or not payload:
+        raise InvalidCookiesError("cookies file must contain a non-empty JSON object")
+    cookies: dict[str, str] = {}
+    for key, value in payload.items():
+        if not isinstance(key, str) or not key.strip():
+            raise InvalidCookiesError("cookie names must be non-empty strings")
+        if not isinstance(value, str):
+            raise InvalidCookiesError("cookie values must be strings")
+        cookies[key.strip()] = value
+    return cookies
+
+
+def basic_html_transform(content: str) -> str:
+    cleaned_input = _ARIA_HIDDEN_RE.sub("", content)
+    cleaned_input = _HIDDEN_RE.sub("", cleaned_input)
+    doc = JustHTML(cleaned_input, fragment=False)
+    for selector in BASIC_HTML_CLEANUP_SELECTORS:
+        for node in doc.query(selector):
+            if node.parent:
+                node.parent.remove_child(node)
+    for head in doc.query("head"):
+        while head.has_child_nodes():
+            head.remove_child(head.children[0])
+    return doc.to_html(pretty=True)
+
+
+def write_output(output_path: Path, content: str) -> None:
+    output_path.write_text(content, encoding="utf-8")

nscraper-0.1.0/tests/test_core.py

@@ -0,0 +1,94 @@
+import pytest
+
+from nscraper import InvalidHeadersError, InvalidUrlError, basic_html_transform, parse_headers, validate_url
+
+
+@pytest.mark.parametrize(
+    ("url", "expected"),
+    [
+        ("http://example.com", "http://example.com"),
+        ("https://example.com", "https://example.com"),
+    ],
+)
+def test_validate_url_accepts_standard_urls(url, expected):
+    assert validate_url(url) == expected
+
+
+@pytest.mark.parametrize(
+    "url",
+    [
+        "not-a-url",
+        "https://example.com/path with space",
+        "ftp://example.com",
+        "http://",
+    ],
+)
+def test_validate_url_rejects_invalid_urls(url):
+    with pytest.raises(InvalidUrlError, match="invalid url"):
+        validate_url(url)
+
+
+def test_parse_headers_rejects_missing():
+    with pytest.raises(InvalidHeadersError):
+        parse_headers(None)
+
+
+def test_parse_headers_accepts_json_object():
+    headers = parse_headers('{"Accept": "text/html"}')
+    assert headers == {"Accept": "text/html"}
+
+
+def test_basic_html_transform_removes_tags():
+    cleaned = basic_html_transform("<html><body>Hello</body></html>")
+    assert "<html" in cleaned
+    assert "<body>" in cleaned
+    assert "Hello" in cleaned
+
+
+def test_basic_html_transform_strips_scripts_and_styles():
+    html = """
+    <html>
+      <head>
+        <style>body { color: red; }</style>
+        <script>alert('x')</script>
+      </head>
+      <body><noscript>ignore</noscript><div>Hello</div></body>
+    </html>
+    """
+    cleaned = basic_html_transform(html)
+    assert "<script>" not in cleaned
+    assert "<style>" not in cleaned
+    assert "<noscript>" not in cleaned
+    assert "body { color: red; }" not in cleaned
+    assert "alert('x')" not in cleaned
+    assert "Hello" in cleaned
+    assert "<html>" in cleaned
+
+
+def test_basic_html_transform_handles_broken_markup():
+    html = "<div>Hello<script>bad()</script><style>.x{}</style><span>World</span>"
+    cleaned = basic_html_transform(html)
+    assert "<script>" not in cleaned
+    assert "<style>" not in cleaned
+    assert "Hello" in cleaned
+    assert "World" in cleaned
+    assert "<div>" in cleaned or "<span>" in cleaned
+
+
+def test_basic_html_transform_removes_hidden_and_ad_like_nodes():
+    html = """
+    <html>
+      <body>
+        <div class="ads">Ad</div>
+        <div class="newsletter">Newsletter</div>
+        <div aria-hidden="true">Hidden</div>
+        <div hidden>Hidden attr</div>
+        <div class="content">Keep</div>
+      </body>
+    </html>
+    """
+    cleaned = basic_html_transform(html)
+    assert "Ad" not in cleaned
+    assert "Newsletter" not in cleaned
+    assert "Hidden" not in cleaned
+    assert "Keep" in cleaned

nscraper-0.1.0/tests/test_http_scraper.py

@@ -0,0 +1,113 @@
+from dataclasses import replace
+import niquests
+import pytest
+
+from nscraper import InvalidHeadersError, NetworkError, RequestError, ScrapeOptions
+from nscraper.scraper.http import HttpScraper
+
+
+class DummyResponse:
+    def __init__(self, text: str, *, raise_exc: Exception | None = None) -> None:
+        self.text = text
+        self._raise_exc = raise_exc
+
+    def raise_for_status(self) -> None:
+        if self._raise_exc is not None:
+            raise self._raise_exc
+
+
+def make_options(**overrides):
+    base = ScrapeOptions(
+        url="https://example.com",
+        headers={"Accept": "text/html"},
+        cookies=None,
+        output_path=None,
+        transform="raw",
+    )
+    return replace(base, **overrides)
+
+
+def test_http_scraper_returns_raw_response(monkeypatch):
+    response = DummyResponse("<html>Hello</html>")
+    calls = {}
+
+    def fake_get(url, **kwargs):
+        calls["url"] = url
+        calls["kwargs"] = kwargs
+        return response
+
+    monkeypatch.setattr("niquests.get", fake_get)
+    scraper = HttpScraper(make_options())
+
+    assert scraper.scrape() == "<html>Hello</html>"
+    assert calls["url"] == "https://example.com"
+    assert calls["kwargs"]["headers"] == {"Accept": "text/html"}
+
+
+def test_http_scraper_forwards_cookies(monkeypatch):
+    response = DummyResponse("<html>Hello</html>")
+    calls = {}
+
+    def fake_get(url, **kwargs):
+        calls["kwargs"] = kwargs
+        return response
+
+    monkeypatch.setattr("niquests.get", fake_get)
+    scraper = HttpScraper(make_options(cookies={"sessionid": "abc123"}))
+
+    assert scraper.scrape() == "<html>Hello</html>"
+    assert calls["kwargs"]["cookies"] == {"sessionid": "abc123"}
+
+
+def test_http_scraper_applies_basic_html_transform(monkeypatch):
+    monkeypatch.setattr("niquests.get", lambda *args, **kwargs: DummyResponse("<script>x</script><div>Hello</div>"))
+    scraper = HttpScraper(make_options(transform="basic_html"))
+
+    cleaned = scraper.scrape()
+    assert "<div>Hello</div>" in cleaned
+    assert "<script>" not in cleaned
+
+
+def test_http_scraper_writes_output(monkeypatch, tmp_path):
+    monkeypatch.setattr("niquests.get", lambda *args, **kwargs: DummyResponse("Hello"))
+    output = tmp_path / "page.html"
+    scraper = HttpScraper(make_options(output_path=output))
+
+    assert scraper.scrape() == "Hello"
+    assert output.read_text(encoding="utf-8") == "Hello"
+
+
+def test_http_scraper_rejects_missing_headers(monkeypatch):
+    monkeypatch.setattr("niquests.get", lambda *args, **kwargs: DummyResponse("Hello"))
+    scraper = HttpScraper(make_options(headers=None))
+
+    with pytest.raises(InvalidHeadersError, match="headers are required"):
+        scraper.scrape()
+
+
+def test_http_scraper_maps_request_error(monkeypatch):
+    class Boom(niquests.exceptions.HTTPError):
+        pass
+
+    def fake_get(*args, **kwargs):
+        return DummyResponse("ignored", raise_exc=Boom("bad status"))
+
+    monkeypatch.setattr("niquests.get", fake_get)
+    scraper = HttpScraper(make_options())
+
+    with pytest.raises(RequestError, match="request failed for"):
+        scraper.scrape()
+
+
+def test_http_scraper_maps_network_error(monkeypatch):
+    class Boom(niquests.exceptions.RequestException):
+        pass
+
+    def fake_get(*args, **kwargs):
+        raise Boom("network down")
+
+    monkeypatch.setattr("niquests.get", fake_get)
+    scraper = HttpScraper(make_options())
+
+    with pytest.raises(NetworkError, match="network failure while fetching"):
+        scraper.scrape()

nscraper-0.1.0/tests/test_main.py

@@ -0,0 +1,101 @@
+from argparse import Namespace
+
+import pytest
+
+from nscraper import InvalidCookiesError, ScrapeOptions
+from nscraper.__main__ import _build_options
+from nscraper.utils import DEFAULT_HEADERS
+
+
+def test_default_headers_are_used_when_explicitly_requested():
+    options = _build_options(
+        Namespace(
+            url="https://example.com",
+            engine="http",
+            proxy=None,
+            headers="default",
+            cookies_file=None,
+            timeout=3.0,
+            output=None,
+            transform="raw",
+        )
+    )
+
+    assert options.headers == DEFAULT_HEADERS
+    assert options.headers["User-Agent"] == (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+        "(KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36"
+    )
+
+
+def test_explicit_headers_override_defaults():
+    options = _build_options(
+        Namespace(
+            url="https://example.com",
+            engine="http",
+            proxy=None,
+            headers='{"Accept": "application/json"}',
+            cookies_file=None,
+            timeout=3.0,
+            output=None,
+            transform="raw",
+        )
+    )
+
+    assert options.headers == {"Accept": "application/json"}
+    assert isinstance(options, ScrapeOptions)
+
+
+def test_cookie_file_is_optional(tmp_path):
+    options = _build_options(
+        Namespace(
+            url="https://example.com",
+            engine="http",
+            proxy=None,
+            headers="default",
+            cookies_file=None,
+            timeout=3.0,
+            output=None,
+            transform="raw",
+        )
+    )
+
+    assert options.cookies is None
+
+
+def test_cookie_file_is_loaded(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text('{"sessionid": "abc123"}', encoding="utf-8")
+    options = _build_options(
+        Namespace(
+            url="https://example.com",
+            engine="http",
+            proxy=None,
+            headers="default",
+            cookies_file=cookie_file,
+            timeout=3.0,
+            output=None,
+            transform="raw",
+        )
+    )
+
+    assert options.cookies == {"sessionid": "abc123"}
+
+
+def test_cookie_file_invalid_json_fails_fast(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text("{not json}", encoding="utf-8")
+
+    with pytest.raises(InvalidCookiesError, match="valid JSON"):
+        _build_options(
+            Namespace(
+                url="https://example.com",
+                engine="http",
+                proxy=None,
+                headers="default",
+                cookies_file=cookie_file,
+                timeout=3.0,
+                output=None,
+                transform="raw",
+            )
+        )

nscraper-0.1.0/tests/test_utils.py

@@ -0,0 +1,54 @@
+from pathlib import Path
+
+import pytest
+
+from nscraper import InvalidCookiesError
+from nscraper.utils import load_cookies_file
+
+
+def test_load_cookies_file_returns_none_when_missing():
+    assert load_cookies_file(None) is None
+
+
+def test_load_cookies_file_reads_json(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text('{"sessionid": "abc123"}', encoding="utf-8")
+
+    assert load_cookies_file(cookie_file) == {"sessionid": "abc123"}
+
+
+def test_load_cookies_file_rejects_missing_file(tmp_path):
+    with pytest.raises(InvalidCookiesError, match="cookies file not found"):
+        load_cookies_file(tmp_path / "missing.json")
+
+
+def test_load_cookies_file_rejects_invalid_json(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text("{not json}", encoding="utf-8")
+
+    with pytest.raises(InvalidCookiesError, match="valid JSON"):
+        load_cookies_file(cookie_file)
+
+
+def test_load_cookies_file_rejects_non_object_json(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text('["sessionid", "abc123"]', encoding="utf-8")
+
+    with pytest.raises(InvalidCookiesError, match="non-empty JSON object"):
+        load_cookies_file(cookie_file)
+
+
+def test_load_cookies_file_rejects_empty_object(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text("{}", encoding="utf-8")
+
+    with pytest.raises(InvalidCookiesError, match="non-empty JSON object"):
+        load_cookies_file(cookie_file)
+
+
+def test_load_cookies_file_rejects_non_string_values(tmp_path):
+    cookie_file = tmp_path / "cookies.json"
+    cookie_file.write_text('{"sessionid": 123}', encoding="utf-8")
+
+    with pytest.raises(InvalidCookiesError, match="cookie values must be strings"):
+        load_cookies_file(cookie_file)