crawl4md 0.1.2__tar.gz

crawl4md-0.1.2/.gitignore

@@ -0,0 +1,57 @@
+# -------------------------
+# Python
+# -------------------------
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.so
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# -------------------------
+# Build / Packaging
+# -------------------------
+build/
+dist/
+*.egg-info/
+.eggs/
+
+# -------------------------
+# uv
+# -------------------------
+.uv/
+uv.lock
+
+# -------------------------
+# IDE / Editor
+# -------------------------
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# -------------------------
+# OS
+# -------------------------
+.DS_Store
+Thumbs.db
+
+# -------------------------
+# Project specific
+# -------------------------
+
+# User-specific config (must NOT be committed)
+crawl.yml
+
+# Generated Markdown output
+docs/*
+!docs/.gitkeep
+
+# Optional: logs or temp files (future-proof)
+*.log
+tmp/
+

crawl4md-0.1.2/AGENTS.md

@@ -0,0 +1,171 @@
+# AGENTS.md
+
+## Purpose
+
+crawl4md is a minimal CLI tool to crawl web pages or sitemaps and convert them into clean, deterministic Markdown files.
+
+This document defines how contributors and automated agents (LLMs, scripts, CI tools) should interact with and extend the project.
+
+---
+
+## Core Principles
+
+- Keep it simple (no overengineering)
+- Deterministic output (same input → same output)
+- No hidden behavior or side effects
+- Clear separation of concerns:
+  - config
+  - crawling
+  - preprocessing
+  - writing
+- CLI-first design
+
+---
+
+## Project Structure
+
+src/crawl4md/
+- cli.py            → entrypoint (orchestration only)
+- config.py         → config models (Pydantic)
+- sitemap.py        → sitemap parsing
+- crawler.py        → crawl4ai integration
+- paths.py          → URL → file path mapping
+- writer.py         → file output
+- (future) preprocessing.py → markdown cleanup
+
+docs/
+- output directory (generated, not versioned except .gitkeep)
+
+crawl.yml.example
+- example configuration (must stay in sync with config models)
+
+---
+
+## Responsibilities
+
+### cli.py
+- Orchestrates flow
+- No business logic
+- Reads config, loops URLs, prints output
+
+### crawler.py
+- Only responsible for fetching + converting to markdown
+- Must not handle filesystem or preprocessing
+
+### preprocessing (future)
+- Pure functions: markdown in → markdown out
+- No IO, no side effects
+
+### writer.py
+- Only writes files
+- Must not modify content
+
+---
+
+## Configuration Rules
+
+- All behavior must be configurable via crawl.yml
+- crawl.yml is user-specific → never commit
+- crawl.yml.example is canonical → always update when config changes
+
+### Config Sections
+
+- type: pages | sitemap
+- sources: list[str]
+
+- crawl:
+    parse_type: markdown | markdown-fit
+
+- preprocessing:
+    markdown:
+        enabled: bool
+        remove_reference_sections: bool
+        remove_html_comments: bool
+        normalize_whitespace: bool
+        reference_headings: list[str]
+
+---
+
+## Coding Guidelines
+
+- Python >= 3.11
+- Use type hints everywhere
+- Prefer explicit over implicit
+- No global state (except logging config)
+- Keep functions small and focused
+- Avoid unnecessary dependencies
+
+---
+
+## Logging & Output
+
+- CLI output must stay minimal and readable
+- Avoid noisy logs
+- External library logs (e.g. crawl4ai) should be suppressed or reduced
+
+---
+
+## Error Handling
+
+- Errors per URL must not stop the entire run
+- Always continue with next URL
+- Provide clear summary:
+  - success count
+  - failure count
+
+---
+
+## File Output Rules
+
+- Output path must be deterministic:
+  docs/<project>/<url-path>.md
+
+- URL path rules:
+  - "/" → index.md
+  - strip leading slash
+  - ignore query params
+
+- Always create parent directories
+
+---
+
+## Extending the Project
+
+When adding features:
+
+1. First update config models
+2. Then update crawl.yml.example
+3. Keep backward compatibility (defaults!)
+4. Add logic in the correct layer (do not mix concerns)
+
+---
+
+## Anti-Patterns (Avoid)
+
+- Business logic inside cli.py
+- Hidden transformations
+- Implicit defaults not visible in config
+- Mixing crawling and preprocessing
+- Writing files outside writer.py
+
+---
+
+## Future Extensions (Planned)
+
+- Markdown preprocessing pipeline
+- Frontmatter support
+- Parallel crawling
+- Retry & rate limiting
+- Chunking for RAG
+- Database export
+
+---
+
+## Summary
+
+crawl4md is intentionally simple.
+
+Every addition must preserve:
+- clarity
+- determinism
+- separation of concerns

crawl4md-0.1.2/CHANGELOG.md

@@ -0,0 +1,43 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [0.1.2] - 2026-05-02
+
+### Added
+
+* Update README for PyPI package usage and clarify batch crawler setup
+
+## [0.1.1] - 2026-05-02
+
+### Added
+
+- Add uv check command for tests and Ruff linting
+- Export public Python API and expand README with usage and crawl4ai context
+
+### Refactored
+
+- Split `fetch_markdown` into fetch and convert layers
+- Move markdown preprocessing from CLI into convert pipeline
+- Refactor markdown fetch/convert into classes and add sync APIs
+
+### Removed
+
+- Crawl4AI Logging
+
+## [0.1.0] - 2026-05-02
+
+### Added
+
+- Initial release
+- CLI for crawling single pages and sitemaps
+- YAML-based project configuration
+- Deterministic Markdown file output
+- Support for multiple Markdown extraction modes
+- Configurable Markdown preprocessing pipeline
+- Automatic cleanup of common wiki and web artifacts
+- Automatic removal of reference and appendix sections
+- Whitespace and document structure normalization
+- Automatic insertion of missing top-level headings
+- Clear separation of crawling, preprocessing, and file writing
+- Basic test coverage for core Markdown processing

crawl4md-0.1.2/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Björn Hempel <bjoern@hempel.li>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

crawl4md-0.1.2/PKG-INFO

@@ -0,0 +1,336 @@
+Metadata-Version: 2.4
+Name: crawl4md
+Version: 0.1.2
+Summary: Convert web pages and HTML to clean Markdown
+Author: Björn Hempel
+License: MIT
+License-File: LICENSE.md
+Requires-Python: >=3.11
+Requires-Dist: crawl4ai
+Requires-Dist: lxml
+Requires-Dist: pydantic
+Requires-Dist: pyyaml
+Requires-Dist: requests
+Requires-Dist: typer
+Description-Content-Type: text/markdown
+
+# crawl4md
+
+crawl4md is a minimal, clean CLI tool that crawls web pages or sitemaps and converts them into structured Markdown files.
+
+The project is intentionally designed to stay simple, deterministic, and easy to extend — without unnecessary complexity or hidden behavior.
+
+---
+
+## Philosophy
+
+- **Minimal**: only what is needed, nothing more  
+- **Deterministic**: same input → same output  
+- **Transparent**: no magic, clear processing steps  
+- **Composable**: ideal as a building block for pipelines (e.g. RAG)
+
+---
+
+## Features
+
+- Crawl from:
+  - `sitemap.xml`
+  - explicit page lists
+- Clean Markdown output (via `crawl4ai`, markdown-fit mode)
+- Deterministic file structure based on URL paths
+- YAML-based project configuration
+- CLI-first workflow (uv-compatible)
+- Clear, readable progress output
+
+---
+
+## Installation
+
+There are two ways to use `crawl4md`.
+
+### Use the Batch Crawler
+
+If you want to use the project directly for batch crawling via `crawl.yml`, clone the repository:
+
+```bash
+git clone git@github.com:ixnode/crawl4md.git && cd crawl4md
+```
+
+Then continue with the configuration section below.
+
+### Use the Python Package
+
+If you want to build your own tooling on top of `crawl4md`, install it as a package:
+
+```bash
+pip install crawl4md
+```
+
+Or with `uv`:
+
+```bash
+uv add crawl4md
+```
+
+For local development inside the repository:
+
+```bash
+uv sync
+```
+
+---
+
+## Configuration
+
+The CLI reads a `crawl.yml` file from the current working directory.
+
+Create it from the example:
+
+```bash
+cp crawl.yml.example crawl.yml
+```
+
+Minimal example:
+
+```yaml
+projects:
+    planes:
+        type: pages
+        crawl:
+            parse_type: markdown-fit
+        sources:
+            - https://de.wikipedia.org/wiki/Boeing_707
+            - https://de.wikipedia.org/wiki/Boeing_717
+        preprocessing:
+            markdown:
+                enabled: true
+                remove_html_comments: true
+                normalize_whitespace: true
+
+    pydantic:
+        type: sitemap
+        crawl:
+            parse_type: markdown-fit
+        sources:
+            - https://pydantic.dev/sitemap.xml
+        preprocessing:
+            markdown:
+                enabled: false
+```
+
+Available project settings:
+
+- `type`: `pages` or `sitemap`
+- `sources`: list of page URLs or sitemap URLs
+- `crawl.parse_type`: `markdown` or `markdown-fit`
+- `preprocessing.markdown.enabled`: enables Markdown cleanup
+- `preprocessing.markdown.*`: optional cleanup rules such as `ensure_h1`, `remove_html_comments`, `remove_reference_sections`, and `normalize_whitespace`
+
+For the full configuration, see [`crawl.yml.example`](crawl.yml.example).
+
+---
+
+## Usage
+
+After cloning the repository and creating `crawl.yml`, use:
+
+```bash
+crawl planes
+crawl pydantic
+```
+
+Or with `uv` inside the project:
+
+```bash
+uv run crawl planes
+uv run crawl pydantic
+```
+
+---
+
+## Python API
+
+`crawl4md` can also be used as a Python package.
+
+The public classes are:
+
+- `MarkdownFetcher`
+- `MarkdownConverter`
+- `ParseType`
+- `MarkdownPreprocessingConfig`
+
+### Configure Parse Type
+
+Use `ParseType` to control how Markdown is generated:
+
+- `"markdown"`: raw markdown output
+- `"markdown-fit"`: cleaned and reduced markdown output via `crawl4ai`
+
+### Configure Preprocessing
+
+Use `MarkdownPreprocessingConfig` to enable optional cleanup steps.
+
+Simple example:
+
+```python
+from crawl4md import MarkdownPreprocessingConfig
+
+config = MarkdownPreprocessingConfig(
+    enabled=True,
+    remove_html_comments=True,
+    normalize_whitespace=True,
+)
+```
+
+### Fetch Markdown From a URL
+
+Use `MarkdownFetcher` if you want to fetch a page and directly receive Markdown.
+
+```python
+from crawl4md import MarkdownFetcher, MarkdownPreprocessingConfig
+
+config = MarkdownPreprocessingConfig(enabled=True)
+fetcher = MarkdownFetcher(config=config, parse_type="markdown-fit")
+
+markdown = fetcher.fetch_sync("https://example.com")
+print(markdown)
+```
+
+Async version:
+
+```python
+import asyncio
+
+from crawl4md import MarkdownFetcher, MarkdownPreprocessingConfig
+
+config = MarkdownPreprocessingConfig(enabled=True)
+fetcher = MarkdownFetcher(config=config, parse_type="markdown-fit")
+
+markdown = asyncio.run(fetcher.fetch("https://example.com"))
+print(markdown)
+```
+
+### Convert HTML to Markdown
+
+Use `MarkdownConverter` if you already have HTML and only want the conversion step.
+
+```python
+from crawl4md import MarkdownConverter, MarkdownPreprocessingConfig
+
+html = "<html><body><h1>Hello</h1><p>World</p></body></html>"
+
+config = MarkdownPreprocessingConfig(enabled=True, ensure_h1=True)
+converter = MarkdownConverter(config=config, parse_type="markdown")
+
+markdown = converter.convert_sync(html=html, url="https://example.com")
+print(markdown)
+```
+
+Async version:
+
+```python
+import asyncio
+
+from crawl4md import MarkdownConverter, MarkdownPreprocessingConfig
+
+html = "<html><body><h1>Hello</h1><p>World</p></body></html>"
+
+config = MarkdownPreprocessingConfig(enabled=True, ensure_h1=True)
+converter = MarkdownConverter(config=config, parse_type="markdown")
+
+markdown = asyncio.run(
+    converter.convert(html=html, url="https://example.com")
+)
+print(markdown)
+```
+
+---
+
+## Output Structure
+
+Markdown files are stored deterministically based on the URL path:
+
+```bash
+docs/<project>/<url-path>.md
+```
+
+Example:
+
+```bash
+docs/planes/wiki/Boeing_707.md
+```
+
+Rules:
+
+* Domain is ignored
+* URL path is preserved
+* `/` → `index.md`
+* Query parameters are ignored
+
+---
+
+## Example Output
+
+```bash
+1/2 Crawl https://de.wikipedia.org/wiki/Boeing_707
+- Fetching ... done
+- Processing ... done
+- Writing docs/planes/wiki/Boeing_707.md ... done
+```
+
+---
+
+## Use Cases
+* RAG data ingestion
+* Website snapshotting
+* Knowledge base generation
+* Offline documentation
+
+---
+
+## Project Structure
+
+```bash
+src/crawl4md/
+├─ cli.py
+├─ config.py
+├─ sitemap.py
+├─ crawler.py
+├─ paths.py
+└─ writer.py
+```
+
+---
+
+## Notes
+
+* No recursive crawling (by design)
+* No hidden caching or transformations
+* Focus on clean Markdown output only
+
+---
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE.md](LICENSE.md) file for details.
+
+### Authors
+
+* Björn Hempel <bjoern@hempel.li> - _Initial work_ - [https://github.com/bjoern-hempel](https://github.com/bjoern-hempel)
+
+---
+
+## Built on top of crawl4ai
+
+This project builds on the excellent [`crawl4ai`](https://github.com/unclecode/crawl4ai) library and extends it with a simpler batch-oriented workflow for repeatable Markdown exports.
+
+Why use `crawl4md` as a complement to `crawl4ai`:
+
+- project-based batch crawling via `crawl.yml`
+- support for both page lists and sitemap-driven crawls
+- deterministic output paths for generated Markdown files
+- optional Markdown cleanup rules for better downstream text quality
+- a small CLI and Python API focused on URL or HTML to Markdown workflows
+- clearer separation between fetching, conversion, preprocessing, and writing
+
+In short: `crawl4ai` provides the powerful crawling and Markdown generation foundation, while `crawl4md` adds a lightweight structure around it for batch jobs, cleaner output, and easier integration into documentation or RAG pipelines.