gnosis-markdown 1.1.0__tar.gz

gnosis_markdown-1.1.0/CHANGELOG.md

@@ -0,0 +1,58 @@
+# Changelog
+
+## [1.1.0] - 2026-08-04
+
+### Added
+- **Provenance frontmatter on every output file** (on by default, `--no-frontmatter`
+  to opt out): `title`, `url`, `fetched_at` (UTC ISO 8601), `content_hash`
+  (SHA-256 of body), `status_code`, `language`, `author`, `description`,
+  `site_name`, `published_time`/`modified_time`, `etag`/`last_modified`,
+  `generator`. Standard YAML, parseable by python-frontmatter/Jekyll/Hugo.
+- **Authentication**: Bearer, HTTP Basic (Confluence Cloud PAT pattern:
+  email + API token), and arbitrary header auth. Secrets are read from
+  environment variables only — via `--bearer-token-env`,
+  `--basic-user`/`--basic-token-env`, or `${ENV_VAR}` expansion in config files
+  and `--header` values.
+- **User frontmatter extras**: `--frontmatter 'key: value'` (repeatable) and
+  `output.frontmatter_extra` in config; merged without overriding core fields.
+- **Crawl manifest**: `--all` runs write `_manifest.json` with per-page URL,
+  file, content hash, timestamp, status, and title.
+- **Metadata extraction**: `HTMLToMarkdownConverter.extract_metadata()`
+  (title with entity unescaping, author, language, OG fields).
+- **Downloader `fetch_result()`**: returns `FetchResult` with final URL,
+  status code, fetch timestamp, and response headers. The crawler now carries
+  this provenance through crawl mode.
+- **Boilerplate word stripping**: `converter.strip_class_words` — word-level
+  class matching catches framework-namespaced boilerplate
+  (`bd-sidebar-primary`) without false positives (`research-content`).
+- **Content selector precedence**: `content_selectors` are now tried in order
+  and the first substantial match wins (matching the documented behavior);
+  platform containers (`.markdown-body`, `.ak-renderer-document`,
+  `.wiki-content`) precede chrome-wrapping landmarks (`main`, `#content`).
+- **Test suite**: 47 pytest tests covering converter quality, provenance,
+  auth, crawler resolution, and CLI end-to-end behavior.
+
+### Fixed
+- HTML comments no longer leak into output as text (Confluence
+  `<!-- data-loadable-* -->` SSR markers polluted converted pages).
+- Heading permalink anchors no longer glue `#` onto heading text
+  (`# Quickstart#` → `# Quickstart`).
+- Table cells with multiple paragraphs no longer break markdown rows
+  (joined with `<br>`); pipes in cell text are escaped.
+- Duplicate table header rows (Confluence thead+tbody repetition) and
+  single-row sticky-header clone tables are removed.
+- Relative links below extensionless "directory" URLs now resolve correctly
+  (`/en/latest` + `quickstart.html` no longer escapes the crawl scope) —
+  this previously broke `--all` crawls of most docs sites.
+- `data:`-URI images (spacers/tracking pixels) are skipped.
+
+### Changed
+- torch/transformers are no longer core dependencies; install the QMD
+  integration via `pip install gnosis[qmd]` (or `requirements-qmd.txt`).
+- Blank-line collapsing in output is stricter (max one blank line).
+- Default User-Agent updated to `Gnosis/1.1`.
+
+## [1.0.0] - 2026-03-27
+
+Initial public release: single-page download, `--all` crawling, configurable
+extraction, QMD integration.

gnosis_markdown-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Steffen Hoehne, SHCV.IT
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gnosis_markdown-1.1.0/MANIFEST.in

@@ -0,0 +1,9 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include requirements.txt
+include requirements-qmd.txt
+include config/default.yaml
+recursive-include gnosis *.py
+recursive-include tests *.py
+graft config

gnosis_markdown-1.1.0/PKG-INFO

@@ -0,0 +1,379 @@
+Metadata-Version: 2.4
+Name: gnosis-markdown
+Version: 1.1.0
+Summary: Website to Markdown converter with provenance frontmatter for LLM knowledge bases
+Home-page: https://github.com/shcv-it/gnosis
+Author: Steffen Hoehne
+Author-email: Steffen Hoehne <steffen.hoehne@shcv.it>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/SHCV-it/gnosis
+Project-URL: Repository, https://github.com/SHCV-it/gnosis
+Project-URL: Issues, https://github.com/SHCV-it/gnosis/issues
+Project-URL: Changelog, https://github.com/SHCV-it/gnosis/blob/main/CHANGELOG.md
+Keywords: markdown,web-scraping,crawler,html-to-markdown,knowledge-base,llm
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Documentation
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: lxml>=5.0.0
+Requires-Dist: pyyaml>=6.0.0
+Provides-Extra: qmd
+Requires-Dist: torch>=2.0.0; extra == "qmd"
+Requires-Dist: transformers>=4.30.0; extra == "qmd"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# Gnosis
+
+<p align="center">
+  <strong>Website → clean, provenance-stamped Markdown.</strong><br>
+  <em>Built for LLM knowledge bases, documentation pipelines, and audit-ready content archives.</em>
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/gnosis/"><img alt="PyPI" src="https://img.shields.io/badge/python-3.12+-blue"></a>
+  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-green"></a>
+  <a href="https://github.com/SHCV-it/gnosis"><img alt="Version" src="https://img.shields.io/badge/version-1.1.0-blue"></a>
+</p>
+
+---
+
+Point gnosis at a URL and get clean, LLM-friendly Markdown files with a **YAML
+provenance frontmatter block** on every file — recording exactly where the
+content came from, when it was fetched, and how to verify it. No external
+bookkeeping, no hidden state. Every file is self-describing.
+
+## Table of contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [Provenance: the contract](#provenance-the-contract)
+- [Authenticated fetching](#authenticated-fetching)
+- [CLI reference](#cli-reference)
+- [Configuration reference](#configuration-reference)
+- [Exit codes](#exit-codes)
+- [How clean is the output?](#how-clean-is-the-output)
+- [Development & testing](#development--testing)
+- [License](#license)
+
+## Features
+
+| Feature | Description |
+|---|---|
+| **Single page or full site** | One URL, or crawl every page under a path with `--all` |
+| **Provenance frontmatter** | Every file records `url`, `fetched_at` (UTC), SHA-256 `content_hash`, `status_code`, page metadata (title/author/language), and caching headers when the server sends them |
+| **Authentication built-in** | Bearer tokens, HTTP Basic (Confluence Cloud API tokens), or arbitrary headers — secrets **always** from environment variables |
+| **Clean extraction** | Main-content detection with ordered selectors, class-word boilerplate stripping (sidebars, breadcrumbs, cookie banners, permalink anchors), and framework-aware fixes for Confluence, Sphinx/RTD, and GitHub-style pages |
+| **Valid GFM tables** | Multi-paragraph cells joined with `<br>`, pipes escaped, sticky-header clone tables removed — tables survive ingestion |
+| **Scalable crawling** | Configurable concurrent fetches, politeness delay, retries with backoff, robots-aware |
+| **Scheduler-friendly** | Headless CLI, meaningful exit codes, JSON crawl manifest — drops into cron, n8n, Airflow, CI |
+| **Metadata extraction** | Title (entity-unescaped), author, language, description, Open Graph fields |
+| **Optional QMD integration** | Index output into a QMD knowledge base with local-LLM context generation (`pip install gnosis[qmd]`) |
+
+## Installation
+
+Requires **Python 3.12+**.
+
+```bash
+git clone https://github.com/SHCV-it/gnosis.git
+cd gnosis
+python -m venv venv && source venv/bin/activate
+pip install -e .
+```
+
+Optional, only if you use `--qmd-index` (pulls torch + transformers):
+
+```bash
+pip install -e .[qmd]
+```
+
+## Quick start
+
+```bash
+# One page → one markdown file with provenance
+gnosis https://docs.python.org/3/tutorial/
+
+# Crawl an entire section of a docs site
+gnosis https://docs.python.org/3/tutorial/ --all -o ./python-docs/
+
+# Preview how many pages would be crawled (no downloads)
+gnosis https://docs.python.org/3/tutorial/ --all --dry-run
+
+# Faster crawl with parallel fetches
+gnosis https://docs.example.com/ --all --config myconfig.yaml
+```
+
+## Provenance: the contract
+
+Every file gnosis writes is self-describing. Default frontmatter:
+
+```yaml
+---
+title: Quickstart — Trafilatura 2.2.0 documentation
+url: https://trafilatura.readthedocs.io/en/latest/quickstart.html
+fetched_at: '2026-08-04T10:24:17Z'
+content_hash: 1549512c...16fd
+status_code: 200
+generator: gnosis/1.1.0
+language: en
+etag: '"61e917f4cd107c3bce6182b633819fcf"'
+last_modified: Fri, 31 Jul 2026 16:07:37 GMT
+---
+```
+
+| Field | Required | Description |
+|---|---|---|
+| `title` | Always | Page title (from `og:title` or `<title>`, HTML entities unescaped) |
+| `url` | Always | Final URL after redirects (`requested_url` added if it differs) |
+| `fetched_at` | Always | UTC fetch timestamp, ISO 8601 |
+| `content_hash` | Always | SHA-256 of the markdown body — use for dedup/change detection |
+| `status_code` | Always | HTTP status of the final response |
+| `generator` | Always | Gnosis version that produced this file |
+| `language` | If present | From `<html lang>` or `og:locale` |
+| `author` | If present | From `<meta name=author>`, `article:author`, or `dc.creator` |
+| `description` | If present | From `<meta name=description>` or `og:description` |
+| `site_name` | If present | From `og:site_name` |
+| `published_time` | If present | From `article:published_time` |
+| `modified_time` | If present | From `article:modified_time` |
+| `etag` | If sent | Response `ETag` header |
+| `last_modified` | If sent | Response `Last-Modified` header |
+| `requested_url` | If redirected | Original URL before redirects |
+
+Add your own constant fields per run (`--frontmatter`) or per config
+(`output.frontmatter_extra`) — custom keys never override the core provenance
+fields above:
+
+```bash
+gnosis https://example.com/docs --frontmatter 'tags: [customs, passar]' --frontmatter 'owner: kb-team'
+```
+
+The frontmatter is standard YAML between `---` fences: parseable by
+python-frontmatter, Jekyll, Hugo, Obsidian, and any downstream knowledge
+pipeline.
+
+Opt out per run with `--no-frontmatter` or globally in config:
+```yaml
+output:
+  frontmatter: false
+```
+
+## Authenticated fetching
+
+Secrets are read from **environment variables only**. They are never passed as
+plain CLI arguments (which leak into shell history and process tables) and
+never committed in config files.
+
+### Confluence Cloud with a Personal Access Token
+
+```bash
+# Set up a PAT at https://id.atlassian.com/manage/api-tokens
+export CONFLUENCE_PAT="your-api-token"
+
+gnosis "https://your-domain.atlassian.net/wiki/spaces/SPACE/pages/PAGE_ID" \
+  --basic-user you@example.com \
+  --basic-token-env CONFLUENCE_PAT
+```
+
+### Bearer token (authenticated API docs, internal tools)
+
+```bash
+export MY_API_TOKEN="..."
+gnosis https://internal.example.com/docs --bearer-token-env MY_API_TOKEN
+```
+
+### Custom headers
+
+```bash
+gnosis https://example.com --header "X-API-Key: ${MY_KEY}" --header "X-Team: docs"
+```
+
+### Via config file (multi-run / CI)
+
+```yaml
+downloader:
+  auth:
+    type: basic                      # bearer | basic | header
+    username: "you@example.com"
+    password: "${CONFLUENCE_PAT}"    # ${ENV_VAR} expanded at load time
+```
+
+## CLI reference
+
+```
+gnosis URL [OPTIONS]
+```
+
+| Flag | Description |
+|---|---|
+| `-a, --all` | Crawl all child pages under the URL path |
+| `-n, --dry-run` | Discover and count pages only (requires `--all`) |
+| `-o, --output DIR` | Output directory (default: `./`) |
+| `-c, --config FILE` | Path to YAML configuration file |
+| `-f, --overwrite` | Overwrite existing output files |
+| `-q, --quiet` | Suppress progress output |
+| `-v, --verbose` | Show detailed conversion diagnostics |
+| `--no-frontmatter` | Write bare markdown without provenance block |
+| `--frontmatter KEY: VALUE` | Extra constant frontmatter field (repeatable) |
+| `--header NAME: VALUE` | Extra request header, `${ENV_VAR}` expanded (repeatable) |
+| `--bearer-token-env VAR` | Bearer token from environment variable |
+| `--basic-user USER` | HTTP Basic username (requires `--basic-token-env`) |
+| `--basic-token-env VAR` | HTTP Basic password/token from environment variable |
+| `--qmd-index` | Index output into QMD (requires `[qmd]` extra) |
+
+## Configuration reference
+
+Copy [`config/default.yaml`](config/default.yaml) and pass it with `-c`.
+Full reference:
+
+```yaml
+# ── Downloader ────────────────────────────
+downloader:
+  timeout: 30                # Request timeout (seconds)
+  retries: 3                 # Retries on 5xx / network errors
+  user_agent: "Gnosis/1.1"   # User-Agent header
+  rate_limit_ms: 500          # Minimum delay between requests (0 = no limit)
+  respect_robots: true        # Obey robots.txt (future)
+  headers: {}                 # Extra HTTP headers (${ENV_VAR} expanded)
+  auth:                       # Optional: bearer | basic | header
+    type: bearer
+    token: "${MY_API_TOKEN}"
+
+# ── Crawler ───────────────────────────────
+crawler:
+  max_depth: 10              # Crawl depth from seed URL
+  max_pages: 500             # Stop after this many pages
+  concurrent_requests: 5     # Parallel fetch batch size (1 = sequential)
+
+# ── Converter ─────────────────────────────
+converter:
+  excluded_tags: [...]        # HTML tags stripped before conversion
+  content_selectors: [...]    # Tried in order; first match ≥ 200 chars wins
+  strip_classes: [...]        # Exact class-token matches to remove
+  strip_class_words: [...]    # Word-level matches inside class names
+  include_images: true        # Emit <img> as ![alt](src)
+  absolute_urls: true         # Resolve relative links to absolute URLs
+
+# ── Output ─────────────────────────────────
+output:
+  directory: "./"             # Where .md files go
+  overwrite: false            # Skip existing files unless true
+  extension: ".md"            # Output file extension
+  frontmatter: true           # Write YAML provenance block
+  frontmatter_extra: {}       # Constant fields added to every file
+
+# ── QMD (optional) ──────────────────────────
+qmd:
+  enabled: false              # Enable QMD knowledge base indexing
+  llm_model: "Qwen/Qwen3-0.6B"  # HuggingFace model for context generation
+  llm_device: "cpu"           # cpu | cuda | auto
+```
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | Success (crawl mode: at least one page saved) |
+| `1` | Failure — download error, file exists without `-f`, nothing saved, bad flags |
+| `130` | Interrupted (Ctrl-C) |
+
+In `--all` mode a `_manifest.json` is written to the output directory listing
+every page with `url`, `file`, `content_hash`, `fetched_at`, `status_code`, and
+`title` — ready for schedulers and downstream audit.
+
+## How clean is the output?
+
+Gnosis is opinionated about boilerplate. By default it:
+
+- **strips** script/style/nav/footer/aside/form/template tags and HTML comments
+  (including Confluence's `<!-- data-loadable-begin=... -->` SSR markers)
+- **removes** elements matching exact `strip_classes` tokens AND boilerplate
+  *words* (`sidebar`, `toc`, `breadcrumb`, `cookie`, `headerlink`, `sourcelink`, …)
+  inside namespaced class names — `bd-sidebar-primary` is gone, but `research-content`
+  stays
+- **cleans** permalink anchors from headings (`# Quickstart#` → `# Quickstart`)
+- **picks** the main content by precedence-ordered selectors
+  (`.markdown-body`, `.ak-renderer-document`, `.wiki-content`, … before `main`/`#content`)
+- **converts** tables to valid GFM: multi-line cells joined with `<br>`, `|`
+  escaped, duplicate/sticky-header rows removed
+- **resolves** relative links to absolute URLs; skips `data:`-URI images
+  (spacers/tracking pixels)
+- **unescapes** HTML entities in titles (`&#8212;` → `—`)
+
+Everything is configurable — see [`config/default.yaml`](config/default.yaml).
+
+## Development & testing
+
+```bash
+git clone https://github.com/SHCV-it/gnosis.git
+cd gnosis
+python -m venv venv && source venv/bin/activate
+pip install -e . pytest python-frontmatter
+
+# Run the test suite (offline — only localhost fixtures)
+python -m pytest tests/ -q -v
+```
+
+### Project structure
+
+```
+gnosis/
+  cli/           Click CLI (single page, crawl, dry-run, manifest)
+  config/        YAML loading + typed settings dataclass
+  core/
+    downloader   Async HTTP client, auth, retries, FetchResult
+    converter    HTML → Markdown, boilerplate stripping, metadata extraction
+    crawler      BFS crawler with concurrent batch fetching
+    provenance   Frontmatter generation, content_hash, render_document
+  integrations/  QMD pipeline (optional, heavy deps)
+```
+
+The test suite covers converter quality (comment/anchor/boilerplate/table
+handling, shadow-table dedup, metadata), provenance generation (fields,
+round-trip parsing, extras merging), auth header injection (3 schemes),
+crawler link resolution, and CLI end-to-end behavior. Runs entirely offline.
+
+## Contributing
+
+Contributions are welcome. Please open an issue first to discuss what you'd
+like to change.
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing`)
+3. Run the tests (`python -m pytest tests/ -q`)
+4. Commit your changes with clear messages
+5. Push and open a pull request against `main`
+
+## Related projects
+
+Gnosis is designed to feed documentation pipelines and LLM knowledge bases.
+Pair it with:
+
+- **n8n / cron / Airflow** — schedule gnosis runs and pipe results into your
+  downstream pipeline
+- **QMD** — local vector search via the `--qmd-index` flag
+- **Any Markdown-to-anything pipeline** — the YAML frontmatter is parseable by
+  python-frontmatter, Jekyll, Hugo, Obsidian, and standard static-site generators
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+---
+
+**Author:** Steffen Hoehne, [SHCV.IT](https://shcv.it)