langchain-enconvert 0.1.0__tar.gz

langchain_enconvert-0.1.0/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## [0.1.0] - 2026-08-27
+
+### Added
+
+- Initial release. `EnconvertLoader` loads web pages (`urls=`) into clean-markdown Documents with a
+  `render_quality` score, or ingests a whole site (`ingest_url=`) into RAG-ready chunk Documents.
+- Depends on `langchain-core` only; the API key is read from `api_key=` or `$ENCONVERT_API_KEY`.

langchain_enconvert-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 EnConvert
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

langchain_enconvert-0.1.0/PKG-INFO

@@ -0,0 +1,51 @@
+Metadata-Version: 2.5
+Name: langchain-enconvert
+Version: 0.1.0
+Summary: EnConvert document loader for LangChain: perceive URLs or ingest a whole site into Documents, every read scored.
+Project-URL: Homepage, https://www.enconvert.com
+Project-URL: Repository, https://github.com/enconvert/langchain-enconvert
+Author-email: EnConvert <support@enconvert.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: document-loader,enconvert,ingest,langchain,markdown,rag,web
+Requires-Python: >=3.9
+Requires-Dist: enconvert>=0.1
+Requires-Dist: langchain-core>=0.3
+Requires-Dist: requests>=2.28
+Description-Content-Type: text/markdown
+
+# EnConvert loader for LangChain
+
+`langchain-enconvert` turns web pages and whole sites into LangChain `Document`s through
+[EnConvert](https://www.enconvert.com). Every perceived page carries a `render_quality` score (0.0-1.0)
+in its metadata, so a blocked or empty page comes back flagged rather than trusted.
+
+## Install
+
+```bash
+pip install langchain-enconvert
+```
+
+## Use
+
+```python
+from langchain_enconvert import EnconvertLoader
+
+# A few URLs into clean-markdown Documents:
+docs = EnconvertLoader(urls=["https://example.com", "https://example.com/pricing"]).load()
+
+# Or a whole site into RAG-ready chunk Documents:
+docs = EnconvertLoader(ingest_url="https://docs.example.com", mode="sitemap", max_pages=100).load()
+```
+
+- **URLs** are perceived into markdown; metadata carries `source` and `render_quality`.
+- **`ingest_url`** crawls the site (async; the loader polls to completion), then returns one Document per
+  chunk, each carrying the chunk's own metadata (`source`, title, etc.).
+- `.lazy_load()` streams Documents one at a time.
+
+Auth: a **private** key (`sk_...`) from your [dashboard](https://www.enconvert.com/dashboard/api-keys).
+Public `pk_` keys are rejected. The key is read from `api_key=` or `$ENCONVERT_API_KEY`.
+
+## Licence
+
+[MIT](LICENSE)

langchain_enconvert-0.1.0/README.md

@@ -0,0 +1,35 @@
+# EnConvert loader for LangChain
+
+`langchain-enconvert` turns web pages and whole sites into LangChain `Document`s through
+[EnConvert](https://www.enconvert.com). Every perceived page carries a `render_quality` score (0.0-1.0)
+in its metadata, so a blocked or empty page comes back flagged rather than trusted.
+
+## Install
+
+```bash
+pip install langchain-enconvert
+```
+
+## Use
+
+```python
+from langchain_enconvert import EnconvertLoader
+
+# A few URLs into clean-markdown Documents:
+docs = EnconvertLoader(urls=["https://example.com", "https://example.com/pricing"]).load()
+
+# Or a whole site into RAG-ready chunk Documents:
+docs = EnconvertLoader(ingest_url="https://docs.example.com", mode="sitemap", max_pages=100).load()
+```
+
+- **URLs** are perceived into markdown; metadata carries `source` and `render_quality`.
+- **`ingest_url`** crawls the site (async; the loader polls to completion), then returns one Document per
+  chunk, each carrying the chunk's own metadata (`source`, title, etc.).
+- `.lazy_load()` streams Documents one at a time.
+
+Auth: a **private** key (`sk_...`) from your [dashboard](https://www.enconvert.com/dashboard/api-keys).
+Public `pk_` keys are rejected. The key is read from `api_key=` or `$ENCONVERT_API_KEY`.
+
+## Licence
+
+[MIT](LICENSE)

langchain_enconvert-0.1.0/langchain_enconvert/__init__.py

@@ -0,0 +1,3 @@
+from langchain_enconvert.document_loaders import EnconvertLoader
+
+__all__ = ["EnconvertLoader"]

langchain_enconvert-0.1.0/langchain_enconvert/document_loaders.py

@@ -0,0 +1,113 @@
+"""EnConvert document loader for LangChain."""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+from typing import Iterator, List, Optional
+
+import requests
+from enconvert import Enconvert
+from langchain_core.document_loaders import BaseLoader
+from langchain_core.documents import Document
+
+
+class EnconvertLoader(BaseLoader):
+    """Load web pages or a whole site into LangChain Documents via EnConvert.
+
+    Pass ``urls=`` to perceive pages into clean-markdown Documents (each carrying
+    a ``render_quality`` score), or ``ingest_url=`` to crawl a site into
+    RAG-ready chunk Documents.
+    """
+
+    def __init__(
+        self,
+        urls: Optional[List[str]] = None,
+        *,
+        ingest_url: Optional[str] = None,
+        mode: str = "sitemap",
+        max_pages: int = 50,
+        api_key: Optional[str] = None,
+        base_url: str = "https://api.enconvert.com",
+        poll_interval: float = 5.0,
+    ) -> None:
+        self.urls = urls
+        self.ingest_url = ingest_url
+        self.mode = mode
+        self.max_pages = max_pages
+        self.base_url = base_url
+        self.poll_interval = poll_interval
+        self._api_key = api_key or os.environ.get("ENCONVERT_API_KEY", "")
+
+    def lazy_load(self) -> Iterator[Document]:
+        if not self._api_key:
+            raise ValueError(
+                "EnConvert api_key is missing. Pass api_key= or set $ENCONVERT_API_KEY "
+                "(a private key starting with sk_)."
+            )
+        client = Enconvert(api_key=self._api_key, base_url=self.base_url)
+        if self.urls:
+            for url in self.urls:
+                op = client.v2.perceive_direct(url, outputs=["markdown"])
+                yield Document(
+                    page_content=op.content.decode("utf-8", "replace"),
+                    metadata={"source": url, "render_quality": op.render_quality},
+                )
+        elif self.ingest_url:
+            yield from _ingest(
+                client, self.ingest_url, self.mode, self.max_pages, self.poll_interval
+            )
+        else:
+            raise ValueError("Provide urls=[...] or ingest_url=...")
+
+
+def _ingest(
+    client: Enconvert,
+    url: str,
+    mode: str,
+    max_pages: int,
+    poll_interval: float,
+) -> Iterator[Document]:
+    job = client.v2.ingest(mode=mode, url=url, max_pages=max_pages)
+    while True:
+        status = client.v2.get_ingest_job(job.job_id)
+        if status.status in ("completed", "failed", "cancelled"):
+            break
+        time.sleep(poll_interval)
+    if status.status != "completed":
+        raise RuntimeError(f"EnConvert ingest job {job.job_id} ended: {status.status}")
+    resp = requests.get(status.output_url, timeout=120)
+    resp.raise_for_status()
+    yield from _chunks_to_documents(resp.text)
+
+
+def _chunks_to_documents(jsonl_text: str) -> Iterator[Document]:
+    """Map an EnConvert ingest JSONL (one chunk per line) to LangChain Documents.
+
+    Schema-agnostic: text comes from ``content`` (or ``text``); the rest becomes
+    metadata, with ``source`` set from the chunk's URL for LangChain provenance.
+    """
+    for line in jsonl_text.splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        chunk = json.loads(line)
+        text = chunk.pop("content", None) or chunk.pop("text", "")
+        src = chunk.get("source_url") or chunk.get("url")
+        if src and "source" not in chunk:
+            chunk["source"] = src
+        yield Document(page_content=text, metadata=chunk)
+
+
+if __name__ == "__main__":
+    sample = (
+        '{"content": "hello", "source_url": "https://x.com", "chunk_index": 0}\n'
+        "\n"
+        '{"text": "world", "title": "W"}'
+    )
+    out = list(_chunks_to_documents(sample))
+    assert len(out) == 2, out
+    assert out[0].page_content == "hello" and out[0].metadata["source"] == "https://x.com"
+    assert out[1].page_content == "world" and out[1].metadata["title"] == "W"
+    print("ok")

langchain_enconvert-0.1.0/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "langchain-enconvert"
+version = "0.1.0"
+description = "EnConvert document loader for LangChain: perceive URLs or ingest a whole site into Documents, every read scored."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+authors = [{ name = "EnConvert", email = "support@enconvert.com" }]
+keywords = ["langchain", "enconvert", "rag", "document-loader", "markdown", "web", "ingest"]
+dependencies = [
+    "langchain-core>=0.3",
+    "enconvert>=0.1",
+    "requests>=2.28",
+]
+
+[project.urls]
+Homepage = "https://www.enconvert.com"
+Repository = "https://github.com/enconvert/langchain-enconvert"
+
+[tool.hatch.build.targets.wheel]
+packages = ["langchain_enconvert"]