everalgo-parser 0.2.0__tar.gz → 0.2.1__tar.gz

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/.gitignore

@@ -56,3 +56,7 @@ local/
 
 # Claude Code session-local state (scheduled tasks, agent caches, locks)
 .claude/
+benchmarks/claudecode
+
+# Git worktrees (isolated workspaces)
+.worktrees/

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/CHANGELOG.md

@@ -9,14 +9,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 <!-- git-cliff-unreleased-end -->
 
+## [0.2.1] - 2026-06-15
+
+### Changed
+
+- **License relicensed from MIT to Apache-2.0** as part of the pre-open-source security audit.
+
 ## [0.2.0] - 2026-05-27
 
 > First archived changelog. Entries below accumulated since the initial `0.1.0` PyPI release (published manually, without a git tag or per-package changelog), so this `0.2.0` section also consolidates the previously-unarchived `0.1.0` surface.
 
 ### Added
 
-- **PDF parser** — single multimodal LLM call, mirrors
-  `evermemos-multimodal/src/parser/pdf/gemini_parser.py`.
+- **PDF parser** — single multimodal LLM call, mirrors the upstream
+  internal multimodal PDF parser reference implementation.
 - **Image parser** — single LLM call for normal-ratio images; tall
   screenshots (height/width > 10) are split into overlapping vertical
   slices, OCR'd per slice, and merged with a second LLM call using
@@ -41,8 +47,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   unknown) → HTML handler with Open Graph / Twitter Card / `<meta>` tag
   extraction into `ParsedContent.metadata`. `file://` and other local-
   filesystem URIs are rejected (AGENTS.md §1). Inner modality recorded in
-  `metadata["inner_modality"]`. Metadata-extractor schema lifted from
-  `evermemos-opensource/src/common_utils/url_extractor.py`.
+  `metadata["inner_modality"]`. Metadata-extractor schema lifted from the
+  upstream internal URL extractor reference implementation.
 - **MIME-aware top-level dispatch** — `parser.aparse` now falls back to
   `mime` when `extension` is missing or `UNKNOWN`, so the canonical
   `RawFile(uri=..., mime="application/pdf")` example actually routes to
@@ -59,8 +65,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Notes
 
-- `video` submodule remains a stub — upstream evermemos-multimodal has no
-  video implementation to port; backend selection pending an ADR.
+- `video` submodule remains a stub — the upstream internal multimodal parser
+  has no video implementation to port; backend selection pending an ADR.
 - The parser layer deliberately omits retry / fallback / multi-key
   rotation / Prometheus metrics. Those are deployment concerns
   (ADR-012); algorithm-layer operators surface failures via `LLMError`
@@ -74,5 +80,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Optional `cairosvg >= 2.7.0` via `[svg]` extra.
 - System: LibreOffice (`soffice` on PATH) for Office document parsing.
 
-[Unreleased]: https://github.com/EverMind-AI/EverAlgo/compare/everalgo-parser/v0.2.0...HEAD
+[Unreleased]: https://github.com/EverMind-AI/EverAlgo/compare/everalgo-parser/v0.2.1...HEAD
+[0.2.1]: https://github.com/EverMind-AI/EverAlgo/compare/everalgo-parser/v0.2.0...everalgo-parser/v0.2.1
 [0.2.0]: https://github.com/EverMind-AI/EverAlgo/releases/tag/everalgo-parser/v0.2.0

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: everalgo-parser
-Version: 0.2.0
+Version: 0.2.1
 Summary: EverAlgo parser: multimodal raw-file parsing (image / audio / document / video / url) into ParsedContent.
 Project-URL: Homepage, https://github.com/EverMind-AI/EverAlgo
 Project-URL: Repository, https://github.com/EverMind-AI/EverAlgo
@@ -8,12 +8,12 @@ Project-URL: Issues, https://github.com/EverMind-AI/EverAlgo/issues
 Project-URL: Documentation, https://github.com/EverMind-AI/EverAlgo/tree/main/packages/everalgo-parser
 Project-URL: Changelog, https://github.com/EverMind-AI/EverAlgo/blob/main/packages/everalgo-parser/CHANGELOG.md
 Author-email: EverMind <Evermind@shanda.com>
-License-Expression: MIT
+License-Expression: Apache-2.0
 License-File: LICENSE
 Keywords: everalgo,llm,multimodal,parsing
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
+Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
@@ -29,33 +29,32 @@ Description-Content-Type: text/markdown
 
 # everalgo-parser
 
-Multimodal parsing — image / audio / document / video / url raw inputs into `ParsedContent`. Used by `everalgo-knowledge` for file ingestion and by evermem step 1 for inline parsing.
+Multimodal parsing — image / audio / document / video / url raw inputs into `ParsedContent`. Used by `everalgo-knowledge` for file ingestion and by EverOS step 1 for inline parsing.
 
 See the umbrella project: [EverAlgo monorepo](../../README.md) and the architecture document at [`docs/concepts/architecture.md`](../../docs/concepts/architecture.md).
 
 ## Quick start
 
 ```python
-import everalgo
 from everalgo.llm.config import LLMConfig
 from everalgo.llm.providers.openai_compat import OpenAICompatClient
 from everalgo.parser import aparse, RawFile
 
-# Configure an LLM once (process-wide). The parser uses OpenAI-compatible
-# clients; OpenRouter is the reference deployment (Gemini multimodal via
-# OpenRouter passthrough).
-everalgo.configure(OpenAICompatClient(LLMConfig(
+# Create an LLM client per call (or share one instance). The parser uses
+# OpenAI-compatible clients; OpenRouter is the reference deployment
+# (Gemini multimodal via OpenRouter passthrough).
+client = OpenAICompatClient(LLMConfig(
     model="google/gemini-3-flash-preview",
     api_key="sk-or-v1-...",
     base_url="https://openrouter.ai/api/v1",
-)))
+))
 
 # Bytes-in: caller already hydrated the file.
-parsed = await aparse(RawFile(content=pdf_bytes, extension="pdf"))
+parsed = await aparse(RawFile(content=pdf_bytes, extension="pdf"), llm=client)
 print(parsed.text)
 
 # URL-in: parser fetches over HTTP, then delegates to the HTML handler.
-parsed = await aparse(RawFile(uri="https://example.com/article"))
+parsed = await aparse(RawFile(uri="https://example.com/article"), llm=client)
 print(parsed.metadata["title"], parsed.text[:500])
 ```
 
@@ -101,8 +100,17 @@ The parser detects `soffice` via `shutil.which("soffice")` and the canonical mac
 - The library is **stateless**: it never reads the filesystem and never owns business state. HTTP I/O (LLM calls, URL fetching) is explicitly allowed.
 - No retry / fallback / metrics inside operators — surface failures via `LLMError`, let the caller wrap.
 
+## Security notes for callers
+
+`everalgo-parser` is a stateless library; production safety properties belong to the calling service. The points below are easy to miss and worth wiring into your integration:
+
+- **SSRF — `url.aparse` fetches arbitrary HTTP(S) URIs.** `fetch_uri` rejects non-http(s) schemes (no `file://`, no `gopher://`) but does **not** block private / link-local / loopback IPs (e.g. `127.0.0.1`, `10.0.0.0/8`, `169.254.169.254` cloud metadata, IPv6 ULA). If callers may pass attacker-controlled URLs, resolve the hostname first and reject private ranges before invoking `aparse`, or front the parser with an egress proxy that does the filtering.
+- **Decompression-bomb cap is set at import time.** `everalgo/parser/__init__.py` pins `PIL.Image.MAX_IMAGE_PIXELS = 100_000_000` (10000×10000) — Pillow will raise `DecompressionBombError` above that. Re-assign after import if your workload genuinely needs larger images.
+- **Office conversion runs LibreOffice as a subprocess.** `document._convert_to_pdf_via_soffice` validates the `extension` parameter (alphanumeric, ≤8 chars) before writing the temp file, but the LibreOffice attack surface itself is large. Run the host with a non-root user and treat untrusted office docs as you would any other complex binary input.
+- **`cairosvg` (optional `[svg]` extra) is LGPL-3.0-or-later.** EverAlgo itself is Apache-2.0. LGPLv3 is compatible with Apache-2.0 when `cairosvg` is consumed as an unmodified library through its public Python API (which is what the SVG path does); if you statically vendor or modify `cairosvg`, LGPLv3 §4 / §5 obligations apply to that derivative. Not installing the `[svg]` extra removes the dependency entirely.
+
 ## Reference
 
 - Architecture (definitive): [`docs/concepts/architecture.md`](../../docs/concepts/architecture.md)
-- Schema source for PDF / image / audio / document / html / email: `evermemos-multimodal` (tag `prod-20260306-0331-v1`).
-- Schema source for URL metadata extraction: `evermemos-opensource/src/common_utils/url_extractor.py`.
+- Schema source for PDF / image / audio / document / html / email: upstream internal multimodal parser library (snapshot `prod-20260306-0331-v1`).
+- Schema source for URL metadata extraction: upstream internal URL extractor reference implementation.

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/README.md

@@ -1,32 +1,31 @@
 # everalgo-parser
 
-Multimodal parsing — image / audio / document / video / url raw inputs into `ParsedContent`. Used by `everalgo-knowledge` for file ingestion and by evermem step 1 for inline parsing.
+Multimodal parsing — image / audio / document / video / url raw inputs into `ParsedContent`. Used by `everalgo-knowledge` for file ingestion and by EverOS step 1 for inline parsing.
 
 See the umbrella project: [EverAlgo monorepo](../../README.md) and the architecture document at [`docs/concepts/architecture.md`](../../docs/concepts/architecture.md).
 
 ## Quick start
 
 ```python
-import everalgo
 from everalgo.llm.config import LLMConfig
 from everalgo.llm.providers.openai_compat import OpenAICompatClient
 from everalgo.parser import aparse, RawFile
 
-# Configure an LLM once (process-wide). The parser uses OpenAI-compatible
-# clients; OpenRouter is the reference deployment (Gemini multimodal via
-# OpenRouter passthrough).
-everalgo.configure(OpenAICompatClient(LLMConfig(
+# Create an LLM client per call (or share one instance). The parser uses
+# OpenAI-compatible clients; OpenRouter is the reference deployment
+# (Gemini multimodal via OpenRouter passthrough).
+client = OpenAICompatClient(LLMConfig(
     model="google/gemini-3-flash-preview",
     api_key="sk-or-v1-...",
     base_url="https://openrouter.ai/api/v1",
-)))
+))
 
 # Bytes-in: caller already hydrated the file.
-parsed = await aparse(RawFile(content=pdf_bytes, extension="pdf"))
+parsed = await aparse(RawFile(content=pdf_bytes, extension="pdf"), llm=client)
 print(parsed.text)
 
 # URL-in: parser fetches over HTTP, then delegates to the HTML handler.
-parsed = await aparse(RawFile(uri="https://example.com/article"))
+parsed = await aparse(RawFile(uri="https://example.com/article"), llm=client)
 print(parsed.metadata["title"], parsed.text[:500])
 ```
 
@@ -72,8 +71,17 @@ The parser detects `soffice` via `shutil.which("soffice")` and the canonical mac
 - The library is **stateless**: it never reads the filesystem and never owns business state. HTTP I/O (LLM calls, URL fetching) is explicitly allowed.
 - No retry / fallback / metrics inside operators — surface failures via `LLMError`, let the caller wrap.
 
+## Security notes for callers
+
+`everalgo-parser` is a stateless library; production safety properties belong to the calling service. The points below are easy to miss and worth wiring into your integration:
+
+- **SSRF — `url.aparse` fetches arbitrary HTTP(S) URIs.** `fetch_uri` rejects non-http(s) schemes (no `file://`, no `gopher://`) but does **not** block private / link-local / loopback IPs (e.g. `127.0.0.1`, `10.0.0.0/8`, `169.254.169.254` cloud metadata, IPv6 ULA). If callers may pass attacker-controlled URLs, resolve the hostname first and reject private ranges before invoking `aparse`, or front the parser with an egress proxy that does the filtering.
+- **Decompression-bomb cap is set at import time.** `everalgo/parser/__init__.py` pins `PIL.Image.MAX_IMAGE_PIXELS = 100_000_000` (10000×10000) — Pillow will raise `DecompressionBombError` above that. Re-assign after import if your workload genuinely needs larger images.
+- **Office conversion runs LibreOffice as a subprocess.** `document._convert_to_pdf_via_soffice` validates the `extension` parameter (alphanumeric, ≤8 chars) before writing the temp file, but the LibreOffice attack surface itself is large. Run the host with a non-root user and treat untrusted office docs as you would any other complex binary input.
+- **`cairosvg` (optional `[svg]` extra) is LGPL-3.0-or-later.** EverAlgo itself is Apache-2.0. LGPLv3 is compatible with Apache-2.0 when `cairosvg` is consumed as an unmodified library through its public Python API (which is what the SVG path does); if you statically vendor or modify `cairosvg`, LGPLv3 §4 / §5 obligations apply to that derivative. Not installing the `[svg]` extra removes the dependency entirely.
+
 ## Reference
 
 - Architecture (definitive): [`docs/concepts/architecture.md`](../../docs/concepts/architecture.md)
-- Schema source for PDF / image / audio / document / html / email: `evermemos-multimodal` (tag `prod-20260306-0331-v1`).
-- Schema source for URL metadata extraction: `evermemos-opensource/src/common_utils/url_extractor.py`.
+- Schema source for PDF / image / audio / document / html / email: upstream internal multimodal parser library (snapshot `prod-20260306-0331-v1`).
+- Schema source for URL metadata extraction: upstream internal URL extractor reference implementation.

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/docs/architecture.md

@@ -301,7 +301,7 @@ flowchart LR
 
 | 项 | 状态 | 备注 |
 |---|---|---|
-| `video.py` | stub | 上游 evermemos-multimodal **也没有**实现,Gemini Video vs Whisper+帧抽取 选型待 ADR |
+| `video.py` | stub | 上游内部多模态解析库**也没有**实现,Gemini Video vs Whisper+帧抽取 选型待 ADR |
 | `audio` BMP-like 转码 | 没做 | mp3/wav 测过;m4a/amr/aiff/aac/ogg/flac 未在 e2e 覆盖,Gemini 兼容度未全测 |
 | `Office` 系统依赖动态检测降级 | 仅运行时报错 | 没有 import 时友好警告;`_find_soffice` 缺失就 `RuntimeError` |
 | `T1` 主条目其他 schema | 待 RESOLVED | 我们只 RESOLVED 了 `T1.parser` 子集(`ParsedContent` / `Modality` / `RawFile`) |

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/pyproject.toml

@@ -4,17 +4,17 @@ build-backend = "hatchling.build"
 
 [project]
 name = "everalgo-parser"
-version = "0.2.0"
+version = "0.2.1"
 description = "EverAlgo parser: multimodal raw-file parsing (image / audio / document / video / url) into ParsedContent."
 requires-python = ">=3.12"
-license = "MIT"
+license = "Apache-2.0"
 readme = "README.md"
 authors = [{ name = "EverMind", email = "Evermind@shanda.com" }]
 keywords = ["llm", "multimodal", "parsing", "everalgo"]
 classifiers = [
   "Development Status :: 3 - Alpha",
   "Intended Audience :: Developers",
-  "License :: OSI Approved :: MIT License",
+  "License :: OSI Approved :: Apache Software License",
   "Programming Language :: Python :: 3",
   "Programming Language :: Python :: 3.12",
   "Topic :: Scientific/Engineering :: Artificial Intelligence",

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/src/everalgo/parser/__init__.py

@@ -14,6 +14,7 @@ from typing import TYPE_CHECKING
 from urllib.parse import urlparse
 
 from asgiref.sync import async_to_sync
+from PIL import Image
 
 from everalgo.parser import audio, document, image, url
 from everalgo.types import (
@@ -39,6 +40,13 @@ __all__ = [
 logger = logging.getLogger(__name__)
 logger.addHandler(logging.NullHandler())
 
+# Explicit decompression-bomb cap (Pillow default raises ``DecompressionBombError``
+# only at ~2x this threshold). 100M pixels = 10000x10000, ample for any realistic
+# screenshot / scan while keeping a hostile caller from exhausting memory via a
+# crafted PNG/TIFF. Applications can override by re-assigning
+# ``PIL.Image.MAX_IMAGE_PIXELS`` after importing the parser.
+Image.MAX_IMAGE_PIXELS = 100_000_000
+
 
 async def aparse(raw_file: RawFile, *, llm: LLMClient) -> ParsedContent:
     """Top-level dispatch: classify the input and route to a submodule.

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/src/everalgo/parser/_utils.py

@@ -6,8 +6,8 @@
 - ``clean_html_content`` — strict text extraction (drops all tags).
 - ``clean_html_for_llm`` — preserves structural tags inside ``<body>``,
   strips known-noise tags and useless attributes, returns HTML.
-- ``fetch_uri`` — HTTP(S) fetch for the URL parser. Adapted from
-  ``evermemos-opensource/src/common_utils/url_extractor.py``.
+- ``fetch_uri`` — HTTP(S) fetch for the URL parser. Adapted from an
+  upstream internal URL extractor reference implementation.
 """
 
 from __future__ import annotations
@@ -40,7 +40,7 @@ _DEFAULT_USER_AGENT = "Mozilla/5.0 (compatible; everalgo-parser/0.1; +https://ev
 async def fetch_uri(uri: str, *, timeout: float = 30.0) -> tuple[bytes, str]:
     """Fetch ``http``/``https`` URI and return ``(content_bytes, content_type)``.
 
-    Adapted from ``evermemos-opensource/src/common_utils/url_extractor.py``;
+    Adapted from an upstream internal URL extractor reference implementation;
     rejects non-http(s) schemes per AGENTS.md §1 (no filesystem access).
 
     Parameters

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/src/everalgo/parser/document.py

@@ -399,6 +399,11 @@ async def _convert_to_pdf_via_soffice(
     content: bytes, extension: str, soffice: str
 ) -> bytes:  # pragma: no cover  (libreoffice binary, not in CI)
     """Run LibreOffice headless to convert bytes → PDF bytes."""
+    # Reject path-traversal / directory separators in `extension` so a hostile
+    # caller cannot escape the tempdir via e.g. ``extension="../../etc/passwd"``.
+    # `extension` is expected to be a short ASCII suffix (docx / xlsx / ...).
+    if not extension or not extension.isalnum() or len(extension) > 8:
+        raise ValueError(f"invalid extension for soffice conversion: {extension!r}")
     with tempfile.TemporaryDirectory() as tmpdir:  # pragma: no cover
         in_path = Path(tmpdir) / f"input.{extension}"  # pragma: no cover
         in_path.write_bytes(content)  # pragma: no cover

{everalgo_parser-0.2.0 → everalgo_parser-0.2.1}/src/everalgo/parser/url.py

@@ -18,10 +18,10 @@ The returned ``ParsedContent.modality`` is always ``Modality.URL`` so
 callers can distinguish "loaded from network" from "passed in as bytes";
 the inner modality is recorded in ``metadata["inner_modality"]``.
 
-Schema source for the metadata extractor:
-``evermemos-opensource/src/common_utils/url_extractor.py``
-(``_extract_og_tags`` / ``_extract_twitter_tags`` / ``_extract_meta_tags`` /
-``_extract_title`` / ``_extract_favicon``).
+Schema source for the metadata extractor: an upstream internal URL
+extractor reference implementation (``_extract_og_tags`` /
+``_extract_twitter_tags`` / ``_extract_meta_tags`` / ``_extract_title`` /
+``_extract_favicon``).
 """
 
 from __future__ import annotations
@@ -166,7 +166,7 @@ async def _dispatch_inner(
     raise ValueError(f"url.aparse: cannot dispatch fetched body for modality={modality.value!r}")
 
 
-# ---- metadata extraction (port from evermemos-opensource) ----
+# ---- metadata extraction (port from upstream internal URL extractor) ----
 
 
 def _extract_metadata(html_text: str, *, base_url: str) -> dict[str, object]: