spextract 0.5.16__tar.gz

spextract-0.5.16/LICENSE.md

@@ -0,0 +1,7 @@
+Copyright © 2026 Vincent Lonij
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

spextract-0.5.16/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.3
+Name: spextract
+Version: 0.5.16
+Summary: A declarative html scraper for python
+License: MIT
+Author: Vincent Lonij
+Author-email: 29819815+vincentropy@users.noreply.github.com
+Requires-Python: >=3.10
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: beautifulsoup4 (>=4.14.3,<5.0.0)
+Requires-Dist: click (>=8.3.2,<9.0.0)
+Requires-Dist: lxml (>=6.0.4,<7.0.0)
+Requires-Dist: pydantic (>=2.12.5,<3.0.0)
+Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
+Description-Content-Type: text/markdown
+
+# A Declarative HTML Scraper for Python
+
+This package provides a simple way to declare what data should be extracted from an HTML document in a configuration file.
+
+This enables sharing of scraping logic across projects and teams without the risk of executing untrusted code. It also allows for easier maintenance and updates to scraping logic without needing to modify the underlying codebase.
+
+## CLI
+
+The package includes a Click-based CLI with two commands:
+
+```bash
+decs parse spec.yaml <path to html file or directory>
+decs validate spec.yaml expected-results.yaml
+```
+
+`parse` emits YAML in the same expected-results format used by `validate`, so you can capture known-good output and re-run validation later.
+
+## How to use
+
+### Build a configuration file
+
+You can write a configuration file with the provided ParserSpec class.
+
+```python
+import py_decs
+
+spec = py_decs.ParserSpec(
+    name="example_parser",
+    description="An example parser for demonstration purposes.",
+    fields=[
+        py_decs.FieldSpec(
+            name="title",
+            selector="h1.title::text",
+            type=py_decs.FieldType.TEXT,
+        ),
+        py_decs.FieldSpec(
+            name="links",
+            selector="a.link::attr(href)",
+            type=py_decs.FieldType.LINK,
+            multiple=True,
+        )
+        py_decs.FieldSpec(
+            name="author",
+            selector="div.author",
+            type=py_decs.FieldType.OBJECT,
+            fields=[
+                py_decs.FieldSpec(
+                    name="name",
+                    selector="span.name::text",
+                    type=py_decs.FieldType.TEXT,
+                ),
+                py_decs.FieldSpec(
+                    name="profile_url",
+                    selector="a.profile::attr(href)",
+                    type=py_decs.FieldType.LINK,
+                ),
+            ]
+        ),
+    ]
+)
+```
+

spextract-0.5.16/README.md

@@ -0,0 +1,61 @@
+# A Declarative HTML Scraper for Python
+
+This package provides a simple way to declare what data should be extracted from an HTML document in a configuration file.
+
+This enables sharing of scraping logic across projects and teams without the risk of executing untrusted code. It also allows for easier maintenance and updates to scraping logic without needing to modify the underlying codebase.
+
+## CLI
+
+The package includes a Click-based CLI with two commands:
+
+```bash
+decs parse spec.yaml <path to html file or directory>
+decs validate spec.yaml expected-results.yaml
+```
+
+`parse` emits YAML in the same expected-results format used by `validate`, so you can capture known-good output and re-run validation later.
+
+## How to use
+
+### Build a configuration file
+
+You can write a configuration file with the provided ParserSpec class.
+
+```python
+import py_decs
+
+spec = py_decs.ParserSpec(
+    name="example_parser",
+    description="An example parser for demonstration purposes.",
+    fields=[
+        py_decs.FieldSpec(
+            name="title",
+            selector="h1.title::text",
+            type=py_decs.FieldType.TEXT,
+        ),
+        py_decs.FieldSpec(
+            name="links",
+            selector="a.link::attr(href)",
+            type=py_decs.FieldType.LINK,
+            multiple=True,
+        )
+        py_decs.FieldSpec(
+            name="author",
+            selector="div.author",
+            type=py_decs.FieldType.OBJECT,
+            fields=[
+                py_decs.FieldSpec(
+                    name="name",
+                    selector="span.name::text",
+                    type=py_decs.FieldType.TEXT,
+                ),
+                py_decs.FieldSpec(
+                    name="profile_url",
+                    selector="a.profile::attr(href)",
+                    type=py_decs.FieldType.LINK,
+                ),
+            ]
+        ),
+    ]
+)
+```

spextract-0.5.16/pyproject.toml

@@ -0,0 +1,48 @@
+[project]
+name = "spextract"
+version = "0.5.16"
+description = "A declarative html scraper for python"
+authors = [
+    { name = "Vincent Lonij", email = "29819815+vincentropy@users.noreply.github.com" },
+]
+license = "MIT"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "beautifulsoup4 (>=4.14.3,<5.0.0)",
+    "pydantic (>=2.12.5,<3.0.0)",
+    "pyyaml (>=6.0.3,<7.0.0)",
+    "lxml (>=6.0.4,<7.0.0)",
+    "click (>=8.3.2,<9.0.0)",
+]
+
+[project.scripts]
+decs = "py_decs.cli:cli"
+
+[tool.poetry]
+
+[tool.poetry.group.dev.dependencies]
+mypy = "^1.20.0"
+types-pyyaml = "^6.0.12.20260408"
+pylint = "^4.0.5"
+pytest = "^9.0.3"
+types-lxml = "^2026.2.16"
+pylint-pydantic = "^0.4.1"
+twine = "^6.2.0"
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pylint]
+load-plugins = ["pylint_pydantic"]
+disable = [
+    "missing-module-docstring",
+    "missing-class-docstring",
+    "missing-function-docstring",
+    "line-too-long",
+]
+max-line-length = 120
+
+[tool.black]
+line-length = 120

spextract-0.5.16/spextract/__init__.py

@@ -0,0 +1,27 @@
+from .models import (
+    ParseSpec,
+    FieldSpec,
+    FieldType,
+    ProcessorSpec,
+    EngineOutput,
+    SpecValidationResult,
+    ValidationMismatch,
+    ExpectedResults,
+    FileExpectedItems,
+)
+from .engine import ParseEngine
+from .validation.spec_validate import validate_spec_output
+
+__all__ = [
+    "FieldSpec",
+    "FieldType",
+    "ParseEngine",
+    "ParseSpec",
+    "ProcessorSpec",
+    "EngineOutput",
+    "SpecValidationResult",
+    "ValidationMismatch",
+    "ExpectedResults",
+    "FileExpectedItems",
+    "validate_spec_output",
+]

spextract-0.5.16/spextract/cli.py

@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import click
+import yaml
+
+from .engine import ParseEngine
+from .models.output import EngineOutput
+from .models.parser_spec import ParseSpec
+from .models.validation import ExpectedResults, FileExpectedItems
+from .validation.true_validate import validate_files
+
+
+@click.group()
+def cli() -> None:
+    """Declarative scraper utilities."""
+
+
+@cli.command()
+@click.argument("spec_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+@click.argument("html_path", type=click.Path(exists=True, dir_okay=True, path_type=Path))
+@click.option("--output", "output_path", type=click.Path(dir_okay=False, path_type=Path))
+def parse(spec_path: Path, html_path: Path, output_path: Path | None) -> None:
+    """Apply a spec to one or more HTML files."""
+
+    spec = ParseSpec.from_yaml_file(spec_path)
+    engine = ParseEngine(spec)
+    results: dict[str, EngineOutput] = {}
+    if html_path.is_file():
+        html_files = [html_path]
+    else:
+        html_files = sorted(html_path.glob("*.html"))
+
+    for html_file in html_files:
+        html = html_file.read_text(encoding="utf-8")
+        parsed = engine.parse(html)
+        results[html_file.name] = parsed
+
+    file_results = ExpectedResults(
+        data_path=html_path if html_path.is_dir() else html_path.parent,
+        files=[FileExpectedItems.from_engine_output(file_name, output) for file_name, output in results.items()],
+    )
+
+    if output_path is not None:
+        file_results.to_yaml_file(output_path)
+        click.echo(f"Wrote parsed results to {output_path}")
+        return
+
+    rendered_yaml = yaml.safe_dump(file_results.model_dump(), sort_keys=False, allow_unicode=True)
+    click.echo(rendered_yaml, nl=False)
+
+
+@cli.command()
+@click.argument("spec_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+@click.argument("expected_results_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+@click.option(
+    "--field-path", "-f", type=str, help="Optional dot path to a specific field to validate (e.g. 'items.name')."
+)
+def validate(spec_path: Path, expected_results_path: Path, field_path: str | None) -> None:
+    """Validate a spec against YAML expected extraction results."""
+
+    validation_result = validate_files(
+        expected_values_path=expected_results_path,
+        spec_file_path=spec_path,
+        field_path=field_path,
+    )
+    if validation_result.passed:
+        click.echo(
+            f"Validation passed for {validation_result.total_files} file(s) and {validation_result.total_items} extracted field(s)."
+        )
+        return
+
+    for file_result in validation_result.file_results:
+        click.echo(f"{file_result.file_name}:")
+        if file_result.passed:
+            click.echo(f"  Passed. Values checked: {file_result.item_count}")
+            continue
+
+        for error in file_result.errors:
+            click.echo(f"  - {error}")
+
+    raise click.ClickException(
+        f"Validation failed for {validation_result.failures} of {validation_result.total_files} file(s)."
+    )

spextract-0.5.16/spextract/engine.py

@@ -0,0 +1,152 @@
+"""Core extraction engine that applies a ParseSpec to HTML content."""
+
+from __future__ import annotations
+
+from typing import cast
+
+from bs4 import BeautifulSoup, Tag
+
+from spextract.models.validation import SpecValidationResult
+
+from .models import EngineOutput, FieldSpec, ParseSpec, ProcessorSpec, DataValue
+from .processors import apply_processor
+from .uni_selector import select
+
+
+class ParseEngine:
+    """Applies a ParseSpec to HTML content and returns extracted items."""
+
+    def __init__(self, spec: ParseSpec) -> None:
+        self.spec = spec
+
+    def parse(self, html: str, field_path: str | None = None) -> EngineOutput:
+        """Parse HTML string and return EngineOutput with typed fields.
+
+        Args:
+            html: Raw HTML content to parse.
+            field_path: Optional dot-separated path (e.g. ``"person.name"``) to
+                extract only a single field instead of the full spec.  Each
+                segment must match a key in the ``fields`` dict at that level of
+                nesting.  Raises ``KeyError`` if a segment is not found or if a
+                non-leaf segment has no child fields.
+        """
+        root = BeautifulSoup(html, "html.parser")
+        if field_path is not None:
+            data = self._extract_field_path(root, self.spec.fields, field_path.split("."))
+        else:
+            data = self._extract_fields(root, self.spec.fields)
+        return EngineOutput(spec=self.spec, data=data if data is not None else {})
+
+    def validate(self, html: str, field_path: str | None = None) -> SpecValidationResult:
+        """Validate HTML against spec, returning SpecValidationResult with details."""
+        from .validation.spec_validate import validate_spec_output  # pylint: disable=import-outside-toplevel
+
+        output = self.parse(html, field_path=field_path)
+        validation_result = validate_spec_output(self.spec, output.data, raise_=False)
+        return validation_result
+
+    def parse_and_validate(self, html: str, field_path: str | None = None) -> EngineOutput:
+        """Parse HTML and validate output against spec, returning EngineOutput with validation results.
+        Raises ValueError if validation fails."""
+
+        output = self.parse(html, field_path=field_path)
+        validation_result = self.validate(html, field_path=field_path)
+        if not validation_result.is_valid:
+            raise ValueError(f"Validation failed: {validation_result.mismatches[0]}")
+        return output
+
+    @staticmethod
+    def _extract_field_path(
+        node: Tag | BeautifulSoup,
+        fields: dict[str, FieldSpec],
+        path: list[str],
+    ) -> dict[str, DataValue]:
+        """Navigate ``fields`` and the HTML tree simultaneously following ``path``.
+
+        Returns a ``{leaf_name: value}`` dict containing only the targeted field.
+        Raises ``KeyError`` if any path segment is missing or if a non-leaf
+        segment has no child ``fields``.
+        """
+        name, *rest = path
+        if name not in fields:
+            raise KeyError(f"Field {name!r} not found. Available: {list(fields.keys())}")
+        field_spec = fields[name]
+
+        if not rest:
+            # Leaf — extract just this field
+            value = ParseEngine._extract_field(node, field_spec)
+            return {name: value} if value is not None else {}
+
+        # Intermediate — must have child fields and a navigable selector
+        if field_spec.fields is None:
+            raise KeyError(f"Field {name!r} has no child fields; cannot navigate to {'.'.join(rest)!r}")
+        sub_nodes = select(node, field_spec.selector, assert_tags=True)
+        if not sub_nodes:
+            return {}
+
+        if field_spec.multiple:
+            items = [ParseEngine._extract_field_path(sub, field_spec.fields, rest) for sub in sub_nodes]
+            return cast(dict[str, DataValue], {name: cast(DataValue, items)})
+
+        inner = ParseEngine._extract_field_path(sub_nodes[0], field_spec.fields, rest)
+        return cast(dict[str, DataValue], {name: cast(DataValue, inner)})
+
+    @staticmethod
+    def _extract_fields(node: Tag | BeautifulSoup, fields: dict[str, FieldSpec]) -> dict[str, DataValue]:
+        result: dict[str, DataValue] = {}
+        for name, field_spec in fields.items():
+            field_out = ParseEngine._extract_field(node, field_spec)
+            if field_out is not None:
+                result[name] = field_out
+        return result
+
+    @staticmethod
+    def _extract_field(node: Tag | BeautifulSoup, field_spec: FieldSpec) -> DataValue | None:
+        if field_spec.fields is not None:
+            # if this field has child fields, extraction is slightly different
+            # we ignore ::text / ::attr on the parent selector.
+            return ParseEngine._extract_nested(node, field_spec)
+
+        values = select(node, field_spec.selector)
+        if not values:
+            return [] if field_spec.multiple else None
+        if field_spec.multiple:
+            out_values = [ParseEngine.apply_processors(v, field_spec.resolved_processors()) for v in values]
+            return cast(DataValue, out_values)
+
+        all_strings = all(isinstance(v, str) for v in values)
+        if all_strings:
+            values = ["".join(cast(list[str], values))]
+        if len(values) > 1:
+            print(f"Warning: Multiple elements matched for single field: {field_spec.selector}. Using first match.")
+        out_value = ParseEngine.apply_processors(values[0], field_spec.resolved_processors())
+        return out_value
+
+    @staticmethod
+    def _extract_nested(node: Tag | BeautifulSoup, field_spec: FieldSpec) -> DataValue | None:
+        """Extract a field with child fields, applying child selectors relative to parent elements.
+        In this case we ignore ::text / ::attr on the parent selector since it doesn't make sense
+        to apply these to a parent element that we're extracting child fields from."""
+        assert field_spec.fields is not None
+
+        sub_nodes = select(node, field_spec.selector, assert_tags=True)
+        if not sub_nodes:
+            return [] if field_spec.multiple else None
+
+        if field_spec.multiple:
+            # Return a list of FieldOutput objects
+            return [ParseEngine._extract_fields(sub, field_spec.fields) for sub in sub_nodes]
+
+        if len(sub_nodes) > 1:
+            print(f"Warning: Multiple elements matched for single field: {field_spec.selector}. Using first match.")
+        return ParseEngine._extract_fields(sub_nodes[0], field_spec.fields)
+
+    @staticmethod
+    def apply_processors(value: object, processors: list[ProcessorSpec]) -> str | float | None:
+        if value is None:
+            return value
+        for proc in processors:
+            value = apply_processor(proc.name, value, proc.args if proc.args else None)
+        if isinstance(value, (str, float)) or value is None:
+            return value
+        raise ValueError(f"Unsupported value type after processing: {type(value)}")

spextract-0.5.16/spextract/models/__init__.py

@@ -0,0 +1,17 @@
+from .parser_spec import FieldSpec, FieldType, ParseSpec, ProcessorName, ProcessorSpec
+from .output import DataValue, EngineOutput
+from .validation import SpecValidationResult, ValidationMismatch, ExpectedResults, FileExpectedItems
+
+__all__ = [
+    "FieldSpec",
+    "FieldType",
+    "ParseSpec",
+    "ProcessorName",
+    "ProcessorSpec",
+    "DataValue",
+    "EngineOutput",
+    "SpecValidationResult",
+    "ValidationMismatch",
+    "ExpectedResults",
+    "FileExpectedItems",
+]

spextract-0.5.16/spextract/models/output.py

@@ -0,0 +1,21 @@
+from __future__ import annotations
+
+from typing import Union  # pylint: disable=unused-import
+from typing_extensions import TypeAliasType
+
+from .yaml import BaseModelWithYamlSupport
+
+from .parser_spec import ParseSpec
+
+DataValue = TypeAliasType(
+    "DataValue",
+    "Union[None, float, str, dict[str, DataValue], list[DataValue]]",
+)
+
+
+class EngineOutput(BaseModelWithYamlSupport):
+    """Output from the scraping engine for a single file,
+    including the parser spec used and the extracted data."""
+
+    spec: ParseSpec | None = None
+    data: dict[str, DataValue]

spextract-0.5.16/spextract/models/parser_spec.py

@@ -0,0 +1,85 @@
+"""
+Pydantic models for the declarative scraper.
+"""
+
+from __future__ import annotations
+
+import enum
+from typing import Union
+
+from pydantic import Field
+
+from .yaml import BaseModelWithYamlSupport
+from ..processors import ProcessorName
+
+
+class ProcessorSpec(BaseModelWithYamlSupport):
+    """A single field processor.
+
+    Can be a simple named processor (e.g. "strip") or a parameterised one, eg a regex.
+    """
+
+    name: ProcessorName
+    args: list[Union[str, int]] = Field(default_factory=list)
+
+
+class FieldType(enum.Enum):
+    """Type of field to extract, used to determine how to extract it from HTML."""
+
+    TEXT = "text"
+    LINK = "link"
+    NUMBER = "number"
+    DATE = "date"
+    OBJECT = "object"
+
+
+class FieldSpec(BaseModelWithYamlSupport):
+    """Specification for extracting a single field from HTML."""
+
+    selector: str = Field(
+        description="""
+            CSS selector or XPATH selector for the field.
+            If the fields attribute is not None, this selector should return a parent element or a list of parent elements.
+            The selector attribute of the child fields will be applied relative to each parent element.
+        """
+    )
+    type: FieldType = Field(
+        default=FieldType.TEXT,
+        description="Type of field to extract, used to validate extraction.",
+    )
+    required: bool = Field(default=True, description="Whether this field is required. Used for validation.")
+
+    multiple: bool = Field(
+        default=False,
+        description="Whether to extract multiple values from this field (i.e. return a list).",
+    )
+    processors: list[Union[ProcessorName, dict[ProcessorName, list[Union[str, int]]]]] = Field(
+        default_factory=list,
+        description="List of processors to apply to the extracted value(s). Each processor can be a string (processor name) or a dict mapping processor name to argument list. These are applied in order.",
+    )
+    fields: dict[str, "FieldSpec"] | None = Field(
+        default=None,
+        description="Child fields to extract from the element(s) selected by this field. \
+            Keys in this dict will be keys in the output data.",
+    )
+
+    def resolved_processors(self) -> list[ProcessorSpec]:
+        """Normalise the processor list into ProcessorSpec objects from dict[func-name=>arg list] or str."""
+        result: list[ProcessorSpec] = []
+        for p in self.processors:
+            if isinstance(p, (str, ProcessorName)):
+                result.append(ProcessorSpec(name=p))
+            elif isinstance(p, dict):
+                for name, args in p.items():
+                    result.append(ProcessorSpec(name=name, args=args))
+            else:
+                raise ValueError(f"Invalid processor spec: {p}")
+        return result
+
+
+class ParseSpec(BaseModelWithYamlSupport):
+    """Top-level declarative parser specification."""
+
+    version: int = 1
+    name: str
+    fields: dict[str, FieldSpec]

spextract-0.5.16/spextract/models/validation.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+from pydantic import BaseModel, Field, field_serializer
+
+from .output import DataValue, EngineOutput
+from .yaml import BaseModelWithYamlSupport
+
+
+class ValidationMismatch(BaseModel):
+    field: str
+    expected_type: str
+    actual_type: str
+    message: str
+
+
+class SpecValidationResult(BaseModel):
+    mismatches: list[ValidationMismatch] = Field(default_factory=list)
+
+    @property
+    def is_valid(self) -> bool:
+        return len(self.mismatches) == 0
+
+
+class FileValidationResult(BaseModel):
+    """Validation result for a single HTML file."""
+
+    file_name: str
+    item_count: int
+    errors: list[str] = Field(default_factory=list)
+
+    @property
+    def passed(self) -> bool:
+        return not self.errors
+
+
+class TrueValidationResult(BaseModel):
+    """Aggregate validation result across all files."""
+
+    file_results: list[FileValidationResult] = Field(default_factory=list)
+
+    @property
+    def total_files(self) -> int:
+        return len(self.file_results)
+
+    @property
+    def total_items(self) -> int:
+        return sum(result.item_count for result in self.file_results)
+
+    @property
+    def failures(self) -> int:
+        return sum(1 for result in self.file_results if not result.passed)
+
+    @property
+    def passed(self) -> bool:
+        return self.failures == 0
+
+
+class FileExpectedItems(BaseModel):
+    """Expected extraction results for a single example file."""
+
+    file: str
+    items: dict[str, DataValue]
+
+    @classmethod
+    def from_engine_output(cls, file_name: str, output: EngineOutput) -> FileExpectedItems:
+        return cls(file=file_name, items=output.data)
+
+
+class ExpectedResults(BaseModelWithYamlSupport):
+    """Expected extraction output used to validate parser correctness.
+
+    Stored as YAML alongside example data files.
+    """
+
+    version: int = 1
+    data_path: Path | None = None
+    files: list[FileExpectedItems]
+
+    @field_serializer("data_path")
+    def serialize_data_path(self, value: Path | None) -> str | None:
+        """Serialize data_path as a string in YAML."""
+        return str(value) if value is not None else None