bopa-fetcher 0.1.0__tar.gz

bopa_fetcher-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Diego González Suárez and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bopa_fetcher-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include LICENSE
+recursive-include src/bopa *.py

bopa_fetcher-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: bopa-fetcher
+Version: 0.1.0
+Summary: A Python library for fetching structured information from BOPA (Boletín Oficial de la Principidad de Asturias) webpage.
+Author-email: Diego González Suárez <gonzalezsdiego@uniovi.es>, Noelia Rico <noeliarico@uniovi.es>
+Maintainer-email: Diego González Suárez <gonzalezsdiego@uniovi.es>
+License-Expression: MIT
+Project-URL: Homepage, https://diegoglezsu.github.io/bopa-fetcher
+Project-URL: Documentation, https://diegoglezsu.github.io/bopa-fetcher
+Project-URL: Repository, https://github.com/diegoglezsu/bopa-fetcher.git
+Project-URL: Issues, https://github.com/diegoglezsu/bopa-fetcher/issues
+Keywords: official gazette,asturian legislation,data analysis,policy research,legal analytics
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Legal Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing :: General
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: bs4==0.0.2
+Requires-Dist: matplotlib>=3.9.0
+Requires-Dist: numpy>=2.1.0
+Requires-Dist: pandas>=2.2.2
+Requires-Dist: requests>=2.33.0
+Requires-Dist: seaborn>=0.13.2
+Provides-Extra: dev
+Requires-Dist: pytest>=6.0; extra == "dev"
+Requires-Dist: pytest-cov>=2.0; extra == "dev"
+Requires-Dist: black>=21.0; extra == "dev"
+Requires-Dist: flake8>=3.9; extra == "dev"
+Requires-Dist: mypy>=0.900; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs<2.0,>=1.6; extra == "docs"
+Requires-Dist: mkdocs-material>=9.5; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.25; extra == "docs"
+Dynamic: license-file
+
+# bopa-fetcher
+
+[![Tests](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/tests.yml/badge.svg)](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/tests.yml)
+[![CodeQL](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/github-code-scanning/codeql)
+[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=diegoglezsu_bopa-fetcher&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=diegoglezsu_bopa-fetcher)
+[![Codecov status](https://codecov.io/github/diegoglezsu/bopa-fetcher/badge.svg?branch=main&service=github)](https://app.codecov.io/github/diegoglezsu/bopa-fetcher)
+[![PyPI version](https://img.shields.io/pypi/v/bopa-fetcher.svg)](https://pypi.org/project/bopa-fetcher/)
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://diegoglezsu.github.io/bopa-fetcher/)
+[![DOI](https://zenodo.org/badge/1193786609.svg)]()
+
+## Description
+
+![bopa-fetcher Logo](https://raw.githubusercontent.com/diegoglezsu/bopa-fetcher/main/docs/assets/logo.png)
+
+**bopa-fetcher** is a Python library for programmatic access to the official bulletins of the Principality of Asturias (BOPA). It allows users to search, retrieve, and analyze bulletin summaries and individual articles in a structured manner.
+
+## Why bopa-fetcher?
+
+[BOPA (Boletín Oficial del Principado de Asturias)](https://miprincipado.asturias.es/bopa) is the official gazette of the region of Asturias, Spain. Researchers, legal professionals, and journalists often need to search, download, and analyze large volumes of legislative and administrative documents. **bopa-fetcher** provides a simple, programmatic interface to:
+
+- Retrieve bulletin summaries and articles as structured Python objects.
+- Search across date ranges for both bulletins and individual articles.
+- Export data to dictionaries for integration with data analysis pipelines (pandas, NumPy, etc.).
+- Avoid manual scraping by handling HTML parsing and URL construction internally.
+
+> [!WARNING]
+> BOPA bulletins are available in the portal from **01/01/2000** onwards. Requests for earlier dates will return no data.
+
+## Main features
+
+- **Legal research**: Download and analyze official bulletins for a specific time period to track legislative changes.
+- **Data journalism**: Collect structured data from BOPA for investigative reporting on regional governance.
+- **Policy analysis**: Extract and categorize dispositions by origin (council, council board, presidency, etc.) for quantitative studies.
+- **Archive building**: Build reproducible datasets of Asturian official publications for academic research.
+
+## Quick Start
+
+### Installation
+
+Install from PyPI:
+
+```bash
+pip install bopa-fetcher
+```
+
+### Basic Usage Example
+
+Fetch acts for a publication date:
+
+```python
+from bopa.api import Client
+client = Client()
+
+# Get the bulletin summary for 29/12/2023
+bulletin = client.get_bulletin(date="29/12/2023")
+print(bulletin.to_dict())
+
+# get specific article by code
+article = client.get_article(cod="2023-11737", date="29/12/2023")
+print(f"  article: {article.to_dict()}")
+
+```
+
+### Use Case Examples
+
+The repository includes runnable scripts with examples and use cases of the library. These scripts can be found in the `scripts/` directory.
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.
+
+## Contact
+
+For any questions or suggestions, feel free to reach out to the corresponding author:
+
+- **Author**: Diego González Suárez, Noelia Rico Pachón
+- **Email**: <gonzalezsdiego@uniovi.es>, <noeliarico@uniovi.es>
+
+## Acknowledgements
+
+## Citation

bopa_fetcher-0.1.0/README.md

@@ -0,0 +1,81 @@
+# bopa-fetcher
+
+[![Tests](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/tests.yml/badge.svg)](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/tests.yml)
+[![CodeQL](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/diegoglezsu/bopa-fetcher/actions/workflows/github-code-scanning/codeql)
+[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=diegoglezsu_bopa-fetcher&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=diegoglezsu_bopa-fetcher)
+[![Codecov status](https://codecov.io/github/diegoglezsu/bopa-fetcher/badge.svg?branch=main&service=github)](https://app.codecov.io/github/diegoglezsu/bopa-fetcher)
+[![PyPI version](https://img.shields.io/pypi/v/bopa-fetcher.svg)](https://pypi.org/project/bopa-fetcher/)
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://diegoglezsu.github.io/bopa-fetcher/)
+[![DOI](https://zenodo.org/badge/1193786609.svg)]()
+
+## Description
+
+![bopa-fetcher Logo](https://raw.githubusercontent.com/diegoglezsu/bopa-fetcher/main/docs/assets/logo.png)
+
+**bopa-fetcher** is a Python library for programmatic access to the official bulletins of the Principality of Asturias (BOPA). It allows users to search, retrieve, and analyze bulletin summaries and individual articles in a structured manner.
+
+## Why bopa-fetcher?
+
+[BOPA (Boletín Oficial del Principado de Asturias)](https://miprincipado.asturias.es/bopa) is the official gazette of the region of Asturias, Spain. Researchers, legal professionals, and journalists often need to search, download, and analyze large volumes of legislative and administrative documents. **bopa-fetcher** provides a simple, programmatic interface to:
+
+- Retrieve bulletin summaries and articles as structured Python objects.
+- Search across date ranges for both bulletins and individual articles.
+- Export data to dictionaries for integration with data analysis pipelines (pandas, NumPy, etc.).
+- Avoid manual scraping by handling HTML parsing and URL construction internally.
+
+> [!WARNING]
+> BOPA bulletins are available in the portal from **01/01/2000** onwards. Requests for earlier dates will return no data.
+
+## Main features
+
+- **Legal research**: Download and analyze official bulletins for a specific time period to track legislative changes.
+- **Data journalism**: Collect structured data from BOPA for investigative reporting on regional governance.
+- **Policy analysis**: Extract and categorize dispositions by origin (council, council board, presidency, etc.) for quantitative studies.
+- **Archive building**: Build reproducible datasets of Asturian official publications for academic research.
+
+## Quick Start
+
+### Installation
+
+Install from PyPI:
+
+```bash
+pip install bopa-fetcher
+```
+
+### Basic Usage Example
+
+Fetch acts for a publication date:
+
+```python
+from bopa.api import Client
+client = Client()
+
+# Get the bulletin summary for 29/12/2023
+bulletin = client.get_bulletin(date="29/12/2023")
+print(bulletin.to_dict())
+
+# get specific article by code
+article = client.get_article(cod="2023-11737", date="29/12/2023")
+print(f"  article: {article.to_dict()}")
+
+```
+
+### Use Case Examples
+
+The repository includes runnable scripts with examples and use cases of the library. These scripts can be found in the `scripts/` directory.
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.
+
+## Contact
+
+For any questions or suggestions, feel free to reach out to the corresponding author:
+
+- **Author**: Diego González Suárez, Noelia Rico Pachón
+- **Email**: <gonzalezsdiego@uniovi.es>, <noeliarico@uniovi.es>
+
+## Acknowledgements
+
+## Citation

bopa_fetcher-0.1.0/pyproject.toml

@@ -0,0 +1,75 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "bopa-fetcher"
+version = "0.1.0"
+description = "A Python library for fetching structured information from BOPA (Boletín Oficial de la Principidad de Asturias) webpage."
+readme = "README.md"
+requires-python = ">=3.7"
+license-files = ["LICENSE"]
+authors = [
+  {name = "Diego González Suárez", email = "gonzalezsdiego@uniovi.es"},
+  {name = "Noelia Rico", email = "noeliarico@uniovi.es"}
+]
+maintainers = [
+  {name = "Diego González Suárez", email = "gonzalezsdiego@uniovi.es"},
+]
+keywords = ["official gazette", "asturian legislation", "data analysis", "policy research", "legal analytics"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Operating System :: OS Independent",
+  "Intended Audience :: Legal Industry",
+  "Intended Audience :: Science/Research",
+  "Topic :: Scientific/Engineering :: Information Analysis",
+  "Topic :: Text Processing :: General",
+]
+license = "MIT"
+dependencies = [
+  "bs4==0.0.2",
+  "matplotlib>=3.9.0",
+  "numpy>=2.1.0",
+  "pandas>=2.2.2",
+  "requests>=2.33.0",
+  "seaborn>=0.13.2"
+]
+
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=6.0",
+    "pytest-cov>=2.0",
+    "black>=21.0",
+    "flake8>=3.9",
+    "mypy>=0.900",
+]
+docs = [
+    "mkdocs>=1.6,<2.0",
+    "mkdocs-material>=9.5",
+    "mkdocstrings[python]>=0.25",
+]
+
+[project.urls]
+Homepage = "https://diegoglezsu.github.io/bopa-fetcher"
+Documentation = "https://diegoglezsu.github.io/bopa-fetcher"
+Repository = "https://github.com/diegoglezsu/bopa-fetcher.git"
+Issues = "https://github.com/diegoglezsu/bopa-fetcher/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.black]
+line-length = 88
+target-version = ["py37"]
+
+[tool.isort]
+profile = "black"
+line_length = 88
+
+[tool.mypy]
+python_version = "3.9"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = true
+ignore_missing_imports = true

bopa_fetcher-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

bopa_fetcher-0.1.0/src/bopa/__init__.py

@@ -0,0 +1,15 @@
+"""bopa-fetcher: A Python library for fetching structured information from BOPA.
+
+BOPA (Boletín Oficial del Principado de Asturias) is the official gazette
+of the region of Asturias, Spain. This library provides programmatic access
+to its bulletins and articles via web scraping.
+"""
+
+__author__ = ["Diego González Suárez", "Noelia Rico"]
+__email__ = ["gonzalezsdiego@uniovi.es", "noeliarico@uniovi.es"]
+__version__ = "0.1.0"
+
+from . import service
+from . import api
+from . import models
+from . import constants

bopa_fetcher-0.1.0/src/bopa/api/__init__.py

@@ -0,0 +1,3 @@
+from .client import Client
+
+__all__ = ["Client"]

bopa_fetcher-0.1.0/src/bopa/api/client.py

@@ -0,0 +1,157 @@
+from datetime import datetime, timedelta
+
+from bopa.constants import DATE_MIN
+
+from ..models import BulletinArticle, BulletinSummary
+from ..service.article import Article
+from ..service.bulletin import Bulletin
+
+
+class Client:
+    """A client class to interact with the BOPA API and fetch bulletins and articles."""
+
+    def get_bulletin(
+        self,
+        date: str,
+        text_contains: str | None = None,
+        origin_contains: str | None = None,
+    ) -> BulletinSummary:
+        """Fetch the bulletin summary for a specific date.
+
+        Args:
+            date: Date in dd/mm/YYYY format (e.g. "29/12/2023").
+            text_contains: Optional string to filter entries by.
+            origin_contains: Optional string to filter entries by origin.
+
+        Returns:
+            BulletinSummary corresponding to the given date.
+        """
+
+        if not date:
+            raise ValueError("The 'date' parameter is required.")
+        
+        if datetime.strptime(date, "%d/%m/%Y") < DATE_MIN:
+            raise ValueError(f"date must be on or after {DATE_MIN}.")
+
+        return Bulletin(date=date).get_bulletin(
+            text_contains=text_contains, origin_contains=origin_contains
+        )
+
+    def get_bulletins(
+        self,
+        date_from: str,
+        date_to: str,
+        text_contains: str | None = None,
+        origin_contains: str | None = None,
+    ) -> list[BulletinSummary]:
+        """Fetch all bulletin summaries in a date range.
+
+        Args:
+            date_from: Start date in dd/mm/YYYY format.
+            date_to: End date in dd/mm/YYYY format.
+            text_contains: Optional string to filter entries by.
+            origin_contains: Optional string to filter entries by origin.
+
+        Returns:
+            List of BulletinSummary objects for each weekday in the range.
+        """
+        date_from = datetime.strptime(date_from, "%d/%m/%Y")
+        date_to = datetime.strptime(date_to, "%d/%m/%Y")
+
+        if date_from > date_to:
+            raise ValueError("date_from must be earlier than or equal to date_to.")
+        
+        if date_from < DATE_MIN:
+            raise ValueError(f"date_from must be on or after {DATE_MIN}.")
+
+        summaries = []
+        current_date = date_from
+        while current_date <= date_to:
+            fecha_str = current_date.strftime("%d/%m/%Y")
+            try:
+                summaries.append(
+                    self.get_bulletin(
+                        fecha_str,
+                        text_contains=text_contains,
+                        origin_contains=origin_contains,
+                    )
+                )
+            except Exception:
+                pass
+            current_date += timedelta(days=1)
+
+        return summaries
+
+    def get_article(
+        self,
+        cod: str,
+        date: str,
+    ) -> BulletinArticle:
+        """Fetch a specific article by code and date.
+
+        Args:
+            cod: Article disposition code (e.g. "2023-11737").
+            date: Date in dd/mm/YYYY format.
+
+        Returns:
+            BulletinArticle with full content and metadata.
+        """
+
+        if not cod or not date:
+            raise ValueError("Both 'cod' and 'date' must be provided.")
+        
+        if datetime.strptime(date, "%d/%m/%Y") < DATE_MIN:
+            raise ValueError(f"date must be on or after {DATE_MIN}.")
+
+        return Article(cod=cod, date=date).get_article()
+
+    def get_articles(
+        self,
+        date_from: str,
+        date_to: str,
+        text_contains: str | None = None,
+        origin_contains: str | None = None,
+    ) -> list[BulletinArticle]:
+        """Fetch all articles in a date range.
+
+        Iterates over each day in the range, retrieves the bulletin summary,
+        and fetches the full content of every article listed.
+
+        Args:
+            date_from: Start date in dd/mm/YYYY format.
+            date_to: End date in dd/mm/YYYY format.
+            text_contains: Optional string to filter bulletin entries by.
+            origin_contains: Optional string to filter bulletin entries
+                by origin.
+
+        Returns:
+            List of BulletinArticle objects in the range.
+        """
+        articles = []
+        start_date = datetime.strptime(date_from, "%d/%m/%Y")
+        end_date = datetime.strptime(date_to, "%d/%m/%Y")
+
+        if start_date > end_date:
+            raise ValueError("date_from must be earlier than or equal to date_to.")
+        
+        if start_date < DATE_MIN:
+            raise ValueError(f"date_from must be on or after {DATE_MIN}.")
+
+        current_date = start_date
+        while current_date <= end_date:
+            fecha_str = current_date.strftime("%d/%m/%Y")
+            try:
+                bulletin = self.get_bulletin(
+                    fecha_str,
+                    text_contains=text_contains,
+                    origin_contains=origin_contains,
+                )
+                articles.extend(
+                    Article(cod=cod, num=bulletin.num, date=bulletin.date).get_article()
+                    for cod in bulletin.codes
+                )
+            except Exception:
+                pass
+            current_date += timedelta(days=1)
+
+        return articles

bopa_fetcher-0.1.0/src/bopa/constants.py

@@ -0,0 +1,15 @@
+"""Constants for BOPA web portal URLs and HTML element identifiers."""
+
+from datetime import datetime
+
+
+BOPA_URL = "https://miprincipado.asturias.es/bopa"
+
+DISPOSITONS_URL = BOPA_URL + "/disposiciones"
+
+SUMMARY_URL = "https://miprincipado.asturias.es/bopa-sumario"
+
+BOPA_ARTICLE_ID = "bopa-articulo"
+BOPA_BULLETIN_ID = "bopa-boletin"
+
+DATE_MIN = datetime(2000, 1, 1)  # Minimum date for BOPA bulletins

bopa_fetcher-0.1.0/src/bopa/models/__init__.py

@@ -0,0 +1,4 @@
+from .bulletin import BulletinSummary, BulletinSummaryEntry
+from .article import BulletinArticle
+
+__all__ = ["BulletinArticle", "BulletinSummary", "BulletinSummaryEntry"]

bopa_fetcher-0.1.0/src/bopa/models/article.py

@@ -0,0 +1,42 @@
+
+from dataclasses import dataclass
+import datetime
+
+
+@dataclass
+class BulletinArticle:
+    """Full content for one BOPA article.
+
+    Attributes:
+        code: Disposition code (e.g. "2023-11737").
+        num: Bulletin number.
+        date: Publication date.
+        origin: Origin hierarchy string (part / chapter / topic / sub-author).
+        content: List of text paragraphs comprising the article body.
+        link_html: URL to the HTML detail page.
+        link_pdf: URL to the PDF document.
+    """
+
+    code: str
+    num: str
+    date: datetime
+    origin: str
+    content: list[str]
+    link_html: str
+    link_pdf: str
+
+    def to_dict(self) -> dict[str, object]:
+        """Serialize the article to a dictionary.
+
+        Returns:
+            Dict with all fields; date is formatted as dd/mm/YYYY.
+        """
+        return {
+            "code": self.code,
+            "num": self.num,
+            "date": self.date.strftime("%d/%m/%Y"),
+            "origin": self.origin,
+            "content": self.content,
+            "link_html": self.link_html,
+            "link_pdf": self.link_pdf,
+        }

bopa_fetcher-0.1.0/src/bopa/models/bulletin.py

@@ -0,0 +1,67 @@
+from dataclasses import dataclass, field
+from datetime import datetime
+
+
+@dataclass
+class BulletinSummaryEntry:
+    """One disposition entry listed in a BOPA bulletin summary.
+
+    Attributes:
+        code: Disposition code (e.g. "2023-11737"), or "N/A" if not found.
+        origin: Origin hierarchy string (part / chapter / topic / sub-author).
+        description: Short text description of the disposition.
+        link_html: URL to the HTML detail page.
+        link_pdf: URL to the PDF document.
+    """
+
+    code: str
+    origin: str
+    description: str
+    link_html: str
+    link_pdf: str
+
+    def to_dict(self) -> dict[str, str]:
+        """Serialize the entry to a dictionary.
+
+        Returns:
+            Dict with all fields.
+        """
+        return {
+            "code": self.code,
+            "origin": self.origin,
+            "description": self.description,
+            "link_html": self.link_html,
+            "link_pdf": self.link_pdf,
+        }
+
+
+@dataclass
+class BulletinSummary:
+    """Structured summary for one BOPA bulletin.
+
+    Attributes:
+        num: Bulletin number.
+        date: Publication date.
+        summary: List of disposition entries.
+    """
+
+    num: str
+    date: datetime
+    summary: list[BulletinSummaryEntry] = field(default_factory=list)
+
+    @property
+    def codes(self):
+        """List of non-empty disposition codes in this bulletin."""
+        return [entry.code for entry in self.summary if entry.code != "N/A"]
+
+    def to_dict(self) -> dict[str, object]:
+        """Serialize the bulletin summary to a dictionary.
+
+        Returns:
+            Dict with num, date (dd/mm/YYYY), and serialized entries.
+        """
+        return {
+            "num": self.num,
+            "date": self.date.strftime("%d/%m/%Y"),
+            "summary": [entry.to_dict() for entry in self.summary],
+        }

bopa_fetcher-0.1.0/src/bopa/service/__init__.py

@@ -0,0 +1,11 @@
+from ..models import BulletinArticle, BulletinSummary, BulletinSummaryEntry
+from .bulletin import Bulletin
+from .article import Article
+
+__all__ = [
+    "Bulletin",
+    "BulletinArticle",
+    "BulletinSummary",
+    "BulletinSummaryEntry",
+    "Article",
+]