refua-data 0.6.0__tar.gz

refua_data-0.6.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 JJ Ben-Joseph
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

refua_data-0.6.0/PKG-INFO

@@ -0,0 +1,193 @@
+Metadata-Version: 2.4
+Name: refua-data
+Version: 0.6.0
+Summary: Data ingestion, caching, and parquet materialization for the Refua drug discovery ecosystem.
+Author-email: JJ Ben-Joseph <jj@tensorspace.ai>
+License-Expression: MIT
+Project-URL: Homepage, https://agentcures.com/
+Project-URL: Repository, https://github.com/agentcures/refua
+Project-URL: Documentation, https://github.com/agentcures/refua#readme
+Project-URL: Issues, https://github.com/agentcures/refua/issues
+Keywords: drug discovery,data engineering,cheminformatics,bioinformatics,parquet,refua
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Chemistry
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: <3.15,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.2.2
+Requires-Dist: pyarrow>=18.0.0
+Requires-Dist: requests>=2.32.3
+Requires-Dist: tqdm>=4.66.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: ruff>=0.6.0; extra == "dev"
+Requires-Dist: mypy>=1.11.0; extra == "dev"
+Requires-Dist: pandas-stubs>=2.2.3.250527; extra == "dev"
+Requires-Dist: types-requests>=2.32.0.20241016; extra == "dev"
+Requires-Dist: build>=1.2.2; extra == "dev"
+Requires-Dist: twine>=6.1.0; extra == "dev"
+Dynamic: license-file
+
+# refua-data
+
+`refua-data` is the Refua data layer for drug discovery. It provides a curated dataset catalog, intelligent local caching, and parquet materialization optimized for downstream modeling and campaign workflows.
+
+## What it provides
+
+- A built-in catalog of useful drug-discovery datasets.
+- Dataset-aware download pipeline with cache reuse and metadata tracking.
+- Pluggable cache backend architecture (filesystem cache by default).
+- API dataset ingestion for paginated JSON endpoints (for example ChEMBL and UniProt).
+- HTTP conditional refresh support (`ETag` / `Last-Modified`) when enabled.
+- Incremental parquet materialization (chunked processing + partitioned parquet parts).
+- CLI for listing, fetching, and materializing datasets.
+- Source health checks via `validate-sources` for CI and environment diagnostics.
+- Rich dataset metadata snapshots (description + usage notes) persisted in cache metadata.
+
+## Included datasets
+
+The default catalog includes local-file/HTTP datasets plus API presets useful in drug discovery, including **ZINC**, **ChEMBL**, and **UniProt**.
+
+1. `zinc15_250k` (ZINC)
+2. `zinc15_tranche_druglike_instock` (ZINC tranche)
+3. `zinc15_tranche_druglike_agent` (ZINC tranche)
+4. `zinc15_tranche_druglike_wait_ok` (ZINC tranche)
+5. `zinc15_tranche_druglike_boutique` (ZINC tranche)
+6. `zinc15_tranche_druglike_annotated` (ZINC tranche)
+7. `tox21`
+8. `bbbp`
+9. `bace`
+10. `clintox`
+11. `sider`
+12. `hiv`
+13. `muv`
+14. `esol`
+15. `freesolv`
+16. `lipophilicity`
+17. `pcba`
+18. `chembl_activity_ki_human`
+19. `chembl_activity_ic50_human`
+20. `chembl_assays_binding_human`
+21. `chembl_targets_human_single_protein`
+22. `chembl_molecules_phase3plus`
+23. `uniprot_human_reviewed`
+24. `uniprot_human_kinases`
+25. `uniprot_human_gpcr`
+26. `uniprot_human_ion_channels`
+27. `uniprot_human_transporters`
+
+Most of these are distributed through MoleculeNet/DeepChem mirrors and retain upstream licensing terms.
+ChEMBL and UniProt presets are fetched through their public REST APIs and cached locally as JSONL.
+ZINC tranche presets aggregate multiple tranche files per dataset (drug-like MW B-K and logP A-K bins,
+reactivity A/B/C/E) into one cached tabular source during fetch.
+
+## Install
+
+```bash
+cd refua-data
+pip install -e .
+```
+
+## CLI quickstart
+
+List datasets:
+
+```bash
+refua-data list
+```
+
+Validate all dataset sources:
+
+```bash
+refua-data validate-sources
+```
+
+Validate a subset and fail CI on probe failures:
+
+```bash
+refua-data validate-sources chembl_activity_ki_human uniprot_human_kinases --fail-on-error
+```
+
+JSON output for automation:
+
+```bash
+refua-data validate-sources --json --fail-on-error
+```
+
+For datasets with multiple mirrors, source validation succeeds when at least one configured source
+is reachable. Failed fallback attempts are included in the result details.
+
+Fetch raw data with cache:
+
+```bash
+refua-data fetch zinc15_250k
+```
+
+Fetch API-based presets:
+
+```bash
+refua-data fetch chembl_activity_ki_human
+refua-data fetch uniprot_human_kinases
+```
+
+Materialize parquet:
+
+```bash
+refua-data materialize zinc15_250k
+```
+
+Refresh against remote metadata:
+
+```bash
+refua-data fetch zinc15_250k --refresh
+```
+
+For API datasets, `--refresh` re-runs the API query (with conditional headers on first page when available).
+
+## Cache layout
+
+By default, cache root is:
+
+- `~/.cache/refua-data`
+
+Override with:
+
+- `REFUA_DATA_HOME=/custom/path`
+
+Layout:
+
+- `raw/<dataset>/<version>/...` downloaded source files
+- `_meta/raw/<dataset>/<version>/...json` raw metadata (`etag`, `sha256`, API request signature, rows/pages, dataset description/usage metadata)
+- `parquet/<dataset>/<version>/part-*.parquet` materialized parquet parts
+- `_meta/parquet/<dataset>/<version>/manifest.json` parquet manifest metadata with dataset snapshot
+
+## Python API
+
+```python
+from refua_data import DatasetManager
+
+manager = DatasetManager()
+manager.fetch("zinc15_250k")
+manager.fetch("chembl_activity_ki_human")
+result = manager.materialize("zinc15_250k")
+print(result.parquet_dir)
+```
+
+`DataCache` is the default cache backend. You can pass a custom backend object that implements
+the same interface (`ensure`, `raw_file`, `raw_meta`, `parquet_dir`, `parquet_manifest`,
+`read_json`, `write_json`) to make storage pluggable.
+
+## Licensing notes
+
+- `refua-data` package code is MIT licensed.
+- Dataset content licenses are dataset-specific and controlled by upstream providers.
+- Always verify dataset licensing and allowed use before redistribution or commercial deployment.

refua_data-0.6.0/README.md

@@ -0,0 +1,154 @@
+# refua-data
+
+`refua-data` is the Refua data layer for drug discovery. It provides a curated dataset catalog, intelligent local caching, and parquet materialization optimized for downstream modeling and campaign workflows.
+
+## What it provides
+
+- A built-in catalog of useful drug-discovery datasets.
+- Dataset-aware download pipeline with cache reuse and metadata tracking.
+- Pluggable cache backend architecture (filesystem cache by default).
+- API dataset ingestion for paginated JSON endpoints (for example ChEMBL and UniProt).
+- HTTP conditional refresh support (`ETag` / `Last-Modified`) when enabled.
+- Incremental parquet materialization (chunked processing + partitioned parquet parts).
+- CLI for listing, fetching, and materializing datasets.
+- Source health checks via `validate-sources` for CI and environment diagnostics.
+- Rich dataset metadata snapshots (description + usage notes) persisted in cache metadata.
+
+## Included datasets
+
+The default catalog includes local-file/HTTP datasets plus API presets useful in drug discovery, including **ZINC**, **ChEMBL**, and **UniProt**.
+
+1. `zinc15_250k` (ZINC)
+2. `zinc15_tranche_druglike_instock` (ZINC tranche)
+3. `zinc15_tranche_druglike_agent` (ZINC tranche)
+4. `zinc15_tranche_druglike_wait_ok` (ZINC tranche)
+5. `zinc15_tranche_druglike_boutique` (ZINC tranche)
+6. `zinc15_tranche_druglike_annotated` (ZINC tranche)
+7. `tox21`
+8. `bbbp`
+9. `bace`
+10. `clintox`
+11. `sider`
+12. `hiv`
+13. `muv`
+14. `esol`
+15. `freesolv`
+16. `lipophilicity`
+17. `pcba`
+18. `chembl_activity_ki_human`
+19. `chembl_activity_ic50_human`
+20. `chembl_assays_binding_human`
+21. `chembl_targets_human_single_protein`
+22. `chembl_molecules_phase3plus`
+23. `uniprot_human_reviewed`
+24. `uniprot_human_kinases`
+25. `uniprot_human_gpcr`
+26. `uniprot_human_ion_channels`
+27. `uniprot_human_transporters`
+
+Most of these are distributed through MoleculeNet/DeepChem mirrors and retain upstream licensing terms.
+ChEMBL and UniProt presets are fetched through their public REST APIs and cached locally as JSONL.
+ZINC tranche presets aggregate multiple tranche files per dataset (drug-like MW B-K and logP A-K bins,
+reactivity A/B/C/E) into one cached tabular source during fetch.
+
+## Install
+
+```bash
+cd refua-data
+pip install -e .
+```
+
+## CLI quickstart
+
+List datasets:
+
+```bash
+refua-data list
+```
+
+Validate all dataset sources:
+
+```bash
+refua-data validate-sources
+```
+
+Validate a subset and fail CI on probe failures:
+
+```bash
+refua-data validate-sources chembl_activity_ki_human uniprot_human_kinases --fail-on-error
+```
+
+JSON output for automation:
+
+```bash
+refua-data validate-sources --json --fail-on-error
+```
+
+For datasets with multiple mirrors, source validation succeeds when at least one configured source
+is reachable. Failed fallback attempts are included in the result details.
+
+Fetch raw data with cache:
+
+```bash
+refua-data fetch zinc15_250k
+```
+
+Fetch API-based presets:
+
+```bash
+refua-data fetch chembl_activity_ki_human
+refua-data fetch uniprot_human_kinases
+```
+
+Materialize parquet:
+
+```bash
+refua-data materialize zinc15_250k
+```
+
+Refresh against remote metadata:
+
+```bash
+refua-data fetch zinc15_250k --refresh
+```
+
+For API datasets, `--refresh` re-runs the API query (with conditional headers on first page when available).
+
+## Cache layout
+
+By default, cache root is:
+
+- `~/.cache/refua-data`
+
+Override with:
+
+- `REFUA_DATA_HOME=/custom/path`
+
+Layout:
+
+- `raw/<dataset>/<version>/...` downloaded source files
+- `_meta/raw/<dataset>/<version>/...json` raw metadata (`etag`, `sha256`, API request signature, rows/pages, dataset description/usage metadata)
+- `parquet/<dataset>/<version>/part-*.parquet` materialized parquet parts
+- `_meta/parquet/<dataset>/<version>/manifest.json` parquet manifest metadata with dataset snapshot
+
+## Python API
+
+```python
+from refua_data import DatasetManager
+
+manager = DatasetManager()
+manager.fetch("zinc15_250k")
+manager.fetch("chembl_activity_ki_human")
+result = manager.materialize("zinc15_250k")
+print(result.parquet_dir)
+```
+
+`DataCache` is the default cache backend. You can pass a custom backend object that implements
+the same interface (`ensure`, `raw_file`, `raw_meta`, `parquet_dir`, `parquet_manifest`,
+`read_json`, `write_json`) to make storage pluggable.
+
+## Licensing notes
+
+- `refua-data` package code is MIT licensed.
+- Dataset content licenses are dataset-specific and controlled by upstream providers.
+- Always verify dataset licensing and allowed use before redistribution or commercial deployment.

refua_data-0.6.0/pyproject.toml

@@ -0,0 +1,84 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "refua-data"
+version = "0.6.0"
+description = "Data ingestion, caching, and parquet materialization for the Refua drug discovery ecosystem."
+readme = "README.md"
+requires-python = ">=3.11,<3.15"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{name = "JJ Ben-Joseph", email = "jj@tensorspace.ai"}]
+keywords = [
+  "drug discovery",
+  "data engineering",
+  "cheminformatics",
+  "bioinformatics",
+  "parquet",
+  "refua",
+]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Science/Research",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Topic :: Scientific/Engineering :: Bio-Informatics",
+  "Topic :: Scientific/Engineering :: Chemistry",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+  "pandas>=2.2.2",
+  "pyarrow>=18.0.0",
+  "requests>=2.32.3",
+  "tqdm>=4.66.0",
+]
+
+[project.urls]
+Homepage = "https://agentcures.com/"
+Repository = "https://github.com/agentcures/refua"
+Documentation = "https://github.com/agentcures/refua#readme"
+Issues = "https://github.com/agentcures/refua/issues"
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0.0",
+  "ruff>=0.6.0",
+  "mypy>=1.11.0",
+  "pandas-stubs>=2.2.3.250527",
+  "types-requests>=2.32.0.20241016",
+  "build>=1.2.2",
+  "twine>=6.1.0",
+]
+
+[project.scripts]
+refua-data = "refua_data.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.mypy]
+python_version = "3.11"
+warn_unused_ignores = true
+disallow_untyped_defs = true
+check_untyped_defs = true
+strict_optional = true

refua_data-0.6.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

refua_data-0.6.0/src/refua_data/__init__.py

@@ -0,0 +1,20 @@
+"""refua-data package API."""
+
+from .cache import CacheBackend, DataCache
+from .catalog import DatasetCatalog, get_default_catalog
+from .models import ApiDatasetConfig, DatasetDefinition, FetchResult, MaterializeResult
+from .pipeline import DatasetManager
+from .validation import SourceValidationResult
+
+__all__ = [
+    "ApiDatasetConfig",
+    "CacheBackend",
+    "DataCache",
+    "DatasetCatalog",
+    "DatasetDefinition",
+    "DatasetManager",
+    "FetchResult",
+    "MaterializeResult",
+    "SourceValidationResult",
+    "get_default_catalog",
+]

refua_data-0.6.0/src/refua_data/__main__.py

@@ -0,0 +1,6 @@
+"""Module runner for `python -m refua_data`."""
+
+from .cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

refua_data-0.6.0/src/refua_data/cache.py

@@ -0,0 +1,106 @@
+"""Filesystem cache primitives for dataset files and metadata."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from pathlib import Path
+from typing import Any, Protocol
+
+from .config import default_cache_root
+from .models import DatasetDefinition
+
+
+class CacheBackend(Protocol):
+    """Protocol for pluggable cache backends used by the pipeline."""
+
+    root: Path
+
+    def ensure(self) -> None:
+        ...
+
+    def raw_file(self, dataset: DatasetDefinition) -> Path:
+        ...
+
+    def raw_meta(self, dataset: DatasetDefinition) -> Path:
+        ...
+
+    def parquet_dir(self, dataset: DatasetDefinition) -> Path:
+        ...
+
+    def parquet_manifest(self, dataset: DatasetDefinition) -> Path:
+        ...
+
+    def read_json(self, path: Path) -> dict[str, Any] | None:
+        ...
+
+    def write_json(self, path: Path, payload: dict[str, Any]) -> None:
+        ...
+
+
+class DataCache:
+    """Filesystem-backed cache backend for raw + parquet artifacts."""
+
+    def __init__(self, root: Path | None = None):
+        self.root = (root or default_cache_root()).expanduser().resolve()
+
+    def ensure(self) -> None:
+        """Create required cache root directories."""
+        self.root.mkdir(parents=True, exist_ok=True)
+        self.root.joinpath("raw").mkdir(parents=True, exist_ok=True)
+        self.root.joinpath("parquet").mkdir(parents=True, exist_ok=True)
+        self.root.joinpath("_meta", "raw").mkdir(parents=True, exist_ok=True)
+        self.root.joinpath("_meta", "parquet").mkdir(parents=True, exist_ok=True)
+
+    def raw_file(self, dataset: DatasetDefinition) -> Path:
+        """Return raw file path for a dataset."""
+        filename = dataset.preferred_filename()
+        return self.root.joinpath("raw", dataset.dataset_id, dataset.version, filename)
+
+    def raw_meta(self, dataset: DatasetDefinition) -> Path:
+        """Return raw metadata path for a dataset."""
+        filename = f"{dataset.preferred_filename()}.json"
+        return self.root.joinpath("_meta", "raw", dataset.dataset_id, dataset.version, filename)
+
+    def parquet_dir(self, dataset: DatasetDefinition) -> Path:
+        """Return parquet output directory for a dataset."""
+        return self.root.joinpath("parquet", dataset.dataset_id, dataset.version)
+
+    def parquet_manifest(self, dataset: DatasetDefinition) -> Path:
+        """Return parquet manifest metadata path for a dataset."""
+        return self.root.joinpath(
+            "_meta",
+            "parquet",
+            dataset.dataset_id,
+            dataset.version,
+            "manifest.json",
+        )
+
+    def read_json(self, path: Path) -> dict[str, Any] | None:
+        """Read JSON metadata if it exists."""
+        if not path.exists():
+            return None
+        return json.loads(path.read_text(encoding="utf-8"))
+
+    def write_json(self, path: Path, payload: dict[str, Any]) -> None:
+        """Write JSON metadata atomically."""
+        path.parent.mkdir(parents=True, exist_ok=True)
+        tmp_path = path.with_suffix(path.suffix + ".tmp")
+        tmp_path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
+        os.replace(tmp_path, path)
+
+
+_CHUNK_SIZE = 4 * 1024 * 1024
+
+
+def sha256_file(path: Path) -> str:
+    """Compute the SHA256 checksum of a file."""
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        while True:
+            chunk = handle.read(_CHUNK_SIZE)
+            if not chunk:
+                break
+            digest.update(chunk)
+    return digest.hexdigest()