educelab-hercdb 0.1.0.dev2__tar.gz → 0.2.2__tar.gz

educelab_hercdb-0.2.2/PKG-INFO

@@ -0,0 +1,202 @@
+Metadata-Version: 2.4
+Name: educelab-hercdb
+Version: 0.2.2
+Summary: Graph database API for Herculaneum data
+Author-email: Mami Hayashida <mami.hayashida@uky.edu>, Seth Parker <c.seth.parker@uky.edu>
+Project-URL: Repository, https://github.com/educelab/educelab-hercdb
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.32.5
+Provides-Extra: server
+Requires-Dist: fastapi>=0.124.4; extra == "server"
+Requires-Dist: neo4j>=5.20; extra == "server"
+Requires-Dist: numpy>=2.0; extra == "server"
+Requires-Dist: pandas>=2.2; extra == "server"
+Requires-Dist: prompt-toolkit; extra == "server"
+Requires-Dist: pydantic>=2.0; extra == "server"
+Requires-Dist: rapidfuzz>=3.0; extra == "server"
+Requires-Dist: uvicorn>=0.40.0; extra == "server"
+
+# EduceLab Herculaneum Graph Database API
+
+This API is considered a work in progress and can change at any moment.
+
+## Architecture
+
+![hercdb architecture](docs/hercdb_architecture.svg)
+
+## Setup Quick Start
+
+For a visual overview of the full setup process (beyond just this repo), see the quick start guide:
+
+![setup quick start](docs/hercdb_setup_quickstart.svg)
+
+For step-by-step server setup instructions, see [docs/SERVER_SETUP.md](docs/SERVER_SETUP.md).
+
+## Installation
+
+This package supports two install modes:
+
+### Client only (lightweight)
+
+For remote machines that only need to call the REST API:
+
+```shell
+pip install educelab-hercdb
+```
+
+This installs only the `requests` library. See [src/educelab/hercdb/client/README.md](src/educelab/hercdb/client/README.md) for client usage.
+
+### Server (full)
+
+For running the REST API server, loading data, or querying Neo4j directly:
+
+```shell
+pip install educelab-hercdb[server]
+```
+
+This adds `fastapi`, `neo4j`, `numpy`, `pandas`, `prompt-toolkit`, and `uvicorn`.
+
+## Development Setup
+
+```shell
+# Install base dependencies
+uv sync
+
+# Or with server extras (fastapi, neo4j, etc.)
+uv sync --extra server
+
+# Run commands in the environment
+uv run python -c "from educelab.hercdb.client import HercClient"
+
+# Or activate the venv directly
+source .venv/bin/activate
+```
+
+## Client Library
+
+```python
+from educelab.hercdb.client import HercClient
+
+client = HercClient(host="api.example.com", token="my-token")
+pherc = client.get_artifact_by_name("211")
+```
+
+See [src/educelab/hercdb/client/README.md](src/educelab/hercdb/client/README.md) for the full API reference.
+
+## Direct Database Connection
+
+For environments with the `server` extra installed, you can connect to Neo4j directly:
+
+```python
+from educelab import hercdb
+
+uri = "neo4j://localhost:7687"
+user = "foo"
+password = "bar"
+db = hercdb.connect(uri, user, password)
+if db.verify_connection():
+  print("Connected!")
+```
+
+If credentials are not passed directly, the package reads them from `~/.educedb` or environment variables. See [docs/SERVER_SETUP.md](docs/SERVER_SETUP.md) for configuration details.
+
+## REST API
+
+A FastAPI-based REST API is available for querying the database over HTTP. All endpoints require Bearer token authentication.
+
+```shell
+uv run uvicorn educelab.hercdb.rest.server:app --reload
+```
+
+Interactive API docs are available at `/docs` (Swagger UI) and `/redoc` (ReDoc) once the server is running.
+
+See [src/educelab/hercdb/rest/README.md](src/educelab/hercdb/rest/README.md) for endpoint documentation and [docs/SERVER_SETUP.md](docs/SERVER_SETUP.md) for production deployment.
+
+## Loading Data
+
+Data loading is done in two steps using the loader scripts. Both read CSV files from `input_data/`.
+
+### 1. Load metadata and UUIDs
+
+```shell
+uv run python src/educelab/hercdb/loader/metadata_loader.py
+```
+
+Reads (defaults):
+- `input_data/metadata_file.csv` - Pre-processed metadata file. (PHerc, Cornice, Pezzo, Disegni nodes and properties.)
+- `input_data/uuid_file.csv` - Pre-processed uuid file. (all EduceLabID added)
+
+Optional arguments:
+```shell
+uv run python src/educelab/hercdb/loader/metadata_loader.py \
+  --metadata path/to/metadata.csv \
+  --uuid path/to/uuid.csv
+```
+
+### 2. Load scan data
+
+```shell
+uv run python src/educelab/hercdb/loader/scan_loader.py
+```
+
+Reads (defaults):
+- `input_data/negatives.csv` - FlatbedScanDataset nodes
+- `input_data/pgs_datasets_20260601(in).csv` - PGSRaw nodes
+- `input_data/spectral_datasets_20260601_reconciled.csv` - SpectralRaw nodes
+
+By default (`--replace`) it deletes all existing PGSRaw/SpectralRaw nodes and reloads from scratch (FlatbedScanDataset is untouched); pass `--no-replace` to merge into existing data instead. Nodes are keyed on the scan `uuid`, so re-running is idempotent.
+
+Optional arguments:
+```shell
+uv run python src/educelab/hercdb/loader/scan_loader.py \
+  --negatives path/to/negatives.csv \
+  --photogrammetry path/to/pgs.csv \
+  --spectral path/to/spectral.csv \
+  --no-replace
+```
+
+**Note:** Run metadata_loader first since scan data links to EduceLabID nodes.
+
+## Reporting Tools
+
+### Scan completeness report
+
+`el-hercdb-scan-report` walks every PHerc and its hierarchy and writes two CSVs that flag artifacts needing first-time scans or re-scans.
+
+```shell
+# After `uv sync --extra server`, the entry point is on PATH:
+uv run el-hercdb-scan-report --out-dir ./tmp
+
+# Equivalent fallback without re-syncing:
+uv run python -m educelab.hercdb.cli.scan_completeness --out-dir ./tmp
+```
+
+Outputs (default names):
+
+- `scan_completeness_full.csv` — one row per (artifact, UUID) pair, plus blank-UUID sentinel rows for artifacts that have no EduceLabID assigned. Columns: PHerc, Cornice, Pezzo, UUID, PGS Status, PGS Latest Complete Date, PGS Path, Spectral Status, Spectral Latest Complete Date, Spectral Path, Institution.
+- `scan_completeness_issues.csv` — same shape, filtered to rows where PGS or Spectral is missing/incomplete, or the artifact has no UUID at all.
+
+Status values are `complete` / `incomplete` / `missing` when the artifact has a UUID, and **blank** when it does not (so "UUID assigned but no scan" stays distinguishable from "no UUID even assigned"). Files are written with `utf-8-sig` so Excel opens them with correct character encoding.
+
+The report walks `REPLACES` relationships between EduceLabIDs, so pre-replacement scans on a retired predecessor UUID still count toward the artifact's coverage.
+
+## Temporary Scripts and Notes
+
+The `tmp/` directory contains temporary scripts, notes, and other informal resources shared among the team. Contents are version controlled but considered ephemeral — they may be rewritten or deleted at any time and should not be relied upon as stable code.
+
+### Delete all data
+
+To clear the database before reloading:
+
+```python
+from educelab.hercdb.loader import PhercGraphDatabaseLoader
+loader = PhercGraphDatabaseLoader()
+loader._delete_all_nodes()
+```

educelab_hercdb-0.2.2/README.md

@@ -0,0 +1,177 @@
+# EduceLab Herculaneum Graph Database API
+
+This API is considered a work in progress and can change at any moment.
+
+## Architecture
+
+![hercdb architecture](docs/hercdb_architecture.svg)
+
+## Setup Quick Start
+
+For a visual overview of the full setup process (beyond just this repo), see the quick start guide:
+
+![setup quick start](docs/hercdb_setup_quickstart.svg)
+
+For step-by-step server setup instructions, see [docs/SERVER_SETUP.md](docs/SERVER_SETUP.md).
+
+## Installation
+
+This package supports two install modes:
+
+### Client only (lightweight)
+
+For remote machines that only need to call the REST API:
+
+```shell
+pip install educelab-hercdb
+```
+
+This installs only the `requests` library. See [src/educelab/hercdb/client/README.md](src/educelab/hercdb/client/README.md) for client usage.
+
+### Server (full)
+
+For running the REST API server, loading data, or querying Neo4j directly:
+
+```shell
+pip install educelab-hercdb[server]
+```
+
+This adds `fastapi`, `neo4j`, `numpy`, `pandas`, `prompt-toolkit`, and `uvicorn`.
+
+## Development Setup
+
+```shell
+# Install base dependencies
+uv sync
+
+# Or with server extras (fastapi, neo4j, etc.)
+uv sync --extra server
+
+# Run commands in the environment
+uv run python -c "from educelab.hercdb.client import HercClient"
+
+# Or activate the venv directly
+source .venv/bin/activate
+```
+
+## Client Library
+
+```python
+from educelab.hercdb.client import HercClient
+
+client = HercClient(host="api.example.com", token="my-token")
+pherc = client.get_artifact_by_name("211")
+```
+
+See [src/educelab/hercdb/client/README.md](src/educelab/hercdb/client/README.md) for the full API reference.
+
+## Direct Database Connection
+
+For environments with the `server` extra installed, you can connect to Neo4j directly:
+
+```python
+from educelab import hercdb
+
+uri = "neo4j://localhost:7687"
+user = "foo"
+password = "bar"
+db = hercdb.connect(uri, user, password)
+if db.verify_connection():
+  print("Connected!")
+```
+
+If credentials are not passed directly, the package reads them from `~/.educedb` or environment variables. See [docs/SERVER_SETUP.md](docs/SERVER_SETUP.md) for configuration details.
+
+## REST API
+
+A FastAPI-based REST API is available for querying the database over HTTP. All endpoints require Bearer token authentication.
+
+```shell
+uv run uvicorn educelab.hercdb.rest.server:app --reload
+```
+
+Interactive API docs are available at `/docs` (Swagger UI) and `/redoc` (ReDoc) once the server is running.
+
+See [src/educelab/hercdb/rest/README.md](src/educelab/hercdb/rest/README.md) for endpoint documentation and [docs/SERVER_SETUP.md](docs/SERVER_SETUP.md) for production deployment.
+
+## Loading Data
+
+Data loading is done in two steps using the loader scripts. Both read CSV files from `input_data/`.
+
+### 1. Load metadata and UUIDs
+
+```shell
+uv run python src/educelab/hercdb/loader/metadata_loader.py
+```
+
+Reads (defaults):
+- `input_data/metadata_file.csv` - Pre-processed metadata file. (PHerc, Cornice, Pezzo, Disegni nodes and properties.)
+- `input_data/uuid_file.csv` - Pre-processed uuid file. (all EduceLabID added)
+
+Optional arguments:
+```shell
+uv run python src/educelab/hercdb/loader/metadata_loader.py \
+  --metadata path/to/metadata.csv \
+  --uuid path/to/uuid.csv
+```
+
+### 2. Load scan data
+
+```shell
+uv run python src/educelab/hercdb/loader/scan_loader.py
+```
+
+Reads (defaults):
+- `input_data/negatives.csv` - FlatbedScanDataset nodes
+- `input_data/pgs_datasets_20260601(in).csv` - PGSRaw nodes
+- `input_data/spectral_datasets_20260601_reconciled.csv` - SpectralRaw nodes
+
+By default (`--replace`) it deletes all existing PGSRaw/SpectralRaw nodes and reloads from scratch (FlatbedScanDataset is untouched); pass `--no-replace` to merge into existing data instead. Nodes are keyed on the scan `uuid`, so re-running is idempotent.
+
+Optional arguments:
+```shell
+uv run python src/educelab/hercdb/loader/scan_loader.py \
+  --negatives path/to/negatives.csv \
+  --photogrammetry path/to/pgs.csv \
+  --spectral path/to/spectral.csv \
+  --no-replace
+```
+
+**Note:** Run metadata_loader first since scan data links to EduceLabID nodes.
+
+## Reporting Tools
+
+### Scan completeness report
+
+`el-hercdb-scan-report` walks every PHerc and its hierarchy and writes two CSVs that flag artifacts needing first-time scans or re-scans.
+
+```shell
+# After `uv sync --extra server`, the entry point is on PATH:
+uv run el-hercdb-scan-report --out-dir ./tmp
+
+# Equivalent fallback without re-syncing:
+uv run python -m educelab.hercdb.cli.scan_completeness --out-dir ./tmp
+```
+
+Outputs (default names):
+
+- `scan_completeness_full.csv` — one row per (artifact, UUID) pair, plus blank-UUID sentinel rows for artifacts that have no EduceLabID assigned. Columns: PHerc, Cornice, Pezzo, UUID, PGS Status, PGS Latest Complete Date, PGS Path, Spectral Status, Spectral Latest Complete Date, Spectral Path, Institution.
+- `scan_completeness_issues.csv` — same shape, filtered to rows where PGS or Spectral is missing/incomplete, or the artifact has no UUID at all.
+
+Status values are `complete` / `incomplete` / `missing` when the artifact has a UUID, and **blank** when it does not (so "UUID assigned but no scan" stays distinguishable from "no UUID even assigned"). Files are written with `utf-8-sig` so Excel opens them with correct character encoding.
+
+The report walks `REPLACES` relationships between EduceLabIDs, so pre-replacement scans on a retired predecessor UUID still count toward the artifact's coverage.
+
+## Temporary Scripts and Notes
+
+The `tmp/` directory contains temporary scripts, notes, and other informal resources shared among the team. Contents are version controlled but considered ephemeral — they may be rewritten or deleted at any time and should not be relied upon as stable code.
+
+### Delete all data
+
+To clear the database before reloading:
+
+```python
+from educelab.hercdb.loader import PhercGraphDatabaseLoader
+loader = PhercGraphDatabaseLoader()
+loader._delete_all_nodes()
+```

{educelab_hercdb-0.1.0.dev2 → educelab_hercdb-0.2.2}/pyproject.toml

@@ -4,16 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [tool.setuptools.packages.find]
 where = ["src/"]
-include = ["educelab.hercdb"]
+include = ["educelab.hercdb*"]
 
 [project]
 name = "educelab-hercdb"
-version = "0.1.0.dev2"
+version = "0.2.2"
 dependencies = [
-    "neo4j>=5.20",
-    "numpy>=2.0",
-    "pandas>=2.2",
-    "prompt-toolkit"
+    "requests>=2.32.5",
 ]
 requires-python = ">= 3.10"
 authors = [
@@ -32,8 +29,30 @@ classifiers = [
     "Programming Language :: Python :: 3.12",
 ]
 
+[project.optional-dependencies]
+server = [
+    "fastapi>=0.124.4",
+    "neo4j>=5.20",
+    "numpy>=2.0",
+    "pandas>=2.2",
+    "prompt-toolkit",
+    "pydantic>=2.0",
+    "rapidfuzz>=3.0",
+    "uvicorn>=0.40.0",
+]
+
 [project.urls]
-Repository = "https://gitlab.com/educelab/herculaneum-graph-db"
+Repository = "https://github.com/educelab/educelab-hercdb"
 
 [project.scripts]
-el-hercdb-search = "educelab.hercdb.apps.search:main"
+el-hercdb-scan-report = "educelab.hercdb.cli.scan_completeness:main"
+el-hercdb-sample-uuid-check = "educelab.hercdb.cli.sample_uuid_check:main"
+el-hercdb-pipeline-cleanup = "educelab.hercdb.cli.cleanup_pipelines:main"
+
+[dependency-groups]
+dev = [
+    "ipykernel>=7.1.0",
+    "matplotlib>=3.10.9",
+    "networkx>=3.4.2",
+    "pyyaml>=6.0.3",
+]

educelab_hercdb-0.2.2/src/educelab/hercdb/__init__.py

@@ -0,0 +1,16 @@
+try:
+    from educelab.hercdb import config
+    from educelab.hercdb.db import (
+        connect,
+        GraphDBConnection,
+        DatasetType,
+    )
+
+    __all__ = [
+        "config",
+        "connect",
+        "GraphDBConnection",
+        "DatasetType",
+    ]
+except ImportError:
+    pass

educelab_hercdb-0.2.2/src/educelab/hercdb/api.py

@@ -0,0 +1,14 @@
+"""Backward compatibility shim - use educelab.hercdb.db instead."""
+
+# Re-export everything from db module for backward compatibility
+from educelab.hercdb.db import (
+    GraphDBConnection,
+    connect,
+    DatasetType,
+)
+
+__all__ = [
+    "GraphDBConnection",
+    "connect",
+    "DatasetType",
+]

educelab_hercdb-0.2.2/src/educelab/hercdb/cli/__init__.py

@@ -0,0 +1 @@
+"""Command-line interface tools for hercdb."""

educelab_hercdb-0.2.2/src/educelab/hercdb/cli/cleanup_pipelines.py

@@ -0,0 +1,101 @@
+"""Delete pipeline records from Neo4j by pipeline id (DB-admin tool).
+
+Companion to the pipeline-recording write path in the acquisition-workflow
+``submit_uber_pipeline.py``: while that path is being tested, every (dry-run or
+real) submission writes a Pipeline node keyed by its ``uber_job_id``. This tool
+lets a database administrator remove those test records again, straight against
+Neo4j (no REST server needed).
+
+Deleting a pipeline removes the Pipeline node, its Process nodes, and the output
+dataset nodes (PGSProcessed/SpectralProcessed/Registered/WebProcessed) those
+processes produced. Input/raw datasets (PGSRaw/SpectralRaw/...) are left
+untouched.
+
+Wired up as the ``el-hercdb-pipeline-cleanup`` shell command (see
+``[project.scripts]`` in pyproject.toml). Usage:
+
+    el-hercdb-pipeline-cleanup uber-1a2b3c4d [uber-...] [-y]
+    uv run el-hercdb-pipeline-cleanup --list          # show what exists first
+"""
+import argparse
+import sys
+
+from educelab import hercdb
+from educelab.hercdb import config
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description=(
+            "Delete pipeline records (Pipeline + Process + output datasets) "
+            "from Neo4j by pipeline id. Input/raw datasets are left untouched."
+        )
+    )
+    parser.add_argument(
+        "pipeline_ids", nargs="*",
+        help="Pipeline id(s) to delete (the uber_job_id printed by "
+             "submit_uber_pipeline.py).",
+    )
+    parser.add_argument(
+        "--list", action="store_true",
+        help="List all pipelines currently in the database and exit (no "
+             "deletion). Useful for finding ids to clean up.",
+    )
+    parser.add_argument(
+        "--yes", "-y", action="store_true",
+        help="Skip the confirmation prompt.",
+    )
+    args = parser.parse_args()
+
+    if not args.list and not args.pipeline_ids:
+        parser.error("provide one or more pipeline ids, or use --list.")
+
+    config.request_required()
+    db = hercdb.connect()
+    if not db.verify_connection():
+        print("Failed to connect to Neo4j.", file=sys.stderr)
+        sys.exit(1)
+
+    if args.list:
+        pipelines = db.get_all_pipeline_summaries()
+        if not pipelines:
+            print("No pipelines in the database.")
+            return
+        print(f"{len(pipelines)} pipeline(s):")
+        for p in pipelines:
+            pid = p.get("pipeline_id", "?")
+            uuid = p.get("artifact_uuid", "")
+            print(f"  - {pid}" + (f"  (artifact {uuid})" if uuid else ""))
+        return
+
+    if not args.yes:
+        print("About to delete the following pipeline(s):")
+        for pid in args.pipeline_ids:
+            print(f"  - {pid}")
+        confirm = input("Proceed? [y/N]: ").strip().lower()
+        if confirm not in ("y", "yes"):
+            print("Aborted.")
+            return
+
+    failed = False
+    for pid in args.pipeline_ids:
+        result = db.delete_pipeline(pid)
+        if result is None:
+            print(f"{pid}: not found")
+            failed = True
+        else:
+            print(
+                f"{pid}: deleted "
+                f"({result.get('processes_deleted', 0)} process(es), "
+                f"{result.get('output_datasets_deleted', 0)} output dataset(s))"
+            )
+
+    if failed:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        sys.exit(0)