oracc-parser 0.1.0__tar.gz

oracc_parser-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Shahar Spencer
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

oracc_parser-0.1.0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.4
+Name: oracc-parser
+Version: 0.1.0
+Summary: Download and parse ORACC cuneiform text projects into ML-ready formats
+Author: Avital Romach, Shahar Spencer, Claude Sonnet 4.6
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/shaharspencer/oracc-parser
+Project-URL: Issues, https://github.com/shaharspencer/oracc-parser/issues
+Keywords: oracc,cuneiform,akkadian,nlp,digital-humanities
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.31
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: tqdm>=4.65
+Requires-Dist: beautifulsoup4>=4.12
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: httpx>=0.24; extra == "dev"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.100; extra == "server"
+Requires-Dist: uvicorn>=0.22; extra == "server"
+Provides-Extra: notebooks
+Requires-Dist: jupyter; extra == "notebooks"
+Requires-Dist: matplotlib; extra == "notebooks"
+Dynamic: license-file
+
+# oracc-parser
+
+A Python tool to download and parse [ORACC](http://oracc.museum.upenn.edu/) cuneiform text projects into machine-learning-ready formats (JSONL, CSV, pandas DataFrames).
+
+## Features
+
+- **Download** — Fetch project ZIPs directly from ORACC or Zenodo
+- **Parse** — Convert raw ORACC JSON into structured data
+- **Export** — Save datasets as JSONL, CSV, or pandas DataFrames
+- **Configure** — Control handling of broken signs and POS masking using `RunConfig`
+
+## Installation
+
+
+```bash
+git clone https://github.com/shaharspencer/oracc-parser.git
+cd oracc-parser
+pip install -e ".[dev]"
+```
+
+## Getting Started — Notebooks
+
+The easiest way to explore oracc-parser is through the interactive notebooks.
+Start with notebook 01 — it downloads all the data you need from Zenodo automatically.
+
+| Notebook | What you'll learn |
+|---|---|
+| [`01_quickstart.ipynb`](notebooks/01_quickstart.ipynb) | Download the dataset → parse a project from pre-processed CSVs → explore transliterations, translations, and metadata → export |
+| [`02_reference_data.ipynb`](notebooks/02_reference_data.ipynb) | Browse all projects in the dataset, query catalogues, explore bundled reference data (provenance, periods, sign list, POS tags) |
+| [`03_configure_and_export.ipynb`](notebooks/03_configure_and_export.ipynb) | All `RunConfig` options — word-level and sign-level break filtering, POS masking — combining multiple projects and exporting datasets |
+| [`04_oracc_json_processing.ipynb`](notebooks/04_oracc_json_processing.ipynb) | Advanced: understand the raw ORACC JSON structure, the JSON → TabletRecord → CSV pipeline, and how to download and parse projects not in the dataset |
+
+```bash
+pip install oracc-parser[notebooks]
+jupyter notebook notebooks/
+```
+
+## Quick Example
+
+```python
+from oracc_parser import parse_project, RunConfig, get_full_flat_table
+
+# Parse 5 tablets from SAA 01 (Neo-Assyrian royal letters)
+records = parse_project("saao/saa01", config=RunConfig(limit=5))
+
+# Get a flat DataFrame — no nesting, ready for analysis
+df = get_full_flat_table(records)
+df.to_json("dataset.jsonl", orient="records", lines=True)
+```
+
+## Configuration
+
+You can customize the parsing process using `RunConfig`:
+
+```python
+from oracc_parser import parse_project, RunConfig
+
+records = parse_project("saao/saa01", config=RunConfig(
+    limit=10,
+    max_break_fraction=0.5,   # word-level: drop words that are >50% broken
+    drop_missing=True,        # sign-level: drop [x] signs from Unicode output
+    drop_damaged=False,       # sign-level: keep ⸢x⸣ signs in Unicode output
+    mask_pos=["PN", "DN"],    # replace personal/divine names with tag
+))
+```
+
+### Two independent levels of break filtering
+
+`RunConfig` provides two distinct ways to handle damaged or missing text,
+operating at different granularities and affecting different outputs:
+
+| Parameter | Level | Affects | How it works |
+|---|---|---|---|
+| `max_break_fraction` | **Word** | Transliteration, normalization, lemmatization | Each word has a `break_perc` (fraction of its signs that are broken). Words exceeding this threshold are replaced with `X`. Default `1.0` keeps all words. |
+| `drop_missing` | **Sign** | Unicode cuneiform only | Drops individual signs marked `[x]` (completely lost). |
+| `drop_damaged` | **Sign** | Unicode cuneiform only | Drops individual signs marked `⸢x⸣` (partially legible). |
+
+> **Note:** Because word-level and sign-level filtering use different thresholds
+> and different granularities, **the text outputs and the Unicode cuneiform output
+> are not necessarily aligned**. A word kept in the transliteration (because its
+> average damage is below `max_break_fraction`) may still have individual signs
+> dropped from the Unicode output if `drop_missing` / `drop_damaged` are enabled.
+
+### Other options
+
+| Parameter | Default | Description |
+|---|---|---|
+| `limit` | `None` | Only parse the first N texts (useful for testing) |
+| `keep_word_segmentation` | `True` | Preserve word boundaries in Unicode cuneiform output |
+| `mask_pos` | `[]` | Replace words of certain POS tags with the tag name |
+| `languages` | `["Akkadian"]` | Which languages to include when downloading projects |
+| `use_cache` | `True` | Use cached results if available |
+
+All reference data is bundled with the package, so you don't need to configure external paths unless you are customizing `oracc_parser.settings`.
+
+## CLI
+
+```bash
+oracc-parser download --project saao/saa01
+oracc-parser parse --project saao/saa01 --limit 5 --format jsonl --output saa01.jsonl
+```
+
+## Heavy Data (Zenodo)
+
+Large data files (ORACC ZIPs, cached translations, Pleiades data) are on Zenodo:
+
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18643122.svg)](https://doi.org/10.5281/zenodo.18643122)
+
+```bash
+python scripts/download_zenodo_data.py
+```
+
+## Running Tests
+
+```bash
+pytest tests/ -v     # 98 tests
+```
+
+## Known Limitations
+
+- **Chronology**: Period-to-year normalization is optimized for the **1st Millennium BCE**.
+- **Language**: Parsing is primarily validated on **Akkadian** projects.
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Credits
+
+Based on code by Niek Veldhuis ([Compass](https://github.com/niekveldhuis/compass)) and adapted for the BEn Project.

oracc_parser-0.1.0/README.md

@@ -0,0 +1,130 @@
+# oracc-parser
+
+A Python tool to download and parse [ORACC](http://oracc.museum.upenn.edu/) cuneiform text projects into machine-learning-ready formats (JSONL, CSV, pandas DataFrames).
+
+## Features
+
+- **Download** — Fetch project ZIPs directly from ORACC or Zenodo
+- **Parse** — Convert raw ORACC JSON into structured data
+- **Export** — Save datasets as JSONL, CSV, or pandas DataFrames
+- **Configure** — Control handling of broken signs and POS masking using `RunConfig`
+
+## Installation
+
+
+```bash
+git clone https://github.com/shaharspencer/oracc-parser.git
+cd oracc-parser
+pip install -e ".[dev]"
+```
+
+## Getting Started — Notebooks
+
+The easiest way to explore oracc-parser is through the interactive notebooks.
+Start with notebook 01 — it downloads all the data you need from Zenodo automatically.
+
+| Notebook | What you'll learn |
+|---|---|
+| [`01_quickstart.ipynb`](notebooks/01_quickstart.ipynb) | Download the dataset → parse a project from pre-processed CSVs → explore transliterations, translations, and metadata → export |
+| [`02_reference_data.ipynb`](notebooks/02_reference_data.ipynb) | Browse all projects in the dataset, query catalogues, explore bundled reference data (provenance, periods, sign list, POS tags) |
+| [`03_configure_and_export.ipynb`](notebooks/03_configure_and_export.ipynb) | All `RunConfig` options — word-level and sign-level break filtering, POS masking — combining multiple projects and exporting datasets |
+| [`04_oracc_json_processing.ipynb`](notebooks/04_oracc_json_processing.ipynb) | Advanced: understand the raw ORACC JSON structure, the JSON → TabletRecord → CSV pipeline, and how to download and parse projects not in the dataset |
+
+```bash
+pip install oracc-parser[notebooks]
+jupyter notebook notebooks/
+```
+
+## Quick Example
+
+```python
+from oracc_parser import parse_project, RunConfig, get_full_flat_table
+
+# Parse 5 tablets from SAA 01 (Neo-Assyrian royal letters)
+records = parse_project("saao/saa01", config=RunConfig(limit=5))
+
+# Get a flat DataFrame — no nesting, ready for analysis
+df = get_full_flat_table(records)
+df.to_json("dataset.jsonl", orient="records", lines=True)
+```
+
+## Configuration
+
+You can customize the parsing process using `RunConfig`:
+
+```python
+from oracc_parser import parse_project, RunConfig
+
+records = parse_project("saao/saa01", config=RunConfig(
+    limit=10,
+    max_break_fraction=0.5,   # word-level: drop words that are >50% broken
+    drop_missing=True,        # sign-level: drop [x] signs from Unicode output
+    drop_damaged=False,       # sign-level: keep ⸢x⸣ signs in Unicode output
+    mask_pos=["PN", "DN"],    # replace personal/divine names with tag
+))
+```
+
+### Two independent levels of break filtering
+
+`RunConfig` provides two distinct ways to handle damaged or missing text,
+operating at different granularities and affecting different outputs:
+
+| Parameter | Level | Affects | How it works |
+|---|---|---|---|
+| `max_break_fraction` | **Word** | Transliteration, normalization, lemmatization | Each word has a `break_perc` (fraction of its signs that are broken). Words exceeding this threshold are replaced with `X`. Default `1.0` keeps all words. |
+| `drop_missing` | **Sign** | Unicode cuneiform only | Drops individual signs marked `[x]` (completely lost). |
+| `drop_damaged` | **Sign** | Unicode cuneiform only | Drops individual signs marked `⸢x⸣` (partially legible). |
+
+> **Note:** Because word-level and sign-level filtering use different thresholds
+> and different granularities, **the text outputs and the Unicode cuneiform output
+> are not necessarily aligned**. A word kept in the transliteration (because its
+> average damage is below `max_break_fraction`) may still have individual signs
+> dropped from the Unicode output if `drop_missing` / `drop_damaged` are enabled.
+
+### Other options
+
+| Parameter | Default | Description |
+|---|---|---|
+| `limit` | `None` | Only parse the first N texts (useful for testing) |
+| `keep_word_segmentation` | `True` | Preserve word boundaries in Unicode cuneiform output |
+| `mask_pos` | `[]` | Replace words of certain POS tags with the tag name |
+| `languages` | `["Akkadian"]` | Which languages to include when downloading projects |
+| `use_cache` | `True` | Use cached results if available |
+
+All reference data is bundled with the package, so you don't need to configure external paths unless you are customizing `oracc_parser.settings`.
+
+## CLI
+
+```bash
+oracc-parser download --project saao/saa01
+oracc-parser parse --project saao/saa01 --limit 5 --format jsonl --output saa01.jsonl
+```
+
+## Heavy Data (Zenodo)
+
+Large data files (ORACC ZIPs, cached translations, Pleiades data) are on Zenodo:
+
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18643122.svg)](https://doi.org/10.5281/zenodo.18643122)
+
+```bash
+python scripts/download_zenodo_data.py
+```
+
+## Running Tests
+
+```bash
+pytest tests/ -v     # 98 tests
+```
+
+## Known Limitations
+
+- **Chronology**: Period-to-year normalization is optimized for the **1st Millennium BCE**.
+- **Language**: Parsing is primarily validated on **Akkadian** projects.
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Credits
+
+Based on code by Niek Veldhuis ([Compass](https://github.com/niekveldhuis/compass)) and adapted for the BEn Project.

oracc_parser-0.1.0/oracc_parser/__init__.py

@@ -0,0 +1,34 @@
+"""
+oracc-parser: Download and parse ORACC cuneiform text projects.
+"""
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+# Public re-exports for convenience
+from oracc_parser.pipeline import (  # noqa: F401
+    export_to_csv,
+    export_to_jsonl,
+    parse_project,
+    parse_project_from_word_csvs,
+    records_to_word_dataframes,
+    save_project_catalogue,
+    load_project_catalogue,
+    reference_data,
+    get_metadata_table,
+    get_transliterations,
+    get_normalizations,
+    get_lemmatizations,
+    get_unicode_texts,
+    get_translations,
+    get_full_flat_table,
+)
+from oracc_parser.io.word_csv import (  # noqa: F401
+    load_word_csvs_from_dir,
+    load_word_csvs_from_zenodo,
+    save_word_csv,
+)
+from oracc_parser.models.config import RunConfig  # noqa: F401
+from oracc_parser.metadata.populate import enrich_catalogue_df  # noqa: F401
+from oracc_parser.download.pleiades import PleiadesData  # noqa: F401
+

oracc_parser-0.1.0/oracc_parser/cache.py

@@ -0,0 +1,251 @@
+"""
+JSON caching for parsed TabletRecord objects.
+
+Parsed tablets are expensive to produce (long runtimes due to CDL tree
+traversal, sign parsing, and translation downloads).  This module caches
+the full result including a **config fingerprint**.
+
+On reload:
+- If the current config matches the cached fingerprint → **instant return**
+  (everything is reused, including string representations)
+- If the config differs → the cached **words** are reused and string
+  representations are rebuilt (cheap, no re-parsing needed)
+- If not cached at all → full parse from scratch
+
+Cache layout::
+
+    {cache_dir}/tablets/{project}/{text_id}.json
+
+Each file is a JSON wrapper::
+
+    {"config_fingerprint": "a1b2c3d4", "record": { ... TabletRecord ... }}
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+from pathlib import Path
+
+from oracc_parser.utils.logger import get_logger
+
+logger = get_logger()
+
+
+# ---------------------------------------------------------------------------
+# Config fingerprinting
+# ---------------------------------------------------------------------------
+
+# These RunConfig fields affect the parsed output.
+# Everything else (USE_CACHE, CACHE_DIR, limit, languages) does NOT.
+_OUTPUT_AFFECTING_FIELDS = (
+    "drop_missing",
+    "drop_damaged",
+    "keep_word_segmentation",
+    "mask_pos",
+    "max_break_fraction",
+)
+
+
+def config_fingerprint(config) -> str:
+    """Compute a short, stable hash of the output-affecting config options.
+
+    Args:
+        config: A ``RunConfig`` instance.
+
+    Returns:
+        8-char hex string (e.g. ``"a1b2c3d4"``).
+    """
+    key = {}
+    for field in _OUTPUT_AFFECTING_FIELDS:
+        val = getattr(config, field)
+        if isinstance(val, list):
+            val = sorted(val)
+        key[field] = val
+
+    raw = json.dumps(key, sort_keys=True)
+    return hashlib.sha256(raw.encode()).hexdigest()[:8]
+
+
+# ---------------------------------------------------------------------------
+# Path helpers
+# ---------------------------------------------------------------------------
+
+
+def _resolve_cache_dir(cache_dir: str | None = None) -> Path:
+    """Return the base cache directory."""
+    if cache_dir:
+        return Path(cache_dir)
+    from oracc_parser.settings import CACHE_DIR as settings_CACHE_DIR
+    return settings_CACHE_DIR
+
+
+def _tablet_path(
+    project: str,
+    text_id: str,
+    cache_dir: str | None = None,
+) -> Path:
+    """Return the JSON file path for a cached tablet."""
+    base = _resolve_cache_dir(cache_dir) / "tablets"
+    project_dir = project.replace("/", "-")
+    return base / project_dir / f"{text_id}.json"
+
+
+# ---------------------------------------------------------------------------
+# Load / Save
+# ---------------------------------------------------------------------------
+
+
+def load_cached_tablet(
+    project: str,
+    text_id: str,
+    config,
+    cache_dir: str | None = None,
+) -> "TabletRecord | None":
+    """Load a cached tablet, rebuilding string reps only if config changed.
+
+    Two fast paths:
+
+    1. **Config match** — the cached fingerprint matches the current config.
+       The full record (including string representations) is returned as-is.
+       This is the fastest path.
+
+    2. **Config mismatch** — the words and metadata are reused, but string
+       representations are rebuilt with the current config.  This avoids
+       the expensive CDL parsing + translation download.
+
+    Args:
+        project: ORACC project path, e.g. ``"saao/saa01"``.
+        text_id: Text identifier, e.g. ``"P334189"``.
+        config: ``RunConfig`` instance.
+        cache_dir: Custom cache directory (overrides settings).
+
+    Returns:
+        The TabletRecord (possibly with rebuilt strings), or ``None``.
+    """
+    from oracc_parser.models.tablet import TabletRecord
+    from oracc_parser.parsing.parse_content import (
+        _add_word_level_representations,
+        _add_unicode_representation,
+    )
+
+    path = _tablet_path(project, text_id, cache_dir)
+    if not path.exists():
+        return None
+
+    try:
+        raw = path.read_text(encoding="utf-8")
+        wrapper = json.loads(raw)
+
+        # Handle both new wrapper format and legacy bare-record format
+        if "record" in wrapper and "config_fingerprint" in wrapper:
+            cached_fp = wrapper["config_fingerprint"]
+            record = TabletRecord.model_validate(wrapper["record"])
+        else:
+            # Legacy format (bare TabletRecord JSON) — always rebuild
+            cached_fp = None
+            record = TabletRecord.model_validate(wrapper)
+
+        current_fp = config_fingerprint(config)
+
+        if cached_fp == current_fp:
+            # Fast path: config matches → everything is valid
+            return record
+
+        # Config changed → rebuild string representations from cached words
+        record.content = _add_word_level_representations(
+            record.content, config.mask_pos, config.max_break_fraction
+        )
+        record.content = _add_unicode_representation(
+            record.content,
+            drop_missing=config.drop_missing,
+            drop_damaged=config.drop_damaged,
+            keep_segmentation=config.keep_word_segmentation,
+        )
+        return record
+
+    except Exception as e:
+        logger.warning(f"Corrupt cache file {path}, will re-parse: {e}")
+        path.unlink(missing_ok=True)
+        return None
+
+
+def save_tablet_to_cache(
+    record: "TabletRecord",
+    project: str,
+    text_id: str,
+    config,
+    cache_dir: str | None = None,
+) -> None:
+    """Persist a TabletRecord to the JSON cache with a config fingerprint.
+
+    The saved file includes the config fingerprint so that on reload
+    we can skip string rebuilding when the config hasn't changed.
+
+    Args:
+        record: The parsed tablet to cache.
+        project: ORACC project path.
+        text_id: Text identifier.
+        config: ``RunConfig`` instance (its fingerprint is stored).
+        cache_dir: Custom cache directory.
+    """
+    path = _tablet_path(project, text_id, cache_dir)
+    path.parent.mkdir(parents=True, exist_ok=True)
+
+    wrapper = {
+        "config_fingerprint": config_fingerprint(config),
+        "record": record.model_dump(mode="python"),
+    }
+
+    try:
+        path.write_text(
+            json.dumps(wrapper, indent=1, default=str, ensure_ascii=False),
+            encoding="utf-8",
+        )
+    except Exception as e:
+        logger.warning(f"Failed to write cache file {path}: {e}")
+
+
+# ---------------------------------------------------------------------------
+# Clear
+# ---------------------------------------------------------------------------
+
+
+def clear_project_cache(
+    project: str | None = None,
+    cache_dir: str | None = None,
+) -> int:
+    """Delete cached JSON files for a project (or all projects).
+
+    Args:
+        project: ORACC project path.  ``None`` = clear everything.
+        cache_dir: Custom cache directory.
+
+    Returns:
+        Number of tablet JSON files deleted.
+    """
+    base = _resolve_cache_dir(cache_dir) / "tablets"
+    if not base.exists():
+        return 0
+
+    if project:
+        target = base / project.replace("/", "-")
+    else:
+        target = base
+
+    if not target.exists():
+        return 0
+
+    count = 0
+    for f in target.rglob("*.json"):
+        f.unlink()
+        count += 1
+
+    # Clean up empty directories (bottom-up)
+    for d in sorted(target.rglob("*"), reverse=True):
+        if d.is_dir() and not any(d.iterdir()):
+            d.rmdir()
+    if project and target.exists() and not any(target.iterdir()):
+        target.rmdir()
+
+    logger.info(f"Cleared {count} cached tablet(s)")
+    return count