hoard-erd 0.2.1__tar.gz

hoard_erd-0.2.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Marcus Quinn
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hoard_erd-0.2.1/PKG-INFO

@@ -0,0 +1,34 @@
+Metadata-Version: 2.4
+Name: hoard-erd
+Version: 0.2.1
+Summary: Heritage Observation And Report Drafter — local multi-stage AI pipeline for archaeological grey literature reports
+Author: Marcus Quinn
+License: MIT
+Requires-Python: >=3.11
+License-File: LICENSE
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13
+Requires-Dist: pillow>=10
+Requires-Dist: opencv-python-headless>=4.9
+Requires-Dist: wand>=0.6
+Requires-Dist: pandas>=2
+Requires-Dist: openpyxl>=3
+Requires-Dist: pyyaml>=6
+Requires-Dist: heritage-models>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov>=5; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1; extra == "dev"
+Provides-Extra: ark
+Requires-Dist: sentence-transformers>=3; extra == "ark"
+Requires-Dist: numpy>=1.24; extra == "ark"
+Provides-Extra: ocr
+Requires-Dist: transformers>=4.40; extra == "ocr"
+Requires-Dist: torch>=2; extra == "ocr"
+Requires-Dist: accelerate>=0.30; extra == "ocr"
+Provides-Extra: llm
+Requires-Dist: llama-cpp-python>=0.3; extra == "llm"
+Provides-Extra: doc
+Requires-Dist: python-docx>=1; extra == "doc"
+Dynamic: license-file

hoard_erd-0.2.1/README.md

@@ -0,0 +1,137 @@
+# HOARD
+
+**Heritage Observation And Report Drafter**
+
+A fully local, multi-stage AI pipeline that converts archaeological field data — context sheets, finds catalogues, site photographs, section drawings, and sample results — into a near-publication-ready grey literature report conforming to the relevant heritage authority standard.
+
+Targets 8 GB VRAM consumer GPUs. Runs entirely on-device via Ollama — zero API calls, zero data leaves your machine.
+
+---
+
+## Features
+
+### End-to-End Report Generation
+Converts raw field records (context sheets, finds catalogues, photographs, section drawings, sample results) into a complete grey literature report in six automated phases — from file triage through to publication-ready DOCX, PDF/A-2b, TEI-XML, and ZIP export. All six phases are implemented and E2E-verified on real archaeological data.
+
+### Multi-Provider AI
+Switch between four AI backends per pipeline phase — **Ollama** (local GPU), **OpenAI**, **Anthropic Claude**, and **Google Gemini** — with intelligent routing based on task requirements, privacy constraints, and hardware availability. Configure once; HOARD selects the optimal provider automatically.
+
+### Hardware Tier System
+Auto-detects your GPU, VRAM, and Ollama models on first run and suggests an appropriate tier:
+- **Ultra-light** — no GPU needed, cloud-only inference
+- **Budget** — 6 GB VRAM, compact local models
+- **Standard** — 8-12 GB VRAM, full local pipeline
+- **Performance** — 16-24 GB VRAM, high-end local models
+
+### 14 Jurisdiction Templates
+Reports conform to heritage authority standards in England, Scotland, Wales, Ireland, Netherlands, France, Germany, US, Canada, Australia, New Zealand, and South Africa — all driven by declarative YAML templates. Adding a new jurisdiction means writing one YAML file; no code changes required.
+
+### Interactive Review Dashboard
+After each pipeline phase, a terminal TUI presents flagged items (blurred images, low-confidence OCR, spatial mismatches, compliance warnings) for Accept/Edit/Defer review. Corrections write back to the workspace and update pipeline state for re-runnable workflows.
+
+### Offline Getty Vocabulary
+Standardises materials, periods, and artefact types against Getty AAT/ULAN/TGN terms using the `heritage-vocab` library — works offline with a built-in fallback covering common archaeological terms. No API calls required.
+
+### Harris Matrix Generator
+Pure-Python SVG stratigraphic matrix from context relationships. Colour-coded by period, arrows from later to earlier contexts. No graphviz or external tools needed.
+
+### Cryptographically Signed PDFs
+Optional PAdES-B/LTV digital signatures via pyHanko for legally compliant report certification.
+
+### Cloud-Ready Credential Vault
+API keys for OpenAI, Anthropic, and Google are stored encrypted at rest (AES-256-GCM + PBKDF2) and managed via `hoard keys set/list/remove`. Cross-compatible with the Kryptis vault format.
+
+### Ecosystem Integration
+HOARD shares data contracts and workflows with [StratiGraph](https://github.com/mabo-du/stratigraph) (Harris Matrix viewer), Trowel (desktop report drafter), Libby (radiocarbon calibration), Cache & Carry (offline collections management), and Dibble (3D lithic analysis) — all accessible through the unified `heritage` CLI.
+
+## Quick Start
+
+```bash
+# Install
+pip install hoard            # from PyPI
+# or from source
+git clone https://github.com/mabo-du/HOARD.git
+cd HOARD && pip install -e ".[dev]"
+
+# Install Ollama and pull models
+ollama pull glm-ocr qwen3-vl:8b qwen3.5-4b gemma4
+
+# Initialise a project
+hoard init "Stoneyfield Farm 2026" --jurisdiction historic_england_cl3
+
+# Run Phase 0 (no GPU needed)
+hoard run --project stoneyfield_farm_2026 --input ./field_records --phase 0
+
+# List available jurisdiction templates
+hoard templates list
+```
+
+## CLI Reference
+
+| Command | Description |
+|---------|-------------|
+| `hoard init <name>` | Initialise a new project |
+| `hoard run --project <id>` | Run the pipeline (full or partial) |
+| `hoard run --project <id> --phase <N>` | Run a single phase |
+| `hoard run --project <id> --from-phase <N>` | Run from phase N onward |
+| `hoard run --project <id> --strict` | Halt Phase 1 on schema validation failure |
+| `hoard run --project <id> --extractor nuextract3` | Use NuExtract3 for Phase 1 extraction (opt-in) |
+| `hoard import-ark --project <id> --input <dir>` | Import structured data from ARK system exports |
+| `hoard review --project <id>` | Interactive review dashboard for flagged items |
+| `hoard export --project <id> --format docx,pdf` | Export final report |
+| `hoard templates list` | List available jurisdiction templates |
+| `hoard templates show --name <code>` | Show template details with syntax highlighting |
+| `hoard templates validate --file <path>` | Validate a template YAML file |
+| `hoard keys set <provider> <key>` | Store an encrypted API key for cloud providers |
+| `hoard keys list` | List configured API keys |
+| `hoard keys unlock` | Unlock the credential vault |
+
+## Ecosystem Integration
+
+HOARD is one component of a broader heritage science open-source ecosystem:
+
+| Tool | Function | Integration |
+|------|----------|-------------|
+| **StratiGraph** | Interactive Harris Matrix editor (Tauri 2 + React) | [Shared JSON Schema](schemas/heritage-data-package-v1.json) — HOARD Phase 1 exports import directly |
+| **Trowel** | Desktop report drafter (PyQt6) | Bidirectional JSON import/export, shared jurisdiction templates |
+| **Libby** | Radiocarbon calibration (FastAPI + Svelte 5) | StratiGraph exports OxCal CQL / JSON payloads to Libby |
+| **Cache & Carry** | Offline collections management (Tauri + Rust) | Getty AAT/ULAN/TGN vocabulary for term normalisation |
+| **Dibble** | 3D lithic analysis (Python + PyVista) | Specialist finds appendix data via JSON bridge |
+| **heritage-cli** | Unified ecosystem CLI | `heritage run/calibrate/lithics/review/matrix/publish` |
+
+## Jurisdiction Templates
+
+Reports conform to national heritage authority standards via declarative YAML templates. Currently 14 jurisdictions:
+
+| Code | Authority | Region |
+|------|-----------|--------|
+| `historic_england_cl3` | Historic England — Evaluation (CL3) | England |
+| `historic_england_cl4` | Historic England — Excavation (CL4) | England |
+| `historic_environment_scotland` | HES — Data Structure Report | Scotland |
+| `wales_rcahmw` | Cadw / RCAHMW | Wales |
+| `ireland_nms` | National Monuments Service | Ireland |
+| `netherlands_kna` | KNA 5.0 | Netherlands |
+| `france_inrap` | INRAP / Code du Patrimoine | France |
+| `germany_denkmalpflege` | Landesdenkmalpflege | Germany |
+| `us_section106` | Section 106 (NRHP) | United States |
+| `canada_ontario` | Ontario S&G | Canada |
+| `australia_burra` | Burra Charter / ICOMOS | Australia |
+| `new_zealand` | Heritage NZ Pouhere Taonga | New Zealand |
+| `south_africa_sahra` | SAHRA | South Africa |
+| `international_generic` | Generic fallback | Any |
+
+Adding a new jurisdiction means writing a single YAML file — no pipeline code changes required. Templates support `extends` inheritance for regional variations (e.g. US state-level overrides).
+
+## Documentation
+
+- **[Full User Guide](docs/user-guide.md)** — installation, phase walkthroughs, ARK import, review dashboard, GPU setup, troubleshooting
+- **`hoard --help`** — inline CLI reference
+- **Research papers** — see `docs/research-papers/` for architectural deep-dives on multi-provider AI, ecosystem integration, schema unification, and model selection
+
+## Licence
+
+MIT
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and pull request workflow.

hoard_erd-0.2.1/pyproject.toml

@@ -0,0 +1,68 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hoard-erd"
+version = "0.2.1"
+description = "Heritage Observation And Report Drafter — local multi-stage AI pipeline for archaeological grey literature reports"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+authors = [
+    { name = "Marcus Quinn" },
+]
+
+dependencies = [
+    # CLI
+    "typer>=0.12",
+    "rich>=13",
+
+    # Image processing
+    "pillow>=10",
+    "opencv-python-headless>=4.9",
+    "wand>=0.6",
+
+    # Data
+    "pandas>=2",
+    "openpyxl>=3",
+    "pyyaml>=6",
+
+    # Shared data models (auto-generated from heritage-types TypeSpec)
+    "heritage-models>=1.0",
+
+    # OCR (loaded on demand)
+    # "transformers>=4.40",     # TrOCR
+    # "torch>=2",               # PyTorch
+
+    # Reporting
+    # pandoc is a system-level dependency, not pip
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8",
+    "pytest-cov>=5",
+    "ruff>=0.4",
+    "mypy>=1",
+]
+ark = [
+    "sentence-transformers>=3",
+    "numpy>=1.24",
+]
+ocr = [
+    "transformers>=4.40",
+    "torch>=2",
+    "accelerate>=0.30",
+]
+llm = [
+    "llama-cpp-python>=0.3",
+]
+doc = [
+    "python-docx>=1",
+]
+
+[project.scripts]
+hoard = "hoard.cli.main:app"
+
+[tool.setuptools.packages.find]
+where = ["src"]

hoard_erd-0.2.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

hoard_erd-0.2.1/src/hoard/__init__.py

@@ -0,0 +1,13 @@
+"""hoard — Heritage Observation And Report Drafter.
+
+Multi-stage AI pipeline converting archaeological field data into
+near-publication-ready grey literature reports. Fully local, targets
+6 GB VRAM, jurisdiction-templated.
+
+exports: (package) — use hoard.cli.main:app as entry point
+used_by: pyproject.toml → `hoard` CLI command
+rules:   No model inference logic in this file; orchestration only.
+agent:   deepseek-v4-flash | 2026-05-09 | s_20260509_001 | Initial scaffold
+"""
+
+__version__ = "0.1.0"

hoard_erd-0.2.1/src/hoard/__main__.py

@@ -0,0 +1,11 @@
+"""__main__.py — Allow `python -m hoard` as alternative to `hoard` CLI.
+
+usage: python -m hoard --help
+used_by: developer workflow
+rules:   Must delegate to hoard.cli.main:entry_point
+agent:   deepseek-v4-flash | 2026-05-09 | s_20260509_001 | Initial scaffold
+"""
+
+from hoard.cli.main import entry_point
+
+entry_point()

hoard_erd-0.2.1/src/hoard/ark/__init__.py

@@ -0,0 +1,24 @@
+"""hoard.ark — ARK system direct data input (bypasses Phase 1 OCR).
+
+For digital-first excavations using the ARK (Archaeological Recording Kit)
+system, structured data can be imported directly — bypassing Phase 0 file
+ingestion and Phase 1 multi-modal digitisation.
+
+exports: import_ark_export, ArkImportResult
+used_by: hoard.cli.main  → `hoard import-ark` command
+rules:   Must never import torch or any GPU-bound library. Exported data
+         must be compatible with Phase 5+ pipeline stages.
+"""
+
+from hoard.ark.loader import ArkImportResult, import_ark_export
+from hoard.ark.mapping import guess_mapping_from_header, transform_row
+from hoard.ark.semantic_mapper import ArkSemanticMapper, map_headers_semantic
+
+__all__ = [
+    "ArkImportResult",
+    "ArkSemanticMapper",
+    "import_ark_export",
+    "guess_mapping_from_header",
+    "map_headers_semantic",
+    "transform_row",
+]

hoard_erd-0.2.1/src/hoard/ark/loader.py

@@ -0,0 +1,331 @@
+"""loader.py — ARK export data importer.
+
+Discovers ARK system export files (CSV/JSON), maps fields to HOARD's
+internal representation, and writes structured data directly into the
+workspace — bypassing Phase 0 file ingestion and Phase 1 OCR for
+digital-first excavations.
+
+exports: import_ark_export, ArkImportResult
+used_by: hoard.cli.main  → `hoard import-ark` command
+rules:   Must never import torch or any GPU-bound library.
+         Generated manifests must be compatible with Phase 5+ pipeline stages.
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from hoard.ark.mapping import (
+    ARK_CONTEXT_FIELDS,
+    ARK_DRAWINGS_FIELDS,
+    ARK_FINDS_FIELDS,
+    ARK_PHOTOS_FIELDS,
+    ARK_SAMPLES_FIELDS,
+    SOURCE_TYPE_MAP,
+    guess_mapping_from_header,
+    transform_row,
+)
+from hoard.config import Config
+from hoard.workspace import PipelineState
+
+# ── Data contracts ─────────────────────────────────────────────────────────
+
+IMPORT_TABLES: dict[str, dict[str, Any]] = {
+    "context": {
+        "filename_patterns": ("context", "contexts", "ctx_register"),
+        "mapping": ARK_CONTEXT_FIELDS,
+        "output_key": "context_sheets",
+    },
+    "finds": {
+        "filename_patterns": ("finds", "small_finds", "finds_catalogue", "sf_register"),
+        "mapping": ARK_FINDS_FIELDS,
+        "output_key": "finds",
+    },
+    "samples": {
+        "filename_patterns": ("samples", "sample", "environmental"),
+        "mapping": ARK_SAMPLES_FIELDS,
+        "output_key": "samples",
+    },
+    "photos": {
+        "filename_patterns": ("photos", "photo", "images", "photo_log"),
+        "mapping": ARK_PHOTOS_FIELDS,
+        "output_key": "photos",
+    },
+    "drawings": {
+        "filename_patterns": ("drawings", "drawing", "plans", "section_drawings"),
+        "mapping": ARK_DRAWINGS_FIELDS,
+        "output_key": "drawings",
+    },
+}
+
+
+class ArkImportResult:
+    """Result of an ARK import operation."""
+
+    def __init__(self) -> None:
+        self.files_found: int = 0
+        self.files_parsed: int = 0
+        self.total_records: int = 0
+        self.errors: list[str] = []
+        self.warnings: list[str] = []
+        self.records_by_type: dict[str, int] = {}
+        self.manifest_path: Path | None = None
+        self.digitised_data: list[dict[str, Any]] = []
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "files_found": self.files_found,
+            "files_parsed": self.files_parsed,
+            "total_records": self.total_records,
+            "records_by_type": self.records_by_type,
+            "errors": self.errors,
+            "warnings": self.warnings,
+            "manifest_path": str(self.manifest_path) if self.manifest_path else None,
+        }
+
+
+# ── File discovery ─────────────────────────────────────────────────────────
+
+
+def _discover_ark_files(input_dir: Path) -> list[tuple[Path, str]]:
+    """Find ARK export files in input_dir.
+
+    Returns list of (file_path, source_type) tuples.
+    """
+    found: list[tuple[Path, str]] = []
+    if not input_dir.is_dir():
+        return found
+
+    for f in sorted(input_dir.iterdir()):
+        if not f.is_file() or f.name.startswith("."):
+            continue
+        stem = f.stem.lower()
+
+        for table_name, table_config in IMPORT_TABLES.items():
+            if stem in table_config["filename_patterns"]:
+                found.append((f, table_name))
+                break
+
+    return found
+
+
+# ── CSV parsing ─────────────────────────────────────────────────────────────
+
+
+def _parse_csv(filepath: Path) -> list[dict[str, Any]]:
+    """Parse a CSV file into a list of row dicts."""
+    rows: list[dict[str, Any]] = []
+    try:
+        with open(filepath, newline="", encoding="utf-8-sig") as f:
+            reader = csv.DictReader(f)
+            for row in reader:
+                cleaned = {k.strip(): v.strip() if v else "" for k, v in row.items()}
+                rows.append(cleaned)
+    except Exception as e:
+        raise ValueError(f"Failed to parse CSV {filepath}: {e}") from e
+    return rows
+
+
+# ── JSON parsing ────────────────────────────────────────────────────────────
+
+
+def _parse_json(filepath: Path) -> list[dict[str, Any]]:
+    """Parse a JSON file into a list of row dicts.
+
+    Handles both top-level arrays and {'data': [...]} wrappers.
+    """
+    raw = json.loads(filepath.read_text())
+    if isinstance(raw, list):
+        return raw
+    if isinstance(raw, dict):
+        # Common ARK JSON export wrappers
+        for key in ("data", "results", "records", "rows", "items"):
+            if key in raw and isinstance(raw[key], list):
+                return raw[key]
+    raise ValueError(f"Unrecognised JSON structure in {filepath}")
+
+
+# ── Per-type processing ─────────────────────────────────────────────────────
+
+
+def _process_table(
+    filepath: Path,
+    source_type: str,
+    mapping_table: list[tuple[str, str, str | None]],
+) -> tuple[list[dict[str, Any]], list[str]]:
+    """Parse and transform one ARK export file.
+
+    Returns (transformed_records, warnings).
+    """
+    warnings: list[str] = []
+
+    # Parse
+    if filepath.suffix.lower() == ".json":
+        raw_rows = _parse_json(filepath)
+    else:
+        raw_rows = _parse_csv(filepath)
+
+    if not raw_rows:
+        return [], [f"Empty file: {filepath.name}"]
+
+    # Infer mapping from first row's headers
+    header = list(raw_rows[0].keys())
+    field_map = guess_mapping_from_header(header, mapping_table)
+
+    if not field_map:
+        return [], [f"No recognised ARK fields in {filepath.name}"]
+
+    unrecognised = len(header) - len(field_map)
+    if unrecognised > 0:
+        warnings.append(f"{filepath.name}: {unrecognised} unrecognised column(s) ignored")
+
+    # Transform
+    records = [transform_row(row, field_map, source_type) for row in raw_rows]
+
+    return records, warnings
+
+
+# ── Data-source manifest generation ─────────────────────────────────────────
+
+
+def _build_manifest(
+    project_id: str,
+    records_by_type: dict[str, int],
+    errors: list[str],
+    warnings: list[str],
+    digitised_dir: Path,
+) -> dict[str, Any]:
+    """Build a Phase-0-compatible manifest for ARK-imported data."""
+    manifest: dict[str, Any] = {
+        "project_id": project_id,
+        "created": datetime.now(timezone.utc).isoformat(),
+        "import_method": "ark",
+        "ark_direct_input": True,
+        "phase0_bypassed": True,
+        "phase1_bypassed": True,
+        "total_records": sum(records_by_type.values()),
+        "records_by_type": records_by_type,
+        "files": [],
+        "mandatory_check": "PASS",
+        "missing_mandatory": [],
+        "quality_warnings": 0,
+        "finds_validation_issues": [],
+        "halt": False,
+        "import_errors": errors,
+        "import_warnings": warnings,
+        "digitised_dir": str(digitised_dir),
+    }
+
+    # Build synthetic file entries for the manifest (Phase 5 needs these)
+    for source_type, count in records_by_type.items():
+        hoard_type = SOURCE_TYPE_MAP.get(source_type, "unknown")
+        manifest["files"].append({
+            "id": f"ark_{source_type}",
+            "path": f"ark_import/{source_type}.csv",
+            "type": hoard_type,
+            "quality": {},
+            "ark_record_count": count,
+        })
+
+    return manifest
+
+
+# ── Digitised data ──────────────────────────────────────────────────────────
+
+
+def _write_digitised_data(
+    records: list[dict[str, Any]],
+    digitised_dir: Path,
+    prefix: str,
+) -> None:
+    """Write transformed records as digitised-phase-style JSON files.
+
+    Each record becomes its own JSON file (compatible with Phase 5's
+    expectation of per-record JSON in 01_digitised/).
+    """
+    digitised_dir.mkdir(parents=True, exist_ok=True)
+    for i, record in enumerate(records):
+        record_path = digitised_dir / f"{prefix}_{i:04d}.json"
+        record_path.write_text(json.dumps(record, indent=2))
+
+
+# ── Main import ─────────────────────────────────────────────────────────────
+
+
+def import_ark_export(config: Config) -> ArkImportResult:
+    """Import ARK system export data for a project.
+
+    Discovers ARK export files in config.input_dir, maps fields to HOARD's
+    internal representation, and writes structured data into the workspace.
+
+    Returns an ArkImportResult describing what was imported.
+    """
+    result = ArkImportResult()
+    project_dir = config.project_dir
+    manifest_dir = config.manifest_dir
+    digitised_dir = config.digitised_dir
+
+    project_dir.mkdir(parents=True, exist_ok=True)
+    manifest_dir.mkdir(parents=True, exist_ok=True)
+
+    # Discover ARK export files
+    ark_files = _discover_ark_files(config.input_dir)
+    result.files_found = len(ark_files)
+
+    if not ark_files:
+        result.errors.append("No ARK export files found")
+        return result
+
+    all_records: list[dict[str, Any]] = []
+    records_by_type: dict[str, int] = {}
+
+    for filepath, source_type in ark_files:
+        mapping_table = IMPORT_TABLES[source_type]["mapping"]
+        try:
+            records, warnings = _process_table(filepath, source_type, mapping_table)
+        except ValueError as e:
+            result.errors.append(str(e))
+            continue
+
+        result.warnings.extend(warnings)
+        result.files_parsed += 1
+
+        if records:
+            prefix = source_type
+            _write_digitised_data(records, digitised_dir, prefix)
+            all_records.extend(records)
+            records_by_type[source_type] = len(records)
+            result.total_records += len(records)
+
+    result.records_by_type = records_by_type
+    result.digitised_data = all_records
+
+    # Generate manifest
+    manifest = _build_manifest(
+        config.project_id,
+        records_by_type,
+        result.errors,
+        result.warnings,
+        digitised_dir,
+    )
+
+    manifest_path = manifest_dir / "manifest.json"
+    manifest_path.write_text(json.dumps(manifest, indent=2))
+    result.manifest_path = manifest_path
+
+    # Update pipeline state: mark Phase 0 and Phase 1 as bypassed
+    state = PipelineState(project_dir / "pipeline_state.json")
+    state.complete_phase(
+        0,
+        summary=f"Bypassed via ARK import — {result.total_records} records from {result.files_parsed} files",
+    )
+    state.complete_phase(
+        1,
+        summary="Bypassed via ARK import — digital-first excavation data already structured",
+    )
+
+    return result