agift-graph 0.1.0__tar.gz

agift_graph-0.1.0/LICENSE

@@ -0,0 +1,15 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

agift_graph-0.1.0/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: agift-graph
+Version: 0.1.0
+Summary: Australian Government Interactive Functions Thesaurus (AGIFT) as a Neo4j knowledge graph with embeddings and dual edge types
+Author: AGIFT Graph Team
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/DeepCivic/AGIFT-graph-builder
+Project-URL: Repository, https://github.com/DeepCivic/AGIFT-graph-builder
+Project-URL: Issues, https://github.com/DeepCivic/AGIFT-graph-builder/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: neo4j>=5.0.0
+Provides-Extra: local
+Requires-Dist: sentence-transformers>=2.2.0; extra == "local"
+Provides-Extra: isaacus
+Requires-Dist: isaacus>=0.1.0; extra == "isaacus"
+Provides-Extra: all
+Requires-Dist: sentence-transformers>=2.2.0; extra == "all"
+Requires-Dist: isaacus>=0.1.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Dynamic: license-file
+
+# AGIFT Graph
+
+Australian Government Interactive Functions Thesaurus (AGIFT) as a Neo4j knowledge graph with embeddings and dual edge types.
+
+## What it does
+
+Fetches the full AGIFT vocabulary from the [TemaTres API](https://vocabularyserver.com/agift/), builds a Neo4j graph with structural hierarchy edges, generates embeddings (free local or Isaacus API), then creates semantic similarity edges between related terms.
+
+```
+TemaTres API ──► Neo4j Graph ──► Embeddings ──► Semantic Edges
+  (AGIFT)        (PARENT_OF)     (384/512/768d)  (SIMILAR_TO)
+```
+
+## Graph model
+
+Two edge types with different weights for query-time flexibility:
+
+| Edge | Type | Weight | Description |
+|------|------|--------|-------------|
+| `PARENT_OF` | structural | 1.0 | AGIFT hierarchy (L1 → L2 → L3) |
+| `SIMILAR_TO` | semantic | 0.5 | Cosine similarity above threshold |
+
+Nodes carry DCAT-AP theme mappings for interoperability with European open data standards.
+
+## Quick start
+
+```bash
+docker compose -f docker-compose.agift.yml up -d --build
+```
+
+Then open the dashboard at http://localhost:5050 and click "Full Pipeline" or "Graph Only".
+
+## Embedding providers
+
+| Provider | Cost | Dimensions | Setup |
+|----------|------|-----------|-------|
+| local (sentence-transformers) | Free | 384, 768 | Nothing — runs on CPU |
+| isaacus (kanon-2-embedder) | Paid | 256–1792 | Set API key in dashboard |
+
+The local provider uses `all-MiniLM-L6-v2` (384d) or `all-mpnet-base-v2` (768d). Models are downloaded on first run and cached in a Docker volume.
+
+## Configuration
+
+Copy `.env.example` to `.env` and edit:
+
+```bash
+cp agift/.env.example .env
+```
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `NEO4J_PASSWORD` | `changeme` | Neo4j database password |
+| `ISAACUS_API_KEY` | (empty) | Isaacus API key (optional) |
+
+All other settings (dimension, provider, similarity threshold, semantic edge weight) are configured via the dashboard UI and stored in Neo4j.
+
+## Services
+
+| Service | Port | Description |
+|---------|------|-------------|
+| Neo4j Browser | 7474 | Graph database UI |
+| Neo4j Bolt | 7687 | Database protocol |
+| Dashboard | 5050 | Config, run controls, logs |
+
+## CLI usage
+
+```bash
+# Full pipeline (fetch + graph + embed + semantic edges)
+docker exec agift-worker python import_agift.py
+
+# Graph only (no embeddings)
+docker exec agift-worker python import_agift.py --skip-embed --skip-semantic
+
+# Local embeddings, 384 dimensions
+docker exec agift-worker python import_agift.py --provider local --dimension 384
+
+# Force re-embed all terms
+docker exec agift-worker python import_agift.py --force-embed
+
+# Dry run (fetch from API, no writes)
+docker exec agift-worker python import_agift.py --dry-run
+```
+
+## Docker Hub (no source code needed)
+
+```bash
+docker compose -f docker-compose.agift.hub.yml up -d
+```
+
+## Project structure
+
+```
+agift/
+├── import_agift.py          # 4-stage pipeline (fetch/graph/embed/link)
+├── dashboard/
+│   ├── Dockerfile
+│   ├── app.py               # Flask dashboard + run controls
+│   └── templates/
+│       └── index.html
+├── worker/
+│   ├── Dockerfile
+│   └── entrypoint.sh        # Cron scheduler + manual trigger
+├── .env.example
+├── LICENSE                   # Apache 2.0
+└── README.md
+```
+
+## Data source
+
+AGIFT is maintained by the National Archives of Australia and published via TemaTres at https://vocabularyserver.com/agift/
+
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE).

agift_graph-0.1.0/README.md

@@ -0,0 +1,114 @@
+# AGIFT Graph
+
+Australian Government Interactive Functions Thesaurus (AGIFT) as a Neo4j knowledge graph with embeddings and dual edge types.
+
+## What it does
+
+Fetches the full AGIFT vocabulary from the [TemaTres API](https://vocabularyserver.com/agift/), builds a Neo4j graph with structural hierarchy edges, generates embeddings (free local or Isaacus API), then creates semantic similarity edges between related terms.
+
+```
+TemaTres API ──► Neo4j Graph ──► Embeddings ──► Semantic Edges
+  (AGIFT)        (PARENT_OF)     (384/512/768d)  (SIMILAR_TO)
+```
+
+## Graph model
+
+Two edge types with different weights for query-time flexibility:
+
+| Edge | Type | Weight | Description |
+|------|------|--------|-------------|
+| `PARENT_OF` | structural | 1.0 | AGIFT hierarchy (L1 → L2 → L3) |
+| `SIMILAR_TO` | semantic | 0.5 | Cosine similarity above threshold |
+
+Nodes carry DCAT-AP theme mappings for interoperability with European open data standards.
+
+## Quick start
+
+```bash
+docker compose -f docker-compose.agift.yml up -d --build
+```
+
+Then open the dashboard at http://localhost:5050 and click "Full Pipeline" or "Graph Only".
+
+## Embedding providers
+
+| Provider | Cost | Dimensions | Setup |
+|----------|------|-----------|-------|
+| local (sentence-transformers) | Free | 384, 768 | Nothing — runs on CPU |
+| isaacus (kanon-2-embedder) | Paid | 256–1792 | Set API key in dashboard |
+
+The local provider uses `all-MiniLM-L6-v2` (384d) or `all-mpnet-base-v2` (768d). Models are downloaded on first run and cached in a Docker volume.
+
+## Configuration
+
+Copy `.env.example` to `.env` and edit:
+
+```bash
+cp agift/.env.example .env
+```
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `NEO4J_PASSWORD` | `changeme` | Neo4j database password |
+| `ISAACUS_API_KEY` | (empty) | Isaacus API key (optional) |
+
+All other settings (dimension, provider, similarity threshold, semantic edge weight) are configured via the dashboard UI and stored in Neo4j.
+
+## Services
+
+| Service | Port | Description |
+|---------|------|-------------|
+| Neo4j Browser | 7474 | Graph database UI |
+| Neo4j Bolt | 7687 | Database protocol |
+| Dashboard | 5050 | Config, run controls, logs |
+
+## CLI usage
+
+```bash
+# Full pipeline (fetch + graph + embed + semantic edges)
+docker exec agift-worker python import_agift.py
+
+# Graph only (no embeddings)
+docker exec agift-worker python import_agift.py --skip-embed --skip-semantic
+
+# Local embeddings, 384 dimensions
+docker exec agift-worker python import_agift.py --provider local --dimension 384
+
+# Force re-embed all terms
+docker exec agift-worker python import_agift.py --force-embed
+
+# Dry run (fetch from API, no writes)
+docker exec agift-worker python import_agift.py --dry-run
+```
+
+## Docker Hub (no source code needed)
+
+```bash
+docker compose -f docker-compose.agift.hub.yml up -d
+```
+
+## Project structure
+
+```
+agift/
+├── import_agift.py          # 4-stage pipeline (fetch/graph/embed/link)
+├── dashboard/
+│   ├── Dockerfile
+│   ├── app.py               # Flask dashboard + run controls
+│   └── templates/
+│       └── index.html
+├── worker/
+│   ├── Dockerfile
+│   └── entrypoint.sh        # Cron scheduler + manual trigger
+├── .env.example
+├── LICENSE                   # Apache 2.0
+└── README.md
+```
+
+## Data source
+
+AGIFT is maintained by the National Archives of Australia and published via TemaTres at https://vocabularyserver.com/agift/
+
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE).

agift_graph-0.1.0/agift/__init__.py

@@ -0,0 +1,47 @@
+"""AGIFT Graph Builder — modular ETL pipeline for Neo4j."""
+
+from agift.common import (
+    AGIFT_TOP_TO_DCAT,
+    LOCAL_MODELS,
+    PROVIDER_ISAACUS,
+    PROVIDER_LOCAL,
+    SEMANTIC_EDGE_WEIGHT,
+    SIMILARITY_THRESHOLD,
+    STRUCTURAL_EDGE_WEIGHT,
+    TEMATRES_BASE,
+    VALID_DIMENSIONS,
+    VALID_PROVIDERS,
+    get_config_from_neo4j,
+    get_neo4j_driver,
+    log_run,
+    print_summary,
+)
+from agift.fetch import AgiftTerm, fetch_full_hierarchy
+from agift.graph import ensure_schema, upsert_graph
+from agift.embed import build_hierarchical_text, embed_terms, embed_terms_local
+from agift.link import build_semantic_edges
+
+__all__ = [
+    "AGIFT_TOP_TO_DCAT",
+    "AgiftTerm",
+    "LOCAL_MODELS",
+    "PROVIDER_ISAACUS",
+    "PROVIDER_LOCAL",
+    "SEMANTIC_EDGE_WEIGHT",
+    "SIMILARITY_THRESHOLD",
+    "STRUCTURAL_EDGE_WEIGHT",
+    "TEMATRES_BASE",
+    "VALID_DIMENSIONS",
+    "VALID_PROVIDERS",
+    "build_hierarchical_text",
+    "build_semantic_edges",
+    "embed_terms",
+    "embed_terms_local",
+    "ensure_schema",
+    "fetch_full_hierarchy",
+    "get_config_from_neo4j",
+    "get_neo4j_driver",
+    "log_run",
+    "print_summary",
+    "upsert_graph",
+]

agift_graph-0.1.0/agift/__main__.py

@@ -0,0 +1,5 @@
+"""Allow running as: python -m agift"""
+
+from agift.cli import main
+
+main()

agift_graph-0.1.0/agift/cli.py

@@ -0,0 +1,200 @@
+"""
+CLI entry point for AGIFT import pipeline.
+
+Usage (via pip install):
+    agift
+    agift --dry-run
+    agift --skip-embed
+    agift --force-embed
+    agift --skip-semantic
+
+Usage (direct):
+    python -m agift
+    python import_agift.py
+
+Source: https://vocabularyserver.com/agift/services.php
+"""
+
+import argparse
+import os
+import sys
+import time
+from datetime import datetime, timezone
+
+from agift.common import (
+    PROVIDER_ISAACUS,
+    PROVIDER_LOCAL,
+    TEMATRES_BASE,
+    VALID_DIMENSIONS,
+    VALID_PROVIDERS,
+    get_config_from_neo4j,
+    get_neo4j_driver,
+    log_run,
+    print_summary,
+)
+from agift.fetch import fetch_full_hierarchy
+from agift.graph import ensure_schema, upsert_graph
+from agift.embed import embed_terms, embed_terms_local
+from agift.link import build_semantic_edges
+
+
+def main():
+    """CLI entry point for AGIFT import pipeline."""
+    parser = argparse.ArgumentParser(
+        description="Import AGIFT vocabulary into Neo4j graph with embeddings"
+    )
+    parser.add_argument("--dry-run", action="store_true",
+                        help="Fetch from API but don't write to Neo4j")
+    parser.add_argument("--skip-alt", action="store_true",
+                        help="Skip fetching alt labels (faster)")
+    parser.add_argument("--skip-embed", action="store_true",
+                        help="Skip embedding generation")
+    parser.add_argument("--force-embed", action="store_true",
+                        help="Re-embed all terms, not just new/changed")
+    parser.add_argument("--skip-semantic", action="store_true",
+                        help="Skip semantic edge generation")
+    parser.add_argument("--provider", choices=list(VALID_PROVIDERS),
+                        help="Override embedding provider (isaacus or local)")
+    parser.add_argument("--dimension", type=int, choices=list(VALID_DIMENSIONS),
+                        help="Override embedding dimension")
+    parser.add_argument("--threshold", type=float, default=None,
+                        help="Cosine similarity threshold for semantic edges")
+    args = parser.parse_args()
+
+    started_at = datetime.now(timezone.utc).isoformat()
+    run_details = {"started_at": started_at}
+
+    print("=" * 60)
+    print("AGIFT Vocabulary Import (Neo4j + Embeddings + Semantic Edges)")
+    print("=" * 60)
+    print(f"Source: {TEMATRES_BASE}")
+    print()
+
+    # Stage 1: Fetch from TemaTres
+    start = time.time()
+    terms = fetch_full_hierarchy(include_alts=not args.skip_alt)
+    elapsed = time.time() - start
+    print(f"\nFetched {len(terms)} terms in {elapsed:.1f}s")
+    run_details["fetched"] = len(terms)
+
+    total_alts = sum(len(t.alt_labels) for t in terms)
+    print(f"Total alt labels: {total_alts}")
+
+    if args.dry_run:
+        print("\n[DRY RUN] Skipping Neo4j write and embedding.")
+        for t in terms[:10]:
+            alts = f" (alts: {', '.join(t.alt_labels)})" if t.alt_labels else ""
+            print(f"  L{t.depth} [{t.dcat_theme}] {t.label}{alts}")
+        print(f"  ... and {len(terms) - 10} more")
+        return
+
+    # Connect to Neo4j
+    print("\nConnecting to Neo4j...")
+    try:
+        driver = get_neo4j_driver()
+        driver.verify_connectivity()
+    except Exception as e:
+        print(f"ERROR: Cannot connect to Neo4j: {e}")
+        sys.exit(1)
+
+    try:
+        # Stage 2: Build graph (structural edges)
+        print("\nStage 2: Building graph (structural edges)...")
+        ensure_schema(driver)
+        graph_stats = upsert_graph(driver, terms)
+        print(f"  Created: {graph_stats['created']}")
+        print(f"  Updated: {graph_stats['updated']}")
+        print(f"  Unchanged: {graph_stats['unchanged']}")
+        run_details.update({
+            "created": graph_stats["created"],
+            "updated": graph_stats["updated"],
+            "unchanged": graph_stats["unchanged"],
+        })
+
+        # Stage 3: Embed
+        if args.skip_embed:
+            print("\nStage 3: Skipped (--skip-embed)")
+            run_details["embedded"] = 0
+            run_details["embed_failed"] = 0
+            run_details["embedding_provider"] = ""
+        else:
+            config = get_config_from_neo4j(driver)
+            provider = args.provider or config["embedding_provider"]
+            dimension = args.dimension or config["embedding_dimension"]
+            run_details["embedding_provider"] = provider
+
+            # Determine which term IDs to embed
+            if args.force_embed:
+                with driver.session() as session:
+                    result = session.run(
+                        "MATCH (t:Term) RETURN t.term_id AS tid"
+                    )
+                    embed_ids = [r["tid"] for r in result]
+            else:
+                embed_ids = graph_stats["changed_ids"]
+
+            if not embed_ids:
+                print("\nStage 3: No terms need embedding")
+                run_details["embedded"] = 0
+                run_details["embed_failed"] = 0
+            elif provider == PROVIDER_LOCAL:
+                print(f"\nStage 3: Local embedding {len(embed_ids)} terms "
+                      f"(dimension={dimension})...")
+                embed_stats = embed_terms_local(driver, embed_ids, dimension)
+                print(f"  Embedded: {embed_stats['embedded']}")
+                print(f"  Failed:   {embed_stats['failed']}")
+                run_details["embedded"] = embed_stats["embedded"]
+                run_details["embed_failed"] = embed_stats["failed"]
+            else:
+                # Isaacus provider
+                api_key = config["isaacus_api_key"] or os.environ.get("ISAACUS_API_KEY")
+                if not api_key:
+                    print("\nStage 3: Skipped (no Isaacus API key configured)")
+                    print("  Set via dashboard, or use --provider local")
+                    run_details["embedded"] = 0
+                    run_details["embed_failed"] = 0
+                else:
+                    print(f"\nStage 3: Isaacus embedding {len(embed_ids)} terms "
+                          f"(dimension={dimension})...")
+                    embed_stats = embed_terms(driver, embed_ids, api_key, dimension)
+                    print(f"  Embedded: {embed_stats['embedded']}")
+                    print(f"  Failed:   {embed_stats['failed']}")
+                    run_details["embedded"] = embed_stats["embedded"]
+                    run_details["embed_failed"] = embed_stats["failed"]
+
+        # Stage 4: Semantic edges
+        if args.skip_semantic:
+            print("\nStage 4: Skipped (--skip-semantic)")
+            run_details["semantic_edges_created"] = 0
+        else:
+            config = get_config_from_neo4j(driver)
+            threshold = args.threshold or config["similarity_threshold"]
+            sem_weight = config["semantic_edge_weight"]
+
+            print(f"\nStage 4: Building semantic edges "
+                  f"(threshold={threshold}, weight={sem_weight})...")
+            sem_stats = build_semantic_edges(driver, threshold, sem_weight)
+            print(f"  Created:            {sem_stats['created']}")
+            print(f"  Skipped (structural): {sem_stats['skipped_structural']}")
+            print(f"  Below threshold:    {sem_stats['below_threshold']}")
+            run_details["semantic_edges_created"] = sem_stats["created"]
+
+        print_summary(driver)
+        log_run(driver, "success", run_details)
+
+    except Exception as e:
+        print(f"\nERROR: {e}")
+        run_details["error"] = str(e)
+        try:
+            log_run(driver, "error", run_details)
+        except Exception:
+            pass
+        sys.exit(1)
+    finally:
+        driver.close()
+
+    print("\nDone.")
+
+
+if __name__ == "__main__":
+    main()