graflo 1.3.7__tar.gz

graflo-1.3.7/.gitignore

@@ -0,0 +1,79 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+*.pyc
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.idea
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+#Ipython Notebook
+.ipynb_checkpoints
+
+#Latex
+*.aux
+*.fdb_latexmk
+*.fls
+*.log
+*.synctex.gz
+
+# cache
+.pytest_cache
+
+.mypy_cache
+
+*.log.*
+#*/**/*png
+*/**/*pdf
+
+site/

graflo-1.3.7/LICENSE

@@ -0,0 +1,126 @@
+Business Source License 1.1
+
+Parameters
+
+Licensor:             Growgraph SAS
+Licensed Work:        GraFlo
+                      The Licensed Work is (c) 2025 GraFlo
+                      Formerly GraphCast
+
+Additional Use Grant:
+    The Licensor hereby grants you the right to make production use of the Licensed Work
+    if any of the following conditions are met:
+
+    - Your organization's annual revenue does not exceed USD 100,000
+    - You are not offering a commercially available product or service that
+        competes with the Licensor's products or services
+
+Definitions:
+    "Production Use" means any use of the Licensed Work in a production environment,
+        including but not limited to:
+
+    - Deploying the Licensed Work to serve production traffic
+    - Using the Licensed Work to process real customer data
+    - Incorporating the Licensed Work into a commercial product or service
+    - Using the Licensed Work to generate revenue directly or indirectly
+
+"Competing Product or Service" means any product or service that:
+
+    - Provides substantially similar functionality to the Licensed Work
+    - Offers the Licensed Work as a hosted or managed service
+    - Incorporates the Licensed Work into a broader product or service that competes
+        with the Licensor's current or reasonably anticipated products or services
+
+"Non-Production Use" means:
+
+    - Development and testing in isolated environments
+    - Academic research and educational purposes
+    - Evaluation of the Licensed Work for potential purchase of a commercial license
+    - Personal, non-commercial projects where the Licensed Work is not used to generate revenue
+
+Change Date:          2029-05-01
+
+Change License:       Apache License, Version 2.0
+
+For information about alternative licensing arrangements for the Software,
+please contact: <team@growgraph.dev>
+
+Notice
+
+The Business Source License (this document, or the “License”) is not an Open
+Source license. However, the Licensed Work will eventually be made available
+under an Open Source License, as stated in this License.
+
+License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
+“Business Source License” is a trademark of MariaDB Corporation Ab.
+
+-----------------------------------------------------------------------------
+
+Business Source License 1.1
+
+Terms
+
+The Licensor hereby grants you the right to copy, modify, create derivative
+works, redistribute, and make non-production use of the Licensed Work. The
+Licensor may make an Additional Use Grant, above, permitting limited
+production use.
+
+Effective on the Change Date, or the fourth anniversary of the first publicly
+available distribution of a specific version of the Licensed Work under this
+License, whichever comes first, the Licensor hereby grants you rights under
+the terms of the Change License, and the rights granted in the paragraph
+above terminate.
+
+If your use of the Licensed Work does not comply with the requirements
+currently in effect as described in this License, you must purchase a
+commercial license from the Licensor, its affiliated entities, or authorized
+resellers, or you must refrain from using the Licensed Work.
+
+All copies of the original and modified Licensed Work, and derivative works
+of the Licensed Work, are subject to this License. This License applies
+separately for each version of the Licensed Work and the Change Date may vary
+for each version of the Licensed Work released by Licensor.
+
+You must conspicuously display this License on each original or modified copy
+of the Licensed Work. If you receive the Licensed Work in original or
+modified form from a third party, the terms and conditions set forth in this
+License apply to your use of that work.
+
+Any use of the Licensed Work in violation of this License will automatically
+terminate your rights under this License for the current and all other
+versions of the Licensed Work.
+
+This License does not grant you any right in any trademark or logo of
+Licensor or its affiliates (provided that you may use a trademark or logo of
+Licensor as expressly required by this License).
+
+TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
+AN “AS IS” BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
+EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
+TITLE.
+
+MariaDB hereby grants you permission to use this License’s text to license
+your works, and to refer to it using the trademark “Business Source License”,
+as long as you comply with the Covenants of Licensor below.
+
+Covenants of Licensor
+
+In consideration of the right to use this License’s text and the “Business
+Source License” name and trademark, Licensor covenants to MariaDB, and to all
+other recipients of the licensed work to be provided by Licensor:
+
+1. To specify as the Change License the GPL Version 2.0 or any later version,
+   or a license that is compatible with GPL Version 2.0 or a later version,
+   where “compatible” means that software provided under the Change License can
+   be included in a program with software provided under GPL Version 2.0 or a
+   later version. Licensor may specify additional Change Licenses without
+   limitation.
+
+2. To either: (a) specify an additional grant of rights to use that does not
+   impose any additional restriction on the right granted in this License, as
+   the Additional Use Grant; or (b) insert the text “None”.
+
+3. To specify a Change Date.
+
+4. Not to modify this License in any other way.

graflo-1.3.7/PKG-INFO

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: graflo
+Version: 1.3.7
+Summary: A framework for transforming tabular (CSV, SQL) and hierarchical data (JSON, XML) into property graphs and ingesting them into graph databases (ArangoDB, Neo4j, TigerGraph). Features automatic PostgreSQL schema inference.
+Author-email: Alexander Belikov <alexander@growgraph.dev>
+License-File: LICENSE
+Requires-Python: ~=3.10.0
+Requires-Dist: click<9,>=8.2.0
+Requires-Dist: dataclass-wizard>=0.34.0
+Requires-Dist: ijson<4,>=3.2.3
+Requires-Dist: neo4j<6,>=5.22.0
+Requires-Dist: networkx~=3.3
+Requires-Dist: pandas-stubs==2.3.0.250703
+Requires-Dist: pandas<3,>=2.0.3
+Requires-Dist: psycopg2-binary>=2.9.11
+Requires-Dist: pydantic-settings>=2.12.0
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: python-arango<9,>=8.1.2
+Requires-Dist: pytigergraph>=1.9.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: sqlalchemy>=2.0.0
+Requires-Dist: strenum>=0.4.15
+Requires-Dist: suthing>=0.5.0
+Requires-Dist: urllib3>=2.0.0
+Requires-Dist: xmltodict<0.15,>=0.14.2
+Provides-Extra: plot
+Requires-Dist: pygraphviz>=1.14; extra == 'plot'
+Description-Content-Type: text/markdown
+
+# GraFlo <img src="https://raw.githubusercontent.com/growgraph/graflo/main/docs/assets/favicon.ico" alt="graflo logo" style="height: 32px; width:32px;"/>
+
+A framework for transforming **tabular** (CSV, SQL) and **hierarchical** data (JSON, XML) into property graphs and ingesting them into graph databases (ArangoDB, Neo4j, **TigerGraph**).
+
+> **⚠️ Package Renamed**: This package was formerly known as `graphcast`.
+
+![Python](https://img.shields.io/badge/python-3.10-blue.svg) 
+[![PyPI version](https://badge.fury.io/py/graflo.svg)](https://badge.fury.io/py/graflo)
+[![PyPI Downloads](https://static.pepy.tech/badge/graflo)](https://pepy.tech/projects/graflo)
+[![License: BSL](https://img.shields.io/badge/license-BSL--1.1-green)](https://github.com/growgraph/graflo/blob/main/LICENSE)
+[![pre-commit](https://github.com/growgraph/graflo/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/growgraph/graflo/actions/workflows/pre-commit.yml)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.15446131.svg)]( https://doi.org/10.5281/zenodo.15446131)
+
+## Core Concepts
+
+### Property Graphs
+graflo works with property graphs, which consist of:
+
+- **Vertices**: Nodes with properties and optional unique identifiers
+- **Edges**: Relationships between vertices with their own properties
+- **Properties**: Both vertices and edges may have properties
+
+### Schema
+The Schema defines how your data should be transformed into a graph and contains:
+
+- **Vertex Definitions**: Specify vertex types, their properties, and unique identifiers
+  - Fields can be specified as strings (backward compatible) or typed `Field` objects with types (INT, FLOAT, STRING, DATETIME, BOOL)
+  - Type information enables better validation and database-specific optimizations
+- **Edge Definitions**: Define relationships between vertices and their properties
+  - Weight fields support typed definitions for better type safety
+- **Resource Mapping**: describe how data sources map to vertices and edges
+- **Transforms**: Modify data during the casting process
+- **Automatic Schema Inference**: Generate schemas automatically from PostgreSQL 3NF databases
+
+### Resources
+Resources are your data sources that can be:
+
+- **Table-like**: CSV files, database tables
+- **JSON-like**: JSON files, nested data structures
+
+## Features
+
+- **Graph Transformation Meta-language**: A powerful declarative language to describe how your data becomes a property graph:
+    - Define vertex and edge structures with typed fields
+    - Set compound indexes for vertices and edges
+    - Use blank vertices for complex relationships
+    - Specify edge constraints and properties with typed weight fields
+    - Apply advanced filtering and transformations
+- **Typed Schema Definitions**: Enhanced type support throughout the schema system
+    - Vertex fields support types (INT, FLOAT, STRING, DATETIME, BOOL) for better validation
+    - Edge weight fields can specify types for improved type safety
+    - Backward compatible: fields without types default to None (suitable for databases like ArangoDB)
+- **🚀 PostgreSQL Schema Inference**: **Automatically generate schemas from PostgreSQL 3NF databases** - No manual schema definition needed!
+    - Introspect PostgreSQL schemas to identify vertex-like and edge-like tables
+    - Automatically map PostgreSQL data types to graflo Field types (INT, FLOAT, STRING, DATETIME, BOOL)
+    - Infer vertex configurations from table structures with proper indexes
+    - Infer edge configurations from foreign key relationships
+    - Create Resource mappings from PostgreSQL tables automatically
+    - Direct database access - ingest data without exporting to files first
+- **Parallel processing**: Use as many cores as you have
+- **Database support**: Ingest into ArangoDB, Neo4j, and **TigerGraph** using the same API (database agnostic). Source data from PostgreSQL and other SQL databases.
+- **Server-side filtering**: Efficient querying with server-side filtering support (TigerGraph REST++ API)
+
+## Documentation
+Full documentation is available at: [growgraph.github.io/graflo](https://growgraph.github.io/graflo)
+
+## Installation
+
+```bash
+pip install graflo
+```
+
+## Usage Examples
+
+### Simple ingest
+
+```python
+from suthing import FileHandle
+
+from graflo import Schema, Caster, Patterns
+from graflo.db.connection.onto import ArangoConfig
+
+schema = Schema.from_dict(FileHandle.load("schema.yaml"))
+
+# Option 1: Load config from docker/arango/.env (recommended)
+conn_conf = ArangoConfig.from_docker_env()
+
+# Option 2: Load from environment variables
+# Set: ARANGO_URI, ARANGO_USERNAME, ARANGO_PASSWORD, ARANGO_DATABASE
+conn_conf = ArangoConfig.from_env()
+
+# Option 3: Load with custom prefix (for multiple configs)
+# Set: USER_ARANGO_URI, USER_ARANGO_USERNAME, USER_ARANGO_PASSWORD, USER_ARANGO_DATABASE
+user_conn_conf = ArangoConfig.from_env(prefix="USER")
+
+# Option 4: Create config directly
+# conn_conf = ArangoConfig(
+#     uri="http://localhost:8535",
+#     username="root",
+#     password="123",
+#     database="mygraph",  # For ArangoDB, 'database' maps to schema/graph
+# )
+# Note: If 'database' (or 'schema_name' for TigerGraph) is not set,
+# Caster will automatically use Schema.general.name as fallback
+
+from graflo.util.onto import FilePattern
+import pathlib
+
+# Create Patterns with file patterns
+patterns = Patterns()
+patterns.add_file_pattern(
+    "work",
+    FilePattern(regex="\Sjson$", sub_path=pathlib.Path("./data"), resource_name="work")
+)
+
+# Or use resource_mapping for simpler initialization
+# patterns = Patterns(
+#     _resource_mapping={
+#         "work": "./data/work.json",
+#     }
+# )
+
+schema.fetch_resource()
+
+from graflo.caster import IngestionParams
+
+caster = Caster(schema)
+
+ingestion_params = IngestionParams(
+    clean_start=False,  # Set to True to wipe existing database
+    # max_items=1000,  # Optional: limit number of items to process
+    # batch_size=10000,  # Optional: customize batch size
+)
+
+caster.ingest(
+    output_config=conn_conf,  # Target database config
+    patterns=patterns,  # Source data patterns
+    ingestion_params=ingestion_params,
+)
+```
+
+### PostgreSQL Schema Inference
+
+```python
+from graflo.db.postgres import PostgresConnection
+from graflo.db.postgres.heuristics import infer_schema_from_postgres
+from graflo.db.connection.onto import PostgresConfig
+from graflo import Caster
+from graflo.onto import DBFlavor
+
+# Connect to PostgreSQL
+postgres_config = PostgresConfig.from_docker_env()  # or PostgresConfig.from_env()
+postgres_conn = PostgresConnection(postgres_config)
+
+# Infer schema from PostgreSQL 3NF database
+schema = infer_schema_from_postgres(
+    postgres_conn,
+    schema_name="public",  # PostgreSQL schema name
+    db_flavor=DBFlavor.ARANGO  # Target graph database flavor
+)
+
+# Close PostgreSQL connection
+postgres_conn.close()
+
+# Use the inferred schema with Caster
+caster = Caster(schema)
+# ... continue with ingestion
+```
+
+## Development
+
+To install requirements
+
+```shell
+git clone git@github.com:growgraph/graflo.git && cd graflo
+uv sync --dev
+```
+
+### Tests
+
+#### Test databases
+Spin up Arango from [arango docker folder](./docker/arango) by
+
+```shell
+docker-compose --env-file .env up arango
+```
+
+Neo4j from [neo4j docker folder](./docker/neo4j) by
+
+```shell
+docker-compose --env-file .env up neo4j
+```
+
+and TigerGraph from [tigergraph docker folder](./docker/tigergraph) by
+
+```shell
+docker-compose --env-file .env up tigergraph
+```
+
+To run unit tests
+
+```shell
+pytest test
+```
+
+## Requirements
+
+- Python 3.10+
+- python-arango
+- sqlalchemy>=2.0.0 (for PostgreSQL and SQL data sources)
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.

graflo-1.3.7/README.md

@@ -0,0 +1,214 @@
+# GraFlo <img src="https://raw.githubusercontent.com/growgraph/graflo/main/docs/assets/favicon.ico" alt="graflo logo" style="height: 32px; width:32px;"/>
+
+A framework for transforming **tabular** (CSV, SQL) and **hierarchical** data (JSON, XML) into property graphs and ingesting them into graph databases (ArangoDB, Neo4j, **TigerGraph**).
+
+> **⚠️ Package Renamed**: This package was formerly known as `graphcast`.
+
+![Python](https://img.shields.io/badge/python-3.10-blue.svg) 
+[![PyPI version](https://badge.fury.io/py/graflo.svg)](https://badge.fury.io/py/graflo)
+[![PyPI Downloads](https://static.pepy.tech/badge/graflo)](https://pepy.tech/projects/graflo)
+[![License: BSL](https://img.shields.io/badge/license-BSL--1.1-green)](https://github.com/growgraph/graflo/blob/main/LICENSE)
+[![pre-commit](https://github.com/growgraph/graflo/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/growgraph/graflo/actions/workflows/pre-commit.yml)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.15446131.svg)]( https://doi.org/10.5281/zenodo.15446131)
+
+## Core Concepts
+
+### Property Graphs
+graflo works with property graphs, which consist of:
+
+- **Vertices**: Nodes with properties and optional unique identifiers
+- **Edges**: Relationships between vertices with their own properties
+- **Properties**: Both vertices and edges may have properties
+
+### Schema
+The Schema defines how your data should be transformed into a graph and contains:
+
+- **Vertex Definitions**: Specify vertex types, their properties, and unique identifiers
+  - Fields can be specified as strings (backward compatible) or typed `Field` objects with types (INT, FLOAT, STRING, DATETIME, BOOL)
+  - Type information enables better validation and database-specific optimizations
+- **Edge Definitions**: Define relationships between vertices and their properties
+  - Weight fields support typed definitions for better type safety
+- **Resource Mapping**: describe how data sources map to vertices and edges
+- **Transforms**: Modify data during the casting process
+- **Automatic Schema Inference**: Generate schemas automatically from PostgreSQL 3NF databases
+
+### Resources
+Resources are your data sources that can be:
+
+- **Table-like**: CSV files, database tables
+- **JSON-like**: JSON files, nested data structures
+
+## Features
+
+- **Graph Transformation Meta-language**: A powerful declarative language to describe how your data becomes a property graph:
+    - Define vertex and edge structures with typed fields
+    - Set compound indexes for vertices and edges
+    - Use blank vertices for complex relationships
+    - Specify edge constraints and properties with typed weight fields
+    - Apply advanced filtering and transformations
+- **Typed Schema Definitions**: Enhanced type support throughout the schema system
+    - Vertex fields support types (INT, FLOAT, STRING, DATETIME, BOOL) for better validation
+    - Edge weight fields can specify types for improved type safety
+    - Backward compatible: fields without types default to None (suitable for databases like ArangoDB)
+- **🚀 PostgreSQL Schema Inference**: **Automatically generate schemas from PostgreSQL 3NF databases** - No manual schema definition needed!
+    - Introspect PostgreSQL schemas to identify vertex-like and edge-like tables
+    - Automatically map PostgreSQL data types to graflo Field types (INT, FLOAT, STRING, DATETIME, BOOL)
+    - Infer vertex configurations from table structures with proper indexes
+    - Infer edge configurations from foreign key relationships
+    - Create Resource mappings from PostgreSQL tables automatically
+    - Direct database access - ingest data without exporting to files first
+- **Parallel processing**: Use as many cores as you have
+- **Database support**: Ingest into ArangoDB, Neo4j, and **TigerGraph** using the same API (database agnostic). Source data from PostgreSQL and other SQL databases.
+- **Server-side filtering**: Efficient querying with server-side filtering support (TigerGraph REST++ API)
+
+## Documentation
+Full documentation is available at: [growgraph.github.io/graflo](https://growgraph.github.io/graflo)
+
+## Installation
+
+```bash
+pip install graflo
+```
+
+## Usage Examples
+
+### Simple ingest
+
+```python
+from suthing import FileHandle
+
+from graflo import Schema, Caster, Patterns
+from graflo.db.connection.onto import ArangoConfig
+
+schema = Schema.from_dict(FileHandle.load("schema.yaml"))
+
+# Option 1: Load config from docker/arango/.env (recommended)
+conn_conf = ArangoConfig.from_docker_env()
+
+# Option 2: Load from environment variables
+# Set: ARANGO_URI, ARANGO_USERNAME, ARANGO_PASSWORD, ARANGO_DATABASE
+conn_conf = ArangoConfig.from_env()
+
+# Option 3: Load with custom prefix (for multiple configs)
+# Set: USER_ARANGO_URI, USER_ARANGO_USERNAME, USER_ARANGO_PASSWORD, USER_ARANGO_DATABASE
+user_conn_conf = ArangoConfig.from_env(prefix="USER")
+
+# Option 4: Create config directly
+# conn_conf = ArangoConfig(
+#     uri="http://localhost:8535",
+#     username="root",
+#     password="123",
+#     database="mygraph",  # For ArangoDB, 'database' maps to schema/graph
+# )
+# Note: If 'database' (or 'schema_name' for TigerGraph) is not set,
+# Caster will automatically use Schema.general.name as fallback
+
+from graflo.util.onto import FilePattern
+import pathlib
+
+# Create Patterns with file patterns
+patterns = Patterns()
+patterns.add_file_pattern(
+    "work",
+    FilePattern(regex="\Sjson$", sub_path=pathlib.Path("./data"), resource_name="work")
+)
+
+# Or use resource_mapping for simpler initialization
+# patterns = Patterns(
+#     _resource_mapping={
+#         "work": "./data/work.json",
+#     }
+# )
+
+schema.fetch_resource()
+
+from graflo.caster import IngestionParams
+
+caster = Caster(schema)
+
+ingestion_params = IngestionParams(
+    clean_start=False,  # Set to True to wipe existing database
+    # max_items=1000,  # Optional: limit number of items to process
+    # batch_size=10000,  # Optional: customize batch size
+)
+
+caster.ingest(
+    output_config=conn_conf,  # Target database config
+    patterns=patterns,  # Source data patterns
+    ingestion_params=ingestion_params,
+)
+```
+
+### PostgreSQL Schema Inference
+
+```python
+from graflo.db.postgres import PostgresConnection
+from graflo.db.postgres.heuristics import infer_schema_from_postgres
+from graflo.db.connection.onto import PostgresConfig
+from graflo import Caster
+from graflo.onto import DBFlavor
+
+# Connect to PostgreSQL
+postgres_config = PostgresConfig.from_docker_env()  # or PostgresConfig.from_env()
+postgres_conn = PostgresConnection(postgres_config)
+
+# Infer schema from PostgreSQL 3NF database
+schema = infer_schema_from_postgres(
+    postgres_conn,
+    schema_name="public",  # PostgreSQL schema name
+    db_flavor=DBFlavor.ARANGO  # Target graph database flavor
+)
+
+# Close PostgreSQL connection
+postgres_conn.close()
+
+# Use the inferred schema with Caster
+caster = Caster(schema)
+# ... continue with ingestion
+```
+
+## Development
+
+To install requirements
+
+```shell
+git clone git@github.com:growgraph/graflo.git && cd graflo
+uv sync --dev
+```
+
+### Tests
+
+#### Test databases
+Spin up Arango from [arango docker folder](./docker/arango) by
+
+```shell
+docker-compose --env-file .env up arango
+```
+
+Neo4j from [neo4j docker folder](./docker/neo4j) by
+
+```shell
+docker-compose --env-file .env up neo4j
+```
+
+and TigerGraph from [tigergraph docker folder](./docker/tigergraph) by
+
+```shell
+docker-compose --env-file .env up tigergraph
+```
+
+To run unit tests
+
+```shell
+pytest test
+```
+
+## Requirements
+
+- Python 3.10+
+- python-arango
+- sqlalchemy>=2.0.0 (for PostgreSQL and SQL data sources)
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.

graflo-1.3.7/graflo/README.md

@@ -0,0 +1,18 @@
+### Table Config  
+
+Table part of the config specifies how input sources will be transformed and mapped to vertex collections.
+
+```yaml
+table:
+-   tabletype: ibes
+    encoding: ISO-8859-1
+    transforms:
+    -   foo: parse_date_ibes
+        module: graflo.util.transform
+        input:
+        -   ANNDATS
+        -   ANNTIMS
+        output:
+        -   datetime_announce
+```
+