Flowfile 0.2.2__tar.gz → 0.3.0__tar.gz

flowfile-0.3.0/PKG-INFO

@@ -0,0 +1,219 @@
+Metadata-Version: 2.3
+Name: Flowfile
+Version: 0.3.0
+Summary: Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)
+Author: Edward van Eechoud
+Author-email: evaneechoud@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: XlsxWriter (>=3.2.0,<3.3.0)
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: airbyte-cdk (==6.47.2)
+Requires-Dist: bcrypt (>=4.3.0,<5.0.0)
+Requires-Dist: connectorx (>=0.4.2,<0.5.0)
+Requires-Dist: databases (>=0.9.0,<0.10.0)
+Requires-Dist: faker (>=23.1.0,<23.2.0)
+Requires-Dist: fastapi (>=0.115.2,<0.116.0)
+Requires-Dist: fastexcel (>=0.12.0,<0.13.0)
+Requires-Dist: google-api-python-client (>=2.149.0,<2.150.0)
+Requires-Dist: gspread (>=6.1.3,<6.2.0)
+Requires-Dist: loky (>=3.4.1,<3.5.0)
+Requires-Dist: methodtools (>=0.4.7,<0.5.0)
+Requires-Dist: openpyxl (>=3.1.2,<3.2.0)
+Requires-Dist: passlib (>=1.7.4,<1.8.0)
+Requires-Dist: pendulum (==2.1.2) ; python_version < "3.12"
+Requires-Dist: polars (>1.8.2,<=1.25.2)
+Requires-Dist: polars-distance (>=0.4.3,<0.5.0)
+Requires-Dist: polars-ds (>=0.6.0)
+Requires-Dist: polars-expr-transformer (>0.4.7.0)
+Requires-Dist: polars-grouper (>=0.3.0,<0.4.0)
+Requires-Dist: polars_simed (>=0.3.4,<0.4.0)
+Requires-Dist: pyairbyte-flowfile (==0.20.2)
+Requires-Dist: pyarrow (>=18.0.0,<19.0.0)
+Requires-Dist: pydantic (>=2.9.2,<2.10.0)
+Requires-Dist: pyinstaller (>=6.11.0,<7.0.0)
+Requires-Dist: pytest (>=8.3.4,<9.0.0)
+Requires-Dist: python-jose (>=3.4.0,<4.0.0)
+Requires-Dist: python-multipart (>=0.0.12,<0.1.0)
+Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
+Description-Content-Type: text/markdown
+
+<h1 align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/logo.png" alt="Flowfile Logo" width="100">
+  <br>
+  Flowfile
+</h1>
+
+<p align="center">
+  <b>Main Repository</b>: <a href="https://github.com/Edwardvaneechoud/Flowfile">Edwardvaneechoud/Flowfile</a><br>
+  <b>Documentation</b>: 
+  <a href="https://edwardvaneechoud.github.io/Flowfile/">Website</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_core/README.md">Core</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_worker/README.md">Worker</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frontend/README.md">Frontend</a> - 
+  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Technical Architecture</a>
+</p>
+
+<p>
+Flowfile is a visual ETL tool and Python library suite that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, or define data flows programmatically with Python and analyze results - all with high-performance data processing.
+</p>
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/group_by_screenshot.png" alt="Flowfile Interface" width="800"/>
+</div>
+
+## ⚡ Technical Design
+
+The `Flowfile` PyPI package provides the backend services and the `flowfile_frame` Python library:
+
+- **Core (`flowfile_core`)** (FastAPI): The main ETL engine using Polars for high-performance data transformations. Typically runs on port `:63578`.
+- **Worker (`flowfile_worker`)** (FastAPI): Handles computation-intensive tasks and caching of data operations, supporting the Core service. Typically runs on port `:63579`.
+- **FlowFrame API (`flowfile_frame`)**: A Python library with a Polars-like API for defining data manipulation pipelines programmatically, which also generates an underlying ETL graph compatible with the Flowfile ecosystem.
+
+Each flow is represented as a directed acyclic graph (DAG), where nodes represent data operations and edges represent data flow between operations.
+
+For a deeper dive into the technical architecture, check out [this article](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c) on how Flowfile leverages Polars for efficient data processing.
+
+## ✨ Introducing FlowFile Frame - A Polars-Like API for ETL
+
+FlowFile Frame is a Python library that provides a familiar Polars-like API for data manipulation, while simultaneously building an ETL (Extract, Transform, Load) graph under the hood. This allows you to:
+
+1. Write data transformation code using a simple, Pandas/Polars-like API
+2. Automatically generate executable ETL workflows compatible with the Flowfile ecosystem
+3. Visualize, save, and share your data pipelines
+4. Get the performance benefits of Polars with the traceability of ETL graphs
+
+### FlowFrame Quick Start
+
+```python
+import flowfile_frame as ff
+from flowfile_frame.utils import open_graph_in_editor
+
+# Create a complex data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+open_graph_in_editor(df.flow_graph)
+
+```
+
+### Key FlowFrame Features
+
+- **Familiar API**: Based on Polars, making it easy to learn if you know Pandas or Polars
+- **ETL Graph Generation**: Automatically builds a directed acyclic graph of your data operations
+- **Lazy Evaluation**: Operations are not executed until `collect()` or a write operation
+- **Interoperability**: Saved `.flowfile` graphs can be opened in the visual Flowfile Designer
+- **High Performance**: Leverages Polars for fast data processing
+- **Reproducible**: Save and share your data transformation workflows
+
+### Common FlowFrame Operations
+
+```python
+import flowfile_frame as ff
+from flowfile_frame import col, when
+
+# Create from dictionary
+df = ff.from_dict({
+    "id": [1, 2, 3],
+    "name": ["Alice", "Bob", "Charlie"],
+    "age": [25, 35, 28]
+})
+
+flow_graph = df.flow_graph
+# Reading data
+# df_csv = ff.read_csv("data.csv")
+# df_parquet = ff.read_parquet("data.parquet")
+
+# Filtering
+adults = df.filter(col("age") >= 30)
+
+# Select and transform
+result = df.select(
+    col("name"),
+    (col("age") * 2).alias("double_age")
+)
+
+# Add new columns
+df_with_cols = df.with_columns([
+    (col("age") + 10).alias("future_age"),
+    when(col("age") >= 30).then(ff.lit("Senior")).otherwise(ff.lit("Junior")).alias("status")]
+)
+
+# Group by and aggregate
+df_sales = ff.from_dict({
+    "region": ["North", "South", "North", "South"],
+    "sales": [100, 200, 150, 300]
+})
+sales_by_region = df_sales.group_by("region").agg([
+    col("sales").sum().alias("total_sales"),
+    col("sales").mean().alias("avg_sales")
+])
+
+# Joins
+customers = ff.from_dict({"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]}, flow_graph=flow_graph)
+orders = ff.from_dict({"id": [101, 102], "customer_id": [1, 2], "amount": [100, 200]}, flow_graph=flow_graph)
+joined = customers.join(orders, left_on="id", right_on="customer_id")
+
+# Save and visualize ETL graph
+
+result.save_graph("my_pipeline.flowfile")
+# open_graph_in_editor(result.flow_graph, "my_pipeline.flowfile") # Opens in Designer UI if installed
+```
+
+For more detailed information on all available operations, including pivoting, window functions, complex workflows, and more, please refer to the [FlowFrame documentation](https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frame/README.md).
+
+## 🔥 Example Use Cases
+
+Flowfile is great for:
+
+- **Data Cleaning & Transformation**
+  - Complex joins (fuzzy matching)
+  - Text-to-rows transformations
+  - Advanced filtering and grouping
+  - Custom formulas and expressions
+  - Filter data based on conditions
+
+- **Performance**
+  - Built to scale out of core
+  - Using Polars for data processing
+
+- **Data Integration**
+  - Standardize data formats
+  - Handle messy Excel files 
+
+- **ETL Operations**
+  - Data quality checks
+
+(For more visual examples of these use cases, please see our [main GitHub repository](https://github.com/Edwardvaneechoud/Flowfile#-example-use-cases)).
+
+## 🚀 Getting Started
+
+### Installing the Flowfile Python Package
+
+This package provides the `flowfile_core` and `flowfile_worker` backend services, and the `flowfile_frame` library.
+
+```bash
+pip install Flowfile
+```
+
+Once installed, you can use `flowfile_frame` as a library in your Python scripts (see Quick Start above).
+
+### Full Application with Visual Designer
+
+For the complete visual ETL experience with the Designer UI, please see the [installation instructions in the main repository](https://github.com/Edwardvaneechoud/Flowfile#-getting-started).
+
+Available options include:
+- Desktop application (recommended for most users)
+- Docker setup (backend services + web frontend)
+- Manual setup for development
+
+## 📋 Development Roadmap
+
+For the latest development roadmap and TODO list, please refer to the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-todo).
+

{flowfile-0.2.2 → flowfile-0.3.0}/flowfile/__init__.py

@@ -18,7 +18,7 @@ from flowfile_frame.expr import (
     sum, min, max, mean, count, when
 )
 from flowfile_frame.group_frame import GroupByFrame
-from flowfile_frame.utils import create_etl_graph, open_graph_in_editor
+from flowfile_frame.utils import create_flow_graph, open_graph_in_editor
 from flowfile_frame.selectors import (
     numeric, float_, integer, string, temporal,
     datetime, date, time, duration, boolean,
@@ -57,7 +57,7 @@ __all__ = [
     'by_dtype', 'contains', 'starts_with', 'ends_with', 'matches',
 
     # Utilities
-    'create_etl_graph', 'open_graph_in_editor',
+    'create_flow_graph', 'open_graph_in_editor',
 
     # Data types from Polars
     'Int8', 'Int16', 'Int32', 'Int64', 'Int128',

{flowfile-0.2.2 → flowfile-0.3.0}/flowfile_frame/flowfile_frame/__init__.py

@@ -4,7 +4,7 @@
 # Core classes
 from flowfile_frame.flow_frame import FlowFrame   # noqa: F401
 
-from flowfile_frame.utils import create_etl_graph  # noqa: F401
+from flowfile_frame.utils import create_flow_graph  # noqa: F401
 
 # Commonly used functions
 from flowfile_frame.expr import (  # noqa: F401

{flowfile-0.2.2 → flowfile-0.3.0}/flowfile_frame/flowfile_frame/flow_frame.py

@@ -16,7 +16,7 @@ from flowfile_core.schemas import input_schema, transform_schema
 from flowfile_frame.expr import Expr, Column, lit, col
 from flowfile_frame.selectors import Selector
 from flowfile_frame.group_frame import GroupByFrame
-from flowfile_frame.utils import _parse_inputs_as_iterable, create_etl_graph
+from flowfile_frame.utils import _parse_inputs_as_iterable, create_flow_graph
 from flowfile_frame.join import _normalize_columns_to_list, _create_join_mappings
 
 node_id_counter = 0
@@ -92,7 +92,7 @@ class FlowFrame:
 
         # Create a new flow graph if none is provided
         if flow_graph is None:
-            flow_graph = create_etl_graph()
+            flow_graph = create_flow_graph()
 
         flow_id = flow_graph.flow_id
 
@@ -198,7 +198,7 @@ class FlowFrame:
 
         # Initialize graph
         if flow_graph is None:
-            flow_graph = create_etl_graph()
+            flow_graph = create_flow_graph()
         self.flow_graph = flow_graph
         # Set up data
         if isinstance(data, FlowDataEngine):
@@ -1922,7 +1922,7 @@ def read_csv(file_path, *, flow_graph: FlowGraph = None, separator: str = ';',
     # Create new node ID
     node_id = generate_node_id()
     if flow_graph is None:
-        flow_graph = create_etl_graph()
+        flow_graph = create_flow_graph()
 
     flow_id = flow_graph.flow_id
 
@@ -1982,7 +1982,7 @@ def read_parquet(file_path, *, flow_graph: FlowGraph = None, description: str =
     node_id = generate_node_id()
 
     if flow_graph is None:
-        flow_graph = create_etl_graph()
+        flow_graph = create_flow_graph()
 
     flow_id = flow_graph.flow_id
 
@@ -2028,7 +2028,7 @@ def from_dict(data, *, flow_graph: FlowGraph = None, description: str = None) ->
     node_id = generate_node_id()
 
     if not flow_graph:
-        flow_graph = create_etl_graph()
+        flow_graph = create_flow_graph()
     flow_id = flow_graph.flow_id
 
     input_node = input_schema.NodeManualInput(

{flowfile-0.2.2 → flowfile-0.3.0}/flowfile_frame/flowfile_frame/utils.py

@@ -33,7 +33,7 @@ def _generate_id() -> int:
     return int(uuid.uuid4().int % 100000)
 
 
-def create_etl_graph() -> FlowGraph:
+def create_flow_graph() -> FlowGraph:
     flow_id = _generate_id()
     flow_settings = schemas.FlowSettings(
         flow_id=flow_id,
@@ -127,13 +127,13 @@ def import_flow_to_editor(flow_path: str, auth_token: str) -> Optional[int]:
         return None
 
 
-def open_graph_in_editor(etl_graph: FlowGraph, storage_location: str = None) -> bool:
+def open_graph_in_editor(flow_graph: FlowGraph, storage_location: str = None) -> bool:
     """
     Save the ETL graph and open it in the Flowfile editor.
 
     Parameters:
     -----------
-    etl_graph : FlowGraph
+    flow_graph : FlowGraph
         The graph to save and open
     storage_location : str, optional
         Where to save the flowfile. If None, a default name is used.
@@ -152,8 +152,8 @@ def open_graph_in_editor(etl_graph: FlowGraph, storage_location: str = None) ->
         # Ensure path is absolute
         storage_location = os.path.abspath(storage_location)
 
-    etl_graph.apply_layout()
-    etl_graph.save_flow(storage_location)
+    flow_graph.apply_layout()
+    flow_graph.save_flow(storage_location)
     print(f"Flow saved to: {storage_location}")
 
     # Check if Flowfile is running, and start it if not

{flowfile-0.2.2 → flowfile-0.3.0}/pyproject.toml

@@ -1,8 +1,8 @@
 [tool.poetry]
 name = "Flowfile"
-version = "0.2.2"
+version = "0.3.0"
 description = "Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)"
-readme = "README.md"
+readme = "readme-pypi.md"
 authors = ["Edward van Eechoud <evaneechoud@gmail.com>"]
 packages = [
     { include = "flowfile_core", from = "flowfile_core" },

flowfile-0.3.0/readme-pypi.md

@@ -0,0 +1,175 @@
+<h1 align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/logo.png" alt="Flowfile Logo" width="100">
+  <br>
+  Flowfile
+</h1>
+
+<p align="center">
+  <b>Main Repository</b>: <a href="https://github.com/Edwardvaneechoud/Flowfile">Edwardvaneechoud/Flowfile</a><br>
+  <b>Documentation</b>: 
+  <a href="https://edwardvaneechoud.github.io/Flowfile/">Website</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_core/README.md">Core</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_worker/README.md">Worker</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frontend/README.md">Frontend</a> - 
+  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Technical Architecture</a>
+</p>
+
+<p>
+Flowfile is a visual ETL tool and Python library suite that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, or define data flows programmatically with Python and analyze results - all with high-performance data processing.
+</p>
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/group_by_screenshot.png" alt="Flowfile Interface" width="800"/>
+</div>
+
+## ⚡ Technical Design
+
+The `Flowfile` PyPI package provides the backend services and the `flowfile_frame` Python library:
+
+- **Core (`flowfile_core`)** (FastAPI): The main ETL engine using Polars for high-performance data transformations. Typically runs on port `:63578`.
+- **Worker (`flowfile_worker`)** (FastAPI): Handles computation-intensive tasks and caching of data operations, supporting the Core service. Typically runs on port `:63579`.
+- **FlowFrame API (`flowfile_frame`)**: A Python library with a Polars-like API for defining data manipulation pipelines programmatically, which also generates an underlying ETL graph compatible with the Flowfile ecosystem.
+
+Each flow is represented as a directed acyclic graph (DAG), where nodes represent data operations and edges represent data flow between operations.
+
+For a deeper dive into the technical architecture, check out [this article](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c) on how Flowfile leverages Polars for efficient data processing.
+
+## ✨ Introducing FlowFile Frame - A Polars-Like API for ETL
+
+FlowFile Frame is a Python library that provides a familiar Polars-like API for data manipulation, while simultaneously building an ETL (Extract, Transform, Load) graph under the hood. This allows you to:
+
+1. Write data transformation code using a simple, Pandas/Polars-like API
+2. Automatically generate executable ETL workflows compatible with the Flowfile ecosystem
+3. Visualize, save, and share your data pipelines
+4. Get the performance benefits of Polars with the traceability of ETL graphs
+
+### FlowFrame Quick Start
+
+```python
+import flowfile_frame as ff
+from flowfile_frame.utils import open_graph_in_editor
+
+# Create a complex data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+open_graph_in_editor(df.flow_graph)
+
+```
+
+### Key FlowFrame Features
+
+- **Familiar API**: Based on Polars, making it easy to learn if you know Pandas or Polars
+- **ETL Graph Generation**: Automatically builds a directed acyclic graph of your data operations
+- **Lazy Evaluation**: Operations are not executed until `collect()` or a write operation
+- **Interoperability**: Saved `.flowfile` graphs can be opened in the visual Flowfile Designer
+- **High Performance**: Leverages Polars for fast data processing
+- **Reproducible**: Save and share your data transformation workflows
+
+### Common FlowFrame Operations
+
+```python
+import flowfile_frame as ff
+from flowfile_frame import col, when
+
+# Create from dictionary
+df = ff.from_dict({
+    "id": [1, 2, 3],
+    "name": ["Alice", "Bob", "Charlie"],
+    "age": [25, 35, 28]
+})
+
+flow_graph = df.flow_graph
+# Reading data
+# df_csv = ff.read_csv("data.csv")
+# df_parquet = ff.read_parquet("data.parquet")
+
+# Filtering
+adults = df.filter(col("age") >= 30)
+
+# Select and transform
+result = df.select(
+    col("name"),
+    (col("age") * 2).alias("double_age")
+)
+
+# Add new columns
+df_with_cols = df.with_columns([
+    (col("age") + 10).alias("future_age"),
+    when(col("age") >= 30).then(ff.lit("Senior")).otherwise(ff.lit("Junior")).alias("status")]
+)
+
+# Group by and aggregate
+df_sales = ff.from_dict({
+    "region": ["North", "South", "North", "South"],
+    "sales": [100, 200, 150, 300]
+})
+sales_by_region = df_sales.group_by("region").agg([
+    col("sales").sum().alias("total_sales"),
+    col("sales").mean().alias("avg_sales")
+])
+
+# Joins
+customers = ff.from_dict({"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]}, flow_graph=flow_graph)
+orders = ff.from_dict({"id": [101, 102], "customer_id": [1, 2], "amount": [100, 200]}, flow_graph=flow_graph)
+joined = customers.join(orders, left_on="id", right_on="customer_id")
+
+# Save and visualize ETL graph
+
+result.save_graph("my_pipeline.flowfile")
+# open_graph_in_editor(result.flow_graph, "my_pipeline.flowfile") # Opens in Designer UI if installed
+```
+
+For more detailed information on all available operations, including pivoting, window functions, complex workflows, and more, please refer to the [FlowFrame documentation](https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frame/README.md).
+
+## 🔥 Example Use Cases
+
+Flowfile is great for:
+
+- **Data Cleaning & Transformation**
+  - Complex joins (fuzzy matching)
+  - Text-to-rows transformations
+  - Advanced filtering and grouping
+  - Custom formulas and expressions
+  - Filter data based on conditions
+
+- **Performance**
+  - Built to scale out of core
+  - Using Polars for data processing
+
+- **Data Integration**
+  - Standardize data formats
+  - Handle messy Excel files 
+
+- **ETL Operations**
+  - Data quality checks
+
+(For more visual examples of these use cases, please see our [main GitHub repository](https://github.com/Edwardvaneechoud/Flowfile#-example-use-cases)).
+
+## 🚀 Getting Started
+
+### Installing the Flowfile Python Package
+
+This package provides the `flowfile_core` and `flowfile_worker` backend services, and the `flowfile_frame` library.
+
+```bash
+pip install Flowfile
+```
+
+Once installed, you can use `flowfile_frame` as a library in your Python scripts (see Quick Start above).
+
+### Full Application with Visual Designer
+
+For the complete visual ETL experience with the Designer UI, please see the [installation instructions in the main repository](https://github.com/Edwardvaneechoud/Flowfile#-getting-started).
+
+Available options include:
+- Desktop application (recommended for most users)
+- Docker setup (backend services + web frontend)
+- Manual setup for development
+
+## 📋 Development Roadmap
+
+For the latest development roadmap and TODO list, please refer to the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-todo).