Flowfile 0.2.2__py3-none-any.whl → 0.3.0__py3-none-any.whl

flowfile/__init__.py

@@ -18,7 +18,7 @@ from flowfile_frame.expr import (
     sum, min, max, mean, count, when
 )
 from flowfile_frame.group_frame import GroupByFrame
-from flowfile_frame.utils import create_etl_graph, open_graph_in_editor
+from flowfile_frame.utils import create_flow_graph, open_graph_in_editor
 from flowfile_frame.selectors import (
     numeric, float_, integer, string, temporal,
     datetime, date, time, duration, boolean,
@@ -57,7 +57,7 @@ __all__ = [
     'by_dtype', 'contains', 'starts_with', 'ends_with', 'matches',
 
     # Utilities
-    'create_etl_graph', 'open_graph_in_editor',
+    'create_flow_graph', 'open_graph_in_editor',
 
     # Data types from Polars
     'Int8', 'Int16', 'Int32', 'Int64', 'Int128',

flowfile-0.3.0.dist-info/METADATA

@@ -0,0 +1,219 @@
+Metadata-Version: 2.3
+Name: Flowfile
+Version: 0.3.0
+Summary: Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)
+Author: Edward van Eechoud
+Author-email: evaneechoud@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: XlsxWriter (>=3.2.0,<3.3.0)
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: airbyte-cdk (==6.47.2)
+Requires-Dist: bcrypt (>=4.3.0,<5.0.0)
+Requires-Dist: connectorx (>=0.4.2,<0.5.0)
+Requires-Dist: databases (>=0.9.0,<0.10.0)
+Requires-Dist: faker (>=23.1.0,<23.2.0)
+Requires-Dist: fastapi (>=0.115.2,<0.116.0)
+Requires-Dist: fastexcel (>=0.12.0,<0.13.0)
+Requires-Dist: google-api-python-client (>=2.149.0,<2.150.0)
+Requires-Dist: gspread (>=6.1.3,<6.2.0)
+Requires-Dist: loky (>=3.4.1,<3.5.0)
+Requires-Dist: methodtools (>=0.4.7,<0.5.0)
+Requires-Dist: openpyxl (>=3.1.2,<3.2.0)
+Requires-Dist: passlib (>=1.7.4,<1.8.0)
+Requires-Dist: pendulum (==2.1.2) ; python_version < "3.12"
+Requires-Dist: polars (>1.8.2,<=1.25.2)
+Requires-Dist: polars-distance (>=0.4.3,<0.5.0)
+Requires-Dist: polars-ds (>=0.6.0)
+Requires-Dist: polars-expr-transformer (>0.4.7.0)
+Requires-Dist: polars-grouper (>=0.3.0,<0.4.0)
+Requires-Dist: polars_simed (>=0.3.4,<0.4.0)
+Requires-Dist: pyairbyte-flowfile (==0.20.2)
+Requires-Dist: pyarrow (>=18.0.0,<19.0.0)
+Requires-Dist: pydantic (>=2.9.2,<2.10.0)
+Requires-Dist: pyinstaller (>=6.11.0,<7.0.0)
+Requires-Dist: pytest (>=8.3.4,<9.0.0)
+Requires-Dist: python-jose (>=3.4.0,<4.0.0)
+Requires-Dist: python-multipart (>=0.0.12,<0.1.0)
+Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
+Description-Content-Type: text/markdown
+
+<h1 align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/logo.png" alt="Flowfile Logo" width="100">
+  <br>
+  Flowfile
+</h1>
+
+<p align="center">
+  <b>Main Repository</b>: <a href="https://github.com/Edwardvaneechoud/Flowfile">Edwardvaneechoud/Flowfile</a><br>
+  <b>Documentation</b>: 
+  <a href="https://edwardvaneechoud.github.io/Flowfile/">Website</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_core/README.md">Core</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_worker/README.md">Worker</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frontend/README.md">Frontend</a> - 
+  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Technical Architecture</a>
+</p>
+
+<p>
+Flowfile is a visual ETL tool and Python library suite that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, or define data flows programmatically with Python and analyze results - all with high-performance data processing.
+</p>
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/group_by_screenshot.png" alt="Flowfile Interface" width="800"/>
+</div>
+
+## ⚡ Technical Design
+
+The `Flowfile` PyPI package provides the backend services and the `flowfile_frame` Python library:
+
+- **Core (`flowfile_core`)** (FastAPI): The main ETL engine using Polars for high-performance data transformations. Typically runs on port `:63578`.
+- **Worker (`flowfile_worker`)** (FastAPI): Handles computation-intensive tasks and caching of data operations, supporting the Core service. Typically runs on port `:63579`.
+- **FlowFrame API (`flowfile_frame`)**: A Python library with a Polars-like API for defining data manipulation pipelines programmatically, which also generates an underlying ETL graph compatible with the Flowfile ecosystem.
+
+Each flow is represented as a directed acyclic graph (DAG), where nodes represent data operations and edges represent data flow between operations.
+
+For a deeper dive into the technical architecture, check out [this article](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c) on how Flowfile leverages Polars for efficient data processing.
+
+## ✨ Introducing FlowFile Frame - A Polars-Like API for ETL
+
+FlowFile Frame is a Python library that provides a familiar Polars-like API for data manipulation, while simultaneously building an ETL (Extract, Transform, Load) graph under the hood. This allows you to:
+
+1. Write data transformation code using a simple, Pandas/Polars-like API
+2. Automatically generate executable ETL workflows compatible with the Flowfile ecosystem
+3. Visualize, save, and share your data pipelines
+4. Get the performance benefits of Polars with the traceability of ETL graphs
+
+### FlowFrame Quick Start
+
+```python
+import flowfile_frame as ff
+from flowfile_frame.utils import open_graph_in_editor
+
+# Create a complex data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+open_graph_in_editor(df.flow_graph)
+
+```
+
+### Key FlowFrame Features
+
+- **Familiar API**: Based on Polars, making it easy to learn if you know Pandas or Polars
+- **ETL Graph Generation**: Automatically builds a directed acyclic graph of your data operations
+- **Lazy Evaluation**: Operations are not executed until `collect()` or a write operation
+- **Interoperability**: Saved `.flowfile` graphs can be opened in the visual Flowfile Designer
+- **High Performance**: Leverages Polars for fast data processing
+- **Reproducible**: Save and share your data transformation workflows
+
+### Common FlowFrame Operations
+
+```python
+import flowfile_frame as ff
+from flowfile_frame import col, when
+
+# Create from dictionary
+df = ff.from_dict({
+    "id": [1, 2, 3],
+    "name": ["Alice", "Bob", "Charlie"],
+    "age": [25, 35, 28]
+})
+
+flow_graph = df.flow_graph
+# Reading data
+# df_csv = ff.read_csv("data.csv")
+# df_parquet = ff.read_parquet("data.parquet")
+
+# Filtering
+adults = df.filter(col("age") >= 30)
+
+# Select and transform
+result = df.select(
+    col("name"),
+    (col("age") * 2).alias("double_age")
+)
+
+# Add new columns
+df_with_cols = df.with_columns([
+    (col("age") + 10).alias("future_age"),
+    when(col("age") >= 30).then(ff.lit("Senior")).otherwise(ff.lit("Junior")).alias("status")]
+)
+
+# Group by and aggregate
+df_sales = ff.from_dict({
+    "region": ["North", "South", "North", "South"],
+    "sales": [100, 200, 150, 300]
+})
+sales_by_region = df_sales.group_by("region").agg([
+    col("sales").sum().alias("total_sales"),
+    col("sales").mean().alias("avg_sales")
+])
+
+# Joins
+customers = ff.from_dict({"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]}, flow_graph=flow_graph)
+orders = ff.from_dict({"id": [101, 102], "customer_id": [1, 2], "amount": [100, 200]}, flow_graph=flow_graph)
+joined = customers.join(orders, left_on="id", right_on="customer_id")
+
+# Save and visualize ETL graph
+
+result.save_graph("my_pipeline.flowfile")
+# open_graph_in_editor(result.flow_graph, "my_pipeline.flowfile") # Opens in Designer UI if installed
+```
+
+For more detailed information on all available operations, including pivoting, window functions, complex workflows, and more, please refer to the [FlowFrame documentation](https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frame/README.md).
+
+## 🔥 Example Use Cases
+
+Flowfile is great for:
+
+- **Data Cleaning & Transformation**
+  - Complex joins (fuzzy matching)
+  - Text-to-rows transformations
+  - Advanced filtering and grouping
+  - Custom formulas and expressions
+  - Filter data based on conditions
+
+- **Performance**
+  - Built to scale out of core
+  - Using Polars for data processing
+
+- **Data Integration**
+  - Standardize data formats
+  - Handle messy Excel files 
+
+- **ETL Operations**
+  - Data quality checks
+
+(For more visual examples of these use cases, please see our [main GitHub repository](https://github.com/Edwardvaneechoud/Flowfile#-example-use-cases)).
+
+## 🚀 Getting Started
+
+### Installing the Flowfile Python Package
+
+This package provides the `flowfile_core` and `flowfile_worker` backend services, and the `flowfile_frame` library.
+
+```bash
+pip install Flowfile
+```
+
+Once installed, you can use `flowfile_frame` as a library in your Python scripts (see Quick Start above).
+
+### Full Application with Visual Designer
+
+For the complete visual ETL experience with the Designer UI, please see the [installation instructions in the main repository](https://github.com/Edwardvaneechoud/Flowfile#-getting-started).
+
+Available options include:
+- Desktop application (recommended for most users)
+- Docker setup (backend services + web frontend)
+- Manual setup for development
+
+## 📋 Development Roadmap
+
+For the latest development roadmap and TODO list, please refer to the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-todo).
+

{flowfile-0.2.2.dist-info → flowfile-0.3.0.dist-info}/RECORD

@@ -1,7 +1,7 @@
 build_backends/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 build_backends/main.py,sha256=hLmfqTeHLSTiwwZ5mUuoLQgtO40Igvl1_4NbnvzWSgI,9912
 build_backends/main_prd.py,sha256=JR2tYCMWM5ThooQjv5pw6nwVKMQjgsiHgKMhYn9NXWI,6927
-flowfile/__init__.py,sha256=jERp50eC0SrT-lsuMJpHaFoN6NIEzWYkqLvZF0vZ6ls,2299
+flowfile/__init__.py,sha256=ci66Dom0DjMuu_S1wsyy9StLkp95wp6KCqGhj7YVQ0E,2301
 flowfile/__main__.py,sha256=X8ItB1LEC1ZXw_tvegu7sagb2CwqUeWSwWybbO1HtUs,630
 flowfile_core/__init__.py,sha256=dGxpVE9ol33CMRPJSPWlL7AZqXowBmlCx8unxCVWJXQ,254
 flowfile_core/auth/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
@@ -119,15 +119,15 @@ flowfile_core/utils/excel_file_manager.py,sha256=EIad2LenHu-3Yw1FcLmE0KgmLflnvNK
 flowfile_core/utils/fileManager.py,sha256=LnJhK_pwjb9MIApG2e4Hp3L5Z7Wny8YYHaL9SkW8WlE,1371
 flowfile_core/utils/fl_executor.py,sha256=eNnNZHZ9451brzZD00_X8aoCHFl1hR1gVOIGxtE0Db4,1301
 flowfile_core/utils/utils.py,sha256=NkEu21OF1l5weu01g-dAVdJ6BRHdpT2jBrWBSi-wp3c,270
-flowfile_frame/__init__.py,sha256=5Jofwsln7SI6-NcwR3DvMzRhg0nN19z3Ro3ZDZ5fClw,1349
+flowfile_frame/__init__.py,sha256=-fD0U8K8VAw9uLGm9FU7R-mCgLoVtvcNh1YJsfhOzeE,1350
 flowfile_frame/__main__.py,sha256=qN0H-bYXDLIDoJNAurXep27crvKNHiz90HOGWbvC0HM,413
 flowfile_frame/adapters.py,sha256=C6JZZKANoKbHHmwMaF9AqAcZvITvQeb2Dklcpg5JAdY,530
 flowfile_frame/expr.py,sha256=wzJLiHtAqBA4NUEyXW8UhNJJ2jiGDvWEX8RfWT5ICwQ,47370
-flowfile_frame/flow_frame.py,sha256=X4p5Wr2DNoAZhkhCveVPGtFF2LG9FgDZHerLDrVqJdY,77553
+flowfile_frame/flow_frame.py,sha256=TP3u7Nyq0TuT3sba0NGLCgMPnug5MuFCKE7Djafud-U,77559
 flowfile_frame/group_frame.py,sha256=MMGE2_DC8n1J2UxBBm_TyacGiRT_1V2CcWUvsIyhbIQ,9115
 flowfile_frame/join.py,sha256=pezHyNQMcaSPK9vhjaCaelMkgKdrvOQxwiROsa0fmN8,2480
 flowfile_frame/selectors.py,sha256=Ny5IpDP481ClNr5gI7_SjXzeqF16LsPcVQxiyUf5tUw,9130
-flowfile_frame/utils.py,sha256=fTcHWDyN7LVxRtVBoLwlfdEanhGhFZXEtslQUK2LDpQ,5923
+flowfile_frame/utils.py,sha256=WGCY-eDeDlVkIT_zF1Si_80557TcxS51FMSAIdG5iTw,5928
 flowfile_worker/__init__.py,sha256=ZDdn3JCP7LWTiTsmntVIVduB4p2bUkJcZUKVEj7V9TU,1375
 flowfile_worker/configs.py,sha256=7fYtlj06vxDrMiRuMbwvSDOD1JRVMZqnPbcQFuikCJM,2714
 flowfile_worker/create/__init__.py,sha256=vkWy5uODffivUdxt3nNVALj6xgQK3HPBetqR-QqZ-uo,1643
@@ -164,8 +164,8 @@ test_utils/__init__.py,sha256=8WwOgIuKw6YtOc1GWR1DqIhQ8BhlLWqsMyQJSpxnzKk,66
 test_utils/postgres/__init__.py,sha256=y3V_6a9N1Pvm5NIBaA8CFf3i4mvPVY-H1teHA-rg0VU,33
 test_utils/postgres/commands.py,sha256=4oA8EHW3EqwGkG02HSqEGbXEBGM01sUW5FsyHm86W4k,4347
 test_utils/postgres/fixtures.py,sha256=kR8UBjQr3pgbe-xM-V8x8VseTHCPv0EmDEzPHl5Qc8Y,13507
-flowfile-0.2.2.dist-info/LICENSE,sha256=pCfLAA27jMHReYk_wGiirZxWRRXz_Bm7PVInRCa9P5g,1075
-flowfile-0.2.2.dist-info/METADATA,sha256=4jZprT4VmdoP8ahxW6WAm0gpwN5ulfHXr21-I07Hzww,7409
-flowfile-0.2.2.dist-info/WHEEL,sha256=b4K_helf-jlQoXBBETfwnf4B04YC67LOev0jo4fX5m8,88
-flowfile-0.2.2.dist-info/entry_points.txt,sha256=CiyNXUvc77hDbE9rDaAMQFdFCQs-XdBm5_o1WV9_gQA,335
-flowfile-0.2.2.dist-info/RECORD,,
+flowfile-0.3.0.dist-info/LICENSE,sha256=pCfLAA27jMHReYk_wGiirZxWRRXz_Bm7PVInRCa9P5g,1075
+flowfile-0.3.0.dist-info/METADATA,sha256=049rH0Qr90sG9YaKOxeF9MoEBXgsKg8FnCvd7ithtAw,8952
+flowfile-0.3.0.dist-info/WHEEL,sha256=b4K_helf-jlQoXBBETfwnf4B04YC67LOev0jo4fX5m8,88
+flowfile-0.3.0.dist-info/entry_points.txt,sha256=CiyNXUvc77hDbE9rDaAMQFdFCQs-XdBm5_o1WV9_gQA,335
+flowfile-0.3.0.dist-info/RECORD,,

flowfile_frame/__init__.py

@@ -4,7 +4,7 @@
 # Core classes
 from flowfile_frame.flow_frame import FlowFrame   # noqa: F401
 
-from flowfile_frame.utils import create_etl_graph  # noqa: F401
+from flowfile_frame.utils import create_flow_graph  # noqa: F401
 
 # Commonly used functions
 from flowfile_frame.expr import (  # noqa: F401

flowfile_frame/flow_frame.py

@@ -16,7 +16,7 @@ from flowfile_core.schemas import input_schema, transform_schema
 from flowfile_frame.expr import Expr, Column, lit, col
 from flowfile_frame.selectors import Selector
 from flowfile_frame.group_frame import GroupByFrame
-from flowfile_frame.utils import _parse_inputs_as_iterable, create_etl_graph
+from flowfile_frame.utils import _parse_inputs_as_iterable, create_flow_graph
 from flowfile_frame.join import _normalize_columns_to_list, _create_join_mappings
 
 node_id_counter = 0
@@ -92,7 +92,7 @@ class FlowFrame:
 
         # Create a new flow graph if none is provided
         if flow_graph is None:
-            flow_graph = create_etl_graph()
+            flow_graph = create_flow_graph()
 
         flow_id = flow_graph.flow_id
 
@@ -198,7 +198,7 @@ class FlowFrame:
 
         # Initialize graph
         if flow_graph is None:
-            flow_graph = create_etl_graph()
+            flow_graph = create_flow_graph()
         self.flow_graph = flow_graph
         # Set up data
         if isinstance(data, FlowDataEngine):
@@ -1922,7 +1922,7 @@ def read_csv(file_path, *, flow_graph: FlowGraph = None, separator: str = ';',
     # Create new node ID
     node_id = generate_node_id()
     if flow_graph is None:
-        flow_graph = create_etl_graph()
+        flow_graph = create_flow_graph()
 
     flow_id = flow_graph.flow_id
 
@@ -1982,7 +1982,7 @@ def read_parquet(file_path, *, flow_graph: FlowGraph = None, description: str =
     node_id = generate_node_id()
 
     if flow_graph is None:
-        flow_graph = create_etl_graph()
+        flow_graph = create_flow_graph()
 
     flow_id = flow_graph.flow_id
 
@@ -2028,7 +2028,7 @@ def from_dict(data, *, flow_graph: FlowGraph = None, description: str = None) ->
     node_id = generate_node_id()
 
     if not flow_graph:
-        flow_graph = create_etl_graph()
+        flow_graph = create_flow_graph()
     flow_id = flow_graph.flow_id
 
     input_node = input_schema.NodeManualInput(

flowfile_frame/utils.py

@@ -33,7 +33,7 @@ def _generate_id() -> int:
     return int(uuid.uuid4().int % 100000)
 
 
-def create_etl_graph() -> FlowGraph:
+def create_flow_graph() -> FlowGraph:
     flow_id = _generate_id()
     flow_settings = schemas.FlowSettings(
         flow_id=flow_id,
@@ -127,13 +127,13 @@ def import_flow_to_editor(flow_path: str, auth_token: str) -> Optional[int]:
         return None
 
 
-def open_graph_in_editor(etl_graph: FlowGraph, storage_location: str = None) -> bool:
+def open_graph_in_editor(flow_graph: FlowGraph, storage_location: str = None) -> bool:
     """
     Save the ETL graph and open it in the Flowfile editor.
 
     Parameters:
     -----------
-    etl_graph : FlowGraph
+    flow_graph : FlowGraph
         The graph to save and open
     storage_location : str, optional
         Where to save the flowfile. If None, a default name is used.
@@ -152,8 +152,8 @@ def open_graph_in_editor(etl_graph: FlowGraph, storage_location: str = None) ->
         # Ensure path is absolute
         storage_location = os.path.abspath(storage_location)
 
-    etl_graph.apply_layout()
-    etl_graph.save_flow(storage_location)
+    flow_graph.apply_layout()
+    flow_graph.save_flow(storage_location)
     print(f"Flow saved to: {storage_location}")
 
     # Check if Flowfile is running, and start it if not

flowfile-0.2.2.dist-info/METADATA

@@ -1,225 +0,0 @@
-Metadata-Version: 2.3
-Name: Flowfile
-Version: 0.2.2
-Summary: Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)
-Author: Edward van Eechoud
-Author-email: evaneechoud@gmail.com
-Requires-Python: >=3.10,<3.13
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Dist: XlsxWriter (>=3.2.0,<3.3.0)
-Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
-Requires-Dist: airbyte-cdk (==6.47.2)
-Requires-Dist: bcrypt (>=4.3.0,<5.0.0)
-Requires-Dist: connectorx (>=0.4.2,<0.5.0)
-Requires-Dist: databases (>=0.9.0,<0.10.0)
-Requires-Dist: faker (>=23.1.0,<23.2.0)
-Requires-Dist: fastapi (>=0.115.2,<0.116.0)
-Requires-Dist: fastexcel (>=0.12.0,<0.13.0)
-Requires-Dist: google-api-python-client (>=2.149.0,<2.150.0)
-Requires-Dist: gspread (>=6.1.3,<6.2.0)
-Requires-Dist: loky (>=3.4.1,<3.5.0)
-Requires-Dist: methodtools (>=0.4.7,<0.5.0)
-Requires-Dist: openpyxl (>=3.1.2,<3.2.0)
-Requires-Dist: passlib (>=1.7.4,<1.8.0)
-Requires-Dist: pendulum (==2.1.2) ; python_version < "3.12"
-Requires-Dist: polars (>1.8.2,<=1.25.2)
-Requires-Dist: polars-distance (>=0.4.3,<0.5.0)
-Requires-Dist: polars-ds (>=0.6.0)
-Requires-Dist: polars-expr-transformer (>0.4.7.0)
-Requires-Dist: polars-grouper (>=0.3.0,<0.4.0)
-Requires-Dist: polars_simed (>=0.3.4,<0.4.0)
-Requires-Dist: pyairbyte-flowfile (==0.20.2)
-Requires-Dist: pyarrow (>=18.0.0,<19.0.0)
-Requires-Dist: pydantic (>=2.9.2,<2.10.0)
-Requires-Dist: pyinstaller (>=6.11.0,<7.0.0)
-Requires-Dist: pytest (>=8.3.4,<9.0.0)
-Requires-Dist: python-jose (>=3.4.0,<4.0.0)
-Requires-Dist: python-multipart (>=0.0.12,<0.1.0)
-Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
-Description-Content-Type: text/markdown
-
-<h1 align="center">
-  <img src=".github/images/logo.png" alt="Flowfile Logo" width="100">
-  <br>
-  Flowfile
-</h1>
-<p align="center">
-  <b>Documentation</b>:
-  <a href="https://edwardvaneechoud.github.io/Flowfile/">Website</a>
-  -
-  <a href="flowfile_core/README.md">Core</a>
-  -
-  <a href="flowfile_worker/README.md">Worker</a>
-  -
-  <a href="flowfile_frontend/README.md">Frontend</a>
-  -
-  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Technical Architecture</a>
-</p>
-<p>
-Flowfile is a visual ETL tool that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, and analyze results - all without writing code.
-</p>
-
-<div align="center">
-  <img src=".github/images/group_by_screenshot.png" alt="Flowfile Interface" width="800"/>
-</div>
-
-## ⚡ Technical Design
-
-Flowfile operates as three interconnected services:
-
-- **Designer** (Electron + Vue): Visual interface for building data flows
-- **Core** (FastAPI): ETL engine using Polars for data transformations (`:63578`)
-- **Worker** (FastAPI): Handles computation and caching of data operations (`:63579`)
-
-Each flow is represented as a directed acyclic graph (DAG), where nodes represent data operations and edges represent data flow between operations.
-
-For a deeper dive into the technical architecture, check out [this article](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c) on how Flowfile leverages Polars for efficient data processing.
-
-## 🔥 Example Use Cases
-
-- **Data Cleaning & Transformation**
-  - Complex joins (fuzzy matching)
-  - Text to rows transformations
-  - Advanced filtering and grouping
-  - Custom formulas and expressions
-  - Filter data based on conditions
-
-<div align="center">
-  <img src=".github/images/flowfile_demo_1.gif" alt="Flowfile Layout" width="800"/>
-</div>
-
----
-
-- **Performance**
-  - Build to scale out of core
-  - Using polars for data processing
-
-<div align="center">
-  <img src=".github/images/demo_flowfile_write.gif" alt="Flowfile Layout" width="800"/>
-</div>
-
----
-
-### **Data Integration**
-  - Standardize data formats
-  - Handle messy Excel files
-
-
-<div align="center">
-  <img src=".github/images/read_excel_flowfile.gif" alt="Flowfile Layout" width="800"/>
-</div>
-
-
----
-
-- **ETL Operations**
-  - Data quality checks
-
-
-## 🚀 Getting Started
-
-### Prerequisites
-- Python 3.10+
-- Node.js 16+
-- Poetry (Python package manager)
-- Docker & Docker Compose (option, for Docker setup)
-- Make (optional, for build automation)
-
-### Installation Options
-
-#### 1. Desktop Application
-The desktop version offers the best experience with a native interface and integrated services. You can either:
-
-**Option A: Download Pre-built Application** 
-- Download the latest release from [GitHub Releases](https://github.com/Edwardvaneechoud/Flowfile/releases)
-- Run the installer for your platform (Windows, macOS, or Linux)
-  - Note: You may see security warnings since the installer isn't signed. On Windows, click "More info" then "Run anyway". On macOS, right-click the app, select "Open", then confirm. These warnings appear because the app isn't signed with a developer certificate.
-
-**Option B: Build from Source:**
-```bash
-git clone https://github.com/edwardvaneechoud/Flowfile.git
-cd Flowfile
-
-# Build packaged executable
-make    # Creates platform-specific executable
-
-# Or manually:
-poetry install
-poetry run build_backends
-cd flowfile_frontend
-npm install
-npm run build      # All platforms
-```
-
-#### 2. Docker Setup
-Perfect for quick testing, development or deployment scenarios. Runs all services in containers with proper networking and volume management:
-```bash
-# Clone and start all services
-git clone https://github.com/edwardvaneechoud/Flowfile.git
-cd Flowfile
-docker compose up -d
-
-# Access services:
-Frontend: http://localhost:8080 # main service
-Core API: http://localhost:63578/docs
-Worker API: http://localhost:63579/docs
-```
-Just place your files that you want to transform in the directory in shared_data and you're all set!
-
-Docker Compose is also excellent for development, as it automatically sets up all required services and ensures proper communication between them. Code changes in the mounted volumes will be reflected in the running containers.
-
-#### 3. Manual Setup (Development)
-Ideal for development work when you need direct access to all services and hot-reloading:
-
-```bash
-git clone https://github.com/edwardvaneechoud/Flowfile.git
-cd Flowfile
-
-# Install Python dependencies
-poetry install
-
-# Start backend services
-poetry run flowfile_worker  # Starts worker on :63579
-poetry run flowfile_core   # Starts core on :63578
-
-# Start web frontend
-cd flowfile_frontend
-npm install
-npm run dev:web  # Starts web interface on :8080
-```
-
-## 📋 TODO
-
-### Core Features
-- [ ] Add cloud storage support
-  - S3 integration
-  - Azure Data Lake Storage (ADLS)
-- [x] Multi-flow execution support
-- [ ] Polars code reverse engineering
-  - Generate Polars code from visual flows
-  - Import existing Polars scripts
-
-### Documentation
-- [ ] Add comprehensive docstrings
-- [x] Create detailed node documentation
-- [x] Add architectural documentation
-- [ ] Improve inline code comments
-- [ ] Create user guides and tutorials
-
-### Infrastructure
-- [ ] Implement proper testing
-- [x] Add CI/CD pipeline
-- [x] Improve error handling
-- [x] Add monitoring and logging
-
-## 📝 License
-
-[MIT License](LICENSE)
-
-## Acknowledgments
-
-Built with Polars, Vue.js, FastAPI, Vueflow and Electron.
-