Flowfile 0.2.2__tar.gz → 0.3.0.1__tar.gz

flowfile-0.3.0.1/PKG-INFO

@@ -0,0 +1,219 @@
+Metadata-Version: 2.3
+Name: Flowfile
+Version: 0.3.0.1
+Summary: Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)
+Author: Edward van Eechoud
+Author-email: evaneechoud@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: XlsxWriter (>=3.2.0,<3.3.0)
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: airbyte-cdk (==6.47.2)
+Requires-Dist: bcrypt (>=4.3.0,<5.0.0)
+Requires-Dist: connectorx (>=0.4.2,<0.5.0)
+Requires-Dist: databases (>=0.9.0,<0.10.0)
+Requires-Dist: faker (>=23.1.0,<23.2.0)
+Requires-Dist: fastapi (>=0.115.2,<0.116.0)
+Requires-Dist: fastexcel (>=0.12.0,<0.13.0)
+Requires-Dist: google-api-python-client (>=2.149.0,<2.150.0)
+Requires-Dist: gspread (>=6.1.3,<6.2.0)
+Requires-Dist: loky (>=3.4.1,<3.5.0)
+Requires-Dist: methodtools (>=0.4.7,<0.5.0)
+Requires-Dist: openpyxl (>=3.1.2,<3.2.0)
+Requires-Dist: passlib (>=1.7.4,<1.8.0)
+Requires-Dist: pendulum (==2.1.2) ; python_version < "3.12"
+Requires-Dist: polars (>1.8.2,<=1.25.2)
+Requires-Dist: polars-distance (>=0.4.3,<0.5.0)
+Requires-Dist: polars-ds (>=0.6.0)
+Requires-Dist: polars-expr-transformer (>0.4.7.0)
+Requires-Dist: polars-grouper (>=0.3.0,<0.4.0)
+Requires-Dist: polars_simed (>=0.3.4,<0.4.0)
+Requires-Dist: pyairbyte-flowfile (==0.20.2)
+Requires-Dist: pyarrow (>=18.0.0,<19.0.0)
+Requires-Dist: pydantic (>=2.9.2,<2.10.0)
+Requires-Dist: pyinstaller (>=6.11.0,<7.0.0)
+Requires-Dist: pytest (>=8.3.4,<9.0.0)
+Requires-Dist: python-jose (>=3.4.0,<4.0.0)
+Requires-Dist: python-multipart (>=0.0.12,<0.1.0)
+Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
+Description-Content-Type: text/markdown
+
+<h1 align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/logo.png" alt="Flowfile Logo" width="100">
+  <br>
+  Flowfile
+</h1>
+
+<p align="center">
+  <b>Main Repository</b>: <a href="https://github.com/Edwardvaneechoud/Flowfile">Edwardvaneechoud/Flowfile</a><br>
+  <b>Documentation</b>: 
+  <a href="https://edwardvaneechoud.github.io/Flowfile/">Website</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_core/README.md">Core</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_worker/README.md">Worker</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frontend/README.md">Frontend</a> - 
+  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Technical Architecture</a>
+</p>
+
+<p>
+Flowfile is a visual ETL tool and Python library suite that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, or define data flows programmatically with Python and analyze results - all with high-performance data processing.
+</p>
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/group_by_screenshot.png" alt="Flowfile Interface" width="800"/>
+</div>
+
+## ⚡ Technical Design
+
+The `Flowfile` PyPI package provides the backend services and the `flowfile_frame` Python library:
+
+- **Core (`flowfile_core`)** (FastAPI): The main ETL engine using Polars for high-performance data transformations. Typically runs on port `:63578`.
+- **Worker (`flowfile_worker`)** (FastAPI): Handles computation-intensive tasks and caching of data operations, supporting the Core service. Typically runs on port `:63579`.
+- **FlowFrame API (`flowfile_frame`)**: A Python library with a Polars-like API for defining data manipulation pipelines programmatically, which also generates an underlying ETL graph compatible with the Flowfile ecosystem.
+
+Each flow is represented as a directed acyclic graph (DAG), where nodes represent data operations and edges represent data flow between operations.
+
+For a deeper dive into the technical architecture, check out [this article](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c) on how Flowfile leverages Polars for efficient data processing.
+
+## ✨ Introducing FlowFile Frame - A Polars-Like API for ETL
+
+FlowFile Frame is a Python library that provides a familiar Polars-like API for data manipulation, while simultaneously building an ETL (Extract, Transform, Load) graph under the hood. This allows you to:
+
+1. Write data transformation code using a simple, Pandas/Polars-like API
+2. Automatically generate executable ETL workflows compatible with the Flowfile ecosystem
+3. Visualize, save, and share your data pipelines
+4. Get the performance benefits of Polars with the traceability of ETL graphs
+
+### FlowFrame Quick Start
+
+```python
+import flowfile_frame as ff
+from flowfile_frame.utils import open_graph_in_editor
+
+# Create a complex data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+open_graph_in_editor(df.flow_graph)
+
+```
+
+### Key FlowFrame Features
+
+- **Familiar API**: Based on Polars, making it easy to learn if you know Pandas or Polars
+- **ETL Graph Generation**: Automatically builds a directed acyclic graph of your data operations
+- **Lazy Evaluation**: Operations are not executed until `collect()` or a write operation
+- **Interoperability**: Saved `.flowfile` graphs can be opened in the visual Flowfile Designer
+- **High Performance**: Leverages Polars for fast data processing
+- **Reproducible**: Save and share your data transformation workflows
+
+### Common FlowFrame Operations
+
+```python
+import flowfile_frame as ff
+from flowfile_frame import col, when
+
+# Create from dictionary
+df = ff.from_dict({
+    "id": [1, 2, 3],
+    "name": ["Alice", "Bob", "Charlie"],
+    "age": [25, 35, 28]
+})
+
+flow_graph = df.flow_graph
+# Reading data
+# df_csv = ff.read_csv("data.csv")
+# df_parquet = ff.read_parquet("data.parquet")
+
+# Filtering
+adults = df.filter(col("age") >= 30)
+
+# Select and transform
+result = df.select(
+    col("name"),
+    (col("age") * 2).alias("double_age")
+)
+
+# Add new columns
+df_with_cols = df.with_columns([
+    (col("age") + 10).alias("future_age"),
+    when(col("age") >= 30).then(ff.lit("Senior")).otherwise(ff.lit("Junior")).alias("status")]
+)
+
+# Group by and aggregate
+df_sales = ff.from_dict({
+    "region": ["North", "South", "North", "South"],
+    "sales": [100, 200, 150, 300]
+})
+sales_by_region = df_sales.group_by("region").agg([
+    col("sales").sum().alias("total_sales"),
+    col("sales").mean().alias("avg_sales")
+])
+
+# Joins
+customers = ff.from_dict({"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]}, flow_graph=flow_graph)
+orders = ff.from_dict({"id": [101, 102], "customer_id": [1, 2], "amount": [100, 200]}, flow_graph=flow_graph)
+joined = customers.join(orders, left_on="id", right_on="customer_id")
+
+# Save and visualize ETL graph
+
+result.save_graph("my_pipeline.flowfile")
+# open_graph_in_editor(result.flow_graph, "my_pipeline.flowfile") # Opens in Designer UI if installed
+```
+
+For more detailed information on all available operations, including pivoting, window functions, complex workflows, and more, please refer to the [FlowFrame documentation](https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frame/README.md).
+
+## 🔥 Example Use Cases
+
+Flowfile is great for:
+
+- **Data Cleaning & Transformation**
+  - Complex joins (fuzzy matching)
+  - Text-to-rows transformations
+  - Advanced filtering and grouping
+  - Custom formulas and expressions
+  - Filter data based on conditions
+
+- **Performance**
+  - Built to scale out of core
+  - Using Polars for data processing
+
+- **Data Integration**
+  - Standardize data formats
+  - Handle messy Excel files 
+
+- **ETL Operations**
+  - Data quality checks
+
+(For more visual examples of these use cases, please see our [main GitHub repository](https://github.com/Edwardvaneechoud/Flowfile#-example-use-cases)).
+
+## 🚀 Getting Started
+
+### Installing the Flowfile Python Package
+
+This package provides the `flowfile_core` and `flowfile_worker` backend services, and the `flowfile_frame` library.
+
+```bash
+pip install Flowfile
+```
+
+Once installed, you can use `flowfile_frame` as a library in your Python scripts (see Quick Start above).
+
+### Full Application with Visual Designer
+
+For the complete visual ETL experience with the Designer UI, please see the [installation instructions in the main repository](https://github.com/Edwardvaneechoud/Flowfile#-getting-started).
+
+Available options include:
+- Desktop application (recommended for most users)
+- Docker setup (backend services + web frontend)
+- Manual setup for development
+
+## 📋 Development Roadmap
+
+For the latest development roadmap and TODO list, please refer to the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-todo).
+

{flowfile-0.2.2 → flowfile-0.3.0.1}/flowfile/__init__.py

@@ -7,9 +7,16 @@ This package ties together the FlowFile ecosystem components:
 - flowfile_worker: Computation engine
 """
 
-__version__ = "0.2.1"
+__version__ = "0.3.1"
 
-# Import the key components from flowfile_frame
+import os
+import logging
+
+os.environ['WORKER_PORT'] = "63578"
+os.environ['SINGLE_FILE_MODE'] = "1"
+
+from flowfile.web import start_server as start_web_ui
+from flowfile.api import open_graph_in_editor
 from flowfile_frame.flow_frame import (
     FlowFrame, read_csv, read_parquet, from_dict, concat
 )
@@ -18,7 +25,7 @@ from flowfile_frame.expr import (
     sum, min, max, mean, count, when
 )
 from flowfile_frame.group_frame import GroupByFrame
-from flowfile_frame.utils import create_etl_graph, open_graph_in_editor
+from flowfile_frame.utils import create_flow_graph
 from flowfile_frame.selectors import (
     numeric, float_, integer, string, temporal,
     datetime, date, time, duration, boolean,
@@ -26,7 +33,6 @@ from flowfile_frame.selectors import (
     by_dtype, contains, starts_with, ends_with, matches
 )
 
-# Import Polars data types for convenience
 from polars.datatypes import (
     Int8, Int16, Int32, Int64, Int128,
     UInt8, UInt16, UInt32, UInt64,
@@ -38,7 +44,6 @@ from polars.datatypes import (
     DataType, DataTypeClass, Field
 )
 
-# Define what's publicly available from the package
 __all__ = [
     # Core FlowFrame classes
     'FlowFrame', 'GroupByFrame',
@@ -57,7 +62,7 @@ __all__ = [
     'by_dtype', 'contains', 'starts_with', 'ends_with', 'matches',
 
     # Utilities
-    'create_etl_graph', 'open_graph_in_editor',
+    'create_flow_graph', 'open_graph_in_editor',
 
     # Data types from Polars
     'Int8', 'Int16', 'Int32', 'Int64', 'Int128',
@@ -68,4 +73,6 @@ __all__ = [
     'Date', 'Time', 'Datetime', 'Duration',
     'Categorical', 'Decimal', 'Enum', 'Unknown',
     'DataType', 'DataTypeClass', 'Field',
-]
+    'start_web_ui'
+]
+logging.getLogger("PipelineHandler").setLevel(logging.WARNING)

flowfile-0.3.0.1/flowfile/__main__.py

@@ -0,0 +1,60 @@
+#  flowfile/__main__.py
+
+def main():
+    """
+    Display information about FlowFile when run directly as a module.
+    """
+    import flowfile
+    import argparse
+
+    parser = argparse.ArgumentParser(description="FlowFile: A visual ETL tool with a Polars-like API")
+    parser.add_argument("command", nargs="?", choices=["run"], help="Command to execute")
+    parser.add_argument("component", nargs="?", choices=["web", "core", "worker"],
+                        help="Component to run (web, core, or worker)")
+    parser.add_argument("--host", default="127.0.0.1", help="Host to bind the server to")
+    parser.add_argument("--port", type=int, default=63578, help="Port to bind the server to")
+    parser.add_argument("--no-browser", action="store_true", help="Don't open a browser window")
+
+    # Parse arguments
+    args = parser.parse_args()
+
+    if args.command == "run" and args.component:
+        if args.component == "web":
+            try:
+                flowfile.start_web_ui(
+                    host=args.host,
+                    port=args.port,
+                    open_browser=not args.no_browser
+                )
+            except KeyboardInterrupt:
+                print("\nFlowFile service stopped.")
+        elif args.component == "core":
+            # Only for direct core service usage
+            from flowfile_core.main import run as run_core
+            run_core(host=args.host, port=args.port)
+        elif args.component == "worker":
+            # Only for direct worker service usage
+            from flowfile_worker.main import run as run_worker
+            run_worker(host=args.host, port=args.port)
+    else:
+        # Default action - show info
+        print(f"FlowFile v{flowfile.__version__}")
+        print("A framework combining visual ETL with a Polars-like API")
+        print("\nUsage:")
+        print("  # Start the FlowFile web UI with integrated services")
+        print("  flowfile run web")
+        print("")
+        print("  # Advanced: Run individual components")
+        print("  flowfile run core  # Start only the core service")
+        print("  flowfile run worker  # Start only the worker service")
+        print("")
+        print("  # Options")
+        print("  flowfile run web --host 0.0.0.0 --port 8080  # Custom host/port")
+        print("  flowfile run web --no-browser  # Don't open browser")
+        print("")
+        print("  # Python API usage examples")
+        print("  import flowfile as ff")
+        print("  df = ff.read_csv('data.csv')")
+        print("  result = df.filter(ff.col('value') > 10)")
+        print("  ff.open_graph_in_editor(result)")
+        print("  ff.start_web_ui()")