Flowfile 0.3.0__tar.gz → 0.3.0.2__tar.gz

flowfile-0.3.0.2/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.3
+Name: Flowfile
+Version: 0.3.0.2
+Summary: Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)
+Author: Edward van Eechoud
+Author-email: evaneechoud@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: XlsxWriter (>=3.2.0,<3.3.0)
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: airbyte-cdk (==6.47.2)
+Requires-Dist: bcrypt (>=4.3.0,<5.0.0)
+Requires-Dist: connectorx (>=0.4.2,<0.5.0)
+Requires-Dist: databases (>=0.9.0,<0.10.0)
+Requires-Dist: faker (>=23.1.0,<23.2.0)
+Requires-Dist: fastapi (>=0.115.2,<0.116.0)
+Requires-Dist: fastexcel (>=0.12.0,<0.13.0)
+Requires-Dist: google-api-python-client (>=2.149.0,<2.150.0)
+Requires-Dist: gspread (>=6.1.3,<6.2.0)
+Requires-Dist: loky (>=3.4.1,<3.5.0)
+Requires-Dist: methodtools (>=0.4.7,<0.5.0)
+Requires-Dist: openpyxl (>=3.1.2,<3.2.0)
+Requires-Dist: passlib (>=1.7.4,<1.8.0)
+Requires-Dist: pendulum (==2.1.2) ; python_version < "3.12"
+Requires-Dist: polars (>1.8.2,<=1.25.2)
+Requires-Dist: polars-distance (>=0.4.3,<0.5.0)
+Requires-Dist: polars-ds (>=0.6.0)
+Requires-Dist: polars-expr-transformer (>0.4.7.0)
+Requires-Dist: polars-grouper (>=0.3.0,<0.4.0)
+Requires-Dist: polars_simed (>=0.3.4,<0.4.0)
+Requires-Dist: pyairbyte-flowfile (==0.20.2)
+Requires-Dist: pyarrow (>=18.0.0,<19.0.0)
+Requires-Dist: pydantic (>=2.9.2,<2.10.0)
+Requires-Dist: pyinstaller (>=6.11.0,<7.0.0)
+Requires-Dist: pytest (>=8.3.4,<9.0.0)
+Requires-Dist: python-jose (>=3.4.0,<4.0.0)
+Requires-Dist: python-multipart (>=0.0.12,<0.1.0)
+Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
+Description-Content-Type: text/markdown
+
+# Flowfile
+
+![Flowfile Logo](https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/logo.png)
+
+Flowfile is a visual ETL tool and Python library suite that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, or define data flows programmatically with Python and analyze results - all with high-performance data processing.
+
+![Flowfile Interface](https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/group_by_screenshot.png)
+
+## 🚀 Getting Started
+
+### Installation
+
+Install Flowfile directly from PyPI:
+
+```bash
+pip install Flowfile
+```
+
+### Quick Start: Web UI
+
+The easiest way to get started is by launching the web-based UI:
+
+```bash
+# Start the Flowfile web UI with integrated services
+flowfile run ui
+```
+
+This will:
+- Start the combined core and worker services
+- Launch a web interface in your browser
+- Provide access to the full visual ETL capabilities
+
+**Options:**
+```bash
+# Customize host and port
+flowfile run ui --host 0.0.0.0 --port 8080
+
+# Start without opening a browser
+flowfile run ui --no-browser
+```
+
+You can also start the web UI programmatically:
+
+```python
+import flowfile
+
+# Start with default settings
+flowfile.start_web_ui()
+
+# Or customize
+flowfile.start_web_ui(host="0.0.0.0", port=8080, open_browser=False)
+```
+
+### Using the FlowFrame API
+
+Flowfile provides a Polars-like API for defining data pipelines programmatically:
+
+```python
+import flowfile as ff
+from flowfile import col, open_graph_in_editor
+
+# Create a data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+# Process the data
+result = df.filter(col("value") > 150).with_columns([
+    (col("value") * 2).alias("double_value")
+])
+
+# Open the graph in the web UI (starts the server if needed)
+open_graph_in_editor(result.flow_graph)
+```
+
+## 📦 Package Components
+
+The `Flowfile` PyPI package includes:
+
+- **Core Service (`flowfile_core`)**: The main ETL engine using Polars
+- **Worker Service (`flowfile_worker`)**: Handles computation-intensive tasks
+- **Web UI**: Browser-based visual ETL interface
+- **FlowFrame API (`flowfile_frame`)**: Polars-like API for Python coding
+
+## ✨ Key Features
+
+### Visual ETL with Web UI
+
+- **No Installation Required**: Launch directly from the pip package
+- **Drag-and-Drop Interface**: Build data pipelines visually
+- **Integrated Services**: Combined core and worker services
+- **Browser-Based**: Access from any device on your network
+
+### FlowFrame API
+
+- **Familiar Syntax**: Polars-like API makes it easy to learn
+- **ETL Graph Generation**: Automatically builds visual workflows
+- **Lazy Evaluation**: Operations are not executed until needed
+- **Interoperability**: Move between code and visual interfaces
+
+### Data Operations
+
+- **Data Cleaning & Transformation**: Complex joins, filtering, etc.
+- **High Performance**: Built on Polars for efficient processing
+- **Data Integration**: Handle various file formats
+- **ETL Pipeline Building**: Create reusable workflows
+
+## 🔄 Common FlowFrame Operations
+
+```python
+
+import flowfile as ff
+from flowfile import col, when, lit
+
+# Read data
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+# df_parquet = ff.read_parquet("data.parquet")
+# df_csv = ff.read_csv("data.csv")
+
+other_df = ff.from_dict({
+    "product_id": [1, 2, 3, 4, 6],
+    "product_name": ["WidgetA", "WidgetB", "WidgetC", "WidgetD", "WidgetE"],
+    "supplier": ["SupplierX", "SupplierY", "SupplierX", "SupplierZ", "SupplierY"]
+}, flow_graph=df.flow_graph  # Assign the data to the same graph
+)
+
+# Filter
+filtered = df.filter(col("value") > 150)
+
+# Transform
+result = df.select(
+    col("id"),
+    (col("value") * 2).alias("double_value")
+)
+
+# Conditional logic
+with_status = df.with_columns([
+    when(col("value") > 200).then(lit("High")).otherwise(lit("Low")).alias("status")
+])
+
+# Group and aggregate
+by_category = df.group_by("category").agg([
+    col("value").sum().alias("total"),
+    col("value").mean().alias("average")
+])
+
+# Join data
+joined = df.join(other_df, left_on="id", right_on="product_id")
+
+joined.flow_graph.flow_settings.execution_location = "auto"
+joined.flow_graph.flow_settings.execution_mode = "Development"
+ff.open_graph_in_editor(joined.flow_graph)  # opens the graph in the UI!
+
+```
+
+## 🧰 Command-Line Interface
+
+```bash
+# Show help and version info
+flowfile
+
+# Start the web UI
+flowfile run ui [options]
+
+# Run individual services
+flowfile run core --host 0.0.0.0 --port 8080
+flowfile run worker --host 0.0.0.0 --port 8079
+```
+
+## 📚 Resources
+
+- **[Main Repository](https://github.com/Edwardvaneechoud/Flowfile)**: Latest code and examples
+- **[Documentation](https://edwardvaneechoud.github.io/Flowfile/)**: Comprehensive guides
+- **[Technical Architecture](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c)**: Design overview
+
+## 🖥️ Full Application Options
+
+For the complete visual ETL experience, you have additional options:
+
+- **Desktop Application**: Download from the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-getting-started)
+- **Docker Setup**: Run with Docker Compose
+- **Manual Setup**: For development environments
+
+## 📋 Development Roadmap
+
+See the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-todo) for the latest development roadmap and TODO list.

{flowfile-0.3.0 → flowfile-0.3.0.2}/flowfile/__init__.py

@@ -7,9 +7,16 @@ This package ties together the FlowFile ecosystem components:
 - flowfile_worker: Computation engine
 """
 
-__version__ = "0.2.1"
+__version__ = "0.3.1"
 
-# Import the key components from flowfile_frame
+import os
+import logging
+
+os.environ['WORKER_PORT'] = "63578"
+os.environ['SINGLE_FILE_MODE'] = "1"
+
+from flowfile.web import start_server as start_web_ui
+from flowfile.api import open_graph_in_editor
 from flowfile_frame.flow_frame import (
     FlowFrame, read_csv, read_parquet, from_dict, concat
 )
@@ -18,7 +25,7 @@ from flowfile_frame.expr import (
     sum, min, max, mean, count, when
 )
 from flowfile_frame.group_frame import GroupByFrame
-from flowfile_frame.utils import create_flow_graph, open_graph_in_editor
+from flowfile_frame.utils import create_flow_graph
 from flowfile_frame.selectors import (
     numeric, float_, integer, string, temporal,
     datetime, date, time, duration, boolean,
@@ -26,7 +33,6 @@ from flowfile_frame.selectors import (
     by_dtype, contains, starts_with, ends_with, matches
 )
 
-# Import Polars data types for convenience
 from polars.datatypes import (
     Int8, Int16, Int32, Int64, Int128,
     UInt8, UInt16, UInt32, UInt64,
@@ -38,7 +44,6 @@ from polars.datatypes import (
     DataType, DataTypeClass, Field
 )
 
-# Define what's publicly available from the package
 __all__ = [
     # Core FlowFrame classes
     'FlowFrame', 'GroupByFrame',
@@ -68,4 +73,6 @@ __all__ = [
     'Date', 'Time', 'Datetime', 'Duration',
     'Categorical', 'Decimal', 'Enum', 'Unknown',
     'DataType', 'DataTypeClass', 'Field',
-]
+    'start_web_ui'
+]
+logging.getLogger("PipelineHandler").setLevel(logging.WARNING)

flowfile-0.3.0.2/flowfile/__main__.py

@@ -0,0 +1,59 @@
+#  flowfile/__main__.py
+
+def main():
+    """
+    Display information about FlowFile when run directly as a module.
+    """
+    import flowfile
+    import argparse
+
+    parser = argparse.ArgumentParser(description="FlowFile: A visual ETL tool with a Polars-like API")
+    parser.add_argument("command", nargs="?", choices=["run"], help="Command to execute")
+    parser.add_argument("component", nargs="?", choices=["web", "core", "worker"],
+                        help="Component to run (web, core, or worker)")
+    parser.add_argument("--host", default="127.0.0.1", help="Host to bind the server to")
+    parser.add_argument("--port", type=int, default=63578, help="Port to bind the server to")
+    parser.add_argument("--no-browser", action="store_true", help="Don't open a browser window")
+
+    # Parse arguments
+    args = parser.parse_args()
+
+    if args.command == "run" and args.component:
+        if args.component == "ui":
+            try:
+                flowfile.start_web_ui(
+                    host=args.host,
+                    port=args.port,
+                    open_browser=not args.no_browser
+                )
+            except KeyboardInterrupt:
+                print("\nFlowFile service stopped.")
+        elif args.component == "core":
+            # Only for direct core service usage
+            from flowfile_core.main import run as run_core
+            run_core(host=args.host, port=args.port)
+        elif args.component == "worker":
+            # Only for direct worker service usage
+            from flowfile_worker.main import run as run_worker
+            run_worker(host=args.host, port=args.port)
+    else:
+        # Default action - show info
+        print(f"FlowFile v{flowfile.__version__}")
+        print("A framework combining visual ETL with a Polars-like API")
+        print("\nUsage:")
+        print("  # Start the FlowFile web UI with integrated services")
+        print("  flowfile run ui")
+        print("")
+        print("  # Advanced: Run individual components")
+        print("  flowfile run core  # Start only the core service")
+        print("  flowfile run worker  # Start only the worker service")
+        print("")
+        print("  # Options")
+        print("  flowfile run ui --host 0.0.0.0 --port 8080  # Custom host/port")
+        print("  flowfile run ui --no-browser  # Don't open browser")
+        print("")
+        print("  # Python API usage examples")
+        print("  import flowfile as ff")
+        print("  df = ff.read_csv('data.csv')")
+        print("  result = df.filter(ff.col('value') > 10)")
+        print("  ff.open_graph_in_editor(result)")