Flowfile 0.3.0.1__tar.gz → 0.3.0.3__tar.gz

flowfile-0.3.0.3/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.3
+Name: Flowfile
+Version: 0.3.0.3
+Summary: Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)
+Author: Edward van Eechoud
+Author-email: evaneechoud@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: XlsxWriter (>=3.2.0,<3.3.0)
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: airbyte-cdk (==6.47.2)
+Requires-Dist: bcrypt (>=4.3.0,<5.0.0)
+Requires-Dist: connectorx (>=0.4.2,<0.5.0)
+Requires-Dist: databases (>=0.9.0,<0.10.0)
+Requires-Dist: faker (>=23.1.0,<23.2.0)
+Requires-Dist: fastapi (>=0.115.2,<0.116.0)
+Requires-Dist: fastexcel (>=0.12.0,<0.13.0)
+Requires-Dist: google-api-python-client (>=2.149.0,<2.150.0)
+Requires-Dist: gspread (>=6.1.3,<6.2.0)
+Requires-Dist: loky (>=3.4.1,<3.5.0)
+Requires-Dist: methodtools (>=0.4.7,<0.5.0)
+Requires-Dist: openpyxl (>=3.1.2,<3.2.0)
+Requires-Dist: passlib (>=1.7.4,<1.8.0)
+Requires-Dist: pendulum (==2.1.2) ; python_version < "3.12"
+Requires-Dist: polars (>1.8.2,<=1.25.2)
+Requires-Dist: polars-distance (>=0.4.3,<0.5.0)
+Requires-Dist: polars-ds (>=0.6.0)
+Requires-Dist: polars-expr-transformer (>0.4.7.0)
+Requires-Dist: polars-grouper (>=0.3.0,<0.4.0)
+Requires-Dist: polars_simed (>=0.3.4,<0.4.0)
+Requires-Dist: pyairbyte-flowfile (==0.20.2)
+Requires-Dist: pyarrow (>=18.0.0,<19.0.0)
+Requires-Dist: pydantic (>=2.9.2,<2.10.0)
+Requires-Dist: pyinstaller (>=6.11.0,<7.0.0)
+Requires-Dist: pytest (>=8.3.4,<9.0.0)
+Requires-Dist: python-jose (>=3.4.0,<4.0.0)
+Requires-Dist: python-multipart (>=0.0.12,<0.1.0)
+Requires-Dist: uvicorn (>=0.32.0,<0.33.0)
+Description-Content-Type: text/markdown
+
+<h1 align="center">
+  <img src="https://raw.githubusercontent.com/Edwardvaneechoud/Flowfile/main/.github/images/logo.png" alt="Flowfile Logo" width="100">
+  <br>
+  Flowfile
+</h1>
+
+<p align="center">
+  <b>Main Repository</b>: <a href="https://github.com/Edwardvaneechoud/Flowfile">Edwardvaneechoud/Flowfile</a><br>
+  <b>Documentation</b>: 
+  <a href="https://edwardvaneechoud.github.io/Flowfile/">Website</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_core/README.md">Core</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_worker/README.md">Worker</a> - 
+  <a href="https://github.com/Edwardvaneechoud/Flowfile/blob/main/flowfile_frontend/README.md">Frontend</a> - 
+  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Technical Architecture</a>
+</p>
+
+<p>
+Flowfile is a visual ETL tool and Python library suite that combines drag-and-drop workflow building with the speed of Polars dataframes. Build data pipelines visually, transform data using powerful nodes, or define data flows programmatically with Python and analyze results - all with high-performance data processing.
+</p>
+
+## 🚀 Getting Started
+
+### Installation
+
+Install Flowfile directly from PyPI:
+
+```bash
+pip install Flowfile
+```
+
+### Quick Start: Web UI
+
+The easiest way to get started is by launching the web-based UI:
+
+```bash
+# Start the Flowfile web UI with integrated services
+flowfile run ui
+```
+
+This will:
+- Start the combined core and worker services
+- Launch a web interface in your browser
+- Provide access to the full visual ETL capabilities
+
+**Options:**
+```bash
+# Customize host and port
+flowfile run ui --host 0.0.0.0 --port 8080
+
+# Start without opening a browser
+flowfile run ui --no-browser
+```
+
+You can also start the web UI programmatically:
+
+```python
+import flowfile
+
+# Start with default settings
+flowfile.start_web_ui()
+
+# Or customize
+flowfile.start_web_ui(host="0.0.0.0", port=8080, open_browser=False)
+```
+
+### Using the FlowFrame API
+
+Flowfile provides a Polars-like API for defining data pipelines programmatically:
+
+```python
+import flowfile as ff
+from flowfile import col, open_graph_in_editor
+
+# Create a data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+# Process the data
+result = df.filter(col("value") > 150).with_columns([
+    (col("value") * 2).alias("double_value")
+])
+
+# Open the graph in the web UI (starts the server if needed)
+open_graph_in_editor(result.flow_graph)
+```
+
+## 📦 Package Components
+
+The `Flowfile` PyPI package includes:
+
+- **Core Service (`flowfile_core`)**: The main ETL engine using Polars
+- **Worker Service (`flowfile_worker`)**: Handles computation-intensive tasks
+- **Web UI**: Browser-based visual ETL interface
+- **FlowFrame API (`flowfile_frame`)**: Polars-like API for Python coding
+
+## ✨ Key Features
+
+### Visual ETL with Web UI
+
+- **No Installation Required**: Launch directly from the pip package
+- **Drag-and-Drop Interface**: Build data pipelines visually
+- **Integrated Services**: Combined core and worker services
+- **Browser-Based**: Access from any device on your network
+
+### FlowFrame API
+
+- **Familiar Syntax**: Polars-like API makes it easy to learn
+- **ETL Graph Generation**: Automatically builds visual workflows
+- **Lazy Evaluation**: Operations are not executed until needed
+- **Interoperability**: Move between code and visual interfaces
+
+### Data Operations
+
+- **Data Cleaning & Transformation**: Complex joins, filtering, etc.
+- **High Performance**: Built on Polars for efficient processing
+- **Data Integration**: Handle various file formats
+- **ETL Pipeline Building**: Create reusable workflows
+
+## 🔄 Common FlowFrame Operations
+
+```python
+
+import flowfile as ff
+from flowfile import col, when, lit
+
+# Read data
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+# df_parquet = ff.read_parquet("data.parquet")
+# df_csv = ff.read_csv("data.csv")
+
+other_df = ff.from_dict({
+    "product_id": [1, 2, 3, 4, 6],
+    "product_name": ["WidgetA", "WidgetB", "WidgetC", "WidgetD", "WidgetE"],
+    "supplier": ["SupplierX", "SupplierY", "SupplierX", "SupplierZ", "SupplierY"]
+}, flow_graph=df.flow_graph  # Assign the data to the same graph
+)
+
+# Filter
+filtered = df.filter(col("value") > 150)
+
+# Transform
+result = df.select(
+    col("id"),
+    (col("value") * 2).alias("double_value")
+)
+
+# Conditional logic
+with_status = df.with_columns([
+    when(col("value") > 200).then(lit("High")).otherwise(lit("Low")).alias("status")
+])
+
+# Group and aggregate
+by_category = df.group_by("category").agg([
+    col("value").sum().alias("total"),
+    col("value").mean().alias("average")
+])
+
+# Join data
+joined = df.join(other_df, left_on="id", right_on="product_id")
+
+joined.flow_graph.flow_settings.execution_location = "auto"
+joined.flow_graph.flow_settings.execution_mode = "Development"
+ff.open_graph_in_editor(joined.flow_graph)  # opens the graph in the UI!
+
+```
+
+## 🧰 Command-Line Interface
+
+```bash
+# Show help and version info
+flowfile
+
+# Start the web UI
+flowfile run ui [options]
+
+# Run individual services
+flowfile run core --host 0.0.0.0 --port 8080
+flowfile run worker --host 0.0.0.0 --port 8079
+```
+
+## 📚 Resources
+
+- **[Main Repository](https://github.com/Edwardvaneechoud/Flowfile)**: Latest code and examples
+- **[Documentation](https://edwardvaneechoud.github.io/Flowfile/)**: Comprehensive guides
+- **[Technical Architecture](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c)**: Design overview
+
+## 🖥️ Full Application Options
+
+For the complete visual ETL experience, you have additional options:
+
+- **Desktop Application**: Download from the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-getting-started)
+- **Docker Setup**: Run with Docker Compose
+- **Manual Setup**: For development environments
+
+## 📋 Development Roadmap
+
+See the [main repository](https://github.com/Edwardvaneechoud/Flowfile#-todo) for the latest development roadmap and TODO list.

{flowfile-0.3.0.1 → flowfile-0.3.0.3}/flowfile/__main__.py

@@ -9,8 +9,8 @@ def main():
 
     parser = argparse.ArgumentParser(description="FlowFile: A visual ETL tool with a Polars-like API")
     parser.add_argument("command", nargs="?", choices=["run"], help="Command to execute")
-    parser.add_argument("component", nargs="?", choices=["web", "core", "worker"],
-                        help="Component to run (web, core, or worker)")
+    parser.add_argument("component", nargs="?", choices=["ui", "core", "worker"],
+                        help="Component to run (ui, core, or worker)")
     parser.add_argument("--host", default="127.0.0.1", help="Host to bind the server to")
     parser.add_argument("--port", type=int, default=63578, help="Port to bind the server to")
     parser.add_argument("--no-browser", action="store_true", help="Don't open a browser window")
@@ -19,7 +19,7 @@ def main():
     args = parser.parse_args()
 
     if args.command == "run" and args.component:
-        if args.component == "web":
+        if args.component == "ui":
             try:
                 flowfile.start_web_ui(
                     host=args.host,
@@ -42,19 +42,18 @@ def main():
         print("A framework combining visual ETL with a Polars-like API")
         print("\nUsage:")
         print("  # Start the FlowFile web UI with integrated services")
-        print("  flowfile run web")
+        print("  flowfile run ui")
         print("")
         print("  # Advanced: Run individual components")
         print("  flowfile run core  # Start only the core service")
         print("  flowfile run worker  # Start only the worker service")
         print("")
         print("  # Options")
-        print("  flowfile run web --host 0.0.0.0 --port 8080  # Custom host/port")
-        print("  flowfile run web --no-browser  # Don't open browser")
+        print("  flowfile run ui --host 0.0.0.0 --port 8080  # Custom host/port")
+        print("  flowfile run ui --no-browser  # Don't open browser")
         print("")
         print("  # Python API usage examples")
         print("  import flowfile as ff")
         print("  df = ff.read_csv('data.csv')")
         print("  result = df.filter(ff.col('value') > 10)")
-        print("  ff.open_graph_in_editor(result)")
-        print("  ff.start_web_ui()")
+        print("  ff.open_graph_in_editor(result)")

{flowfile-0.3.0.1 → flowfile-0.3.0.3}/flowfile/api.py

@@ -119,7 +119,7 @@ def build_server_command(module_name: str) -> List[str]:
                 "run",
                 module_name,
                 "run",
-                "web",
+                "ui",
                 "--no-browser",
                 f"--port={FLOWFILE_PORT}",
             ]
@@ -134,7 +134,7 @@ def build_server_command(module_name: str) -> List[str]:
         "-m",
         module_name,
         "run",
-        "web",
+        "ui",
         "--no-browser",
         f"--port={FLOWFILE_PORT}",
     ]
@@ -314,7 +314,7 @@ def _save_flow_to_location(
 def _open_flow_in_browser(flow_id: int) -> None:
     """Opens the specified flow ID in a browser tab if in unified mode."""
     if os.environ.get("FLOWFILE_MODE") == "electron":
-        flow_url = f"http://{FLOWFILE_HOST}:{FLOWFILE_PORT}/web/flow/{flow_id}"
+        flow_url = f"http://{FLOWFILE_HOST}:{FLOWFILE_PORT}/ui/flow/{flow_id}"
         logger.info(f"Unified mode detected. Opening imported flow in browser: {flow_url}")
         try:
             time.sleep(0.5)
@@ -335,7 +335,8 @@ def _cleanup_temporary_storage(temp_dir_obj: Optional[TemporaryDirectory]) -> No
             logger.error(f"Error cleaning up temporary directory {temp_dir_obj.name}: {e}")
 
 
-def open_graph_in_editor(flow_graph: FlowGraph, storage_location: Optional[str] = None, module_name: str = DEFAULT_MODULE_NAME) -> bool:
+def open_graph_in_editor(flow_graph: FlowGraph, storage_location: Optional[str] = None,
+                         module_name: str = DEFAULT_MODULE_NAME) -> bool:
     """
     Save the ETL graph, ensure the Flowfile server is running (starting it
     if necessary), import the graph via API, and open it in a new browser
@@ -353,10 +354,13 @@ def open_graph_in_editor(flow_graph: FlowGraph, storage_location: Optional[str]
     """
     temp_dir_obj: Optional[TemporaryDirectory] = None
     try:
+        original_execution_settings = flow_graph.flow_settings.model_copy()
+        flow_graph.flow_settings.execution_location = "auto"
+        flow_graph.flow_settings.execution_mode = "Development"
         flow_file_path, temp_dir_obj = _save_flow_to_location(flow_graph, storage_location)
         if not flow_file_path:
             return False
-
+        flow_graph.flow_settings = original_execution_settings
         if not start_flowfile_server_process(module_name):
             return False
 

flowfile-0.3.0.3/flowfile/readme.md

@@ -0,0 +1,130 @@
+# Flowfile Web UI Documentation
+
+## Overview
+
+Flowfile now supports a web-based user interface that can be launched directly from the pip-installed package. This enhancement allows users to quickly get started with the visual ETL tool without needing to install the desktop application, set up Docker, or manually configure the services.
+
+## Key Features
+
+- **Integrated Web UI**: Launch the Flowfile interface directly in your browser
+- **Unified Service**: Combined API that serves both the web UI and processes worker operations
+- **Easy Installation**: Simple pip installation and startup process
+- **Visual ETL**: Access to all the visual ETL capabilities through a web interface
+
+## Installation
+
+Install Flowfile from PyPI using pip:
+
+```bash
+pip install Flowfile
+```
+
+## Starting the Web UI
+
+You can start the Flowfile web UI using either the Python module or the command-line interface:
+
+### Using the Command-Line Interface
+
+```bash
+# Start the web UI with default settings
+flowfile run ui
+
+# Customize host and port
+flowfile run ui --host 0.0.0.0 --port 8080
+
+# Start without automatically opening a browser window
+flowfile run ui --no-browser
+```
+
+### Using Python
+
+```python
+import flowfile
+
+# Start the web UI with default settings
+flowfile.start_web_ui()
+
+# Customize host, port, and browser launch
+flowfile.start_web_ui(host="0.0.0.0", port=8080, open_browser=False)
+```
+
+## Architecture Overview
+
+The web UI functionality combines multiple components:
+
+1. **Core Service**: The main ETL engine (flowfile_core) that processes data transformations
+2. **Worker Service**: Handles computation and caching of data operations (flowfile_worker)
+3. **Web UI**: A Vue.js frontend that provides the visual interface
+
+When you start the web UI, all these services are launched together in a unified mode, making it simple to get started without configuration.
+
+## Using the Web UI with FlowFrame API
+
+You can create data pipelines programmatically with the FlowFrame API and then visualize them in the web UI:
+
+```python
+import flowfile as ff
+from flowfile import open_graph_in_editor
+
+# Create a data pipeline
+df = ff.from_dict({
+    "id": [1, 2, 3, 4, 5],
+    "category": ["A", "B", "A", "C", "B"],
+    "value": [100, 200, 150, 300, 250]
+})
+
+# Process the data
+result = df.filter(ff.col("value") > 150).with_columns([
+    (ff.col("value") * 2).alias("double_value")
+])
+
+# Open the graph in the web UI (starts the server if it's not running)
+open_graph_in_editor(result.flow_graph)
+```
+
+The `open_graph_in_editor` function automatically:
+1. Saves the flow graph to a temporary file
+2. Starts the Flowfile server if it's not already running
+3. Imports the flow into the editor
+4. Opens a browser tab with the imported flow
+
+## Advanced Server Configuration
+
+For advanced users who need to customize the server behavior:
+
+### Environment Variables
+
+- `FLOWFILE_HOST`: Host to bind the server to (default: "127.0.0.1")
+- `FLOWFILE_PORT`: Port to bind the server to (default: 63578)
+- `FLOWFILE_MODE`: Set to "electron" to enable browser auto-opening behavior
+- `WORKER_URL`: URL for the worker service
+- `SINGLE_FILE_MODE`: Set to "1" to run in unified mode with worker functionality
+- `FLOWFILE_MODULE_NAME`: Module name to run (default: "flowfile")
+
+### Running Individual Components
+
+For development or specialized deployments, you can run the components separately:
+
+```bash
+# Run only the core service
+flowfile run core --host 0.0.0.0 --port 8080
+
+# Run only the worker service
+flowfile run worker --host 0.0.0.0 --port 8081
+```
+
+## Troubleshooting
+
+- If the web UI doesn't open automatically, manually navigate to http://localhost:63578/ui
+- If you encounter connection issues, check if the port is already in use
+- Look for server logs in the terminal where you started the service for error messages
+- For issues with the API, navigate to http://localhost:63578/docs to verify the API is running
+
+## Next Steps
+
+Once you're familiar with the web UI, you might want to explore:
+
+1. The desktop application for a more native experience
+2. Docker deployment for production environments
+3. Advanced ETL operations using the FlowFrame API
+4. Custom node development for specialized transformations

{flowfile-0.3.0.1 → flowfile-0.3.0.3}/flowfile/web/__init__.py

@@ -54,7 +54,7 @@ def extend_app(app: FastAPI):
         from flowfile_core.configs.settings import WORKER_URL
         return WORKER_URL
 
-    @app.get("/web", include_in_schema=False)
+    @app.get("/ui", include_in_schema=False)
     async def web_ui_root():
         """Serve the main index.html file for the web UI"""
         index_path = static_dir / "index.html"
@@ -62,7 +62,7 @@ def extend_app(app: FastAPI):
             return FileResponse(index_path)
         return {"error": "Web UI not installed. Build the frontend and install it in the package."}
 
-    @app.get("/web/{path:path}", include_in_schema=False)
+    @app.get("/ui/{path:path}", include_in_schema=False)
     async def serve_vue_app(path: str):
         """Serve static files or the index.html for client-side routing"""
         # Try to serve the requested file
@@ -72,7 +72,7 @@ def extend_app(app: FastAPI):
 
         # If it's a directory, redirect to add trailing slash
         if (static_dir / path).exists() and (static_dir / path).is_dir():
-            return RedirectResponse(f"/web/{path}/")
+            return RedirectResponse(f"/ui/{path}/")
 
         # For client-side routing, serve the index.html
         index_path = static_dir / "index.html"
@@ -143,11 +143,11 @@ def start_server(host="127.0.0.1", port=63578, open_browser=True):
     # Open browser if requested
     if open_browser:
         time.sleep(2)
-        webbrowser.open_new_tab(f"http://{host}:{port}/web")
+        webbrowser.open_new_tab(f"http://{host}:{port}/ui")
 
     print("\n" + "=" * 60)
     print("    FlowFile - Visual ETL Tool (Unified Mode)")
-    print(f"    Web UI: http://{host}:{port}/web")
+    print(f"    Web UI: http://{host}:{port}/ui")
     print(f"    API Docs: http://{host}:{port}/docs")
     print("=" * 60 + "\n")
 

{flowfile-0.3.0.1 → flowfile-0.3.0.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "Flowfile"
-version = "0.3.0.1"
+version = "0.3.0.3"
 description = "Project combining flowfile core (backend) and flowfile_worker (compute offloader) and flowfile_frame (api)"
 readme = "readme-pypi.md"
 authors = ["Edward van Eechoud <evaneechoud@gmail.com>"]