fewsvisualiser 1.0.1__tar.gz

fewsvisualiser-1.0.1/.github/workflows/python-publish.yml

@@ -0,0 +1,40 @@
+# This workflow will upload a Python Package using Twine when a release is created
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  deploy:
+
+    runs-on: ubuntu-latest
+    # Specifying a GitHub environment is optional, but strongly encouraged
+    environment: release
+    steps:
+    - uses: actions/checkout@v6
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install build
+    - name: Build package
+      run: python -m build 
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        packages-dir: dist

fewsvisualiser-1.0.1/.gitignore

@@ -0,0 +1,19 @@
+/db/*.db3
+__pycache__
+aantekeningen.doc
+test*.ipynb
+/.vscode/launch.json
+/config
+.pixi
+.DS_Store
+db/*.log
+notebooks/*
+!notebooks/*.py
+!notebooks/talkwithfews.ipynb
+!notebooks/validate_tools.ipynb
+env/.env
+logs/fews_agent.log
+doc/TalkWithFEWS-7april-GO-FEWS.key
+hidden_*
+doc/TalkWithFEWS-7april-GO-FEWS.pdf
+dist

fewsvisualiser-1.0.1/.vscode/settings.json

@@ -0,0 +1,10 @@
+{
+    "python.defaultInterpreterPath": ".pixi/envs/default3/python.exe",
+    "python.testing.pytestArgs": [
+        "tests"
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true,
+    "search.useGlobalIgnoreFiles": true,
+    "python-envs.defaultEnvManager": "ms-python.python:system"
+}

fewsvisualiser-1.0.1/PKG-INFO

@@ -0,0 +1,54 @@
+Metadata-Version: 2.5
+Name: fewsvisualiser
+Version: 1.0.1
+Summary: FEWSVisualiser: Insights for optimized Delft-FEWS configurations.
+Author-email: Job Verkaik <verkaik@hkv.nl>
+Keywords: configuration,declarative,delft-fews,insight
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Requires-Python: >=3.8
+Requires-Dist: fire
+Description-Content-Type: text/markdown
+
+# FEWSVisualiser: Insights for optimized Delft-FEWS configurations
+
+A differerent approach to the analysis and visualisation of Delft-FEWS configuration
+
+This project is a successor of [FEWS_Netwerkvisualisation](https://github.com/HKV-products-services/FEWS_Netwerkvisualisation)
+
+HKV project
+RO1122.10.00 Herstructurering analyse netwerkvisualisatie
+
+## Talk with fews
+
+> [!CAUTION]
+> De eerste ontiwikkelingen van `talk with FEWS` waren in deze repo onder `/talkwithfews`.
+> Dit is echter losgetrokken, de actuele repo is: ![hier](https://github.com/HKV-products-services/talk-with-fews) te vinden. 
+> Actiepunt: later volledig lostrekken.
+
+## aantekeningen
+
+![Config Vis Datamodel](ConfigVisDatamodel.png)
+
+2024-02-11
+
+Zie explore. Waarom zijn er 5 call stacks met wf_DebietBerekening?
+
+2024-02-12
+
+Die call stacks zijn wsl goed. Elke module binnen een wf krijgt namelijk een eigen call stack en dat is terecht. Alleen dit deel moet even goed anders: de volgorde van het aanroepen van modules binnen en wf is van belang wanneer er temporaries worden doorgegeven naar een volgende module. Dit is een case die nu nog niet ondersteund wordt.
+
+De betere manier is om in de call stacks per wf vanuit de descriptor de volgorde van modins call vast te leggen. Temporaries kunnen dan op een andere manier door gegeven worden en hoeven niet de database in.
+
+## How to run?
+
+```bash
+pixi install
+```
+
+```bash
+pixi run analyse_config {path_to_config} {path_to_db}
+# e.g: 
+pixi run analyse_config  "..\FEWS Digiflow\Config" "db\digiflow.db3"
+```

fewsvisualiser-1.0.1/README.md

@@ -0,0 +1,41 @@
+# FEWSVisualiser: Insights for optimized Delft-FEWS configurations
+
+A differerent approach to the analysis and visualisation of Delft-FEWS configuration
+
+This project is a successor of [FEWS_Netwerkvisualisation](https://github.com/HKV-products-services/FEWS_Netwerkvisualisation)
+
+HKV project
+RO1122.10.00 Herstructurering analyse netwerkvisualisatie
+
+## Talk with fews
+
+> [!CAUTION]
+> De eerste ontiwikkelingen van `talk with FEWS` waren in deze repo onder `/talkwithfews`.
+> Dit is echter losgetrokken, de actuele repo is: ![hier](https://github.com/HKV-products-services/talk-with-fews) te vinden. 
+> Actiepunt: later volledig lostrekken.
+
+## aantekeningen
+
+![Config Vis Datamodel](ConfigVisDatamodel.png)
+
+2024-02-11
+
+Zie explore. Waarom zijn er 5 call stacks met wf_DebietBerekening?
+
+2024-02-12
+
+Die call stacks zijn wsl goed. Elke module binnen een wf krijgt namelijk een eigen call stack en dat is terecht. Alleen dit deel moet even goed anders: de volgorde van het aanroepen van modules binnen en wf is van belang wanneer er temporaries worden doorgegeven naar een volgende module. Dit is een case die nu nog niet ondersteund wordt.
+
+De betere manier is om in de call stacks per wf vanuit de descriptor de volgorde van modins call vast te leggen. Temporaries kunnen dan op een andere manier door gegeven worden en hoeven niet de database in.
+
+## How to run?
+
+```bash
+pixi install
+```
+
+```bash
+pixi run analyse_config {path_to_config} {path_to_db}
+# e.g: 
+pixi run analyse_config  "..\FEWS Digiflow\Config" "db\digiflow.db3"
+```

fewsvisualiser-1.0.1/app/components/__init__.py

@@ -0,0 +1,10 @@
+from .sidebar_chat import init_chat_state, render_chat_sidebar
+from .tab_explore import render_explore_tab
+from .tab_reports import render_reports_tab
+
+__all__ = [
+    'init_chat_state',
+    'render_chat_sidebar',
+    'render_explore_tab',
+    'render_reports_tab'
+] 

fewsvisualiser-1.0.1/app/components/sidebar_chat.py

@@ -0,0 +1,327 @@
+import sys
+from pathlib import Path
+# Add parent directory to path for imports
+sys.path.append(str(Path(__file__).parent.parent.parent))
+
+import streamlit as st
+from streamlit_markdown import st_markdown
+from talkwithfews.agent import FEWSConfigAgent
+import asyncio
+from queue import Queue, Empty
+import threading
+import time
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+class AgentManager:
+    """Manages the FEWS agent in a separate thread"""
+    def __init__(self, config):
+        self.config = config
+        self.agent = None
+        self.response_queue = Queue()
+        self.command_queue = Queue()
+        self.thread = None
+        self.ready = threading.Event()
+        self.processing = False
+        self.error = None
+
+    def initialize(self):
+        """Initialize the agent in a dedicated thread"""
+        def agent_thread():
+            try:
+                # Create and initialize agent
+                self.agent = FEWSConfigAgent(
+                    db_path=self.config["db_path"],
+                    log_dir=self.config["log_dir"]
+                )
+                self.agent.connect()
+                
+                # Set up event loop
+                loop = asyncio.new_event_loop()
+                asyncio.set_event_loop(loop)
+                
+                # Signal ready
+                self.ready.set()
+                
+                while True:
+                    try:
+                        command = self.command_queue.get(timeout=1)
+                        if command is None:  # Shutdown signal
+                            break
+                            
+                        self.processing = True
+                        self.error = None
+                        
+                        try:
+                            response = loop.run_until_complete(asyncio.wait_for(
+                                self.agent.ask(command),
+                                timeout=300
+                            ))
+                            self.response_queue.put(response)
+                        except asyncio.TimeoutError:
+                            self.error = "Request timed out after 5 minutes"
+                            self.response_queue.put(self.error)
+                        except Exception as e:
+                            self.error = str(e)
+                            self.response_queue.put(f"Error: {str(e)}")
+                        finally:
+                            self.processing = False
+                            
+                    except Empty:
+                        continue
+                    except Exception as e:
+                        self.error = str(e)
+                        self.response_queue.put(f"Error: {str(e)}")
+                        self.processing = False
+                        
+            except Exception as e:
+                self.error = str(e)
+                self.response_queue.put(f"Error: {str(e)}")
+            finally:
+                try:
+                    loop.close()
+                except:
+                    pass
+                
+        self.thread = threading.Thread(target=agent_thread, daemon=True)
+        self.thread.start()
+        if not self.ready.wait(timeout=10):
+            raise Exception("Agent initialization timed out")
+
+    def ask(self, prompt):
+        """Send a question to the agent thread"""
+        if not self.thread or not self.thread.is_alive():
+            raise Exception("Agent thread is not running")
+            
+        self.command_queue.put(prompt)
+        start_time = time.time()
+        
+        while time.time() - start_time < 300:  # 5 minutes timeout
+            try:
+                # Wait for response with a short timeout
+                response = self.response_queue.get(timeout=1)
+                if isinstance(response, str) and response.startswith("Error:"):
+                    raise Exception(response)
+                return response
+            except Empty:
+                # If we're still processing, keep waiting
+                if self.processing:
+                    continue
+                # If we have an error, raise it
+                if self.error:
+                    raise Exception(self.error)
+                # Otherwise, keep waiting
+                continue
+                
+        raise Exception("Timeout waiting for response")
+
+    def close(self):
+        """Close the agent and cleanup"""
+        if self.thread and self.thread.is_alive():
+            self.command_queue.put(None)
+            self.thread.join(timeout=5)
+        self.agent = None
+        self.thread = None
+        self.ready.clear()
+        self.processing = False
+        self.error = None
+
+def init_chat_state():
+    """Initialize chat state and agent if not exists"""
+    if "messages" not in st.session_state:
+        st.session_state.messages = []
+    if "agent_manager" not in st.session_state:
+        st.session_state.agent_manager = None
+    if "selected_data" not in st.session_state:
+        st.session_state.selected_data = None
+
+def format_agent_response(response):
+    """Format the agent response for display"""
+    if hasattr(response, 'data'):
+        content = {
+            "text": response.data.markdown_text or "",
+            "table": response.data.markdown_table or "",
+            "diagram": response.data.mermaid_diagram or ""
+        }
+    else:
+        content = {"text": str(response), "table": "", "diagram": ""}
+    return content
+
+def add_to_chat(data: dict):
+    """
+    Add FEWS object data to the chat as a question and get agent response.
+    
+    Args:
+        data: Dictionary containing FEWS object information with at least a 'type' field
+            Supported types:
+            - workflow: Contains Workflow ID, Beschrijving, Module Config IDs
+            - timeseries: Contains ID, Module, Parameter info for timeseries
+            Future support planned for:
+            - transformation: Contains transformation details
+    """
+    if "messages" not in st.session_state:
+        st.session_state.messages = []
+
+    # Format question based on object type
+    if data.get("type") == "timeseries":
+        # For timeseries sets
+        ts_id = data.get("ID")
+        module = data.get("Module", "")
+        parameter = data.get("Parameter", "")
+        question = (
+            f"Kun je me meer vertellen over de tijdreeks met ID {ts_id}? "
+            f"Deze tijdreeks is van parameter '{parameter}' in module '{module}'. "
+            f"Wat zijn de producerende transformaties en waar wordt deze tijdreeks voor gebruikt?"
+            f"Met andere woorden, waarom is dit een dood punt in de configuratie?"            
+        )
+    elif data.get("type") == "workflow":
+        # For workflows (renamed from the default case)
+        workflow_id = data.get("Workflow ID")
+        if not workflow_id:
+            st.error("Geen workflow ID gevonden in de data")
+            return
+        
+        # Format the workflow data into a clear question
+        question = (
+            f"Kun je me meer vertellen over de workflow met ID {workflow_id}? "
+            f"Deze workflow heeft als beschrijving: '{data.get('Beschrijving', 'Geen beschrijving')}' "
+            f"en gebruikt de volgende Module Config IDs: {data.get('Module Config IDs', [])}"
+        )
+    else:
+        st.error(f"Niet-ondersteund object type: {data.get('type', 'onbekend')}")
+        return
+
+    # Add the question to the chat history
+    st.session_state.messages.append({
+        "role": "user",
+        "content": question
+    })
+    
+    # Store the question in session state to process after rerun
+    if "pending_question" not in st.session_state:
+        st.session_state.pending_question = None
+    st.session_state.pending_question = question
+    
+    # Trigger a rerun to update the UI with the question
+    st.rerun()
+
+def show_table_dialog(table_content: str, title: str = "Table Details"):
+    """Display a table in a popover."""
+    with st.popover(title):
+        st.markdown(table_content)
+        if st.button("Close", use_container_width=True):
+            st.rerun()
+
+def format_mermaid_diagram(diagram_content: str) -> str:
+    """Format the diagram content to ensure proper mermaid syntax."""
+    if not diagram_content.startswith("```mermaid"):
+        diagram_content = "```mermaid\n" + diagram_content
+    if not diagram_content.endswith("```"):
+        diagram_content = diagram_content + "\n```"
+    return diagram_content
+
+def render_chat_sidebar(config=None):
+    """Render the chat interface in the sidebar"""
+    # Add custom CSS for the diagram container
+    st.markdown("""
+        <style>
+        .diagram-container {
+            min-height: 400px;
+            max-height: 400px;
+            overflow-y: auto;
+            border: 1px solid #e0e0e0;
+            border-radius: 4px;
+            padding: 10px;
+        }
+        </style>
+    """, unsafe_allow_html=True)
+    
+    with st.sidebar:
+        st.header("Chat Interface")
+        
+        # Initialize agent if config provided
+        if config and st.session_state.agent_manager is None:
+            try:
+                st.session_state.agent_manager = AgentManager(config)
+                st.session_state.agent_manager.initialize()
+            except Exception as e:
+                st.error(f"Error initializing FEWS agent: {str(e)}")
+                return
+        
+        # Display chat messages
+        for idx, message in enumerate(st.session_state.messages):
+            with st.chat_message(message["role"]):
+                content = message["content"]
+                
+                # Handle both string and dictionary content
+                if isinstance(content, str):
+                    st.markdown(content)
+                else:
+                    # Display text
+                    if content.get("text"):
+                        st.markdown(content["text"])
+                    
+                    # Show table and diagram popovers if available
+                    if content.get("table") or content.get("diagram"):
+                        cols = st.columns(2)
+                        
+                        # Table popover in first column if available
+                        if content.get("table"):
+                            with cols[0]:
+                                with st.popover("📊 Table Details"):
+                                    st.markdown(content["table"])
+                        
+                        # Diagram popover in second column if available
+                        if content.get("diagram"):
+                            with cols[1]:
+                                with st.popover("📈 Diagram Details"):
+                                    # Format and display the mermaid diagram
+                                    formatted_diagram = format_mermaid_diagram(content["diagram"])
+                                    st_markdown(formatted_diagram, height=380)
+        
+        # Process pending question if exists
+        if hasattr(st.session_state, 'pending_question') and st.session_state.pending_question:
+            question = st.session_state.pending_question
+            st.session_state.pending_question = None  # Clear the pending question
+            
+            # Get agent response if available
+            if st.session_state.agent_manager:
+                with st.spinner("Thinking..."):
+                    try:
+                        response = st.session_state.agent_manager.ask(question)
+                        content = format_agent_response(response)
+                        st.session_state.messages.append({
+                            "role": "assistant",
+                            "content": content
+                        })
+                    except Exception as e:
+                        st.error(str(e))
+                        st.session_state.messages.append({
+                            "role": "assistant",
+                            "content": {"text": f"Error: {str(e)}"}
+                        })
+                st.rerun()  # Update UI with the response
+        
+        # Chat input
+        if prompt := st.chat_input("Ask about FEWS configuration..."):
+            # Add user message
+            st.session_state.messages.append({
+                "role": "user",
+                "content": prompt
+            })
+            
+            # Store as pending question
+            st.session_state.pending_question = prompt
+            st.rerun()
+        
+        # Cleanup button
+        if st.button("End Session", use_container_width=True):
+            if st.session_state.agent_manager:
+                st.session_state.agent_manager.close()
+            st.session_state.agent_manager = None
+            st.session_state.messages = []
+            st.session_state.pending_question = None
+            st.rerun() 

fewsvisualiser-1.0.1/app/components/tab_explore.py

@@ -0,0 +1,91 @@
+import streamlit as st
+import pandas as pd
+from datetime import datetime
+
+from .sidebar_chat import add_to_chat
+
+def load_workflow_data():
+    """Load workflow data using the search_workflows function"""
+    try:
+        # Check if tools interface is available
+        if "tools_interface" not in st.session_state:
+            st.error("Database connection not available. Please check your configuration.")
+            return pd.DataFrame()
+            
+        # Use the tools interface from session state
+        tools = st.session_state.tools_interface
+        
+        # Use '%' as default pattern to get all workflows
+        results = tools.search_workflows(name_pattern='%')
+        
+        if results and hasattr(results, 'results'):
+            data = []
+            for workflow in results.results:
+                data.append({
+                    "ID": workflow.id,  # Hidden column
+                    "Workflow ID": workflow.wf_id,
+                    "Beschrijving": workflow.desc,
+                    "Module Config IDs": ', '.join(workflow.module_config_ids) if workflow.module_config_ids else 'Geen'
+                })
+            return pd.DataFrame(data)
+        return pd.DataFrame()  # Return empty DataFrame if no results
+    except Exception as e:
+        st.error(f"Error loading workflows: {str(e)}")
+        return pd.DataFrame()
+
+def render_explore_tab():
+    """Render the explore tab content with workflow data"""
+    st.header("Explore Workflows")
+    
+    # Load workflow data
+    df = load_workflow_data()
+    
+    if df.empty:
+        st.warning("No workflows found or error loading workflows.")
+        return
+    
+    # Define which columns to display
+    display_columns = ["Workflow ID", "Beschrijving", "Module Config IDs"]
+    
+    # Add search functionality
+    search_term = st.text_input("Search workflows", "")
+    
+    # Filter dataframe based on search term
+    if search_term:
+        # Search in all columns, including hidden ones
+        mask = df.astype(str).apply(lambda x: x.str.contains(search_term, case=False)).any(axis=1)
+        filtered_df = df[mask]
+    else:
+        filtered_df = df
+
+    # Add header row
+    with st.container():
+        header_cols = st.columns([0.15] + [1] * len(display_columns))
+        header_cols[0].markdown("")
+        for i, col_name in enumerate(display_columns):
+            header_cols[i+1].markdown(f"**{col_name}**")
+        st.divider()        
+    
+    # Create a container for each row with hover effect
+    for idx, row in filtered_df.iterrows():
+        with st.container():
+            # Create columns only for displayed columns plus the chat button
+            cols = st.columns([0.15] + [1] * len(display_columns))
+            
+            # Add to chat button
+            if cols[0].button("💬", key=f"btn_{idx}", use_container_width=True):
+                # Add the workflow type to the dictionary
+                workflow_data = row.to_dict()  # This includes all columns, even hidden ones
+                workflow_data["type"] = "workflow"
+                add_to_chat(workflow_data)
+            
+            # Display only the visible columns
+            for i, col_name in enumerate(display_columns, 1):
+                cols[i].write(row[col_name])
+            
+            # Add a subtle divider
+            st.divider()
+    
+    # Show result count
+    st.markdown(f"**{len(filtered_df)} workflows found**" + 
+               (f" (filtered from {len(df)} total)" if search_term else ""))