ws-bom-robot-app 0.0.78__tar.gz → 0.0.80__tar.gz

{ws_bom_robot_app-0.0.78/ws_bom_robot_app.egg-info → ws_bom_robot_app-0.0.80}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ws_bom_robot_app
-Version: 0.0.78
+Version: 0.0.80
 Summary: A FastAPI application serving ws bom/robot/llm platform ai.
 Home-page: https://github.com/websolutespa/bom
 Author: Websolute Spa
@@ -83,18 +83,30 @@ from ws_bom_robot_app import main
 app = main.app
 ```
 
-FIll `.env` with the following code:
-
-```env
-#robot_env=local/development/production
-robot_env=local
-robot_user='[user]'
-robot_password='[pwd]'
-robot_data_folder='./.data'
-robot_cms_auth='[auth]'
-robot_cms_host='https://[DOMAIN]'
-robot_cms_db_folder=llmVectorDb
-robot_cms_files_folder=llmKbFile
+Create a `.env` file in the root directory with the following configuration:
+
+```properties
+# robot configuration
+robot_env=development
+robot_user=your_username
+USER_AGENT=ws-bom-robot-app
+
+# cms (bowl) configuration
+robot_cms_host='http://localhost:4000'
+robot_cms_auth='users API-Key your-api-key-here'
+
+# llm providers: fill one or more of these with your API keys
+DEEPSEEK_API_KEY="your-deepseek-api-key"
+OPENAI_API_KEY="your-openai-api-key"
+GOOGLE_API_KEY="your-google-api-key"
+ANTHROPIC_API_KEY="your-anthropic-api-key"
+GROQ_API_KEY="your-groq-api-key"
+# ibm
+WATSONX_URL="https://eu-gb.ml.cloud.ibm.com"
+WATSONX_APIKEY="your-watsonx-api-key"
+WATSONX_PROJECTID="your-watsonx-project-id"
+# gvertex: ensure to mount the file in docker
+GOOGLE_APPLICATION_CREDENTIALS="./.data/secrets/google-credentials.json" 
 ```
 
 ## 🚀 Run the app
@@ -120,15 +132,52 @@ robot_cms_files_folder=llmKbFile
   #gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app --bind
   ```
 
-### 🔖 Windows requirements  
+## 📖 API documentation
+
+- [swagger](http://localhost:6001/docs)
+- [redoc](http://localhost:6001/redoc)
+
+---
+
+## 🐳 Docker
+
+dockerize base image
+
+```pwsh
+<# cpu #>
+docker build -f Dockerfile-robot-base-cpu -t ghcr.io/websolutespa/ws-bom-robot-base:cpu .
+docker push ghcr.io/websolutespa/ws-bom-robot-base:cpu
+<# gpu #>
+docker build -f Dockerfile-robot-base-gpu -t ghcr.io/websolutespa/ws-bom-robot-base:gpu .
+docker push ghcr.io/websolutespa/ws-bom-robot-base:gpu
+```
+
+dockerize app
+
+```pwsh
+docker build -f Dockerfile -t ws-bom-robot-app .
+docker run --rm --name ws-bom-robot-app -d --env-file .env -p 6001:6001 ws-bom-robot-app
+```
+
+docker run mounted to src (dev mode)
 
-  #### libmagic (mandatory)
+```pwsh
+docker run --rm --name ws-bom-robot-app-src -d --env-file .env -v "$(pwd)/ws_bom_robot_app:/app/ws_bom_robot_app" -v "$(pwd)/.data:/app/.data" -v "$(pwd)/tmp:/tmp"  -p 6001:6001 ws-bom-robot-app fastapi dev ./ws_bom_robot_app/main.py --host 0.0.0.0 --port 6001
+```
+
+---
+
+## 🔖 Windows requirements (for RAG functionality only)
+
+> ⚠️ While it's strongly recommended to use a docker container for development, you can run the app on Windows with the following requirements
+
+### libmagic (mandatory)
 
   ```bash
   py -m pip install --upgrade python-magic-bin
   ```
   
-  #### tesseract-ocr (mandatory)
+### tesseract-ocr (mandatory)
 
   [Install tesseract](https://github.com/UB-Mannheim/tesseract/wiki)
   [Last win-64 release](https://github.com/tesseract-ocr/tesseract/releases/download/5.5.0/tesseract-ocr-w64-setup-5.5.0.20241111.exe)
@@ -143,15 +192,15 @@ robot_cms_files_folder=llmKbFile
   }
   ```
 
-  #### docling
-  
+### docling
+
   Set the following environment variables
 
   ```pwsh
   KMP_DUPLICATE_LIB_OK=TRUE
   ```    
 
-  #### libreoffice (optional: for robot_env set to development/production)
+### libreoffice (optional: for robot_env set to development/production)
 
   [Install libreoffice](https://www.libreoffice.org/download/download-libreoffice/)
   [Last win-64 release](https://download.documentfoundation.org/libreoffice/stable/24.8.2/win/x86_64/LibreOffice_24.8.2_Win_x86-64.msi)
@@ -166,7 +215,7 @@ robot_cms_files_folder=llmKbFile
   }
   ```
 
-  #### poppler (optional: for robot_env set to development/production)
+### poppler (optional: for robot_env set to development/production)
 
   [Download win poppler release](https://github.com/oschwartz10612/poppler-windows/releases)
   Extract the zip, copy the nested folder "poppler-x.x.x." to a program folder (e.g. C:\Program Files\poppler-24.08.0)
@@ -210,7 +259,7 @@ py -m build && twine check dist/*
 Install the package in editable project location
 
 ```pwsh
-py -m pip install --upgrade -e .
+py -m pip install -U -e .
 py -m pip show ws-bom-robot-app
 ```
 
@@ -231,7 +280,7 @@ prospector ./ws_bom_robot_app -t dodgy -t bandit
 prospector ./ws_bom_robot_app -t pyroma
 ```
 
-lauch pytest
+#### 🧪 run tests
 
 ```pwsh
 !py -m pip install -U pytest pytest-asyncio pytest-mock pytest-cov pyclean
@@ -242,48 +291,24 @@ pytest --cov=ws_bom_robot_app --log-cli-level=info
 # pytest --cov=ws_bom_robot_app --log-cli-level=info ./tests/app/llm/vector_store/db
 ```
 
-launch debugger
+#### 🐞 start debugger
 
 ```pwsh
 streamlit run debugger.py --server.port 8051
 ```
 
-dockerize base image
-
-```pwsh
-<# cpu #>
-docker build -f Dockerfile-robot-base-cpu -t ghcr.io/websolutespa/ws-bom-robot-base:cpu .
-docker push ghcr.io/websolutespa/ws-bom-robot-base:cpu
-<# gpu #>
-docker build -f Dockerfile-robot-base-gpu -t ghcr.io/websolutespa/ws-bom-robot-base:gpu .
-docker push ghcr.io/websolutespa/ws-bom-robot-base:gpu
-```
-
-dockerize app
-
-```pwsh
-docker build -f Dockerfile -t ws-bom-robot-app .
-docker run --rm --name ws-bom-robot-app -d -p 6001:6001 ws-bom-robot-app
-```
-
-docker run mounted to src
-
-```pwsh
-docker run --rm --name ws-bom-robot-app-src -d -v "$(pwd)/ws_bom_robot_app:/app/ws_bom_robot_app" -v "$(pwd)/.data:/app/.data" -v "$(pwd)/tmp:/tmp"  -p 6001:6001 ws-bom-robot-app
-```
-
 ### ✈️ publish
 
 - [testpypi](https://test.pypi.org/project/ws-bom-robot-app/)
 
   ```pwsh
   twine upload --verbose -r testpypi dist/*
-  #py -m pip install -i https://test.pypi.org/simple/ --upgrade ws-bom-robot-app 
+  #pip install -i https://test.pypi.org/simple/ -U ws-bom-robot-app 
   ```
 
 - [pypi](https://pypi.org/project/ws-bom-robot-app/)
 
   ```pwsh
   twine upload --verbose dist/* 
-  #py -m pip install --upgrade ws-bom-robot-app
+
   ```

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/README.md

@@ -17,18 +17,30 @@ from ws_bom_robot_app import main
 app = main.app
 ```
 
-FIll `.env` with the following code:
-
-```env
-#robot_env=local/development/production
-robot_env=local
-robot_user='[user]'
-robot_password='[pwd]'
-robot_data_folder='./.data'
-robot_cms_auth='[auth]'
-robot_cms_host='https://[DOMAIN]'
-robot_cms_db_folder=llmVectorDb
-robot_cms_files_folder=llmKbFile
+Create a `.env` file in the root directory with the following configuration:
+
+```properties
+# robot configuration
+robot_env=development
+robot_user=your_username
+USER_AGENT=ws-bom-robot-app
+
+# cms (bowl) configuration
+robot_cms_host='http://localhost:4000'
+robot_cms_auth='users API-Key your-api-key-here'
+
+# llm providers: fill one or more of these with your API keys
+DEEPSEEK_API_KEY="your-deepseek-api-key"
+OPENAI_API_KEY="your-openai-api-key"
+GOOGLE_API_KEY="your-google-api-key"
+ANTHROPIC_API_KEY="your-anthropic-api-key"
+GROQ_API_KEY="your-groq-api-key"
+# ibm
+WATSONX_URL="https://eu-gb.ml.cloud.ibm.com"
+WATSONX_APIKEY="your-watsonx-api-key"
+WATSONX_PROJECTID="your-watsonx-project-id"
+# gvertex: ensure to mount the file in docker
+GOOGLE_APPLICATION_CREDENTIALS="./.data/secrets/google-credentials.json" 
 ```
 
 ## 🚀 Run the app
@@ -54,15 +66,52 @@ robot_cms_files_folder=llmKbFile
   #gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app --bind
   ```
 
-### 🔖 Windows requirements  
+## 📖 API documentation
+
+- [swagger](http://localhost:6001/docs)
+- [redoc](http://localhost:6001/redoc)
+
+---
+
+## 🐳 Docker
+
+dockerize base image
+
+```pwsh
+<# cpu #>
+docker build -f Dockerfile-robot-base-cpu -t ghcr.io/websolutespa/ws-bom-robot-base:cpu .
+docker push ghcr.io/websolutespa/ws-bom-robot-base:cpu
+<# gpu #>
+docker build -f Dockerfile-robot-base-gpu -t ghcr.io/websolutespa/ws-bom-robot-base:gpu .
+docker push ghcr.io/websolutespa/ws-bom-robot-base:gpu
+```
+
+dockerize app
+
+```pwsh
+docker build -f Dockerfile -t ws-bom-robot-app .
+docker run --rm --name ws-bom-robot-app -d --env-file .env -p 6001:6001 ws-bom-robot-app
+```
+
+docker run mounted to src (dev mode)
 
-  #### libmagic (mandatory)
+```pwsh
+docker run --rm --name ws-bom-robot-app-src -d --env-file .env -v "$(pwd)/ws_bom_robot_app:/app/ws_bom_robot_app" -v "$(pwd)/.data:/app/.data" -v "$(pwd)/tmp:/tmp"  -p 6001:6001 ws-bom-robot-app fastapi dev ./ws_bom_robot_app/main.py --host 0.0.0.0 --port 6001
+```
+
+---
+
+## 🔖 Windows requirements (for RAG functionality only)
+
+> ⚠️ While it's strongly recommended to use a docker container for development, you can run the app on Windows with the following requirements
+
+### libmagic (mandatory)
 
   ```bash
   py -m pip install --upgrade python-magic-bin
   ```
   
-  #### tesseract-ocr (mandatory)
+### tesseract-ocr (mandatory)
 
   [Install tesseract](https://github.com/UB-Mannheim/tesseract/wiki)
   [Last win-64 release](https://github.com/tesseract-ocr/tesseract/releases/download/5.5.0/tesseract-ocr-w64-setup-5.5.0.20241111.exe)
@@ -77,15 +126,15 @@ robot_cms_files_folder=llmKbFile
   }
   ```
 
-  #### docling
-  
+### docling
+
   Set the following environment variables
 
   ```pwsh
   KMP_DUPLICATE_LIB_OK=TRUE
   ```    
 
-  #### libreoffice (optional: for robot_env set to development/production)
+### libreoffice (optional: for robot_env set to development/production)
 
   [Install libreoffice](https://www.libreoffice.org/download/download-libreoffice/)
   [Last win-64 release](https://download.documentfoundation.org/libreoffice/stable/24.8.2/win/x86_64/LibreOffice_24.8.2_Win_x86-64.msi)
@@ -100,7 +149,7 @@ robot_cms_files_folder=llmKbFile
   }
   ```
 
-  #### poppler (optional: for robot_env set to development/production)
+### poppler (optional: for robot_env set to development/production)
 
   [Download win poppler release](https://github.com/oschwartz10612/poppler-windows/releases)
   Extract the zip, copy the nested folder "poppler-x.x.x." to a program folder (e.g. C:\Program Files\poppler-24.08.0)
@@ -144,7 +193,7 @@ py -m build && twine check dist/*
 Install the package in editable project location
 
 ```pwsh
-py -m pip install --upgrade -e .
+py -m pip install -U -e .
 py -m pip show ws-bom-robot-app
 ```
 
@@ -165,7 +214,7 @@ prospector ./ws_bom_robot_app -t dodgy -t bandit
 prospector ./ws_bom_robot_app -t pyroma
 ```
 
-lauch pytest
+#### 🧪 run tests
 
 ```pwsh
 !py -m pip install -U pytest pytest-asyncio pytest-mock pytest-cov pyclean
@@ -176,48 +225,24 @@ pytest --cov=ws_bom_robot_app --log-cli-level=info
 # pytest --cov=ws_bom_robot_app --log-cli-level=info ./tests/app/llm/vector_store/db
 ```
 
-launch debugger
+#### 🐞 start debugger
 
 ```pwsh
 streamlit run debugger.py --server.port 8051
 ```
 
-dockerize base image
-
-```pwsh
-<# cpu #>
-docker build -f Dockerfile-robot-base-cpu -t ghcr.io/websolutespa/ws-bom-robot-base:cpu .
-docker push ghcr.io/websolutespa/ws-bom-robot-base:cpu
-<# gpu #>
-docker build -f Dockerfile-robot-base-gpu -t ghcr.io/websolutespa/ws-bom-robot-base:gpu .
-docker push ghcr.io/websolutespa/ws-bom-robot-base:gpu
-```
-
-dockerize app
-
-```pwsh
-docker build -f Dockerfile -t ws-bom-robot-app .
-docker run --rm --name ws-bom-robot-app -d -p 6001:6001 ws-bom-robot-app
-```
-
-docker run mounted to src
-
-```pwsh
-docker run --rm --name ws-bom-robot-app-src -d -v "$(pwd)/ws_bom_robot_app:/app/ws_bom_robot_app" -v "$(pwd)/.data:/app/.data" -v "$(pwd)/tmp:/tmp"  -p 6001:6001 ws-bom-robot-app
-```
-
 ### ✈️ publish
 
 - [testpypi](https://test.pypi.org/project/ws-bom-robot-app/)
 
   ```pwsh
   twine upload --verbose -r testpypi dist/*
-  #py -m pip install -i https://test.pypi.org/simple/ --upgrade ws-bom-robot-app 
+  #pip install -i https://test.pypi.org/simple/ -U ws-bom-robot-app 
   ```
 
 - [pypi](https://pypi.org/project/ws-bom-robot-app/)
 
   ```pwsh
   twine upload --verbose dist/* 
-  #py -m pip install --upgrade ws-bom-robot-app
+
   ```

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/setup.py

@@ -4,7 +4,7 @@ _requirements = [line.split('#')[0].strip() for line in open("requirements.txt")
 
 setup(
     name="ws_bom_robot_app",
-    version="0.0.78",
+    version="0.0.80",
     description="A FastAPI application serving ws bom/robot/llm platform ai.",
     long_description=open("README.md", encoding='utf-8').read(),
     long_description_content_type="text/markdown",

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/ws_bom_robot_app/config.py

@@ -14,6 +14,8 @@ class Settings(BaseSettings):
     robot_data_db_folder_out: str = 'out'
     robot_data_db_folder_store: str = 'store'
     robot_data_db_retention_days: float = 60
+    robot_data_attachment_folder: str = 'attachment'
+    robot_data_attachment_retention_days: float = 1
     robot_loader_max_threads: int = 1
     robot_task_max_total_parallelism: int = 2 * (os.cpu_count() or 1)
     robot_task_retention_days: float = 1
@@ -30,7 +32,7 @@ class Settings(BaseSettings):
     WATSONX_URL: str = ''
     WATSONX_APIKEY: str = ''
     WATSONX_PROJECTID: str = ''
-    NEBULY_API_URL: str =''
+    NEBULY_API_URL: str ='https://backend.nebuly.com/'
     GOOGLE_APPLICATION_CREDENTIALS: str = '' # path to google credentials iam file, e.d. ./.secrets/google-credentials.json
     model_config = ConfigDict(
         env_file='./.env',

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/ws_bom_robot_app/cron_manager.py

@@ -8,7 +8,7 @@ from apscheduler.triggers.date import DateTrigger
 from fastapi import APIRouter
 from datetime import datetime
 from ws_bom_robot_app.task_manager import task_manager
-from ws_bom_robot_app.llm.utils.kb import kb_cleanup_data_file
+from ws_bom_robot_app.llm.utils.cleanup import kb_cleanup_data_file, chat_cleanup_attachment
 from ws_bom_robot_app.util import _log
 from ws_bom_robot_app.config import config
 
@@ -57,7 +57,8 @@ class Job:
 class CronManager:
     _list_default = [
             Job('cleanup-task',task_manager.cleanup_task, interval=5 * 60),
-            Job('cleanup-data',kb_cleanup_data_file, interval=180 * 60),
+            Job('cleanup-kb-data',kb_cleanup_data_file, interval=180 * 60),
+            Job('cleanup-chat-attachment',chat_cleanup_attachment, interval=120 * 60),
         ]
     def __get_jobstore_strategy(self) -> JobstoreStrategy:
         if True or config.runtime_options().is_multi_process:
@@ -139,22 +140,22 @@ class CronManager:
 
     def execute_recurring_jobs(self):
         for job in self.scheduler.get_jobs():
-            if job.interval:
-                job.job_func()
+            if job.trigger.interval:
+                job.func()
 
     def pause_recurring_jobs(self):
         for job in self.scheduler.get_jobs():
-            if job.interval:
+            if job.trigger.interval:
                 self.pause_job(job.id)
 
     def resume_recurring_jobs(self):
         for job in self.scheduler.get_jobs():
-            if job.interval:
+            if job.trigger.interval:
                 self.resume_job(job.id)
 
     def remove_recurring_jobs(self):
         for job in self.scheduler.get_jobs():
-            if job.interval:
+            if job.trigger.interval:
                 self.remove_job(job.id)
 
     def clear(self):

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/ws_bom_robot_app/llm/main.py

@@ -39,6 +39,7 @@ def _parse_formatted_message(message: str) -> str:
   except:
     result = message
   return result
+
 async def __stream(rq: StreamRequest, ctx: Request, queue: Queue, formatted: bool = True) -> None:
   #os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
 
@@ -47,11 +48,21 @@ async def __stream(rq: StreamRequest, ctx: Request, queue: Queue, formatted: boo
   for tool in rq.app_tools:
     tool.thread_id = rq.thread_id
 
+  #llm
+  __llm: LlmInterface = rq.get_llm()
+
   #chat history
   chat_history: list[BaseMessage] = []
   for message in rq.messages:
     if message.role in ["human","user"]:
-      chat_history.append(HumanMessage(content=message.content))
+      _content = message.content
+      # multimodal content parsing
+      if isinstance(_content, list):
+        try:
+          _content = await __llm.format_multimodal_content(_content)
+        except Exception as e:
+          logging.warning(f"Error parsing multimodal content {_content[:100]}: {e}")
+      chat_history.append(HumanMessage(content=_content))
     elif message.role in ["ai","assistant"]:
       message_content = ""
       if formatted:
@@ -78,8 +89,6 @@ async def __stream(rq: StreamRequest, ctx: Request, queue: Queue, formatted: boo
       if message_content:
         chat_history.append(AIMessage(content=message_content))
 
-  #llm
-  __llm: LlmInterface = rq.get_llm()
 
   #agent handler
   if formatted:

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/ws_bom_robot_app/llm/models/api.py

@@ -11,6 +11,39 @@ import os, shutil, uuid
 from ws_bom_robot_app.config import Settings, config
 
 class LlmMessage(BaseModel):
+  """
+  💬 multimodal chat
+
+  The multimodal message allows users to interact with the application using both text and media files.
+  `robot` accept multimodal input in a uniform way, regarding the llm provider used.
+
+  - simple message
+
+  ```json
+  {
+    "role": "user",
+    "content": "What is the capital of France?"
+  }
+  ```
+
+  - multimodal message
+
+  ```jsonc
+  {
+    "role": "user",
+    "content": [
+      { "type": "text", "text": "Read carefully all the attachments, analize the content and provide a summary for each one:" },
+      { "type": "image", "url": "https://www.example.com/image/foo.jpg" },
+      { "type": "file", "url": "https://www.example.com/pdf/bar.pdf" },
+      { "type": "file", "url": "data:plain/text;base64,CiAgICAgIF9fX19fCiAgICAgLyAgIC..." }, // base64 encoded file
+      { "type": "media", "mime_type": "plain/text", "data": "CiAgICAgIF9fX19fCiAgICAgLyAgIC..." } // google/gemini specific input format
+    ]
+  }
+  ```
+
+  > 💡 `url` can be a remote url or a base64 representation of the file: [rfc 2397](https://datatracker.ietf.org/doc/html/rfc2397).
+  Can also be used the llm/model specific input format.
+  """
   role: str
   content: Union[str, list]
 

{ws_bom_robot_app-0.0.78 → ws_bom_robot_app-0.0.80}/ws_bom_robot_app/llm/nebuly_handler.py

@@ -145,16 +145,20 @@ class NebulyHandler(AsyncCallbackHandler):
         return payload
 
     def __parse_multimodal_input(self, input: list[dict]) -> str:
-        # Parse the multimodal input and return a string representation
-        # This is a placeholder implementation, you can customize it as needed
-        parsed_input = ""
-        for item in input:
-            if item.get("type") == "text":
-                parsed_input += item.get("text", "")
-            elif item.get("type") == "image_url":
-                parsed_input += " <image>"
-        # print(parsed_input)
-        return parsed_input
+      """Parse multimodal input and return a string representation."""
+      type_mapping = {
+        "text": lambda item: item.get("text", ""),
+        "image": lambda _: " <image>",
+        "image_url": lambda _: " <image>",
+        "file": lambda _: " <file>",
+        "media": lambda _: " <file>",
+        "document": lambda _: " <file>",
+      }
+
+      return "".join(
+        type_mapping.get(item.get("type", ""), lambda item: f" <{item.get('type', '')}>")
+        (item) for item in input
+      )
 
     def __parse_multimodal_history(self, messages: list[dict]) -> list[dict]:
         # Parse the multimodal history and return a list of dictionaries