iatoolkit 0.4.0__tar.gz → 0.4.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of iatoolkit might be problematic. Click here for more details.
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/PKG-INFO +43 -37
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/pyproject.toml +1 -1
- iatoolkit-0.4.1/readme.md +49 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/__init__.py +7 -1
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/cli_commands.py +2 -10
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/company_registry.py +2 -8
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit.egg-info/PKG-INFO +43 -37
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/benchmark_service.py +1 -1
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/dispatcher_service.py +1 -2
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/file_processor_service.py +28 -5
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/load_documents_service.py +75 -22
- iatoolkit-0.4.0/readme.md +0 -43
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/requirements.txt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/setup.cfg +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/base_company.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/iatoolkit.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/system_prompts/arquitectura.prompt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/system_prompts/format_styles.prompt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/system_prompts/query_main.prompt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit/system_prompts/sql_rules.prompt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit.egg-info/SOURCES.txt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit.egg-info/dependency_links.txt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit.egg-info/requires.txt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/iatoolkit.egg-info/top_level.txt +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/__init__.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/api_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/document_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/excel_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/history_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/jwt_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/mail_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/profile_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/prompt_manager_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/query_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/search_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/sql_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/tasks_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/user_feedback_service.py +0 -0
- {iatoolkit-0.4.0 → iatoolkit-0.4.1}/src/services/user_session_context_service.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: iatoolkit
|
|
3
|
-
Version: 0.4.
|
|
3
|
+
Version: 0.4.1
|
|
4
4
|
Summary: IAToolkit
|
|
5
5
|
Author: Fernando Libedinsky
|
|
6
6
|
License-Expression: MIT
|
|
@@ -207,46 +207,52 @@ Requires-Dist: yarl==1.18.3
|
|
|
207
207
|
Requires-Dist: zipp==3.21.0
|
|
208
208
|
Requires-Dist: zstandard==0.23.0
|
|
209
209
|
|
|
210
|
-
|
|
210
|
+
|
|
211
|
+
<div align="center">
|
|
212
|
+
<h1>IAToolkit</h1>
|
|
213
|
+
<p><strong>The Open-Source Framework for Building AI Chatbots on Your Private Data.</strong></p>
|
|
214
|
+
</div>
|
|
211
215
|
|
|
212
216
|
IAToolkit is a comprehensive, open-source framework designed for building enterprise-grade
|
|
213
217
|
AI chatbots and conversational applications.
|
|
214
|
-
|
|
218
|
+
With IAToolkit, you can build production-ready, context-aware chatbots and agents that
|
|
219
|
+
can query relational databases, perform semantic searches on documents,
|
|
220
|
+
and connect to your internal APIs in minutes.
|
|
221
|
+
|
|
222
|
+
IAToolkit bridges the gap between powerful LLMs and your company's data.
|
|
223
|
+
|
|
215
224
|
|
|
216
225
|
## 🚀 Key Features
|
|
217
|
-
- **Universal LLM Integration**: OpenAI GPT, Google Gemini
|
|
218
|
-
- **Template System**: Jinja2-powered prompt templates with variables
|
|
219
|
-
- **Context Management**: Maintain conversation context across sessions
|
|
220
|
-
|
|
221
|
-
### 🔒 **Enterprise Security**
|
|
222
|
-
- **JWT Authentication**: Secure token-based authentication
|
|
223
|
-
- **Session Management**: Redis-backed secure sessions
|
|
224
|
-
- **CORS Configuration**: Flexible cross-origin resource sharing
|
|
225
|
-
|
|
226
|
-
### 🛠 **Function Calling & Tools**
|
|
227
|
-
- **Native Function Calls**: Direct integration with LLM function calling
|
|
228
|
-
- **Custom Tools**: Build and register custom tools for your chatbot
|
|
229
|
-
- **SQL Query Generation**: Natural language to SQL conversion
|
|
230
|
-
- **API Integrations**: Connect to external services and APIs
|
|
231
|
-
|
|
232
|
-
### 🗄 **Database & Storage**
|
|
233
|
-
- **Multi-Database Support**: PostgreSQL, MySQL, SQLite via SQLAlchemy
|
|
234
|
-
- **Vector Store Integration**: Semantic search and retrieval
|
|
235
|
-
- **Document Processing**: PDF, Word, Excel, and text file handling
|
|
236
|
-
|
|
237
|
-
### 📊 **Analytics & Monitoring**
|
|
238
|
-
- **Query Logging**: Track all LLM interactions
|
|
239
|
-
- **Performance Metrics**: Response times, token usage, costs
|
|
240
|
-
- **Benchmarking**: Compare model performance
|
|
241
|
-
- **Task Management**: Async task processing with status tracking
|
|
242
|
-
|
|
243
|
-
### 🔧 **Developer Experience**
|
|
244
|
-
- **Dependency Injection**: Clean, testable architecture
|
|
245
|
-
- **CLI Tools**: Command-line interface for common tasks
|
|
246
|
-
- **Hot Reloading**: Development-friendly configuration
|
|
247
|
-
- **Comprehensive Logging**: Debug and monitor easily
|
|
248
|
-
|
|
249
|
-
## License
|
|
250
|
-
MIT License
|
|
251
226
|
|
|
227
|
+
* **🔗 Unified Data Connection**:
|
|
228
|
+
* **Natural Language to SQL**: Let your chatbot query relational databases (PostgreSQL, MySQL, SQLite) using everyday language.
|
|
229
|
+
* **Semantic Document Search**: Automatically chunk, embed, and search across your private documents (PDFs, Word, etc.) to provide contextually accurate answers.
|
|
230
|
+
|
|
231
|
+
* **🏢 Enterprise-Ready Multi-Tenancy**:
|
|
232
|
+
* Deploy isolated "Company" modules, each with its own data, tools, and context. Perfect for SaaS products or internal departmental agents.
|
|
233
|
+
|
|
234
|
+
* **🧠 LLM Agnostic**:
|
|
235
|
+
* Switch between **OpenAI (GPT-*)** and **Google (Gemini-*)** with a single line change in your configuration. No code refactoring needed.
|
|
236
|
+
|
|
237
|
+
* **🛠️ Developer-First Experience**:
|
|
238
|
+
* Built with a clean, **Dependency Injection** architecture.
|
|
239
|
+
* High-quality code base with **90%+ test coverage**.
|
|
240
|
+
* Powerful Flask-based **CLI** for database setup, API key generation, and more.
|
|
241
|
+
|
|
242
|
+
* **🔒 Security & Observability Built-In**:
|
|
243
|
+
* Comes with JWT-based authentication, user management, and secure session handling out of the box.
|
|
244
|
+
* Full traceability with detailed logging of all queries, function calls, token usage, and costs.
|
|
245
|
+
|
|
246
|
+
## ⚡ Quick Start: Create a Custom Tool in 30 Seconds
|
|
247
|
+
|
|
248
|
+
See how easy it is to give your AI a new skill. Just define a method inside your Company class and describe it.
|
|
249
|
+
IAToolkit handles the rest.
|
|
250
|
+
|
|
251
|
+
## 🤝 Contributing
|
|
252
|
+
|
|
253
|
+
We welcome contributions! Whether it's adding a new feature, improving documentation, or fixing a bug,
|
|
254
|
+
please feel free to open a pull request.
|
|
255
|
+
|
|
256
|
+
## 📄 License
|
|
252
257
|
|
|
258
|
+
IAToolkit is open-source and licensed under the [MIT License](LICENSE).
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
|
|
2
|
+
<div align="center">
|
|
3
|
+
<h1>IAToolkit</h1>
|
|
4
|
+
<p><strong>The Open-Source Framework for Building AI Chatbots on Your Private Data.</strong></p>
|
|
5
|
+
</div>
|
|
6
|
+
|
|
7
|
+
IAToolkit is a comprehensive, open-source framework designed for building enterprise-grade
|
|
8
|
+
AI chatbots and conversational applications.
|
|
9
|
+
With IAToolkit, you can build production-ready, context-aware chatbots and agents that
|
|
10
|
+
can query relational databases, perform semantic searches on documents,
|
|
11
|
+
and connect to your internal APIs in minutes.
|
|
12
|
+
|
|
13
|
+
IAToolkit bridges the gap between powerful LLMs and your company's data.
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
## 🚀 Key Features
|
|
17
|
+
|
|
18
|
+
* **🔗 Unified Data Connection**:
|
|
19
|
+
* **Natural Language to SQL**: Let your chatbot query relational databases (PostgreSQL, MySQL, SQLite) using everyday language.
|
|
20
|
+
* **Semantic Document Search**: Automatically chunk, embed, and search across your private documents (PDFs, Word, etc.) to provide contextually accurate answers.
|
|
21
|
+
|
|
22
|
+
* **🏢 Enterprise-Ready Multi-Tenancy**:
|
|
23
|
+
* Deploy isolated "Company" modules, each with its own data, tools, and context. Perfect for SaaS products or internal departmental agents.
|
|
24
|
+
|
|
25
|
+
* **🧠 LLM Agnostic**:
|
|
26
|
+
* Switch between **OpenAI (GPT-*)** and **Google (Gemini-*)** with a single line change in your configuration. No code refactoring needed.
|
|
27
|
+
|
|
28
|
+
* **🛠️ Developer-First Experience**:
|
|
29
|
+
* Built with a clean, **Dependency Injection** architecture.
|
|
30
|
+
* High-quality code base with **90%+ test coverage**.
|
|
31
|
+
* Powerful Flask-based **CLI** for database setup, API key generation, and more.
|
|
32
|
+
|
|
33
|
+
* **🔒 Security & Observability Built-In**:
|
|
34
|
+
* Comes with JWT-based authentication, user management, and secure session handling out of the box.
|
|
35
|
+
* Full traceability with detailed logging of all queries, function calls, token usage, and costs.
|
|
36
|
+
|
|
37
|
+
## ⚡ Quick Start: Create a Custom Tool in 30 Seconds
|
|
38
|
+
|
|
39
|
+
See how easy it is to give your AI a new skill. Just define a method inside your Company class and describe it.
|
|
40
|
+
IAToolkit handles the rest.
|
|
41
|
+
|
|
42
|
+
## 🤝 Contributing
|
|
43
|
+
|
|
44
|
+
We welcome contributions! Whether it's adding a new feature, improving documentation, or fixing a bug,
|
|
45
|
+
please feel free to open a pull request.
|
|
46
|
+
|
|
47
|
+
## 📄 License
|
|
48
|
+
|
|
49
|
+
IAToolkit is open-source and licensed under the [MIT License](LICENSE).
|
|
@@ -21,13 +21,15 @@ from services.excel_service import ExcelService
|
|
|
21
21
|
from services.dispatcher_service import Dispatcher
|
|
22
22
|
from services.document_service import DocumentService
|
|
23
23
|
from services.search_service import SearchService
|
|
24
|
+
from services.load_documents_service import LoadDocumentsService
|
|
24
25
|
from repositories.profile_repo import ProfileRepo
|
|
25
26
|
from repositories.llm_query_repo import LLMQueryRepo
|
|
26
27
|
from services.query_service import QueryService
|
|
28
|
+
from services.prompt_manager_service import PromptService
|
|
27
29
|
from repositories.database_manager import DatabaseManager
|
|
28
30
|
from infra.call_service import CallServiceClient
|
|
29
31
|
from common.util import Utility
|
|
30
|
-
from repositories.models import Base, Company, Function, TaskType
|
|
32
|
+
from repositories.models import Base, Company, Function, TaskType, Prompt, PromptCategory
|
|
31
33
|
|
|
32
34
|
|
|
33
35
|
__all__ = [
|
|
@@ -42,8 +44,10 @@ __all__ = [
|
|
|
42
44
|
'DocumentService',
|
|
43
45
|
'SearchService',
|
|
44
46
|
'QueryService',
|
|
47
|
+
'LoadDocumentsService',
|
|
45
48
|
'ProfileRepo',
|
|
46
49
|
'LLMQueryRepo',
|
|
50
|
+
'PromptService',
|
|
47
51
|
'DatabaseManager',
|
|
48
52
|
'CallServiceClient',
|
|
49
53
|
'Utility',
|
|
@@ -51,4 +55,6 @@ __all__ = [
|
|
|
51
55
|
'Function',
|
|
52
56
|
'TaskType',
|
|
53
57
|
'Base',
|
|
58
|
+
'Prompt',
|
|
59
|
+
'PromptCategory'
|
|
54
60
|
]
|
|
@@ -24,6 +24,8 @@ def register_core_commands(app):
|
|
|
24
24
|
def setup_company(company_short_name: str):
|
|
25
25
|
"""⚙️ Genera una nueva API key para una compañía ya registrada."""
|
|
26
26
|
try:
|
|
27
|
+
dispatcher = IAToolkit.get_instance().get_injector().get(Dispatcher)
|
|
28
|
+
dispatcher.setup_all_companies()
|
|
27
29
|
profile_service = IAToolkit.get_instance().get_injector().get(ProfileService)
|
|
28
30
|
click.echo(f"🔑 Generando API key para '{company_short_name}'...")
|
|
29
31
|
result = profile_service.new_api_key(company_short_name)
|
|
@@ -64,14 +66,4 @@ def register_core_commands(app):
|
|
|
64
66
|
logging.exception(e)
|
|
65
67
|
click.echo(f"Error: {str(e)}")
|
|
66
68
|
|
|
67
|
-
@app.cli.command("load")
|
|
68
|
-
def load_documents():
|
|
69
|
-
from services.load_documents_service import LoadDocumentsService
|
|
70
69
|
|
|
71
|
-
load_documents_service = IAToolkit.get_instance().get_injector().get(LoadDocumentsService)
|
|
72
|
-
try:
|
|
73
|
-
result = load_documents_service.load()
|
|
74
|
-
click.echo(result['message'])
|
|
75
|
-
except Exception as e:
|
|
76
|
-
logging.exception(e)
|
|
77
|
-
click.echo(f"Error: {str(e)}")
|
|
@@ -15,27 +15,21 @@ class CompanyRegistry:
|
|
|
15
15
|
def __init__(self):
|
|
16
16
|
self._company_classes: Dict[str, Type[BaseCompany]] = {}
|
|
17
17
|
self._company_instances: Dict[str, BaseCompany] = {}
|
|
18
|
-
self._injector = None
|
|
19
18
|
|
|
20
|
-
def set_injector(self, injector) -> None:
|
|
21
|
-
"""Establece el injector para crear instancias con dependencias"""
|
|
22
|
-
self._injector = injector
|
|
23
19
|
|
|
24
|
-
def instantiate_companies(self) -> Dict[str, BaseCompany]:
|
|
20
|
+
def instantiate_companies(self, injector) -> Dict[str, BaseCompany]:
|
|
25
21
|
"""
|
|
26
22
|
Instancia todas las empresas registradas con inyección de dependencias.
|
|
27
23
|
|
|
28
24
|
Returns:
|
|
29
25
|
Dict con instancias de empresas {name: instance}
|
|
30
26
|
"""
|
|
31
|
-
if not self._injector:
|
|
32
|
-
raise RuntimeError("Injector no configurado. Llame a set_injector() primero.")
|
|
33
27
|
|
|
34
28
|
for company_key, company_class in self._company_classes.items():
|
|
35
29
|
if company_key not in self._company_instances:
|
|
36
30
|
try:
|
|
37
31
|
# use de injector to create the instance
|
|
38
|
-
company_instance =
|
|
32
|
+
company_instance = injector.get(company_class)
|
|
39
33
|
self._company_instances[company_key] = company_instance
|
|
40
34
|
logging.info(f"company '{company_key}' created in dispatcher")
|
|
41
35
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: iatoolkit
|
|
3
|
-
Version: 0.4.
|
|
3
|
+
Version: 0.4.1
|
|
4
4
|
Summary: IAToolkit
|
|
5
5
|
Author: Fernando Libedinsky
|
|
6
6
|
License-Expression: MIT
|
|
@@ -207,46 +207,52 @@ Requires-Dist: yarl==1.18.3
|
|
|
207
207
|
Requires-Dist: zipp==3.21.0
|
|
208
208
|
Requires-Dist: zstandard==0.23.0
|
|
209
209
|
|
|
210
|
-
|
|
210
|
+
|
|
211
|
+
<div align="center">
|
|
212
|
+
<h1>IAToolkit</h1>
|
|
213
|
+
<p><strong>The Open-Source Framework for Building AI Chatbots on Your Private Data.</strong></p>
|
|
214
|
+
</div>
|
|
211
215
|
|
|
212
216
|
IAToolkit is a comprehensive, open-source framework designed for building enterprise-grade
|
|
213
217
|
AI chatbots and conversational applications.
|
|
214
|
-
|
|
218
|
+
With IAToolkit, you can build production-ready, context-aware chatbots and agents that
|
|
219
|
+
can query relational databases, perform semantic searches on documents,
|
|
220
|
+
and connect to your internal APIs in minutes.
|
|
221
|
+
|
|
222
|
+
IAToolkit bridges the gap between powerful LLMs and your company's data.
|
|
223
|
+
|
|
215
224
|
|
|
216
225
|
## 🚀 Key Features
|
|
217
|
-
- **Universal LLM Integration**: OpenAI GPT, Google Gemini
|
|
218
|
-
- **Template System**: Jinja2-powered prompt templates with variables
|
|
219
|
-
- **Context Management**: Maintain conversation context across sessions
|
|
220
|
-
|
|
221
|
-
### 🔒 **Enterprise Security**
|
|
222
|
-
- **JWT Authentication**: Secure token-based authentication
|
|
223
|
-
- **Session Management**: Redis-backed secure sessions
|
|
224
|
-
- **CORS Configuration**: Flexible cross-origin resource sharing
|
|
225
|
-
|
|
226
|
-
### 🛠 **Function Calling & Tools**
|
|
227
|
-
- **Native Function Calls**: Direct integration with LLM function calling
|
|
228
|
-
- **Custom Tools**: Build and register custom tools for your chatbot
|
|
229
|
-
- **SQL Query Generation**: Natural language to SQL conversion
|
|
230
|
-
- **API Integrations**: Connect to external services and APIs
|
|
231
|
-
|
|
232
|
-
### 🗄 **Database & Storage**
|
|
233
|
-
- **Multi-Database Support**: PostgreSQL, MySQL, SQLite via SQLAlchemy
|
|
234
|
-
- **Vector Store Integration**: Semantic search and retrieval
|
|
235
|
-
- **Document Processing**: PDF, Word, Excel, and text file handling
|
|
236
|
-
|
|
237
|
-
### 📊 **Analytics & Monitoring**
|
|
238
|
-
- **Query Logging**: Track all LLM interactions
|
|
239
|
-
- **Performance Metrics**: Response times, token usage, costs
|
|
240
|
-
- **Benchmarking**: Compare model performance
|
|
241
|
-
- **Task Management**: Async task processing with status tracking
|
|
242
|
-
|
|
243
|
-
### 🔧 **Developer Experience**
|
|
244
|
-
- **Dependency Injection**: Clean, testable architecture
|
|
245
|
-
- **CLI Tools**: Command-line interface for common tasks
|
|
246
|
-
- **Hot Reloading**: Development-friendly configuration
|
|
247
|
-
- **Comprehensive Logging**: Debug and monitor easily
|
|
248
|
-
|
|
249
|
-
## License
|
|
250
|
-
MIT License
|
|
251
226
|
|
|
227
|
+
* **🔗 Unified Data Connection**:
|
|
228
|
+
* **Natural Language to SQL**: Let your chatbot query relational databases (PostgreSQL, MySQL, SQLite) using everyday language.
|
|
229
|
+
* **Semantic Document Search**: Automatically chunk, embed, and search across your private documents (PDFs, Word, etc.) to provide contextually accurate answers.
|
|
230
|
+
|
|
231
|
+
* **🏢 Enterprise-Ready Multi-Tenancy**:
|
|
232
|
+
* Deploy isolated "Company" modules, each with its own data, tools, and context. Perfect for SaaS products or internal departmental agents.
|
|
233
|
+
|
|
234
|
+
* **🧠 LLM Agnostic**:
|
|
235
|
+
* Switch between **OpenAI (GPT-*)** and **Google (Gemini-*)** with a single line change in your configuration. No code refactoring needed.
|
|
236
|
+
|
|
237
|
+
* **🛠️ Developer-First Experience**:
|
|
238
|
+
* Built with a clean, **Dependency Injection** architecture.
|
|
239
|
+
* High-quality code base with **90%+ test coverage**.
|
|
240
|
+
* Powerful Flask-based **CLI** for database setup, API key generation, and more.
|
|
241
|
+
|
|
242
|
+
* **🔒 Security & Observability Built-In**:
|
|
243
|
+
* Comes with JWT-based authentication, user management, and secure session handling out of the box.
|
|
244
|
+
* Full traceability with detailed logging of all queries, function calls, token usage, and costs.
|
|
245
|
+
|
|
246
|
+
## ⚡ Quick Start: Create a Custom Tool in 30 Seconds
|
|
247
|
+
|
|
248
|
+
See how easy it is to give your AI a new skill. Just define a method inside your Company class and describe it.
|
|
249
|
+
IAToolkit handles the rest.
|
|
250
|
+
|
|
251
|
+
## 🤝 Contributing
|
|
252
|
+
|
|
253
|
+
We welcome contributions! Whether it's adding a new feature, improving documentation, or fixing a bug,
|
|
254
|
+
please feel free to open a pull request.
|
|
255
|
+
|
|
256
|
+
## 📄 License
|
|
252
257
|
|
|
258
|
+
IAToolkit is open-source and licensed under the [MIT License](LICENSE).
|
|
@@ -65,7 +65,7 @@ class BenchmarkService:
|
|
|
65
65
|
|
|
66
66
|
company = self.profile_repo.get_company_by_short_name(company_short_name)
|
|
67
67
|
if not company:
|
|
68
|
-
raise IAToolkitException(IAToolkitException.ErrorType.CONFIG_ERROR, "Compañía
|
|
68
|
+
raise IAToolkitException(IAToolkitException.ErrorType.CONFIG_ERROR, f"Compañía {company_short_name} no encontrada.")
|
|
69
69
|
|
|
70
70
|
total_rows = len(df)
|
|
71
71
|
logging.info(f"Iniciando benchmark para {total_rows} casos de prueba desde el archivo: {file_path}")
|
|
@@ -78,8 +78,7 @@ class Dispatcher:
|
|
|
78
78
|
|
|
79
79
|
# ✅ NOW it is safe to get the injector and instantiate companies.
|
|
80
80
|
injector = current_iatoolkit().get_injector()
|
|
81
|
-
self.company_registry.
|
|
82
|
-
self.company_registry.instantiate_companies()
|
|
81
|
+
self.company_registry.instantiate_companies(injector)
|
|
83
82
|
|
|
84
83
|
|
|
85
84
|
def start_execution(self):
|
|
@@ -7,26 +7,45 @@ from infra.connectors.file_connector import FileConnector
|
|
|
7
7
|
import logging
|
|
8
8
|
import os
|
|
9
9
|
from typing import Optional, Callable, Dict
|
|
10
|
+
from repositories.models import Company
|
|
10
11
|
|
|
11
12
|
|
|
12
13
|
class FileProcessorConfig:
|
|
14
|
+
"""Configuration class for the FileProcessor."""
|
|
13
15
|
def __init__(
|
|
14
16
|
self,
|
|
15
17
|
filters: Dict,
|
|
16
|
-
|
|
18
|
+
callback: Callable[[Company, str, bytes, dict], None],
|
|
17
19
|
continue_on_error: bool = True,
|
|
18
20
|
log_file: str = 'file_processor.log',
|
|
19
21
|
echo: bool = False,
|
|
20
|
-
context: dict = None
|
|
22
|
+
context: dict = None
|
|
21
23
|
):
|
|
24
|
+
"""
|
|
25
|
+
Initializes the FileProcessor configuration.
|
|
26
|
+
|
|
27
|
+
Args:
|
|
28
|
+
filters (Dict): A dictionary of filters to apply to file names.
|
|
29
|
+
Example: {'filename_contains': '.pdf'}
|
|
30
|
+
action (Callable): The function to execute for each processed file.
|
|
31
|
+
It receives filename (str), content (bytes), and context (dict).
|
|
32
|
+
continue_on_error (bool): If True, continues processing other files upon an error.
|
|
33
|
+
log_file (str): The path to the log file.
|
|
34
|
+
echo (bool): If True, prints progress to the console.
|
|
35
|
+
context (dict): A context dictionary passed to the action function.
|
|
36
|
+
"""
|
|
22
37
|
self.filters = filters
|
|
23
|
-
self.
|
|
38
|
+
self.callback = callback
|
|
24
39
|
self.continue_on_error = continue_on_error
|
|
25
40
|
self.log_file = log_file
|
|
26
41
|
self.echo = echo
|
|
27
42
|
self.context = context or {}
|
|
28
43
|
|
|
29
44
|
class FileProcessor:
|
|
45
|
+
"""
|
|
46
|
+
A generic service to process files from a given data source (connector).
|
|
47
|
+
It lists files, applies filters, and executes a specific action for each one.
|
|
48
|
+
"""
|
|
30
49
|
def __init__(self,
|
|
31
50
|
connector: FileConnector,
|
|
32
51
|
config: FileProcessorConfig,
|
|
@@ -45,6 +64,7 @@ class FileProcessor:
|
|
|
45
64
|
return logging.getLogger(__name__)
|
|
46
65
|
|
|
47
66
|
def process_files(self):
|
|
67
|
+
# Fetches files from the connector, filters them, and processes them.
|
|
48
68
|
try:
|
|
49
69
|
files = self.connector.list_files()
|
|
50
70
|
except Exception as e:
|
|
@@ -67,9 +87,12 @@ class FileProcessor:
|
|
|
67
87
|
|
|
68
88
|
content = self.connector.get_file_content(file_path)
|
|
69
89
|
|
|
70
|
-
# execute the
|
|
90
|
+
# execute the callback function
|
|
71
91
|
filename = os.path.basename(file_name)
|
|
72
|
-
self.config.
|
|
92
|
+
self.config.callback(company=self.config.context.get('company'),
|
|
93
|
+
filename=filename,
|
|
94
|
+
content=content,
|
|
95
|
+
context=self.config.context)
|
|
73
96
|
self.processed_files += 1
|
|
74
97
|
|
|
75
98
|
self.logger.info(f"Successfully processed file: {file_path}")
|
|
@@ -21,6 +21,10 @@ from typing import Dict
|
|
|
21
21
|
|
|
22
22
|
|
|
23
23
|
class LoadDocumentsService:
|
|
24
|
+
"""
|
|
25
|
+
Orchestrates the process of loading, processing, and storing documents
|
|
26
|
+
from various sources for different companies.
|
|
27
|
+
"""
|
|
24
28
|
@inject
|
|
25
29
|
def __init__(self,
|
|
26
30
|
file_connector_factory: FileConnectorFactory,
|
|
@@ -38,7 +42,6 @@ class LoadDocumentsService:
|
|
|
38
42
|
self.vector_store = vector_store
|
|
39
43
|
self.file_connector_factory = file_connector_factory
|
|
40
44
|
self.dispatcher = dispatcher
|
|
41
|
-
self.company = None
|
|
42
45
|
|
|
43
46
|
# lower warnings
|
|
44
47
|
logging.getLogger().setLevel(logging.ERROR)
|
|
@@ -51,6 +54,17 @@ class LoadDocumentsService:
|
|
|
51
54
|
|
|
52
55
|
# load the files for all of the companies.
|
|
53
56
|
def load(self, doc_type: str = None):
|
|
57
|
+
"""
|
|
58
|
+
Loads documents for all companies based on their configuration.
|
|
59
|
+
It can load all document types or a specific one if provided.
|
|
60
|
+
|
|
61
|
+
Args:
|
|
62
|
+
doc_type (str, optional): A specific document type to load.
|
|
63
|
+
If None, all configured types are loaded.
|
|
64
|
+
|
|
65
|
+
Returns:
|
|
66
|
+
Dict: A dictionary with a summary message.
|
|
67
|
+
"""
|
|
54
68
|
# doc_type: an optional document_type for loading
|
|
55
69
|
files_loaded = 0
|
|
56
70
|
companies = self.profile_repo.get_companies()
|
|
@@ -61,7 +75,6 @@ class LoadDocumentsService:
|
|
|
61
75
|
continue
|
|
62
76
|
|
|
63
77
|
print(f"Cargando datos de ** {company.short_name} **")
|
|
64
|
-
self.company = company
|
|
65
78
|
|
|
66
79
|
# Si hay configuraciones de tipos de documento específicos
|
|
67
80
|
doc_types_config = load_config.get('document_types', {})
|
|
@@ -81,7 +94,34 @@ class LoadDocumentsService:
|
|
|
81
94
|
raise IAToolkitException(IAToolkitException.ErrorType.MISSING_PARAMETER,
|
|
82
95
|
f"Falta configurar conector en empresa {company.short_name}")
|
|
83
96
|
|
|
84
|
-
files_loaded += self.load_data_source(
|
|
97
|
+
files_loaded += self.load_data_source(company=company,
|
|
98
|
+
connector_config=connector)
|
|
99
|
+
|
|
100
|
+
return {'message': f'{files_loaded} files processed'}
|
|
101
|
+
|
|
102
|
+
def load_company_files(self, company: Company,
|
|
103
|
+
connector: dict,
|
|
104
|
+
predefined_metadata: Dict = None,
|
|
105
|
+
filters: Dict = None):
|
|
106
|
+
"""
|
|
107
|
+
Loads all files for a specific company using a given connector.
|
|
108
|
+
|
|
109
|
+
Args:
|
|
110
|
+
company (Company): The company to load files for.
|
|
111
|
+
connector (dict): The connector configuration.
|
|
112
|
+
|
|
113
|
+
Returns:
|
|
114
|
+
Dict: A dictionary with a summary message.
|
|
115
|
+
"""
|
|
116
|
+
if not connector:
|
|
117
|
+
raise IAToolkitException(IAToolkitException.ErrorType.MISSING_PARAMETER,
|
|
118
|
+
f"Falta configurar conector")
|
|
119
|
+
|
|
120
|
+
files_loaded = self.load_data_source(
|
|
121
|
+
company=company,
|
|
122
|
+
connector_config=connector,
|
|
123
|
+
predefined_metadata=predefined_metadata,
|
|
124
|
+
filters=filters)
|
|
85
125
|
|
|
86
126
|
return {'message': f'{files_loaded} files processed'}
|
|
87
127
|
|
|
@@ -99,36 +139,42 @@ class LoadDocumentsService:
|
|
|
99
139
|
# config specific filters
|
|
100
140
|
filters = type_config.get('filters', {"filename_contains": ".pdf"})
|
|
101
141
|
|
|
102
|
-
return self.load_data_source(
|
|
142
|
+
return self.load_data_source(company=company,
|
|
143
|
+
connector_config=connector,
|
|
144
|
+
predefined_metadata=predefined_metadata,
|
|
145
|
+
filters=filters)
|
|
103
146
|
|
|
104
|
-
def load_data_source(self, connector_config: Dict, predefined_metadata: Dict = None, filters: Dict = None):
|
|
147
|
+
def load_data_source(self, company: Company, connector_config: Dict, predefined_metadata: Dict = None, filters: Dict = None):
|
|
105
148
|
"""
|
|
106
|
-
|
|
149
|
+
Loads files from a data source using a connector and a FileProcessor.
|
|
107
150
|
|
|
108
151
|
Args:
|
|
109
|
-
connector_config:
|
|
110
|
-
predefined_metadata:
|
|
111
|
-
filters:
|
|
152
|
+
connector_config (Dict): The configuration for the file connector.
|
|
153
|
+
predefined_metadata (Dict, optional): Metadata to be added to all documents from this source.
|
|
154
|
+
filters (Dict, optional): Filters to apply to the files.
|
|
112
155
|
|
|
113
156
|
Returns:
|
|
114
|
-
int
|
|
157
|
+
int: The number of processed files.
|
|
115
158
|
"""
|
|
116
159
|
try:
|
|
117
|
-
# Si no se proporcionaron filtros, usar el predeterminado
|
|
118
160
|
if not filters:
|
|
119
161
|
filters = {"filename_contains": ".pdf"}
|
|
120
162
|
|
|
121
163
|
# Pasar metadata predefinida como parte del contexto al procesador
|
|
122
|
-
# para que esté disponible en la función
|
|
123
|
-
|
|
164
|
+
# para que esté disponible en la función load_file_callback
|
|
165
|
+
context = {
|
|
166
|
+
'company': company,
|
|
167
|
+
'metadata': {}
|
|
168
|
+
}
|
|
169
|
+
|
|
124
170
|
if predefined_metadata:
|
|
125
|
-
|
|
171
|
+
context['metadata'] = predefined_metadata
|
|
126
172
|
|
|
127
173
|
# config the processor
|
|
128
174
|
processor_config = FileProcessorConfig(
|
|
129
|
-
|
|
175
|
+
callback=self.load_file_callback,
|
|
176
|
+
context=context,
|
|
130
177
|
filters=filters,
|
|
131
|
-
action=self.load_file,
|
|
132
178
|
continue_on_error=True,
|
|
133
179
|
echo=True
|
|
134
180
|
)
|
|
@@ -144,14 +190,21 @@ class LoadDocumentsService:
|
|
|
144
190
|
logging.exception("Loading files error: %s", str(e))
|
|
145
191
|
return {"error": str(e)}
|
|
146
192
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
193
|
+
def load_file_callback(self, company: Company, filename: str, content: bytes, context: dict = {}):
|
|
194
|
+
"""
|
|
195
|
+
Processes a single file: extracts text, generates metadata, and saves it
|
|
196
|
+
to the relational database and the vector store.
|
|
197
|
+
This method is intended to be used as the 'action' for FileProcessor.
|
|
198
|
+
|
|
199
|
+
Args:
|
|
200
|
+
company (Company): The company associated with the file.
|
|
201
|
+
filename (str): The name of the file.
|
|
202
|
+
content (bytes): The binary content of the file.
|
|
203
|
+
context (dict, optional): A context dictionary, may contain predefined metadata.
|
|
204
|
+
"""
|
|
152
205
|
|
|
153
206
|
# check if file exist in repositories
|
|
154
|
-
if self.doc_repo.get(
|
|
207
|
+
if self.doc_repo.get(company_id=company.id,filename=filename):
|
|
155
208
|
return
|
|
156
209
|
|
|
157
210
|
try:
|
iatoolkit-0.4.0/readme.md
DELETED
|
@@ -1,43 +0,0 @@
|
|
|
1
|
-
# iatoolkit
|
|
2
|
-
|
|
3
|
-
IAToolkit is a comprehensive, open-source framework designed for building enterprise-grade
|
|
4
|
-
AI chatbots and conversational applications.
|
|
5
|
-
Built on Flask with dependency injection, it provides a robust foundation for scalable AI solutions.
|
|
6
|
-
|
|
7
|
-
## 🚀 Key Features
|
|
8
|
-
- **Universal LLM Integration**: OpenAI GPT, Google Gemini
|
|
9
|
-
- **Template System**: Jinja2-powered prompt templates with variables
|
|
10
|
-
- **Context Management**: Maintain conversation context across sessions
|
|
11
|
-
|
|
12
|
-
### 🔒 **Enterprise Security**
|
|
13
|
-
- **JWT Authentication**: Secure token-based authentication
|
|
14
|
-
- **Session Management**: Redis-backed secure sessions
|
|
15
|
-
- **CORS Configuration**: Flexible cross-origin resource sharing
|
|
16
|
-
|
|
17
|
-
### 🛠 **Function Calling & Tools**
|
|
18
|
-
- **Native Function Calls**: Direct integration with LLM function calling
|
|
19
|
-
- **Custom Tools**: Build and register custom tools for your chatbot
|
|
20
|
-
- **SQL Query Generation**: Natural language to SQL conversion
|
|
21
|
-
- **API Integrations**: Connect to external services and APIs
|
|
22
|
-
|
|
23
|
-
### 🗄 **Database & Storage**
|
|
24
|
-
- **Multi-Database Support**: PostgreSQL, MySQL, SQLite via SQLAlchemy
|
|
25
|
-
- **Vector Store Integration**: Semantic search and retrieval
|
|
26
|
-
- **Document Processing**: PDF, Word, Excel, and text file handling
|
|
27
|
-
|
|
28
|
-
### 📊 **Analytics & Monitoring**
|
|
29
|
-
- **Query Logging**: Track all LLM interactions
|
|
30
|
-
- **Performance Metrics**: Response times, token usage, costs
|
|
31
|
-
- **Benchmarking**: Compare model performance
|
|
32
|
-
- **Task Management**: Async task processing with status tracking
|
|
33
|
-
|
|
34
|
-
### 🔧 **Developer Experience**
|
|
35
|
-
- **Dependency Injection**: Clean, testable architecture
|
|
36
|
-
- **CLI Tools**: Command-line interface for common tasks
|
|
37
|
-
- **Hot Reloading**: Development-friendly configuration
|
|
38
|
-
- **Comprehensive Logging**: Debug and monitor easily
|
|
39
|
-
|
|
40
|
-
## License
|
|
41
|
-
MIT License
|
|
42
|
-
|
|
43
|
-
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|