isa-data 1.0.0__tar.gz

isa_data-1.0.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: isa-data
+Version: 1.0.0
+Summary: isA_Data — data analytics, data-fabric, and data-product SDK for the isA platform
+Author: Data Analytics Team
+License: Proprietary
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: fastapi>=0.104.1
+Requires-Dist: uvicorn[standard]>=0.24.0
+Requires-Dist: python-multipart>=0.0.9
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: asyncpg>=0.29.0
+Requires-Dist: psycopg[binary]>=3.0.0
+Requires-Dist: sqlalchemy[asyncio]>=2.0
+Requires-Dist: duckdb>=0.9.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: polars-lts-cpu>=0.20.0
+Requires-Dist: pyarrow>=14.0.0
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: matplotlib>=3.8.0
+Requires-Dist: httpx>=0.25.2
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: structlog>=23.2.0
+Requires-Dist: python-logging-loki>=0.3.1
+Requires-Dist: prometheus-client>=0.19.0
+Requires-Dist: feast<0.50,>=0.40
+Requires-Dist: redis>=5.0.0
+Requires-Dist: isa-common>=0.6.0
+Requires-Dist: isa-model[cloud]>=0.5.7
+Provides-Extra: bigdata
+Requires-Dist: deltalake>=1.2.0; extra == "bigdata"
+Requires-Dist: minio>=7.2.0; extra == "bigdata"
+Requires-Dist: pyiceberg[hive]>=0.7.0; extra == "bigdata"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: pytest-aiohttp>=1.0.0; extra == "dev"
+Requires-Dist: aiosqlite>=0.19.0; extra == "dev"
+Requires-Dist: papermill>=2.4.0; extra == "dev"
+Requires-Dist: nbformat>=5.10; extra == "dev"
+Requires-Dist: nbclient>=0.10; extra == "dev"
+Requires-Dist: jupyter-client>=8.0.0; extra == "dev"
+Requires-Dist: ipykernel>=6.29; extra == "dev"
+
+# isA_Data
+
+**Cloud Data Analytics & Processing Center**
+
+## 🎯 Purpose
+
+独立的数据分析云服务，处理数据处理、分析、可视化等任务。
+从 `isA_MCP/tools/services/data_analytics_service` 迁移而来，提供 API 服务。
+
+## 🏗️ Architecture
+
+```
+isA_Data/
+├── app/
+│   ├── main.py                    # FastAPI 主服务
+│   ├── config.py                  # 配置管理
+│   └── api/
+│       ├── analytics.py           # 数据分析端点
+│       ├── processing.py          # 数据处理端点
+│       └── visualization.py       # 数据可视化端点
+│
+├── services/
+│   ├── analytics/                 # [迁移自 isA_MCP]
+│   │   ├── pandas_ops.py          # Pandas 操作
+│   │   ├── statistical.py         # 统计分析
+│   │   └── ml_analysis.py         # ML 分析
+│   │
+│   ├── processing/
+│   │   ├── etl.py                 # ETL 处理
+│   │   ├── cleaning.py            # 数据清洗
+│   │   └── transformation.py      # 数据转换
+│   │
+│   └── storage/
+│       ├── database.py            # 数据库操作
+│       └── cache.py               # 缓存管理
+│
+├── core/
+│   ├── logging.py
+│   └── clients/
+│       └── model_client.py        # ISA Model 客户端
+│
+├── deployment/
+│   ├── docker/
+│   │   └── Dockerfile
+│   └── kubernetes/
+│       └── deployment.yaml
+│
+├── tests/
+│   └── test_analytics.py
+│
+├── requirements.txt
+└── README.md
+```
+
+## 📡 API Endpoints
+
+### Data Analytics
+```
+POST /api/v1/data/analyze
+POST /api/v1/data/statistics
+POST /api/v1/data/ml/predict
+```
+
+### Data Processing
+```
+POST /api/v1/data/clean
+POST /api/v1/data/transform
+POST /api/v1/data/aggregate
+```
+
+### Data Visualization
+```
+POST /api/v1/data/visualize/chart
+POST /api/v1/data/visualize/dashboard
+```
+
+## 🚀 Usage
+
+### As Standalone Service
+```bash
+# Start server
+uvicorn app.main:app --host 0.0.0.0 --port 8001
+
+# Or with Docker
+docker-compose up
+```
+
+### Called from isA_MCP
+```python
+# isA_MCP 中通过 API 调用
+from tools.services.data_analytics_service import ISA_Data_Client
+
+client = ISA_Data_Client(base_url="http://localhost:8001")
+result = await client.analyze_data(
+    data=df,
+    analysis_type="descriptive"
+)
+```
+
+## 💡 Benefits
+
+1. **专注**: 专门处理数据相关任务
+2. **性能**: 可以部署在大内存机器上处理大数据
+3. **隔离**: 数据处理故障不影响 MCP 主服务
+4. **可复用**: 多个服务都可以调用数据分析能力
+5. **扩展**: 未来可以添加更多数据源和分析能力
+
+## 🔄 Migration Status
+
+- [ ] Phase 1: 基础架构搭建
+- [ ] Phase 2: 迁移 data_analytics 模块
+- [ ] Phase 3: FastAPI 端点实现
+- [ ] Phase 4: 数据库集成
+- [ ] Phase 5: isA_MCP 集成测试
+- [ ] Phase 6: 生产部署
+
+## 🔗 Related Projects
+
+- **isA_MCP**: MCP 工具层（API 调用端）
+- **isA_OS**: Web & OS 操作云服务
+- **isA_Model**: 模型服务

isa_data-1.0.0/README.md

@@ -0,0 +1,121 @@
+# isA_Data
+
+**Cloud Data Analytics & Processing Center**
+
+## 🎯 Purpose
+
+独立的数据分析云服务，处理数据处理、分析、可视化等任务。
+从 `isA_MCP/tools/services/data_analytics_service` 迁移而来，提供 API 服务。
+
+## 🏗️ Architecture
+
+```
+isA_Data/
+├── app/
+│   ├── main.py                    # FastAPI 主服务
+│   ├── config.py                  # 配置管理
+│   └── api/
+│       ├── analytics.py           # 数据分析端点
+│       ├── processing.py          # 数据处理端点
+│       └── visualization.py       # 数据可视化端点
+│
+├── services/
+│   ├── analytics/                 # [迁移自 isA_MCP]
+│   │   ├── pandas_ops.py          # Pandas 操作
+│   │   ├── statistical.py         # 统计分析
+│   │   └── ml_analysis.py         # ML 分析
+│   │
+│   ├── processing/
+│   │   ├── etl.py                 # ETL 处理
+│   │   ├── cleaning.py            # 数据清洗
+│   │   └── transformation.py      # 数据转换
+│   │
+│   └── storage/
+│       ├── database.py            # 数据库操作
+│       └── cache.py               # 缓存管理
+│
+├── core/
+│   ├── logging.py
+│   └── clients/
+│       └── model_client.py        # ISA Model 客户端
+│
+├── deployment/
+│   ├── docker/
+│   │   └── Dockerfile
+│   └── kubernetes/
+│       └── deployment.yaml
+│
+├── tests/
+│   └── test_analytics.py
+│
+├── requirements.txt
+└── README.md
+```
+
+## 📡 API Endpoints
+
+### Data Analytics
+```
+POST /api/v1/data/analyze
+POST /api/v1/data/statistics
+POST /api/v1/data/ml/predict
+```
+
+### Data Processing
+```
+POST /api/v1/data/clean
+POST /api/v1/data/transform
+POST /api/v1/data/aggregate
+```
+
+### Data Visualization
+```
+POST /api/v1/data/visualize/chart
+POST /api/v1/data/visualize/dashboard
+```
+
+## 🚀 Usage
+
+### As Standalone Service
+```bash
+# Start server
+uvicorn app.main:app --host 0.0.0.0 --port 8001
+
+# Or with Docker
+docker-compose up
+```
+
+### Called from isA_MCP
+```python
+# isA_MCP 中通过 API 调用
+from tools.services.data_analytics_service import ISA_Data_Client
+
+client = ISA_Data_Client(base_url="http://localhost:8001")
+result = await client.analyze_data(
+    data=df,
+    analysis_type="descriptive"
+)
+```
+
+## 💡 Benefits
+
+1. **专注**: 专门处理数据相关任务
+2. **性能**: 可以部署在大内存机器上处理大数据
+3. **隔离**: 数据处理故障不影响 MCP 主服务
+4. **可复用**: 多个服务都可以调用数据分析能力
+5. **扩展**: 未来可以添加更多数据源和分析能力
+
+## 🔄 Migration Status
+
+- [ ] Phase 1: 基础架构搭建
+- [ ] Phase 2: 迁移 data_analytics 模块
+- [ ] Phase 3: FastAPI 端点实现
+- [ ] Phase 4: 数据库集成
+- [ ] Phase 5: isA_MCP 集成测试
+- [ ] Phase 6: 生产部署
+
+## 🔗 Related Projects
+
+- **isA_MCP**: MCP 工具层（API 调用端）
+- **isA_OS**: Web & OS 操作云服务
+- **isA_Model**: 模型服务

isa_data-1.0.0/isa_data/__init__.py

@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+"""
+Data Analytics Service Package
+
+A comprehensive data analytics service implementing the MCP architecture.
+Provides tools and services for data extraction, processing, and analysis.
+"""
+
+# Package metadata
+__version__ = "1.0.0"
+__author__ = "Data Analytics Team"
+
+# Core functionality - only import what actually exists and works
+try:
+    from .models.dto.base_models import AssetType
+
+    # Only import working components
+    __all__ = ["AssetType"]
+
+except ImportError as e:
+    # Graceful fallback if imports fail
+    __all__ = []
+    print(f"Warning: Some data analytics components not available: {e}")
+
+
+def get_version():
+    """Get package version."""
+    return __version__

isa_data-1.0.0/isa_data/adapters/__init__.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+"""
+Database, file, and graph adapters for metadata extraction
+"""
+
+from .source_adapters import (
+    DatabaseAdapter,
+    PostgreSQLAdapter,
+    MySQLAdapter,
+    SQLServerAdapter,
+)
+from .file_adapters import FileAdapter, ExcelAdapter, CSVAdapter, DocumentAdapter
+
+# Graph adapters not available
+# from .graph_adapters import Neo4jAdapter
+
+__all__ = [
+    "DatabaseAdapter",
+    "PostgreSQLAdapter",
+    "MySQLAdapter",
+    "SQLServerAdapter",
+    "FileAdapter",
+    "ExcelAdapter",
+    "CSVAdapter",
+    "DocumentAdapter",
+    # "Neo4jAdapter"  # Not available
+]

isa_data-1.0.0/isa_data/adapters/base/__init__.py

@@ -0,0 +1,14 @@
+"""
+isa_data.adapters.base
+======================
+
+The IDataSource/IDataTarget adapter family (distinct from the metadata-extractor
+``source_adapters`` family). ``BaseAdapter`` provides the common lifecycle
+(initialize/health_check/capabilities/stats); ``UnifiedDatabaseAdapter`` is the
+concrete async SQL DataSource/DataTarget over asyncpg/aiomysql/aiosqlite.
+"""
+
+from .base_adapter import BaseAdapter
+from .database_adapter import UnifiedDatabaseAdapter
+
+__all__ = ["BaseAdapter", "UnifiedDatabaseAdapter"]

isa_data-1.0.0/isa_data/adapters/base/base_adapter.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""
+Base Adapter for Data Analytics Service
+
+Provides common functionality for all adapters.
+"""
+
+import asyncio
+import logging
+from abc import ABC, abstractmethod
+from typing import Dict, Any, Optional
+from datetime import datetime
+
+from isa_data.models.dto.interfaces import IAdapter
+from isa_data.models.dto.data_models import DataSource, DataTarget
+
+logger = logging.getLogger(__name__)
+
+class BaseAdapter(IAdapter, ABC):
+    """
+    Base adapter class providing common functionality
+    
+    All adapters should inherit from this class and implement
+    the abstract methods for their specific data source/target type.
+    """
+    
+    def __init__(self, name: str, adapter_type: str):
+        self.name = name
+        self.adapter_type = adapter_type
+        self.config: Dict[str, Any] = {}
+        self.is_initialized = False
+        self.last_health_check: Optional[datetime] = None
+        self.health_status = "unknown"
+        self.connection = None
+        
+        # Statistics tracking
+        self.stats = {
+            'total_operations': 0,
+            'successful_operations': 0,
+            'failed_operations': 0,
+            'last_operation_time': None,
+            'average_response_time': 0.0
+        }
+        
+        self.logger = logging.getLogger(f"{__name__}.{name}")
+    
+    async def initialize(self, config: Dict[str, Any]) -> bool:
+        """
+        Initialize adapter with configuration
+        
+        Args:
+            config: Adapter configuration
+            
+        Returns:
+            True if initialization successful
+        """
+        try:
+            self.config = config
+            self.logger.info(f"Initializing {self.name} adapter")
+            
+            # Validate configuration
+            if not await self._validate_config(config):
+                raise ValueError(f"Invalid configuration for {self.name} adapter")
+            
+            # Perform adapter-specific initialization
+            await self._initialize_adapter()
+            
+            self.is_initialized = True
+            self.logger.info(f"{self.name} adapter initialized successfully")
+            return True
+            
+        except Exception as e:
+            self.logger.error(f"Failed to initialize {self.name} adapter: {e}")
+            self.is_initialized = False
+            return False
+    
+    async def health_check(self) -> bool:
+        """
+        Check if adapter is healthy
+        
+        Returns:
+            True if adapter is healthy
+        """
+        try:
+            if not self.is_initialized:
+                self.health_status = "not_initialized"
+                return False
+            
+            # Perform adapter-specific health check
+            is_healthy = await self._perform_health_check()
+            
+            self.health_status = "healthy" if is_healthy else "unhealthy"
+            self.last_health_check = datetime.now()
+            
+            return is_healthy
+            
+        except Exception as e:
+            self.logger.error(f"Health check failed for {self.name} adapter: {e}")
+            self.health_status = "error"
+            self.last_health_check = datetime.now()
+            return False
+    
+    def get_capabilities(self) -> Dict[str, Any]:
+        """
+        Get adapter capabilities
+        
+        Returns:
+            Dictionary containing adapter capabilities
+        """
+        base_capabilities = {
+            'name': self.name,
+            'type': self.adapter_type,
+            'initialized': self.is_initialized,
+            'health_status': self.health_status,
+            'last_health_check': self.last_health_check.isoformat() if self.last_health_check else None,
+            'statistics': self.stats.copy()
+        }
+        
+        # Add adapter-specific capabilities
+        specific_capabilities = self._get_specific_capabilities()
+        base_capabilities.update(specific_capabilities)
+        
+        return base_capabilities
+    
+    def get_stats(self) -> Dict[str, Any]:
+        """Get adapter statistics"""
+        return self.stats.copy()
+    
+    def _update_stats(self, operation_time: float, success: bool):
+        """Update adapter statistics"""
+        self.stats['total_operations'] += 1
+        self.stats['last_operation_time'] = datetime.now().isoformat()
+        
+        if success:
+            self.stats['successful_operations'] += 1
+        else:
+            self.stats['failed_operations'] += 1
+        
+        # Update average response time
+        if self.stats['successful_operations'] > 0:
+            current_avg = self.stats['average_response_time']
+            total_successful = self.stats['successful_operations']
+            self.stats['average_response_time'] = (
+                (current_avg * (total_successful - 1) + operation_time) / total_successful
+            )
+    
+    async def _execute_with_stats(self, operation_name: str, operation_func, *args, **kwargs):
+        """Execute operation with statistics tracking"""
+        import time
+        start_time = time.time()
+        
+        try:
+            self.logger.debug(f"Executing {operation_name} operation")
+            result = await operation_func(*args, **kwargs)
+            
+            execution_time = time.time() - start_time
+            self._update_stats(execution_time, True)
+            
+            self.logger.debug(f"{operation_name} completed in {execution_time:.2f}s")
+            return result
+            
+        except Exception as e:
+            execution_time = time.time() - start_time
+            self._update_stats(execution_time, False)
+            
+            self.logger.error(f"{operation_name} failed after {execution_time:.2f}s: {e}")
+            raise
+    
+    # Abstract methods that must be implemented by subclasses
+    
+    @abstractmethod
+    async def _validate_config(self, config: Dict[str, Any]) -> bool:
+        """
+        Validate adapter configuration
+        
+        Args:
+            config: Configuration to validate
+            
+        Returns:
+            True if configuration is valid
+        """
+        pass
+    
+    @abstractmethod
+    async def _initialize_adapter(self) -> None:
+        """
+        Perform adapter-specific initialization
+        
+        This method should establish connections, validate credentials,
+        and perform any other setup required for the adapter.
+        """
+        pass
+    
+    @abstractmethod
+    async def _perform_health_check(self) -> bool:
+        """
+        Perform adapter-specific health check
+        
+        Returns:
+            True if adapter is healthy
+        """
+        pass
+    
+    @abstractmethod
+    def _get_specific_capabilities(self) -> Dict[str, Any]:
+        """
+        Get adapter-specific capabilities
+        
+        Returns:
+            Dictionary containing adapter-specific capabilities
+        """
+        pass
+    
+    # Utility methods
+    
+    def _ensure_initialized(self):
+        """Ensure adapter is initialized"""
+        if not self.is_initialized:
+            raise RuntimeError(f"{self.name} adapter is not initialized")
+    
+    async def _safe_close_connection(self):
+        """Safely close connection if it exists"""
+        if self.connection:
+            try:
+                if hasattr(self.connection, 'close'):
+                    close_method = getattr(self.connection, 'close')
+                    if asyncio.iscoroutinefunction(close_method):
+                        await close_method()
+                    else:
+                        close_method()
+                elif hasattr(self.connection, 'disconnect'):
+                    disconnect_method = getattr(self.connection, 'disconnect')
+                    if asyncio.iscoroutinefunction(disconnect_method):
+                        await disconnect_method()
+                    else:
+                        disconnect_method()
+            except Exception as e:
+                self.logger.warning(f"Error closing connection: {e}")
+            finally:
+                self.connection = None