shareddata 4.10.3__tar.gz → 6.83.4__tar.gz

shareddata-6.83.4/MANIFEST.in

@@ -0,0 +1,2 @@
+include src/SharedData/sharedmutexwin.pyd
+include src/SharedData/IO/apidocs.yml

shareddata-6.83.4/PKG-INFO

@@ -0,0 +1,195 @@
+Metadata-Version: 2.4
+Name: shareddata
+Version: 6.83.4
+Summary: Memory Mapped / Shared Memory Database with S3 repository
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9, <3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ipykernel==6.23.3
+Requires-Dist: boto3==1.26.160
+Requires-Dist: python-json-logger==2.0.7
+Requires-Dist: python-dotenv==1.0.1
+Requires-Dist: pandas<3,>=2.2
+Requires-Dist: scipy==1.14.0
+Requires-Dist: numpy==1.26.2
+Requires-Dist: numba==0.59.1
+Requires-Dist: XlsxWriter==3.1.2
+Requires-Dist: openpyxl==3.1.2
+Requires-Dist: tqdm==4.65.0
+Requires-Dist: cffi==1.17.1
+Requires-Dist: tzlocal==5.0.1
+Requires-Dist: websockets==12.0
+Requires-Dist: cryptography==41.0.7
+Requires-Dist: lz4==4.3.3
+Requires-Dist: flask==3.0.0
+Requires-Dist: waitress==3.0.0
+Requires-Dist: requests==2.31.0
+Requires-Dist: flasgger==0.9.7.1
+Requires-Dist: pymongo==4.8.0
+Requires-Dist: setuptools==74.1.2
+Requires-Dist: filelock==3.18.0
+Requires-Dist: gunicorn==23.0.0
+Requires-Dist: confluent-kafka==2.10.0
+Requires-Dist: aiokafka==0.12.0
+Requires-Dist: aiohttp==3.12.13
+Requires-Dist: redis==6.2.0
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# SharedData
+
+A comprehensive ultrafast Python library for financial data.
+
+## 📖 Table of Contents
+
+- [🏗️ Core Features](#-core-features)
+- [⚡ Quick Start](#-quick-start)
+- [🔧 Configuration](#-configuration)
+- [🚀 Advanced Usage](#-advanced-usage)
+- [🔄 Development & Documentation](#-development--documentation)
+- [📄 License](#-license)
+- [🤝 Contributing](#-contributing)
+
+## 🏗️ Core Features
+
+SharedData provides a comprehensive set of features for high-performance financial data management:
+
+- **🗃️ Database Schema & Indexing** - Optimized schemas for financial data types
+- **🌐 Storage & Integration** - Multi-storage support (Local, S3, MongoDB, Redis)
+- **📈 Performance & Scalability** - Parallel processing and advanced querying
+- **📊 Data Containers** - Tables, Collections, Time Series, Streams, Cache, Metadata
+- **⚡ Multiprocessing & Parallel Computing** - Sophisticated parallel processing library
+- **🤖 Distributed Worker System** - Automated task execution and job scheduling
+- **📋 Comprehensive Logging System** - Enterprise-grade logging with multiple destinations
+- **🌐 Remote API Client** - REST API for remote data access and operations
+
+**📚 [Read the complete Core Features guide →](docs/CORE_FEATURES.md)**
+
+## ⚡ Quick Start
+
+Get up and running with SharedData in minutes:
+
+### Installation
+
+```bash
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate
+
+# Install SharedData
+pip install -r requirements.txt
+pip install -e .
+```
+
+### Basic Usage
+
+```python
+import pandas as pd
+from SharedData.SharedData import SharedData
+
+# Initialize SharedData
+shdata = SharedData(__file__, user='master')
+
+# Quick example - Tables
+dates = pd.date_range('2025-01-01', '2025-01-10', freq='D')
+symbols = ['AAPL', 'GOOGL', 'MSFT']
+idx = pd.MultiIndex.from_product([dates, symbols], names=['date', 'symbol'])
+df = pd.DataFrame({'price': 100, 'volume': 1000}, index=idx)
+
+# Write and read data
+tbl = shdata.table('MarketData', 'D1', 'TEST', 'PRICES', value=df)
+tbl.write()
+data = tbl.loc['2025-01-05':, 'AAPL']  # Fast symbol lookup
+print(f"Retrieved {data.shape[0]} rows")
+```
+
+**📚 [Read the complete Quick Start guide →](docs/QUICK_START.md)**
+
+## 🔧 Configuration
+
+Configure SharedData with environment variables for your specific needs:
+
+```bash
+# Required Variables
+SHAREDDATA_SECRET_KEY="your-secret-key"
+SHAREDDATA_TOKEN="your-auth-token"
+AWS_ACCESS_KEY_ID="your-aws-access-key"
+S3_BUCKET="s3://your-bucket-name"
+MONGODB_HOST="your-mongodb-host"
+SHAREDDATA_ENDPOINT="http://your-server:port"
+```
+
+**📚 [Read the complete Configuration guide →](docs/CONFIGURATION.md)**
+
+## 🚀 Advanced Usage
+
+For power users and complex scenarios:
+
+- **Advanced Data Operations** - Complex queries and aggregations
+- **Performance Optimization** - Memory management and parallel processing
+- **Distributed Computing** - Worker pools and distributed data processing
+- **Custom Extensions** - Custom data containers and worker types
+- **Production Deployment** - High availability and monitoring
+- **Monitoring & Debugging** - Performance profiling and health checks
+
+**📚 [Read the complete Advanced Usage guide →](docs/ADVANCED_USAGE.md)**
+
+## 🔄 Development & Documentation
+
+For developers contributing to SharedData:
+
+### Building Documentation
+
+```bash
+# Generate documentation from docstrings
+make gitea
+
+# Alternative methods
+python generate_docs.py
+python update_docs.py
+make all
+```
+
+### Running Tests
+
+```bash
+python -m pytest tests/
+```
+
+**📚 [Read the complete Development guide →](docs/DEVELOPMENT.md)**
+
+## 📄 License
+
+See [LICENSE](LICENSE) file for details.
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Update documentation: `make gitea`
+5. Submit a pull request
+
+**📚 [Read the complete Contributing guide →](docs/DEVELOPMENT.md#contributing-guidelines)**
+
+---
+
+## 📚 Documentation Index
+
+- **[🏗️ Core Features](docs/CORE_FEATURES.md)** - Comprehensive overview of all SharedData capabilities
+- **[⚡ Quick Start](docs/QUICK_START.md)** - Get started with SharedData in minutes
+- **[🔧 Configuration](docs/CONFIGURATION.md)** - Complete configuration reference
+- **[🚀 Advanced Usage](docs/ADVANCED_USAGE.md)** - Advanced patterns and optimization
+- **[🔄 Development](docs/DEVELOPMENT.md)** - Contributing and development guidelines
+
+---
+
+[⬆️ Back to top](#shareddata)

shareddata-6.83.4/README.md

@@ -0,0 +1,149 @@
+# SharedData
+
+A comprehensive ultrafast Python library for financial data.
+
+## 📖 Table of Contents
+
+- [🏗️ Core Features](#-core-features)
+- [⚡ Quick Start](#-quick-start)
+- [🔧 Configuration](#-configuration)
+- [🚀 Advanced Usage](#-advanced-usage)
+- [🔄 Development & Documentation](#-development--documentation)
+- [📄 License](#-license)
+- [🤝 Contributing](#-contributing)
+
+## 🏗️ Core Features
+
+SharedData provides a comprehensive set of features for high-performance financial data management:
+
+- **🗃️ Database Schema & Indexing** - Optimized schemas for financial data types
+- **🌐 Storage & Integration** - Multi-storage support (Local, S3, MongoDB, Redis)
+- **📈 Performance & Scalability** - Parallel processing and advanced querying
+- **📊 Data Containers** - Tables, Collections, Time Series, Streams, Cache, Metadata
+- **⚡ Multiprocessing & Parallel Computing** - Sophisticated parallel processing library
+- **🤖 Distributed Worker System** - Automated task execution and job scheduling
+- **📋 Comprehensive Logging System** - Enterprise-grade logging with multiple destinations
+- **🌐 Remote API Client** - REST API for remote data access and operations
+
+**📚 [Read the complete Core Features guide →](docs/CORE_FEATURES.md)**
+
+## ⚡ Quick Start
+
+Get up and running with SharedData in minutes:
+
+### Installation
+
+```bash
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate
+
+# Install SharedData
+pip install -r requirements.txt
+pip install -e .
+```
+
+### Basic Usage
+
+```python
+import pandas as pd
+from SharedData.SharedData import SharedData
+
+# Initialize SharedData
+shdata = SharedData(__file__, user='master')
+
+# Quick example - Tables
+dates = pd.date_range('2025-01-01', '2025-01-10', freq='D')
+symbols = ['AAPL', 'GOOGL', 'MSFT']
+idx = pd.MultiIndex.from_product([dates, symbols], names=['date', 'symbol'])
+df = pd.DataFrame({'price': 100, 'volume': 1000}, index=idx)
+
+# Write and read data
+tbl = shdata.table('MarketData', 'D1', 'TEST', 'PRICES', value=df)
+tbl.write()
+data = tbl.loc['2025-01-05':, 'AAPL']  # Fast symbol lookup
+print(f"Retrieved {data.shape[0]} rows")
+```
+
+**📚 [Read the complete Quick Start guide →](docs/QUICK_START.md)**
+
+## 🔧 Configuration
+
+Configure SharedData with environment variables for your specific needs:
+
+```bash
+# Required Variables
+SHAREDDATA_SECRET_KEY="your-secret-key"
+SHAREDDATA_TOKEN="your-auth-token"
+AWS_ACCESS_KEY_ID="your-aws-access-key"
+S3_BUCKET="s3://your-bucket-name"
+MONGODB_HOST="your-mongodb-host"
+SHAREDDATA_ENDPOINT="http://your-server:port"
+```
+
+**📚 [Read the complete Configuration guide →](docs/CONFIGURATION.md)**
+
+## 🚀 Advanced Usage
+
+For power users and complex scenarios:
+
+- **Advanced Data Operations** - Complex queries and aggregations
+- **Performance Optimization** - Memory management and parallel processing
+- **Distributed Computing** - Worker pools and distributed data processing
+- **Custom Extensions** - Custom data containers and worker types
+- **Production Deployment** - High availability and monitoring
+- **Monitoring & Debugging** - Performance profiling and health checks
+
+**📚 [Read the complete Advanced Usage guide →](docs/ADVANCED_USAGE.md)**
+
+## 🔄 Development & Documentation
+
+For developers contributing to SharedData:
+
+### Building Documentation
+
+```bash
+# Generate documentation from docstrings
+make gitea
+
+# Alternative methods
+python generate_docs.py
+python update_docs.py
+make all
+```
+
+### Running Tests
+
+```bash
+python -m pytest tests/
+```
+
+**📚 [Read the complete Development guide →](docs/DEVELOPMENT.md)**
+
+## 📄 License
+
+See [LICENSE](LICENSE) file for details.
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Update documentation: `make gitea`
+5. Submit a pull request
+
+**📚 [Read the complete Contributing guide →](docs/DEVELOPMENT.md#contributing-guidelines)**
+
+---
+
+## 📚 Documentation Index
+
+- **[🏗️ Core Features](docs/CORE_FEATURES.md)** - Comprehensive overview of all SharedData capabilities
+- **[⚡ Quick Start](docs/QUICK_START.md)** - Get started with SharedData in minutes
+- **[🔧 Configuration](docs/CONFIGURATION.md)** - Complete configuration reference
+- **[🚀 Advanced Usage](docs/ADVANCED_USAGE.md)** - Advanced patterns and optimization
+- **[🔄 Development](docs/DEVELOPMENT.md)** - Contributing and development guidelines
+
+---
+
+[⬆️ Back to top](#shareddata)

shareddata-6.83.4/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = [
+    "setuptools>=42",
+    "wheel"
+]
+build-backend = "setuptools.build_meta"
+
+[tool.pytest.ini_options]
+# Configure pytest to suppress asyncio warnings
+asyncio_default_fixture_loop_scope = "function"
+pythonpath = ["src"]
+filterwarnings = [
+    "ignore::pytest.PytestDeprecationWarning",
+    "ignore::pytest.PytestReturnNotNoneWarning"
+]
+# Add markers for test categories
+markers = [
+    "crud: marks tests as CRUD operations",
+    "mutex: marks tests as mutex functionality tests",
+    "multiprocess: marks tests as multi-process operations",
+    "integration: marks tests as integration tests"
+]

shareddata-6.83.4/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

shareddata-6.83.4/setup.py

@@ -0,0 +1,51 @@
+import sys
+from setuptools import setup, find_packages
+
+install_requires = [
+    'ipykernel==6.23.3',
+    'boto3==1.26.160',
+    'python-json-logger==2.0.7',
+    'python-dotenv==1.0.1',
+    'pandas>=2.2,<3',
+    'scipy==1.14.0',
+    'numpy==1.26.2',
+    'numba==0.59.1',
+    'XlsxWriter==3.1.2',
+    'openpyxl==3.1.2',
+    'tqdm==4.65.0',
+    'cffi==1.17.1',
+    'tzlocal==5.0.1',
+    'websockets==12.0',
+    'cryptography==41.0.7',
+    'lz4==4.3.3',
+    'flask==3.0.0',
+    'waitress==3.0.0',
+    'requests==2.31.0',
+    'flasgger==0.9.7.1',
+    'pymongo==4.8.0',
+    'setuptools==74.1.2',
+    'filelock==3.18.0',
+    'gunicorn==23.0.0',
+    'confluent-kafka==2.10.0',
+    'aiokafka==0.12.0',
+    'aiohttp==3.12.13',
+    'redis==6.2.0',
+]
+
+setup(
+    name='shareddata',
+    version='6.83.4',
+    description='Memory Mapped / Shared Memory Database with S3 repository',
+    long_description=open('README.md').read(),
+    long_description_content_type='text/markdown',
+    classifiers=[
+        'Programming Language :: Python :: 3',
+        'License :: OSI Approved :: MIT License',
+        'Operating System :: OS Independent',
+    ],
+    packages=find_packages(where='src'),
+    package_dir={'': 'src'},
+    python_requires='>=3.9, <3.13',
+    install_requires=install_requires,
+    include_package_data=True,
+)

shareddata-6.83.4/src/SharedData/API/ServerGunicorn.py

@@ -0,0 +1,162 @@
+"""
+SharedData API Server - Production Entry Point (Gunicorn)
+
+This module provides the production server for the SharedData API using Gunicorn.
+For development/testing, use ServerAPI.py which runs with Waitress.
+
+Usage:
+    python -m SharedData.API.ServerGunicorn [--host HOST] [--port PORT] [--nproc NPROC] [--nthreads NTHREADS]
+"""
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+import threading
+import time
+from gunicorn.app.base import BaseApplication
+
+from SharedData.Logger import Logger
+Logger.connect('SharedData.API.ServerGunicorn')
+
+from SharedData.API import create_app
+
+# Create Flask app at module level for Gunicorn
+flask_app = create_app()
+
+
+class GunicornEmbedded(BaseApplication):
+    """
+    A Gunicorn application wrapper that runs Gunicorn within the current Python interpreter.
+    
+    This class allows embedding Gunicorn server functionality directly into a Python process,
+    enabling programmatic control over Gunicorn configuration and execution without spawning
+    a separate Gunicorn process.
+    
+    Attributes:
+        _wsgi_app: The WSGI application to be served by Gunicorn.
+        _options: A dictionary of Gunicorn configuration options.
+    
+    Example:
+        GunicornEmbedded(flask_app, {"bind": "0.0.0.0:8002", "workers": 4}).run()
+    """
+    def __init__(self, wsgi_app, options: dict[str, str | int]):
+        """
+        Initializes the instance with a WSGI application and configuration options.
+        
+        Parameters:
+            wsgi_app: The WSGI application callable to be wrapped or used.
+            options: A dictionary of Gunicorn configuration options.
+        """
+        self._wsgi_app = wsgi_app
+        self._options = options or {}
+        super().__init__()
+
+    def load_config(self):
+        """
+        Loads configuration settings from a dictionary of options into the Gunicorn configuration.
+        
+        For each key-value pair in the provided options, this method sets the corresponding
+        configuration value in the Gunicorn configuration object, provided that the key is valid.
+        """
+        for key, value in self._options.items():
+            if key in self.cfg.settings and value is not None:
+                self.cfg.set(key.lower(), value)
+
+    def load(self):
+        """
+        Returns the WSGI application instance.
+        
+        Returns:
+            The WSGI application instance stored in this object.
+        """
+        return self._wsgi_app
+
+
+def parse_cli():
+    """
+    Parse CLI arguments when running as a stand-alone script.
+    
+    Returns:
+        Parsed arguments namespace.
+    """
+    p = argparse.ArgumentParser(
+        description="Run SharedData API with embedded Gunicorn"
+    )
+    p.add_argument("--host", default="0.0.0.0", help="Listen address")
+    p.add_argument("--port", type=int, default=8002, help="Listen port")
+    p.add_argument("--nproc", type=int, default=4,
+                   help="Number of Gunicorn worker processes")
+    p.add_argument("--nthreads", type=int, default=8,
+                   help="Number of request threads *per* worker (gthread)")
+    p.add_argument("--timeout", type=int, default=120,
+                   help="Hard kill after N seconds")
+    p.add_argument("--log-level", default="warning")
+    return p.parse_args()
+
+
+def send_heartbeat():    
+    """
+    Sends periodic heartbeat signals to indicate that the routine is running.
+    
+    The function waits for an initial 15 seconds before logging a start message. 
+    It then enters an infinite loop where it logs a debug heartbeat message every 
+    15 seconds to signal ongoing activity.
+    """
+    heartbeat_interval = 15
+    time.sleep(15)
+    Logger.log.info('ROUTINE STARTED!')
+    while True:
+        Logger.log.debug('#heartbeat#')
+        time.sleep(heartbeat_interval)
+        
+        
+def start_background_threads():    
+    """
+    Starts a background daemon thread that continuously emits heartbeat metrics.
+    """
+    heartbeat_thread = threading.Thread(target=send_heartbeat, daemon=True)
+    heartbeat_thread.start()
+
+
+def main():
+    """
+    Entry point for the application. Parses command-line arguments, starts necessary 
+    background helper threads, configures Gunicorn server options based on the parsed 
+    arguments, and launches the Gunicorn server embedded within the process to serve 
+    the Flask application.
+    """
+    args = parse_cli()
+    
+    # Launch in-process helper threads
+    start_background_threads()
+
+    # Assemble Gunicorn config
+    gunicorn_opts = {
+        "bind":            f"{args.host}:{args.port}",
+        "workers":         args.nproc,
+        "worker_class":    "gthread",      
+        "threads":         args.nthreads,
+        "timeout":         args.timeout,
+        "graceful_timeout": 30,
+        "preload_app":     False,           
+        "accesslog":       None,  
+        "errorlog":        "-",
+        "loglevel":        args.log_level,
+        "max_requests":    5000,           # recycle workers → tame leaks
+        "max_requests_jitter": 500,
+        "limit_request_line": 8190,       # 8190 is HTTP/1.1 spec max        
+    }
+
+    # Block here; Gunicorn handles signals and will exit cleanly
+    GunicornEmbedded(flask_app, gunicorn_opts).run()
+
+
+if __name__ == "__main__":
+    # Basic env-var sanity checks
+    required_env = ("SHAREDDATA_SECRET_KEY", "SHAREDDATA_TOKEN")
+    missing = [v for v in required_env if v not in os.environ]
+    if missing:
+        sys.exit(f"Missing environment variables: {', '.join(missing)}")
+
+    main()

shareddata-6.83.4/src/SharedData/API/ServerWaitress.py

@@ -0,0 +1,62 @@
+"""
+SharedData API Server - Development Entry Point
+
+This module provides the development server for the SharedData API.
+For production, use ServerHttp.py which runs with Gunicorn.
+
+Usage:
+    python -m SharedData.API.ServerWaitress [--host HOST] [--port PORT]
+"""
+
+import os
+import sys
+import argparse
+
+from waitress import serve
+
+from SharedData.Logger import Logger
+
+
+def parse_args():
+    """Parse command line arguments."""
+    parser = argparse.ArgumentParser(description='SharedData API Development Server')
+    parser.add_argument('--host', type=str, default='0.0.0.0',
+                        help='Host to bind to (default: 0.0.0.0)')
+    parser.add_argument('--port', type=int, default=8002,
+                        help='Port to bind to (default: 8002)')
+    parser.add_argument('--threads', type=int, default=4,
+                        help='Number of worker threads (default: 4)')
+    return parser.parse_args()
+
+
+def validate_environment():
+    """Validate required environment variables are set."""
+    required_env = ('SHAREDDATA_SECRET_KEY', 'SHAREDDATA_TOKEN')
+    missing = [v for v in required_env if v not in os.environ]
+    if missing:
+        sys.exit(f"Missing environment variables: {', '.join(missing)}")
+
+
+def main():
+    """Main entry point for the development server."""
+    # Validate environment
+    validate_environment()
+    
+    # Parse arguments
+    args = parse_args()
+    
+    # Setup logging
+    Logger.connect('SharedData.API.ServerWaitress')
+    Logger.log.info(f'Starting SharedData API development server on {args.host}:{args.port}')
+    
+    # Import and create app (after logging is setup)
+    from SharedData.API import create_app
+    app = create_app()
+    
+    # Run with Waitress (production-quality WSGI server)
+    Logger.log.info(f'Server ready - http://{args.host}:{args.port}')
+    serve(app, host=args.host, port=args.port, threads=args.threads)
+
+
+if __name__ == '__main__':
+    main()