e6data-python-connector 2.3.10rc8__tar.gz → 2.3.11__tar.gz

{e6data_python_connector-2.3.10rc8/e6data_python_connector.egg-info → e6data_python_connector-2.3.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: e6data-python-connector
-Version: 2.3.10rc8
+Version: 2.3.11
 Summary: Client for the e6data distributed SQL Engine.
 Home-page: https://github.com/e6x-labs/e6data-python-connector
 Author: e6data, Inc.
@@ -16,16 +16,17 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: sqlalchemy>=1.0.0
-Requires-Dist: future==1.0.0
-Requires-Dist: python-dateutil==2.9.0.post0
-Requires-Dist: pycryptodome==3.19.1
-Requires-Dist: pytz==2024.1
-Requires-Dist: thrift==0.20.0
-Requires-Dist: grpcio==1.65.1
-Requires-Dist: grpcio-tools
+Requires-Dist: future>=1.0.0
+Requires-Dist: python-dateutil>=2.9.0
+Requires-Dist: pycryptodome>=3.19.1
+Requires-Dist: pytz>=2024.1
+Requires-Dist: thrift>=0.20.0
+Requires-Dist: grpcio>=1.65.1
+Requires-Dist: grpcio-tools>=1.65.1
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -38,7 +39,7 @@ Dynamic: summary
 
 # e6data Python Connector
 
-![version](https://img.shields.io/badge/version-2.3.10rc8-blue.svg)
+![version](https://img.shields.io/badge/version-2.3.11-blue.svg)
 
 ## Introduction
 
@@ -75,6 +76,8 @@ Use your e6data Email ID as the username and your access token as the password.
 
 ```python
 from e6data_python_connector import Connection
+# For connection pooling (recommended for concurrent operations)
+from e6data_python_connector import ConnectionPool
 
 username = '<username>'  # Your e6data Email ID.
 password = '<password>'  # Access Token generated in the e6data console.
@@ -84,6 +87,7 @@ database = '<database>'  # Database to perform the query on.
 port = 80  # Port of the e6data engine.
 catalog_name = '<catalog_name>'
 
+# Single connection (for simple, single-threaded use)
 conn = Connection(
     host=host,
     port=port,
@@ -91,6 +95,17 @@ conn = Connection(
     database=database,
     password=password
 )
+
+# Or use connection pool (for concurrent/multi-threaded use)
+pool = ConnectionPool(
+    min_size=2,
+    max_size=10,
+    host=host,
+    port=port,
+    username=username,
+    database=database,
+    password=password
+)
 ```
 
 #### Connection Parameters
@@ -109,6 +124,7 @@ The `Connection` class supports the following parameters:
 | `secure` | bool | No | False | Enable SSL/TLS for secure connections |
 | `auto_resume` | bool | No | True | Automatically resume cluster if suspended |
 | `grpc_options` | dict | No | None | Additional gRPC configuration options |
+| `debug` | bool | No | False | Enable debug logging for troubleshooting |
 
 #### Secure Connection Example
 
@@ -439,8 +455,241 @@ For detailed migration instructions, see the [Migration Guide](docs/zero-downtim
 - Message size optimization for large queries
 
 ### Connection Management
-- Enable connection pooling for better resource utilization
+
+#### Connection Pooling
+
+The e6data Python connector now includes a built-in connection pool for efficient connection management and reuse across multiple threads. The `ConnectionPool` class provides:
+
+- **Thread-safe connection reuse**: Each thread automatically reuses its assigned connection
+- **Automatic lifecycle management**: Handles connection creation, health checks, and cleanup
+- **Overflow connections**: Creates temporary connections when pool is exhausted
+- **Connection health monitoring**: Automatic detection and replacement of broken connections
+- **Statistics tracking**: Monitor pool usage and performance
+
+##### Basic Connection Pool Usage
+
+```python
+from e6data_python_connector import ConnectionPool
+
+# Create a connection pool
+pool = ConnectionPool(
+    min_size=2,        # Minimum connections to maintain
+    max_size=10,       # Maximum connections in pool
+    max_overflow=5,    # Additional temporary connections allowed
+    timeout=30.0,      # Timeout for getting connection (seconds)
+    recycle=3600,      # Maximum age before recycling (seconds)
+    debug=False,       # Enable debug logging
+    pre_ping=True,     # Check connection health before use
+    # Connection parameters
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database,
+    catalog=catalog_name,
+    cluster_name=cluster_name,
+    secure=True
+)
+
+# Get connection and execute query
+conn = pool.get_connection()
+cursor = conn.cursor()
+cursor.execute("SELECT * FROM table")
+results = cursor.fetchall()
+
+# Return connection to pool (important!)
+pool.return_connection(conn)
+
+# Clean up when done
+pool.close_all()
+```
+
+##### Using Context Manager (Recommended)
+
+The context manager pattern ensures connections are automatically returned to the pool:
+
+```python
+from e6data_python_connector import ConnectionPool
+
+pool = ConnectionPool(
+    min_size=2,
+    max_size=10,
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database
+)
+
+# Connection automatically returned to pool after use
+with pool.get_connection_context() as conn:
+    cursor = conn.cursor()
+    cursor.execute("SELECT * FROM table")
+    results = cursor.fetchall()
+    print(results)
+```
+
+##### Concurrent Query Execution
+
+Connection pooling is especially beneficial for concurrent query execution:
+
+```python
+import concurrent.futures
+from e6data_python_connector import ConnectionPool
+
+def execute_query(pool, query_id, query):
+    """Execute a query using a pooled connection."""
+    # Each thread will reuse its assigned connection
+    conn = pool.get_connection()
+    try:
+        cursor = conn.cursor()
+        cursor.execute(query)
+        results = cursor.fetchall()
+        return f"Query {query_id}: {len(results)} rows"
+    finally:
+        pool.return_connection(conn)
+
+# Create pool
+pool = ConnectionPool(
+    min_size=3,
+    max_size=10,
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database
+)
+
+# Execute multiple queries concurrently
+queries = [
+    "SELECT COUNT(*) FROM table1",
+    "SELECT AVG(value) FROM table2",
+    "SELECT MAX(date) FROM table3"
+]
+
+with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
+    futures = [
+        executor.submit(execute_query, pool, i, query)
+        for i, query in enumerate(queries)
+    ]
+    
+    for future in concurrent.futures.as_completed(futures):
+        print(future.result())
+
+# Clean up
+pool.close_all()
+```
+
+##### Connection Pool Configuration
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `min_size` | int | 2 | Minimum number of connections to maintain |
+| `max_size` | int | 10 | Maximum number of connections in pool |
+| `max_overflow` | int | 5 | Additional temporary connections allowed |
+| `timeout` | float | 30.0 | Timeout for getting connection (seconds) |
+| `recycle` | int | 3600 | Maximum connection age before recycling (seconds) |
+| `debug` | bool | False | Enable debug logging for pool operations |
+| `pre_ping` | bool | True | Check connection health before returning from pool |
+
+##### Monitoring Pool Statistics
+
+```python
+# Get pool statistics
+stats = pool.get_statistics()
+print(f"Active connections: {stats['active_connections']}")
+print(f"Idle connections: {stats['idle_connections']}")
+print(f"Total requests: {stats['total_requests']}")
+print(f"Failed connections: {stats['failed_connections']}")
+```
+
+##### When to Use Connection Pooling
+
+Connection pooling is recommended when:
+- Executing multiple queries concurrently
+- Building web applications or APIs
+- Running batch processing jobs
+- Reducing connection overhead
+- Improving application performance
+
+##### Direct Connection Usage (Without Pool)
+
+For simple, single-threaded applications, you can still use direct connections:
+
+```python
+from e6data_python_connector import Connection
+
+conn = Connection(
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database
+)
+
+cursor = conn.cursor()
+cursor.execute("SELECT * FROM table")
+results = cursor.fetchall()
+conn.close()
+```
+
+#### Additional Connection Management Features
 - Automatic connection health monitoring
 - Graceful connection recovery and retry logic
+- Blue-green deployment support with automatic failover
+
+## Debugging and Troubleshooting
+
+### Enable Debug Mode
+
+Enable comprehensive debugging to troubleshoot connection and query issues:
+
+```python
+from e6data_python_connector import Connection
+
+conn = Connection(
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database,
+    debug=True  # Enable debug logging
+)
+```
+
+When `debug=True`, the following features are enabled:
+- Python logging at DEBUG level for all operations
+- Blue-green strategy transition logging
+- Connection lifecycle logging
+- Query execution detailed logging
+
+### gRPC Network Tracing
+
+For low-level gRPC network debugging (HTTP/2 frames, TCP events), set environment variables **before** running your Python script:
+
+```bash
+# Enable gRPC network tracing
+export GRPC_VERBOSITY=DEBUG
+export GRPC_TRACE=client_channel,http2
+
+# For comprehensive tracing
+export GRPC_TRACE=api,call_error,channel,client_channel,connectivity_state,http,http2_stream,tcp,transport_security
+
+# Run your script
+python your_script.py
+```
+
+**Note**: These environment variables must be set before Python starts, as the gRPC C++ core reads them at module import time.
+
+### Common Issues and Solutions
+
+| Issue | Solution |
+|-------|----------|
+| Connection timeout | Check network connectivity, firewall rules, and ensure port 80/443 is open |
+| Authentication failure | Verify username (email) and access token are correct |
+| 503 Service Unavailable | Cluster may be suspended; enable `auto_resume=True` |
+| 456 Strategy Error | Automatic blue-green failover will handle this |
+| Memory issues with large results | Use `fetchall_buffer()` instead of `fetchall()` |
+| gRPC message size errors | Configure `grpc_options` with appropriate message size limits |
 
 See [TECH_DOC.md](TECH_DOC.md) for detailed technical documentation.

{e6data_python_connector-2.3.10rc8 → e6data_python_connector-2.3.11}/README.md

@@ -1,6 +1,6 @@
 # e6data Python Connector
 
-![version](https://img.shields.io/badge/version-2.3.10rc8-blue.svg)
+![version](https://img.shields.io/badge/version-2.3.11-blue.svg)
 
 ## Introduction
 
@@ -37,6 +37,8 @@ Use your e6data Email ID as the username and your access token as the password.
 
 ```python
 from e6data_python_connector import Connection
+# For connection pooling (recommended for concurrent operations)
+from e6data_python_connector import ConnectionPool
 
 username = '<username>'  # Your e6data Email ID.
 password = '<password>'  # Access Token generated in the e6data console.
@@ -46,6 +48,7 @@ database = '<database>'  # Database to perform the query on.
 port = 80  # Port of the e6data engine.
 catalog_name = '<catalog_name>'
 
+# Single connection (for simple, single-threaded use)
 conn = Connection(
     host=host,
     port=port,
@@ -53,6 +56,17 @@ conn = Connection(
     database=database,
     password=password
 )
+
+# Or use connection pool (for concurrent/multi-threaded use)
+pool = ConnectionPool(
+    min_size=2,
+    max_size=10,
+    host=host,
+    port=port,
+    username=username,
+    database=database,
+    password=password
+)
 ```
 
 #### Connection Parameters
@@ -71,6 +85,7 @@ The `Connection` class supports the following parameters:
 | `secure` | bool | No | False | Enable SSL/TLS for secure connections |
 | `auto_resume` | bool | No | True | Automatically resume cluster if suspended |
 | `grpc_options` | dict | No | None | Additional gRPC configuration options |
+| `debug` | bool | No | False | Enable debug logging for troubleshooting |
 
 #### Secure Connection Example
 
@@ -401,8 +416,241 @@ For detailed migration instructions, see the [Migration Guide](docs/zero-downtim
 - Message size optimization for large queries
 
 ### Connection Management
-- Enable connection pooling for better resource utilization
+
+#### Connection Pooling
+
+The e6data Python connector now includes a built-in connection pool for efficient connection management and reuse across multiple threads. The `ConnectionPool` class provides:
+
+- **Thread-safe connection reuse**: Each thread automatically reuses its assigned connection
+- **Automatic lifecycle management**: Handles connection creation, health checks, and cleanup
+- **Overflow connections**: Creates temporary connections when pool is exhausted
+- **Connection health monitoring**: Automatic detection and replacement of broken connections
+- **Statistics tracking**: Monitor pool usage and performance
+
+##### Basic Connection Pool Usage
+
+```python
+from e6data_python_connector import ConnectionPool
+
+# Create a connection pool
+pool = ConnectionPool(
+    min_size=2,        # Minimum connections to maintain
+    max_size=10,       # Maximum connections in pool
+    max_overflow=5,    # Additional temporary connections allowed
+    timeout=30.0,      # Timeout for getting connection (seconds)
+    recycle=3600,      # Maximum age before recycling (seconds)
+    debug=False,       # Enable debug logging
+    pre_ping=True,     # Check connection health before use
+    # Connection parameters
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database,
+    catalog=catalog_name,
+    cluster_name=cluster_name,
+    secure=True
+)
+
+# Get connection and execute query
+conn = pool.get_connection()
+cursor = conn.cursor()
+cursor.execute("SELECT * FROM table")
+results = cursor.fetchall()
+
+# Return connection to pool (important!)
+pool.return_connection(conn)
+
+# Clean up when done
+pool.close_all()
+```
+
+##### Using Context Manager (Recommended)
+
+The context manager pattern ensures connections are automatically returned to the pool:
+
+```python
+from e6data_python_connector import ConnectionPool
+
+pool = ConnectionPool(
+    min_size=2,
+    max_size=10,
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database
+)
+
+# Connection automatically returned to pool after use
+with pool.get_connection_context() as conn:
+    cursor = conn.cursor()
+    cursor.execute("SELECT * FROM table")
+    results = cursor.fetchall()
+    print(results)
+```
+
+##### Concurrent Query Execution
+
+Connection pooling is especially beneficial for concurrent query execution:
+
+```python
+import concurrent.futures
+from e6data_python_connector import ConnectionPool
+
+def execute_query(pool, query_id, query):
+    """Execute a query using a pooled connection."""
+    # Each thread will reuse its assigned connection
+    conn = pool.get_connection()
+    try:
+        cursor = conn.cursor()
+        cursor.execute(query)
+        results = cursor.fetchall()
+        return f"Query {query_id}: {len(results)} rows"
+    finally:
+        pool.return_connection(conn)
+
+# Create pool
+pool = ConnectionPool(
+    min_size=3,
+    max_size=10,
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database
+)
+
+# Execute multiple queries concurrently
+queries = [
+    "SELECT COUNT(*) FROM table1",
+    "SELECT AVG(value) FROM table2",
+    "SELECT MAX(date) FROM table3"
+]
+
+with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
+    futures = [
+        executor.submit(execute_query, pool, i, query)
+        for i, query in enumerate(queries)
+    ]
+    
+    for future in concurrent.futures.as_completed(futures):
+        print(future.result())
+
+# Clean up
+pool.close_all()
+```
+
+##### Connection Pool Configuration
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `min_size` | int | 2 | Minimum number of connections to maintain |
+| `max_size` | int | 10 | Maximum number of connections in pool |
+| `max_overflow` | int | 5 | Additional temporary connections allowed |
+| `timeout` | float | 30.0 | Timeout for getting connection (seconds) |
+| `recycle` | int | 3600 | Maximum connection age before recycling (seconds) |
+| `debug` | bool | False | Enable debug logging for pool operations |
+| `pre_ping` | bool | True | Check connection health before returning from pool |
+
+##### Monitoring Pool Statistics
+
+```python
+# Get pool statistics
+stats = pool.get_statistics()
+print(f"Active connections: {stats['active_connections']}")
+print(f"Idle connections: {stats['idle_connections']}")
+print(f"Total requests: {stats['total_requests']}")
+print(f"Failed connections: {stats['failed_connections']}")
+```
+
+##### When to Use Connection Pooling
+
+Connection pooling is recommended when:
+- Executing multiple queries concurrently
+- Building web applications or APIs
+- Running batch processing jobs
+- Reducing connection overhead
+- Improving application performance
+
+##### Direct Connection Usage (Without Pool)
+
+For simple, single-threaded applications, you can still use direct connections:
+
+```python
+from e6data_python_connector import Connection
+
+conn = Connection(
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database
+)
+
+cursor = conn.cursor()
+cursor.execute("SELECT * FROM table")
+results = cursor.fetchall()
+conn.close()
+```
+
+#### Additional Connection Management Features
 - Automatic connection health monitoring
 - Graceful connection recovery and retry logic
+- Blue-green deployment support with automatic failover
+
+## Debugging and Troubleshooting
+
+### Enable Debug Mode
+
+Enable comprehensive debugging to troubleshoot connection and query issues:
+
+```python
+from e6data_python_connector import Connection
+
+conn = Connection(
+    host=host,
+    port=port,
+    username=username,
+    password=password,
+    database=database,
+    debug=True  # Enable debug logging
+)
+```
+
+When `debug=True`, the following features are enabled:
+- Python logging at DEBUG level for all operations
+- Blue-green strategy transition logging
+- Connection lifecycle logging
+- Query execution detailed logging
+
+### gRPC Network Tracing
+
+For low-level gRPC network debugging (HTTP/2 frames, TCP events), set environment variables **before** running your Python script:
+
+```bash
+# Enable gRPC network tracing
+export GRPC_VERBOSITY=DEBUG
+export GRPC_TRACE=client_channel,http2
+
+# For comprehensive tracing
+export GRPC_TRACE=api,call_error,channel,client_channel,connectivity_state,http,http2_stream,tcp,transport_security
+
+# Run your script
+python your_script.py
+```
+
+**Note**: These environment variables must be set before Python starts, as the gRPC C++ core reads them at module import time.
+
+### Common Issues and Solutions
+
+| Issue | Solution |
+|-------|----------|
+| Connection timeout | Check network connectivity, firewall rules, and ensure port 80/443 is open |
+| Authentication failure | Verify username (email) and access token are correct |
+| 503 Service Unavailable | Cluster may be suspended; enable `auto_resume=True` |
+| 456 Strategy Error | Automatic blue-green failover will handle this |
+| Memory issues with large results | Use `fetchall_buffer()` instead of `fetchall()` |
+| gRPC message size errors | Configure `grpc_options` with appropriate message size limits |
 
 See [TECH_DOC.md](TECH_DOC.md) for detailed technical documentation.

e6data_python_connector-2.3.11/e6data_python_connector/__init__.py

@@ -0,0 +1,4 @@
+from e6data_python_connector.e6data_grpc import Connection, Cursor
+from e6data_python_connector.connection_pool import ConnectionPool
+
+__all__ = ['Connection', 'Cursor', 'ConnectionPool']