e6data-python-connector 3.0.1rc1__tar.gz → 3.0.1rc2__tar.gz

{e6data_python_connector-3.0.1rc1/e6data_python_connector.egg-info → e6data_python_connector-3.0.1rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: e6data-python-connector
-Version: 3.0.1rc1
+Version: 3.0.1rc2
 Summary: Client for the e6data distributed SQL Engine.
 Home-page: https://github.com/e6x-labs/e6data-python-connector
 Author: e6data, Inc.
@@ -471,85 +471,119 @@ cursor = conn.cursor(database, catalog_name)
 
 ### Opt in to multi-chunk result batches
 
-`enable_result_batch_v2=True` lets a compatible planner return several result
-chunks in one `getNextResultBatchV2` response. The default is `False`, which keeps
-the existing V1 protocol. Increasing `arraysize` or `fetchmany(size)` changes how
-many rows your application receives per call; it does not change the planner's
-response size.
+Set `enable_result_batch_v2=True` to enable V2 batches, continuous result downloading,
+and bounded parallel chunk decoding together. It defaults to `False`. There
+are no separate public prefetch or decoding switches. `arraysize` and
+`fetchmany(size)` only change the number of rows returned to the application.
 
 The planner must support V2 and have `ENABLE_GET_NEXT_RESULT_BATCH_V2` enabled.
-`ENABLE_GET_NEXT_CHUNK_V2` controls the separate executor-to-planner boundary.
-Record both flags when qualifying a deployment. V2 can reduce network round
-trips, but it does not reduce the number of result bytes or extend the planner's
-query lifetime. Completion of a 20-million-row result within 900 seconds must be
-measured against the actual workload and deployment.
-
-The following examples use your existing `connection_options`, `sql`, and
-`consume` function. Choose a positive `result_receive_limit_bytes` from measured
-V2 response sizes and the client's memory budget. The sync API preserves its
-existing gRPC options, so supply a finite receive limit for a V2 rollout:
+`ENABLE_GET_NEXT_CHUNK_V2` controls a separate executor-to-planner boundary.
+These flags do not extend the planner's query lifetime. Finishing a large result
+within 900 seconds still needs a measurement on the target deployment.
+
+The optimized path requires ordinary GIL-enabled CPython 3.11 through 3.13 and
+an import-safe application entry point. Put application startup inside
+`if __name__ == "__main__":`, including when using SQLAlchemy. Interactive,
+daemon and frozen applications are rejected before query submission. Two
+workers use the explicit `spawn` method; the application's global multiprocessing
+start method is unchanged. An unguarded main script can run its other import
+side effects, so keep all application startup inside the guard.
+
+Both APIs default to a 64 MiB receive limit when V2 is enabled. A positive finite
+custom limit is allowed; unlimited receive sizes are rejected for V2. The sync
+flag-off path keeps its existing options. For async, set
+`max_receive_message_bytes`; any receive limit in `grpc_options` must agree.
+Choose limits from measured envelope sizes and the client's memory budget.
+
+These examples use your existing `connection_options`, `sql`, and `consume`:
 
 ```python
 from e6data_python_connector import Connection
 
-sync_options = {
-    **connection_options,
-    "enable_result_batch_v2": True,
-    "grpc_options": {
-        **connection_options.get("grpc_options", {}),
-        "max_receive_message_length": result_receive_limit_bytes,
-    },
-}
-with Connection(**sync_options) as connection:
-    with connection.cursor() as cursor:
-        cursor.execute(sql)
-        for rows in cursor.fetchall_buffer():
-            consume(rows)
-```
 
-The async API keeps its finite 64 MiB default. Set
-`max_receive_message_bytes` to change that limit. If `grpc_options` also contains
-a receive limit, it must agree with this value.
+def main():
+    options = {**connection_options, "enable_result_batch_v2": True}
+    with Connection(**options) as connection:
+        with connection.cursor() as cursor:
+            cursor.execute(sql)
+            for rows in cursor.fetchall_buffer():
+                consume(rows)
+
+
+if __name__ == "__main__":
+    main()
+```
 
 ```python
+import asyncio
 from e6data_python_connector.aio import AsyncConnection
 
-async_options = {
-    **connection_options,
-    "enable_result_batch_v2": True,
-    "max_receive_message_bytes": result_receive_limit_bytes,
-}
-async with AsyncConnection(**async_options) as connection:
-    async with connection.cursor() as cursor:
-        await cursor.execute(sql)
-        async for rows in cursor.fetchall_buffer():
-            consume(rows)
-```
 
-SQLAlchemy accepts the same opt-in through `connect_args`:
+async def main():
+    options = {**connection_options, "enable_result_batch_v2": True}
+    async with AsyncConnection(**options) as connection:
+        async with connection.cursor() as cursor:
+            await cursor.execute(sql)
+            async for rows in cursor.fetchall_buffer():
+                consume(rows)
 
-```python
-from sqlalchemy import create_engine
-from sqlalchemy.ext.asyncio import create_async_engine
 
-engine = create_engine("e6data://", connect_args=sync_options)
-async_engine = create_async_engine("e6data+asyncio://", connect_args=async_options)
+if __name__ == "__main__":
+    asyncio.run(main())
 ```
 
-Fetch return shapes and row order stay the same. `fetchall_buffer()` yields
-decoded chunks; a V2 response can contain several chunks. All chunks in an
-envelope are decoded before any are exposed, so V2 may use more memory and delay
-the first chunk compared with V1. `fetchall()` still retains the full result.
-There is no prefetch or parallel fetching for a query.
+SQLAlchemy accepts the same flag through `connect_args`. Create and use engines
+inside the guarded application entry point:
 
-An `UNIMPLEMENTED` response switches that query to V1. A later query may try V2
-again. Other fetch failures do not trigger a protocol fallback or query replay.
-To disable V2 for new connections, omit the option or set it to `False`.
+```python
+connect_args = {**connection_options, "enable_result_batch_v2": True}
+engine = create_engine("e6data://", connect_args=connect_args)
+async_engine = create_async_engine("e6data+asyncio://", connect_args=connect_args)
+```
 
-The [result batch qualification instructions](test/README.md#result-batch-v2-qualification)
-cover protocol parity and the explicit large-result benchmark. Real-engine
-qualification and the 900-second acceptance measurement have not been run for
-this change.
+After the first V2 response, a download thread (sync) or task (async) keeps
+fetching serialized batches until the engine returns end-of-stream. Decoding
+runs concurrently and takes batches from the queue in order. Downloading does
+not wait for decoding or application row processing. There is no queue-size
+limit and no process-wide limit on active downloaders. Result RPCs for the same
+query never overlap.
+
+Each background RPC gets a fresh configured transport budget when it starts;
+that deadline never extends after dispatch. An empty nonterminal response uses
+backoff with a finite no-progress deadline. Completed responses can wait while
+the application is idle. Public fetch deadlines still cover waiting and decoding,
+and expiry or cancellation stops the downloader. Async `fetchall` keeps one
+deadline for its entire operation.
+
+Two shared worker processes can decode one multi-chunk envelope at a time.
+Other envelopes use the sequential path. Workers receive chunk bytes and column
+positions, not connections or credentials. Process startup and copying add cost,
+so parallel decoding is not a promise of better performance for every result.
+
+Fetch return shapes and row order stay the same. `fetchall_buffer()` yields the
+original decoded chunks. All chunks in an envelope must decode successfully
+before any are exposed. The serialized queue can hold the entire result when
+downloading is faster than decoding or application reads. Workers add Python
+objects and process copies; the receive limit is not a limit on total memory.
+`fetchall()` still retains the full result.
+
+Only a V2 `UNIMPLEMENTED` response switches that query to V1 and disables its
+optimizations. Other RPC failures reach the caller as the original
+`grpc.RpcError` or `grpc.aio.AioRpcError`. A failed pending RPC is raised when
+its envelope is needed, after current valid chunks have been consumed. Failed
+or cancelled queries are not replayed. Later fetches on an incomplete result
+raise `IncompleteResultError`. Clear, cancel, close and pool return retire
+pending work; unsafe connections must not be reused.
+
+Use `debug=True` for payload-free RPC, decode and worker diagnostics. Serialized
+Protobuf bytes are uncompressed payload size, not bytes sent over the network.
+
+For rollback, close old cursors and connections, then omit
+`enable_result_batch_v2` or set it to `False` on new connections. Closing the
+last V2 connection also stops its shared workers. See the
+[qualification instructions](test/README.md#result-batch-v2-qualification).
+Local synthetic results do not prove customer performance or the 900-second
+query requirement.
 
 ### Get Query Time Metrics
 ```python
@@ -996,11 +1030,27 @@ conn = Connection(
 ```
 
 When `debug=True`, the following features are enabled:
-- Python logging at DEBUG level for all operations
+- Connector logging at DEBUG level, for both sync and async connections. Existing
+  application logging handlers and the root logger level are kept.
 - Blue-green strategy transition logging
 - Connection lifecycle logging
 - Query execution detailed logging
 
+Each result fetch writes a normal debug message with `protocol`, `rpc_seconds`,
+`decode_seconds`, `chunk_count`, `serialized_bytes`, and `status`. The timing values
+are seconds. `rpc_seconds` measures the client RPC call, including waiting for its
+response. `decode_seconds` measures client response processing. `serialized_bytes`
+is the protobuf response size, not the number of bytes on the network. A failed
+attempt reports zero response size and chunk count. The same values remain available
+as `result_batch_*` attributes for structured log handlers.
+
+These metrics do not add SQL, credentials, session IDs, result values, or raw metadata
+to logs. `debug=False` keeps the normal quiet default. Applications can also enable
+DEBUG on the `e6data_python_connector` logger themselves. `debug=True` adds a connector
+stream handler only when no existing handler in its logger path accepts DEBUG.
+As with normal Python logging, enabling the package logger also affects other open
+connector connections in that process.
+
 ### gRPC Network Tracing
 
 For low-level gRPC network debugging (HTTP/2 frames, TCP events), set environment variables **before** running your Python script:

{e6data_python_connector-3.0.1rc1 → e6data_python_connector-3.0.1rc2}/README.md

@@ -425,85 +425,119 @@ cursor = conn.cursor(database, catalog_name)
 
 ### Opt in to multi-chunk result batches
 
-`enable_result_batch_v2=True` lets a compatible planner return several result
-chunks in one `getNextResultBatchV2` response. The default is `False`, which keeps
-the existing V1 protocol. Increasing `arraysize` or `fetchmany(size)` changes how
-many rows your application receives per call; it does not change the planner's
-response size.
+Set `enable_result_batch_v2=True` to enable V2 batches, continuous result downloading,
+and bounded parallel chunk decoding together. It defaults to `False`. There
+are no separate public prefetch or decoding switches. `arraysize` and
+`fetchmany(size)` only change the number of rows returned to the application.
 
 The planner must support V2 and have `ENABLE_GET_NEXT_RESULT_BATCH_V2` enabled.
-`ENABLE_GET_NEXT_CHUNK_V2` controls the separate executor-to-planner boundary.
-Record both flags when qualifying a deployment. V2 can reduce network round
-trips, but it does not reduce the number of result bytes or extend the planner's
-query lifetime. Completion of a 20-million-row result within 900 seconds must be
-measured against the actual workload and deployment.
-
-The following examples use your existing `connection_options`, `sql`, and
-`consume` function. Choose a positive `result_receive_limit_bytes` from measured
-V2 response sizes and the client's memory budget. The sync API preserves its
-existing gRPC options, so supply a finite receive limit for a V2 rollout:
+`ENABLE_GET_NEXT_CHUNK_V2` controls a separate executor-to-planner boundary.
+These flags do not extend the planner's query lifetime. Finishing a large result
+within 900 seconds still needs a measurement on the target deployment.
+
+The optimized path requires ordinary GIL-enabled CPython 3.11 through 3.13 and
+an import-safe application entry point. Put application startup inside
+`if __name__ == "__main__":`, including when using SQLAlchemy. Interactive,
+daemon and frozen applications are rejected before query submission. Two
+workers use the explicit `spawn` method; the application's global multiprocessing
+start method is unchanged. An unguarded main script can run its other import
+side effects, so keep all application startup inside the guard.
+
+Both APIs default to a 64 MiB receive limit when V2 is enabled. A positive finite
+custom limit is allowed; unlimited receive sizes are rejected for V2. The sync
+flag-off path keeps its existing options. For async, set
+`max_receive_message_bytes`; any receive limit in `grpc_options` must agree.
+Choose limits from measured envelope sizes and the client's memory budget.
+
+These examples use your existing `connection_options`, `sql`, and `consume`:
 
 ```python
 from e6data_python_connector import Connection
 
-sync_options = {
-    **connection_options,
-    "enable_result_batch_v2": True,
-    "grpc_options": {
-        **connection_options.get("grpc_options", {}),
-        "max_receive_message_length": result_receive_limit_bytes,
-    },
-}
-with Connection(**sync_options) as connection:
-    with connection.cursor() as cursor:
-        cursor.execute(sql)
-        for rows in cursor.fetchall_buffer():
-            consume(rows)
-```
 
-The async API keeps its finite 64 MiB default. Set
-`max_receive_message_bytes` to change that limit. If `grpc_options` also contains
-a receive limit, it must agree with this value.
+def main():
+    options = {**connection_options, "enable_result_batch_v2": True}
+    with Connection(**options) as connection:
+        with connection.cursor() as cursor:
+            cursor.execute(sql)
+            for rows in cursor.fetchall_buffer():
+                consume(rows)
+
+
+if __name__ == "__main__":
+    main()
+```
 
 ```python
+import asyncio
 from e6data_python_connector.aio import AsyncConnection
 
-async_options = {
-    **connection_options,
-    "enable_result_batch_v2": True,
-    "max_receive_message_bytes": result_receive_limit_bytes,
-}
-async with AsyncConnection(**async_options) as connection:
-    async with connection.cursor() as cursor:
-        await cursor.execute(sql)
-        async for rows in cursor.fetchall_buffer():
-            consume(rows)
-```
 
-SQLAlchemy accepts the same opt-in through `connect_args`:
+async def main():
+    options = {**connection_options, "enable_result_batch_v2": True}
+    async with AsyncConnection(**options) as connection:
+        async with connection.cursor() as cursor:
+            await cursor.execute(sql)
+            async for rows in cursor.fetchall_buffer():
+                consume(rows)
 
-```python
-from sqlalchemy import create_engine
-from sqlalchemy.ext.asyncio import create_async_engine
 
-engine = create_engine("e6data://", connect_args=sync_options)
-async_engine = create_async_engine("e6data+asyncio://", connect_args=async_options)
+if __name__ == "__main__":
+    asyncio.run(main())
 ```
 
-Fetch return shapes and row order stay the same. `fetchall_buffer()` yields
-decoded chunks; a V2 response can contain several chunks. All chunks in an
-envelope are decoded before any are exposed, so V2 may use more memory and delay
-the first chunk compared with V1. `fetchall()` still retains the full result.
-There is no prefetch or parallel fetching for a query.
+SQLAlchemy accepts the same flag through `connect_args`. Create and use engines
+inside the guarded application entry point:
 
-An `UNIMPLEMENTED` response switches that query to V1. A later query may try V2
-again. Other fetch failures do not trigger a protocol fallback or query replay.
-To disable V2 for new connections, omit the option or set it to `False`.
+```python
+connect_args = {**connection_options, "enable_result_batch_v2": True}
+engine = create_engine("e6data://", connect_args=connect_args)
+async_engine = create_async_engine("e6data+asyncio://", connect_args=connect_args)
+```
 
-The [result batch qualification instructions](test/README.md#result-batch-v2-qualification)
-cover protocol parity and the explicit large-result benchmark. Real-engine
-qualification and the 900-second acceptance measurement have not been run for
-this change.
+After the first V2 response, a download thread (sync) or task (async) keeps
+fetching serialized batches until the engine returns end-of-stream. Decoding
+runs concurrently and takes batches from the queue in order. Downloading does
+not wait for decoding or application row processing. There is no queue-size
+limit and no process-wide limit on active downloaders. Result RPCs for the same
+query never overlap.
+
+Each background RPC gets a fresh configured transport budget when it starts;
+that deadline never extends after dispatch. An empty nonterminal response uses
+backoff with a finite no-progress deadline. Completed responses can wait while
+the application is idle. Public fetch deadlines still cover waiting and decoding,
+and expiry or cancellation stops the downloader. Async `fetchall` keeps one
+deadline for its entire operation.
+
+Two shared worker processes can decode one multi-chunk envelope at a time.
+Other envelopes use the sequential path. Workers receive chunk bytes and column
+positions, not connections or credentials. Process startup and copying add cost,
+so parallel decoding is not a promise of better performance for every result.
+
+Fetch return shapes and row order stay the same. `fetchall_buffer()` yields the
+original decoded chunks. All chunks in an envelope must decode successfully
+before any are exposed. The serialized queue can hold the entire result when
+downloading is faster than decoding or application reads. Workers add Python
+objects and process copies; the receive limit is not a limit on total memory.
+`fetchall()` still retains the full result.
+
+Only a V2 `UNIMPLEMENTED` response switches that query to V1 and disables its
+optimizations. Other RPC failures reach the caller as the original
+`grpc.RpcError` or `grpc.aio.AioRpcError`. A failed pending RPC is raised when
+its envelope is needed, after current valid chunks have been consumed. Failed
+or cancelled queries are not replayed. Later fetches on an incomplete result
+raise `IncompleteResultError`. Clear, cancel, close and pool return retire
+pending work; unsafe connections must not be reused.
+
+Use `debug=True` for payload-free RPC, decode and worker diagnostics. Serialized
+Protobuf bytes are uncompressed payload size, not bytes sent over the network.
+
+For rollback, close old cursors and connections, then omit
+`enable_result_batch_v2` or set it to `False` on new connections. Closing the
+last V2 connection also stops its shared workers. See the
+[qualification instructions](test/README.md#result-batch-v2-qualification).
+Local synthetic results do not prove customer performance or the 900-second
+query requirement.
 
 ### Get Query Time Metrics
 ```python
@@ -950,11 +984,27 @@ conn = Connection(
 ```
 
 When `debug=True`, the following features are enabled:
-- Python logging at DEBUG level for all operations
+- Connector logging at DEBUG level, for both sync and async connections. Existing
+  application logging handlers and the root logger level are kept.
 - Blue-green strategy transition logging
 - Connection lifecycle logging
 - Query execution detailed logging
 
+Each result fetch writes a normal debug message with `protocol`, `rpc_seconds`,
+`decode_seconds`, `chunk_count`, `serialized_bytes`, and `status`. The timing values
+are seconds. `rpc_seconds` measures the client RPC call, including waiting for its
+response. `decode_seconds` measures client response processing. `serialized_bytes`
+is the protobuf response size, not the number of bytes on the network. A failed
+attempt reports zero response size and chunk count. The same values remain available
+as `result_batch_*` attributes for structured log handlers.
+
+These metrics do not add SQL, credentials, session IDs, result values, or raw metadata
+to logs. `debug=False` keeps the normal quiet default. Applications can also enable
+DEBUG on the `e6data_python_connector` logger themselves. `debug=True` adds a connector
+stream handler only when no existing handler in its logger path accepts DEBUG.
+As with normal Python logging, enabling the package logger also affects other open
+connector connections in that process.
+
 ### gRPC Network Tracing
 
 For low-level gRPC network debugging (HTTP/2 frames, TCP events), set environment variables **before** running your Python script:

{e6data_python_connector-3.0.1rc1 → e6data_python_connector-3.0.1rc2}/e6data_python_connector/async_connection.py

@@ -52,6 +52,9 @@ class AsyncConnection:
                  oauth_timeout=10.0, cleanup_timeout=10.0, auto_resume_timeout=300.0,
                  max_receive_message_bytes=64 * 1024 * 1024,
                  enable_result_batch_v2=False):
+        from .result_decode_worker import is_decode_worker
+        if is_decode_worker():
+            raise RuntimeError('A result decode worker cannot create a connection.')
         if sys.version_info < (3, 11):
             raise RuntimeError('The async API requires Python 3.11 or newer.')
         try:
@@ -81,6 +84,9 @@ class AsyncConnection:
             raise ValueError('ssl_cert must be a certificate path or PEM bytes.')
         if not isinstance(enable_result_batch_v2, bool):
             raise ValueError('enable_result_batch_v2 must be a boolean.')
+        if enable_result_batch_v2:
+            from .result_decode import validate_decode_runtime
+            validate_decode_runtime()
         settings = dict(host=host, port=port, username=username, password=password,
                         catalog=catalog, database=database, cluster_name=cluster_name,
                         secure=bool(secure), ssl_cert=ssl_cert, auto_resume=auto_resume,
@@ -107,6 +113,9 @@ class AsyncConnection:
         settings['grpc_options'] = MappingProxyType(options)
         settings['max_receive_message_bytes'] = max_receive_message_bytes
         self._config = MappingProxyType(settings)
+        if debug:
+            from .e6data_grpc import _configure_debug_logging
+            _configure_debug_logging()
         self._oauth_enabled = not modes[0]
         self._channel = self._client = self._token_provider = None
         self._owns_token_provider = True
@@ -129,6 +138,11 @@ class AsyncConnection:
         self._cleanup_error = None
         self._cluster_manager = None
         self._owns_cluster_manager = True
+        self._decoder_started = False
+        self._decoder_lease = None
+        if enable_result_batch_v2:
+            from .result_decode import DecoderLease
+            self._decoder_lease = DecoderLease()
 
     def __getattr__(self, name):
         config = self.__dict__.get('_config', {})
@@ -304,6 +318,24 @@ class AsyncConnection:
         self._check_owner(_cleanup=_cleanup, _internal=True)
         return metadata
 
+    async def _ensure_decoder(self, deadline):
+        self._check_owner()
+        if self._decoder_lease is not None and not self._decoder_started:
+            from .async_work import run_blocking
+            await run_blocking(self._decoder_lease.start, deadline, deadline=deadline)
+            self._check_owner()
+            self._decoder_started = True
+
+    async def _raw_result_rpc(self, method_name, request, metadata, deadline):
+        """Transport only: the owning task already froze identity and metadata."""
+        call = getattr(self._client, method_name)(
+            request, metadata=metadata, timeout=self._remaining(deadline))
+        self._calls.add(call)
+        try:
+            return await call
+        finally:
+            self._calls.discard(call)
+
     async def _rpc(self, method_name, request, *, deadline, route=None, safe_retry=False, _cleanup=False, _on_dispatch=None, _response_metadata=None):
         self._check_owner(_cleanup=_cleanup, _internal=True)
         refreshed = switched = False
@@ -504,6 +536,15 @@ class AsyncConnection:
             # Native channel close with no grace cancels transports without waiting.
             if self._channel is not None:
                 await self._channel.close()
+            if self._decoder_lease is not None:
+                # Retirement must survive loop shutdown and saturated work slots.
+                self._decoder_lease.retire(deadline)
+                try:
+                    async with asyncio.timeout_at(deadline):
+                        while self._decoder_lease.cleanup_pending:
+                            await asyncio.sleep(min(0.005, self._remaining(deadline)))
+                except (Exception, asyncio.CancelledError):
+                    self._cleanup_error = OperationalError('Result decoder cleanup remains pending.')
             for resource, owned in ((self._cluster_manager, self._owns_cluster_manager),
                                     (self._token_provider, self._owns_token_provider)):
                 if owned and resource is not None:
@@ -542,6 +583,12 @@ class AsyncConnection:
         if self._owner is not None and self._owner != (os.getpid(), threading.get_ident(), asyncio.get_running_loop()):
             raise ProgrammingError('Connection belongs to another owner.')
         await self.close()
+        if self._decoder_lease is not None and self._decoder_lease.cleanup_pending:
+            raise ProgrammingError('Cannot reopen while result decoder cleanup remains pending.')
+        if self.enable_result_batch_v2:
+            from .result_decode import DecoderLease
+            self._decoder_lease = DecoderLease()
+            self._decoder_started = False
         self._state = 'new'
         self._session_id = self._session_task = self._close_task = self._open_task = None
         if self._owns_token_provider:

{e6data_python_connector-3.0.1rc1 → e6data_python_connector-3.0.1rc2}/e6data_python_connector/async_connection_pool.py

@@ -316,6 +316,8 @@ class AsyncConnectionPool:
                     healthy = healthy and cursor._state != 'SUBMISSION_UNKNOWN'
                     await cursor._close_owned(deadline)
                     healthy = healthy and cursor.cleanup_error is None
+                    healthy = (healthy and cursor._pending_result is None
+                               and not cursor._retired_results)
                 healthy = (healthy and not connection._routes and not connection._ambiguous_submissions
                            and connection._state == 'open')
         except (Exception, asyncio.CancelledError):