e6data-python-connector 2.3.14__tar.gz → 3.0.0__tar.gz

{e6data_python_connector-2.3.14 → e6data_python_connector-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: e6data-python-connector
-Version: 2.3.14
+Version: 3.0.0
 Summary: Client for the e6data distributed SQL Engine.
 Home-page: https://github.com/e6x-labs/e6data-python-connector
 Author: e6data, Inc.
@@ -27,6 +27,11 @@ Requires-Dist: pytz>=2024.1
 Requires-Dist: thrift>=0.20.0
 Requires-Dist: grpcio>=1.65.1
 Requires-Dist: grpcio-tools>=1.65.1
+Provides-Extra: async
+Requires-Dist: httpx<1,>=0.28.1; extra == "async"
+Provides-Extra: async-sqlalchemy
+Requires-Dist: httpx<1,>=0.28.1; extra == "async-sqlalchemy"
+Requires-Dist: SQLAlchemy[asyncio]<2.1,>=2.0; extra == "async-sqlalchemy"
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -34,16 +39,38 @@ Dynamic: description
 Dynamic: description-content-type
 Dynamic: home-page
 Dynamic: license
+Dynamic: license-file
+Dynamic: provides-extra
 Dynamic: requires-dist
 Dynamic: summary
 
 # e6data Python Connector
 
-![version](https://img.shields.io/badge/version-2.3.14-blue.svg)
+Package version and extras are defined in `setup.py`.
 
 ## Introduction
 
-The e6data Connector for Python provides an interface for writing Python applications that can connect to e6data and perform operations. It includes automatic support for blue-green deployments, ensuring seamless failover during server updates without query interruption.
+The e6data Connector for Python provides an interface for writing Python applications that can connect to e6data and perform operations. It includes routing support for blue-green deployments. Active query handles remain tied to the engine that accepted them; failover does not guarantee uninterrupted results.
+
+## Optional native asyncio API
+
+This branch adds native async connection, cursor, pooling and SQLAlchemy support for Python 3.11+. Release qualification remains incomplete. Install this branch's built package with `[async]` or `[async-sqlalchemy]`; these instructions do not assert that the feature is published.
+
+```python
+from e6data_python_connector.aio import AsyncConnection
+
+async def query(config, sql):
+    async with AsyncConnection(**config) as connection:
+        async with connection.cursor() as cursor:
+            await cursor.execute(sql)
+            return await cursor.fetchall()
+```
+
+`config` is application-supplied connection configuration. OAuth requires verified TLS. Native `fetchone()` preserves the one-row outer list, while SQLAlchemy `e6data+asyncio` adapts it to a normal row. Ambiguous submission and incomplete result errors must not be handled by blindly retrying the query. Use context managers for bounded cleanup.
+
+See [Async API](docs/ASYNC_API.md) for the complete API map, deadlines, pooling, SQLAlchemy and test configuration, [OAuth lifecycle](docs/OAUTH_LIFECYCLE.md) for renewal and failure behavior, and [the application example](examples/async_query.py).
+
+The new CI keeps the full-package coverage denominator and a greater-than-80% gate. The initial baseline was about 19%, so focused async test success is not a release pass. Real-service tests require explicit configuration and are qualified separately.
 
 ### Dependencies
 Make sure to install below dependencies and wheel before install e6data-python-connector.
@@ -120,8 +147,8 @@ The `Connection` class supports the following parameters:
 |-----------|------|----------|---------|-------------|
 | `host` | str | Yes | - | IP address or hostname of the e6data cluster |
 | `port` | int | Yes | - | Port of the e6data engine (typically 80) |
-| `username` | str | Yes | - | Your e6data Email ID |
-| `password` | str | Yes | - | Access Token generated in the e6data console |
+| `username` | str | Conditional | None | Your e6data Email ID. Required unless authenticating with OAuth |
+| `password` | str | Conditional | None | Access Token generated in the e6data console. Required unless authenticating with OAuth |
 | `database` | str | No | None | Database to perform queries on |
 | `catalog` | str | No | None | Catalog name |
 | `cluster_name` | str | No | None | Name of the cluster for cluster-specific operations |
@@ -131,6 +158,76 @@ The `Connection` class supports the following parameters:
 | `grpc_options` | dict | No | None | Additional gRPC configuration options |
 | `debug` | bool | No | False | Enable debug logging for troubleshooting |
 | `require_fastbinary` | bool | No | True | Require fastbinary module for Thrift deserialization. Set to False to use pure Python implementation if system dependencies cannot be installed |
+| `client_id` | str | No | None | OAuth 2.0 client id. Use with `client_secret` and `token_url` |
+| `client_secret` | str | No | None | OAuth 2.0 client secret |
+| `token_url` | str | No | None | Token endpoint of the authorization server |
+| `oauth_scope` | str | No | None | Space-delimited scopes to request. Omit for the client's full registered set |
+| `access_token` | str | No | None | A previously obtained access token, for callers that mint their own |
+| `client_auth_method` | str | No | `'basic'` | How client credentials reach the token endpoint: `'basic'` or `'post'` |
+
+#### Authenticating with OAuth 2.0
+
+As an alternative to username and password, a connection can authenticate with an OAuth 2.0 access
+token. The connector obtains a token using the client-credentials grant, caches it, and refreshes it
+shortly before it expires.
+
+Matching client-credentials configurations share one token cache and concurrent
+refresh across sync and async connections, separate pools and threads in the same
+process. Connection and transport ownership remain local. See the
+[OAuth lifecycle](docs/OAUTH_LIFECYCLE.md) for matching settings and expiry behavior.
+
+```python
+conn = Connection(
+    host=host,
+    port=443,
+    database=database,
+    cluster_name='<cluster-name>',
+    secure=True,
+    client_id='<client_id>',
+    client_secret='<client_secret>',
+    token_url='https://<your-workspace>/oauth2/token',
+)
+```
+
+If you mint tokens yourself, pass one directly instead. The connector will not refresh it, so a
+long-lived connection may outlive the token:
+
+```python
+conn = Connection(
+    host=host, port=443, database=database, cluster_name='<cluster-name>',
+    secure=True, access_token='<token>',
+)
+```
+
+**Supply exactly one authentication method.** Passing both username/password and OAuth settings
+raises a `ValueError` rather than picking one, so a stale value left in a config file cannot quietly
+win.
+
+OAuth sends `authorization: Bearer <token>` on every RPC. It does not create an
+authentication session; protobuf username, password and session ID fields remain
+empty. The server stack must support and enable this bearer-only flow.
+
+Use `secure=True` with your TLS endpoint and an HTTPS `token_url` to protect tokens
+and client credentials. OAuth does not enable TLS automatically; `secure` defaults
+to `False`. Set `cluster_name` to identify the target cluster explicitly.
+
+With `auto_resume=True` (the default), OAuth can recover an initial query prepare
+that returns the exact suspended-cluster response. This requires compatible,
+enabled server components that authorize resume and report readiness. The connector
+performs one recovery sequence and retries prepare after readiness. It never
+replays query execution, metadata retrieval or result fetching, and it does not
+treat generic connection failures as suspension.
+
+Recovery has a shared 300-second default deadline covering lock waits, token
+acquisition, status/resume calls, polling and the resumed prepare. Configure it with
+`grpc_options={'grpc_auto_resume_timeout_seconds': 300}`. The deadline starts after
+the suspended response; initial prepare has its own timeout. A dispatched resume
+may still complete after the client times out.
+
+Set `auto_resume=False` to disable recovery. Invalid credentials, denied permissions,
+unsupported servers and failed or unknown cluster states stop recovery without
+falling back to username/password authentication. Externally supplied access tokens
+are not refreshed.
 
 #### Secure Connection Example
 

e6data_python_connector-2.3.14/e6data_python_connector.egg-info/PKG-INFO → e6data_python_connector-3.0.0/README.md

@@ -1,49 +1,30 @@
-Metadata-Version: 2.2
-Name: e6data-python-connector
-Version: 2.3.14
-Summary: Client for the e6data distributed SQL Engine.
-Home-page: https://github.com/e6x-labs/e6data-python-connector
-Author: e6data, Inc.
-Author-email: info@e6data.com
-License: Apache 2.0
-Classifier: Operating System :: POSIX :: Linux
-Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python :: 3.5
-Classifier: Programming Language :: Python :: 3.6
-Classifier: Programming Language :: Python :: 3.7
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: sqlalchemy>=1.0.0
-Requires-Dist: future>=1.0.0
-Requires-Dist: python-dateutil>=2.9.0
-Requires-Dist: pycryptodome>=3.19.1
-Requires-Dist: pytz>=2024.1
-Requires-Dist: thrift>=0.20.0
-Requires-Dist: grpcio>=1.65.1
-Requires-Dist: grpcio-tools>=1.65.1
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
-Dynamic: requires-dist
-Dynamic: summary
-
 # e6data Python Connector
 
-![version](https://img.shields.io/badge/version-2.3.14-blue.svg)
+Package version and extras are defined in `setup.py`.
 
 ## Introduction
 
-The e6data Connector for Python provides an interface for writing Python applications that can connect to e6data and perform operations. It includes automatic support for blue-green deployments, ensuring seamless failover during server updates without query interruption.
+The e6data Connector for Python provides an interface for writing Python applications that can connect to e6data and perform operations. It includes routing support for blue-green deployments. Active query handles remain tied to the engine that accepted them; failover does not guarantee uninterrupted results.
+
+## Optional native asyncio API
+
+This branch adds native async connection, cursor, pooling and SQLAlchemy support for Python 3.11+. Release qualification remains incomplete. Install this branch's built package with `[async]` or `[async-sqlalchemy]`; these instructions do not assert that the feature is published.
+
+```python
+from e6data_python_connector.aio import AsyncConnection
+
+async def query(config, sql):
+    async with AsyncConnection(**config) as connection:
+        async with connection.cursor() as cursor:
+            await cursor.execute(sql)
+            return await cursor.fetchall()
+```
+
+`config` is application-supplied connection configuration. OAuth requires verified TLS. Native `fetchone()` preserves the one-row outer list, while SQLAlchemy `e6data+asyncio` adapts it to a normal row. Ambiguous submission and incomplete result errors must not be handled by blindly retrying the query. Use context managers for bounded cleanup.
+
+See [Async API](docs/ASYNC_API.md) for the complete API map, deadlines, pooling, SQLAlchemy and test configuration, [OAuth lifecycle](docs/OAUTH_LIFECYCLE.md) for renewal and failure behavior, and [the application example](examples/async_query.py).
+
+The new CI keeps the full-package coverage denominator and a greater-than-80% gate. The initial baseline was about 19%, so focused async test success is not a release pass. Real-service tests require explicit configuration and are qualified separately.
 
 ### Dependencies
 Make sure to install below dependencies and wheel before install e6data-python-connector.
@@ -120,8 +101,8 @@ The `Connection` class supports the following parameters:
 |-----------|------|----------|---------|-------------|
 | `host` | str | Yes | - | IP address or hostname of the e6data cluster |
 | `port` | int | Yes | - | Port of the e6data engine (typically 80) |
-| `username` | str | Yes | - | Your e6data Email ID |
-| `password` | str | Yes | - | Access Token generated in the e6data console |
+| `username` | str | Conditional | None | Your e6data Email ID. Required unless authenticating with OAuth |
+| `password` | str | Conditional | None | Access Token generated in the e6data console. Required unless authenticating with OAuth |
 | `database` | str | No | None | Database to perform queries on |
 | `catalog` | str | No | None | Catalog name |
 | `cluster_name` | str | No | None | Name of the cluster for cluster-specific operations |
@@ -131,6 +112,76 @@ The `Connection` class supports the following parameters:
 | `grpc_options` | dict | No | None | Additional gRPC configuration options |
 | `debug` | bool | No | False | Enable debug logging for troubleshooting |
 | `require_fastbinary` | bool | No | True | Require fastbinary module for Thrift deserialization. Set to False to use pure Python implementation if system dependencies cannot be installed |
+| `client_id` | str | No | None | OAuth 2.0 client id. Use with `client_secret` and `token_url` |
+| `client_secret` | str | No | None | OAuth 2.0 client secret |
+| `token_url` | str | No | None | Token endpoint of the authorization server |
+| `oauth_scope` | str | No | None | Space-delimited scopes to request. Omit for the client's full registered set |
+| `access_token` | str | No | None | A previously obtained access token, for callers that mint their own |
+| `client_auth_method` | str | No | `'basic'` | How client credentials reach the token endpoint: `'basic'` or `'post'` |
+
+#### Authenticating with OAuth 2.0
+
+As an alternative to username and password, a connection can authenticate with an OAuth 2.0 access
+token. The connector obtains a token using the client-credentials grant, caches it, and refreshes it
+shortly before it expires.
+
+Matching client-credentials configurations share one token cache and concurrent
+refresh across sync and async connections, separate pools and threads in the same
+process. Connection and transport ownership remain local. See the
+[OAuth lifecycle](docs/OAUTH_LIFECYCLE.md) for matching settings and expiry behavior.
+
+```python
+conn = Connection(
+    host=host,
+    port=443,
+    database=database,
+    cluster_name='<cluster-name>',
+    secure=True,
+    client_id='<client_id>',
+    client_secret='<client_secret>',
+    token_url='https://<your-workspace>/oauth2/token',
+)
+```
+
+If you mint tokens yourself, pass one directly instead. The connector will not refresh it, so a
+long-lived connection may outlive the token:
+
+```python
+conn = Connection(
+    host=host, port=443, database=database, cluster_name='<cluster-name>',
+    secure=True, access_token='<token>',
+)
+```
+
+**Supply exactly one authentication method.** Passing both username/password and OAuth settings
+raises a `ValueError` rather than picking one, so a stale value left in a config file cannot quietly
+win.
+
+OAuth sends `authorization: Bearer <token>` on every RPC. It does not create an
+authentication session; protobuf username, password and session ID fields remain
+empty. The server stack must support and enable this bearer-only flow.
+
+Use `secure=True` with your TLS endpoint and an HTTPS `token_url` to protect tokens
+and client credentials. OAuth does not enable TLS automatically; `secure` defaults
+to `False`. Set `cluster_name` to identify the target cluster explicitly.
+
+With `auto_resume=True` (the default), OAuth can recover an initial query prepare
+that returns the exact suspended-cluster response. This requires compatible,
+enabled server components that authorize resume and report readiness. The connector
+performs one recovery sequence and retries prepare after readiness. It never
+replays query execution, metadata retrieval or result fetching, and it does not
+treat generic connection failures as suspension.
+
+Recovery has a shared 300-second default deadline covering lock waits, token
+acquisition, status/resume calls, polling and the resumed prepare. Configure it with
+`grpc_options={'grpc_auto_resume_timeout_seconds': 300}`. The deadline starts after
+the suspended response; initial prepare has its own timeout. A dispatched resume
+may still complete after the client times out.
+
+Set `auto_resume=False` to disable recovery. Invalid credentials, denied permissions,
+unsupported servers and failed or unknown cluster states stop recovery without
+falling back to username/password authentication. Externally supplied access tokens
+are not refreshed.
 
 #### Secure Connection Example
 

e6data_python_connector-3.0.0/e6data_python_connector/aio.py

@@ -0,0 +1,26 @@
+"""Explicit optional native asyncio entry point (Python 3.11+)."""
+import sys
+
+if sys.version_info < (3, 11):
+    raise RuntimeError('The async API requires Python 3.11 or newer.')
+try:
+    import httpx as _httpx
+except ImportError:
+    raise ImportError('Install e6data-python-connector[async] for the async API.') from None
+
+from .async_connection import AsyncConnection
+
+__all__ = ['connect', 'AsyncConnection', 'AsyncCursor', 'AsyncConnectionPool', 'AsyncClusterManager']
+
+
+async def connect(*args, **kwargs):
+    return await AsyncConnection(*args, **kwargs).open()
+
+
+def __getattr__(name):
+    modules = {'AsyncCursor': 'async_cursor', 'AsyncConnectionPool': 'async_connection_pool',
+               'AsyncClusterManager': 'async_cluster_manager'}
+    if name not in modules:
+        raise AttributeError(name)
+    from importlib import import_module
+    return getattr(import_module('.' + modules[name], __package__), name)

e6data_python_connector-3.0.0/e6data_python_connector/async_cluster_manager.py

@@ -0,0 +1,267 @@
+"""Native async cluster recovery with owner-scoped mutation tracking."""
+import asyncio
+import math
+import os
+import threading
+
+import grpc
+
+from .cluster_server import cluster_pb2 as pb, cluster_pb2_grpc as bindings
+from .exceptions import OperationalError, ProgrammingError
+from .oauth_common import validate_positive_timeout
+
+
+def is_strategy_mismatch(error):
+    return error.code() == grpc.StatusCode.UNKNOWN and error.details() == 'status: 456'
+
+
+def is_suspended_error(error):
+    return (error.code() == grpc.StatusCode.UNAVAILABLE
+            and error.details() == 'status: 503, cluster is suspended')
+
+
+def _remaining(deadline):
+    result = deadline - asyncio.get_running_loop().time()
+    if result <= 0:
+        raise TimeoutError('Cluster recovery deadline exceeded; dispatched resume may still complete.')
+    return result
+
+
+def _read_certificate(path):
+    with open(path, 'rb') as source:
+        return source.read()
+
+
+class _ResumeState:
+    """Mutation state deliberately outlives any individual recovery flight."""
+    def __init__(self):
+        self.pending = None
+
+    def observe(self, status):
+        if status not in ('active', 'suspended', 'resuming'):
+            raise RuntimeError('Cluster returned failed or unsupported status: {!r}'.format(status))
+        if status == 'active':
+            self.pending = None
+        return status == 'suspended' and self.pending is None
+
+    def dispatched(self):
+        self.pending = 'unknown'
+
+    def acknowledged(self):
+        self.pending = 'acknowledged'
+
+
+class _ResumeFlight:
+    """One phase deadline, independently bounded shielded waiters."""
+    def __init__(self, timeout):
+        self.timeout = timeout
+        self.task = None
+        self.waiters = 0
+        self.retiring = False
+
+    @staticmethod
+    def _consume_result(task):
+        if not task.cancelled():
+            task.exception()
+
+    async def wait(self, work, deadline):
+        # Last-waiter disposal must finish before a replacement can start.
+        while self.retiring and self.task is not None:
+            retiring_task = self.task
+            try:
+                async with asyncio.timeout_at(deadline):
+                    await asyncio.shield(retiring_task)
+            except asyncio.CancelledError:
+                if asyncio.current_task().cancelling():
+                    raise
+            except Exception:
+                if not retiring_task.done():
+                    raise
+            if self.task is retiring_task and retiring_task.done():
+                self.task = None
+                self.retiring = False
+        if deadline is not None:
+            _remaining(deadline)
+        if self.task is None or (self.task.done() and self.waiters == 0):
+            phase_deadline = asyncio.get_running_loop().time() + self.timeout
+            self.task = asyncio.create_task(work(phase_deadline))
+            self.task.add_done_callback(self._consume_result)
+        task = self.task
+        self.waiters += 1
+        try:
+            async with asyncio.timeout_at(deadline):
+                return await asyncio.shield(task)
+        finally:
+            self.waiters -= 1
+            if self.waiters == 0 and not task.done():
+                self.retiring = True
+                task.cancel()
+
+
+class AsyncClusterManager:
+    """Local constructor; channels belong to the first invoking loop/thread/process.
+
+    Sharing this explicit object shares recovery only within its target and auth
+    owner. OAuth metadata providers must be async and own their refresh policy.
+    """
+    def __init__(self, host, port, user='', password='', secure_channel=False,
+                 timeout=300, cluster_uuid=None, grpc_options=None, debug=False,
+                 ssl_cert=None, metadata_provider=None, initial_strategy=None,
+                 auto_resume_timeout=None, cleanup_timeout=5):
+        if not isinstance(host, str) or not host or isinstance(port, bool) or not isinstance(port, int) or not 0 < port < 65536:
+            raise ValueError('host and a valid port are required.')
+        if metadata_provider is not None and not secure_channel:
+            raise ValueError('Async OAuth cluster recovery requires verified TLS.')
+        if metadata_provider is not None and (user or password):
+            raise ValueError('OAuth and password authentication cannot be combined.')
+        if metadata_provider is None and (not user or not password):
+            raise ValueError('user and password are required for legacy recovery.')
+        if initial_strategy not in (None, 'blue', 'green'):
+            raise ValueError('initial_strategy must be blue or green.')
+        if ssl_cert is not None and not isinstance(ssl_cert, (str, bytes)):
+            raise ValueError('ssl_cert must be a certificate path or PEM bytes.')
+        options = dict(grpc_options or {})
+        if any(key.removeprefix('grpc.') in ('ssl_target_name_override', 'default_authority') for key in options):
+            raise ValueError('TLS authority overrides are unsupported.')
+        limit = options.get('grpc.max_receive_message_length', 64 * 1024 * 1024)
+        if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
+            raise ValueError('A finite positive receive limit is required.')
+        options['grpc.max_receive_message_length'] = limit
+        self._host, self._port = host, port
+        self._user, self._password = user, password
+        self._secure_channel, self._ssl_cert = secure_channel, ssl_cert
+        self._grpc_options = tuple(options.items())
+        self.cluster_uuid = cluster_uuid
+        self._metadata_provider = metadata_provider
+        self.last_successful_strategy = initial_strategy
+        self._timeout = validate_positive_timeout(timeout if auto_resume_timeout is None else auto_resume_timeout)
+        self._cleanup_timeout = validate_positive_timeout(cleanup_timeout)
+        self._flight = _ResumeFlight(self._timeout)
+        self._mutation = _ResumeState()
+        self._owner = None
+        self._closed = False
+
+    def _check_owner(self):
+        owner = (os.getpid(), threading.get_ident(), asyncio.get_running_loop())
+        if self._owner is None:
+            self._owner = owner
+        elif self._owner != owner:
+            raise ProgrammingError('AsyncClusterManager cannot cross loops, threads or processes.')
+        if self._closed:
+            raise ProgrammingError('AsyncClusterManager is closed.')
+
+    async def resume(self, *, timeout=None, deadline=None):
+        self._check_owner()
+        if deadline is not None and (isinstance(deadline, bool) or not isinstance(deadline, (int, float)) or not math.isfinite(deadline)):
+            raise ValueError('deadline must be finite.')
+        if timeout is not None:
+            bound = asyncio.get_running_loop().time() + validate_positive_timeout(timeout)
+            deadline = bound if deadline is None else min(bound, deadline)
+        try:
+            if deadline is not None:
+                _remaining(deadline)
+            return await self._flight.wait(self._recover, deadline)
+        except (TimeoutError, grpc.RpcError) as error:
+            raise OperationalError('Cluster recovery failed; a dispatched resume may still complete.') from error
+
+    async def _metadata(self, strategy, deadline, rejected_token=None):
+        _remaining(deadline)
+        if self._metadata_provider is not None:
+            async with asyncio.timeout_at(deadline):
+                if rejected_token is None:
+                    return await self._metadata_provider(strategy, deadline)
+                return await self._metadata_provider(strategy, deadline, rejected_token=rejected_token)
+        values = [('strategy', strategy)]
+        if self.cluster_uuid:
+            values.append(('cluster-name', self.cluster_uuid))
+        return values
+
+    async def _request(self, client, operation, deadline):
+        request_type = pb.ClusterStatusRequest if operation == 'status' else pb.ResumeRequest
+        request = request_type() if self._metadata_provider else request_type(user=self._user, password=self._password)
+        first = self.last_successful_strategy or 'blue'
+        refreshed = False
+        for strategy in (first, 'green' if first == 'blue' else 'blue'):
+            metadata = await self._metadata(strategy, deadline)
+            while True:
+                try:
+                    remaining = _remaining(deadline)
+                    if operation == 'resume':
+                        self._mutation.dispatched()
+                    response = await getattr(client, operation)(request, metadata=metadata, timeout=remaining)
+                    if operation == 'resume':
+                        self._mutation.acknowledged()
+                    self.last_successful_strategy = strategy
+                    return response
+                except grpc.RpcError as error:
+                    if (error.code() == grpc.StatusCode.UNAUTHENTICATED and not refreshed
+                            and self._metadata_provider is not None):
+                        refreshed = True
+                        rejected = dict(metadata).get('authorization', '')[len('Bearer '):]
+                        replacement = await self._metadata(strategy, deadline, rejected)
+                        if replacement is not None:
+                            metadata = replacement
+                            continue
+                    if strategy == first and is_strategy_mismatch(error):
+                        break
+                    raise
+
+    async def _recover(self, deadline):
+        channel = None
+        try:
+            async with asyncio.timeout_at(deadline):
+                certificate = self._ssl_cert
+                if isinstance(certificate, str):
+                    from .async_work import run_blocking
+                    certificate = await run_blocking(_read_certificate, certificate, deadline=deadline)
+                target = '{}:{}'.format(self._host, self._port)
+                if self._secure_channel:
+                    channel = grpc.aio.secure_channel(target, grpc.ssl_channel_credentials(root_certificates=certificate), options=self._grpc_options)
+                else:
+                    channel = grpc.aio.insecure_channel(target, options=self._grpc_options)
+                client = bindings.ClusterServiceStub(channel)
+                while True:
+                    response = await self._request(client, 'status', deadline)
+                    should_resume = self._mutation.observe(response.status)
+                    if response.status == 'active':
+                        return True
+                    if should_resume:
+                        try:
+                            result = await self._request(client, 'resume', deadline)
+                            self._mutation.observe(result.status)
+                            # Only a status read resolves readiness, even if resume says active.
+                            self._mutation.acknowledged()
+                        except grpc.RpcError as error:
+                            if error.code() not in (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.DEADLINE_EXCEEDED, grpc.StatusCode.CANCELLED):
+                                raise
+                        continue
+                    await asyncio.sleep(min(1.0, _remaining(deadline)))
+        finally:
+            if channel is not None:
+                async with asyncio.timeout(self._cleanup_timeout):
+                    await channel.close()
+
+    async def close(self, *, deadline=None):
+        if self._owner is not None:
+            owner = (os.getpid(), threading.get_ident(), asyncio.get_running_loop())
+            if self._owner != owner:
+                raise ProgrammingError('AsyncClusterManager cannot cross loops, threads or processes.')
+        if self._closed and (self._flight.task is None or self._flight.task.done()):
+            return
+        if not self._closed:
+            self._check_owner()
+        self._closed = True
+        bound = asyncio.get_running_loop().time() + self._cleanup_timeout
+        deadline = bound if deadline is None else min(bound, deadline)
+        task = self._flight.task
+        if task is not None:
+            task.cancel()
+            try:
+                async with asyncio.timeout_at(deadline):
+                    await asyncio.shield(task)
+            except asyncio.CancelledError:
+                if asyncio.current_task().cancelling():
+                    raise
+            except Exception:
+                if not task.done():
+                    raise