great-expectations-cloud 20240625.0__tar.gz → 20260114.0.dev0__tar.gz

{great_expectations_cloud-20240625.0 → great_expectations_cloud-20260114.0.dev0}/PKG-INFO

@@ -1,40 +1,33 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.3
 Name: great_expectations_cloud
-Version: 20240625.0
+Version: 20260114.0.dev0
 Summary: Great Expectations Cloud
-Home-page: https://greatexpectations.io
 License: Proprietary
 Author: The Great Expectations Team
 Author-email: team@greatexpectations.io
-Requires-Python: >=3.8,<3.12
+Requires-Python: >=3.11.4,<3.12
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: License :: Other/Proprietary License
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Quality Assurance
 Classifier: Topic :: Software Development :: Testing
-Provides-Extra: postgres
-Provides-Extra: snowflake
-Requires-Dist: great-expectations (>=0.18.16,<0.19.0)
+Requires-Dist: great-expectations[databricks,gx-redshift,mssql,postgresql,snowflake,trino] (==1.11.0)
 Requires-Dist: orjson (>=3.9.7,<4.0.0,!=3.9.10)
-Requires-Dist: packaging (>=21.3,<25.0)
+Requires-Dist: packaging (>=21.3,<26.0)
 Requires-Dist: pika (>=1.3.1,<2.0.0)
-Requires-Dist: psycopg2-binary ; extra == "postgres"
-Requires-Dist: pydantic (<3)
-Requires-Dist: snowflake-sqlalchemy ; extra == "snowflake"
-Requires-Dist: sqlalchemy ; extra == "postgres"
-Requires-Dist: tenacity (>=8.2.3,<9.0.0)
+Requires-Dist: pydantic (>=2.8.1,<3)
+Requires-Dist: setuptools (==80.9.0)
+Requires-Dist: sqlalchemy (>=2.0)
+Requires-Dist: tenacity (>=8.2.3,<10.0.0)
+Project-URL: Homepage, https://greatexpectations.io
 Project-URL: Repository, https://github.com/great-expectations/cloud
 Description-Content-Type: text/markdown
 
-# cloud
+# GX cloud
 
 [![PyPI](https://img.shields.io/pypi/v/great_expectations_cloud)](https://pypi.org/project/great-expectations_cloud/#history)
 [![Docker Pulls](https://img.shields.io/docker/pulls/greatexpectations/agent)](https://hub.docker.com/r/greatexpectations/agent)
@@ -45,19 +38,30 @@ Description-Content-Type: text/markdown
 
 ## Quick Start
 
-### Python
+To use the GX Agent, you will need to have a Great Expectations Cloud account. You can sign up for free at [https://app.greatexpectations.io](https://app.greatexpectations.io).
 
-#### Install
+Deployment instructions for the GX Agent can be found in the [GX Cloud documentation](https://docs.greatexpectations.io/docs/cloud/deploy/deploy_gx_agent).
 
-```console
-pip install great_expectations_cloud
-```
+## Dev Setup
 
-##### Optional Dependencies
+The following instructions are for those who are contributing to the GX Agent, to deploy and use the GX agent please see the Quick Start section above.
 
-```console
-pip install 'great_expectations_cloud[sql]'
-```
+See also [CONTRIBUTING.md](https://github.com/great-expectations/cloud/blob/main/CONTRIBUTING.md)
+
+1. [Install or upgrade `poetry` to the latest version](https://python-poetry.org/docs/#installing-with-pipx)
+   - `pipx install poetry` or `pipx upgrade poetry`
+2. Set up virtual environment and install dependencies
+   - `poetry sync`
+3. Activate your virtual environment
+   - `eval $(poetry env activate)`
+4. Set up precommit hooks
+   - `pre-commit install`
+
+### Troubleshooting
+
+If you run into issues, you can try `pipx reinstall-all`
+
+### Running locally for development
 
 ```console
 $ gx-agent --help
@@ -74,45 +78,33 @@ optional arguments:
   --version             Show the GX Agent version.
 ```
 
-#### Set env variables
+#### Set ENV variables
 
 `GX_CLOUD_ACCESS_TOKEN`
 `GX_CLOUD_ORGANIZATION_ID`
 
-### Start the Agent
+If you want to override where the GX Agent looks for the RabbitMQ queue you can also set
+`AMQP_HOST_OVERRIDE` and `AMQP_PORT_OVERRIDE`. For example, if you are running a local dockerized RabbitMQ
+service exposed on localhost port 5672, you can set `AMQP_HOST_OVERRIDE=127.0.0.1` and
+`AMQP_PORT_OVERRIDE=5672`.
+
+### Start the GX Agent
 
-If you intend to run the Agent against local services (Cloud backend or datasources) run the Agent outside of the container.
+If you intend to run the GX Agent against local services (Cloud backend or datasources) run the Agent outside of the container.
 
 ```
 gx-agent
 ```
 
-### Docker
-
-[Building and running the Agent with Docker](#building-and-running-the-gx-agent-image)
-
-## Dev Setup
-
-See also [CONTRIBUTING.md](https://github.com/great-expectations/cloud/blob/main/CONTRIBUTING.md)
-
-1. [Install `poetry`](https://python-poetry.org/docs/#installation)
-   - [`pipx install poetry`](https://python-poetry.org/docs/#installing-with-pipx)
-2. Set up virtual environment and install dependencies
-   - `poetry install --sync`
-3. Activate your virtual environment
-   - `poetry shell`
-4. Set up precommit hooks
-   - `pre-commit install`
-
 ### Developer Tasks
 
-Common developer tasks are available via `invoke` (defined in `tasks.py`)
+Common developer tasks are available via `invoke` (defined in `tasks.py`).
 
 `invoke --list` to see available tasks.
 
 #### Synchronize Dependencies
 
-To ensure you are using the latest version of the core and development dependencies run `poetry install --sync`.
+To ensure you are using the latest version of the core and development dependencies run `poetry sync`.
 Also available as an invoke task.
 
 ```console
@@ -121,6 +113,11 @@ invoke deps
 
 #### Updating `poetry.lock` dependencies
 
+Use the latest version of poetry
+```console
+pipx upgrade poetry
+```
+
 The dependencies installed in our CI and the Docker build step are determined by the [poetry.lock file](https://python-poetry.org/docs/basic-usage/#installing-with-poetrylock).
 
 [To update only a specific dependency](https://python-poetry.org/docs/cli/#update) (such as `great_expectations`) ...
@@ -129,6 +126,8 @@ The dependencies installed in our CI and the Docker build step are determined by
 poetry update great_expectations
 ```
 
+**Note:** If `poetry update` does not find the latest version of `great_expectations`, you can manually update the version in `pyproject.toml`, and then update the lockfile using `poetry lock`.
+
 [To resolve and update all dependencies ...](https://python-poetry.org/docs/cli/#lock)
 
 ```console
@@ -252,13 +251,15 @@ We use the GitHub Actions workflow to automate the release and pre-release proce
 A visual representation of the workflow is shown [here](https://github.com/great-expectations/cloud/blob/main/.github/workflows/agent_release_workflows.png)
 
 ### Dependabot and Releases/Pre-releases
+
 GitHub's Dependabot regularly checks our dependencies for vulnerabilty-based updates and proposes PRs to update dependency version numbers accordingly.
 
 Dependabot may only update the `poetry.lock` file. If only changes to `poetry.lock` are made, this may be done in a pre-release.
 
 For changes to the `pyproject.toml` file:
+
 - If the version of a tool in the `[tool.poetry.group.dev.dependencies]` group is updated, this may be done without any version bump.
-   -  While doing this, make sure any version references in the pre-commit config `.pre-commit-config.yaml` are kept in sync (e.g., ruff).
+  - While doing this, make sure any version references in the pre-commit config `.pre-commit-config.yaml` are kept in sync (e.g., ruff).
 - For other dependency updates or package build metadata changes, a new release should be orchestrated. This includes updates in the following sections:
   - `[tool.poetry.dependencies]`
   - `[tool.poetry.group.*.dependencies]` where `*` is the name of the group (not including the `dev` group)
@@ -266,3 +267,5 @@ For changes to the `pyproject.toml` file:
   - Only modifying dev dependencies.
   - Only modifying tests that do not change functionality.
 
+NOTE: Dependabot does not have permissions to access secrets in our CI. You may notice that integration tests fail on PRs that dependabot creates. If you add a commit (as a GX member) to the PR, the tests will run again and pass because they now have access to the secrets. That commit can be anything, including an empty commit e.g. `git commit -m "some message" --allow-empty`.
+

{great_expectations_cloud-20240625.0 → great_expectations_cloud-20260114.0.dev0}/README.md

@@ -1,4 +1,4 @@
-# cloud
+# GX cloud
 
 [![PyPI](https://img.shields.io/pypi/v/great_expectations_cloud)](https://pypi.org/project/great-expectations_cloud/#history)
 [![Docker Pulls](https://img.shields.io/docker/pulls/greatexpectations/agent)](https://hub.docker.com/r/greatexpectations/agent)
@@ -9,19 +9,30 @@
 
 ## Quick Start
 
-### Python
+To use the GX Agent, you will need to have a Great Expectations Cloud account. You can sign up for free at [https://app.greatexpectations.io](https://app.greatexpectations.io).
 
-#### Install
+Deployment instructions for the GX Agent can be found in the [GX Cloud documentation](https://docs.greatexpectations.io/docs/cloud/deploy/deploy_gx_agent).
 
-```console
-pip install great_expectations_cloud
-```
+## Dev Setup
 
-##### Optional Dependencies
+The following instructions are for those who are contributing to the GX Agent, to deploy and use the GX agent please see the Quick Start section above.
 
-```console
-pip install 'great_expectations_cloud[sql]'
-```
+See also [CONTRIBUTING.md](https://github.com/great-expectations/cloud/blob/main/CONTRIBUTING.md)
+
+1. [Install or upgrade `poetry` to the latest version](https://python-poetry.org/docs/#installing-with-pipx)
+   - `pipx install poetry` or `pipx upgrade poetry`
+2. Set up virtual environment and install dependencies
+   - `poetry sync`
+3. Activate your virtual environment
+   - `eval $(poetry env activate)`
+4. Set up precommit hooks
+   - `pre-commit install`
+
+### Troubleshooting
+
+If you run into issues, you can try `pipx reinstall-all`
+
+### Running locally for development
 
 ```console
 $ gx-agent --help
@@ -38,45 +49,33 @@ optional arguments:
   --version             Show the GX Agent version.
 ```
 
-#### Set env variables
+#### Set ENV variables
 
 `GX_CLOUD_ACCESS_TOKEN`
 `GX_CLOUD_ORGANIZATION_ID`
 
-### Start the Agent
+If you want to override where the GX Agent looks for the RabbitMQ queue you can also set
+`AMQP_HOST_OVERRIDE` and `AMQP_PORT_OVERRIDE`. For example, if you are running a local dockerized RabbitMQ
+service exposed on localhost port 5672, you can set `AMQP_HOST_OVERRIDE=127.0.0.1` and
+`AMQP_PORT_OVERRIDE=5672`.
+
+### Start the GX Agent
 
-If you intend to run the Agent against local services (Cloud backend or datasources) run the Agent outside of the container.
+If you intend to run the GX Agent against local services (Cloud backend or datasources) run the Agent outside of the container.
 
 ```
 gx-agent
 ```
 
-### Docker
-
-[Building and running the Agent with Docker](#building-and-running-the-gx-agent-image)
-
-## Dev Setup
-
-See also [CONTRIBUTING.md](https://github.com/great-expectations/cloud/blob/main/CONTRIBUTING.md)
-
-1. [Install `poetry`](https://python-poetry.org/docs/#installation)
-   - [`pipx install poetry`](https://python-poetry.org/docs/#installing-with-pipx)
-2. Set up virtual environment and install dependencies
-   - `poetry install --sync`
-3. Activate your virtual environment
-   - `poetry shell`
-4. Set up precommit hooks
-   - `pre-commit install`
-
 ### Developer Tasks
 
-Common developer tasks are available via `invoke` (defined in `tasks.py`)
+Common developer tasks are available via `invoke` (defined in `tasks.py`).
 
 `invoke --list` to see available tasks.
 
 #### Synchronize Dependencies
 
-To ensure you are using the latest version of the core and development dependencies run `poetry install --sync`.
+To ensure you are using the latest version of the core and development dependencies run `poetry sync`.
 Also available as an invoke task.
 
 ```console
@@ -85,6 +84,11 @@ invoke deps
 
 #### Updating `poetry.lock` dependencies
 
+Use the latest version of poetry
+```console
+pipx upgrade poetry
+```
+
 The dependencies installed in our CI and the Docker build step are determined by the [poetry.lock file](https://python-poetry.org/docs/basic-usage/#installing-with-poetrylock).
 
 [To update only a specific dependency](https://python-poetry.org/docs/cli/#update) (such as `great_expectations`) ...
@@ -93,6 +97,8 @@ The dependencies installed in our CI and the Docker build step are determined by
 poetry update great_expectations
 ```
 
+**Note:** If `poetry update` does not find the latest version of `great_expectations`, you can manually update the version in `pyproject.toml`, and then update the lockfile using `poetry lock`.
+
 [To resolve and update all dependencies ...](https://python-poetry.org/docs/cli/#lock)
 
 ```console
@@ -216,16 +222,20 @@ We use the GitHub Actions workflow to automate the release and pre-release proce
 A visual representation of the workflow is shown [here](https://github.com/great-expectations/cloud/blob/main/.github/workflows/agent_release_workflows.png)
 
 ### Dependabot and Releases/Pre-releases
+
 GitHub's Dependabot regularly checks our dependencies for vulnerabilty-based updates and proposes PRs to update dependency version numbers accordingly.
 
 Dependabot may only update the `poetry.lock` file. If only changes to `poetry.lock` are made, this may be done in a pre-release.
 
 For changes to the `pyproject.toml` file:
+
 - If the version of a tool in the `[tool.poetry.group.dev.dependencies]` group is updated, this may be done without any version bump.
-   -  While doing this, make sure any version references in the pre-commit config `.pre-commit-config.yaml` are kept in sync (e.g., ruff).
+  - While doing this, make sure any version references in the pre-commit config `.pre-commit-config.yaml` are kept in sync (e.g., ruff).
 - For other dependency updates or package build metadata changes, a new release should be orchestrated. This includes updates in the following sections:
   - `[tool.poetry.dependencies]`
   - `[tool.poetry.group.*.dependencies]` where `*` is the name of the group (not including the `dev` group)
 - To stop the auto-version bump add the `no version bump` label to the PR. Use this when:
   - Only modifying dev dependencies.
   - Only modifying tests that do not change functionality.
+
+NOTE: Dependabot does not have permissions to access secrets in our CI. You may notice that integration tests fail on PRs that dependabot creates. If you add a commit (as a GX member) to the PR, the tests will run again and pass because they now have access to the secrets. That commit can be anything, including an empty commit e.g. `git commit -m "some message" --allow-empty`.

{great_expectations_cloud-20240625.0 → great_expectations_cloud-20260114.0.dev0}/great_expectations_cloud/agent/__init__.py

@@ -2,4 +2,7 @@ from __future__ import annotations
 
 import great_expectations_cloud.agent.actions  # import actions to register them
 from great_expectations_cloud.agent.agent import GXAgent
+from great_expectations_cloud.agent.event_handler import register_event_action
+from great_expectations_cloud.agent.models import Event, EventType
 from great_expectations_cloud.agent.run import get_version, run_agent
+from great_expectations_cloud.agent.utils import triangular_interpolation

{great_expectations_cloud-20240625.0 → great_expectations_cloud-20260114.0.dev0}/great_expectations_cloud/agent/actions/__init__.py

@@ -7,16 +7,16 @@ from great_expectations_cloud.agent.actions.agent_action import (
 )
 
 # Import all actions to register them:
-from great_expectations_cloud.agent.actions.data_assistants import (
-    RunMissingnessDataAssistantAction,
-    RunOnboardingDataAssistantAction,
-)
 from great_expectations_cloud.agent.actions.draft_datasource_config_action import (
     DraftDatasourceConfigAction,
 )
-from great_expectations_cloud.agent.actions.list_table_names import ListTableNamesAction
+from great_expectations_cloud.agent.actions.generate_data_quality_check_expectations_action import (
+    GenerateDataQualityCheckExpectationsAction,
+)
+from great_expectations_cloud.agent.actions.list_asset_names import ListAssetNamesAction
 from great_expectations_cloud.agent.actions.run_checkpoint import RunCheckpointAction
 from great_expectations_cloud.agent.actions.run_metric_list_action import MetricListAction
 from great_expectations_cloud.agent.actions.run_scheduled_checkpoint import (
     RunScheduledCheckpointAction,
 )
+from great_expectations_cloud.agent.actions.run_window_checkpoint import RunWindowCheckpointAction

great_expectations_cloud-20260114.0.dev0/great_expectations_cloud/agent/actions/agent_action.py

@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import datetime
+from abc import abstractmethod
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Generic, Optional, TypeVar, Union
+
+from pydantic.v1 import BaseModel
+
+from great_expectations_cloud.agent.models import (
+    AgentBaseExtraForbid,
+    AgentBaseExtraIgnore,
+    CreatedResource,
+    DomainContext,
+)
+
+if TYPE_CHECKING:
+    from great_expectations.data_context import CloudDataContext
+
+
+class ActionResult(BaseModel):
+    id: str
+    type: str
+    created_resources: Sequence[CreatedResource]
+    job_duration: Optional[datetime.timedelta] = (  # noqa: UP045
+        None  # Python 3.8 doesn't support `X | Y` for type annotation
+    )
+
+
+_EventT = TypeVar("_EventT", bound=Union[AgentBaseExtraForbid, AgentBaseExtraIgnore])
+
+
+class AgentAction(Generic[_EventT]):
+    def __init__(
+        self, context: CloudDataContext, base_url: str, domain_context: DomainContext, auth_key: str
+    ):
+        self._context = context
+        self._base_url = base_url
+        self._domain_context = domain_context
+        self._auth_key = auth_key
+
+    @abstractmethod
+    def run(self, event: _EventT, id: str) -> ActionResult: ...

{great_expectations_cloud-20240625.0 → great_expectations_cloud-20260114.0.dev0}/great_expectations_cloud/agent/actions/draft_datasource_config_action.py

@@ -1,18 +1,16 @@
 from __future__ import annotations
 
 from typing import Any
+from urllib.parse import urljoin
 from uuid import UUID
 
-from great_expectations.compatibility import pydantic
 from great_expectations.core.http import create_session
+from great_expectations.datasource.fluent import SQLDatasource
 from great_expectations.datasource.fluent.interfaces import TestConnectionError
 from typing_extensions import override
 
 from great_expectations_cloud.agent.actions import ActionResult, AgentAction
-from great_expectations_cloud.agent.config import (
-    GxAgentEnvVars,
-    generate_config_validation_error_text,
-)
+from great_expectations_cloud.agent.actions.utils import get_asset_names
 from great_expectations_cloud.agent.event_handler import register_event_action
 from great_expectations_cloud.agent.exceptions import ErrorCode, raise_with_error_code
 from great_expectations_cloud.agent.models import DraftDatasourceConfigEvent
@@ -46,11 +44,10 @@ class DraftDatasourceConfigAction(AgentAction[DraftDatasourceConfigEvent]):
         datasource_type = draft_config.get("type", None)
         if datasource_type is None:
             raise TypeError(  # noqa: TRY003 # one off error
-                "The DraftDatasourceConfigAction can only be used with a "
-                "fluent-style Data Source."
+                "The DraftDatasourceConfigAction can only be used with a fluent-style Data Source."
             )
         try:
-            datasource_cls = self._context.sources.type_lookup[datasource_type]
+            datasource_cls = self._context.data_sources.type_lookup[datasource_type]
         except LookupError as exc:
             raise TypeError(  # noqa: TRY003 # one off error
                 "DraftDatasourceConfigAction received an unknown Data Source type."
@@ -58,32 +55,53 @@ class DraftDatasourceConfigAction(AgentAction[DraftDatasourceConfigEvent]):
         datasource = datasource_cls(**draft_config)
         datasource._data_context = self._context
         datasource.test_connection(test_assets=True)  # raises `TestConnectionError` on failure
-        return ActionResult(id=id, type=event.type, created_resources=[])
 
-    def get_draft_config(self, config_id: UUID) -> dict[str, Any]:
-        try:
-            config = GxAgentEnvVars()
-        except pydantic.ValidationError as validation_err:
-            raise RuntimeError(
-                generate_config_validation_error_text(validation_err)
-            ) from validation_err
-        resource_url = (
-            f"{config.gx_cloud_base_url}/organizations/"
-            f"{config.gx_cloud_organization_id}/datasources/drafts/{config_id}"
+        if isinstance(datasource, SQLDatasource):
+            asset_names = get_asset_names(datasource)
+            self._update_asset_names_list(config_id=event.config_id, asset_names=asset_names)
+
+        return ActionResult(
+            id=id,
+            type=event.type,
+            created_resources=[],
         )
-        session = create_session(access_token=config.gx_cloud_access_token)
-        response = session.get(resource_url)
+
+    def _update_asset_names_list(self, config_id: UUID, asset_names: list[str]) -> None:
+        with create_session(access_token=self._auth_key) as session:
+            url = urljoin(
+                base=self._base_url,
+                url=f"/api/v1/organizations/{self._domain_context.organization_id}/workspaces/{self._domain_context.workspace_id}/draft-table-names/{config_id}",
+            )
+            response = session.put(
+                url=url,
+                json={"data": {"table_names": asset_names}},
+            )
         if not response.ok:
             raise RuntimeError(  # noqa: TRY003 # one off error
-                "DraftDatasourceConfigAction encountered an error while " "connecting to GX Cloud"
+                f"DraftDatasourceConfigAction encountered an error while connecting to GX Cloud. "
+                f"Unable to update "
+                f"table_names for Draft Config with ID"
+                f"={config_id}.",
             )
+
+    def get_draft_config(self, config_id: UUID) -> dict[str, Any]:
+        resource_url = urljoin(
+            base=self._base_url,
+            url=f"/api/v1/organizations/{self._domain_context.organization_id}/workspaces/{self._domain_context.workspace_id}/draft-datasources/{config_id}",
+        )
+        with create_session(access_token=self._auth_key) as session:
+            response = session.get(resource_url)
+            if not response.ok:
+                raise RuntimeError(  # noqa: TRY003 # one off error
+                    "DraftDatasourceConfigAction encountered an error while connecting to GX Cloud"
+                )
         data = response.json()
         try:
-            return data["data"]["attributes"]["draft_config"]  # type: ignore[no-any-return]
+            return data["data"]["config"]  # type: ignore[no-any-return]
         except KeyError as e:
             raise RuntimeError(  # noqa: TRY003 # one off error
                 "Malformed response received from GX Cloud"
             ) from e
 
 
-register_event_action("0", DraftDatasourceConfigEvent, DraftDatasourceConfigAction)
+register_event_action("1", DraftDatasourceConfigEvent, DraftDatasourceConfigAction)