gooddata-pipelines 1.74.1.dev1__tar.gz → 1.74.1.dev3__tar.gz

gooddata_pipelines-1.74.1.dev3/AGENTS.md

@@ -0,0 +1,96 @@
+# gooddata-pipelines
+
+A lifecycle-automation library on top of `gooddata-sdk`, covering three independent areas:
+provisioning users, groups, permissions and workspaces; backing up and restoring workspace
+metadata to local disk, S3 or Azure Blob; and extending an existing workspace's logical
+data model with custom datasets and fields. Each area is entered through a small set of
+manager or provisioner classes.
+
+## Owns
+
+- Provisioning flows for users, user groups, permissions, user data filters, and workspace
+  hierarchies, with pydantic-validated inputs
+- Backup and restore of workspace metadata across local, S3 and Azure Blob targets
+- LDM extension for child workspaces
+
+## Does NOT Own
+
+- Core platform API and service abstractions → `gooddata-sdk`
+- dbt metadata conversion → `gooddata-dbt`
+- Flight RPC runtime → `gooddata-flight-server`
+
+## Architecture
+
+| Module | Role |
+|---|---|
+| `provisioning/` | `UserProvisioner`, `UserGroupProvisioner`, `PermissionProvisioner`, `WorkspaceProvisioner`, `UserDataFilterProvisioner` |
+| `backup_and_restore/` | `BackupManager`, `RestoreManager`, plus `storage/` backends |
+| `ldm_extension/` | `LdmExtensionManager` |
+| `api/`, `utils/`, `logger/` | supporting HTTP, helpers and the log observer |
+
+**Depends on**: `gooddata-sdk`, `pydantic`, `requests`, `boto3`, `azure-storage-blob`,
+`azure-identity`.
+
+### Construction
+
+Managers and provisioners are built through classmethods, not `__init__`:
+
+```python
+provisioner = UserProvisioner.create(host, token)
+# or
+provisioner = UserProvisioner.create_from_profile(profile="default")
+```
+
+Each exposes a `.logger` (`LogObserver`) that a stdlib logger can `.subscribe()` to.
+
+Provisioning is also reachable generically via `provision(data, workflow_type, host,
+token)` driven by `WorkflowType`, intended for config- or orchestration-driven callers.
+
+## Gotchas
+
+**`full_load()` deletes.** Full load treats the list you pass as the complete desired
+state: it diffs your input against what exists upstream and deletes everything not present
+(`ids_to_delete = panther_id.difference(source_id)`). `incremental_load()` instead applies
+only the explicit create/update/delete entries you provide and leaves everything else
+alone. Confusing the two is the easiest way to write accidentally destructive code in this
+repo — when in doubt, use incremental.
+
+**`attrs` and `pyyaml` are used but not declared.** Both are imported throughout this
+package yet appear in neither its dependencies; they arrive transitively through
+`gooddata-sdk`. Fine today, silently broken if the SDK's dependency set ever changes.
+
+**Backups are zip archives with a fixed internal layout.** A backup is written per
+organization and workspace as `gooddata_layouts.zip`, containing the declarative layout
+plus `user_data_filters/`, `filter_views/` and `automations/`. Restore expects exactly that
+shape, so anything that changes the archive layout breaks restore of existing backups.
+
+**LDM extension is additive, not a general LDM editor.** `LdmExtensionManager.process()`
+adds custom datasets and fields validated through `CustomDatasetDefinition` /
+`CustomFieldDefinition`, optionally merging into the existing LDM. It can also prune
+previously-managed datasets carrying its `management_tag` when they are absent from the
+current call. It has an optional relations-integrity check that reverts the change if it
+would break more references than it fixes — do not remove that safety net casually.
+
+**Backup batching, rate limiting and retry are configurable and load-bearing.**
+`BackupRestoreConfig` carries `batch_size` and `api_calls_per_second`; a failed batch is
+retried with exponential backoff up to `BackupSettings.MAX_RETRIES` before the whole run
+aborts. These are the knobs for backup reliability and speed.
+
+**`UserDataFilterProvisioner` is not wired into the generic path.** It is provisioned
+directly rather than through `PROVISIONING_CONFIG` / `WorkflowType`, so do not assume
+feature parity across provisioners.
+
+**CSV input is backup-only.** `CSVReader.read_backup_csv` (a single-column list of
+workspace ids) serves `BackupManager` only. Provisioning takes validated pydantic model
+lists that the caller builds from whatever source they like.
+
+**Input models forbid unknown keys and want real lists.** `UserFullLoad` and friends set
+`extra="forbid"`, so a stray CSV column raises `ValidationError`, and `user_groups` is a
+`list[str]` — a delimited CSV cell has to be split before validating. The models' only
+alternate constructor is `from_sdk_obj`; there is no `from_list_of_dicts` helper, so the
+caller builds the list. The README shows the working shape.
+
+## Testing
+
+Package-local pytest suites under `tests/`. Cloud storage targets are exercised with
+`moto` rather than live buckets.

gooddata_pipelines-1.74.1.dev3/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gooddata-pipelines
-Version: 1.74.1.dev1
+Version: 1.74.1.dev3
 Summary: GoodData Cloud lifecycle automation pipelines
 Author-email: GoodData <support@gooddata.com>
 License-Expression: MIT
@@ -10,7 +10,7 @@ Requires-Dist: azure-identity<2.0.0,>=1.15.0
 Requires-Dist: azure-storage-blob<13.0.0,>=12.19.0
 Requires-Dist: boto3-stubs<2.0.0,>=1.39.3
 Requires-Dist: boto3<2.0.0,>=1.39.3
-Requires-Dist: gooddata-sdk~=1.74.1.dev1
+Requires-Dist: gooddata-sdk~=1.74.1.dev3
 Requires-Dist: pydantic<3.0.0,>=2.9.2
 Requires-Dist: requests<3.0.0,>=2.32.3
 Requires-Dist: types-pyyaml<7.0.0,>=6.0.12.20250326
@@ -56,9 +56,10 @@ from pathlib import Path
 # Import the Entity Provisioner class and corresponding model from the gooddata_pipelines library
 from gooddata_pipelines import UserFullLoad, UserProvisioner
 
-# Create the Provisioner instance - you can also create the instance from a GDC yaml profile
-provisioner = UserProvisioner(
-    host=os.environ["GDC_HOSTNAME"], token=os.environ["GDC_AUTH_TOKEN"]
+# Create the Provisioner instance - you can also create the instance from a GDC yaml
+# profile with UserProvisioner.create_from_profile(profile="default")
+provisioner = UserProvisioner.create(
+    os.environ["GDC_HOSTNAME"], os.environ["GDC_AUTH_TOKEN"]
 )
 
 # Optional: set up logging and subscribe to logs emitted by the provisioner
@@ -71,10 +72,12 @@ source_data_path: Path = Path("path/to/some.csv")
 source_data_reader = DictReader(source_data_path.read_text().splitlines())
 source_data = [row for row in source_data_reader]
 
-# Validate your input data
-full_load_data: list[UserFullLoad] = UserFullLoad.from_list_of_dicts(
-    source_data
-)
+# Validate your input data. UserFullLoad forbids unknown keys, and user_groups must be a
+# list, so split the delimited CSV column before validating.
+full_load_data: list[UserFullLoad] = [
+    UserFullLoad(**{**row, "user_groups": row["user_groups"].split("|")})
+    for row in source_data
+]
 
 # Run the provisioning
 provisioner.full_load(full_load_data)

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/README.md

@@ -37,9 +37,10 @@ from pathlib import Path
 # Import the Entity Provisioner class and corresponding model from the gooddata_pipelines library
 from gooddata_pipelines import UserFullLoad, UserProvisioner
 
-# Create the Provisioner instance - you can also create the instance from a GDC yaml profile
-provisioner = UserProvisioner(
-    host=os.environ["GDC_HOSTNAME"], token=os.environ["GDC_AUTH_TOKEN"]
+# Create the Provisioner instance - you can also create the instance from a GDC yaml
+# profile with UserProvisioner.create_from_profile(profile="default")
+provisioner = UserProvisioner.create(
+    os.environ["GDC_HOSTNAME"], os.environ["GDC_AUTH_TOKEN"]
 )
 
 # Optional: set up logging and subscribe to logs emitted by the provisioner
@@ -52,10 +53,12 @@ source_data_path: Path = Path("path/to/some.csv")
 source_data_reader = DictReader(source_data_path.read_text().splitlines())
 source_data = [row for row in source_data_reader]
 
-# Validate your input data
-full_load_data: list[UserFullLoad] = UserFullLoad.from_list_of_dicts(
-    source_data
-)
+# Validate your input data. UserFullLoad forbids unknown keys, and user_groups must be a
+# list, so split the delimited CSV column before validating.
+full_load_data: list[UserFullLoad] = [
+    UserFullLoad(**{**row, "user_groups": row["user_groups"].split("|")})
+    for row in source_data
+]
 
 # Run the provisioning
 provisioner.full_load(full_load_data)

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/pyproject.toml

@@ -1,7 +1,7 @@
 # (C) 2025 GoodData Corporation
 [project]
 name = "gooddata-pipelines"
-version = "1.74.1.dev1"
+version = "1.74.1.dev3"
 description = "GoodData Cloud lifecycle automation pipelines"
 authors = [{ name = "GoodData", email = "support@gooddata.com" }]
 license = "MIT"
@@ -11,7 +11,7 @@ dependencies = [
     "pydantic (>=2.9.2,<3.0.0)",
     "requests (>=2.32.3,<3.0.0)",
     "types-requests (>1.0.0,<3.0.0)",
-    "gooddata-sdk~=1.74.1.dev1",
+    "gooddata-sdk~=1.74.1.dev3",
     "boto3 (>=1.39.3,<2.0.0)",
     "boto3-stubs (>=1.39.3,<2.0.0)",
     "azure-storage-blob (>=12.19.0,<13.0.0)",
@@ -19,14 +19,10 @@ dependencies = [
     "types-pyyaml (>=6.0.12.20250326,<7.0.0)",
 ]
 
-[tool.ruff]
-exclude = [".venv"]
-line-length = 80
-
 [dependency-groups]
 test = [
-    "pytest~=8.3.4",
-    "pytest-cov~=6.0.0",
+    "pytest~=9.1.1",
+    "pytest-cov~=7.1.0",
     "pytest-json-report==1.5.0",
     "pytest-mock>=3.14.0,<4.0.0",
     "moto>=5.1.6,<6.0.0",

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/api/gooddata_api.py

@@ -38,9 +38,7 @@ class ApiMethods:
         """
         return f"{self.base_url}{endpoint}"
 
-    def get_all_workspace_data_filters(
-        self, workspace_id: str
-    ) -> requests.Response:
+    def get_all_workspace_data_filters(self, workspace_id: str) -> requests.Response:
         """Gets all workspace data filters for a given workspace.
 
         Args:
@@ -52,9 +50,7 @@ class ApiMethods:
         url = f"/entities/workspaces/{workspace_id}/workspaceDataFilters"
         return self._get(url)
 
-    def get_workspace_data_filter_settings(
-        self, workspace_id: str
-    ) -> requests.Response:
+    def get_workspace_data_filter_settings(self, workspace_id: str) -> requests.Response:
         """Gets all workspace data filter settings for a given workspace.
 
         Args:
@@ -66,9 +62,7 @@ class ApiMethods:
         url = f"/entities/workspaces/{workspace_id}/workspaceDataFilterSettings?include=workspaceDataFilters"
         return self._get(url)
 
-    def get_workspace_data_filter_setting(
-        self, workspace_id: str, wdf_id: str
-    ) -> requests.Response:
+    def get_workspace_data_filter_setting(self, workspace_id: str, wdf_id: str) -> requests.Response:
         """Gets a specific workspace data filter setting.
 
         Args:
@@ -119,9 +113,7 @@ class ApiMethods:
             requests.Response: The response from the server containing the
                 created workspace data filter setting.
         """
-        endpoint = (
-            f"/entities/workspaces/{workspace_id}/workspaceDataFilterSettings/"
-        )
+        endpoint = f"/entities/workspaces/{workspace_id}/workspaceDataFilterSettings/"
         return self._post(
             endpoint,
             wdf_setting,
@@ -153,9 +145,7 @@ class ApiMethods:
         endpoint = f"/layout/workspaces/{workspace_id}/userDataFilters"
         return self._get(endpoint)
 
-    def put_user_data_filters(
-        self, workspace_id: str, user_data_filters: dict[str, Any]
-    ) -> requests.Response:
+    def put_user_data_filters(self, workspace_id: str, user_data_filters: dict[str, Any]) -> requests.Response:
         """Puts the user data filters into GoodData workspace."""
         headers = {**self.headers, "Content-Type": "application/json"}
         return self._put(
@@ -166,25 +156,17 @@ class ApiMethods:
 
     def get_automations(self, workspace_id: str) -> requests.Response:
         """Gets the automations for a given workspace."""
-        endpoint = (
-            f"/entities/workspaces/{workspace_id}/automations?include=ALL"
-        )
+        endpoint = f"/entities/workspaces/{workspace_id}/automations?include=ALL"
         return self._get(endpoint)
 
-    def post_automation(
-        self, workspace_id: str, automation: dict[str, Any]
-    ) -> requests.Response:
+    def post_automation(self, workspace_id: str, automation: dict[str, Any]) -> requests.Response:
         """Posts an automation for a given workspace."""
         endpoint = f"/entities/workspaces/{workspace_id}/automations"
         return self._post(endpoint, automation)
 
-    def delete_automation(
-        self, workspace_id: str, automation_id: str
-    ) -> requests.Response:
+    def delete_automation(self, workspace_id: str, automation_id: str) -> requests.Response:
         """Deletes an automation for a given workspace."""
-        endpoint = (
-            f"/entities/workspaces/{workspace_id}/automations/{automation_id}"
-        )
+        endpoint = f"/entities/workspaces/{workspace_id}/automations/{automation_id}"
         return self._delete(endpoint)
 
     def get_all_metrics(self, workspace_id: str) -> requests.Response:
@@ -199,9 +181,7 @@ class ApiMethods:
         headers = {**self.headers, "X-GDC-VALIDATE-RELATIONS": "true"}
         return self._get(endpoint, headers=headers)
 
-    def get_all_visualization_objects(
-        self, workspace_id: str
-    ) -> requests.Response:
+    def get_all_visualization_objects(self, workspace_id: str) -> requests.Response:
         """Get all visualizations from the specified workspace.
 
         Args:
@@ -230,9 +210,7 @@ class ApiMethods:
         endpoint = "/profile"
         return self._get(endpoint)
 
-    def _get(
-        self, endpoint: str, headers: dict[str, str] | None = None
-    ) -> requests.Response:
+    def _get(self, endpoint: str, headers: dict[str, str] | None = None) -> requests.Response:
         """Sends a GET request to the server.
 
         Args:
@@ -267,9 +245,7 @@ class ApiMethods:
         request_headers = headers if headers else self.headers
         data_json = json.dumps(data)
 
-        return requests.post(
-            url, data=data_json, headers=request_headers, timeout=TIMEOUT
-        )
+        return requests.post(url, data=data_json, headers=request_headers, timeout=TIMEOUT)
 
     def _put(
         self,
@@ -291,9 +267,7 @@ class ApiMethods:
         request_headers = headers if headers else self.headers
         data_json = json.dumps(data)
 
-        return requests.put(
-            url, data=data_json, headers=request_headers, timeout=TIMEOUT
-        )
+        return requests.put(url, data=data_json, headers=request_headers, timeout=TIMEOUT)
 
     def _delete(
         self,

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/api/gooddata_api_wrapper.py

@@ -47,9 +47,7 @@ class GoodDataApi(ApiMethods):
             raise RuntimeError("Host is not set. Please provide a valid host.")
 
         if host == "":
-            raise ValueError(
-                "Host is an empty string. Please provide a valid host."
-            )
+            raise ValueError("Host is an empty string. Please provide a valid host.")
 
         # Remove trailing slash if present.
         if host[-1] == "/":

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/backup_and_restore/backup_input_processor.py

@@ -43,7 +43,9 @@ class BackupInputProcessor:
             + "filter=parent.id=={parent_id}"
             + f"&include=parent&page=0&size={self.page_size}&sort=name,asc&metaInclude=page,hierarchy"
         )
-        self.all_workspaces_endpoint = f"{self.base_workspace_endpoint}?page=0&size={self.page_size}&sort=name,asc&metaInclude=page"
+        self.all_workspaces_endpoint = (
+            f"{self.base_workspace_endpoint}?page=0&size={self.page_size}&sort=name,asc&metaInclude=page"
+        )
 
     @attrs.define
     class _ProcessDataOutput:
@@ -59,9 +61,7 @@ class BackupInputProcessor:
         if response.ok:
             return WorkspaceResponse(**response.json())
         else:
-            raise RuntimeError(
-                f"Failed to fetch data from the API. URL: {endpoint}"
-            )
+            raise RuntimeError(f"Failed to fetch data from the API. URL: {endpoint}")
 
     @staticmethod
     def process_data(data: list[Workspace]) -> _ProcessDataOutput:
@@ -74,9 +74,8 @@ class BackupInputProcessor:
             children.append(workspace.id)
 
             # if hierarchy is present and has children, append child workspace ID to sub_parents
-            if workspace.meta and workspace.meta.hierarchy:
-                if workspace.meta.hierarchy.children_count > 0:
-                    sub_parents.append(workspace.id)
+            if workspace.meta and workspace.meta.hierarchy and workspace.meta.hierarchy.children_count > 0:
+                sub_parents.append(workspace.id)
         return BackupInputProcessor._ProcessDataOutput(children, sub_parents)
 
     def log_paging_progress(self, response: WorkspaceResponse) -> None:
@@ -94,9 +93,7 @@ class BackupInputProcessor:
         if current_page and total_pages:
             self.logger.info(f"Fetched page: {current_page} of {total_pages}")
 
-    def _paginate(
-        self, url: str | None
-    ) -> list["BackupInputProcessor._ProcessDataOutput"]:
+    def _paginate(self, url: str | None) -> list["BackupInputProcessor._ProcessDataOutput"]:
         """Paginates through the API responses and returns a list of workspace IDs."""
         result: list[BackupInputProcessor._ProcessDataOutput] = []
         while url:
@@ -114,9 +111,7 @@ class BackupInputProcessor:
 
         all_children, sub_parents = [], []
 
-        results: list[BackupInputProcessor._ProcessDataOutput] = self._paginate(
-            url
-        )
+        results: list[BackupInputProcessor._ProcessDataOutput] = self._paginate(url)
 
         for result in results:
             all_children.extend(result.workspace_ids)
@@ -127,9 +122,7 @@ class BackupInputProcessor:
             all_children += self.get_hierarchy(subparent)
 
         if not all_children:
-            self.logger.warning(
-                f"No child workspaces found for parent workspace ID: {parent_id}"
-            )
+            self.logger.warning(f"No child workspaces found for parent workspace ID: {parent_id}")
 
         return all_children
 
@@ -144,9 +137,7 @@ class BackupInputProcessor:
 
         all_workspaces: list[str] = []
 
-        results: list[BackupInputProcessor._ProcessDataOutput] = self._paginate(
-            url
-        )
+        results: list[BackupInputProcessor._ProcessDataOutput] = self._paginate(url)
 
         for result in results:
             all_workspaces.extend(result.workspace_ids)

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/backup_and_restore/backup_manager.py

@@ -95,15 +95,12 @@ class BackupManager(BaseManager):
     def _get_automations_from_api(self, workspace_id: str) -> Any:
         """Returns automations for the workspace as JSON."""
         with self._api_rate_limiter:
-            response: requests.Response = self._api.get_automations(
-                workspace_id
-            )
+            response: requests.Response = self._api.get_automations(workspace_id)
             if response.ok:
                 return response.json()
             else:
                 raise RuntimeError(
-                    f"Failed to get automations for {workspace_id}. "
-                    + f"{response.status_code}: {response.text}"
+                    f"Failed to get automations for {workspace_id}. " + f"{response.status_code}: {response.text}"
                 )
 
     def _store_automations(self, export_path: Path, workspace_id: str) -> None:
@@ -120,9 +117,7 @@ class BackupManager(BaseManager):
             "automations",
         )
 
-        automations_file_path: Path = Path(
-            automations_folder_path, "automations.json"
-        )
+        automations_file_path: Path = Path(automations_folder_path, "automations.json")
 
         os.mkdir(automations_folder_path)
 
@@ -130,15 +125,11 @@ class BackupManager(BaseManager):
         if len(automations["data"]) > 0:
             self.json_utils.dump(automations_file_path, automations)
 
-    def store_declarative_filter_views(
-        self, export_path: Path, workspace_id: str
-    ) -> None:
+    def store_declarative_filter_views(self, export_path: Path, workspace_id: str) -> None:
         """Stores the filter views in the specified export path."""
         # Get the filter views YAML files from the API
         with self._api_rate_limiter:
-            self._api._sdk.catalog_workspace.store_declarative_filter_views(
-                workspace_id, export_path
-            )
+            self._api._sdk.catalog_workspace.store_declarative_filter_views(workspace_id, export_path)
 
         # Move filter views to the subfolder containing the analytics model
         self._move_folder(
@@ -176,8 +167,7 @@ class BackupManager(BaseManager):
                 user_data_filters = self.get_user_data_filters(workspace_id)
             except Exception as e:
                 self.logger.error(
-                    f"Skipping backup of {workspace_id} - check if workspace exists."
-                    + f"{e.__class__.__name__}: {e}"
+                    f"Skipping backup of {workspace_id} - check if workspace exists." + f"{e.__class__.__name__}: {e}"
                 )
                 continue
 
@@ -188,33 +178,23 @@ class BackupManager(BaseManager):
                 # directly instead of reorganizing the folder structures. That should
                 # be more transparent/readable and possibly safer for threading
                 with self._api_rate_limiter:
-                    self._api._sdk.catalog_workspace.store_declarative_workspace(
-                        workspace_id, export_path
-                    )
+                    self._api._sdk.catalog_workspace.store_declarative_workspace(workspace_id, export_path)
                 self.store_declarative_filter_views(export_path, workspace_id)
                 self._store_automations(export_path, workspace_id)
 
-                self._store_user_data_filters(
-                    user_data_filters, export_path, workspace_id
-                )
+                self._store_user_data_filters(user_data_filters, export_path, workspace_id)
 
-                workspace_duration_ms = int(
-                    (time.time() - workspace_start_time) * 1000
-                )
+                workspace_duration_ms = int((time.time() - workspace_start_time) * 1000)
                 zip_file_path = f"{self.org_id}/{workspace_id}.zip"
 
                 self.logger.info(
-                    f"Completed backup for workspace {workspace_id} "
-                    f"to {zip_file_path} in {workspace_duration_ms}ms"
+                    f"Completed backup for workspace {workspace_id} to {zip_file_path} in {workspace_duration_ms}ms"
                 )
                 exported = True
             except Exception as e:
-                workspace_duration_ms = int(
-                    (time.time() - workspace_start_time) * 1000
-                )
+                workspace_duration_ms = int((time.time() - workspace_start_time) * 1000)
                 self.logger.error(
-                    f"Skipping {workspace_id} after {workspace_duration_ms}ms. "
-                    f"{e.__class__.__name__} encountered: {e}"
+                    f"Skipping {workspace_id} after {workspace_duration_ms}ms. {e.__class__.__name__} encountered: {e}"
                 )
 
         if not exported:
@@ -245,9 +225,7 @@ class BackupManager(BaseManager):
             raise
 
     @staticmethod
-    def _split_to_batches(
-        workspaces_to_export: list[str], batch_size: int
-    ) -> list[BackupBatch]:
+    def _split_to_batches(workspaces_to_export: list[str], batch_size: int) -> list[BackupBatch]:
         """Splits the list of workspaces into batches of the specified size.
         The batch is represented as a list of workspace IDs.
         Returns a list of batches (i.e. list of lists of IDs)
@@ -274,9 +252,7 @@ class BackupManager(BaseManager):
             with tempfile.TemporaryDirectory() as tmpdir:
                 self._get_workspace_export(tmpdir, batch.list_of_ids)
 
-                self._archive_gooddata_layouts_to_zip(
-                    str(Path(tmpdir, self.org_id))
-                )
+                self._archive_gooddata_layouts_to_zip(str(Path(tmpdir, self.org_id)))
 
                 self.storage.export(tmpdir, self.org_id)
 
@@ -367,13 +343,9 @@ class BackupManager(BaseManager):
                 path_to_csv,
                 workspace_ids,
             )
-            batches = self._split_to_batches(
-                workspaces_to_export, self.config.batch_size
-            )
+            batches = self._split_to_batches(workspaces_to_export, self.config.batch_size)
 
-            self.logger.info(
-                f"Exporting {len(workspaces_to_export)} workspaces in {len(batches)} batches."
-            )
+            self.logger.info(f"Exporting {len(workspaces_to_export)} workspaces in {len(batches)} batches.")
 
             self._process_batches(batches)
 

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/backup_and_restore/base_manager.py

@@ -2,7 +2,7 @@
 
 import abc
 from pathlib import Path
-from typing import Type, TypeVar
+from typing import TypeVar
 
 from gooddata_sdk.utils import PROFILES_FILE_PATH, profile_content
 
@@ -11,6 +11,9 @@ from gooddata_pipelines.backup_and_restore.models.storage import (
     BackupRestoreConfig,
     StorageType,
 )
+from gooddata_pipelines.backup_and_restore.storage.azure_storage import (
+    AzureStorage,
+)
 from gooddata_pipelines.backup_and_restore.storage.base_storage import (
     BackupStorage,
 )
@@ -18,9 +21,6 @@ from gooddata_pipelines.backup_and_restore.storage.local_storage import (
     LocalStorage,
 )
 from gooddata_pipelines.backup_and_restore.storage.s3_storage import S3Storage
-from gooddata_pipelines.backup_and_restore.storage.azure_storage import (
-    AzureStorage,
-)
 from gooddata_pipelines.logger import LogObserver
 from gooddata_pipelines.utils.file_utils import JsonUtils, YamlUtils
 
@@ -52,13 +52,11 @@ class BaseManager(abc.ABC):
         elif conf.storage_type == StorageType.LOCAL:
             return LocalStorage(conf)
         else:
-            raise RuntimeError(
-                f'Unsupported storage type "{conf.storage_type.value}".'
-            )
+            raise RuntimeError(f'Unsupported storage type "{conf.storage_type.value}".')
 
     @classmethod
     def create(
-        cls: Type[ManagerT],
+        cls: type[ManagerT],
         config: BackupRestoreConfig,
         host: str,
         token: str,
@@ -68,7 +66,7 @@ class BaseManager(abc.ABC):
 
     @classmethod
     def create_from_profile(
-        cls: Type[ManagerT],
+        cls: type[ManagerT],
         config: BackupRestoreConfig,
         profile: str = "default",
         profiles_path: Path = PROFILES_FILE_PATH,

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/backup_and_restore/constants.py

@@ -33,8 +33,4 @@ class BackupSettings:
     API = ApiDefaults()
     MAX_RETRIES = 3
     RETRY_DELAY = 5  # seconds
-    TIMESTAMP_SDK_FOLDER = (
-        str(datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
-        + "-"
-        + sdk_version.replace(".", "_")
-    )
+    TIMESTAMP_SDK_FOLDER = str(datetime.datetime.now().strftime("%Y%m%d-%H%M%S")) + "-" + sdk_version.replace(".", "_")

{gooddata_pipelines-1.74.1.dev1 → gooddata_pipelines-1.74.1.dev3}/src/gooddata_pipelines/backup_and_restore/csv_reader.py

@@ -14,18 +14,14 @@ class CSVReader:
         """
 
         with open(file_path) as csv_file:
-            reader: Iterator[list[str]] = csv.reader(
-                csv_file, skipinitialspace=True
-            )
+            reader: Iterator[list[str]] = csv.reader(csv_file, skipinitialspace=True)
 
             try:
                 # Skip the header
                 headers = next(reader)
 
                 if len(headers) > 1:
-                    raise ValueError(
-                        "Input file contains more than one column. Please check the input and try again."
-                    )
+                    raise ValueError("Input file contains more than one column. Please check the input and try again.")
 
             except StopIteration:
                 # Raise an error if the iterator is empty