al-data-utils 0.3.1__tar.gz

al_data_utils-0.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 AL Data Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

al_data_utils-0.3.1/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.4
+Name: al-data-utils
+Version: 0.3.1
+Summary: Reusable utilities for data engineering (Airflow, Azure, Snowflake, SQL, config).
+Author-email: AL Data Labs <pypi@aldatalabs.com>
+License-Expression: MIT
+Project-URL: Homepage, https://aldatalabs.com
+Keywords: airflow,data-engineering,azure,snowflake,sql
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# al-data-utils
+
+Reusable Python utilities for data and automation workflows.
+
+This repository contains independent utility modules for sync operations, git automation, SQL work, logging helpers, and MSSQL schema snapshot generation.
+
+## Python and Setup
+
+- Python: 3.10+
+- Packaging: setuptools (src layout)
+
+Install locally in editable mode:
+
+```bash
+pip install -e .
+```
+
+## Project Structure
+
+- src/al_data_utils
+- examples
+- development_guides
+- tools
+
+Main utility groups under src/al_data_utils:
+
+- airflow_utils
+- backup_utils
+- config_utils
+- files_sync_utils
+- geo_utils
+- git_utils
+- logging_utils
+- ones_api_utils
+- sql_utils
+- tg_utils
+
+## Examples
+
+Runnable examples are available in the `examples` folder for the main utility groups.
+Use them as reference implementations for local scripts and automation jobs.
+
+## MSSQL Schema Snapshot
+
+The snapshot tool exports SQL objects from MSSQL into a filesystem layout and syncs staging output to destination folders.
+
+Current behavior highlights:
+
+- Supports single DB and multi-DB runs in one execution.
+- Always writes outputs into per-DB folders.
+- Uses staging first, then syncs to final output.
+- Supports cleanup of staging folders.
+- Supports sync log retention cleanup.
+- Supports include and exclude filters per object type.
+- Supports DataGrip template override files.
+
+Output shape:
+
+```text
+<MSSQL_OUT_DIR>/<DatabaseName>/schemas/<schema>/
+	tables/
+	views/
+	routines/
+	sequences/
+	synonyms/
+```
+
+### Snapshot Environment Variables
+
+Core connection and output:
+
+- MSSQL_OUT_DIR
+- MSSQL_SERVER
+- MSSQL_DATABASE
+- MSSQL_DATABASES
+- MSSQL_AUTH_MODE
+- MSSQL_USER
+- MSSQL_PASSWORD
+- MSSQL_DRIVER
+
+Snapshot behavior:
+
+- MSSQL_ENABLE_TABLE_DEFINITIONS
+- MSSQL_ENABLE_INDEXES
+- MSSQL_ENABLE_FOREIGN_KEYS
+- MSSQL_ENABLE_VIEWS
+- MSSQL_ENABLE_STORED_PROCEDURES
+- MSSQL_ENABLE_FUNCTIONS
+- MSSQL_ENABLE_SEQUENCES
+- MSSQL_ENABLE_SYNONYMS
+- MSSQL_CLEANUP_STAGING
+- MSSQL_SYNC_LOG_RETENTION_DAYS
+
+Filters:
+
+- MSSQL_INCLUDE_SCHEMAS
+- MSSQL_EXCLUDE_SCHEMAS
+- MSSQL_INCLUDE_TABLES
+- MSSQL_EXCLUDE_TABLES
+- MSSQL_INCLUDE_VIEWS
+- MSSQL_EXCLUDE_VIEWS
+- MSSQL_INCLUDE_STORED_PROCEDURES
+- MSSQL_EXCLUDE_STORED_PROCEDURES
+- MSSQL_INCLUDE_FUNCTIONS
+- MSSQL_EXCLUDE_FUNCTIONS
+- MSSQL_INCLUDE_SEQUENCES
+- MSSQL_EXCLUDE_SEQUENCES
+- MSSQL_INCLUDE_SYNONYMS
+- MSSQL_EXCLUDE_SYNONYMS
+
+Notes:
+
+- If MSSQL_DATABASES is set, it is treated as a comma-separated list.
+- If MSSQL_DATABASES is empty, MSSQL_DATABASE is used.
+- If MSSQL_DATABASE contains commas, it is also treated as a list for backward compatibility.
+
+## File Sync Utilities
+
+`files_sync_utils` provides:
+
+- Inventory building
+- Plan generation
+- Copy and update execution
+- Extra destination cleanup rules
+- Activity CSV logging
+- Log retention cleanup
+
+## Git Utilities
+
+`git_utils` wraps common git automation operations with consistent behavior and error handling.
+
+## Development Guides
+
+- development_guides/dev_guide.txt
+- development_guides/auth_key_guide.txt
+
+These guides contain release flow notes and auth-related instructions.
+
+## Changelog
+
+See CHANGELOG.md for release notes.

al_data_utils-0.3.1/README.md

@@ -0,0 +1,141 @@
+# al-data-utils
+
+Reusable Python utilities for data and automation workflows.
+
+This repository contains independent utility modules for sync operations, git automation, SQL work, logging helpers, and MSSQL schema snapshot generation.
+
+## Python and Setup
+
+- Python: 3.10+
+- Packaging: setuptools (src layout)
+
+Install locally in editable mode:
+
+```bash
+pip install -e .
+```
+
+## Project Structure
+
+- src/al_data_utils
+- examples
+- development_guides
+- tools
+
+Main utility groups under src/al_data_utils:
+
+- airflow_utils
+- backup_utils
+- config_utils
+- files_sync_utils
+- geo_utils
+- git_utils
+- logging_utils
+- ones_api_utils
+- sql_utils
+- tg_utils
+
+## Examples
+
+Runnable examples are available in the `examples` folder for the main utility groups.
+Use them as reference implementations for local scripts and automation jobs.
+
+## MSSQL Schema Snapshot
+
+The snapshot tool exports SQL objects from MSSQL into a filesystem layout and syncs staging output to destination folders.
+
+Current behavior highlights:
+
+- Supports single DB and multi-DB runs in one execution.
+- Always writes outputs into per-DB folders.
+- Uses staging first, then syncs to final output.
+- Supports cleanup of staging folders.
+- Supports sync log retention cleanup.
+- Supports include and exclude filters per object type.
+- Supports DataGrip template override files.
+
+Output shape:
+
+```text
+<MSSQL_OUT_DIR>/<DatabaseName>/schemas/<schema>/
+	tables/
+	views/
+	routines/
+	sequences/
+	synonyms/
+```
+
+### Snapshot Environment Variables
+
+Core connection and output:
+
+- MSSQL_OUT_DIR
+- MSSQL_SERVER
+- MSSQL_DATABASE
+- MSSQL_DATABASES
+- MSSQL_AUTH_MODE
+- MSSQL_USER
+- MSSQL_PASSWORD
+- MSSQL_DRIVER
+
+Snapshot behavior:
+
+- MSSQL_ENABLE_TABLE_DEFINITIONS
+- MSSQL_ENABLE_INDEXES
+- MSSQL_ENABLE_FOREIGN_KEYS
+- MSSQL_ENABLE_VIEWS
+- MSSQL_ENABLE_STORED_PROCEDURES
+- MSSQL_ENABLE_FUNCTIONS
+- MSSQL_ENABLE_SEQUENCES
+- MSSQL_ENABLE_SYNONYMS
+- MSSQL_CLEANUP_STAGING
+- MSSQL_SYNC_LOG_RETENTION_DAYS
+
+Filters:
+
+- MSSQL_INCLUDE_SCHEMAS
+- MSSQL_EXCLUDE_SCHEMAS
+- MSSQL_INCLUDE_TABLES
+- MSSQL_EXCLUDE_TABLES
+- MSSQL_INCLUDE_VIEWS
+- MSSQL_EXCLUDE_VIEWS
+- MSSQL_INCLUDE_STORED_PROCEDURES
+- MSSQL_EXCLUDE_STORED_PROCEDURES
+- MSSQL_INCLUDE_FUNCTIONS
+- MSSQL_EXCLUDE_FUNCTIONS
+- MSSQL_INCLUDE_SEQUENCES
+- MSSQL_EXCLUDE_SEQUENCES
+- MSSQL_INCLUDE_SYNONYMS
+- MSSQL_EXCLUDE_SYNONYMS
+
+Notes:
+
+- If MSSQL_DATABASES is set, it is treated as a comma-separated list.
+- If MSSQL_DATABASES is empty, MSSQL_DATABASE is used.
+- If MSSQL_DATABASE contains commas, it is also treated as a list for backward compatibility.
+
+## File Sync Utilities
+
+`files_sync_utils` provides:
+
+- Inventory building
+- Plan generation
+- Copy and update execution
+- Extra destination cleanup rules
+- Activity CSV logging
+- Log retention cleanup
+
+## Git Utilities
+
+`git_utils` wraps common git automation operations with consistent behavior and error handling.
+
+## Development Guides
+
+- development_guides/dev_guide.txt
+- development_guides/auth_key_guide.txt
+
+These guides contain release flow notes and auth-related instructions.
+
+## Changelog
+
+See CHANGELOG.md for release notes.

al_data_utils-0.3.1/pyproject.toml

@@ -0,0 +1,58 @@
+[project]
+name = "al-data-utils"
+version = "0.3.1"
+description = "Reusable utilities for data engineering (Airflow, Azure, Snowflake, SQL, config)."
+authors = [{ name = "AL Data Labs", email = "pypi@aldatalabs.com" }]
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = ["airflow", "data-engineering", "azure", "snowflake", "sql"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Topic :: Software Development :: Libraries",
+]
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [] # keep empty for now, using airflow environment's dependencies
+
+[project.urls]
+Homepage = "https://aldatalabs.com"
+
+
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+# Ruff configuration
+[tool.ruff]
+# Exclude a variety of commonly ignored directories.
+exclude = [
+    ".git",
+    ".venv",
+]
+
+# Assume Python 3.10
+target-version = "py310"
+
+# Same as Black.
+line-length = 88
+indent-width = 4
+
+
+[tool.ruff.format]
+quote-style = "single"
+
+# Like Black, indent with spaces, rather than tabs.
+indent-style = "space"
+
+# Like Black, respect magic trailing commas.
+skip-magic-trailing-comma = true
+
+# Like Black, automatically detect the appropriate line ending.
+line-ending = "auto"

al_data_utils-0.3.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

al_data_utils-0.3.1/src/al_data_utils/__init__.py

@@ -0,0 +1,10 @@
+__all__ = [
+    'airflow_utils',
+    'config_utils',
+    'geo_utils',
+    'logging_utils',
+    'azure_as_utils',
+    'sql_utils',
+    'tg_utils',
+    'ones_api_utils',
+]

al_data_utils-0.3.1/src/al_data_utils/airflow_utils/__init__.py

@@ -0,0 +1,13 @@
+from .airflow_utils import (
+    get_variable_value,
+    get_variable_value_dict,
+    get_variable_value_int,
+    get_variable_value_str,
+)
+
+__all__ = [
+    "get_variable_value",
+    "get_variable_value_dict",
+    "get_variable_value_int",
+    "get_variable_value_str",
+]

al_data_utils-0.3.1/src/al_data_utils/airflow_utils/airflow_utils.py

@@ -0,0 +1,83 @@
+from __future__ import annotations
+
+import functools
+import json
+from pathlib import Path
+from sys import platform
+from typing import Any
+
+import yaml
+
+
+def _get_dags_folder() -> str:
+    try:
+        from airflow import settings  # type: ignore
+        return settings.DAGS_FOLDER
+    except Exception:
+        return "./dags"
+
+
+def _airflow_variable_get(name: str) -> str:
+    from airflow.sdk import Variable  # type: ignore
+    return Variable.get(name)
+
+
+@functools.cache
+def get_variable_value(
+    airflow_variable_name: str,
+    pack_to_json: bool = False,
+    source: str = "airflow",
+    source_section: str = "variables",
+) -> Any:
+    """Get variable value different ways for windows and linux."""
+
+    if platform == "win32":
+        base = Path(".local/variables")
+        data = json.loads((base / "variables.json").read_text(encoding="utf-8"))
+        environment_code = data["ENVIRONMENT"]
+
+        env_data = json.loads(
+            (base / f"{environment_code.lower()}_variables.json").read_text(encoding="utf-8")
+        )
+        variable_value = env_data[airflow_variable_name]
+
+    else:
+        if source == "config":
+            dags_folder = _get_dags_folder()
+            p = Path(dags_folder) / ".configs" / "_variables.yaml"
+            variables = yaml.safe_load(p.read_text(encoding="utf-8")) or {}
+            variable_value = variables[source_section][airflow_variable_name]
+        else:
+            variable_value = _airflow_variable_get(airflow_variable_name)
+
+    if pack_to_json and not isinstance(variable_value, dict):
+        try:
+            variable_value = json.loads(variable_value)
+        except Exception:
+            pass
+
+    return variable_value
+
+
+def get_variable_value_str(
+    airflow_variable_name: str,
+    source: str = "airflow",
+    source_section: str = "variables",
+) -> str:
+    return str(get_variable_value(airflow_variable_name, False, source, source_section))
+
+
+def get_variable_value_int(
+    airflow_variable_name: str,
+    source: str = "airflow",
+    source_section: str = "variables",
+) -> int:
+    return int(get_variable_value(airflow_variable_name, False, source, source_section))
+
+
+def get_variable_value_dict(
+    airflow_variable_name: str,
+    source: str = "airflow",
+    source_section: str = "variables",
+) -> dict:
+    return dict(get_variable_value(airflow_variable_name, True, source, source_section))

al_data_utils-0.3.1/src/al_data_utils/azure_as_utils/__init__.py

@@ -0,0 +1,31 @@
+from .auth import ServicePrincipalTokenProvider, StaticTokenProvider, TokenProvider
+from .azure_as_utils import SemanticModels
+from .client import AasRestClient
+from .errors import (
+    AasAuthError,
+    AasError,
+    AasHttpError,
+    AasRefreshError,
+    AasRefreshTimeout,
+    AasTransportError,
+)
+from .models import RefreshObject, RefreshRequest, RefreshStatus
+from .poller import RefreshPoller
+
+__all__ = [
+    'AasAuthError',
+    'AasError',
+    'AasHttpError',
+    'AasRefreshError',
+    'AasRefreshTimeout',
+    'AasRestClient',
+    'AasTransportError',
+    'RefreshObject',
+    'RefreshPoller',
+    'RefreshRequest',
+    'RefreshStatus',
+    'SemanticModels',
+    'ServicePrincipalTokenProvider',
+    'StaticTokenProvider',
+    'TokenProvider',
+]

al_data_utils-0.3.1/src/al_data_utils/azure_as_utils/auth.py

@@ -0,0 +1,82 @@
+"""Access tokens for the AAS REST API via an Azure AD service principal."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any, Protocol
+
+import msal
+
+from .errors import AasAuthError
+
+#: Resource identifier of Azure Analysis Services; the wildcard is literal.
+AAS_SCOPE = 'https://*.asazure.windows.net/.default'
+DEFAULT_AUTHORITY_HOST = 'https://login.microsoftonline.com'
+
+
+class TokenProvider(Protocol):
+    """Anything that can hand out a bearer token for the AAS resource."""
+
+    def get_token(self) -> str: ...
+
+
+class StaticTokenProvider:
+    """Returns a fixed token. Useful for tests and pre-issued tokens."""
+
+    def __init__(self, token: str) -> None:
+        if not token:
+            raise ValueError('token must be a non-empty string')
+        self._token = token
+
+    def get_token(self) -> str:
+        return self._token
+
+
+class ServicePrincipalTokenProvider:
+    """Client-credentials flow through MSAL.
+
+    MSAL keeps the issued token in its in-memory cache and only goes back to
+    Azure AD when it is about to expire, so calling :meth:`get_token` before
+    every request is cheap.
+    """
+
+    def __init__(
+        self,
+        tenant_id: str,
+        client_id: str,
+        client_secret: str,
+        *,
+        authority_host: str = DEFAULT_AUTHORITY_HOST,
+        scope: str = AAS_SCOPE,
+    ) -> None:
+        for name, value in (
+            ('tenant_id', tenant_id),
+            ('client_id', client_id),
+            ('client_secret', client_secret),
+        ):
+            if not isinstance(value, str) or not value.strip():
+                raise ValueError(f'{name} must be a non-empty string')
+        self._scopes = [scope]
+        self._authority = f'{authority_host.rstrip("/")}/{tenant_id}'
+        self._app = msal.ConfidentialClientApplication(
+            client_id, authority=self._authority, client_credential=client_secret
+        )
+
+    def get_token(self) -> str:
+        result: Mapping[str, Any] | None = self._app.acquire_token_for_client(
+            scopes=self._scopes
+        )
+        token = result.get('access_token') if isinstance(result, Mapping) else None
+        if not token:
+            raise AasAuthError(_describe_failure(result))
+        return token
+
+
+def _describe_failure(result: Mapping[str, Any] | None) -> str:
+    if not isinstance(result, Mapping):
+        return f'token request returned no result ({result!r})'
+    error = result.get('error') or 'unknown_error'
+    description = result.get('error_description') or 'no description'
+    correlation = result.get('correlation_id')
+    suffix = f' (correlation_id={correlation})' if correlation else ''
+    return f'token request failed: {error}: {description}{suffix}'