fabric-cicd 0.1.0__tar.gz

fabric_cicd-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

fabric_cicd-0.1.0/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.2
+Name: fabric-cicd
+Version: 0.1.0
+Summary: Microsoft Fabric CI/CD
+Author: Microsoft Corporation
+License:     MIT License
+        
+            Copyright (c) Microsoft Corporation.
+        
+            Permission is hereby granted, free of charge, to any person obtaining a copy
+            of this software and associated documentation files (the "Software"), to deal
+            in the Software without restriction, including without limitation the rights
+            to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+            copies of the Software, and to permit persons to whom the Software is
+            furnished to do so, subject to the following conditions:
+        
+            The above copyright notice and this permission notice shall be included in all
+            copies or substantial portions of the Software.
+        
+            THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+            IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+            FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+            AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+            LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+            OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+            SOFTWARE
+        
+Project-URL: Repository, https://github.com/microsoft/fabric-cicd.git
+Requires-Python: <3.13,>=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: azure-identity>=1.19.0
+Requires-Dist: colorlog>=6.9.0
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: requests>=2.32.3
+
+# Fabric CICD
+
+[![Language](https://img.shields.io/badge/language-Python-blue.svg)](https://www.python.org/)
+[![PyPI Version](https://badge.fury.io/py/fabric-cicd.svg)](https://badge.fury.io/py/fabric-cicd)
+[![Python Versions](https://img.shields.io/pypi/pyversions/fabric-cicd.svg)](https://pypi.org/project/fabric-cicd/)
+[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/charliermarsh/ruff)
+
+---
+
+## Project Overview
+
+fabric-cicd is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library supports code-first Continuous Integration / Continuous Deployment (CI/CD) automations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal is to assist CI/CD developers who prefer not to interact directly with the Microsoft Fabric APIs.
+
+## Documentation
+
+All documentation is hosted on our [fabric-cicd](https://microsoft.github.io/fabric-cicd/) GitHub Pages
+
+Section Overview:
+-   [Home](https://microsoft.github.io/fabric-cicd/latest/)
+-   [How To](https://microsoft.github.io/fabric-cicd/latest/how_to/)
+-   [Contribution](https://microsoft.github.io/fabric-cicd/latest/contribution/)
+-   [Changelog](https://microsoft.github.io/fabric-cicd/latest/changelog/)
+-   [About](https://microsoft.github.io/fabric-cicd/latest/help/) - Inclusive of Support & Security Policies
+
+## Installation
+
+To install fabric-cicd, run:
+
+```bash
+pip install fabric-cicd
+```
+
+## Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

fabric_cicd-0.1.0/README.md

@@ -0,0 +1,35 @@
+# Fabric CICD
+
+[![Language](https://img.shields.io/badge/language-Python-blue.svg)](https://www.python.org/)
+[![PyPI Version](https://badge.fury.io/py/fabric-cicd.svg)](https://badge.fury.io/py/fabric-cicd)
+[![Python Versions](https://img.shields.io/pypi/pyversions/fabric-cicd.svg)](https://pypi.org/project/fabric-cicd/)
+[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/charliermarsh/ruff)
+
+---
+
+## Project Overview
+
+fabric-cicd is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library supports code-first Continuous Integration / Continuous Deployment (CI/CD) automations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal is to assist CI/CD developers who prefer not to interact directly with the Microsoft Fabric APIs.
+
+## Documentation
+
+All documentation is hosted on our [fabric-cicd](https://microsoft.github.io/fabric-cicd/) GitHub Pages
+
+Section Overview:
+-   [Home](https://microsoft.github.io/fabric-cicd/latest/)
+-   [How To](https://microsoft.github.io/fabric-cicd/latest/how_to/)
+-   [Contribution](https://microsoft.github.io/fabric-cicd/latest/contribution/)
+-   [Changelog](https://microsoft.github.io/fabric-cicd/latest/changelog/)
+-   [About](https://microsoft.github.io/fabric-cicd/latest/help/) - Inclusive of Support & Security Policies
+
+## Installation
+
+To install fabric-cicd, run:
+
+```bash
+pip install fabric-cicd
+```
+
+## Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

fabric_cicd-0.1.0/VERSION

@@ -0,0 +1 @@
+0.1.0

fabric_cicd-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[project]
+name = "fabric-cicd"
+authors = [{ name = "Microsoft Corporation" }]
+description = "Microsoft Fabric CI/CD"
+readme = "README.md"
+requires-python = ">=3.9,<3.13"
+license = { file = "LICENSE" }
+dynamic = ["version"]
+urls.Repository = "https://github.com/microsoft/fabric-cicd.git"
+
+dependencies = [
+    "azure-identity>=1.19.0",
+    "colorlog>=6.9.0",
+    "pyyaml>=6.0.2",
+    "requests>=2.32.3",
+]
+
+
+[dependency-groups]
+dev = [
+    "gitpython>=3.1.44",
+    "mike>=2.1.3",
+    "mkdocs-include-markdown-plugin>=7.1.2",
+    "mkdocs-minify-plugin>=0.8.0",
+    "mkdocstrings-python>=1.13.0",
+]
+
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+dynamic.version = { file = "VERSION" }
+packages.find.where = ["src"]
+
+[tool.uv]
+package = true
+python-preference = "only-managed"

fabric_cicd-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

fabric_cicd-0.1.0/src/fabric_cicd/__init__.py

@@ -0,0 +1,54 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+"""Provides tools for managing and publishing items in a Fabric workspace."""
+
+import logging
+import sys
+
+from fabric_cicd._common._logging import configure_logger, exception_handler
+from fabric_cicd.fabric_workspace import FabricWorkspace
+from fabric_cicd.publish import publish_all_items, unpublish_all_orphan_items
+
+logger = logging.getLogger(__name__)
+
+
+def change_log_level(level: str = "DEBUG") -> None:
+    """
+    Sets the log level for all loggers within the fabric_cicd package. Currently only supports DEBUG.
+
+    Parameters
+    ----------
+    level : str
+        The logging level to set (e.g., DEBUG).
+
+    Examples
+    --------
+    Basic usage
+    >>> from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, change_log_level
+    >>> change_log_level("DEBUG")
+    >>> workspace = FabricWorkspace(
+    ...     workspace_id="your-workspace-id",
+    ...     repository_directory="/path/to/repo",
+    ...     item_type_in_scope=["Environment", "Notebook", "DataPipeline"]
+    ... )
+    >>> publish_all_items(workspace)
+    >>> unpublish_orphaned_items(workspace)
+
+    """
+    if level.upper() == "DEBUG":
+        configure_logger(logging.DEBUG)
+        logger.info("Changed log level to DEBUG")
+    else:
+        logger.warning(f"Log level '{level}' not supported.  Only DEBUG is supported at this time. No changes made.")
+
+
+configure_logger()
+sys.excepthook = exception_handler
+
+__all__ = [
+    "FabricWorkspace",
+    "change_log_level",
+    "publish_all_items",
+    "unpublish_all_orphan_items",
+]

fabric_cicd-0.1.0/src/fabric_cicd/_common/__init__.py

@@ -0,0 +1,3 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+

fabric_cicd-0.1.0/src/fabric_cicd/_common/_exceptions.py

@@ -0,0 +1,29 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+
+class BaseCustomError(Exception):
+    def __init__(self, message, logger, additional_info=None):
+        super().__init__(message)
+        self.logger = logger
+        self.additional_info = additional_info
+
+
+class ParsingError(BaseCustomError):
+    pass
+
+
+class InputError(BaseCustomError):
+    pass
+
+
+class TokenError(BaseCustomError):
+    pass
+
+
+class InvokeError(BaseCustomError):
+    pass
+
+
+class ItemDependencyError(BaseCustomError):
+    pass

fabric_cicd-0.1.0/src/fabric_cicd/_common/_fabric_endpoint.py

@@ -0,0 +1,264 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+import base64
+import datetime
+import json
+import logging
+import time
+
+import requests
+from azure.core.exceptions import (
+    ClientAuthenticationError,
+)
+
+from fabric_cicd._common._exceptions import InvokeError, TokenError
+
+logger = logging.getLogger(__name__)
+
+
+class FabricEndpoint:
+    """Handles interactions with the Fabric API, including authentication and request management."""
+
+    def __init__(self, token_credential):
+        """Initializes the FabricEndpoint instance, sets up the authentication token."""
+        self.aad_token = None
+        self.aad_token_expiration = None
+        self.token_credential = token_credential
+        self._refresh_token()
+
+    def invoke(self, method, url, body="{}", files=None):
+        """
+        Sends an HTTP request to the specified URL with the given method and body.
+
+        :param method: HTTP method to use for the request (e.g., 'GET', 'POST', 'PATCH', 'DELETE').
+        :param url: URL to send the request to.
+        :param body: The JSON body to include in the request. Defaults to an empty JSON object.
+        :param files: The file path to be included in the request. Defaults to None.
+        :return: A dictionary containing the response headers, body, and status code.
+        """
+        exit_loop = False
+        iteration_count = 0
+        long_running = False
+
+        while not exit_loop:
+            try:
+                if files is None:
+                    headers = {
+                        "Authorization": f"Bearer {self.aad_token}",
+                        "Content-Type": "application/json; charset=utf-8",
+                    }
+                    response = requests.request(method=method, url=url, headers=headers, json=body)
+
+                else:
+                    headers = {"Authorization": f"Bearer {self.aad_token}"}
+                    response = requests.request(method=method, url=url, headers=headers, files=files)
+
+                iteration_count += 1
+
+                invoke_log_message = _format_invoke_log(response, method, url, body)
+
+                # Handle long-running operations
+                # https://learn.microsoft.com/en-us/rest/api/fabric/core/long-running-operations/get-operation-result
+                if (response.status_code == 200 and long_running) or response.status_code == 202:
+                    url = response.headers.get("Location")
+                    method = "GET"
+                    body = "{}"
+                    response_json = response.json()
+
+                    if long_running:
+                        status = response_json.get("status")
+                        if status == "Succeeded":
+                            long_running = False
+                            exit_loop = True
+                        elif status == "Failed":
+                            response_error = response_json["error"]
+                            msg = f"Operation failed. Error Code: {response_error['errorCode']}. Error Message: {response_error['message']}"
+                            raise Exception(msg)
+                        elif status == "Undefined":
+                            msg = f"Operation is in an undefined state. Full Body: {response_json}"
+                            raise Exception(msg)
+                        else:
+                            retry_after = float(response.headers.get("Retry-After", 0.5))
+                            logger.info(f"Operation in progress. Checking again in {retry_after} seconds.")
+                            time.sleep(retry_after)
+                    else:
+                        time.sleep(1)
+                        long_running = True
+
+                # Handle successful responses
+                elif response.status_code in {200, 201}:
+                    exit_loop = True
+
+                # Handle API throttling
+                elif response.status_code == 429:
+                    retry_after = float(response.headers.get("Retry-After", 5)) + 5
+                    logger.info(f"API Overloaded: Retrying in {retry_after} seconds")
+                    time.sleep(retry_after)
+
+                # Handle expired authentication token
+                elif (
+                    response.status_code == 401 and response.headers.get("x-ms-public-api-error-code") == "TokenExpired"
+                ):
+                    logger.info("AAD token expired. Refreshing token.")
+                    self._refresh_token()
+
+                # Handle unauthorized access
+                elif (
+                    response.status_code == 401 and response.headers.get("x-ms-public-api-error-code") == "Unauthorized"
+                ):
+                    msg = f"The executing identity is not authorized to call {method} on '{url}'."
+                    raise Exception(msg)
+
+                # Handle item name conflicts
+                elif (
+                    response.status_code == 400
+                    and response.headers.get("x-ms-public-api-error-code") == "ItemDisplayNameAlreadyInUse"
+                ):
+                    if iteration_count <= 6:
+                        logger.info("Item name is reserved. Retrying in 60 seconds.")
+                        time.sleep(60)
+                    else:
+                        msg = f"Item name still in use after 6 attempts. Description: {response.reason}"
+                        raise Exception(msg)
+
+                # Handle scenario where library removed from environment before being removed from repo
+                elif response.status_code == 400 and "is not present in the environment." in response.json().get(
+                    "message", "No message provided"
+                ):
+                    msg = f"Deployment attempted to remove a library that is not present in the environment. Description: {response.json().get('message')}"
+                    raise Exception(msg)
+
+                # Handle no environment libraries on GET request
+                elif (
+                    response.status_code == 404
+                    and response.headers.get("x-ms-public-api-error-code") == "EnvironmentLibrariesNotFound"
+                ):
+                    logger.info("Live environment doesnt have any libraries, continuing")
+                    exit_loop = True
+
+                # Handle unsupported principal type
+                elif (
+                    response.status_code == 400
+                    and response.headers.get("x-ms-public-api-error-code") == "PrincipalTypeNotSupported"
+                ):
+                    msg = f"The executing principal type is not supported to call {method} on '{url}'"
+                    raise Exception(msg)
+
+                # Handle unsupported item types
+                elif response.status_code == 403 and response.reason == "FeatureNotAvailable":
+                    msg = f"Item type not supported. Description: {response.reason}"
+                    raise Exception(msg)
+
+                # Handle unexpected errors
+                else:
+                    err_msg = (
+                        f" Message: {response.json()['message']}"
+                        if "application/json" in (response.headers.get("Content-Type") or "")
+                        else ""
+                    )
+                    msg = f"Unhandled error occurred calling {method} on '{url}'.{err_msg}"
+                    raise Exception(msg)
+
+                # Log if reached to end of loop iteration
+                if logger.isEnabledFor(logging.DEBUG):
+                    logger.debug(invoke_log_message)
+
+            except Exception as e:
+                logger.debug(invoke_log_message)
+                raise InvokeError(e, logger, invoke_log_message) from e
+
+        return {
+            "header": dict(response.headers),
+            "body": (response.json() if "application/json" in response.headers.get("Content-Type") else {}),
+            "status_code": response.status_code,
+        }
+
+    def _refresh_token(self):
+        """Refreshes the AAD token if empty or expiration has passed"""
+        if (
+            self.aad_token is None
+            or self.aad_token_expiration is None
+            or self.aad_token_expiration < datetime.datetime.utcnow()
+        ):
+            resource_url = "https://api.fabric.microsoft.com/.default"
+
+            try:
+                self.aad_token = self.token_credential.get_token(resource_url).token
+            except ClientAuthenticationError as e:
+                msg = f"Failed to aquire AAD token. {e}"
+                raise TokenError(msg, logger) from e
+            except Exception as e:
+                msg = f"An unexpected error occurred when generating the AAD token. {e}"
+                raise TokenError(msg, logger) from e
+
+            try:
+                decoded_token = _decode_jwt(self.aad_token)
+                expiration = decoded_token.get("exp")
+                upn = decoded_token.get("upn")
+                appid = decoded_token.get("appid")
+                oid = decoded_token.get("oid")
+
+                if expiration:
+                    self.aad_token_expiration = datetime.datetime.fromtimestamp(expiration)
+                else:
+                    msg = "Token does not contain expiration claim."
+                    raise TokenError(msg, logger)
+
+                if upn:
+                    logger.info(f"Executing as User '{upn}'")
+                    self.upn_auth = True
+                else:
+                    self.upn_auth = False
+                    if appid:
+                        logger.info(f"Executing as Application Id '{appid}'")
+                    elif oid:
+                        logger.info(f"Executing as Object Id '{oid}'")
+
+            except Exception as e:
+                msg = f"An unexpected error occurred while decoding the credential token. {e}"
+                raise TokenError(msg, logger) from e
+
+
+def _decode_jwt(token):
+    """Decodes a JWT token and returns the payload as a dictionary."""
+    try:
+        # Split the token into its parts
+        parts = token.split(".")
+        if len(parts) != 3:
+            msg = "The token has an invalid JWT format"
+            raise TokenError(msg, logger)
+
+        # Decode the payload (second part of the token)
+        payload = parts[1]
+        padding = "=" * (4 - len(payload) % 4)
+        payload += padding
+        decoded_bytes = base64.urlsafe_b64decode(payload.encode("utf-8"))
+        decoded_str = decoded_bytes.decode("utf-8")
+        return json.loads(decoded_str)
+    except Exception as e:
+        msg = f"An unexpected error occurred while decoding the credential token. {e}"
+        raise TokenError(msg, logger) from e
+
+
+def _format_invoke_log(response, method, url, body):
+    message = [
+        f"\nURL: {url}",
+        f"Method: {method}",
+        (f"Request Body:\n{json.dumps(body, indent=4)}" if body else "Request Body: None"),
+    ]
+    if response is not None:
+        message.extend([
+            f"Response Status: {response.status_code}",
+            "Response Headers:",
+            json.dumps(dict(response.headers), indent=4),
+            "Response Body:",
+            (
+                json.dumps(response.json(), indent=4)
+                if response.headers.get("Content-Type") == "application/json"
+                else response.text
+            ),
+            "",
+        ])
+
+    return "\n".join(message)

fabric_cicd-0.1.0/src/fabric_cicd/_common/_logging.py

@@ -0,0 +1,88 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+import inspect
+import logging
+import sys
+from pathlib import Path
+
+import colorlog
+
+from fabric_cicd._common import _exceptions
+
+
+def configure_logger(level: int = logging.INFO) -> None:
+    """
+    Configure the logger.
+
+    :param level: The log level to set. Must be one of the standard logging levels.
+    """
+    # Configure default logging
+    logging.basicConfig(
+        level=(
+            # For non-fabric_cicd packages: INFO if DEBUG, else ERROR
+            logging.INFO if level == logging.DEBUG else logging.ERROR
+        ),
+        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+        filename="fabric_cicd.error.log",
+        filemode="w",
+    )
+
+    # Configure Console Handler
+    console_handler = logging.StreamHandler()
+    console_handler.setLevel(level)
+    console_handler.setFormatter(
+        colorlog.ColoredFormatter(
+            "%(log_color)s[%(levelname)s] %(asctime)s - %(message)s",
+            datefmt="%H:%M:%S",
+            log_colors={"DEBUG": "cyan", "INFO": "green", "WARNING": "yellow", "ERROR": "red", "CRITICAL": "bold_red"},
+        )
+    )
+
+    # Create a logger that writes to the console and log file
+    package_logger = logging.getLogger("fabric_cicd")
+    package_logger.setLevel(level)
+    package_logger.handlers = []
+    package_logger.addHandler(console_handler)
+
+    # Create a logger that only writes to the console
+    console_only_logger = logging.getLogger("console_only")
+    console_only_logger.setLevel(level)
+    console_only_logger.handlers = []
+    console_only_logger.addHandler(console_handler)
+    console_only_logger.propagate = False  # Prevent logs from being propagated to other loggers
+
+
+def exception_handler(exception_type, exception, traceback):
+    """
+    Handle exceptions that are instances of any class from the _common._exceptions module.
+
+    :param exception_type: The type of the exception.
+    :param exception: The exception instance.
+    :param traceback: The traceback object.
+    """
+    # Get all exception classes from the _common._exceptions module
+    exception_classes = [cls for _, cls in inspect.getmembers(_exceptions, inspect.isclass)]
+
+    # Check if the exception is an instance of any class from _common._exceptions
+    if any(isinstance(exception, cls) for cls in exception_classes):
+        # Log the exception using the logger associated with the exception
+        original_logger = exception.logger
+
+        # Write only the exception message to the console
+        logging.getLogger("console_only").error(
+            f"{exception!s}\n\nSee {Path('fabric_cicd.error.log').resolve()} for full details."
+        )
+
+        # Write exception and full stack trace to logs but not terminal
+        package_logger = logging.getLogger("fabric_cicd")
+
+        # Clear any existing handlers to prevent writing to console
+        additional_info = getattr(exception, "additional_info", None)
+        additional_info = "\n\nAdditional Info: \n" + additional_info if additional_info is not None else ""
+
+        package_logger.handlers = []
+        original_logger.exception(f"%s{additional_info}", exception, exc_info=(exception_type, exception, traceback))
+    else:
+        # If the exception is not from _common._exceptions, use the default exception handler
+        sys.__excepthook__(exception_type, exception, traceback)