msdev-kit 0.2.2__tar.gz → 0.2.4__tar.gz

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: msdev-kit
-Version: 0.2.2
+Version: 0.2.4
 Summary: Microsoft developer toolkit: Fabric, MS Graph, and SharePoint
 License: MIT
 License-File: LICENSE
@@ -65,15 +65,11 @@ Or install from GitHub:
 pip install git+https://github.com/Bernardo-Rufino/msdev-kit.git
 ```
 
-For local development:
+For local development, use the repository instructions in [Development](docs/development.md).
 
-```shell
-git clone https://github.com/Bernardo-Rufino/msdev-kit.git
-cd msdev-kit
-pip install -e .
-```
-
-**Requirements:** Python >= 3.10, an Azure app registration with a client ID and client secret.
+**Requirements:** Python >= 3.10. Service principal authentication requires an
+Azure app registration, tenant ID, client ID, and client secret. Interactive
+user authentication is also supported for APIs that permit it.
 
 ---
 
@@ -89,7 +85,7 @@ from msdev_kit.sharepoint import SharePointClient
 auth = Auth(tenant_id="...", client_id="...", client_secret="...")
 
 # fabric: list workspaces
-ws = Workspace(auth.get_token('fabric'))
+ws = Workspace(auth.get_token('pbi'))
 workspaces = ws.list_workspaces_for_user()
 
 # graph: look up a user
@@ -105,7 +101,10 @@ sp.download_file('/Reports/monthly.xlsx', local_dir='./downloads')
 
 ## Authentication
 
-All classes use a shared `Auth` object. You can use different service principals for different services — instantiate one `Auth` per SPN:
+`Auth` owns token acquisition. Fabric and Power BI classes receive a token string,
+while Graph and SharePoint clients receive the `Auth` instance. You can use
+different service principals for different services by instantiating one `Auth`
+per service principal:
 
 ```python
 from msdev_kit import Auth
@@ -142,14 +141,16 @@ token = auth.get_token_for_user('fabric')
 
 ### Credentials
 
-Set up credentials via environment variables or a `.env` file:
+For the bundled examples, copy the template and set the values:
 
 ```shell
-TENANT_ID='<YOUR_TENANT_ID>'
-CLIENT_ID='<YOUR_CLIENT_ID>'
-CLIENT_SECRET='<YOUR_CLIENT_SECRET>'
+cp .env.example .env
 ```
 
+`msdev-kit` does not load `.env` automatically. The scripts in `examples/` load
+`.env` from the repository root. Applications may pass values directly to
+`Auth`, or load environment variables with their own configuration mechanism.
+
 ---
 
 ## Fabric & Power BI
@@ -239,7 +240,7 @@ pages = rpt.list_report_pages(workspace_id, report_id)
 Manage Power BI and Fabric dataflows, including Gen1, Gen2, and Gen2 CI/CD.
 
 ```python
-df = Dataflow(auth.get_token('fabric'))
+df = Dataflow(auth.get_token('pbi'))
 
 # upgrade Gen1 to Gen2 CI/CD
 result = df.upgrade_to_gen2_cicd(
@@ -262,10 +263,33 @@ result = df.upgrade_to_gen2_cicd(
 | `create_dataflow_gen2_from_definition(workspace_id, display_name, definition)` | Create a Dataflow Gen2 CI/CD from a definition. |
 | `update_dataflow_gen2_from_definition(workspace_id, dataflow_id, display_name, definition)` | Update an existing Dataflow Gen2 CI/CD definition. |
 | `get_data_destinations(workspace_id, dataflow_id)` | Get data destination details for each table in a dataflow. |
+| `get_workspace_data_destinations(workspace_id, max_workers=4)` | Inspect every dataflow, return its destination details, and save destination-only rows as a workbook under `data/dataflows`. The inventory is paced at 200 requests per minute and retries 429 responses. |
 | `change_data_destination(workspace_id, dataflow_id, destination_type, ...)` | Change data destination (Lakehouse/Warehouse). Modes: `preview`, `replace`, `create`. |
 | `create_dataflow_with_new_destination(workspace_id, dataflow_id, ...)` | Create a new Gen2 CI/CD dataflow with a different data destination. |
 | `upgrade_to_gen2_cicd(...)` | Upgrade a Gen1 or Gen2 (standard) dataflow to Gen2 CI/CD. |
 
+#### Inventory workspace data destinations
+
+```python
+from msdev_kit import Auth
+from msdev_kit.fabric import Dataflow
+
+workspace_id = "<workspace-id>"
+auth = Auth(tenant_id="<tenant-id>", client_id="<client-id>", client_secret="<client-secret>")
+dataflow = Dataflow(auth.get_token("pbi"))
+
+result = dataflow.get_workspace_data_destinations(workspace_id, max_workers=4)
+```
+
+The method inspects every dataflow so `result["content"]` also records empty
+inspections and failures. Its workbook contains only table rows with a real data
+destination. Table name is exported as `table_name`, not `table_table`. A Fabric
+source without an API generation value is normalized to `2.1`. The progress line
+is updated in place. If a 429 response occurs, the method prints the completed
+count, waits, then resumes the progress line. See
+[`examples/dataflow_destinations.py`](examples/dataflow_destinations.py) for a
+DataFrame normalizer and a runnable placeholder.
+
 ### Pipeline
 
 Manage Fabric Data Pipelines.
@@ -410,7 +434,7 @@ Hostname and site path inputs are normalized automatically:
 
 ## Limitations
 
-- The Power BI REST API has a **200 requests per hour** rate limit.
+- Power BI and Fabric endpoints apply operation-specific throttling. Do not assume one global quota. `get_workspace_data_destinations` intentionally paces definition lookups at 200 requests per minute and backs off after HTTP 429.
 - Not all users can be updated via the API. See Microsoft docs: [Dataset permissions](https://learn.microsoft.com/en-us/power-bi/developer/embedded/datasets-permissions#get-and-update-dataset-permissions-with-apis).
 - **Dataset query limits** (executeQueries API):
   - Max **100,000 rows** or **1,000,000 values** (rows x columns) per query, whichever is hit first.
@@ -424,28 +448,24 @@ Hostname and site path inputs are normalized automatically:
 
 ## Examples
 
-End-to-end scripts for the most common scenarios live in [`examples/`](./examples).
-They share a small `_setup.py` that builds Auth and the service clients from
-environment variables (or `./utils/.env`). Run any of them from the repo root:
+Runnable, placeholder-based scripts live in [`examples/`](./examples). Start
+with the [examples guide](examples/README.md), then run a read-only example from
+the repository root:
 
 ```bash
+cp .env.example .env
 python -m examples.workspaces
-python -m examples.dataflows
+python -m examples.dataflow_destinations
 ```
 
+Write examples never perform a mutation when run directly. Edit the placeholder
+values and call their explicit helper only after reviewing the target IDs.
+
 ---
 
 ## Contributing
 
-External contributions are welcome via pull requests. Please:
-
-1. Open an issue first for non-trivial changes so the design can be discussed.
-2. Use `feature/<name>` / `fix/<name>` branches. Direct pushes to `main` are
-   blocked by repository rulesets.
-3. Keep PRs focused. Add or update tests under `tests/` for any behavior change.
-4. Run `pytest` locally before opening a PR; the `Collaboration` workflow runs
-   the same suite on every PR and must pass before merging.
-
-PyPI releases are published automatically from `main` by the `Publish to PyPI`
-workflow, gated by a manual approval on the `pypi` environment.
+Read [Contributing](CONTRIBUTING.md) before opening a pull request. It links the
+local setup, validation commands, issue and pull request process, branch
+conventions, and release boundary.
 

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/README.md

@@ -32,15 +32,11 @@ Or install from GitHub:
 pip install git+https://github.com/Bernardo-Rufino/msdev-kit.git
 ```
 
-For local development:
+For local development, use the repository instructions in [Development](docs/development.md).
 
-```shell
-git clone https://github.com/Bernardo-Rufino/msdev-kit.git
-cd msdev-kit
-pip install -e .
-```
-
-**Requirements:** Python >= 3.10, an Azure app registration with a client ID and client secret.
+**Requirements:** Python >= 3.10. Service principal authentication requires an
+Azure app registration, tenant ID, client ID, and client secret. Interactive
+user authentication is also supported for APIs that permit it.
 
 ---
 
@@ -56,7 +52,7 @@ from msdev_kit.sharepoint import SharePointClient
 auth = Auth(tenant_id="...", client_id="...", client_secret="...")
 
 # fabric: list workspaces
-ws = Workspace(auth.get_token('fabric'))
+ws = Workspace(auth.get_token('pbi'))
 workspaces = ws.list_workspaces_for_user()
 
 # graph: look up a user
@@ -72,7 +68,10 @@ sp.download_file('/Reports/monthly.xlsx', local_dir='./downloads')
 
 ## Authentication
 
-All classes use a shared `Auth` object. You can use different service principals for different services — instantiate one `Auth` per SPN:
+`Auth` owns token acquisition. Fabric and Power BI classes receive a token string,
+while Graph and SharePoint clients receive the `Auth` instance. You can use
+different service principals for different services by instantiating one `Auth`
+per service principal:
 
 ```python
 from msdev_kit import Auth
@@ -109,14 +108,16 @@ token = auth.get_token_for_user('fabric')
 
 ### Credentials
 
-Set up credentials via environment variables or a `.env` file:
+For the bundled examples, copy the template and set the values:
 
 ```shell
-TENANT_ID='<YOUR_TENANT_ID>'
-CLIENT_ID='<YOUR_CLIENT_ID>'
-CLIENT_SECRET='<YOUR_CLIENT_SECRET>'
+cp .env.example .env
 ```
 
+`msdev-kit` does not load `.env` automatically. The scripts in `examples/` load
+`.env` from the repository root. Applications may pass values directly to
+`Auth`, or load environment variables with their own configuration mechanism.
+
 ---
 
 ## Fabric & Power BI
@@ -206,7 +207,7 @@ pages = rpt.list_report_pages(workspace_id, report_id)
 Manage Power BI and Fabric dataflows, including Gen1, Gen2, and Gen2 CI/CD.
 
 ```python
-df = Dataflow(auth.get_token('fabric'))
+df = Dataflow(auth.get_token('pbi'))
 
 # upgrade Gen1 to Gen2 CI/CD
 result = df.upgrade_to_gen2_cicd(
@@ -229,10 +230,33 @@ result = df.upgrade_to_gen2_cicd(
 | `create_dataflow_gen2_from_definition(workspace_id, display_name, definition)` | Create a Dataflow Gen2 CI/CD from a definition. |
 | `update_dataflow_gen2_from_definition(workspace_id, dataflow_id, display_name, definition)` | Update an existing Dataflow Gen2 CI/CD definition. |
 | `get_data_destinations(workspace_id, dataflow_id)` | Get data destination details for each table in a dataflow. |
+| `get_workspace_data_destinations(workspace_id, max_workers=4)` | Inspect every dataflow, return its destination details, and save destination-only rows as a workbook under `data/dataflows`. The inventory is paced at 200 requests per minute and retries 429 responses. |
 | `change_data_destination(workspace_id, dataflow_id, destination_type, ...)` | Change data destination (Lakehouse/Warehouse). Modes: `preview`, `replace`, `create`. |
 | `create_dataflow_with_new_destination(workspace_id, dataflow_id, ...)` | Create a new Gen2 CI/CD dataflow with a different data destination. |
 | `upgrade_to_gen2_cicd(...)` | Upgrade a Gen1 or Gen2 (standard) dataflow to Gen2 CI/CD. |
 
+#### Inventory workspace data destinations
+
+```python
+from msdev_kit import Auth
+from msdev_kit.fabric import Dataflow
+
+workspace_id = "<workspace-id>"
+auth = Auth(tenant_id="<tenant-id>", client_id="<client-id>", client_secret="<client-secret>")
+dataflow = Dataflow(auth.get_token("pbi"))
+
+result = dataflow.get_workspace_data_destinations(workspace_id, max_workers=4)
+```
+
+The method inspects every dataflow so `result["content"]` also records empty
+inspections and failures. Its workbook contains only table rows with a real data
+destination. Table name is exported as `table_name`, not `table_table`. A Fabric
+source without an API generation value is normalized to `2.1`. The progress line
+is updated in place. If a 429 response occurs, the method prints the completed
+count, waits, then resumes the progress line. See
+[`examples/dataflow_destinations.py`](examples/dataflow_destinations.py) for a
+DataFrame normalizer and a runnable placeholder.
+
 ### Pipeline
 
 Manage Fabric Data Pipelines.
@@ -377,7 +401,7 @@ Hostname and site path inputs are normalized automatically:
 
 ## Limitations
 
-- The Power BI REST API has a **200 requests per hour** rate limit.
+- Power BI and Fabric endpoints apply operation-specific throttling. Do not assume one global quota. `get_workspace_data_destinations` intentionally paces definition lookups at 200 requests per minute and backs off after HTTP 429.
 - Not all users can be updated via the API. See Microsoft docs: [Dataset permissions](https://learn.microsoft.com/en-us/power-bi/developer/embedded/datasets-permissions#get-and-update-dataset-permissions-with-apis).
 - **Dataset query limits** (executeQueries API):
   - Max **100,000 rows** or **1,000,000 values** (rows x columns) per query, whichever is hit first.
@@ -391,27 +415,23 @@ Hostname and site path inputs are normalized automatically:
 
 ## Examples
 
-End-to-end scripts for the most common scenarios live in [`examples/`](./examples).
-They share a small `_setup.py` that builds Auth and the service clients from
-environment variables (or `./utils/.env`). Run any of them from the repo root:
+Runnable, placeholder-based scripts live in [`examples/`](./examples). Start
+with the [examples guide](examples/README.md), then run a read-only example from
+the repository root:
 
 ```bash
+cp .env.example .env
 python -m examples.workspaces
-python -m examples.dataflows
+python -m examples.dataflow_destinations
 ```
 
+Write examples never perform a mutation when run directly. Edit the placeholder
+values and call their explicit helper only after reviewing the target IDs.
+
 ---
 
 ## Contributing
 
-External contributions are welcome via pull requests. Please:
-
-1. Open an issue first for non-trivial changes so the design can be discussed.
-2. Use `feature/<name>` / `fix/<name>` branches. Direct pushes to `main` are
-   blocked by repository rulesets.
-3. Keep PRs focused. Add or update tests under `tests/` for any behavior change.
-4. Run `pytest` locally before opening a PR; the `Collaboration` workflow runs
-   the same suite on every PR and must pass before merging.
-
-PyPI releases are published automatically from `main` by the `Publish to PyPI`
-workflow, gated by a manual approval on the `pypi` environment.
+Read [Contributing](CONTRIBUTING.md) before opening a pull request. It links the
+local setup, validation commands, issue and pull request process, branch
+conventions, and release boundary.

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/msdev_kit/__init__.py

@@ -1,4 +1,4 @@
 from .auth import Auth
 
-__version__ = "0.2.2"
+__version__ = "0.2.4"
 __all__ = ["Auth"]

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/msdev_kit/fabric/dataflow.py

@@ -7,8 +7,11 @@ import time
 import base64
 import requests
 import pandas as pd
-from typing import Dict, List, Optional
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from threading import Lock
+from typing import Callable, Dict, List, Optional
 from pydantic import BaseModel, ConfigDict
+from msdev_kit.http import RequestPacer, request_with_retry
 from .utilities import create_directory
 from .workspace import Workspace
 
@@ -27,14 +30,17 @@ class ComputeEngineSettingsModel(BaseModel):
 class Dataflow:
 
     def __init__(self, token: str):
-        """
-        Initialize variables.
+        """Initialize the Dataflow client.
+
+        Args:
+            token: Power BI or Fabric bearer token.
         """
         self.main_url = 'https://api.powerbi.com/v1.0/myorg'
         self.fabric_api_base_url = 'https://api.fabric.microsoft.com'
         self.token = token
         self.headers = {'Authorization': f'Bearer {self.token}'}
         self.workspace = Workspace(self.token)
+        self._request_pacer = RequestPacer(requests_per_minute=200)
 
         # Directories
         self.dataflows_dir = './data/dataflows'
@@ -44,24 +50,38 @@ class Dataflow:
             create_directory(dir)
 
 
-    def _request_with_retry(self, method: str, url: str, max_retries: int = 3, **kwargs) -> requests.Response:
+    def _request_with_retry(
+        self,
+        method: str,
+        url: str,
+        max_retries: int = 3,
+        log_retries: bool = True,
+        on_rate_limit: Optional[Callable[[float], None]] = None,
+        **kwargs,
+    ) -> requests.Response:
         """
-        Makes an HTTP request with automatic retry on 429 (Too Many Requests).
-        Respects the Retry-After header when present.
+        Compatibility wrapper around the shared HTTP retry helper.
         """
-        for attempt in range(max_retries + 1):
-            response = requests.request(method, url, **kwargs)
-            if response.status_code != 429:
-                return response
-
-            retry_after = int(response.headers.get('Retry-After', 5))
-            print(f"  Rate limited (429). Retrying in {retry_after}s... (attempt {attempt + 1}/{max_retries})")
-            time.sleep(retry_after)
-
-        return response
+        return request_with_retry(
+            method,
+            url,
+            max_retries=max_retries,
+            pacer=getattr(self, '_request_pacer', None),
+            request_func=requests.request,
+            sleep=time.sleep,
+            log_retries=log_retries,
+            on_rate_limit=on_rate_limit,
+            **kwargs,
+        )
 
 
-    def _get_dataflow_pbi_definition(self, workspace_id: str, dataflow_id: str) -> Dict:
+    def _get_dataflow_pbi_definition(
+        self,
+        workspace_id: str,
+        dataflow_id: str,
+        log_retries: bool = True,
+        on_rate_limit: Optional[Callable[[float], None]] = None,
+    ) -> Dict:
         """
         Fetches a dataflow definition from the Power BI REST API.
         Works for Gen1 and Gen2 (standard) dataflows.
@@ -80,7 +100,13 @@ class Dataflow:
             return {'message': 'Missing dataflow id, please check.', 'content': ''}
 
         request_url = f'{self.main_url}/groups/{workspace_id}/dataflows/{dataflow_id}'
-        r = self._request_with_retry('GET', request_url, headers=self.headers)
+        r = self._request_with_retry(
+            'GET',
+            request_url,
+            headers=self.headers,
+            log_retries=log_retries,
+            on_rate_limit=on_rate_limit,
+        )
 
         if r.status_code == 200:
             return {'message': 'Success', 'content': json.loads(r.content)}
@@ -166,6 +192,10 @@ class Dataflow:
             if not fabric_df.empty:
                 fabric_df['name'] = fabric_df['displayName']
                 fabric_df.drop(columns=['displayName'], inplace=True)
+                if 'generation' not in fabric_df.columns:
+                    fabric_df['generation'] = 2.1
+                else:
+                    fabric_df['generation'] = fabric_df['generation'].fillna(2.1)
                 fabric_df['source'] = 'fabric'
                 fabric_records = fabric_df.to_dict('records')
 
@@ -371,7 +401,13 @@ class Dataflow:
 
         return {'message': 'Success', 'content': response}
 
-    def get_dataflow_gen2_definition(self, workspace_id: str, dataflow_id: str) -> Dict:
+    def get_dataflow_gen2_definition(
+        self,
+        workspace_id: str,
+        dataflow_id: str,
+        verbose: bool = True,
+        on_rate_limit: Optional[Callable[[float], None]] = None,
+    ) -> Dict:
         """
         Gets the definition of a Dataflow Gen2 (CI/CD) from a specified workspace.
         Only Dataflow Gen2 (CI/CD / native Fabric) items support definition export.
@@ -380,18 +416,27 @@ class Dataflow:
         Args:
             workspace_id (str): The ID of the workspace where the Dataflow Gen2 resides.
             dataflow_id (str): The ID of the Dataflow Gen2 to retrieve the definition for.
+            verbose (bool): Print per-dataflow progress. Defaults to True.
 
         Returns:
             Dict: A dictionary containing the status ('Success' or error) and the Dataflow Gen2 definition content.
         """
         api_url = f'{self.fabric_api_base_url}/v1/workspaces/{workspace_id}/dataflows/{dataflow_id}/getDefinition'
 
-        print(f"Extracting definition for dataflow {dataflow_id} from workspace {workspace_id}...")
-        response = requests.post(api_url, headers=self.headers)
+        if verbose:
+            print(f"Extracting definition for dataflow {dataflow_id} from workspace {workspace_id}...")
+        response = self._request_with_retry(
+            'POST',
+            api_url,
+            headers=self.headers,
+            log_retries=verbose,
+            on_rate_limit=on_rate_limit,
+        )
 
         if response.status_code == 200:
             definition = response.json()
-            print("Successfully extracted Dataflow Gen2 definition.")
+            if verbose:
+                print("Successfully extracted Dataflow Gen2 definition.")
             return {'message': 'Success', 'content': definition}
         else:
             # getDefinition only works for Dataflow Gen2 (CI/CD / native Fabric).
@@ -410,7 +455,8 @@ class Dataflow:
                 }
 
             error_message = response.text
-            print(f"Error getting Dataflow Gen2 definition: {response.status_code} - {error_message}")
+            if verbose:
+                print(f"Error getting Dataflow Gen2 definition: {response.status_code} - {error_message}")
             return {'message': {'error': error_message, 'status_code': response.status_code}}
         
 
@@ -1141,7 +1187,14 @@ class Dataflow:
         return new_m_code, metadata
 
 
-    def get_data_destinations(self, workspace_id: str, dataflow_id: str) -> Dict:
+    def get_data_destinations(
+        self,
+        workspace_id: str,
+        dataflow_id: str,
+        source: Optional[str] = None,
+        verbose: bool = True,
+        on_rate_limit: Optional[Callable[[float], None]] = None,
+    ) -> Dict:
         """
         Gets the data destination details for each table in a dataflow.
 
@@ -1152,10 +1205,14 @@ class Dataflow:
         Args:
             workspace_id: The workspace ID where the dataflow resides.
             dataflow_id: The dataflow ID.
+            source: Source returned by :meth:`list_dataflows`. Uses the known
+                API route directly when it is ``pbi`` or ``fabric``.
+            verbose: Print definition extraction progress. Defaults to True.
+            on_rate_limit: Called with the retry delay for each HTTP 429.
 
         Returns:
             Dict: 'message' and 'content' (list of dicts with keys:
-                table, destination_type, workspace_id, item_id, sql_schema,
+                name, destination_type, workspace_id, item_id, sql_schema,
                 mapping_type ('Automatic' or 'Manual'),
                 columns (list of {source, destination} dicts, empty for automatic mappings)).
         """
@@ -1164,18 +1221,176 @@ class Dataflow:
         if dataflow_id == '':
             return {'message': 'Missing dataflow id, please check.', 'content': ''}
 
-        # Try CI/CD format first
-        cicd_result = self.get_dataflow_gen2_definition(workspace_id, dataflow_id)
-        is_cicd = cicd_result.get('message') == 'Success'
+        cicd_result = None
+        if source != 'pbi':
+            cicd_result = self.get_dataflow_gen2_definition(
+                workspace_id,
+                dataflow_id,
+                verbose=verbose,
+                on_rate_limit=on_rate_limit,
+            )
+            if cicd_result.get('message') == 'Success':
+                return self._get_data_destinations_cicd(cicd_result['content'])
+            if source == 'fabric':
+                return {
+                    'message': f'Failed to fetch Dataflow Gen2 definition: {cicd_result.get("message")}',
+                    'content': '',
+                }
 
-        if is_cicd:
-            return self._get_data_destinations_cicd(cicd_result['content'])
-        else:
-            pbi_result = self._get_dataflow_pbi_definition(workspace_id, dataflow_id)
-            if pbi_result.get('message') == 'Success':
-                return self._get_data_destinations_standard(pbi_result['content'])
+        pbi_result = self._get_dataflow_pbi_definition(
+            workspace_id,
+            dataflow_id,
+            log_retries=verbose,
+            on_rate_limit=on_rate_limit,
+        )
+        if pbi_result.get('message') == 'Success':
+            return self._get_data_destinations_standard(pbi_result['content'])
+
+        cicd_message = cicd_result.get('message') if cicd_result else 'not attempted'
+        return {
+            'message': (
+                'Failed to fetch dataflow definition. '
+                f'CI/CD: {cicd_message}. PBI: {pbi_result.get("message")}'
+            ),
+            'content': '',
+        }
+
+
+    def get_workspace_data_destinations(
+        self,
+        workspace_id: str,
+        max_workers: int = 4,
+    ) -> Dict:
+        """Get destination tables for every dataflow in a workspace.
+
+        Uses :meth:`list_dataflows` as the source of truth for the workspace,
+        then obtains each dataflow's destinations concurrently. Requests share
+        the hardcoded 200 requests-per-minute pace and retry 429 responses, so
+        increasing ``max_workers`` does not bypass API throttling.
+
+        Args:
+            workspace_id: The workspace ID to inventory.
+            max_workers: Maximum concurrent definition lookups. Defaults to 4.
+
+        Returns:
+            Dict with ``content`` containing one item per dataflow. Each item
+            has the source ``dataflow`` record, ``tables`` with destination
+            details, and ``error`` when that dataflow could not be inspected.
+            The message is ``Success`` when all dataflows were inspected or
+            ``Partial success`` when one or more definition lookups failed.
+        """
+        if workspace_id == '':
+            return {'message': 'Missing workspace id, please check.', 'content': ''}
+        if max_workers < 1:
+            return {'message': 'max_workers must be at least 1.', 'content': ''}
+
+        dataflows_result = self.list_dataflows(workspace_id)
+        if dataflows_result.get('message') != 'Success':
+            return {
+                'message': dataflows_result.get('message'),
+                'content': [],
+            }
+
+        dataflows = dataflows_result.get('content', [])
+        results = [None] * len(dataflows)
+        progress_lock = Lock()
+        progress = {'completed': 0}
+
+        def report_rate_limit(_retry_delay: float) -> None:
+            with progress_lock:
+                print(
+                    f'\n{progress["completed"]} processados até então, '
+                    'aguardando rate limit...',
+                    flush=True,
+                )
+
+        def inspect_dataflow(index: int, dataflow: Dict) -> tuple[int, Dict]:
+            dataflow_id = dataflow.get('id') or dataflow.get('objectId')
+            result = {
+                'dataflow': dataflow,
+                'tables': [],
+            }
+            if not dataflow_id:
+                result['error'] = 'Dataflow record has no id.'
+                return index, result
+
+            try:
+                destinations_result = self.get_data_destinations(
+                    workspace_id,
+                    dataflow_id,
+                    source=dataflow.get('source'),
+                    verbose=False,
+                    on_rate_limit=report_rate_limit,
+                )
+            except Exception as error:
+                result['error'] = str(error)
+                return index, result
+
+            if destinations_result.get('message') == 'Success':
+                result['tables'] = destinations_result.get('content', [])
             else:
-                return {'message': f'Failed to fetch dataflow definition. CI/CD: {cicd_result.get("message")}. PBI: {pbi_result.get("message")}', 'content': ''}
+                result['error'] = destinations_result.get('message')
+            return index, result
+
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            futures = [
+                executor.submit(inspect_dataflow, index, dataflow)
+                for index, dataflow in enumerate(dataflows)
+            ]
+            for completed, future in enumerate(as_completed(futures), start=1):
+                index, result = future.result()
+                results[index] = result
+                with progress_lock:
+                    progress['completed'] = completed
+                    print(
+                        f'\rExtracted definition from {completed}/{len(dataflows)} '
+                        f'dataflows from workspace {workspace_id}...',
+                        end='',
+                        flush=True,
+                    )
+
+        if dataflows:
+            print()
+
+        self._export_workspace_dataflow_destinations(workspace_id, results)
+        failed = [result for result in results if result.get('error')]
+        return {
+            'message': 'Partial success' if failed else 'Success',
+            'content': results,
+        }
+
+
+    def _export_workspace_dataflow_destinations(
+        self,
+        workspace_id: str,
+        dataflows: List[Dict],
+    ) -> pd.DataFrame:
+        """Flatten workspace destination results and save them to Excel."""
+        metadata = [
+            ['dataflow', 'id'],
+            ['dataflow', 'name'],
+            ['dataflow', 'configuredBy'],
+            ['dataflow', 'generation'],
+            ['dataflow', 'source'],
+        ]
+        dataframe = pd.json_normalize(
+            dataflows,
+            record_path=['tables'],
+            meta=metadata,
+            record_prefix='table_',
+            errors='ignore',
+        )
+        dataframe.columns = [column.replace('.', '_') for column in dataframe.columns]
+        dataflow_columns = [
+            column for column in dataframe.columns if column.startswith('dataflow_')
+        ]
+        destination_columns = [
+            column for column in dataframe.columns if not column.startswith('dataflow_')
+        ]
+        dataframe = dataframe[dataflow_columns + destination_columns]
+        filename = f'workspace_dataflow_destinations_{workspace_id}.xlsx'
+        dataframe.to_excel(os.path.join(self.dataflows_dir, filename), index=False)
+        return dataframe
 
 
     def _parse_column_mappings(self, annotation: str) -> List[Dict]:
@@ -1235,7 +1450,7 @@ class Dataflow:
             lh_match = re.search(r'lakehouseId\s*=\s*"([^"]+)"', body)
             for q in data_queries:
                 destinations.append({
-                    'table': q['name'],
+                    'name': q['name'],
                     'destination_type': 'Lakehouse',
                     'workspace_id': ws_match.group(1) if ws_match else '',
                     'item_id': lh_match.group(1) if lh_match else '',
@@ -1260,7 +1475,7 @@ class Dataflow:
                 if 'Lakehouse.Contents' in body:
                     lh_match = re.search(r'lakehouseId\s*=\s*"([^"]+)"', body)
                     destinations.append({
-                        'table': q['name'],
+                        'name': q['name'],
                         'destination_type': 'Lakehouse',
                         'workspace_id': ws_match.group(1) if ws_match else '',
                         'item_id': lh_match.group(1) if lh_match else '',
@@ -1271,7 +1486,7 @@ class Dataflow:
                 elif 'Fabric.Warehouse' in body:
                     wh_match = re.search(r'warehouseId\s*=\s*"([^"]+)"', body)
                     destinations.append({
-                        'table': q['name'],
+                        'name': q['name'],
                         'destination_type': 'Warehouse',
                         'workspace_id': ws_match.group(1) if ws_match else '',
                         'item_id': wh_match.group(1) if wh_match else '',
@@ -1312,7 +1527,7 @@ class Dataflow:
             if 'Lakehouse.Contents' in body:
                 lh_match = re.search(r'lakehouseId\s*=\s*"([^"]+)"', body)
                 destinations.append({
-                    'table': table_name,
+                    'name': table_name,
                     'destination_type': 'Lakehouse',
                     'workspace_id': ws_match.group(1) if ws_match else '',
                     'item_id': lh_match.group(1) if lh_match else '',
@@ -1323,7 +1538,7 @@ class Dataflow:
             elif 'Fabric.Warehouse' in body:
                 wh_match = re.search(r'warehouseId\s*=\s*"([^"]+)"', body)
                 destinations.append({
-                    'table': table_name,
+                    'name': table_name,
                     'destination_type': 'Warehouse',
                     'workspace_id': ws_match.group(1) if ws_match else '',
                     'item_id': wh_match.group(1) if wh_match else '',
@@ -1343,7 +1558,7 @@ class Dataflow:
                 bind_pattern = r'\[BindToDefaultDestination\s*=\s*true\]\s*\n\s*shared\s+(\w+)\s*='
                 for bind_match in re.finditer(bind_pattern, m_code):
                     destinations.append({
-                        'table': bind_match.group(1),
+                        'name': bind_match.group(1),
                         'destination_type': 'Lakehouse',
                         'workspace_id': ws_match.group(1) if ws_match else '',
                         'item_id': lh_match.group(1) if lh_match else '',

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/msdev_kit/fabric/dataset.py

@@ -8,6 +8,7 @@ from . import report
 from typing import Dict
 from .utilities import create_directory
 from concurrent.futures import ThreadPoolExecutor
+from msdev_kit.http import request_with_retry
 
 
 class Dataset:
@@ -28,21 +29,16 @@ class Dataset:
         self, method: str, url: str, max_retries: int = 3, **kwargs
     ) -> requests.Response:
         """
-        Makes an HTTP request with automatic retry on 429 (Too Many Requests).
-        Respects the Retry-After header when present.
+        Compatibility wrapper around the shared HTTP retry helper.
         """
-        for attempt in range(max_retries + 1):
-            response = requests.request(method, url, **kwargs)
-            if response.status_code != 429:
-                return response
-
-            retry_after = int(response.headers.get("Retry-After", 5))
-            print(
-                f"  Rate limited (429). Retrying in {retry_after}s... (attempt {attempt + 1}/{max_retries})"
-            )
-            time.sleep(retry_after)
-
-        return response
+        return request_with_retry(
+            method,
+            url,
+            max_retries=max_retries,
+            request_func=requests.request,
+            sleep=time.sleep,
+            **kwargs,
+        )
 
     def get_dataset_name(self, workspace_id: str, dataset_id: str) -> str:
         """

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/msdev_kit/fabric/notebook.py

@@ -1,6 +1,7 @@
 import time
 import requests
 from typing import Dict
+from msdev_kit.http import request_with_retry
 
 
 class Notebook:
@@ -16,19 +17,16 @@ class Notebook:
 
     def _request_with_retry(self, method: str, url: str, max_retries: int = 3, **kwargs) -> requests.Response:
         """
-        Makes an HTTP request with automatic retry on 429 (Too Many Requests).
-        Respects the Retry-After header when present.
+        Compatibility wrapper around the shared HTTP retry helper.
         """
-        for attempt in range(max_retries + 1):
-            response = requests.request(method, url, **kwargs)
-            if response.status_code != 429:
-                return response
-
-            retry_after = int(response.headers.get('Retry-After', 5))
-            print(f"  Rate limited (429). Retrying in {retry_after}s... (attempt {attempt + 1}/{max_retries})")
-            time.sleep(retry_after)
-
-        return response
+        return request_with_retry(
+            method,
+            url,
+            max_retries=max_retries,
+            request_func=requests.request,
+            sleep=time.sleep,
+            **kwargs,
+        )
 
 
     def list_notebooks(self, workspace_id: str) -> Dict:

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/msdev_kit/fabric/pipeline.py

@@ -7,6 +7,7 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
 from .dataflow import Dataflow
 from .notebook import Notebook
 from .dataset import Dataset
+from msdev_kit.http import request_with_retry
 
 
 class Pipeline:
@@ -23,21 +24,16 @@ class Pipeline:
         self, method: str, url: str, max_retries: int = 3, **kwargs
     ) -> requests.Response:
         """
-        Makes an HTTP request with automatic retry on 429 (Too Many Requests).
-        Respects the Retry-After header when present.
+        Compatibility wrapper around the shared HTTP retry helper.
         """
-        for attempt in range(max_retries + 1):
-            response = requests.request(method, url, **kwargs)
-            if response.status_code != 429:
-                return response
-
-            retry_after = int(response.headers.get("Retry-After", 5))
-            print(
-                f"  Rate limited (429). Retrying in {retry_after}s... (attempt {attempt + 1}/{max_retries})"
-            )
-            time.sleep(retry_after)
-
-        return response
+        return request_with_retry(
+            method,
+            url,
+            max_retries=max_retries,
+            request_func=requests.request,
+            sleep=time.sleep,
+            **kwargs,
+        )
 
     def _resolve_pipeline(self, workspace_id: str, pipeline_id_or_name: str) -> tuple:
         """

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/msdev_kit/graph/client.py

@@ -1,6 +1,7 @@
 import requests
 from typing import Optional
 from msdev_kit.auth import Auth
+from msdev_kit.http import request_with_retry
 
 
 class GraphClient:
@@ -15,10 +16,20 @@ class GraphClient:
             'Content-Type': 'application/json',
         }
 
+    def _request(self, method: str, url: str, **kwargs) -> requests.Response:
+        """Send a Graph request through the shared retry implementation."""
+        return request_with_retry(
+            method,
+            url,
+            request_func=requests.request,
+            **kwargs,
+        )
+
     def get_user_id(self, email: str) -> Optional[str]:
         """Return the Entra object ID of a user by UPN/email, or None if not found.
         Falls back to filtering by the mail field when the UPN lookup returns 404."""
-        resp = requests.get(
+        resp = self._request(
+            'GET',
             f'{self._GRAPH_BASE}/users/{email}',
             headers=self._headers(),
             params={'$select': 'id'},
@@ -28,7 +39,8 @@ class GraphClient:
             resp.raise_for_status()
             return resp.json().get('id')
 
-        resp = requests.get(
+        resp = self._request(
+            'GET',
             f'{self._GRAPH_BASE}/users',
             headers=self._headers(),
             params={'$filter': f"mail eq '{email}'", '$select': 'id'},
@@ -40,7 +52,8 @@ class GraphClient:
 
     def get_group_id(self, group_name: str) -> Optional[str]:
         """Return the Entra object ID of a security group by displayName, or None."""
-        resp = requests.get(
+        resp = self._request(
+            'GET',
             f'{self._GRAPH_BASE}/groups',
             headers=self._headers(),
             params={'$filter': f"displayName eq '{group_name}'", '$select': 'id,displayName'},
@@ -58,7 +71,7 @@ class GraphClient:
         params = {'$select': 'id,displayName,mail,userPrincipalName', '$top': '999'}
 
         while url:
-            resp = requests.get(url, headers=self._headers(), params=params, timeout=30)
+            resp = self._request('GET', url, headers=self._headers(), params=params, timeout=30)
             resp.raise_for_status()
             data = resp.json()
             members.extend(data.get('value', []))
@@ -69,7 +82,8 @@ class GraphClient:
 
     def add_group_member(self, group_id: str, user_id: str):
         """Add user to group. Silently ignores 'already a member' errors."""
-        resp = requests.post(
+        resp = self._request(
+            'POST',
             f'{self._GRAPH_BASE}/groups/{group_id}/members/$ref',
             headers=self._headers(),
             json={'@odata.id': f'{self._GRAPH_BASE}/directoryObjects/{user_id}'},
@@ -81,7 +95,8 @@ class GraphClient:
 
     def remove_group_member(self, group_id: str, user_id: str):
         """Remove user from group. Silently ignores 404 (not a member) and 403 (insufficient privileges)."""
-        resp = requests.delete(
+        resp = self._request(
+            'DELETE',
             f'{self._GRAPH_BASE}/groups/{group_id}/members/{user_id}/$ref',
             headers=self._headers(),
             timeout=30,

msdev_kit-0.2.4/msdev_kit/http.py

@@ -0,0 +1,92 @@
+"""Shared HTTP request helpers for Microsoft service clients."""
+
+from datetime import datetime, timezone
+from email.utils import parsedate_to_datetime
+from threading import Lock
+import time
+from typing import Callable, Optional
+
+import requests
+
+
+class RequestPacer:
+    """Thread-safe minimum interval between requests from one client instance."""
+
+    def __init__(self, requests_per_minute: float):
+        if requests_per_minute <= 0:
+            raise ValueError('requests_per_minute must be greater than zero.')
+        self._interval = 60 / requests_per_minute
+        self._lock = Lock()
+        self._last_request_at = 0.0
+
+    def wait(self) -> None:
+        """Wait until the next request is allowed."""
+        with self._lock:
+            elapsed = time.monotonic() - self._last_request_at
+            if elapsed < self._interval:
+                time.sleep(self._interval - elapsed)
+            self._last_request_at = time.monotonic()
+
+
+def request_with_retry(
+    method: str,
+    url: str,
+    *,
+    max_retries: int = 3,
+    pacer: Optional[RequestPacer] = None,
+    request_func: Callable[..., requests.Response] = requests.request,
+    sleep: Optional[Callable[[float], None]] = None,
+    log_retries: bool = True,
+    on_rate_limit: Optional[Callable[[float], None]] = None,
+    **kwargs,
+) -> requests.Response:
+    """Send an HTTP request and retry HTTP 429 responses.
+
+    ``method`` accepts any HTTP verb supported by ``requests``. When a 429
+    includes ``Retry-After``, its delta-seconds or HTTP date value is honored.
+    Otherwise retries back off exponentially from one second. A shared
+    ``RequestPacer`` can be supplied by callers that issue concurrent work.
+    """
+    normalized_method = method.strip().upper() if isinstance(method, str) else ''
+    if not normalized_method:
+        raise ValueError('method must be a non-empty HTTP method string.')
+    if max_retries < 0:
+        raise ValueError('max_retries must be zero or greater.')
+    if sleep is None:
+        sleep = time.sleep
+
+    for attempt in range(max_retries + 1):
+        if pacer is not None:
+            pacer.wait()
+
+        response = request_func(normalized_method, url, **kwargs)
+        if response.status_code != 429:
+            return response
+
+        if attempt < max_retries:
+            retry_delay = _get_retry_delay(response, attempt)
+            if on_rate_limit is not None:
+                on_rate_limit(retry_delay)
+            if log_retries:
+                print(
+                    f'  Rate limited (429). Retrying in {retry_delay:g}s... '
+                    f'(attempt {attempt + 1}/{max_retries})'
+                )
+            sleep(retry_delay)
+
+    return response
+
+
+def _get_retry_delay(response: requests.Response, attempt: int) -> float:
+    """Return a Retry-After delay, falling back to exponential backoff."""
+    retry_after = response.headers.get('Retry-After')
+    try:
+        return max(float(retry_after), 0)
+    except (TypeError, ValueError):
+        try:
+            retry_after_at = parsedate_to_datetime(retry_after)
+            if retry_after_at.tzinfo is None:
+                retry_after_at = retry_after_at.replace(tzinfo=timezone.utc)
+            return max((retry_after_at - datetime.now(timezone.utc)).total_seconds(), 0)
+        except (TypeError, ValueError):
+            return min(2 ** attempt, 60)

{msdev_kit-0.2.2 → msdev_kit-0.2.4}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "msdev-kit"
 description = "Microsoft developer toolkit: Fabric, MS Graph, and SharePoint"
-version = "0.2.2"
+version = "0.2.4"
 requires-python = ">=3.10"
 readme = "README.md"
 license = {text = "MIT"}