airbyte-source-microsoft-sharepoint 0.2.0__py3-none-any.whl

airbyte_source_microsoft_sharepoint-0.2.0.dist-info/METADATA

@@ -0,0 +1,113 @@
+Metadata-Version: 2.1
+Name: airbyte-source-microsoft-sharepoint
+Version: 0.2.0
+Summary: Source implementation for Microsoft SharePoint.
+Home-page: https://airbyte.com
+License: MIT
+Author: Airbyte
+Author-email: contact@airbyte.io
+Requires-Python: >=3.9,<3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: Office365-REST-Python-Client (==2.5.5)
+Requires-Dist: airbyte-cdk[file-based] (==0.61.0)
+Requires-Dist: msal (==1.25.0)
+Requires-Dist: smart-open (==6.4.0)
+Project-URL: Documentation, https://docs.airbyte.com/integrations/sources/microsoft-sharepoint
+Project-URL: Repository, https://github.com/airbytehq/airbyte
+Description-Content-Type: text/markdown
+
+# Microsoft SharePoint source connector
+
+
+This is the repository for the Microsoft SharePoint source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/microsoft-sharepoint).
+
+## Local development
+
+### Prerequisites
+* Python (~=3.9)
+* Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
+
+
+### Installing the connector
+From this connector directory, run:
+```bash
+poetry install --with dev
+```
+
+
+### Create credentials
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/microsoft-sharepoint)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_microsoft_sharepoint/spec.yaml` file.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
+See `sample_files/sample_config.json` for a sample config file.
+
+
+### Locally running the connector
+```
+poetry run source-microsoft-sharepoint spec
+poetry run source-microsoft-sharepoint check --config secrets/config.json
+poetry run source-microsoft-sharepoint discover --config secrets/config.json
+poetry run source-microsoft-sharepoint read --config secrets/config.json --catalog sample_files/configured_catalog.json
+```
+
+### Running unit tests
+To run unit tests locally, from the connector directory run:
+```
+poetry run pytest unit_tests
+```
+
+### Building the docker image
+1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
+2. Run the following command to build the docker image:
+```bash
+airbyte-ci connectors --name=source-microsoft-sharepoint build
+```
+
+An image will be available on your host with the tag `airbyte/source-microsoft-sharepoint:dev`.
+
+
+### Running as a docker container
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-microsoft-sharepoint:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-microsoft-sharepoint:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-microsoft-sharepoint:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-microsoft-sharepoint:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+### Running our CI test suite
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+```bash
+airbyte-ci connectors --name=source-microsoft-sharepoint test
+```
+
+### Customizing acceptance Tests
+Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+### Dependency Management
+All of your dependencies should be managed via Poetry. 
+To add a new dependency, run:
+```bash
+poetry add <package-name>
+```
+
+Please commit the changes to `pyproject.toml` and `poetry.lock` files.
+
+## Publishing a new version of the connector
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-microsoft-sharepoint test`
+2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): 
+    - bump the `dockerImageTag` value in in `metadata.yaml`
+    - bump the `version` value in `pyproject.toml`
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/microsoft-sharepoint.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

airbyte_source_microsoft_sharepoint-0.2.0.dist-info/RECORD

@@ -0,0 +1,10 @@
+source_microsoft_sharepoint/__init__.py,sha256=a1qKp3Ui2-daZ3RA6LQ1BhcgMlB1reM-3io73TalbUw,150
+source_microsoft_sharepoint/run.py,sha256=vyooBYcOlWf5ulMvNFmmHzTBS0sjxa5-rYljTqpO1m8,766
+source_microsoft_sharepoint/source.py,sha256=c0a16gbjtA0N1XQ9pfcpBX7LaXjQFepPL6kfLsFf5GE,3211
+source_microsoft_sharepoint/spec.py,sha256=B4Gpytf_CY0lT2awcgiRYIC6SSsQ4pVOcdxYnyHC1AE,5146
+source_microsoft_sharepoint/stream_reader.py,sha256=CnGi2pYoyAoxqxFcszhfdWgvI5MOh3gRt0c9dX0GVHs,12207
+source_microsoft_sharepoint/utils.py,sha256=Fmp29wIxjuZv6uAf74ejzd_ZasUdmZFQSdwIi-Cgk1I,3849
+airbyte_source_microsoft_sharepoint-0.2.0.dist-info/METADATA,sha256=a0IpD7ALMkzCamx-beLqZxFsQ4SznZqAumU0nN3Ona4,5627
+airbyte_source_microsoft_sharepoint-0.2.0.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
+airbyte_source_microsoft_sharepoint-0.2.0.dist-info/entry_points.txt,sha256=qQVJCey5TyNCB-xZURXYwXMUbcy2tgmcJboz_VJEWvc,83
+airbyte_source_microsoft_sharepoint-0.2.0.dist-info/RECORD,,

airbyte_source_microsoft_sharepoint-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 1.9.0
+Root-Is-Purelib: true
+Tag: py3-none-any

airbyte_source_microsoft_sharepoint-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+source-microsoft-sharepoint=source_microsoft_sharepoint.run:run
+

source_microsoft_sharepoint/__init__.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from .source import SourceMicrosoftSharePoint
+
+__all__ = ["SourceMicrosoftSharePoint"]

source_microsoft_sharepoint/run.py

@@ -0,0 +1,23 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+import sys
+
+from airbyte_cdk import AirbyteEntrypoint
+from airbyte_cdk.entrypoint import launch
+from source_microsoft_sharepoint.source import SourceMicrosoftSharePoint
+
+
+def run():
+    args = sys.argv[1:]
+    catalog_path = AirbyteEntrypoint.extract_catalog(args)
+    config_path = AirbyteEntrypoint.extract_config(args)
+    state_path = AirbyteEntrypoint.extract_state(args)
+    source = SourceMicrosoftSharePoint(
+        SourceMicrosoftSharePoint.read_catalog(catalog_path) if catalog_path else None,
+        SourceMicrosoftSharePoint.read_config(config_path) if config_path else None,
+        SourceMicrosoftSharePoint.read_state(state_path) if state_path else None,
+    )
+    launch(source, args)

source_microsoft_sharepoint/source.py

@@ -0,0 +1,65 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from typing import Any, Mapping, Optional
+
+from airbyte_cdk.models import AdvancedAuth, ConfiguredAirbyteCatalog, ConnectorSpecification, OAuthConfigSpecification
+from airbyte_cdk.sources.file_based.file_based_source import FileBasedSource
+from airbyte_cdk.sources.file_based.stream.cursor.default_file_based_cursor import DefaultFileBasedCursor
+from airbyte_cdk.sources.source import TState
+from source_microsoft_sharepoint.spec import SourceMicrosoftSharePointSpec
+from source_microsoft_sharepoint.stream_reader import SourceMicrosoftSharePointStreamReader
+
+
+class SourceMicrosoftSharePoint(FileBasedSource):
+    def __init__(self, catalog: Optional[ConfiguredAirbyteCatalog], config: Optional[Mapping[str, Any]], state: Optional[TState]):
+        super().__init__(
+            stream_reader=SourceMicrosoftSharePointStreamReader(),
+            spec_class=SourceMicrosoftSharePointSpec,
+            catalog=catalog,
+            config=config,
+            state=state,
+            cursor_cls=DefaultFileBasedCursor,
+        )
+
+    def spec(self, *args: Any, **kwargs: Any) -> ConnectorSpecification:
+        """
+        Returns the specification describing what fields can be configured by a user when setting up a file-based source.
+        """
+
+        return ConnectorSpecification(
+            documentationUrl=self.spec_class.documentation_url(),
+            connectionSpecification=self.spec_class.schema(),
+            advanced_auth=AdvancedAuth(
+                auth_flow_type="oauth2.0",
+                predicate_key=["credentials", "auth_type"],
+                predicate_value="Client",
+                oauth_config_specification=OAuthConfigSpecification(
+                    complete_oauth_output_specification={
+                        "type": "object",
+                        "additionalProperties": False,
+                        "properties": {"refresh_token": {"type": "string", "path_in_connector_config": ["credentials", "refresh_token"]}},
+                    },
+                    complete_oauth_server_input_specification={
+                        "type": "object",
+                        "additionalProperties": False,
+                        "properties": {"client_id": {"type": "string"}, "client_secret": {"type": "string"}},
+                    },
+                    complete_oauth_server_output_specification={
+                        "type": "object",
+                        "additionalProperties": False,
+                        "properties": {
+                            "client_id": {"type": "string", "path_in_connector_config": ["credentials", "client_id"]},
+                            "client_secret": {"type": "string", "path_in_connector_config": ["credentials", "client_secret"]},
+                        },
+                    },
+                    oauth_user_input_from_connector_config_specification={
+                        "type": "object",
+                        "additionalProperties": False,
+                        "properties": {"tenant_id": {"type": "string", "path_in_connector_config": ["credentials", "tenant_id"]}},
+                    },
+                ),
+            ),
+        )

source_microsoft_sharepoint/spec.py

@@ -0,0 +1,125 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from typing import Any, Dict, Literal, Optional, Union
+
+import dpath.util
+from airbyte_cdk.sources.file_based.config.abstract_file_based_spec import AbstractFileBasedSpec
+from pydantic import BaseModel, Field
+
+
+class OAuthCredentials(BaseModel):
+    """
+    OAuthCredentials class to hold authentication details for Microsoft OAuth authentication.
+    This class uses pydantic for data validation and settings management.
+    """
+
+    class Config:
+        title = "Authenticate via Microsoft (OAuth)"
+
+    # Fields for the OAuth authentication, including tenant_id, client_id, client_secret, and refresh_token
+    auth_type: Literal["Client"] = Field("Client", const=True)
+    tenant_id: str = Field(title="Tenant ID", description="Tenant ID of the Microsoft SharePoint user", airbyte_secret=True)
+    client_id: str = Field(
+        title="Client ID",
+        description="Client ID of your Microsoft developer application",
+        airbyte_secret=True,
+    )
+    client_secret: str = Field(
+        title="Client Secret",
+        description="Client Secret of your Microsoft developer application",
+        airbyte_secret=True,
+    )
+    refresh_token: str = Field(
+        title="Refresh Token",
+        description="Refresh Token of your Microsoft developer application",
+        airbyte_secret=True,
+    )
+
+
+class ServiceCredentials(BaseModel):
+    """
+    ServiceCredentials class for service key authentication.
+    This class is structured similarly to OAuthCredentials but for a different authentication method.
+    """
+
+    class Config:
+        title = "Service Key Authentication"
+
+    # Fields for the Service authentication, similar to OAuthCredentials
+    auth_type: Literal["Service"] = Field("Service", const=True)
+    tenant_id: str = Field(title="Tenant ID", description="Tenant ID of the Microsoft SharePoint user", airbyte_secret=True)
+    user_principal_name: str = Field(
+        title="User Principal Name",
+        description="Special characters such as a period, comma, space, and the at sign (@) are converted to underscores (_). More details: https://learn.microsoft.com/en-us/sharepoint/list-onedrive-urls",
+        airbyte_secret=True,
+    )
+    client_id: str = Field(
+        title="Client ID",
+        description="Client ID of your Microsoft developer application",
+        airbyte_secret=True,
+    )
+    client_secret: str = Field(
+        title="Client Secret",
+        description="Client Secret of your Microsoft developer application",
+        airbyte_secret=True,
+    )
+
+
+class SourceMicrosoftSharePointSpec(AbstractFileBasedSpec, BaseModel):
+    """
+    SourceMicrosoftSharePointSpec class for Microsoft SharePoint Source Specification.
+    This class combines the authentication details with additional configuration for the SharePoint API.
+    """
+
+    class Config:
+        title = "Microsoft SharePoint Source Spec"
+
+    # Union type for credentials, allowing for either OAuth or Service Key authentication
+    credentials: Union[OAuthCredentials, ServiceCredentials] = Field(
+        title="Authentication",
+        description="Credentials for connecting to the One Drive API",
+        discriminator="auth_type",
+        type="object",
+        order=0,
+    )
+
+    search_scope: str = Field(
+        title="Search Scope",
+        description="Specifies the location(s) to search for files. Valid options are 'ACCESSIBLE_DRIVES' for all SharePoint drives the user can access, 'SHARED_ITEMS' for shared items the user has access to, and 'ALL' to search both.",
+        default="ALL",
+        enum=["ACCESSIBLE_DRIVES", "SHARED_ITEMS", "ALL"],
+        order=3,
+    )
+
+    folder_path: str = Field(
+        title="Folder Path",
+        description="Path to a specific folder within the drives to search for files. Leave empty to search all folders of the drives. This does not apply to shared items.",
+        order=4,
+        default=".",
+    )
+
+    @classmethod
+    def documentation_url(cls) -> str:
+        """Provides the URL to the documentation for this specific source."""
+        return "https://docs.airbyte.com/integrations/sources/microsoft-sharepoint"
+
+    @classmethod
+    def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]:
+        """
+        Generates the schema mapping for configuration fields.
+        It also cleans up the schema by removing legacy settings and discriminators.
+        """
+        schema = super().schema(*args, **kwargs)
+
+        # Remove legacy settings related to streams
+        dpath.util.delete(schema, "properties/streams/items/properties/legacy_prefix")
+        dpath.util.delete(schema, "properties/streams/items/properties/format/oneOf/*/properties/inference_type")
+
+        # Hide API processing option until https://github.com/airbytehq/airbyte-platform-internal/issues/10354 is fixed
+        processing_options = dpath.util.get(schema, "properties/streams/items/properties/format/oneOf/4/properties/processing/oneOf")
+        dpath.util.set(schema, "properties/streams/items/properties/format/oneOf/4/properties/processing/oneOf", processing_options[:1])
+
+        return schema

source_microsoft_sharepoint/stream_reader.py

@@ -0,0 +1,278 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+import logging
+from datetime import datetime
+from functools import lru_cache
+from io import IOBase
+from typing import Iterable, List, Optional, Tuple
+
+import requests
+import smart_open
+from airbyte_cdk.sources.file_based.file_based_stream_reader import AbstractFileBasedStreamReader, FileReadMode
+from airbyte_cdk.sources.file_based.remote_file import RemoteFile
+from airbyte_cdk.utils.traced_exception import AirbyteTracedException, FailureType
+from msal import ConfidentialClientApplication
+from office365.graph_client import GraphClient
+from source_microsoft_sharepoint.spec import SourceMicrosoftSharePointSpec
+
+from .utils import MicrosoftSharePointRemoteFile, execute_query_with_retry, filter_http_urls
+
+
+class SourceMicrosoftSharePointClient:
+    """
+    Client to interact with Microsoft SharePoint.
+    """
+
+    def __init__(self, config: SourceMicrosoftSharePointSpec):
+        self.config = config
+        self._client = None
+        self._msal_app = ConfidentialClientApplication(
+            self.config.credentials.client_id,
+            authority=f"https://login.microsoftonline.com/{self.config.credentials.tenant_id}",
+            client_credential=self.config.credentials.client_secret,
+        )
+
+    @property
+    def client(self):
+        """Initializes and returns a GraphClient instance."""
+        if not self.config:
+            raise ValueError("Configuration is missing; cannot create the Office365 graph client.")
+        if not self._client:
+            self._client = GraphClient(self._get_access_token)
+        return self._client
+
+    def _get_access_token(self):
+        """Retrieves an access token for SharePoint access."""
+        scope = ["https://graph.microsoft.com/.default"]
+        refresh_token = self.config.credentials.refresh_token if hasattr(self.config.credentials, "refresh_token") else None
+
+        if refresh_token:
+            result = self._msal_app.acquire_token_by_refresh_token(refresh_token, scopes=scope)
+        else:
+            result = self._msal_app.acquire_token_for_client(scopes=scope)
+
+        if "access_token" not in result:
+            error_description = result.get("error_description", "No error description provided.")
+            message = f"Failed to acquire access token. Error: {result.get('error')}. Error description: {error_description}."
+            raise AirbyteTracedException(message=message, failure_type=FailureType.config_error)
+
+        return result
+
+
+class SourceMicrosoftSharePointStreamReader(AbstractFileBasedStreamReader):
+    """
+    A stream reader for Microsoft SharePoint. Handles file enumeration and reading from SharePoint.
+    """
+
+    ROOT_PATH = [".", "/"]
+
+    def __init__(self):
+        super().__init__()
+        self._auth_client = None
+        self._one_drive_client = None
+
+    @property
+    def config(self) -> SourceMicrosoftSharePointSpec:
+        return self._config
+
+    @property
+    def auth_client(self):
+        # Lazy initialization of the auth_client
+        if self._auth_client is None:
+            self._auth_client = SourceMicrosoftSharePointClient(self._config)
+        return self._auth_client
+
+    @property
+    def one_drive_client(self):
+        # Lazy initialization of the one_drive_client
+        if self._one_drive_client is None:
+            self._one_drive_client = self.auth_client.client
+        return self._one_drive_client
+
+    def get_access_token(self):
+        # Directly fetch a new access token from the auth_client each time it's called
+        return self.auth_client._get_access_token()["access_token"]
+
+    @config.setter
+    def config(self, value: SourceMicrosoftSharePointSpec):
+        """
+        The FileBasedSource reads and parses configuration from a file, then sets this configuration in its StreamReader. While it only
+        uses keys from its abstract configuration, concrete StreamReader implementations may need additional keys for third-party
+        authentication. Therefore, subclasses of AbstractFileBasedStreamReader should verify that the value in their config setter
+        matches the expected config type for their StreamReader.
+        """
+        assert isinstance(value, SourceMicrosoftSharePointSpec)
+        self._config = value
+
+    def _get_shared_drive_object(self, drive_id: str, object_id: str, path: str) -> List[Tuple[str, str, datetime]]:
+        """
+        Retrieves a list of all nested files under the specified object.
+
+        Args:
+            drive_id: The ID of the drive containing the object.
+            object_id: The ID of the object to start the search from.
+
+        Returns:
+            A list of tuples containing file information (name, download URL, and last modified datetime).
+
+        Raises:
+            RuntimeError: If an error occurs during the request.
+        """
+
+        access_token = self.get_access_token()
+        headers = {"Authorization": f"Bearer {access_token}"}
+        base_url = f"https://graph.microsoft.com/v1.0/drives/{drive_id}"
+
+        def get_files(url: str, path: str) -> List[Tuple[str, str, datetime]]:
+            response = requests.get(url, headers=headers)
+            if response.status_code != 200:
+                error_info = response.json().get("error", {}).get("message", "No additional error information provided.")
+                raise RuntimeError(f"Failed to retrieve files from URL '{url}'. HTTP status: {response.status_code}. Error: {error_info}")
+
+            data = response.json()
+            for child in data.get("value", []):
+                new_path = path + "/" + child["name"]
+                if child.get("file"):  # Object is a file
+                    last_modified = datetime.strptime(child["lastModifiedDateTime"], "%Y-%m-%dT%H:%M:%SZ")
+                    yield (new_path, child["@microsoft.graph.downloadUrl"], last_modified)
+                else:  # Object is a folder, retrieve children
+                    child_url = f"{base_url}/items/{child['id']}/children"  # Use item endpoint for nested objects
+                    yield from get_files(child_url, new_path)
+            yield from []
+
+        # Initial request to item endpoint
+        item_url = f"{base_url}/items/{object_id}"
+        item_response = requests.get(item_url, headers=headers)
+        if item_response.status_code != 200:
+            error_info = item_response.json().get("error", {}).get("message", "No additional error information provided.")
+            raise RuntimeError(
+                f"Failed to retrieve the initial shared object with ID '{object_id}' from drive '{drive_id}'. "
+                f"HTTP status: {item_response.status_code}. Error: {error_info}"
+            )
+
+        # Check if the object is a file or a folder
+        item_data = item_response.json()
+        if item_data.get("file"):  # Initial object is a file
+            new_path = path + "/" + item_data["name"]
+            last_modified = datetime.strptime(item_data["lastModifiedDateTime"], "%Y-%m-%dT%H:%M:%SZ")
+            yield (new_path, item_data["@microsoft.graph.downloadUrl"], last_modified)
+        else:
+            # Initial object is a folder, start file retrieval
+            yield from get_files(f"{item_url}/children", path)
+
+    def _list_directories_and_files(self, root_folder, path):
+        """Enumerates folders and files starting from a root folder."""
+        drive_items = execute_query_with_retry(root_folder.children.get())
+        for item in drive_items:
+            item_path = path + "/" + item.name if path else item.name
+            if item.is_file:
+                yield (item_path, item.properties["@microsoft.graph.downloadUrl"], item.properties["lastModifiedDateTime"])
+            else:
+                yield from self._list_directories_and_files(item, item_path)
+        yield from []
+
+    def _get_files_by_drive_name(self, drives, folder_path):
+        """Yields files from the specified drive."""
+        path_levels = [level for level in folder_path.split("/") if level]
+        folder_path = "/".join(path_levels)
+
+        for drive in drives:
+            is_sharepoint = drive.drive_type == "documentLibrary"
+            if is_sharepoint:
+                # Define base path for drive files to differentiate files between drives
+                if folder_path in self.ROOT_PATH:
+                    folder = drive.root
+                    folder_path_url = drive.web_url
+                else:
+                    folder = execute_query_with_retry(drive.root.get_by_path(folder_path).get())
+                    folder_path_url = drive.web_url + "/" + folder_path
+
+                yield from self._list_directories_and_files(folder, folder_path_url)
+
+    @property
+    @lru_cache(maxsize=None)
+    def drives(self):
+        """
+        Retrieves and caches SharePoint drives, including the user's drive based on authentication type.
+        """
+        drives = execute_query_with_retry(self.one_drive_client.drives.get())
+
+        if self.config.credentials.auth_type == "Client":
+            my_drive = execute_query_with_retry(self.one_drive_client.me.drive.get())
+        else:
+            my_drive = execute_query_with_retry(
+                self.one_drive_client.users.get_by_principal_name(self.config.credentials.user_principal_name).drive.get()
+            )
+
+        drives.add_child(my_drive)
+
+        return drives
+
+    def _get_shared_files_from_all_drives(self, parsed_drives):
+        drive_ids = [drive.id for drive in parsed_drives]
+
+        shared_drive_items = execute_query_with_retry(self.one_drive_client.me.drive.shared_with_me())
+        for drive_item in shared_drive_items:
+            parent_reference = drive_item.remote_item.parentReference
+
+            # check if drive is already parsed
+            if parent_reference and parent_reference["driveId"] not in drive_ids:
+                yield from self._get_shared_drive_object(parent_reference["driveId"], drive_item.id, drive_item.web_url)
+
+    def get_all_files(self):
+        if self.config.search_scope in ("ACCESSIBLE_DRIVES", "ALL"):
+            # Get files from accessible drives
+            yield from self._get_files_by_drive_name(self.drives, self.config.folder_path)
+
+        if self.config.search_scope in ("SHARED_ITEMS", "ALL"):
+            parsed_drives = [] if self.config.search_scope == "SHARED_ITEMS" else self.drives
+
+            # Get files from shared items
+            yield from self._get_shared_files_from_all_drives(parsed_drives)
+
+    def get_matching_files(self, globs: List[str], prefix: Optional[str], logger: logging.Logger) -> Iterable[RemoteFile]:
+        """
+        Retrieve all files matching the specified glob patterns in SharePoint.
+        """
+        files = self.get_all_files()
+
+        files_generator = filter_http_urls(
+            self.filter_files_by_globs_and_start_date(
+                [
+                    MicrosoftSharePointRemoteFile(
+                        uri=path,
+                        download_url=download_url,
+                        last_modified=last_modified,
+                    )
+                    for path, download_url, last_modified in files
+                ],
+                globs,
+            ),
+            logger,
+        )
+
+        items_processed = False
+        for file in files_generator:
+            items_processed = True
+            yield file
+
+        if not items_processed:
+            raise AirbyteTracedException(
+                message=f"Drive is empty or does not exist.",
+                failure_type=FailureType.config_error,
+            )
+
+    def open_file(self, file: RemoteFile, mode: FileReadMode, encoding: Optional[str], logger: logging.Logger) -> IOBase:
+        # choose correct compression mode because the url is random and doesn't end with filename extension
+        file_extension = file.uri.split(".")[-1]
+        if file_extension in ["gz", "bz2"]:
+            compression = "." + file_extension
+        else:
+            compression = "disable"
+
+        try:
+            return smart_open.open(file.download_url, mode=mode.value, compression=compression, encoding=encoding)
+        except Exception as e:
+            logger.exception(f"Error opening file {file.uri}: {e}")

source_microsoft_sharepoint/utils.py

@@ -0,0 +1,85 @@
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+
+import time
+from datetime import datetime, timedelta
+from enum import Enum
+from http import HTTPStatus
+
+from airbyte_cdk.models import FailureType
+from airbyte_cdk.sources.file_based.remote_file import RemoteFile
+from airbyte_cdk.utils.traced_exception import AirbyteTracedException
+
+
+class SearchScope(Enum):
+    OWN_DRIVES = "OWN_DRIVES"
+    SHARED_ITEMS = "SHARED_ITEMS"
+    BOTH = "BOTH"
+
+
+class MicrosoftSharePointRemoteFile(RemoteFile):
+    download_url: str
+
+
+def filter_http_urls(files, logger):
+    for file in files:
+        if file.download_url.startswith("http") and not file.download_url.startswith("https"):  # ignore-https-check
+            logger.error(f"Cannot open file {file.uri}. The URL returned by SharePoint is not secure.")
+        else:
+            yield file
+
+
+def execute_query_with_retry(obj, max_retries=5, initial_retry_after=5, max_retry_after=300, max_total_wait_time=600):
+    """
+    Executes a query with retry logic on encountering specific HTTP errors.
+
+    This function attempts to execute `obj.execute_query()` method, applying exponential backoff
+    retry logic for HTTP status codes 429 (Too Many Requests) and 503 (Service Unavailable). It
+    respects the 'Retry-After' header from the response, if present.
+
+    Parameters:
+        obj (object): The object that has the `execute_query` method to be executed.
+        max_retries (int): Maximum number of retry attempts. Defaults to 5.
+        initial_retry_after (int): Initial waiting time (in seconds) before the first retry. Defaults to 5 seconds.
+        max_retry_after (int): Maximum waiting time (in seconds) between retries. Defaults to 300 seconds.
+        max_total_wait_time (int): Maximum total waiting time (in seconds) for all retries. Defaults to 600 seconds.
+
+    Raises:
+        AirbyteTracedException: If the maximum total wait time or the maximum number of retries is exceeded.
+
+    Returns:
+        The result of `obj.execute_query()` if successful within the retry constraints.
+    """
+    retries = 0
+    start_time = datetime.now()
+    retry_after = initial_retry_after
+
+    while retries < max_retries:
+        try:
+            return obj.execute_query()
+        except Exception as ex:
+            if hasattr(ex, "response") and ex.response.status_code in (HTTPStatus.TOO_MANY_REQUESTS, HTTPStatus.SERVICE_UNAVAILABLE):
+                current_time = datetime.now()
+                elapsed_time = (current_time - start_time).total_seconds()
+
+                retry_after_header = ex.response.headers.get("Retry-After", None)
+                if retry_after_header:
+                    retry_after = int(retry_after_header)
+
+                if elapsed_time + retry_after > max_total_wait_time:
+                    message = (
+                        f"Maximum total wait time of {max_total_wait_time} seconds exceeded for execute_query. "
+                        f"The latest response status code is {ex.response.status_code}."
+                    )
+                    if retry_after_header:
+                        message += f" Retry-After header: {retry_after_header}"
+                    raise AirbyteTracedException(message, message, failure_type=FailureType.system_error)
+
+                time.sleep(retry_after)
+                retries += 1
+                retry_after = min(retry_after * 2, max_retry_after)  # Double the wait time for next retry, up to a max limit
+            else:
+                # Re-raise exceptions that are not related to rate limits or service availability
+                raise AirbyteTracedException.from_exception(ex, message="Caught unexpected exception")
+
+    message = f"Maximum number of retries of {max_retries} exceeded for execute_query."
+    raise AirbyteTracedException(message, message, failure_type=FailureType.system_error)