airbyte-source-box-data-extract 0.1.7__tar.gz

airbyte_source_box_data_extract-0.1.7/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.3
+Name: airbyte-source-box-data-extract
+Version: 0.1.7
+Summary: Source implementation for Box File Text Extraction.
+License: MIT
+Author: Airbyte
+Author-email: contact@airbyte.io
+Requires-Python: >=3.11,<3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: airbyte-cdk (>=6,<7)
+Requires-Dist: box-sdk-gen (>=1.11.1,<2.0.0)
+Project-URL: Documentation, https://docs.airbyte.com/integrations/sources/box-files-text
+Project-URL: Homepage, https://airbyte.com
+Project-URL: Repository, https://github.com/airbytehq/airbyte
+Description-Content-Type: text/markdown
+
+# Box Data Extract Source
+
+This is the repository for the Box Generated source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/box-data-extract).
+
+This connector uses Box AI to extract data directly from documents stored in Box.
+
+For example, a company managing lease contracts can automatically capture key details and populate their system of record. Likewise, a financial institution can extract critical data from loan application documents—such as bank statements and W-2s—and integrate it into approval workflows.
+
+By prioritizing content intelligence, this connector unlocks new opportunities for automation and AI-driven insights.
+
+## Local development
+
+### Prerequisites
+
+* Python (`^3.11`)
+* Poetry (`^1.8`) - installation instructions [here](https://python-poetry.org/docs/#installation)
+
+
+
+### Installing the connector
+
+From this connector directory, run:
+```bash
+poetry install --with dev
+```
+
+
+### Create credentials
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/box-data-extract)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `src/source_box_data_extract/spec.yaml` file.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
+See `sample_files/sample_config.json` for a sample config file.
+
+
+### Locally running the connector
+
+```
+poetry run source-box-data-extract spec
+poetry run source-box-data-extract check --config secrets/config.json
+poetry run source-box-data-extract discover --config secrets/config.json
+poetry run source-box-data-extract read --config secrets/config.json --catalog sample_files/configured_catalog.json
+```
+
+### Running tests
+
+To run tests locally, from the connector directory run:
+
+```
+poetry run pytest tests
+```
+
+### Building the docker image
+
+1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
+2. Run the following command to build the docker image:
+```bash
+airbyte-ci connectors --name=source-box-data-extract build
+```
+
+An image will be available on your host with the tag `airbyte/source-box-data-extract:dev`.
+
+
+### Running as a docker container
+
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-box-data-extract:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-box-data-extract:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-box-data-extract:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-box-data-extract:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+### Running our CI test suite
+
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+
+```bash
+airbyte-ci connectors --name=source-box-data-extract test
+```
+
+### Customizing acceptance Tests
+
+Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+### Dependency Management
+
+All of your dependencies should be managed via Poetry. 
+To add a new dependency, run:
+
+```bash
+poetry add <package-name>
+```
+
+Please commit the changes to `pyproject.toml` and `poetry.lock` files.
+
+## Publishing a new version of the connector
+
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-box-data-extract test`
+2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): 
+    - bump the `dockerImageTag` value in in `metadata.yaml`
+    - bump the `version` value in `pyproject.toml`
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/box-data-extract.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

airbyte_source_box_data_extract-0.1.7/README.md

@@ -0,0 +1,111 @@
+# Box Data Extract Source
+
+This is the repository for the Box Generated source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/box-data-extract).
+
+This connector uses Box AI to extract data directly from documents stored in Box.
+
+For example, a company managing lease contracts can automatically capture key details and populate their system of record. Likewise, a financial institution can extract critical data from loan application documents—such as bank statements and W-2s—and integrate it into approval workflows.
+
+By prioritizing content intelligence, this connector unlocks new opportunities for automation and AI-driven insights.
+
+## Local development
+
+### Prerequisites
+
+* Python (`^3.11`)
+* Poetry (`^1.8`) - installation instructions [here](https://python-poetry.org/docs/#installation)
+
+
+
+### Installing the connector
+
+From this connector directory, run:
+```bash
+poetry install --with dev
+```
+
+
+### Create credentials
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/box-data-extract)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `src/source_box_data_extract/spec.yaml` file.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
+See `sample_files/sample_config.json` for a sample config file.
+
+
+### Locally running the connector
+
+```
+poetry run source-box-data-extract spec
+poetry run source-box-data-extract check --config secrets/config.json
+poetry run source-box-data-extract discover --config secrets/config.json
+poetry run source-box-data-extract read --config secrets/config.json --catalog sample_files/configured_catalog.json
+```
+
+### Running tests
+
+To run tests locally, from the connector directory run:
+
+```
+poetry run pytest tests
+```
+
+### Building the docker image
+
+1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
+2. Run the following command to build the docker image:
+```bash
+airbyte-ci connectors --name=source-box-data-extract build
+```
+
+An image will be available on your host with the tag `airbyte/source-box-data-extract:dev`.
+
+
+### Running as a docker container
+
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-box-data-extract:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-box-data-extract:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-box-data-extract:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-box-data-extract:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+### Running our CI test suite
+
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+
+```bash
+airbyte-ci connectors --name=source-box-data-extract test
+```
+
+### Customizing acceptance Tests
+
+Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+### Dependency Management
+
+All of your dependencies should be managed via Poetry. 
+To add a new dependency, run:
+
+```bash
+poetry add <package-name>
+```
+
+Please commit the changes to `pyproject.toml` and `poetry.lock` files.
+
+## Publishing a new version of the connector
+
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-box-data-extract test`
+2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): 
+    - bump the `dockerImageTag` value in in `metadata.yaml`
+    - bump the `version` value in `pyproject.toml`
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/box-data-extract.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

airbyte_source_box_data_extract-0.1.7/main.py

@@ -0,0 +1,9 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+from source_box_data_extract.run import run
+
+
+if __name__ == "__main__":
+    run()

airbyte_source_box_data_extract-0.1.7/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = [
+    "poetry-core>=1.0.0",
+]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry]
+version = "0.1.7"
+name = "airbyte-source-box-data-extract"
+description = "Source implementation for Box File Text Extraction."
+authors = [
+    "Airbyte <contact@airbyte.io>",
+]
+license = "MIT"
+readme = "README.md"
+documentation = "https://docs.airbyte.com/integrations/sources/box-files-text"
+homepage = "https://airbyte.com"
+repository = "https://github.com/airbytehq/airbyte"
+packages = [
+    { include = "source_box_data_extract" },
+    { include = "main.py" },
+]
+
+[tool.poetry.dependencies]
+python = "^3.11,<3.12"
+airbyte-cdk = "^6"
+box-sdk-gen = "^1.11.1"
+
+[tool.poetry.scripts]
+source-box-data-extract = "source_box_data_extract.run:run"
+
+[tool.poetry.group.dev.dependencies]
+requests-mock = "*"
+pytest-mock = "*"
+pytest = "*"

airbyte_source_box_data_extract-0.1.7/source_box_data_extract/__init__.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+
+from .source import SourceBoxDataExtract
+
+__all__ = ["SourceBoxDataExtract"]

airbyte_source_box_data_extract-0.1.7/source_box_data_extract/box_api.py

@@ -0,0 +1,248 @@
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+
+import json
+import logging
+from dataclasses import dataclass
+from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple, Union
+
+import requests
+from box_sdk_gen import (
+    AiAsk,
+    AiExtractResponse,
+    AiItemBase,
+    AiItemBaseTypeField,
+    AiResponseFull,
+    BoxCCGAuth,
+    BoxClient,
+    BoxSDKError,
+    CCGConfig,
+    CreateAiAskMode,
+    CreateAiExtractStructuredFields,
+    CreateAiExtractStructuredFieldsOptionsField,
+    File,
+)
+from requests_cache import Dict
+
+
+logger = logging.getLogger("airbyte")
+
+
+@dataclass
+class BoxFileExtended:
+    file: File
+    text_representation: str
+
+
+def get_box_ccg_client(config: Mapping[str, Any]) -> BoxClient:
+    client_id = config["client_id"]
+    client_secret = config["client_secret"]
+    box_subject_type = config["box_subject_type"]
+    box_subject_id = config["box_subject_id"]
+
+    if box_subject_type == "enterprise":
+        enterprise_id = box_subject_id
+        user_id = None
+    else:
+        enterprise_id = None
+        user_id = box_subject_id
+    ccg_config = CCGConfig(client_id=client_id, client_secret=client_secret, enterprise_id=enterprise_id, user_id=user_id)
+    ccg_auth = BoxCCGAuth(ccg_config)
+    return add_extra_header_to_box_client(BoxClient(ccg_auth))
+
+
+def add_extra_header_to_box_client(box_client: BoxClient) -> BoxClient:
+    """
+    Add extra headers to the Box client.
+
+    Args:
+        box_client (BoxClient): A Box client object.
+        header (Dict[str, str]): A dictionary of extra headers to add to the Box client.
+
+    Returns:
+        BoxClient: A Box client object with the extra headers added.
+    """
+    header = {"x-box-ai-library": "airbyte"}
+    return box_client.with_extra_headers(extra_headers=header)
+
+
+def _do_request(box_client: BoxClient, url: str):
+    """
+    Performs a GET request to a Box API endpoint using the provided Box client.
+
+    This is an internal helper function and should not be called directly.
+
+    Args:
+        box_client (BoxClient): An authenticated Box client object.
+        url (str): The URL of the Box API endpoint to make the request to.
+
+    Returns:
+        bytes: The content of the response from the Box API.
+
+    Raises:
+        BoxSDKError: If an error occurs while retrieving the access token.
+        requests.exceptions.RequestException: If the request fails (e.g., network error,
+                                             4XX or 5XX status code).
+    """
+    try:
+        access_token = box_client.auth.retrieve_token().access_token
+    except BoxSDKError as e:
+        raise
+
+    resp = requests.get(url, headers={"Authorization": f"Bearer {access_token}"})
+    resp.raise_for_status()
+    return resp.content
+
+
+def box_file_get_by_id(client: BoxClient, file_id: str) -> File:
+    return client.files.get_file_by_id(file_id=file_id)
+
+
+def box_file_text_extract(client: BoxClient, file_id: str) -> str:
+    # Request the file with the "extracted_text" representation hint
+    file_text_representation = client.files.get_file_by_id(
+        file_id,
+        x_rep_hints="[extracted_text]",
+        fields=["name", "representations"],
+    )
+    # Check if any representations exist
+    if not file_text_representation.representations.entries:
+        logger.debug(f"No representation for file {file_text_representation.id}")
+        return ""
+
+    # Find the "extracted_text" representation
+    extracted_text_entry = next(
+        (entry for entry in file_text_representation.representations.entries if entry.representation == "extracted_text"),
+        None,
+    )
+    if not extracted_text_entry:
+        return ""
+
+    # Handle cases where the extracted text needs generation
+    if extracted_text_entry.status.state == "none":
+        _do_request(extracted_text_entry.info.url)  # Trigger text generation
+
+    # Construct the download URL and sanitize filename
+    url = extracted_text_entry.content.url_template.replace("{+asset_path}", "")
+
+    # Download and truncate the raw content
+    raw_content = _do_request(client, url)
+
+    # check to see if rawcontent is bytes
+    if isinstance(raw_content, bytes):
+        return raw_content.decode("utf-8")
+    else:
+        return raw_content
+
+
+def box_file_ai_ask(client: BoxClient, file_id: str, prompt: str) -> str:
+    mode = CreateAiAskMode.SINGLE_ITEM_QA
+    ai_item = AiItemBase(id=file_id, type=AiItemBaseTypeField.FILE)
+    response = client.ai.create_ai_ask(mode=mode, prompt=prompt, items=[ai_item])
+    return response.answer
+
+
+def box_file_ai_extract(client: BoxClient, file_id: str, prompt: str) -> str:
+    ai_item = AiItemBase(id=file_id, type=AiItemBaseTypeField.FILE)
+    response = client.ai.create_ai_extract(prompt=prompt, items=[ai_item])
+    return response.answer
+
+
+def box_file_ai_extract_structured(client: BoxClient, file_id: str, fields_json_str: str) -> str:
+    ai_item = AiItemBase(id=file_id, type=AiItemBaseTypeField.FILE)
+    fields_list = json.loads(fields_json_str)
+    ai_fields = []
+    options = []
+    for field in fields_list:
+        field_options = field.get("options")
+        if field_options is not None:
+            for option in field.get("options"):
+                options.append(CreateAiExtractStructuredFieldsOptionsField(key=option.get("key")))
+
+        ai_fields.append(
+            CreateAiExtractStructuredFields(
+                key=field.get("key"),
+                description=field.get("description"),
+                display_name=field.get("display_name"),
+                prompt=field.get("prompt"),
+                type=field.get("type"),
+                options=options if options is not None and len(options) > 0 else None,
+            )
+        )
+    response: AiExtractResponse = client.ai.create_ai_extract_structured(items=[ai_item], fields=ai_fields)
+    return json.dumps(response.to_dict(), indent=2)
+
+
+def box_folder_text_representation(
+    client: BoxClient, folder_id: str, is_recursive: bool = False, by_pass_text_extraction: bool = False
+) -> Iterable[BoxFileExtended]:
+    # folder items iterator
+    for item in client.folders.get_folder_items(folder_id).entries:
+        if item.type == "file":
+            file = box_file_get_by_id(client=client, file_id=item.id)
+            if not by_pass_text_extraction:
+                text_representation = box_file_text_extract(client=client, file_id=item.id)
+            else:
+                text_representation = ""
+            yield BoxFileExtended(file=file, text_representation=text_representation)
+        elif item.type == "folder" and is_recursive:
+            yield from box_folder_text_representation(
+                client=client, folder_id=item.id, is_recursive=is_recursive, by_pass_text_extraction=by_pass_text_extraction
+            )
+
+
+def box_folder_ai_ask(
+    client: BoxClient, folder_id: str, prompt: str, is_recursive: bool = False, by_pass_text_extraction: bool = False
+) -> Iterable[BoxFileExtended]:
+    # folder items iterator
+    for item in client.folders.get_folder_items(folder_id).entries:
+        if item.type == "file":
+            file = box_file_get_by_id(client=client, file_id=item.id)
+            if not by_pass_text_extraction:
+                text_representation = box_file_ai_ask(client=client, file_id=item.id, prompt=prompt)
+            else:
+                text_representation = ""
+            yield BoxFileExtended(file=file, text_representation=text_representation)
+        elif item.type == "folder" and is_recursive:
+            yield from box_folder_ai_ask(
+                client=client, folder_id=item.id, prompt=prompt, is_recursive=is_recursive, by_pass_text_extraction=by_pass_text_extraction
+            )
+
+
+def box_folder_ai_extract(
+    client: BoxClient, folder_id: str, prompt: str, is_recursive: bool = False, by_pass_text_extraction: bool = False
+) -> Iterable[BoxFileExtended]:
+    # folder items iterator
+    for item in client.folders.get_folder_items(folder_id).entries:
+        if item.type == "file":
+            file = box_file_get_by_id(client=client, file_id=item.id)
+            if not by_pass_text_extraction:
+                text_representation = box_file_ai_extract(client=client, file_id=item.id, prompt=prompt)
+            else:
+                text_representation = ""
+            yield BoxFileExtended(file=file, text_representation=text_representation)
+        elif item.type == "folder" and is_recursive:
+            yield from box_folder_ai_extract(
+                client=client, folder_id=item.id, prompt=prompt, is_recursive=is_recursive, by_pass_text_extraction=by_pass_text_extraction
+            )
+
+
+def box_folder_ai_extract_structured(
+    client: BoxClient, folder_id: str, fields_json_str: str, is_recursive: bool = False, by_pass_text_extraction: bool = False
+) -> Iterable[BoxFileExtended]:
+    # folder items iterator
+    for item in client.folders.get_folder_items(folder_id).entries:
+        if item.type == "file":
+            file = box_file_get_by_id(client=client, file_id=item.id)
+            if not by_pass_text_extraction:
+                text_representation = box_file_ai_extract_structured(client=client, file_id=item.id, fields_json_str=fields_json_str)
+            else:
+                text_representation = ""
+            yield BoxFileExtended(file=file, text_representation=text_representation)
+        elif item.type == "folder" and is_recursive:
+            yield from box_folder_ai_extract_structured(
+                client=client,
+                folder_id=item.id,
+                fields_json_str=fields_json_str,
+                is_recursive=is_recursive,
+                by_pass_text_extraction=by_pass_text_extraction,
+            )

airbyte_source_box_data_extract-0.1.7/source_box_data_extract/run.py

@@ -0,0 +1,15 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+
+import sys
+
+from airbyte_cdk.entrypoint import launch
+
+from .source import SourceBoxDataExtract
+
+
+def run():
+    source = SourceBoxDataExtract()
+    launch(source, sys.argv[1:])

airbyte_source_box_data_extract-0.1.7/source_box_data_extract/schemas.py

@@ -0,0 +1,235 @@
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+
+import json
+from typing import Dict
+
+
+def get_generic_json_schema() -> Dict:
+    generic_schema = """
+        {
+        "$schema": "http://json-schema.org/draft-07/schema#",
+        "type": "object",
+        "properties": {
+            "type": {
+            "type": "string"
+            },
+            "id": {
+            "type": "string"
+            },
+            "file_version": {
+            "type": "object",
+            "properties": {
+                "type": {
+                "type": "string"
+                },
+                "id": {
+                "type": "string"
+                },
+                "sha1": {
+                "type": "string"
+                }
+            },
+            "required": ["type", "id", "sha1"]
+            },
+            "sequence_id": {
+            "type": "string"
+            },
+            "etag": {
+            "type": "string"
+            },
+            "sha1": {
+            "type": "string"
+            },
+            "name": {
+            "type": "string"
+            },
+            "description": {
+            "type": "string"
+            },
+            "size": {
+            "type": "integer"
+            },
+            "path_collection": {
+            "type": "object",
+            "properties": {
+                "total_count": {
+                "type": "integer"
+                },
+                "entries": {
+                "type": "array",
+                "items": [
+                    {
+                    "type": "object",
+                    "properties": {
+                        "type": {
+                        "type": "string"
+                        },
+                        "id": {
+                        "type": "string"
+                        },
+                        "sequence_id": {
+                        "type": "null"
+                        },
+                        "etag": {
+                        "type": "null"
+                        },
+                        "name": {
+                        "type": "string"
+                        }
+                    },
+                    "required": ["type", "id", "sequence_id", "etag", "name"]
+                    },
+                    {
+                    "type": "object",
+                    "properties": {
+                        "type": {
+                        "type": "string"
+                        },
+                        "id": {
+                        "type": "string"
+                        },
+                        "sequence_id": {
+                        "type": "string"
+                        },
+                        "etag": {
+                        "type": "string"
+                        },
+                        "name": {
+                        "type": "string"
+                        }
+                    },
+                    "required": ["type", "id", "sequence_id", "etag", "name"]
+                    },
+                    {
+                    "type": "object",
+                    "properties": {
+                        "type": {
+                        "type": "string"
+                        },
+                        "id": {
+                        "type": "string"
+                        },
+                        "sequence_id": {
+                        "type": "string"
+                        },
+                        "etag": {
+                        "type": "string"
+                        },
+                        "name": {
+                        "type": "string"
+                        }
+                    },
+                    "required": ["type", "id", "sequence_id", "etag", "name"]
+                    }
+                ]
+                }
+            },
+            "required": ["total_count", "entries"]
+            },
+            "created_at": {
+            "type": "string"
+            },
+            "modified_at": {
+            "type": "string"
+            },
+            "trashed_at": {
+            "type": "null"
+            },
+            "purged_at": {
+            "type": "null"
+            },
+            "content_created_at": {
+            "type": "string"
+            },
+            "content_modified_at": {
+            "type": "string"
+            },
+            "created_by": {
+            "type": "object",
+            "properties": {
+                "type": {
+                "type": "string"
+                },
+                "id": {
+                "type": "string"
+                },
+                "name": {
+                "type": "string"
+                },
+                "login": {
+                "type": "string"
+                }
+            },
+            "required": ["type", "id", "name", "login"]
+            },
+            "modified_by": {
+            "type": "object",
+            "properties": {
+                "type": {
+                "type": "string"
+                },
+                "id": {
+                "type": "string"
+                },
+                "name": {
+                "type": "string"
+                },
+                "login": {
+                "type": "string"
+                }
+            },
+            "required": ["type", "id", "name", "login"]
+            },
+            "owned_by": {
+            "type": "object",
+            "properties": {
+                "type": {
+                "type": "string"
+                },
+                "id": {
+                "type": "string"
+                },
+                "name": {
+                "type": "string"
+                },
+                "login": {
+                "type": "string"
+                }
+            },
+            "required": ["type", "id", "name", "login"]
+            },
+            "shared_link": {
+            "type": "null"
+            },
+            "parent": {
+            "type": "object",
+            "properties": {
+                "type": {
+                "type": "string"
+                },
+                "id": {
+                "type": "string"
+                },
+                "sequence_id": {
+                "type": "string"
+                },
+                "etag": {
+                "type": "string"
+                },
+                "name": {
+                "type": "string"
+                }
+            },
+            "required": ["type", "id", "sequence_id", "etag", "name"]
+            },
+            "item_status": {
+            "type": "string"
+            },
+            "text_representation": {
+            "type": "string"
+            }
+        }
+        }
+
+        """
+    return json.loads(generic_schema)

airbyte_source_box_data_extract-0.1.7/source_box_data_extract/source.py

@@ -0,0 +1,264 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+
+import logging
+from typing import Any, Dict, Iterable, List, Mapping, MutableMapping, Optional, Tuple, Union
+
+import requests
+from box_sdk_gen import BoxAPIError, BoxClient, File, Folder, Items
+
+from airbyte_cdk.models import AirbyteMessage, AirbyteStream, ConfiguredAirbyteStream, SyncMode
+from airbyte_cdk.sources import AbstractSource
+from airbyte_cdk.sources.streams import Stream
+from airbyte_cdk.sources.streams.http import HttpStream
+from airbyte_cdk.sources.streams.http.requests_native_auth import TokenAuthenticator
+
+from .box_api import (
+    box_folder_ai_ask,
+    box_folder_ai_extract,
+    box_folder_ai_extract_structured,
+    box_folder_text_representation,
+    get_box_ccg_client,
+)
+from .schemas import get_generic_json_schema
+
+
+logger = logging.getLogger("airbyte")
+
+# A stream's read method can return one of the following types:
+# Mapping[str, Any]: The content of an AirbyteRecordMessage
+# AirbyteMessage: An AirbyteMessage. Could be of any type
+StreamData = Union[Mapping[str, Any], AirbyteMessage]
+"""
+TODO: Most comments in this class are instructive and should be deleted after the source is implemented.
+
+This file provides a stubbed example of how to use the Airbyte CDK to develop both a source connector which supports full refresh or and an
+incremental syncs from an HTTP API.
+
+The various TODOs are both implementation hints and steps - fulfilling all the TODOs should be sufficient to implement one basic and one incremental
+stream from a source. This pattern is the same one used by Airbyte internally to implement connectors.
+
+The approach here is not authoritative, and devs are free to use their own judgement.
+
+There are additional required TODOs in the files within the integration_tests folder and the spec.yaml file.
+"""
+
+
+# Source
+class SourceBoxDataExtract(AbstractSource):
+    def check_connection(self, logger, config) -> Tuple[bool, any]:
+        """
+        :param config:  the user-input config object conforming to the connector's spec.yaml
+        :param logger:  logger object
+        :return Tuple[bool, any]: (True, None) if the input config can be used to connect to the API successfully, (False, error) otherwise.
+        """
+        logger.info("Checking Box API connection...")
+        try:
+            box_client = get_box_ccg_client(config)
+            user = box_client.users.get_user_me()
+            logger.debug(f"box_subject_type: {config.get('box_subject_type')}, box_subject_id: {config.get('box_subject_id')}")
+            logger.info(f"Logged into Box as: {user.name} ({user.id} - {user.login})")
+        except BoxAPIError as e:
+            logger.error(f"Unable to connect to Box API with the provided credentials - {e}")
+            return False, f"Unable to connect to Box API with the provided credentials"
+        return True, None
+
+    def streams(self, config: Mapping[str, Any]) -> List[Stream]:
+        """
+        :param config: A Mapping of the user input configuration as defined in the connector spec.
+        """
+        box_client = get_box_ccg_client(config)
+        box_folder_text_representation_stream = StreamTextRepresentationFolder(
+            box_client, config["box_folder_id"], is_recursive=config.get("is_recursive", False)
+        )
+
+        box_folder_ask_ai_stream = StreamAIAskFolder(
+            box_client, config["box_folder_id"], config["ask_ai_prompt"], is_recursive=config.get("is_recursive", False)
+        )
+
+        box_folder_extract_ai_stream = StreamAIExtractFolder(
+            box_client, config["box_folder_id"], config["extract_ai_prompt"], is_recursive=config.get("is_recursive", False)
+        )
+
+        box_folder_extract_structured_ai_stream = StreamAIExtractStructuredFolder(
+            client=box_client,
+            folder_id=config["box_folder_id"],
+            fields_json_str=config["extract_structured_ai_fields"],
+            is_recursive=config.get("is_recursive", False),
+        )
+
+        return [
+            box_folder_text_representation_stream,
+            box_folder_ask_ai_stream,
+            box_folder_extract_ai_stream,
+            box_folder_extract_structured_ai_stream,
+        ]
+
+
+# Streams
+class StreamTextRepresentationFolder(Stream):
+    """
+    Represents a Box Data Text Representation Stream from a Box Folder.
+    params:
+        client: BoxClient - Box Client object
+        folder_id: str - Box Folder ID
+        is_recursive: bool - Whether to read the folder recursively
+    """
+
+    client: BoxClient = None
+    folder_id: str = None
+    is_recursive: bool = False
+
+    def __init__(self, client: BoxClient, folder_id: str, is_recursive: bool = False):
+        self.client = client
+        self.folder_id = folder_id
+        self.is_recursive = is_recursive
+
+    @property
+    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
+        """
+        :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields.
+          If the stream has no primary keys, return None.
+        """
+        return "id"
+
+    def get_json_schema(self):
+        return get_generic_json_schema()
+
+    def read_records(
+        self,
+        sync_mode: SyncMode,
+        cursor_field: Optional[List[str]] = None,
+        stream_slice: Optional[Mapping[str, Any]] = None,
+        stream_state: Optional[Mapping[str, Any]] = None,
+    ) -> Iterable[StreamData]:
+        logger.info(f"Extracting text representation for files in folder {self.folder_id} {'recursively' if self.is_recursive else ''}")
+        items = box_folder_text_representation(self.client, self.folder_id, is_recursive=self.is_recursive)
+        for item in items:
+            airbyte_item: StreamData = item.file.to_dict()
+            airbyte_item["text_representation"] = item.text_representation
+            logger.info(f"Reading file {item.file.id} - {item.file.name}")
+            yield airbyte_item
+
+
+class StreamAIAskFolder(Stream):
+    client: BoxClient = None
+    folder_id: str = None
+    is_recursive: bool = False
+    prompt: str = None
+
+    def __init__(self, client: BoxClient, folder_id: str, prompt: str, is_recursive: bool = False):
+        self.client = client
+        self.folder_id = folder_id
+        self.is_recursive = is_recursive
+        self.prompt = prompt
+
+    @property
+    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
+        """
+        :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields.
+          If the stream has no primary keys, return None.
+        """
+        return "id"
+
+    def get_json_schema(self):
+        return get_generic_json_schema()
+
+    def read_records(
+        self,
+        sync_mode: SyncMode,
+        cursor_field: Optional[List[str]] = None,
+        stream_slice: Optional[Mapping[str, Any]] = None,
+        stream_state: Optional[Mapping[str, Any]] = None,
+    ) -> Iterable[StreamData]:
+        logger.info(f"Asking AI {self.prompt} for all files in folder {self.folder_id} {'recursively' if self.is_recursive else ''}")
+        items = box_folder_ai_ask(self.client, self.folder_id, prompt=self.prompt, is_recursive=self.is_recursive)
+        for item in items:
+            airbyte_item: StreamData = item.file.to_dict()
+            airbyte_item["text_representation"] = item.text_representation
+            logger.info(f"Reading file {item.file.id} - {item.file.name}")
+            yield airbyte_item
+
+
+class StreamAIExtractFolder(Stream):
+    client: BoxClient = None
+    folder_id: str = None
+    is_recursive: bool = False
+    prompt: str = None
+
+    def __init__(self, client: BoxClient, folder_id: str, prompt: str, is_recursive: bool = False):
+        self.client = client
+        self.folder_id = folder_id
+        self.is_recursive = is_recursive
+        self.prompt = prompt
+
+    @property
+    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
+        """
+        :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields.
+          If the stream has no primary keys, return None.
+        """
+        return "id"
+
+    def get_json_schema(self):
+        return get_generic_json_schema()
+
+    def read_records(
+        self,
+        sync_mode: SyncMode,
+        cursor_field: Optional[List[str]] = None,
+        stream_slice: Optional[Mapping[str, Any]] = None,
+        stream_state: Optional[Mapping[str, Any]] = None,
+    ) -> Iterable[StreamData]:
+        logger.info(f"Extracting AI {self.prompt} for all files in folder {self.folder_id} {'recursively' if self.is_recursive else ''}")
+        items = box_folder_ai_extract(self.client, self.folder_id, prompt=self.prompt, is_recursive=self.is_recursive)
+        for item in items:
+            airbyte_item: StreamData = item.file.to_dict()
+            airbyte_item["text_representation"] = item.text_representation
+            logger.info(f"Reading file {item.file.id} - {item.file.name}")
+            yield airbyte_item
+
+
+class StreamAIExtractStructuredFolder(Stream):
+    client: BoxClient = None
+    folder_id: str = None
+    is_recursive: bool = False
+    fields_json_str: str = None
+
+    def __init__(self, client: BoxClient, folder_id: str, fields_json_str: str, is_recursive: bool = False):
+        self.client = client
+        self.folder_id = folder_id
+        self.is_recursive = is_recursive
+        self.fields_json_str = fields_json_str
+
+    @property
+    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
+        """
+        :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields.
+          If the stream has no primary keys, return None.
+        """
+        return "id"
+
+    def get_json_schema(self):
+        return get_generic_json_schema()
+
+    def read_records(
+        self,
+        sync_mode: SyncMode,
+        cursor_field: Optional[List[str]] = None,
+        stream_slice: Optional[Mapping[str, Any]] = None,
+        stream_state: Optional[Mapping[str, Any]] = None,
+    ) -> Iterable[StreamData]:
+        logger.info(
+            f"Extracting Struvctured AI {self.fields_json_str} for all files in folder {self.folder_id} {'recursively' if self.is_recursive else ''}"
+        )
+        items = box_folder_ai_extract_structured(
+            self.client, self.folder_id, fields_json_str=self.fields_json_str, is_recursive=self.is_recursive
+        )
+        for item in items:
+            airbyte_item: StreamData = item.file.to_dict()
+            airbyte_item["text_representation"] = item.text_representation
+            logger.info(f"Reading file {item.file.id} - {item.file.name}")
+            yield airbyte_item

airbyte_source_box_data_extract-0.1.7/source_box_data_extract/spec.yaml

@@ -0,0 +1,80 @@
+documentationUrl: https://docs.airbyte.com/integrations/sources/box-data-extract
+connectionSpecification:
+  $schema: http://json-schema.org/draft-07/schema#
+  title: Box File Text Extraction Spec
+  type: object
+  required:
+    - client_id
+    - client_secret
+    - box_subject_type
+    - box_subject_id
+    - box_folder_id
+    - is_recursive
+  properties:
+    client_id:
+      type: string
+      title: Client ID
+      description: >-
+        You Box App client ID.
+        Find yours in the <a href="https://app.box.com/developers/console">developer console</a>.
+      order: 0
+    client_secret:
+      type: string
+      title: Client Secret
+      description: >-
+        You Box App client secret.
+        Find yours in the <a href="https://app.box.com/developers/console">developer console</a>.
+      airbyte_secret: true
+      order: 1
+    box_subject_type:
+      type: string
+      enum:
+        - user
+        - enterprise
+      title: Box Subject Type
+      description: >-
+        Represents the type of user to login as ("user" or "enterprise").
+        Enterprise will login with the application service account.
+        User will login with the user if app can impersonate users.
+      order: 2
+    box_subject_id:
+      type: string
+      title: Box Subject ID
+      description: >-
+        If subject type is "enterprise", use your enterprise ID
+        If subject type is "user", use the user id to login as.
+      order: 3
+    box_folder_id:
+      type: string
+      title: Folder ID
+      description: >-
+        Folder to retreive data from.
+      order: 4
+    is_recursive:
+      type: boolean
+      title: Recursive
+      description: >-
+        Read the folders recursively.
+      order: 5
+    ask_ai_prompt:
+      type: string
+      always_show: true
+      title: Ask AI Prompt
+      description: >-
+        Prompt to use in Ask AI Stream.
+      order: 6
+    extract_ai_prompt:
+      type: string
+      always_show: true
+      title: Extract AI Prompt
+      description: >-
+        Prompt to use in Extract AI Stream.
+      order: 7
+    extract_structured_ai_fields:
+      type: string
+      always_show: true
+      multiline: true
+      title: Extract Structured AI Prompt
+      description: >-
+        Prompt to use in Extract Strctured AI Stream.
+      order: 8