airbyte-source-github 2.1.27__tar.gz → 2.1.29__tar.gz

airbyte_source_github-2.1.29/.airbyte-pypi-readme.md

@@ -0,0 +1,7 @@
+# GitHub
+
+This package contains the Airbyte GitHub connector.
+
+Airbyte is open-source data movement for ELT pipelines and AI agents. It moves data from APIs, databases, and files to warehouses, lakes, and AI applications.
+
+Airbyte provides a catalog of 600+ connectors for APIs, databases, data warehouses, data lakes, and AI applications. Use this package with [PyAirbyte](https://docs.airbyte.com/developers/pyairbyte) to run the connector from Python.

airbyte_source_github-2.1.29/PKG-INFO

@@ -0,0 +1,29 @@
+Metadata-Version: 2.1
+Name: airbyte-source-github
+Version: 2.1.29
+Summary: Source implementation for GitHub.
+Home-page: https://airbyte.com
+License: ELv2
+Author: Airbyte
+Author-email: contact@airbyte.io
+Requires-Python: >=3.10,<3.14
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: airbyte-cdk (>=7.12.0,<8.0.0)
+Requires-Dist: sgqlc (==16.3)
+Project-URL: Documentation, https://docs.airbyte.com/integrations/sources/github
+Project-URL: Repository, https://github.com/airbytehq/airbyte
+Description-Content-Type: text/markdown
+
+# GitHub
+
+This package contains the Airbyte GitHub connector.
+
+Airbyte is open-source data movement for ELT pipelines and AI agents. It moves data from APIs, databases, and files to warehouses, lakes, and AI applications.
+
+Airbyte provides a catalog of 600+ connectors for APIs, databases, data warehouses, data lakes, and AI applications. Use this package with [PyAirbyte](https://docs.airbyte.com/developers/pyairbyte) to run the connector from Python.
+

{airbyte_source_github-2.1.27 → airbyte_source_github-2.1.29}/pyproject.toml

@@ -3,12 +3,12 @@ requires = [ "poetry-core>=1.0.0",]
 build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
-version = "2.1.27"
+version = "2.1.29"
 name = "airbyte-source-github"
 description = "Source implementation for GitHub."
 authors = [ "Airbyte <contact@airbyte.io>",]
 license = "ELv2"
-readme = "README.md"
+readme = ".airbyte-pypi-readme.md"
 documentation = "https://docs.airbyte.com/integrations/sources/github"
 homepage = "https://airbyte.com"
 repository = "https://github.com/airbytehq/airbyte"

{airbyte_source_github-2.1.27 → airbyte_source_github-2.1.29}/source_github/errors_handlers.py

@@ -199,11 +199,25 @@ class GitHubGraphQLErrorHandler(GithubStreamABCErrorHandler):
     def interpret_response(self, response_or_exception: Optional[Union[requests.Response, Exception]] = None) -> ErrorResolution:
         if isinstance(response_or_exception, requests.Response):
             if response_or_exception.status_code in (requests.codes.BAD_GATEWAY, requests.codes.GATEWAY_TIMEOUT):
-                self.stream.page_size = int(self.stream.page_size / 2)
+                # Halve the page size on every 502/504 to reduce GraphQL query cost,
+                # but never let it drop below 1 — a page_size of 0 would request no
+                # records and cause infinite paging.
+                previous_page_size = self.stream.page_size
+                self.stream.page_size = max(1, int(self.stream.page_size / 2))
+                self._logger.info(
+                    "GitHub GraphQL endpoint returned HTTP %s for stream `%s`; reducing GraphQL page_size from %s to %s and retrying.",
+                    response_or_exception.status_code,
+                    self.stream.name,
+                    previous_page_size,
+                    self.stream.page_size,
+                )
                 return ErrorResolution(
                     response_action=ResponseAction.RETRY,
                     failure_type=FailureType.transient_error,
-                    error_message=f"Response status code: {response_or_exception.status_code}. Retrying...",
+                    error_message=(
+                        f"GitHub GraphQL endpoint returned HTTP {response_or_exception.status_code} "
+                        f"for stream `{self.stream.name}`. Reducing GraphQL page size and retrying."
+                    ),
                 )
 
             self.stream.page_size = (
@@ -214,7 +228,10 @@ class GitHubGraphQLErrorHandler(GithubStreamABCErrorHandler):
                 return ErrorResolution(
                     response_action=ResponseAction.RETRY,
                     failure_type=FailureType.transient_error,
-                    error_message=f"Response status code: {response_or_exception.status_code}. Retrying...",
+                    error_message=(
+                        f"GitHub GraphQL endpoint returned errors in the response body "
+                        f"for stream `{self.stream.name}` (HTTP {response_or_exception.status_code}). Retrying."
+                    ),
                 )
 
         return super().interpret_response(response_or_exception)

{airbyte_source_github-2.1.27 → airbyte_source_github-2.1.29}/source_github/streams.py

@@ -201,6 +201,12 @@ class GithubStreamABC(HttpStream, ABC):
                     f"GitHub returned HTTP 502 Bad Gateway for stream `{self.name}` after exhausting retries. "
                     f"This is usually transient — the next sync attempt should succeed."
                 )
+            elif e._exception.response.status_code == requests.codes.GATEWAY_TIMEOUT:
+                error_msg = (
+                    f"GitHub returned HTTP 504 Gateway Timeout for stream `{self.name}` after exhausting retries "
+                    f"and reducing the GraphQL page size. The next sync attempt should succeed; "
+                    f'if 504s persist, lower "Page size for large streams" in the source configuration.'
+                )
             else:
                 self.logger.error(f"Undefined error while reading records: {e._exception.response.text}")
                 raise e
@@ -232,11 +238,15 @@ class GithubStream(GithubStreamABC):
     def get_error_display_message(self, exception: BaseException) -> Optional[str]:
         if (
             isinstance(exception, DefaultBackoffException)
-            and exception.response.status_code == requests.codes.BAD_GATEWAY
+            and exception.response.status_code in (requests.codes.BAD_GATEWAY, requests.codes.GATEWAY_TIMEOUT)
             and self.large_stream
             and self.page_size > 1
         ):
-            return f'Please try to decrease the "Page size for large streams" below {self.page_size}. The stream "{self.name}" is a large stream, such streams can fail with 502 for high "page_size" values.'
+            return (
+                f'Please try to decrease the "Page size for large streams" below {self.page_size}. '
+                f'The stream "{self.name}" is a large stream, such streams can fail with '
+                f'{exception.response.status_code} for high "page_size" values.'
+            )
         return super().get_error_display_message(exception)
 
     def transform(self, record: MutableMapping[str, Any], stream_slice: Mapping[str, Any]) -> MutableMapping[str, Any]:
@@ -850,6 +860,16 @@ class Releases(SemiIncrementalMixin, GitHubGraphQLStream):
 
     cursor_field = "created_at"
     is_sorted = "asc"
+    # The Releases GraphQL query is high-cost on the server side: every node
+    # materializes `description` and `descriptionHTML`, which forces GitHub
+    # to render each release body to HTML. On repositories with long release
+    # notes, a page_size of 100 pushes the resolver past its internal 10s
+    # deadline and returns 504 Gateway Timeout (reproduced deterministically
+    # against nodejs/node: first=100 -> 504 in ~11s; first=10 -> 200 in ~3s).
+    # `releaseAssets` / `reactionGroups` / `mentions` are NOT the cost driver
+    # — stripping them does not fix the timeout, only lowering `first` does.
+    # Mark as large_stream so it picks up the smaller default page size.
+    large_stream = True
 
     GRAPHQL_REACTION_TO_REST = {
         "THUMBS_UP": "plus_one",

airbyte_source_github-2.1.27/PKG-INFO

@@ -1,126 +0,0 @@
-Metadata-Version: 2.1
-Name: airbyte-source-github
-Version: 2.1.27
-Summary: Source implementation for GitHub.
-Home-page: https://airbyte.com
-License: ELv2
-Author: Airbyte
-Author-email: contact@airbyte.io
-Requires-Python: >=3.10,<3.14
-Classifier: License :: Other/Proprietary License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Requires-Dist: airbyte-cdk (>=7.12.0,<8.0.0)
-Requires-Dist: sgqlc (==16.3)
-Project-URL: Documentation, https://docs.airbyte.com/integrations/sources/github
-Project-URL: Repository, https://github.com/airbytehq/airbyte
-Description-Content-Type: text/markdown
-
-# Github source connector
-
-This is the repository for the Github source connector, written in Python.
-For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/github).
-
-## Local development
-
-### Prerequisites
-
-- Python (~=3.9)
-- Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
-
-### Installing the connector
-
-From this connector directory, run:
-
-```bash
-poetry install --with dev
-```
-
-### Create credentials
-
-**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/github)
-to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_github/spec.yaml` file.
-Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
-See `sample_files/sample_config.json` for a sample config file.
-
-### Locally running the connector
-
-```
-poetry run source-github spec
-poetry run source-github check --config secrets/config.json
-poetry run source-github discover --config secrets/config.json
-poetry run source-github read --config secrets/config.json --catalog sample_files/configured_catalog.json
-```
-
-### Running unit tests
-
-To run unit tests locally, from the connector directory run:
-
-```
-poetry run pytest unit_tests
-```
-
-### Building the docker image
-
-1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
-2. Run the following command to build the docker image:
-
-```bash
-airbyte-ci connectors --name=source-github build
-```
-
-An image will be available on your host with the tag `airbyte/source-github:dev`.
-
-### Running as a docker container
-
-Then run any of the connector commands as follows:
-
-```
-docker run --rm airbyte/source-github:dev spec
-docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-github:dev check --config /secrets/config.json
-docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-github:dev discover --config /secrets/config.json
-docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-github:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
-```
-
-### Running our CI test suite
-
-You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
-
-```bash
-airbyte-ci connectors --name=source-github test
-```
-
-### Customizing acceptance Tests
-
-Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
-If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
-
-### Dependency Management
-
-All of your dependencies should be managed via Poetry.
-To add a new dependency, run:
-
-```bash
-poetry add <package-name>
-```
-
-Please commit the changes to `pyproject.toml` and `poetry.lock` files.
-
-## Publishing a new version of the connector
-
-You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
-
-1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-github test`
-2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
-   - bump the `dockerImageTag` value in in `metadata.yaml`
-   - bump the `version` value in `pyproject.toml`
-3. Make sure the `metadata.yaml` content is up to date.
-4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/github.md`).
-5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
-6. Pat yourself on the back for being an awesome contributor.
-7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
-8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.
-

airbyte_source_github-2.1.27/README.md

@@ -1,104 +0,0 @@
-# Github source connector
-
-This is the repository for the Github source connector, written in Python.
-For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/github).
-
-## Local development
-
-### Prerequisites
-
-- Python (~=3.9)
-- Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
-
-### Installing the connector
-
-From this connector directory, run:
-
-```bash
-poetry install --with dev
-```
-
-### Create credentials
-
-**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/github)
-to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_github/spec.yaml` file.
-Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
-See `sample_files/sample_config.json` for a sample config file.
-
-### Locally running the connector
-
-```
-poetry run source-github spec
-poetry run source-github check --config secrets/config.json
-poetry run source-github discover --config secrets/config.json
-poetry run source-github read --config secrets/config.json --catalog sample_files/configured_catalog.json
-```
-
-### Running unit tests
-
-To run unit tests locally, from the connector directory run:
-
-```
-poetry run pytest unit_tests
-```
-
-### Building the docker image
-
-1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
-2. Run the following command to build the docker image:
-
-```bash
-airbyte-ci connectors --name=source-github build
-```
-
-An image will be available on your host with the tag `airbyte/source-github:dev`.
-
-### Running as a docker container
-
-Then run any of the connector commands as follows:
-
-```
-docker run --rm airbyte/source-github:dev spec
-docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-github:dev check --config /secrets/config.json
-docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-github:dev discover --config /secrets/config.json
-docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-github:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
-```
-
-### Running our CI test suite
-
-You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
-
-```bash
-airbyte-ci connectors --name=source-github test
-```
-
-### Customizing acceptance Tests
-
-Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
-If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
-
-### Dependency Management
-
-All of your dependencies should be managed via Poetry.
-To add a new dependency, run:
-
-```bash
-poetry add <package-name>
-```
-
-Please commit the changes to `pyproject.toml` and `poetry.lock` files.
-
-## Publishing a new version of the connector
-
-You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
-
-1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-github test`
-2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
-   - bump the `dockerImageTag` value in in `metadata.yaml`
-   - bump the `version` value in `pyproject.toml`
-3. Make sure the `metadata.yaml` content is up to date.
-4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/github.md`).
-5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
-6. Pat yourself on the back for being an awesome contributor.
-7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
-8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.