airbyte-source-github 1.8.18__tar.gz → 1.8.20__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/PKG-INFO +1 -1
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/pyproject.toml +1 -1
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/backoff_strategies.py +1 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/config_migrations.py +3 -6
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/errors_handlers.py +2 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/github_schema.py +1 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/source.py +0 -1
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/streams.py +4 -8
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/utils.py +7 -5
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/README.md +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/__init__.py +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/constants.py +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/graphql.py +1 -1
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/run.py +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/assignees.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/branches.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/collaborators.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/comments.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/commit_comment_reactions.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/commit_comments.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/commits.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/contributor_activity.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/deployments.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/events.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issue_comment_reactions.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issue_events.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issue_labels.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issue_milestones.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issue_reactions.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issue_timeline_events.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issues.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/organizations.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/project_cards.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/project_columns.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/projects.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/projects_v2.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/pull_request_comment_reactions.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/pull_request_commits.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/pull_request_stats.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/pull_requests.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/releases.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/repositories.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/review_comments.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/reviews.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/events/comment.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/events/commented.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/events/committed.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/events/cross_referenced.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/events/reviewed.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/reaction.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/reactions.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/user.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/user_graphql.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/stargazers.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/tags.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/team_members.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/team_memberships.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/teams.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/users.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/workflow_jobs.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/workflow_runs.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/workflows.json +0 -0
- {airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/spec.json +0 -0
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/config_migrations.py
RENAMED
@@ -12,6 +12,7 @@ from airbyte_cdk.sources.message import InMemoryMessageRepository, MessageReposi
|
|
12
12
|
|
13
13
|
from .source import SourceGithub
|
14
14
|
|
15
|
+
|
15
16
|
logger = logging.getLogger("airbyte_logger")
|
16
17
|
|
17
18
|
|
@@ -30,13 +31,11 @@ class MigrateStringToArray(ABC):
|
|
30
31
|
|
31
32
|
@property
|
32
33
|
@abc.abstractmethod
|
33
|
-
def migrate_from_key(self) -> str:
|
34
|
-
...
|
34
|
+
def migrate_from_key(self) -> str: ...
|
35
35
|
|
36
36
|
@property
|
37
37
|
@abc.abstractmethod
|
38
|
-
def migrate_to_key(self) -> str:
|
39
|
-
...
|
38
|
+
def migrate_to_key(self) -> str: ...
|
40
39
|
|
41
40
|
@classmethod
|
42
41
|
def _should_migrate(cls, config: Mapping[str, Any]) -> bool:
|
@@ -95,12 +94,10 @@ class MigrateStringToArray(ABC):
|
|
95
94
|
|
96
95
|
|
97
96
|
class MigrateRepository(MigrateStringToArray):
|
98
|
-
|
99
97
|
migrate_from_key: str = "repository"
|
100
98
|
migrate_to_key: str = "repositories"
|
101
99
|
|
102
100
|
|
103
101
|
class MigrateBranch(MigrateStringToArray):
|
104
|
-
|
105
102
|
migrate_from_key: str = "branch"
|
106
103
|
migrate_to_key: str = "branches"
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/errors_handlers.py
RENAMED
@@ -5,6 +5,7 @@
|
|
5
5
|
from typing import Optional, Union
|
6
6
|
|
7
7
|
import requests
|
8
|
+
|
8
9
|
from airbyte_cdk.sources.streams.http import HttpStream
|
9
10
|
from airbyte_cdk.sources.streams.http.error_handlers import ErrorHandler, ErrorResolution, HttpStatusErrorHandler, ResponseAction
|
10
11
|
from airbyte_cdk.sources.streams.http.error_handlers.default_error_mapping import DEFAULT_ERROR_MAPPING
|
@@ -12,6 +13,7 @@ from airbyte_protocol.models import FailureType
|
|
12
13
|
|
13
14
|
from . import constants
|
14
15
|
|
16
|
+
|
15
17
|
GITHUB_DEFAULT_ERROR_MAPPING = DEFAULT_ERROR_MAPPING | {
|
16
18
|
401: ErrorResolution(
|
17
19
|
response_action=ResponseAction.RETRY,
|
@@ -9,6 +9,7 @@ from urllib import parse
|
|
9
9
|
|
10
10
|
import pendulum
|
11
11
|
import requests
|
12
|
+
|
12
13
|
from airbyte_cdk import BackoffStrategy, StreamSlice
|
13
14
|
from airbyte_cdk.models import AirbyteLogMessage, AirbyteMessage, Level, SyncMode
|
14
15
|
from airbyte_cdk.models import Type as MessageType
|
@@ -41,7 +42,6 @@ from .utils import GitHubAPILimitException, getter
|
|
41
42
|
|
42
43
|
|
43
44
|
class GithubStreamABC(HttpStream, ABC):
|
44
|
-
|
45
45
|
primary_key = "id"
|
46
46
|
|
47
47
|
# Detect streams with high API load
|
@@ -80,7 +80,6 @@ class GithubStreamABC(HttpStream, ABC):
|
|
80
80
|
def request_params(
|
81
81
|
self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
|
82
82
|
) -> MutableMapping[str, Any]:
|
83
|
-
|
84
83
|
params = {"per_page": self.page_size}
|
85
84
|
|
86
85
|
if next_page_token:
|
@@ -756,7 +755,6 @@ class ReviewComments(IncrementalMixin, GithubStream):
|
|
756
755
|
|
757
756
|
|
758
757
|
class GitHubGraphQLStream(GithubStream, ABC):
|
759
|
-
|
760
758
|
http_method = "POST"
|
761
759
|
|
762
760
|
def path(
|
@@ -976,7 +974,6 @@ class ProjectsV2(SemiIncrementalMixin, GitHubGraphQLStream):
|
|
976
974
|
|
977
975
|
|
978
976
|
class ReactionStream(GithubStream, CheckpointMixin, ABC):
|
979
|
-
|
980
977
|
parent_key = "id"
|
981
978
|
copy_parent_key = "comment_id"
|
982
979
|
cursor_field = "created_at"
|
@@ -1394,9 +1391,9 @@ class ProjectCards(GithubStream):
|
|
1394
1391
|
stream_state_value = current_stream_state.get(repository, {}).get(project_id, {}).get(column_id, {}).get(self.cursor_field)
|
1395
1392
|
if stream_state_value:
|
1396
1393
|
updated_state = max(updated_state, stream_state_value)
|
1397
|
-
current_stream_state.setdefault(repository, {}).setdefault(project_id, {}).setdefault(column_id, {})[
|
1398
|
-
|
1399
|
-
|
1394
|
+
current_stream_state.setdefault(repository, {}).setdefault(project_id, {}).setdefault(column_id, {})[self.cursor_field] = (
|
1395
|
+
updated_state
|
1396
|
+
)
|
1400
1397
|
return current_stream_state
|
1401
1398
|
|
1402
1399
|
def transform(self, record: MutableMapping[str, Any], stream_slice: Mapping[str, Any]) -> MutableMapping[str, Any]:
|
@@ -1621,7 +1618,6 @@ class ContributorActivity(GithubStream):
|
|
1621
1618
|
return record
|
1622
1619
|
|
1623
1620
|
def get_error_handler(self) -> Optional[ErrorHandler]:
|
1624
|
-
|
1625
1621
|
return ContributorActivityErrorHandler(logger=self.logger, max_retries=5, error_mapping=GITHUB_DEFAULT_ERROR_MAPPING)
|
1626
1622
|
|
1627
1623
|
def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]:
|
@@ -9,6 +9,7 @@ from typing import Any, List, Mapping
|
|
9
9
|
|
10
10
|
import pendulum
|
11
11
|
import requests
|
12
|
+
|
12
13
|
from airbyte_cdk.models import SyncMode
|
13
14
|
from airbyte_cdk.sources.streams import Stream
|
14
15
|
from airbyte_cdk.sources.streams.http.requests_native_auth import TokenAuthenticator
|
@@ -99,7 +100,6 @@ class MultipleTokenAuthenticatorWithRateLimiter(AbstractHeaderAuthenticator):
|
|
99
100
|
|
100
101
|
@property
|
101
102
|
def token(self) -> str:
|
102
|
-
|
103
103
|
token = self.current_active_token
|
104
104
|
return f"{self._auth_method} {token}"
|
105
105
|
|
@@ -123,13 +123,15 @@ class MultipleTokenAuthenticatorWithRateLimiter(AbstractHeaderAuthenticator):
|
|
123
123
|
)
|
124
124
|
token_info = self._tokens[token]
|
125
125
|
remaining_info_core = rate_limit_info.get("core")
|
126
|
-
token_info.count_rest, token_info.reset_at_rest =
|
127
|
-
remaining_info_core.get("
|
126
|
+
token_info.count_rest, token_info.reset_at_rest = (
|
127
|
+
remaining_info_core.get("remaining"),
|
128
|
+
pendulum.from_timestamp(remaining_info_core.get("reset")),
|
128
129
|
)
|
129
130
|
|
130
131
|
remaining_info_graphql = rate_limit_info.get("graphql")
|
131
|
-
token_info.count_graphql, token_info.reset_at_graphql =
|
132
|
-
remaining_info_graphql.get("
|
132
|
+
token_info.count_graphql, token_info.reset_at_graphql = (
|
133
|
+
remaining_info_graphql.get("remaining"),
|
134
|
+
pendulum.from_timestamp(remaining_info_graphql.get("reset")),
|
133
135
|
)
|
134
136
|
|
135
137
|
def check_all_tokens(self):
|
File without changes
|
File without changes
|
File without changes
|
@@ -11,6 +11,7 @@ from sgqlc.operation import Selector
|
|
11
11
|
|
12
12
|
from . import github_schema
|
13
13
|
|
14
|
+
|
14
15
|
_schema = github_schema
|
15
16
|
_schema_root = _schema.github_schema
|
16
17
|
|
@@ -165,7 +166,6 @@ def get_query_issue_reactions(owner, name, first, after, number=None):
|
|
165
166
|
|
166
167
|
|
167
168
|
class QueryReactions:
|
168
|
-
|
169
169
|
# AVERAGE_REVIEWS - optimal number of reviews to fetch inside every pull request.
|
170
170
|
# If we try to fetch too many (up to 100) we will spend too many scores of query cost.
|
171
171
|
# https://docs.github.com/en/graphql/overview/resource-limitations#calculating-a-rate-limit-score-before-running-the-call
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/assignees.json
RENAMED
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/branches.json
RENAMED
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/comments.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/commits.json
RENAMED
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/deployments.json
RENAMED
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/events.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/issues.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/projects.json
RENAMED
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/projects_v2.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/releases.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/reviews.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/shared/user.json
RENAMED
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/stargazers.json
RENAMED
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/tags.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/teams.json
RENAMED
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/users.json
RENAMED
File without changes
|
File without changes
|
File without changes
|
{airbyte_source_github-1.8.18 → airbyte_source_github-1.8.20}/source_github/schemas/workflows.json
RENAMED
File without changes
|
File without changes
|