openhound-github 0.21.3__tar.gz → 0.22.1__tar.gz

{openhound_github-0.21.3 → openhound_github-0.22.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: openhound-github
-Version: 0.21.3
+Version: 0.22.1
 Summary: Github collector for OpenHound
 Author-email: "jdreijer@specterops.io" <jdreijer@specterops.io>, Jared Atkinson <jatkinson@specterops.io>
 License-File: LICENSE.md
@@ -66,6 +66,27 @@ Both values must be provided together when overriding the GitHub.com defaults.
 Endpoint URLs must use HTTPS and share the same origin. GitHub App
 `credentials.api_uri` may override only the path on that same origin.
 
+### Request timeouts and retries
+
+REST and GraphQL collection requests use a 10-second connection timeout and a
+30-second read timeout. Transient HTTP 5xx errors, connection failures, and
+timeouts share a limit of three attempts (the initial attempt plus two retries).
+Adaptive GraphQL pagination shares that limit across page sizes: HTTP 502/504
+responses trigger smaller-page attempts instead of a new retry budget.
+
+After retries are exhausted, existing resource handlers log the failure and
+continue with unaffected scopes. Already emitted records are retained, but the
+affected scope may be incomplete. Resources that require all pages before
+emitting records retain that behavior.
+
+Workflow collection skips `dynamic/` paths. Missing workflow files are logged
+and skipped without passing failed items to job or step parsing.
+
+Rate-limit responses and GitHub App token refresh retain their existing retry
+handling and do not consume the transient-failure budget. These timeouts are
+not an overall request deadline. App authentication requests and credential
+inventory CSV downloads retain their separate timeout settings.
+
 ### Enterprise SCIM and hybrid correlations
 
 A token with enterprise SCIM access is used to collect both `/scim/v2/enterprises/{enterprise}/Users` and `/scim/v2/enterprises/{enterprise}/Groups`. The collector emits normalized `SCIM_Organization`, `SCIM_User`, and `SCIM_Group` nodes plus `SCIM_Contains`, `SCIM_MemberOf`, and `SCIM_Provisioned` relationships. Install the BloodHound SCIM extension alongside this extension to register the shared SCIM kinds.

{openhound_github-0.21.3 → openhound_github-0.22.1}/README.md

@@ -55,6 +55,27 @@ Both values must be provided together when overriding the GitHub.com defaults.
 Endpoint URLs must use HTTPS and share the same origin. GitHub App
 `credentials.api_uri` may override only the path on that same origin.
 
+### Request timeouts and retries
+
+REST and GraphQL collection requests use a 10-second connection timeout and a
+30-second read timeout. Transient HTTP 5xx errors, connection failures, and
+timeouts share a limit of three attempts (the initial attempt plus two retries).
+Adaptive GraphQL pagination shares that limit across page sizes: HTTP 502/504
+responses trigger smaller-page attempts instead of a new retry budget.
+
+After retries are exhausted, existing resource handlers log the failure and
+continue with unaffected scopes. Already emitted records are retained, but the
+affected scope may be incomplete. Resources that require all pages before
+emitting records retain that behavior.
+
+Workflow collection skips `dynamic/` paths. Missing workflow files are logged
+and skipped without passing failed items to job or step parsing.
+
+Rate-limit responses and GitHub App token refresh retain their existing retry
+handling and do not consume the transient-failure budget. These timeouts are
+not an overall request deadline. App authentication requests and credential
+inventory CSV downloads retain their separate timeout settings.
+
 ### Enterprise SCIM and hybrid correlations
 
 A token with enterprise SCIM access is used to collect both `/scim/v2/enterprises/{enterprise}/Users` and `/scim/v2/enterprises/{enterprise}/Groups`. The collector emits normalized `SCIM_Organization`, `SCIM_User`, and `SCIM_Group` nodes plus `SCIM_Contains`, `SCIM_MemberOf`, and `SCIM_Provisioned` relationships. Install the BloodHound SCIM extension alongside this extension to register the shared SCIM kinds.

openhound_github-0.22.1/descriptions/edges/GH_Owns.md

@@ -0,0 +1,29 @@
+# GH_Owns
+
+## General Information
+
+The traversable GH_Owns edge represents that a GitHub account owns an application or repository. For GitHub Apps, ownership identifies the account whose authorized administrators can manage the App registration and credentials. For repositories, ownership links the repository to its organization. These control relationships are traversable for attack-path analysis.
+
+## Edge Schema
+
+| Source | Destination | Traversable |
+| --- | --- | --- |
+| `GH_Enterprise` | `GH_App` | `true` |
+| `GH_Organization` | `GH_App` | `true` |
+| `GH_Organization` | `GH_Repository` | `true` |
+| `GH_User` | `GH_App` | `true` |
+
+## Diagram
+
+```mermaid
+graph LR
+    n0["GH_Enterprise"]
+    n1["GH_App"]
+    n2["GH_Organization"]
+    n3["GH_Repository"]
+    n4["GH_User"]
+    n0 -->|GH_Owns| n1
+    n2 -->|GH_Owns| n1
+    n2 -->|GH_Owns| n3
+    n4 -->|GH_Owns| n1
+```

{openhound_github-0.21.3 → openhound_github-0.22.1}/descriptions/nodes/GH_App.md

@@ -24,10 +24,12 @@ App definitions are retrieved via the public `GET /apps/{app_slug}` endpoint (no
 | `owner_login` | `string` | The login of the user or organization that owns the app. |
 | `owner_node_id` | `string` | The node_id of the user or organization that owns the app. |
 | `owner_type` | `string` | The type of the owner (e.g., `User`, `Organization`). |
+| `registration_origin` | `string` | Whether the app is tenant-owned, third-party, or unknown. |
 | `events` | `list[string]` | JSON string of the default webhook events the app subscribes to. |
 | `installations_count` | `integer` | The total number of installations of this app across all organizations. |
 | `created_at` | `datetime` | When the app was created. |
 | `updated_at` | `datetime` | When the app was last updated. |
+| `query_owner` | `string` | Query for the account that registered the app. |
 | `query_installations` | `string` | Query for installations. |
 
 ## Diagram
@@ -36,5 +38,11 @@ App definitions are retrieved via the public `GET /apps/{app_slug}` endpoint (no
 graph LR
     n0["GH_App"]
     n1["GH_AppInstallation"]
+    n2["GH_Enterprise"]
+    n3["GH_Organization"]
+    n4["GH_User"]
     n0 -->|GH_InstalledAs| n1
+    n2 -->|GH_Owns| n0
+    n3 -->|GH_Owns| n0
+    n4 -->|GH_Owns| n0
 ```

{openhound_github-0.21.3 → openhound_github-0.22.1}/extension/schema.json

@@ -2121,7 +2121,7 @@
     },
     {
       "name": "GH_Owns",
-      "description": "Organization owns a repository",
+      "description": "GitHub account owns an application or repository",
       "is_traversable": true
     },
     {

openhound_github-0.22.1/src/openhound_github/github_rest_client.py

@@ -0,0 +1,90 @@
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+from functools import wraps
+
+from dlt.sources.helpers import requests
+from requests.exceptions import ChunkedEncodingError, ConnectionError, Timeout
+from tenacity import RetryCallState
+from tenacity.stop import stop_base
+
+from openhound_github.github_retry import should_retry_github_response
+
+__all__ = ["github_http_session", "should_retry_github_response"]
+
+
+@dataclass
+class GraphQLPageRetryBudget:
+    """Share transient failures across page sizes without changing other requests."""
+
+    failures: int = 0
+
+    @property
+    def exhausted(self) -> bool:
+        return self.failures >= 3
+
+    @contextmanager
+    def activate(self):
+        token = _graphql_page_budget.set(self)
+        try:
+            yield
+        finally:
+            _graphql_page_budget.reset(token)
+
+
+_graphql_page_budget: ContextVar[GraphQLPageRetryBudget | None] = ContextVar(
+    "github_graphql_page_budget", default=None
+)
+
+
+class _StopAfterTransientFailures(stop_base):
+    """Count transient failures per send, independently of other retry reasons."""
+
+    def __call__(self, retry_state: RetryCallState) -> bool:
+        outcome = retry_state.outcome
+        if outcome is None:
+            return False
+        exception = outcome.exception()
+        response = None
+        if isinstance(exception, requests.HTTPError):
+            response = exception.response
+        elif exception is None:
+            response = outcome.result()
+        transient = isinstance(exception, (ConnectionError, Timeout, ChunkedEncodingError)) or (
+            response is not None and 500 <= response.status_code < 600
+        )
+        page_budget = _graphql_page_budget.get()
+        if page_budget is not None:
+            page_budget.failures += int(transient)
+            # Return gateway errors immediately so pagination can shrink the page.
+            return page_budget.exhausted or (
+                response is not None and response.status_code in (502, 504)
+            )
+        failures = getattr(retry_state, "_github_transient_failures", 0) + int(transient)
+        setattr(retry_state, "_github_transient_failures", failures)
+        return failures >= 3
+
+
+def github_http_session(retry_condition):
+    """Bound transient failures while retaining DLT's existing retry policy."""
+    session = requests.Client(
+        request_timeout=(10, 30),
+        status_codes=tuple(range(500, 600)),
+        retry_condition=retry_condition,
+    ).session
+    # Keep the configured overall limit for rate limits and credential refresh.
+    # RetryCallState is new for every send, including concurrent requests.
+    retry = session.send.retry
+    retry.stop = _StopAfterTransientFailures() | retry.stop
+    send = session.send
+
+    @wraps(send)
+    def send_with_status_check(request, **kwargs):
+        response = send(request, **kwargs)
+        # RESTClient calls send directly, bypassing Session.request's status check.
+        # Raise only after the retry loop finishes so scope handlers can skip it.
+        response.raise_for_status()
+        return response
+
+    session.send = send_with_status_check
+    return session

{openhound_github-0.21.3 → openhound_github-0.22.1}/src/openhound_github/helpers.py

@@ -14,6 +14,7 @@ from dlt.sources.helpers.rest_client.paginators import (
 from requests import Request
 
 from openhound_github.auth import GitHubAppInstallationAuth
+from openhound_github.github_rest_client import GraphQLPageRetryBudget
 from openhound_github.github_retry import (
     is_primary_rate_limit_response,
     is_secondary_rate_limit_response,
@@ -230,9 +231,9 @@ def adaptive_graphql_paginate(
 ) -> Iterator[dict[str, Any]]:
     """Yield GraphQL pages while adapting connection size after gateway failures.
 
-    The underlying HTTP session still owns ordinary request retries. This helper
-    handles the remaining terminal 502/504 case by retrying the same cursor with
-    progressively smaller connection sizes.
+    HTTP retries and smaller-page recovery share three transient attempts per
+    page. Gateway errors return immediately so the next attempt can shrink the
+    page; rate-limit and credential-refresh retries retain their existing policy.
     """
     if not page_sizes or any(page_size <= 0 for page_size in page_sizes):
         raise ValueError("page_sizes must contain positive integers")
@@ -270,6 +271,7 @@ def adaptive_graphql_paginate(
         page_data: dict[str, Any] | None = None
         paginator: GraphQLCursorPaginator | None = None
         successful_page_size: int | None = None
+        retry_budget = GraphQLPageRetryBudget()
 
         for index, page_size in enumerate(candidate_sizes):
             request_variables = {
@@ -278,10 +280,11 @@ def adaptive_graphql_paginate(
                 page_size_variable: page_size,
             }
             try:
-                response = client.post(
-                    graphql_path,
-                    json={"query": query, "variables": request_variables},
-                )
+                with retry_budget.activate():
+                    response = client.post(
+                        graphql_path,
+                        json={"query": query, "variables": request_variables},
+                    )
                 response.raise_for_status()
                 response_json = response.json()
                 paginator = GraphQLCursorPaginator(
@@ -303,7 +306,11 @@ def adaptive_graphql_paginate(
                 status_code = _graphql_gateway_status(error)
                 next_page_size = (
                     candidate_sizes[index + 1]
-                    if status_code is not None and index + 1 < len(candidate_sizes)
+                    if (
+                        status_code is not None
+                        and index + 1 < len(candidate_sizes)
+                        and not retry_budget.exhausted
+                    )
                     else None
                 )
                 if next_page_size is not None:

{openhound_github-0.21.3 → openhound_github-0.22.1}/src/openhound_github/lookup.py

@@ -118,6 +118,38 @@ class GithubLookup(LookupManager):
             [enterprise_node_id],
         )
 
+    @lru_cache
+    def is_tenant_organization_node_id(self, node_id: str) -> bool:
+        """Return whether an organization belongs to the collected tenant."""
+        collected_org_id = self._find_single_object(
+            f"""SELECT node_id FROM {self.schema}.organizations WHERE node_id = ?""",
+            [node_id],
+        )
+        if collected_org_id is not None:
+            return True
+
+        enterprise_org_id = self._find_single_object(
+            f"""SELECT id FROM {self.schema}.enterprise_organizations WHERE id = ?""",
+            [node_id],
+        )
+        return enterprise_org_id is not None
+
+    @lru_cache
+    def is_collected_user_node_id(self, node_id: str) -> bool:
+        """Return whether a user node exists in collected organization or enterprise data."""
+        organization_user_id = self._find_single_object(
+            f"""SELECT id FROM {self.schema}.users WHERE id = ?""",
+            [node_id],
+        )
+        if organization_user_id is not None:
+            return True
+
+        enterprise_user_id = self._find_single_object(
+            f"""SELECT id FROM {self.schema}.enterprise_users WHERE id = ?""",
+            [node_id],
+        )
+        return enterprise_user_id is not None
+
     @lru_cache
     def _enterprise_runner_group_identity_for_inherited_org_group(
         self, org_node_id: str, group_name: str

{openhound_github-0.21.3 → openhound_github-0.22.1}/src/openhound_github/models/app_installation.py

@@ -215,10 +215,12 @@ class GHAppProperties(GHNodeProperties):
         owner_login: The login of the user or organization that owns the app.
         owner_node_id: The node_id of the user or organization that owns the app.
         owner_type: The type of the owner (e.g., `User`, `Organization`).
+        registration_origin: Whether the app is tenant-owned, third-party, or unknown.
         events: JSON string of the default webhook events the app subscribes to.
         installations_count: The total number of installations of this app across all organizations.
         created_at: When the app was created.
         updated_at: When the app was last updated.
+        query_owner: Query for the account that registered the app.
         query_installations: Query for installations.
     """
 
@@ -231,10 +233,12 @@ class GHAppProperties(GHNodeProperties):
     owner_login: str | None = None
     owner_node_id: str | None = None
     owner_type: str | None = None
+    registration_origin: str | None = None
     events: list[str] | None = None
     installations_count: int | None = None
     created_at: datetime | None = None
     updated_at: datetime | None = None
+    query_owner: str | None = None
     query_installations: str | None = None
 
 
@@ -252,6 +256,29 @@ class Owner(BaseModel):
         icon="cube",
         properties=GHAppProperties,
     ),
+    edges=[
+        EdgeDef(
+            start=nk.ENTERPRISE,
+            end=nk.APP,
+            kind=ek.OWNS,
+            description="Enterprise owns GitHub App",
+            traversable=True,
+        ),
+        EdgeDef(
+            start=nk.ORGANIZATION,
+            end=nk.APP,
+            kind=ek.OWNS,
+            description="Organization owns GitHub App",
+            traversable=True,
+        ),
+        EdgeDef(
+            start=nk.USER,
+            end=nk.APP,
+            kind=ek.OWNS,
+            description="User owns GitHub App",
+            traversable=True,
+        ),
+    ],
 )
 class App(BaseAsset):
     """One record from `apps` → one GH_App node. Edges (GH_InstalledAs) are emitted by AppInstallationAsset."""
@@ -262,7 +289,7 @@ class App(BaseAsset):
     slug: str
     client_id: str | None = None
     node_id: str
-    owner: Owner
+    owner: Owner | None = None
     name: str
     description: str | None = None
     permissions: dict
@@ -282,6 +309,51 @@ class App(BaseAsset):
             return self.collected_org_node_id
         return self._lookup.org_id_for_login(self.org_login)
 
+    @property
+    def registration_origin(self) -> str:
+        if self.owner is None:
+            return "unknown"
+
+        owner_type = self.owner.type.casefold()
+        if owner_type == "user":
+            return "third_party"
+
+        if owner_type == "organization":
+            if self._lookup.is_tenant_organization_node_id(self.owner.node_id):
+                return "tenant_owned"
+            return "third_party" if self._lookup.enterprise_id() else "unknown"
+
+        if owner_type == "enterprise":
+            enterprise_id = self._lookup.enterprise_id()
+            if enterprise_id is None:
+                return "unknown"
+            return (
+                "tenant_owned"
+                if self.owner.node_id == enterprise_id
+                else "third_party"
+            )
+
+        return "unknown"
+
+    @property
+    def registration_owner_node_id(self) -> str | None:
+        if self.owner is None:
+            return None
+
+        owner_type = self.owner.type.casefold()
+        if owner_type == "organization":
+            owner_exists = self._lookup.is_tenant_organization_node_id(
+                self.owner.node_id
+            )
+        elif owner_type == "enterprise":
+            owner_exists = self._lookup.enterprise_id() == self.owner.node_id
+        elif owner_type == "user":
+            owner_exists = self._lookup.is_collected_user_node_id(self.owner.node_id)
+        else:
+            owner_exists = False
+
+        return self.owner.node_id if owner_exists else None
+
     @property
     def as_node(self):
         aid = self.node_id
@@ -298,21 +370,34 @@ class App(BaseAsset):
                 description=self.description,
                 external_url=self.external_url,
                 html_url=self.html_url,
-                # owner_login=self.owner_login,
-                # owner_node_id=self.owner_node_id,
-                # owner_type=self.owner_type,
+                owner_login=self.owner.login if self.owner else None,
+                owner_node_id=self.owner.node_id if self.owner else None,
+                owner_type=self.owner.type if self.owner else None,
+                registration_origin=self.registration_origin,
                 # permissions=self.permissions,
                 # installations_count=self.installations_count,
                 events=self.events,
                 created_at=self.created_at,
                 updated_at=self.updated_at,
+                query_owner=f"MATCH p=(owner)-[:GH_Owns]->(:GH_App {{node_id:'{aid}'}}) RETURN p",
                 query_installations=f"MATCH p=(:GH_App {{node_id:'{aid}'}})-[:GH_InstalledAs]->(:GH_AppInstallation) RETURN p",
             ),
         )
 
     @property
     def edges(self) -> list[Edge]:
-        return []
+        owner_node_id = self.registration_owner_node_id
+        if owner_node_id is None:
+            return []
+
+        return [
+            Edge(
+                kind=ek.OWNS,
+                start=EdgePath(value=owner_node_id, match_by="id"),
+                end=EdgePath(value=self.node_id, match_by="id"),
+                properties=EdgeProperties(traversable=True),
+            )
+        ]
 
 
 @app.asset(

{openhound_github-0.21.3 → openhound_github-0.22.1}/src/openhound_github/resources/organization.py

@@ -6,6 +6,7 @@ from threading import Lock
 from typing import Any, Iterator
 from urllib.parse import quote
 
+import dlt
 from dlt.sources.helpers import requests
 from dlt.sources.helpers.rest_client.client import RESTClient
 from dlt.sources.helpers.rest_client.paginators import (
@@ -1475,7 +1476,7 @@ def workflows(repo: Repository, ctx: SourceContext):
         Workflow (Workflow): An active workflow record.
     """
 
-    @app.defer
+    @dlt.defer
     def _workflow_file_contents(
         client: RESTClient,
         repo: Repository,
@@ -1487,11 +1488,30 @@ def workflows(repo: Repository, ctx: SourceContext):
             return None
 
         params = {"ref": repo.default_branch} if repo.default_branch else None
-        response = client.get(
-            f"/repos/{repo.full_name}/contents/{path}", params=params
-        ).json()
+        try:
+            response = client.get(
+                f"/repos/{repo.full_name}/contents/{path}", params=params
+            ).json()
+            content = response.get("content")
+        except Exception as exc:
+            # Return None so DLT drops the failed item. app.defer returns [],
+            # which can reach the job/step transformers as an invalid Workflow.
+            if (
+                isinstance(exc, requests.HTTPError)
+                and exc.response is not None
+                and exc.response.status_code == 404
+            ):
+                logger.warning(
+                    "Skipping workflow '%s' in repository '%s': file unavailable at ref '%s' (HTTP 404)",
+                    path, repo.full_name, repo.default_branch,
+                )
+            else:
+                logger.error(
+                    "Unable to collect workflow '%s' in repository '%s': %s",
+                    path, repo.full_name, exc,
+                )
+            return None
 
-        content = response.get("content")
         if not content:
             return None
 
@@ -1517,6 +1537,9 @@ def workflows(repo: Repository, ctx: SourceContext):
     ):
         for workflow in page:
             if workflow.get("state") == "active":
+                path = workflow.get("path")
+                if not path or path.startswith("dynamic/"):
+                    continue
                 if workflow_permissions is None:
                     workflow_permissions = _repository_workflow_permissions(
                         ctx, client, repo.full_name

{openhound_github-0.21.3 → openhound_github-0.22.1}/src/openhound_github/source.py

@@ -8,7 +8,6 @@ from urllib.parse import urlparse
 import dlt
 from dlt.common.configuration import configspec
 from dlt.common.configuration.specs import CredentialsConfiguration
-from dlt.sources.helpers import requests
 from dlt.sources.helpers.rest_client.auth import AuthConfigBase
 from dlt.sources.helpers.rest_client.auth import BearerTokenAuth
 from dlt.sources.helpers.rest_client.client import RESTClient
@@ -27,6 +26,7 @@ from openhound_github.helpers import (
     DEFAULT_GITHUB_REST_API_URL,
     github_retry_policy,
 )
+from openhound_github.github_rest_client import github_http_session
 from openhound_github.main import app
 from openhound_github.models.saml_helpers import (
     DEFAULT_GITHUB_DEPLOYMENT_ID,
@@ -352,10 +352,7 @@ def source(
             },
             auth=auth,
             paginator=HeaderLinkPaginator(),
-            session=requests.Client(
-                status_codes=tuple(range(500, 600)),
-                retry_condition=github_retry_policy(auth),
-            ).session,
+            session=github_http_session(github_retry_policy(auth)),
         )
 
     def clients(auth: AuthConfigBase) -> tuple[RESTClient, RESTClient]: