openhound-github 0.6.7__tar.gz → 0.7.1__tar.gz

{openhound_github-0.6.7 → openhound_github-0.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: openhound-github
-Version: 0.6.7
+Version: 0.7.1
 Summary: Github collector for OpenHound
 Author-email: "jdreijer@specterops.io" <jdreijer@specterops.io>, Jared Atkinson <jatkinson@specterops.io>
 License-File: LICENSE.md
@@ -74,10 +74,23 @@ A token with enterprise SCIM access is used to collect both `/scim/v2/enterprise
 
 Enterprise roles, including the built-in `members` role, are emitted through `GH_HasRole` and granular enterprise capability relationships. Only capability relationships with a confirmed privilege path are traversable; descriptive permissions such as `GH_WriteEnterpriseSso` remain non-traversable.
 
+### Optional privileged collection
+
+The collector runs successfully with the documented read-only permission set. Some higher-fidelity relationships require optional non-read-only permissions because GitHub exposes the supporting read APIs behind privileged permission names:
+
+- Organization `Members: write` enables external group mapping collection for IdP-synced teams, which allows the converter to emit `SCIM_Provisioned` edges from `SCIM_Group` to `GH_Team`. This is useful when the organization uses team synchronization with an external identity provider; if no GitHub teams are linked to external groups, this permission does not add graph data.
+- Classic PAT scope `manage_runners:enterprise` enables enterprise self-hosted runner group and runner collection. This is useful when the enterprise has enterprise-scoped runner groups or runners, especially runner groups shared into organizations; if all runners are organization- or repository-scoped, this scope does not add graph data.
+
+If these optional permissions are not granted, OpenHound skips the affected resources and continues collecting the rest of the GitHub environment.
+
 [![Python Version](https://img.shields.io/badge/Python-3.13-brightgreen.svg)](#about)
 
 ## Getting Started
 
-Follow the OpenHound docs to get started:
+Follow the OpenHound GitHub collector docs to get started:
 
-- [OpenHound Documentation](https://bloodhound.specterops.io/openhound/overview)
+- [GitHub collector overview](https://bloodhound.specterops.io/openhound/collectors/github/overview)
+- [Configure the collector](https://bloodhound.specterops.io/openhound/collectors/github/collect-data)
+- [Configure an organization GitHub App](https://bloodhound.specterops.io/openhound/collectors/github/configure-app)
+- [Configure an enterprise GitHub App](https://bloodhound.specterops.io/openhound/collectors/github/configure-enterprise-app)
+- [Configure a personal access token](https://bloodhound.specterops.io/openhound/collectors/github/configure-pat)

{openhound_github-0.6.7 → openhound_github-0.7.1}/README.md

@@ -63,10 +63,23 @@ A token with enterprise SCIM access is used to collect both `/scim/v2/enterprise
 
 Enterprise roles, including the built-in `members` role, are emitted through `GH_HasRole` and granular enterprise capability relationships. Only capability relationships with a confirmed privilege path are traversable; descriptive permissions such as `GH_WriteEnterpriseSso` remain non-traversable.
 
+### Optional privileged collection
+
+The collector runs successfully with the documented read-only permission set. Some higher-fidelity relationships require optional non-read-only permissions because GitHub exposes the supporting read APIs behind privileged permission names:
+
+- Organization `Members: write` enables external group mapping collection for IdP-synced teams, which allows the converter to emit `SCIM_Provisioned` edges from `SCIM_Group` to `GH_Team`. This is useful when the organization uses team synchronization with an external identity provider; if no GitHub teams are linked to external groups, this permission does not add graph data.
+- Classic PAT scope `manage_runners:enterprise` enables enterprise self-hosted runner group and runner collection. This is useful when the enterprise has enterprise-scoped runner groups or runners, especially runner groups shared into organizations; if all runners are organization- or repository-scoped, this scope does not add graph data.
+
+If these optional permissions are not granted, OpenHound skips the affected resources and continues collecting the rest of the GitHub environment.
+
 [![Python Version](https://img.shields.io/badge/Python-3.13-brightgreen.svg)](#about)
 
 ## Getting Started
 
-Follow the OpenHound docs to get started:
+Follow the OpenHound GitHub collector docs to get started:
 
-- [OpenHound Documentation](https://bloodhound.specterops.io/openhound/overview)
+- [GitHub collector overview](https://bloodhound.specterops.io/openhound/collectors/github/overview)
+- [Configure the collector](https://bloodhound.specterops.io/openhound/collectors/github/collect-data)
+- [Configure an organization GitHub App](https://bloodhound.specterops.io/openhound/collectors/github/configure-app)
+- [Configure an enterprise GitHub App](https://bloodhound.specterops.io/openhound/collectors/github/configure-enterprise-app)
+- [Configure a personal access token](https://bloodhound.specterops.io/openhound/collectors/github/configure-pat)

openhound_github-0.7.1/descriptions/edges/GH_MemberOf.md

@@ -0,0 +1,3 @@
+## General Information
+
+The traversable GH_MemberOf edge represents team membership and projection relationships, linking a team role to its parent team, a child team to a parent team in nested team hierarchies, or a GH_EnterpriseTeam to its projected GH_Team in an organization. This edge is traversable because these relationships carry effective team membership context through the graph: a user who holds a role in a child team inherits the repository permissions of ancestor teams, and enterprise-managed team membership flows into the projected organization team.

{openhound_github-0.6.7 → openhound_github-0.7.1}/extension/schema.json

@@ -467,7 +467,7 @@
     },
     {
       "name": "GH_MemberOf",
-      "description": "Team role is a member of a team, or team is a nested member of a parent team",
+      "description": "Team role is a member of a team, team is nested under a parent team, or enterprise team maps to its projected organization team",
       "is_traversable": true
     },
     {

openhound_github-0.7.1/src/openhound_github/github_rest_client.py

@@ -0,0 +1,3 @@
+from openhound_github.github_retry import should_retry_github_response
+
+__all__ = ["should_retry_github_response"]

openhound_github-0.7.1/src/openhound_github/github_retry.py

@@ -0,0 +1,48 @@
+from typing import Optional
+
+from requests import Response
+
+
+def _response_message(response: Response) -> str:
+    try:
+        response_data = response.json()
+    except ValueError:
+        return getattr(response, "text", "")
+    if isinstance(response_data, dict):
+        return str(response_data.get("message", ""))
+    return ""
+
+
+def is_primary_rate_limit_response(response: Response) -> bool:
+    if response.status_code not in (403, 429):
+        return False
+
+    message = _response_message(response).lower()
+    return (
+        response.headers.get("x-ratelimit-remaining") == "0"
+        or "api rate limit exceeded" in message
+    )
+
+
+def is_secondary_rate_limit_response(response: Response) -> bool:
+    if response.status_code not in (403, 429):
+        return False
+
+    message = _response_message(response).lower()
+    return "secondary rate limit" in message or "abuse detection" in message
+
+
+def should_retry_github_response(
+    response: Optional[Response], exception: Optional[BaseException]
+) -> bool:
+    if response is None:
+        return False
+
+    return (
+        is_primary_rate_limit_response(response)
+        or is_secondary_rate_limit_response(response)
+        or (
+            response.status_code in (403, 429)
+            and bool(response.headers.get("Retry-After"))
+        )
+    )

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/graphql.py

@@ -98,7 +98,7 @@ ENTERPRISE_ADMINS_QUERY = """
 query EnterpriseAdmins($slug: String!, $count: Int = 100, $after: String = null) {
     enterprise(slug: $slug) {
         ownerInfo {
-            admins(first: $count, after: $after) {
+            admins(first: $count, after: $after, role: OWNER) {
                 edges {
                     node {
                         id
@@ -115,6 +115,23 @@ query EnterpriseAdmins($slug: String!, $count: Int = 100, $after: String = null)
 }
 """
 
+ORGANIZATION_ENTERPRISE_OWNERS_QUERY = """
+query OrganizationEnterpriseOwners($login: String!, $count: Int = 100, $after: String = null) {
+    organization(login: $login) {
+        enterpriseOwners(first: $count, after: $after) {
+            nodes {
+                id
+                login
+            }
+            pageInfo {
+                endCursor
+                hasNextPage
+            }
+        }
+    }
+}
+"""
+
 ENTERPRISE_SAML_QUERY = """
 query EnterpriseSAML($slug: String!, $count: Int = 100, $after: String = null) {
     enterprise(slug: $slug) {

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/helpers.py

@@ -14,6 +14,10 @@ from dlt.sources.helpers.rest_client.paginators import (
 from requests import Request
 
 from openhound_github.auth import GitHubAppInstallationAuth
+from openhound_github.github_retry import (
+    is_primary_rate_limit_response,
+    is_secondary_rate_limit_response,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -439,18 +443,14 @@ def github_retry_policy(auth: AuthConfigBase):
         if response.status_code not in (403, 429):
             return False
 
-        message = _response_message(response).lower()
-        if (
-            headers.get("x-ratelimit-remaining") == "0"
-            or "api rate limit exceeded" in message
-        ):
+        if is_primary_rate_limit_response(response):
             reset_at = headers.get("x-ratelimit-reset")
             delay = int(reset_at) - now if reset_at else 0
             headers["Retry-After"] = str(delay)
             logger.warning("Primary rate limit reached, retrying in %s seconds", delay)
             return True
 
-        if "secondary rate limit" in message or "abuse detection" in message:
+        if is_secondary_rate_limit_response(response):
             logger.warning("Secondary rate limit reached, retrying in 60 seconds")
             headers["Retry-After"] = "60"
             return True

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/lookup.py

@@ -205,6 +205,42 @@ class GithubLookup(LookupManager):
             [org_login, slug],
         )
 
+    @lru_cache
+    def external_group_for_team(
+        self, org_login: str, team_database_id: int
+    ) -> tuple[int, str] | None:
+        row = self._find_single_row(
+            f"""
+            SELECT external_group_id, external_group_name
+            FROM {self.schema}.team_external_groups
+            WHERE org_login = ?
+              AND team_database_id = ?
+            """,
+            [org_login, team_database_id],
+        )
+        if row is None:
+            return None
+        return int(row[0]), str(row[1])
+
+    @lru_cache
+    def scim_group_id_for_team_external_group(
+        self, org_login: str, external_group_name: str
+    ) -> str | None:
+        rows = self._find_all_objects(
+            f"""
+            SELECT DISTINCT esg.id
+            FROM {self.schema}.enterprise_scim_groups esg
+            JOIN {self.schema}.enterprise_organizations eo
+              ON eo.enterprise_node_id = esg.enterprise_node_id
+            WHERE eo.login = ?
+              AND esg.display_name = ?
+            """,
+            [org_login, external_group_name],
+        )
+        if not rows or len(rows) != 1:
+            return None
+        return str(rows[0][0])
+
     @lru_cache
     def external_identity_id_for_guid(
         self, guid: str, environment_slug: str

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/main.py

@@ -61,12 +61,14 @@ def preproc(ctx: PreProcContext):
         "repo_roles": "repo_roles",
         "users": "users",
         "teams": "teams",
+        "team_external_groups": "team_external_groups",
         "team_members": "team_members",
         "saml_provider": "saml_provider",
         "external_identities": "external_identities",
         "applications": "applications",
         "enterprise": "enterprise",
         "enterprise_organizations": "enterprise_organizations",
+        "enterprise_scim_groups": "enterprise_scim_groups",
         "enterprise_runner_groups": "enterprise_runner_groups",
         "enterprise_runner_group_organizations": "enterprise_runner_group_organizations",
         "enterprise_runner_group_memberships": "enterprise_runner_group_memberships",

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/models/__init__.py

@@ -62,7 +62,7 @@ from .scim_user import (
     ScimUser,
 )
 from .secret_scanning_alert import SecretScanningAlert
-from .team import Team
+from .team import Team, TeamExternalGroup
 from .team_member import TeamMember
 from .team_role import TeamRole
 from .user import User
@@ -78,6 +78,7 @@ __all__ = [
     "ActionPermission",
     "User",
     "Team",
+    "TeamExternalGroup",
     "TeamRole",
     "TeamMember",
     "Repository",

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/models/enterprise_admin.py

@@ -18,4 +18,4 @@ from openhound_github.models.enterprise_role_user import EnterpriseRoleUser
     ],
 )
 class EnterpriseAdmin(EnterpriseRoleUser):
-    pass
+    role_id: str

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/models/team.py

@@ -15,7 +15,7 @@ from openhound_github.main import app
 @dataclass
 class GHTeamProperties(GHNodeProperties):
     """Team-specific properties and accordion panel queries.
-    
+
     Attributes:
         collected: Collected/generated by OpenHound
         github_team_id: The raw GitHub team node ID.
@@ -23,6 +23,8 @@ class GHTeamProperties(GHNodeProperties):
         description: The team's description.
         privacy: The team's privacy level (e.g., `visible`, `secret`).
         type: The team type, such as `enterprise` for projected enterprise teams.
+        external_group_id: GitHub's org-scoped external IdP group identifier.
+        external_group_name: The external IdP group name reported by GitHub.
         environment_name: The name of the environment (GitHub organization).
         query_first_degree_members: Query for first degree members.
         query_unrolled_members: Query for unrolled members.
@@ -40,6 +42,8 @@ class GHTeamProperties(GHNodeProperties):
     description: str | None = None
     privacy: str | None = None
     type: str | None = None
+    external_group_id: int | None = None
+    external_group_name: str | None = None
     environment_name: str | None = None
 
     query_first_degree_members: str | None = None
@@ -74,6 +78,16 @@ class ParentTeam(BaseModel):
     id: str
 
 
+class TeamExternalGroup(BaseModel):
+    """Raw mapping between an org team and its linked external IdP group."""
+
+    org_login: str
+    team_database_id: int
+    external_group_id: int
+    external_group_name: str
+    external_group_updated_at: str | None = None
+
+
 @app.asset(
     node=NodeDef(
         kind=nk.TEAM,
@@ -89,6 +103,13 @@ class ParentTeam(BaseModel):
             description="Team is a child of parent team",
             traversable=False,
         ),
+        EdgeDef(
+            start=nk.SCIM_GROUP,
+            end=nk.TEAM,
+            kind=ek.SCIM_PROVISIONED,
+            description="SCIM group is provisioned as team",
+            traversable=True,
+        ),
     ],
 )
 class Team(BaseAsset):
@@ -120,9 +141,16 @@ class Team(BaseAsset):
         """The ID from a GraphQL API response is the same as a regular node_id"""
         return self.id
 
+    @property
+    def external_group(self) -> tuple[int, str] | None:
+        if self.database_id is None:
+            return None
+        return self._lookup.external_group_for_team(self.org_login, self.database_id)
+
     @property
     def as_node(self) -> GHNode:
         tid = self.node_id
+        external_group = self.external_group
         return GHNode(
             kinds=[nk.TEAM],
             properties=GHTeamProperties(
@@ -132,6 +160,8 @@ class Team(BaseAsset):
                 slug=self.slug,
                 description=self.description,
                 privacy=self.privacy,
+                external_group_id=external_group[0] if external_group else None,
+                external_group_name=external_group[1] if external_group else None,
                 environment_name=self.org_login,
                 environmentid=self.org_node_id,
                 query_first_degree_members=f"MATCH p=(:GH_User)-[:GH_HasRole]->(t:GH_TeamRole)-[:GH_MemberOf]->(:GH_Team {{node_id:'{tid}'}}) RETURN p",
@@ -176,6 +206,19 @@ class Team(BaseAsset):
                 properties=EdgeProperties(traversable=True),
             )
 
+        external_group = self.external_group
+        if external_group:
+            scim_group_id = self._lookup.scim_group_id_for_team_external_group(
+                self.org_login, external_group[1]
+            )
+            if scim_group_id:
+                yield Edge(
+                    kind=ek.SCIM_PROVISIONED,
+                    start=EdgePath(value=scim_group_id, match_by="id"),
+                    end=EdgePath(value=self.node_id, match_by="id"),
+                    properties=EdgeProperties(traversable=True),
+                )
+
         # yield from self._branch_edges
         yield from self._bypass_pull_request_allowances_edges
         yield from self._bypass_push_allowances_edges

{openhound_github-0.6.7 → openhound_github-0.7.1}/src/openhound_github/resources/enterprise.py

@@ -1,5 +1,6 @@
 import logging
-from dataclasses import dataclass
+from dataclasses import dataclass, field
+from typing import Any
 
 from dlt.sources.helpers.rest_client.client import RESTClient
 from dlt.sources.helpers.rest_client.paginators import OffsetPaginator
@@ -7,6 +8,7 @@ from dlt.sources.helpers.rest_client.paginators import OffsetPaginator
 from openhound_github.graphql import (
     ENTERPRISE_ADMINS_QUERY,
     ENTERPRISE_MEMBERS_QUERY,
+    ORGANIZATION_ENTERPRISE_OWNERS_QUERY,
     ENTERPRISE_SAML_PROVIDER_QUERY,
     ENTERPRISE_QUERY,
     ENTERPRISE_SAML_QUERY,
@@ -65,6 +67,7 @@ class SourceContext:
     emit_legacy_scim_correlations: bool = False
     github_deployment_id: str = DEFAULT_GITHUB_DEPLOYMENT_ID
     github_web_origin: str = DEFAULT_GITHUB_WEB_ORIGIN
+    organizations: list[Any] = field(default_factory=list)
 
 
 def _graphql_client(ctx: SourceContext) -> tuple[RESTClient, str]:
@@ -641,6 +644,60 @@ def enterprise_role_users(role: EnterpriseRole, ctx: SourceContext):
 
 @app.transformer(name="enterprise_admins", columns=EnterpriseAdmin, parallelized=True)
 def enterprise_admins(enterprise_data: Enterprise, ctx: SourceContext):
+    seen_node_ids: set[str] = set()
+    owner_info_completed = False
+    client, graphql_path = _sso_graphql_client(ctx)
+    if not client:
+        client, graphql_path = _graphql_client(ctx)
+
+    try:
+        for row in _enterprise_admins_from_owner_info(
+            enterprise_data, ctx, client, graphql_path
+        ):
+            seen_node_ids.add(row["node_id"])
+            yield row
+        owner_info_completed = True
+    except Exception as e:
+        logger.warning(
+            "Unable to collect enterprise owners from ownerInfo.admins for "
+            "enterprise '%s'; trying organization.enterpriseOwners fallback: %s",
+            ctx.enterprise_name,
+            e,
+            extra={"resource": "enterprise_admins", "phase": "resource_iteration"},
+        )
+
+    if owner_info_completed and seen_node_ids:
+        return
+
+    for row in _enterprise_admins_from_organizations(enterprise_data, ctx):
+        if row["node_id"] in seen_node_ids:
+            continue
+        seen_node_ids.add(row["node_id"])
+        yield row
+
+
+def _enterprise_admin_row(
+    node: dict[str, Any], enterprise_data: Enterprise, ctx: SourceContext
+) -> dict[str, Any] | None:
+    node_id = node.get("id")
+    if not node_id:
+        return None
+    return {
+        "node_id": node_id,
+        "login": node.get("login"),
+        "assignment": "direct",
+        "role_id": "owners",
+        "enterprise_node_id": enterprise_data.id,
+        "enterprise_slug": ctx.enterprise_name,
+    }
+
+
+def _enterprise_admins_from_owner_info(
+    enterprise_data: Enterprise,
+    ctx: SourceContext,
+    client: RESTClient,
+    graphql_path: str,
+):
     paginator = GraphQLCursorPaginator(
         page_info_path="data.enterprise.ownerInfo.admins.pageInfo",
         cursor_variable="after",
@@ -652,7 +709,6 @@ def enterprise_admins(enterprise_data: Enterprise, ctx: SourceContext):
         "query": ENTERPRISE_ADMINS_QUERY,
         "variables": {"slug": ctx.enterprise_name, "count": 100, "after": None},
     }
-    client, graphql_path = _graphql_client(ctx)
     for page_data in client.paginate(
         graphql_path,
         method="POST",
@@ -664,16 +720,60 @@ def enterprise_admins(enterprise_data: Enterprise, ctx: SourceContext):
             es_data = enterprise_object.get("enterprise", {})
             owner_info = es_data.get("ownerInfo") or {}
             for edge in (owner_info.get("admins") or {}).get("edges") or []:
-                node = edge.get("node")
-                if node and node.get("id"):
-                    yield {
-                        "node_id": node["id"],
-                        "login": node.get("login"),
-                        "assignment": "direct",
-                        "role_id": "owners",
-                        "enterprise_node_id": enterprise_data.id,
-                        "enterprise_slug": ctx.enterprise_name,
-                    }
+                row = _enterprise_admin_row(edge.get("node") or {}, enterprise_data, ctx)
+                if row:
+                    yield row
+
+
+def _enterprise_admins_from_organizations(
+    enterprise_data: Enterprise, ctx: SourceContext
+):
+    for org in getattr(ctx, "organizations", None) or []:
+        paginator = GraphQLCursorPaginator(
+            page_info_path="data.organization.enterpriseOwners.pageInfo",
+            cursor_variable="after",
+            cursor_field="endCursor",
+            has_next_field="hasNextPage",
+            allow_missing_page_info=True,
+        )
+        data = {
+            "query": ORGANIZATION_ENTERPRISE_OWNERS_QUERY,
+            "variables": {"login": org.org_name, "count": 100, "after": None},
+        }
+        client, graphql_path = graphql_client_and_path(
+            org.client, getattr(org, "graphql_client", None)
+        )
+        found_owner = False
+        try:
+            for page_data in client.paginate(
+                graphql_path,
+                method="POST",
+                json=data,
+                paginator=paginator,
+                data_selector="data",
+            ):
+                for organization_object in page_data:
+                    organization = organization_object.get("organization", {})
+                    for node in (organization.get("enterpriseOwners") or {}).get(
+                        "nodes"
+                    ) or []:
+                        row = _enterprise_admin_row(node, enterprise_data, ctx)
+                        if row:
+                            found_owner = True
+                            yield row
+        except Exception as e:
+            logger.warning(
+                "Unable to collect enterprise owners through organization '%s' "
+                "for enterprise '%s': %s",
+                org.org_name,
+                ctx.enterprise_name,
+                e,
+                extra={"resource": "enterprise_admins", "phase": "resource_iteration"},
+            )
+            continue
+
+        if found_owner:
+            return
 
 
 @app.transformer(