cartography 0.109.0rc2__tar.gz → 0.110.0rc1__tar.gz

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/AGENTS.md

@@ -829,30 +829,95 @@ def start_your_service_ingestion(neo4j_session: neo4j.Session, config: Config) -
 
 ## 🚨 Error Handling {#error-handling}
 
-Follow these principles for robust error handling:
+### Core Principles
 
-### DO: Catch Specific Exceptions
-```python
-import requests
+**Fail Loudly and Early**: Cartography follows the principle of failing loudly and as early as possible. This ensures sync jobs don't continue with partial or incorrect data, which is critical because Cartography has cleanup jobs that delete nodes and relationships it believes are stale. If we don't fail early, we may delete data that we shouldn't have.
+
+**Never Bury Exceptions**: We should never preemptively catch base exceptions unless we know specifically how to handle them. This helps with traceability and debugging.
+
+### GET Functions: Keep It Simple
 
+All `get_*` functions should be "dumb" - just fetch data and let exceptions propagate naturally:
 
+```python
+@timeit
 def get_users(api_key: str) -> dict[str, Any]:
-    try:
-        response = requests.get(f"https://api.service.com/users", headers={"Authorization": f"Bearer {api_key}"})
-        response.raise_for_status()
-        return response.json()
-    except requests.exceptions.HTTPError as e:
-        if e.response.status_code == 401:
-            logger.error("Invalid API key")
-        elif e.response.status_code == 429:
-            logger.error("Rate limit exceeded")
-        raise
-    except requests.exceptions.RequestException as e:
-        logger.error(f"Network error: {e}")
-        raise
+    """
+    Fetch data from external API
+    Should be simple and let exceptions propagate naturally
+    """
+    response = requests.get(
+        "https://api.service.com/users",
+        headers={"Authorization": f"Bearer {api_key}"},
+        timeout=(60, 60),
+    )
+    response.raise_for_status()  # Let HTTP errors propagate
+    return response.json()
 ```
 
-### DON'T: Catch Base Exception
+**Key Principles for `get()` Functions:**
+
+1. **No Preemptive Exception Handling**: Avoid adding try/except blocks in `get()` functions unless you know the specific ways the code can fail and how to handle them.
+   ```python
+   # ❌ DON'T: Preemptively add complex error handling in get()
+   def get_users(api_key: str) -> dict[str, Any]:
+       try:
+           response = requests.get(...)
+           response.raise_for_status()
+           return response.json()
+       except requests.exceptions.HTTPError as e:
+           if e.response.status_code == 401:
+               logger.error("Invalid API key")
+           elif e.response.status_code == 429:
+               logger.error("Rate limit exceeded")
+           raise
+       except requests.exceptions.RequestException as e:
+           logger.error(f"Network error: {e}")
+           raise
+
+   # ✅ DO: Code for the happy path. Keep it simple and let errors propagate.
+   def get_users(api_key: str) -> dict[str, Any]:
+       response = requests.get(...)
+       response.raise_for_status()
+       return response.json()
+   ```
+
+2. **Use Decorators for Common Patterns**: For AWS modules, use `@aws_handle_regions` to handle common AWS errors:
+   ```python
+   @timeit
+   @aws_handle_regions  # Handles region availability, throttling, etc.
+   def get_ec2_instances(boto3_session: boto3.session.Session, region: str) -> list[dict[str, Any]]:
+       client = boto3_session.client("ec2", region_name=region)
+       return client.describe_instances()["Reservations"]
+   ```
+
+3. **Fail Loudly**: If an error occurs, let it propagate up to the caller. This helps users identify and fix issues quickly:
+   ```python
+   # ❌ DON'T: Silently continue on error
+   def get_data() -> dict[str, Any]:
+       try:
+           return api.get_data()
+       except Exception:
+           return {}  # Silently continue with empty data
+
+   # ✅ DO: Let errors propagate
+   def get_data() -> dict[str, Any]:
+       return api.get_data()  # Let errors propagate to caller
+   ```
+
+4. **Timeout Configuration**: Set appropriate timeouts to avoid hanging:
+   ```python
+   # ✅ DO: Set timeouts
+   response = session.post(
+       "https://api.service.com/users",
+       json=payload,
+       timeout=(60, 60),  # (connect_timeout, read_timeout)
+   )
+   ```
+
+### Exception Handling Guidelines
+
+**DON'T: Catch Base Exception**
 ```python
 # ❌ Don't do this - makes debugging impossible
 try:
@@ -862,6 +927,32 @@ except Exception:
     pass  # Silently continue - BAD!
 ```
 
+**DON'T: Prematurely Optimize Error Handling**
+```python
+# ❌ Don't add complex error handling unless you know the specific failure modes
+def get_data() -> dict[str, Any]:
+    try:
+        # Complex error handling for unknown failure modes
+        response = api.get_data()
+        if response.status_code == 200:
+            return response.json()
+        elif response.status_code == 401:
+            # Handle auth error
+            pass
+        elif response.status_code == 429:
+            # Handle rate limiting
+            pass
+        # ... more unknown error cases
+    except Exception as e:
+        # Generic exception handling
+        logger.error(f"Unknown error: {e}")
+        raise
+```
+
+### Best-Effort Mode
+
+Some modules support best-effort mode via flags like `--aws-best-effort-mode` and `--entra-best-effort-mode`. These modes allow the user to override and continue running even if exceptions are encountered. This works well together with Cartography's main error handling principle: fail loudly and early, so you should not feel bad about not handling all possible exceptions.
+
 ### Required vs Optional Field Access
 
 ```python

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cartography
-Version: 0.109.0rc2
+Version: 0.110.0rc1
 Summary: Explore assets and their relationships across your technical infrastructure.
 Maintainer: Cartography Contributors
 License: apache2

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.109.0rc2'
-__version_tuple__ = version_tuple = (0, 109, 0, 'rc2')
+__version__ = version = '0.110.0rc1'
+__version_tuple__ = version_tuple = (0, 110, 0, 'rc1')

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/cli.py

@@ -254,6 +254,14 @@ class CLI:
                 "The name of environment variable containing Entra Client Secret for Service Principal Authentication."
             ),
         )
+        parser.add_argument(
+            "--entra-best-effort-mode",
+            action="store_true",
+            help=(
+                "Enable Entra ID sync best effort mode. This will allow cartography to continue "
+                "syncing other Entra ID entities and delay raising an exception until the end of the sync."
+            ),
+        )
         parser.add_argument(
             "--aws-requested-syncs",
             type=str,
@@ -700,6 +708,15 @@ class CLI:
                 "Required if you are using the Trivy module. Ignored otherwise."
             ),
         )
+        parser.add_argument(
+            "--trivy-results-dir",
+            type=str,
+            default=None,
+            help=(
+                "Path to a directory containing Trivy JSON results on disk. "
+                "Required if you are using the Trivy module with local results."
+            ),
+        )
         parser.add_argument(
             "--scaleway-org",
             type=str,
@@ -1089,6 +1106,9 @@ class CLI:
         if config.trivy_s3_prefix:
             logger.debug(f"Trivy S3 prefix: {config.trivy_s3_prefix}")
 
+        if config.trivy_results_dir:
+            logger.debug(f"Trivy results dir: {config.trivy_results_dir}")
+
         # Scaleway config
         if config.scaleway_secret_key_env_var:
             logger.debug(
@@ -1118,6 +1138,8 @@ class CLI:
             config.sentinelone_api_token = os.environ.get(
                 config.sentinelone_api_token_env_var
             )
+        else:
+            config.sentinelone_api_token = None
 
         # Run cartography
         try:

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/config.py

@@ -51,6 +51,9 @@ class Config:
     :param entra_client_id: Client Id for connecting in a Service Principal Authentication approach. Optional.
     :type entra_client_secret: str
     :param entra_client_secret: Client Secret for connecting in a Service Principal Authentication approach. Optional.
+    :type entra_best_effort_mode: bool
+    :param entra_best_effort_mode: If True, Entra ID sync will continue on errors and raise an aggregated
+        exception at the end of the sync. If False (default), exceptions will be raised immediately.
     :type aws_requested_syncs: str
     :param aws_requested_syncs: Comma-separated list of AWS resources to sync. Optional.
     :type aws_guardduty_severity_threshold: str
@@ -152,6 +155,8 @@ class Config:
     :param trivy_s3_bucket: The S3 bucket name containing Trivy scan results. Optional.
     :type trivy_s3_prefix: str
     :param trivy_s3_prefix: The S3 prefix path containing Trivy scan results. Optional.
+    :type trivy_results_dir: str
+    :param trivy_results_dir: Local directory containing Trivy scan results. Optional.
     :type scaleway_access_key: str
     :param scaleway_access_key: Scaleway access key. Optional.
     :type scaleway_secret_key: str
@@ -162,6 +167,8 @@ class Config:
     :param sentinelone_api_url: SentinelOne API URL. Optional.
     :type sentinelone_api_token: string
     :param sentinelone_api_token: SentinelOne API token for authentication. Optional.
+    :type sentinelone_api_token_env_var: string
+    :param sentinelone_api_token_env_var: The name of an environment variable containing the SentinelOne API token. Optional.
     :type sentinelone_account_ids: list[str]
     :param sentinelone_account_ids: List of SentinelOne account IDs to sync. Optional.
     """
@@ -187,6 +194,7 @@ class Config:
         entra_tenant_id=None,
         entra_client_id=None,
         entra_client_secret=None,
+        entra_best_effort_mode=False,
         aws_requested_syncs=None,
         aws_guardduty_severity_threshold=None,
         analysis_job_directory=None,
@@ -243,11 +251,13 @@ class Config:
         airbyte_api_url=None,
         trivy_s3_bucket=None,
         trivy_s3_prefix=None,
+        trivy_results_dir=None,
         scaleway_access_key=None,
         scaleway_secret_key=None,
         scaleway_org=None,
         sentinelone_api_url=None,
         sentinelone_api_token=None,
+        sentinelone_api_token_env_var=None,
         sentinelone_account_ids=None,
     ):
         self.neo4j_uri = neo4j_uri
@@ -271,6 +281,7 @@ class Config:
         self.entra_tenant_id = entra_tenant_id
         self.entra_client_id = entra_client_id
         self.entra_client_secret = entra_client_secret
+        self.entra_best_effort_mode = entra_best_effort_mode
         self.aws_requested_syncs = aws_requested_syncs
         self.aws_guardduty_severity_threshold = aws_guardduty_severity_threshold
         self.analysis_job_directory = analysis_job_directory
@@ -327,9 +338,11 @@ class Config:
         self.airbyte_api_url = airbyte_api_url
         self.trivy_s3_bucket = trivy_s3_bucket
         self.trivy_s3_prefix = trivy_s3_prefix
+        self.trivy_results_dir = trivy_results_dir
         self.scaleway_access_key = scaleway_access_key
         self.scaleway_secret_key = scaleway_secret_key
         self.scaleway_org = scaleway_org
         self.sentinelone_api_url = sentinelone_api_url
         self.sentinelone_api_token = sentinelone_api_token
+        self.sentinelone_api_token_env_var = sentinelone_api_token_env_var
         self.sentinelone_account_ids = sentinelone_account_ids

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/intel/aws/cloudtrail_management_events.py

@@ -223,6 +223,13 @@ def transform_assume_role_events_to_role_assumptions(
 
         cloudtrail_event = json.loads(event["CloudTrailEvent"])
 
+        # Skip events with null requestParameters since we can't extract roleArn
+        if not cloudtrail_event.get("requestParameters"):
+            logger.debug(
+                f"Skipping CloudTrail AssumeRole event due to missing requestParameters. Event: {event.get('EventId', 'unknown')}"
+            )
+            continue
+
         if cloudtrail_event.get("userIdentity", {}).get("arn"):
             source_principal = cloudtrail_event["userIdentity"]["arn"]
             destination_principal = cloudtrail_event["requestParameters"]["roleArn"]
@@ -298,6 +305,13 @@ def transform_saml_role_events_to_role_assumptions(
 
         cloudtrail_event = json.loads(event["CloudTrailEvent"])
 
+        # Skip events with null requestParameters since we can't extract roleArn
+        if not cloudtrail_event.get("requestParameters"):
+            logger.debug(
+                f"Skipping CloudTrail AssumeRoleWithSAML event due to missing requestParameters. Event: {event.get('EventId', 'unknown')}"
+            )
+            continue
+
         response_elements = cloudtrail_event.get("responseElements", {})
         assumed_role_user = response_elements.get("assumedRoleUser", {})
 
@@ -370,6 +384,13 @@ def transform_web_identity_role_events_to_role_assumptions(
 
         cloudtrail_event = json.loads(event["CloudTrailEvent"])
 
+        # Skip events with null requestParameters since we can't extract roleArn
+        if not cloudtrail_event.get("requestParameters"):
+            logger.debug(
+                f"Skipping CloudTrail AssumeRoleWithWebIdentity event due to missing requestParameters. Event: {event.get('EventId', 'unknown')}"
+            )
+            continue
+
         user_identity = cloudtrail_event.get("userIdentity", {})
 
         if user_identity.get("type") == "WebIdentityUser" and user_identity.get(

cartography-0.110.0rc1/cartography/intel/aws/eventbridge.py

@@ -0,0 +1,91 @@
+import logging
+from typing import Any
+from typing import Dict
+from typing import List
+
+import boto3
+import neo4j
+
+from cartography.client.core.tx import load
+from cartography.graph.job import GraphJob
+from cartography.intel.aws.ec2.util import get_botocore_config
+from cartography.models.aws.eventbridge.rule import EventBridgeRuleSchema
+from cartography.util import aws_handle_regions
+from cartography.util import timeit
+
+logger = logging.getLogger(__name__)
+
+
+@timeit
+@aws_handle_regions
+def get_eventbridge_rules(
+    boto3_session: boto3.Session, region: str
+) -> List[Dict[str, Any]]:
+    client = boto3_session.client(
+        "events", region_name=region, config=get_botocore_config()
+    )
+    paginator = client.get_paginator("list_rules")
+    rules = []
+
+    for page in paginator.paginate():
+        rules.extend(page.get("Rules", []))
+
+    return rules
+
+
+@timeit
+def load_eventbridge_rules(
+    neo4j_session: neo4j.Session,
+    data: List[Dict[str, Any]],
+    region: str,
+    current_aws_account_id: str,
+    aws_update_tag: int,
+) -> None:
+    logger.info(
+        f"Loading EventBridge {len(data)} rules for region '{region}' into graph.",
+    )
+    load(
+        neo4j_session,
+        EventBridgeRuleSchema(),
+        data,
+        lastupdated=aws_update_tag,
+        Region=region,
+        AWS_ID=current_aws_account_id,
+    )
+
+
+@timeit
+def cleanup(
+    neo4j_session: neo4j.Session,
+    common_job_parameters: Dict[str, Any],
+) -> None:
+    logger.debug("Running EventBridge cleanup job.")
+    GraphJob.from_node_schema(EventBridgeRuleSchema(), common_job_parameters).run(
+        neo4j_session
+    )
+
+
+@timeit
+def sync(
+    neo4j_session: neo4j.Session,
+    boto3_session: boto3.session.Session,
+    regions: List[str],
+    current_aws_account_id: str,
+    update_tag: int,
+    common_job_parameters: Dict[str, Any],
+) -> None:
+    for region in regions:
+        logger.info(
+            f"Syncing EventBridge for region '{region}' in account '{current_aws_account_id}'.",
+        )
+
+        rules = get_eventbridge_rules(boto3_session, region)
+        load_eventbridge_rules(
+            neo4j_session,
+            rules,
+            region,
+            current_aws_account_id,
+            update_tag,
+        )
+
+    cleanup(neo4j_session, common_job_parameters)

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/intel/aws/resources.py

@@ -18,6 +18,7 @@ from . import eks
 from . import elasticache
 from . import elasticsearch
 from . import emr
+from . import eventbridge
 from . import glue
 from . import guardduty
 from . import iam
@@ -115,5 +116,6 @@ RESOURCE_FUNCTIONS: Dict[str, Callable[..., None]] = {
     "efs": efs.sync,
     "guardduty": guardduty.sync,
     "codebuild": codebuild.sync,
+    "eventbridge": eventbridge.sync,
     "glue": glue.sync,
 }

cartography-0.110.0rc1/cartography/intel/entra/__init__.py

@@ -0,0 +1,85 @@
+import asyncio
+import datetime
+import logging
+from traceback import TracebackException
+from typing import Awaitable
+from typing import Callable
+
+import neo4j
+
+from cartography.config import Config
+from cartography.intel.entra.resources import RESOURCE_FUNCTIONS
+from cartography.util import timeit
+
+logger = logging.getLogger(__name__)
+
+
+@timeit
+def start_entra_ingestion(neo4j_session: neo4j.Session, config: Config) -> None:
+    """
+    If this module is configured, perform ingestion of Entra data. Otherwise warn and exit
+    :param neo4j_session: Neo4J session for database interface
+    :param config: A cartography.config object
+    :return: None
+    """
+
+    if (
+        not config.entra_tenant_id
+        or not config.entra_client_id
+        or not config.entra_client_secret
+    ):
+        logger.info(
+            "Entra import is not configured - skipping this module. "
+            "See docs to configure.",
+        )
+        return
+
+    common_job_parameters = {
+        "UPDATE_TAG": config.update_tag,
+        "TENANT_ID": config.entra_tenant_id,
+    }
+
+    async def main() -> None:
+        failed_stages = []
+        exception_tracebacks = []
+
+        async def run_stage(name: str, func: Callable[..., Awaitable[None]]) -> None:
+            try:
+                await func(
+                    neo4j_session,
+                    config.entra_tenant_id,
+                    config.entra_client_id,
+                    config.entra_client_secret,
+                    config.update_tag,
+                    common_job_parameters,
+                )
+            except Exception as e:
+                if config.entra_best_effort_mode:
+                    timestamp = datetime.datetime.now()
+                    failed_stages.append(name)
+                    exception_traceback = TracebackException.from_exception(e)
+                    traceback_string = "".join(exception_traceback.format())
+                    exception_tracebacks.append(
+                        f"{timestamp} - Exception for stage {name}\n{traceback_string}"
+                    )
+                    logger.warning(
+                        f"Caught exception syncing {name}. entra-best-effort-mode is on so we are continuing "
+                        "on to the next Entra sync. All exceptions will be aggregated and re-logged at the end of the sync.",
+                        exc_info=True,
+                    )
+                else:
+                    logger.error("Error during Entra sync", exc_info=True)
+                    raise
+
+        for name, func in RESOURCE_FUNCTIONS:
+            await run_stage(name, func)
+
+        if failed_stages:
+            logger.error(
+                f"Entra sync failed for the following stages: {', '.join(failed_stages)}. "
+                "See the logs for more details.",
+            )
+            raise Exception("\n".join(exception_tracebacks))
+
+    # Execute all syncs in sequence
+    asyncio.run(main())

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/intel/entra/applications.py

@@ -172,12 +172,11 @@ async def get_app_role_assignments(
             )
             continue
         except Exception as e:
-            # Only catch truly unexpected errors - these should be rare
             logger.error(
                 f"Unexpected error when fetching app role assignments for application {app.app_id} ({app.display_name}): {e}",
                 exc_info=True,
             )
-            continue
+            raise
 
     logger.info(f"Retrieved {len(assignments)} app role assignments total")
     return assignments

{cartography-0.109.0rc2 → cartography-0.110.0rc1}/cartography/intel/entra/ou.py

@@ -43,7 +43,7 @@ async def get_entra_ous(client: GraphServiceClient) -> list[AdministrativeUnit]:
                 current_request = None
         except Exception as e:
             logger.error(f"Failed to retrieve administrative units: {str(e)}")
-            current_request = None
+            raise
 
     return all_units
 

cartography-0.110.0rc1/cartography/intel/entra/resources.py

@@ -0,0 +1,20 @@
+from cartography.intel.entra.applications import sync_entra_applications
+from cartography.intel.entra.groups import sync_entra_groups
+from cartography.intel.entra.ou import sync_entra_ous
+from cartography.intel.entra.users import sync_entra_users
+
+# This is a list so that we sync these resources in order.
+# Data shape: [("resource_name", sync_function), ...]
+# Each sync function will be called with the following arguments:
+#   - neo4j_session
+#   - config.entra_tenant_id
+#   - config.entra_client_id
+#   - config.entra_client_secret
+#   - config.update_tag
+#   - common_job_parameters
+RESOURCE_FUNCTIONS = [
+    ("users", sync_entra_users),
+    ("groups", sync_entra_groups),
+    ("ous", sync_entra_ous),
+    ("applications", sync_entra_applications),
+]