ingestr 0.13.54__tar.gz → 0.13.55__tar.gz

{ingestr-0.13.54 → ingestr-0.13.55}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.54
+Version: 0.13.55
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

ingestr-0.13.55/docs/supported-sources/facebook-ads.md

@@ -0,0 +1,202 @@
+# Facebook Ads
+
+Facebook Ads is the advertising platform that helps users to create targeted ads on Facebook, Instagram and Messenger.
+
+ingestr supports Facebook Ads as a source.
+
+## URI format
+
+The URI format for Facebook Ads is as follows:
+
+```plaintext
+facebookads://?access_token=<access_token>&account_id=<account_id>
+```
+
+URI parameters:
+
+- `access_token` is associated with Business Facebook App.
+- `account_id` is associated with Ad manager.
+
+Both are used for authentication with Facebook Ads API.
+
+The URI is used to connect to Facebook Ads API for extracting data.
+
+## Setting up a Facebook Ads Integration
+
+Facebook Ads requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/facebook_ads#setup-guide).
+
+Once you complete the guide, you should have an access token and an Account ID. Let's say your `access_token` is `abcdef` and `account_id` is `1234`, here's a sample command that will copy the data from Facebook Ads into a DuckDB database:
+
+```sh
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'campaigns' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.campaigns'
+```
+
+The result of this command will be a table in the `facebook.duckdb` database.
+
+## Tables
+
+Facebook Ads source allows ingesting the following sources into separate tables:
+
+- `campaigns`: Retrieves campaign data with fields:
+  - `id`
+  - `updated_time`
+  - `created_time`
+  - `name`
+  - `status`
+  - `effective_status`
+  - `objective`
+  - `start_time`
+  - `stop_time`
+  - `daily_budget`
+  - `lifetime_budget`
+
+- `ad_sets`: Retrieves ad set data with fields:
+  - `id`
+  - `updated_time`
+  - `created_time`
+  - `name`
+  - `status`
+  - `effective_status`
+  - `campaign_id`
+  - `start_time`
+  - `end_time`
+  - `daily_budget`
+  - `lifetime_budget`
+  - `optimization_goal`
+  - `promoted_object`
+  - `billing_event`
+  - `bid_amount`
+  - `bid_strategy`
+  - `targeting`
+
+- `leads`: Retrieves lead data with fields:
+  - `id`
+  - `created_time`
+  - `ad_id`
+  - `ad_name`
+  - `adset_id`
+  - `adset_name`
+  - `campaign_id`
+  - `campaign_name`
+  - `form_id`
+  - `field_data`
+
+- `ads_creatives`: Retrieves ad creative data with fields:
+  - `id`
+  - `name`
+  - `status`
+  - `thumbnail_url`
+  - `object_story_spec`
+  - `effective_object_story_id`
+  - `call_to_action_type`
+  - `object_type`
+  - `template_url`
+  - `url_tags`
+  - `instagram_actor_id`
+  - `product_set_id`
+
+- `ads`: Retrieves ad data with fields:
+  - `id`
+  - `updated_time`
+  - `created_time`
+  - `name`
+  - `status`
+  - `effective_status`
+  - `adset_id`
+  - `campaign_id`
+  - `creative`
+  - `targeting`
+  - `tracking_specs`
+  - `conversion_specs`
+
+- `facebook_insights`: Retrieves insights data with fields:
+  - `campaign_id`
+  - `adset_id`
+  - `ad_id`
+  - `date_start`
+  - `date_stop`
+  - `reach`
+  - `impressions`
+  - `frequency`
+  - `clicks`
+  - `unique_clicks`
+  - `ctr`
+  - `unique_ctr`
+  - `cpc`
+  - `cpm`
+  - `cpp`
+  - `spend`
+  - `actions`
+  - `action_values`
+  - `cost_per_action_type`
+  - `website_ctr`
+  - `account_currency`
+  - `ad_click_actions`
+  - `ad_name`
+  - `adset_name`
+  - `campaign_name`
+  - `country`
+  - `dma`
+  - `full_view_impressions`
+  - `full_view_reach`
+  - `inline_link_click_ctr`
+  - `outbound_clicks`
+  - `social_spend`
+  - `conversions`
+  - `video_thruplay_watched_actions`
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+### Facebook Insights Custom Configuration
+
+The `facebook_insights` table supports advanced configuration for breakdowns and custom metrics:
+
+#### Format Options
+
+1. **Default usage**: `facebook_insights`
+   - Uses default breakdown and default fields
+
+2. **Custom breakdown**: `facebook_insights:breakdown_type`
+   - Uses specified breakdown with default fields
+
+3. **Custom breakdown + metrics**: `facebook_insights:breakdown_type:metric1,metric2,metric3`
+   - Uses specified breakdown with custom metrics
+
+#### Available Breakdown Types
+
+- `ads_insights` (default)
+- `ads_insights_age_and_gender`
+- `ads_insights_country`
+- `ads_insights_platform_and_device`
+- `ads_insights_region`
+- `ads_insights_dma`
+- `ads_insights_hourly_advertiser`
+
+#### Examples
+
+```sh
+# Default facebook_insights
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.insights'
+
+# Age and gender breakdown with default metrics
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights:ads_insights_age_and_gender' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.insights_demographics'
+
+# Country breakdown with custom metrics
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights:ads_insights_country:impressions,clicks,spend,reach,cpm,ctr' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.insights_by_country'
+```

ingestr-0.13.55/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.55"

{ingestr-0.13.54 → ingestr-0.13.55}/ingestr/src/facebook_ads/__init__.py

@@ -116,6 +116,8 @@ def facebook_insights_source(
     batch_size: int = 50,
     request_timeout: int = 300,
     app_api_version: str = None,
+    start_date: pendulum.DateTime | None = None,
+    end_date: pendulum.DateTime | None = None,
 ) -> DltResource:
     """Incrementally loads insight reports with defined granularity level, fields, breakdowns etc.
 
@@ -148,27 +150,32 @@ def facebook_insights_source(
         account_id, access_token, request_timeout, app_api_version
     )
 
-    # we load with a defined lag
-    initial_load_start_date = pendulum.today().subtract(days=initial_load_past_days)
-    initial_load_start_date_str = initial_load_start_date.isoformat()
+    if start_date is None:
+        start_date = pendulum.today().subtract(days=initial_load_past_days)
+
+    columns = {}
+    for field in fields:
+        if field in INSIGHT_FIELDS_TYPES:
+            columns[field] = INSIGHT_FIELDS_TYPES[field]
 
     @dlt.resource(
         primary_key=INSIGHTS_PRIMARY_KEY,
         write_disposition="merge",
-        columns=INSIGHT_FIELDS_TYPES,
+        columns=columns,
     )
     def facebook_insights(
         date_start: dlt.sources.incremental[str] = dlt.sources.incremental(
             "date_start",
-            initial_value=initial_load_start_date_str,
+            initial_value=start_date.isoformat(),
+            end_value=end_date.isoformat() if end_date else None,
             range_end="closed",
             range_start="closed",
+            lag=attribution_window_days_lag * 24 * 60 * 60,  # Convert days to seconds
         ),
     ) -> Iterator[TDataItems]:
-        start_date = get_start_date(date_start, attribution_window_days_lag)
+        start_date = get_start_date(date_start)
         end_date = pendulum.now()
 
-        # fetch insights in incremental day steps
         while start_date <= end_date:
             query = {
                 "level": level,
@@ -193,7 +200,10 @@ def facebook_insights_source(
                     }
                 ],
             }
-            job = execute_job(account.get_insights(params=query, is_async=True))
+            job = execute_job(
+                account.get_insights(params=query, is_async=True),
+                insights_max_async_sleep_seconds=10,
+            )
             yield list(map(process_report_item, job.get_result()))
             start_date = start_date.add(days=time_increment_days)
 

{ingestr-0.13.54 → ingestr-0.13.55}/ingestr/src/facebook_ads/helpers.py

@@ -31,14 +31,13 @@ from .settings import (
 
 def get_start_date(
     incremental_start_date: dlt.sources.incremental[str],
-    attribution_window_days_lag: int = 7,
 ) -> pendulum.DateTime:
     """
     Get the start date for incremental loading of Facebook Insights data.
     """
     start_date: pendulum.DateTime = ensure_pendulum_datetime(
         incremental_start_date.start_value
-    ).subtract(days=attribution_window_days_lag)
+    )
 
     # facebook forgets insights so trim the lag and warn
     min_start_date = pendulum.today().subtract(
@@ -65,7 +64,6 @@ def process_report_item(item: AbstractObject) -> DictStrAny:
     for pki in INSIGHTS_PRIMARY_KEY:
         if pki not in d:
             d[pki] = "no_" + pki
-
     return d
 
 
@@ -138,7 +136,7 @@ def execute_job(
 ) -> AbstractCrudObject:
     status: str = None
     time_start = time.time()
-    sleep_time = 10
+    sleep_time = 3
     while status != "Job Completed":
         duration = time.time() - time_start
         job = job.api_get()

{ingestr-0.13.54 → ingestr-0.13.55}/ingestr/src/facebook_ads/settings.py

@@ -112,6 +112,8 @@ DEFAULT_INSIGHT_FIELDS = (
     "social_spend",
     "spend",
     "website_ctr",
+    "conversions",
+    "video_thruplay_watched_actions",
 )
 
 TInsightsLevels = Literal["account", "campaign", "adset", "ad"]

ingestr-0.13.55/ingestr/src/facebook_ads/utils.py

@@ -0,0 +1,39 @@
+from typing import Dict
+
+import dlt
+from dlt.common.configuration.inject import with_config
+from dlt.sources.helpers import requests
+
+
+@with_config(sections=("sources", "facebook_ads"))
+def debug_access_token(
+    access_token: str = dlt.secrets.value,
+    client_id: str = dlt.secrets.value,
+    client_secret: str = dlt.secrets.value,
+) -> str:
+    """Debugs the `access_token` providing info on expiration time, scopes etc. If arguments are not provides, `dlt` will inject them from configuration"""
+    debug_url = f"https://graph.facebook.com/debug_token?input_token={access_token}&access_token={client_id}|{client_secret}"
+    response = requests.get(debug_url)
+    data: Dict[str, str] = response.json()
+
+    if "error" in data:
+        raise Exception(f"Error debugging token: {data['error']}")
+
+    return data["data"]
+
+
+@with_config(sections=("sources", "facebook_ads"))
+def get_long_lived_token(
+    access_token: str = dlt.secrets.value,
+    client_id: str = dlt.secrets.value,
+    client_secret: str = dlt.secrets.value,
+) -> str:
+    """Gets the long lived access token (60 days) from `access_token`. If arguments are not provides, `dlt` will inject them from configuration"""
+    exchange_url = f"https://graph.facebook.com/v13.0/oauth/access_token?grant_type=fb_exchange_token&client_id={client_id}&client_secret={client_secret}&fb_exchange_token={access_token}"
+    response = requests.get(exchange_url)
+    data: Dict[str, str] = response.json()
+
+    if "error" in data:
+        raise Exception(f"Error refreshing token: {data['error']}")
+
+    return data["access_token"]

{ingestr-0.13.54 → ingestr-0.13.55}/ingestr/src/sources.py

@@ -747,11 +747,64 @@ class FacebookAdsSource:
         endpoint = None
         if table in ["campaigns", "ad_sets", "ad_creatives", "ads", "leads"]:
             endpoint = table
-        elif table in "facebook_insights":
+        elif table == "facebook_insights":
             return facebook_insights_source(
                 access_token=access_token[0],
                 account_id=account_id[0],
+                start_date=kwargs.get("interval_start"),
+                end_date=kwargs.get("interval_end"),
             ).with_resources("facebook_insights")
+        elif table.startswith("facebook_insights:"):
+            # Parse custom breakdowns and metrics from table name
+            # Supported formats:
+            # facebook_insights:breakdown_type
+            # facebook_insights:breakdown_type:metric1,metric2...
+            parts = table.split(":")
+
+            if len(parts) < 2 or len(parts) > 3:
+                raise ValueError(
+                    "Invalid facebook_insights format. Expected: facebook_insights:breakdown_type or facebook_insights:breakdown_type:metric1,metric2..."
+                )
+
+            breakdown_type = parts[1].strip()
+            if not breakdown_type:
+                raise ValueError(
+                    "Breakdown type must be provided in format: facebook_insights:breakdown_type"
+                )
+
+            # Validate breakdown type against available options from settings
+            import typing
+
+            from ingestr.src.facebook_ads.settings import TInsightsBreakdownOptions
+
+            # Get valid breakdown options from the type definition
+            valid_breakdowns = list(typing.get_args(TInsightsBreakdownOptions))
+
+            if breakdown_type not in valid_breakdowns:
+                raise ValueError(
+                    f"Invalid breakdown type '{breakdown_type}'. Valid options: {', '.join(valid_breakdowns)}"
+                )
+
+            source_kwargs = {
+                "access_token": access_token[0],
+                "account_id": account_id[0],
+                "start_date": kwargs.get("interval_start"),
+                "end_date": kwargs.get("interval_end"),
+                "breakdowns": breakdown_type,
+            }
+
+            # If custom metrics are provided, parse them
+            if len(parts) == 3:
+                fields = [f.strip() for f in parts[2].split(",") if f.strip()]
+                if not fields:
+                    raise ValueError(
+                        "Custom metrics must be provided after the second colon in format: facebook_insights:breakdown_type:metric1,metric2..."
+                    )
+                source_kwargs["fields"] = fields
+
+            return facebook_insights_source(**source_kwargs).with_resources(
+                "facebook_insights"
+            )
         else:
             raise ValueError(
                 f"Resource '{table}' is not supported for Facebook Ads source yet, if you are interested in it please create a GitHub issue at https://github.com/bruin-data/ingestr"

{ingestr-0.13.54 → ingestr-0.13.55}/ingestr/src/stripe_analytics/__init__.py

@@ -85,12 +85,14 @@ def incremental_stripe_source(
         created: Optional[Any] = dlt.sources.incremental(
             "created",
             initial_value=start_date_unix,
+            end_value=transform_date(end_date) if end_date is not None else None,
             range_end="closed",
             range_start="closed",
         ),
     ) -> Generator[Dict[Any, Any], Any, None]:
-        start_value = created.last_value
-        yield from pagination(endpoint, start_date=start_value, end_date=end_date)
+        yield from pagination(
+            endpoint, start_date=created.last_value, end_date=created.end_value
+        )
 
     for endpoint in endpoints:
         yield dlt.resource(

{ingestr-0.13.54 → ingestr-0.13.55}/package-lock.json

@@ -2250,9 +2250,9 @@
       }
     },
     "node_modules/vite": {
-      "version": "5.4.14",
-      "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.14.tgz",
-      "integrity": "sha512-EK5cY7Q1D8JNhSaPKVK4pwBFvaTmZxEnoKXLG/U9gmdDcihQGNzFlgIvaxezFR4glP1LsuiedwMBqCXH3wZccA==",
+      "version": "5.4.19",
+      "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.19.tgz",
+      "integrity": "sha512-qO3aKv3HoQC8QKiNSTuUM1l9o/XX3+c+VTgLHbJWHZGeTPVAg2XwazI9UWzoxjIJCGCV2zU60uqMzjeLZuULqA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {

{ingestr-0.13.54 → ingestr-0.13.55}/requirements_arm64.txt

@@ -353,7 +353,7 @@ proto-plus==1.26.1
     #   google-api-core
     #   google-cloud-bigquery-storage
     #   google-cloud-spanner
-protobuf==4.25.6
+protobuf==4.25.8
     # via
     #   google-ads
     #   google-analytics-data
@@ -447,7 +447,7 @@ rauth==0.7.3
     # via python-quickbooks
 redshift-connector==2.1.5
     # via -r requirements.in
-requests==2.32.3
+requests==2.32.4
     # via
     #   asana
     #   clickhouse-sqlalchemy
@@ -507,7 +507,7 @@ scramp==1.4.5
     # via redshift-connector
 semver==3.0.4
     # via dlt
-setuptools==78.1.0
+setuptools==78.1.1
     # via
     #   dlt
     #   python-quickbooks

ingestr-0.13.54/docs/supported-sources/facebook-ads.md

@@ -1,51 +0,0 @@
-# Facebook Ads
-
-Facebook Ads is the advertising platform that helps users to create targeted ads on Facebook, Instagram and Messenger.
-
-ingestr supports Facebook Ads as a source.
-
-## URI format
-
-The URI format for Facebook Ads is as follows:
-
-```plaintext
-facebookads://?access_token=<access_token>&account_id=<account_id>
-```
-
-URI parameters:
-
-- `access_token` is associated with Business Facebook App.
-- `account_id` is associated with Ad manager.
-
-Both are used for authentication with Facebook Ads API.
-
-The URI is used to connect to Facebook Ads API for extracting data.
-
-## Setting up a Facebook Ads Integration
-
-Facebook Ads requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/facebook_ads#setup-guide).
-
-Once you complete the guide, you should have an access token and an Account ID. Let's say your `access_token` is `abcdef` and `account_id` is `1234`, here's a sample command that will copy the data from Facebook Ads into a DuckDB database:
-
-```sh
-ingestr ingest \
-  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
-  --source-table 'campaigns' \
-  --dest-uri 'duckdb:///facebook.duckdb' \
-  --dest-table 'dest.campaigns'
-```
-
-The result of this command will be a table in the `facebook.duckdb` database.
-
-## Tables
-
-Facebook Ads source allows ingesting the following sources into separate tables:
-
-- `campaigns`: Retrieves all DEFAULT_CAMPAIGN_FIELDS.
-- `ad_sets`: Retrieves all DEFAULT_ADSET_FIELDS.
-- `leads`: Retrieves all DEFAULT_LEAD_FIELDS.
-- `ads_creatives`: Retrieves all DEFAULT_ADCREATIVE_FIELDS.
-- `ads`: Retrieves all DEFAULT_ADS_FIELDS.
-- `facebook_insights`: Retrieves all DEFAULT_INSIGHTS_FIELDS.
-
-Use these as `--source-table` parameter in the `ingestr ingest` command.

ingestr-0.13.54/ingestr/src/buildinfo.py

@@ -1 +0,0 @@
-version = "v0.13.54"