ingestr 0.13.80__tar.gz → 0.13.82__tar.gz

{ingestr-0.13.80 → ingestr-0.13.82}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.80
+Version: 0.13.82
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

{ingestr-0.13.80 → ingestr-0.13.82}/docs/supported-sources/facebook-ads.md

@@ -160,13 +160,19 @@ The `facebook_insights` table supports advanced configuration for breakdowns and
 1. **Default usage**: `facebook_insights`
    - Uses default breakdown and default fields
 
-2. **Custom breakdown**: `facebook_insights:breakdown_type`
-   - Uses specified breakdown with default fields
+2. **Predefined breakdown**: `facebook_insights:breakdown_type`
+   - Uses specified predefined breakdown with default fields
 
-3. **Custom breakdown + metrics**: `facebook_insights:breakdown_type:metric1,metric2,metric3`
-   - Uses specified breakdown with custom metrics
+3. **Predefined breakdown + custom metrics**: `facebook_insights:breakdown_type:metric1,metric2,metric3`
+   - Uses specified predefined breakdown with custom metrics
 
-#### Available Breakdown Types
+4. **Custom dimensions + metrics**: `facebook_insights:dimension1,dimension2:metric1,metric2,metric3`
+   - Uses custom dimensions with custom metrics
+
+5. **Level + dimensions + metrics**: `facebook_insights:level,dimension1,dimension2:metric1,metric2,metric3`
+   - Uses specified level with custom dimensions and metrics
+
+#### Available Predefined Breakdown Types
 
 - `ads_insights` (default)
 - `ads_insights_age_and_gender`
@@ -176,6 +182,30 @@ The `facebook_insights` table supports advanced configuration for breakdowns and
 - `ads_insights_dma`
 - `ads_insights_hourly_advertiser`
 
+#### Available Levels
+
+When using custom dimensions, you can specify one of these levels:
+- `account` - Account level insights
+- `campaign` - Campaign level insights  
+- `adset` - Ad set level insights
+- `ad` - Ad level insights
+
+Note: If multiple levels are specified in the dimensions list, the last valid level will be used and removed from the dimensions list.
+
+#### Common Dimensions
+
+You can use any valid Facebook Ads dimension in your custom configurations. Some commonly used dimensions include:
+- `age` - Age ranges
+- `gender` - Gender breakdown
+- `country` - Country breakdown
+- `region` - Region breakdown  
+- `platform_position` - Platform position
+- `publisher_platform` - Publisher platform
+- `impression_device` - Device type
+- `placement` - Ad placement
+
+Note: Not all dimension combinations are valid according to Facebook's API. Refer to [Facebook's Marketing API](https://developers.facebook.com/docs/marketing-api/insights/breakdowns/) documentation for valid dimension combinations.
+
 #### Examples
 
 ```sh
@@ -199,4 +229,32 @@ ingestr ingest \
   --source-table 'facebook_insights:ads_insights_country:impressions,clicks,spend,reach,cpm,ctr' \
   --dest-uri 'duckdb:///facebook.duckdb' \
   --dest-table 'dest.insights_by_country'
+
+# Custom dimensions (age and gender) with custom metrics
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights:age,gender:impressions,clicks,spend' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.insights_custom_dimensions'
+
+# Campaign level with custom dimensions and metrics
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights:campaign,age,gender:impressions,clicks,spend,reach' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.campaign_insights_demographics'
+
+# Ad level with geographic dimensions
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights:ad,country,region:clicks,impressions,spend' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.ad_insights_geographic'
+
+# Account level insights only (no additional dimensions)
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'facebook_insights:account:impressions,clicks,spend,reach' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.account_level_insights'
 ```

ingestr-0.13.82/docs/supported-sources/linear.md

@@ -0,0 +1,60 @@
+# Linear
+
+[Linear](https://linear.app/) is a purpose-built tool for planning and building products that provides issue tracking and project management for teams.
+
+ingestr supports Linear as a source.
+
+## URI format
+
+The URI format for Linear is:
+
+```plaintext
+linear://?api_key=<api_key>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Linear API.
+
+## Example usage
+
+Assuming your API key is `lin_api_123`, you can ingest teams into DuckDB using:
+
+```bash
+ingestr ingest
+--source-uri 'linear://?api_key=lin_api_123' \
+--source-table 'teams' \
+--dest-uri duckdb:///linear.duckdb \
+--dest-table 'dest.teams'
+```
+<img alt="linear" src="../media/linear.png"/>
+
+## Tables
+Linear source allows ingesting the following tables:
+
+- `issues`: Fetches all issues from your Linear workspace.
+- `projects`: Fetches project-level data.
+- `teams`: Fetches information about the teams configured in Linear.
+- `users`: Fetches users from your workspace.
+- `workflow_states`: Fetches workflow states used in your Linear workspace.
+- `cycles`: Fetches cycle information and planning data.
+- `attachments`: Fetches file attachments associated with issues.
+- `comments`: Fetches comments on issues and other entities.
+- `documents`: Fetches documents created in Linear.
+- `external_users`: Fetches information about external users.
+- `initiative`: Fetches initiative data for high-level planning.
+- `integrations`: Fetches integration configurations.
+- `labels`: Fetches labels used for categorizing issues.
+- `organization`: Fetches organization-level information.
+- `project_updates`: Fetches updates posted to projects.
+- `roadmaps`: Fetches roadmap data for strategic planning.
+- `roadmap_to_projects`: Fetches relationships between roadmaps and projects.
+- `team_memberships`: Fetches team membership information.
+- `initiative_to_project`: Fetches relationships between initiatives and projects.
+- `project_milestone`: Fetches project milestone data.
+- `project_status`: Fetches project status information.
+- `team`: Fetches team-level data.
+- `project`: Fetches individual project information.
+
+
+Use these as the `--source-table` parameter in the `ingestr ingest` command.

ingestr-0.13.82/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.82"

{ingestr-0.13.80 → ingestr-0.13.82}/ingestr/src/facebook_ads/__init__.py

@@ -22,12 +22,8 @@ from .settings import (
     DEFAULT_ADCREATIVE_FIELDS,
     DEFAULT_ADSET_FIELDS,
     DEFAULT_CAMPAIGN_FIELDS,
-    DEFAULT_INSIGHT_FIELDS,
     DEFAULT_LEAD_FIELDS,
     INSIGHT_FIELDS_TYPES,
-    INSIGHTS_BREAKDOWNS_OPTIONS,
-    INVALID_INSIGHTS_FIELDS,
-    TInsightsBreakdownOptions,
     TInsightsLevels,
 )
 
@@ -105,10 +101,9 @@ def facebook_insights_source(
     account_id: str = dlt.config.value,
     access_token: str = dlt.secrets.value,
     initial_load_past_days: int = 1,
-    fields: Sequence[str] = DEFAULT_INSIGHT_FIELDS,
-    attribution_window_days_lag: int = 7,
+    dimensions: Sequence[str] = None,
+    fields: Sequence[str] = None,
     time_increment_days: int = 1,
-    breakdowns: TInsightsBreakdownOptions = "ads_insights",
     action_breakdowns: Sequence[str] = ALL_ACTION_BREAKDOWNS,
     level: TInsightsLevels = "ad",
     action_attribution_windows: Sequence[str] = ALL_ACTION_ATTRIBUTION_WINDOWS,
@@ -155,6 +150,11 @@ def facebook_insights_source(
     if start_date is None:
         start_date = pendulum.today().subtract(days=initial_load_past_days)
 
+    if dimensions is None:
+        dimensions = []
+    if fields is None:
+        fields = []
+
     columns = {}
     for field in fields:
         if field in INSIGHT_FIELDS_TYPES:
@@ -187,15 +187,9 @@ def facebook_insights_source(
             query = {
                 "level": level,
                 "action_breakdowns": list(action_breakdowns),
-                "breakdowns": list(
-                    INSIGHTS_BREAKDOWNS_OPTIONS[breakdowns]["breakdowns"]
-                ),
+                "breakdowns": dimensions,
                 "limit": batch_size,
-                "fields": list(
-                    set(fields)
-                    .union(INSIGHTS_BREAKDOWNS_OPTIONS[breakdowns]["fields"])
-                    .difference(INVALID_INSIGHTS_FIELDS)
-                ),
+                "fields": fields,
                 "time_increment": time_increment_days,
                 "action_attribution_windows": list(action_attribution_windows),
                 "time_ranges": [

{ingestr-0.13.80 → ingestr-0.13.82}/ingestr/src/facebook_ads/helpers.py

@@ -229,3 +229,49 @@ def notify_on_token_expiration(access_token_expires_at: int = None) -> None:
             logger.error(
                 f"Access Token expires in {humanize.precisedelta(pendulum.now() - expires_at)}. Replace the token now!"
             )
+
+
+def parse_insights_table_to_source_kwargs(table: str) -> DictStrAny:
+    import typing
+
+    from ingestr.src.facebook_ads.settings import (
+        INSIGHTS_BREAKDOWNS_OPTIONS,
+        TInsightsBreakdownOptions,
+        TInsightsLevels,
+    )
+
+    parts = table.split(":")
+
+    source_kwargs = {}
+
+    breakdown_type = parts[1]
+
+    valid_breakdowns = list(typing.get_args(TInsightsBreakdownOptions))
+    if breakdown_type in valid_breakdowns:
+        dimensions = INSIGHTS_BREAKDOWNS_OPTIONS[breakdown_type]["breakdowns"]
+        fields = INSIGHTS_BREAKDOWNS_OPTIONS[breakdown_type]["fields"]
+        source_kwargs["dimensions"] = dimensions
+        source_kwargs["fields"] = fields
+    else:
+        dimensions = breakdown_type.split(",")
+        valid_levels = list(typing.get_args(TInsightsLevels))
+        level = None
+        for valid_level in reversed(valid_levels):
+            if valid_level in dimensions:
+                level = valid_level
+                dimensions.remove(valid_level)
+                break
+
+        source_kwargs["level"] = level
+        source_kwargs["dimensions"] = dimensions
+
+    # If custom metrics are provided, parse them
+    if len(parts) == 3:
+        fields = [f.strip() for f in parts[2].split(",") if f.strip()]
+        if not fields:
+            raise ValueError(
+                "Custom metrics must be provided after the second colon in format: facebook_insights:breakdown_type:metric1,metric2..."
+            )
+        source_kwargs["fields"] = fields
+
+    return source_kwargs