ingestr 0.12.1__tar.gz → 0.12.3__tar.gz

{ingestr-0.12.1 → ingestr-0.12.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.12.1
+Version: 0.12.3
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

{ingestr-0.12.1 → ingestr-0.12.3}/docs/.vitepress/config.mjs

@@ -78,6 +78,12 @@ export default defineConfig({
               { text: "SAP Hana", link: "/supported-sources/sap-hana.md" },
               { text: "Snowflake", link: "/supported-sources/snowflake.md" },
               { text: "SQLite", link: "/supported-sources/sqlite.md" },
+              {
+                text: "Experimental",
+                items: [
+                  { text: "Custom Queries", link: "/supported-sources/custom_queries.md" },
+                ],
+              },
             ],
           },
 
@@ -104,6 +110,7 @@ export default defineConfig({
               { text: "Shopify", link: "/supported-sources/shopify.md" },
               { text: "Slack", link: "/supported-sources/slack.md" },
               { text: "Stripe", link: "/supported-sources/stripe.md" },
+              { text: "TikTok Ads", link: "/supported-sources/tiktok-ads.md" },
               { text: "Zendesk", link: "/supported-sources/zendesk.md" },
             ],
           },

ingestr-0.12.3/docs/supported-sources/custom_queries.md

@@ -0,0 +1,50 @@
+# Custom Queries for SQL Sources
+
+ingestr has primarily supported table replication for SQL sources due to that being a common use case. However, there are certain scenarios where loading a table only is not possible:
+- you might want to load a subset of rows from a table
+- you might want to load a table that has a complex query that cannot be expressed as a simple table
+  - you could technically create a view in the database, but sometimes you don't have access/permissions to do so.
+- you might want to do incremental loads but the table you want to load does not have an incremental key, so it needs to be joined with another table that does.
+
+In order to support these scenarios, ingestr has added experimental support for custom queries.
+
+> [!DANGER]
+> This is an experimental feature, so do not expect it to work for all use cases. Please create an issue if you find a use case that doesn't work.
+
+## How to use custom queries
+
+To use a custom query, you can pass a `query:` prefix to the source name:
+
+```bash
+ingestr ingest \
+    --source-uri $POSTGRES_URI \
+    --dest-uri "duckdb:///mydb.db" \
+    --dest-table "public.output" \
+    --source-table "query:select oi.*, o.updated_at from order_items oi join orders o on oi.order_id = o.id" 
+ ```
+
+Ingestr uses SQLAlchemy to run the queries, therefore you can use any valid SQLAlchemy query.
+
+### Incremental loads
+
+Custom queries support incremental loads, but there are some caveats:
+- the incremental key must be a column that is returned by the query
+- the incremental key must be a datetime/timestamp column
+- you must do your own filtering in the query for the incremental load
+  - you can use the `interval_start` and `interval_end` variables to filter the data
+
+Here's an example of how to do an incremental load:
+
+```bash
+ingestr ingest \
+    --source-uri $POSTGRES_URI \
+    --dest-uri "duckdb:///mydb.db" \
+    --dest-table "public.output" \
+    --source-table "query:select oi.*, o.updated_at from order_items oi join orders o on oi.order_id = o.id where o.updated_at > :interval_start" \
+    --incremental-key updated_at \
+    --incremental-strategy merge \
+    --primary-key id
+```
+
+In this example, the query is filtering the data to only include rows where the `updated_at` column is greater than the `interval_start` variable.
+

ingestr-0.12.3/docs/supported-sources/tiktok-ads.md

@@ -0,0 +1,58 @@
+# TikTok Ads
+TikTok Ads is an advertising platform that enables businesses and marketers to create, manage, and analyze ad campaigns targeting TikTok's user base.
+
+Ingestr supports TikTok Ads as a Source.
+
+## URI format
+The URI format for TikTok Ads as a Source is as follows:
+
+```plaintext
+tiktok://?access_token=<ACCESS_TOKEN>&advertiser_ids=<advertiser_ids>&timezone=<timezone>
+```
+## URI parameters:
+- `access_token` (required): Used for authentication and is necessary to access reports through the TikTok Marketing API.
+- `advertiser_ids` (required): The comma-separated list of advertiser IDs to retrieve data for.
+- `timezone` (optional): The timezone to use for the data retrieval, you should set this value to the timezone of the advertiser account. Defaults to `UTC`.
+
+TikTok requires an `access_token` and `advertiser_ids` to retrieve reports from the TikTok marketing API. Please follow the guide to obtain the [credentials](https://business-api.tiktok.com/portal/docs?id=1738373141733378).
+
+## Table: Custom Reports
+Custom reports allow you to retrieve data based on specific `dimensions`, `metrics`, and `filters`.
+
+Custom Table Format:
+```
+custom:<dimensions>:<metrics>[:<filter_name,filter_values>]
+```
+### Parameters:
+- `dimensions`(required): A comma-separated list of [dimensions](https://business-api.tiktok.com/portal/docs?id=1751443956638721) to retrieve.
+- `metrics`(required): A comma-separated list of [metrics](https://business-api.tiktok.com/portal/docs?id=1751443967255553) to retrieve.
+- `filters` (optional): Filters are specified in the format `<filter_name=filter_values>`. 
+    - `filter_name`: The name of the filter (e.g. `campaign_ids`).
+    - `filter_values`: A comma-separated list of one or more values associated with the filter name (e.g., `camp_id123,camp_id456`). Only the `IN` filter type is supported. Learn more about [filters](https://business-api.tiktok.com/portal/docs?id=1751443975608321.). 
+
+> [!NOTE]
+> Ingestr will fetch data for the last 30 days and use the default page size of `1000`. You can override this by specifying the `interval_start` and `interval_end` parameters.
+
+### Example
+
+Retrieve data for campaigns with `campaign_ids` camp_id123 and camp_id456:
+```sh
+ingestr ingest \
+    --source-uri "tiktok://?access_token=token_123&advertiser_ids=0594720014,0594720015" \
+    --source-table "custom:campaign_id,stat_time_day:clicks,cpc" \
+    --dest-uri "duckdb:///campaigns.duckdb" \
+    --dest-table "dest.clicks"
+```
+
+The applied parameters for the report are:
+- dimensions: `campaign_id` and `country_code`
+- metrics: `clicks` and `cpc`
+- filters: `campaign_ids` for `camp_id123` and `camp_id456`
+
+
+This command will retrieve data for the specified date range and save it to the `dest.clicks` table in the DuckDB database.
+
+<img alt="titok_ads_img" src="../media/tiktok.png" />
+
+
+

{ingestr-0.12.1 → ingestr-0.12.3}/ingestr/src/factory.py

@@ -20,6 +20,7 @@ from ingestr.src.sources import (
     AirtableSource,
     AppsflyerSource,
     ArrowMemoryMappedSource,
+    AsanaSource,
     ChessSource,
     DynamoDBSource,
     FacebookAdsSource,
@@ -115,6 +116,7 @@ class SourceDestinationFactory:
         "mmap": ArrowMemoryMappedSource,
         "s3": S3Source,
         "dynamodb": DynamoDBSource,
+        "asana": AsanaSource,
         "tiktok": TikTokSource,
     }
     destinations: Dict[str, Type[DestinationProtocol]] = {

{ingestr-0.12.1 → ingestr-0.12.3}/ingestr/src/sources.py

@@ -3,17 +3,42 @@ import csv
 import json
 import os
 import re
-from datetime import date
-from typing import Any, Callable, Optional
+from datetime import date, datetime
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    Iterator,
+    List,
+    Literal,
+    Optional,
+    Union,
+)
 from urllib.parse import ParseResult, parse_qs, quote, urlparse
 
 import dlt
 import pendulum
-from dlt.common.configuration.specs import AwsCredentials
+import sqlalchemy
+from dlt.common.configuration.specs import (
+    AwsCredentials,
+)
+from dlt.common.libs.sql_alchemy import (
+    Engine,
+    MetaData,
+)
 from dlt.common.time import ensure_pendulum_datetime
-from dlt.common.typing import TSecretStrValue
+from dlt.common.typing import TDataItem, TSecretStrValue
+from dlt.extract import Incremental
 from dlt.sources.credentials import ConnectionStringCredentials
 from dlt.sources.sql_database import sql_table
+from dlt.sources.sql_database.helpers import TableLoader
+from dlt.sources.sql_database.schema_types import (
+    ReflectionLevel,
+    SelectAny,
+    Table,
+    TTypeAdapter,
+)
+from sqlalchemy import Column
 from sqlalchemy import types as sa
 from sqlalchemy.dialects import mysql
 
@@ -39,7 +64,7 @@ from ingestr.src.notion import notion_databases
 from ingestr.src.shopify import shopify_source
 from ingestr.src.slack import slack_source
 from ingestr.src.stripe_analytics import stripe_source
-from ingestr.src.table_definition import table_string_to_dataclass
+from ingestr.src.table_definition import TableDefinition, table_string_to_dataclass
 from ingestr.src.tiktok_ads import tiktok_source
 from ingestr.src.time import isotime
 from ingestr.src.zendesk import zendesk_chat, zendesk_support, zendesk_talk
@@ -48,6 +73,9 @@ from ingestr.src.zendesk.helpers.credentials import (
     ZendeskCredentialsToken,
 )
 
+TableBackend = Literal["sqlalchemy", "pyarrow", "pandas", "connectorx"]
+TQueryAdapter = Callable[[SelectAny, Table], SelectAny]
+
 
 class SqlSource:
     table_builder: Callable
@@ -59,7 +87,9 @@ class SqlSource:
         return False
 
     def dlt_source(self, uri: str, table: str, **kwargs):
-        table_fields = table_string_to_dataclass(table)
+        table_fields = TableDefinition(dataset="custom", table="custom")
+        if not table.startswith("query:"):
+            table_fields = table_string_to_dataclass(table)
 
         incremental = None
         if kwargs.get("incremental_key"):
@@ -87,6 +117,110 @@ class SqlSource:
                     query = query.order_by(kwargs.get("incremental_key"))
                 return query
 
+        defer_table_reflect = False
+        sql_backend = kwargs.get("sql_backend", "sqlalchemy")
+        if table.startswith("query:"):
+            if kwargs.get("sql_limit"):
+                raise ValueError(
+                    "sql_limit is not supported for custom queries, please apply the limit in the query instead"
+                )
+
+            sql_backend = "sqlalchemy"
+            defer_table_reflect = True
+            query_value = table.split(":", 1)[1]
+
+            # this is a very hacky version of the table_rows function. it is built this way to go around the dlt's table loader.
+            # I didn't want to write a full fledged sqlalchemy source for now, and wanted to benefit from the existing stuff to begin with.
+            # this is by no means a production ready solution, but it works for now.
+            # the core idea behind this implementation is to create a mock table instance with the columns that are absolutely necessary for the incremental load to work.
+            # the table loader will then use the query adapter callback to apply the actual query and load the rows.
+            def table_rows(
+                engine: Engine,
+                table: Union[Table, str],
+                metadata: MetaData,
+                chunk_size: int,
+                backend: TableBackend,
+                incremental: Optional[Incremental[Any]] = None,
+                table_adapter_callback: Callable[[Table], None] = None,  # type: ignore
+                reflection_level: ReflectionLevel = "minimal",
+                backend_kwargs: Dict[str, Any] = None,  # type: ignore
+                type_adapter_callback: Optional[TTypeAdapter] = None,
+                included_columns: Optional[List[str]] = None,
+                query_adapter_callback: Optional[TQueryAdapter] = None,
+                resolve_foreign_keys: bool = False,
+            ) -> Iterator[TDataItem]:
+                hints = {  # type: ignore
+                    "columns": [],
+                }
+                cols = []  # type: ignore
+
+                if incremental:
+                    switchDict = {
+                        int: sa.INTEGER,
+                        datetime: sa.TIMESTAMP,
+                        pendulum.Date: sa.DATE,
+                        pendulum.DateTime: sa.TIMESTAMP,
+                    }
+
+                    if incremental.last_value is not None:
+                        cols.append(
+                            Column(
+                                incremental.cursor_path,
+                                switchDict[type(incremental.last_value)],  # type: ignore
+                            )
+                        )
+                    else:
+                        cols.append(Column(incremental.cursor_path, sa.TIMESTAMP))  # type: ignore
+
+                table = Table(
+                    "query_result",
+                    metadata,
+                    *cols,
+                )
+
+                loader = TableLoader(
+                    engine,
+                    backend,
+                    table,
+                    hints["columns"],  # type: ignore
+                    incremental=incremental,
+                    chunk_size=chunk_size,
+                    query_adapter_callback=query_adapter_callback,
+                )
+                try:
+                    yield from loader.load_rows(backend_kwargs)
+                finally:
+                    if getattr(engine, "may_dispose_after_use", False):
+                        engine.dispose()
+
+            dlt.sources.sql_database.table_rows = table_rows
+
+            def query_adapter_callback(query, table, incremental=None, engine=None):
+                params = {}
+                if incremental:
+                    params["interval_start"] = (
+                        incremental.last_value
+                        if incremental.last_value is not None
+                        else datetime(year=1, month=1, day=1)
+                    )
+                    if incremental.end_value is not None:
+                        params["interval_end"] = incremental.end_value
+                else:
+                    if ":interval_start" in query_value:
+                        params["interval_start"] = (
+                            datetime.min
+                            if kwargs.get("interval_start") is None
+                            else kwargs.get("interval_start")
+                        )
+                    if ":interval_end" in query_value:
+                        params["interval_end"] = (
+                            datetime.max
+                            if kwargs.get("interval_end") is None
+                            else kwargs.get("interval_end")
+                        )
+
+                return sqlalchemy.text(query_value).bindparams(**params)
+
         def type_adapter_callback(sql_type):
             if isinstance(sql_type, mysql.SET):
                 return sa.JSON
@@ -97,7 +231,7 @@ class SqlSource:
             schema=table_fields.dataset,
             table=table_fields.table,
             incremental=incremental,
-            backend=kwargs.get("sql_backend", "sqlalchemy"),
+            backend=sql_backend,
             chunk_size=kwargs.get("page_size", None),
             reflection_level=reflection_level,
             query_adapter_callback=query_adapter_callback,
@@ -105,6 +239,7 @@ class SqlSource:
             table_adapter_callback=table_adapter_exclude_columns(
                 kwargs.get("sql_exclude_columns", [])
             ),
+            defer_table_reflect=defer_table_reflect,
         )
 
         return builder_res
@@ -1015,29 +1150,28 @@ class TikTokSource:
         if not access_token:
             raise ValueError("access_token is required to connect to TikTok")
 
-        time_zone = source_fields.get("time_zone", "UTC")
+        timezone = "UTC"
+        if source_fields.get("timezone") is not None:
+            timezone = source_fields.get("timezone")[0]  # type: ignore
+
+        advertiser_ids = source_fields.get("advertiser_ids")
+        if not advertiser_ids:
+            raise ValueError("advertiser_ids is required to connect to TikTok")
 
-        advertiser_id = source_fields.get("advertiser_id")
-        if not advertiser_id:
-            raise ValueError("advertiser_id is required to connect to TikTok")
+        advertiser_ids = advertiser_ids[0].replace(" ", "").split(",")
 
-        start_date = pendulum.now().subtract(days=90).in_tz(time_zone[0])
-        end_date = ensure_pendulum_datetime(pendulum.now()).in_tz(time_zone[0])
+        start_date = pendulum.now().subtract(days=30).in_tz(timezone)
+        end_date = ensure_pendulum_datetime(pendulum.now()).in_tz(timezone)
 
         interval_start = kwargs.get("interval_start")
         if interval_start is not None:
-            start_date = ensure_pendulum_datetime(interval_start).in_tz(time_zone[0])
+            start_date = ensure_pendulum_datetime(interval_start).in_tz(timezone)
 
         interval_end = kwargs.get("interval_end")
         if interval_end is not None:
-            end_date = ensure_pendulum_datetime(interval_end).in_tz(time_zone[0])
+            end_date = ensure_pendulum_datetime(interval_end).in_tz(timezone)
 
-        page_size = kwargs.get("page_size")
-        if page_size is not None and not isinstance(page_size, int):
-            page_size = int(page_size)
-
-        if page_size > 1000:
-            page_size = 1000
+        page_size = min(1000, kwargs.get("page_size", 1000))
 
         if table.startswith("custom:"):
             fields = table.split(":", 3)
@@ -1049,28 +1183,61 @@ class TikTokSource:
             dimensions = fields[1].replace(" ", "").split(",")
             if (
                 "campaign_id" not in dimensions
-                and "advertiser_id" not in dimensions
                 and "adgroup_id" not in dimensions
                 and "ad_id" not in dimensions
             ):
                 raise ValueError(
-                    "You must provide one ID dimension. Please use one ID dimension from the following options: [campaign_id, advertiser_id, adgroup_id, ad_id]"
+                    "TikTok API requires at least one ID dimension, please use one of the following dimensions: [campaign_id, adgroup_id, ad_id]"
                 )
 
+            if "advertiser_id" in dimensions:
+                dimensions.remove("advertiser_id")
+
             metrics = fields[2].replace(" ", "").split(",")
-            filters = []
+            filtering_param = False
+            filter_name = ""
+            filter_value = []
             if len(fields) == 4:
-                filters = fields[3].replace(" ", "").split(",")
+
+                def parse_filters(filters_raw: str) -> dict:
+                    # Parse filter string like "key1=value1,key2=value2,value3,value4"
+                    filters = {}
+                    current_key = None
+
+                    for item in filters_raw.split(","):
+                        if "=" in item:
+                            # Start of a new key-value pair
+                            key, value = item.split("=")
+                            filters[key] = [value]  # Always start with a list
+                            current_key = key
+                        elif current_key is not None:
+                            # Additional value for the current key
+                            filters[current_key].append(item)
+
+                    # Convert single-item lists to simple values
+                    return {k: v[0] if len(v) == 1 else v for k, v in filters.items()}
+
+                filtering_param = True
+                filters = parse_filters(fields[3])
+                if len(filters) > 1:
+                    raise ValueError(
+                        "Only one filter is allowed for TikTok custom reports"
+                    )
+                filter_name = list(filters.keys())[0]
+                filter_value = list(map(int, filters[list(filters.keys())[0]]))
+
         return tiktok_source(
             start_date=start_date,
             end_date=end_date,
             access_token=access_token[0],
-            advertiser_id=advertiser_id[0],
-            time_zone=time_zone[0],
+            advertiser_ids=advertiser_ids,
+            timezone=timezone,
             dimensions=dimensions,
             metrics=metrics,
-            filters=filters,
             page_size=page_size,
+            filter_name=filter_name,
+            filter_value=filter_value,
+            filtering_param=filtering_param,
         ).with_resources(endpoint)
 
 

ingestr-0.12.3/ingestr/src/tiktok_ads/__init__.py

@@ -0,0 +1,139 @@
+from typing import Iterable
+
+import dlt
+import pendulum
+from dlt.common.time import ensure_pendulum_datetime
+from dlt.common.typing import TDataItem
+from dlt.sources import DltResource
+
+from .tiktok_helpers import TikTokAPI
+
+KNOWN_TYPE_HINTS = {
+    "spend": {"data_type": "decimal"},
+    "billed_cost": {"data_type": "decimal"},
+    "cash_spend": {"data_type": "decimal"},
+    "voucher_spend": {"data_type": "decimal"},
+    "cpc": {"data_type": "decimal"},
+    "cpm": {"data_type": "decimal"},
+    "impressions": {"data_type": "bigint"},
+    "gross_impressions": {"data_type": "bigint"},
+    "clicks": {"data_type": "bigint"},
+    "ctr": {"data_type": "decimal"},
+    "reach": {"data_type": "bigint"},
+    "cost_per_1000_reached": {"data_type": "decimal"},
+    "frequency": {"data_type": "decimal"},
+    "conversion": {"data_type": "bigint"},
+    "cost_per_conversion": {"data_type": "decimal"},
+    "conversion_rate": {"data_type": "decimal"},
+    "conversion_rate_v2": {"data_type": "decimal"},
+    "real_time_conversion": {"data_type": "bigint"},
+    "real_time_cost_per_conversion": {"data_type": "decimal"},
+    "real_time_conversion_rate": {"data_type": "decimal"},
+    "real_time_conversion_rate_v2": {"data_type": "decimal"},
+    "result": {"data_type": "bigint"},
+    "cost_per_result": {"data_type": "decimal"},
+    "result_rate": {"data_type": "decimal"},
+    "real_time_result": {"data_type": "bigint"},
+    "real_time_cost_per_result": {"data_type": "decimal"},
+    "real_time_result_rate": {"data_type": "decimal"},
+    "secondary_goal_result": {"data_type": "bigint"},
+    "cost_per_secondary_goal_result": {"data_type": "decimal"},
+    "secondary_goal_result_rate": {"data_type": "decimal"},
+}
+
+
+def find_intervals(
+    current_date: pendulum.DateTime,
+    end_date: pendulum.DateTime,
+    interval_days: int,
+):
+    intervals = []
+    while current_date <= end_date:
+        interval_end = min(current_date.add(days=interval_days), end_date)
+        intervals.append((current_date, interval_end))
+        current_date = interval_end.add(days=1)
+
+    return intervals
+
+
+@dlt.source(max_table_nesting=0)
+def tiktok_source(
+    start_date: pendulum.DateTime,
+    end_date: pendulum.DateTime,
+    access_token: str,
+    advertiser_ids: list[str],
+    timezone: str,
+    page_size: int,
+    filtering_param: bool,
+    filter_name: str,
+    filter_value: list[int],
+    dimensions: list[str],
+    metrics: list[str],
+) -> DltResource:
+    tiktok_api = TikTokAPI(
+        access_token=access_token,
+        timezone=timezone,
+        page_size=page_size,
+        filtering_param=filtering_param,
+        filter_name=filter_name,
+        filter_value=filter_value,
+    )
+    incremental_loading_param = ""
+    is_incremental = False
+    interval_days = 365
+
+    if "stat_time_day" in dimensions:
+        incremental_loading_param = "stat_time_day"
+        is_incremental = True
+        interval_days = 30
+
+    if "stat_time_hour" in dimensions:
+        incremental_loading_param = "stat_time_hour"
+        is_incremental = True
+        interval_days = 0
+
+    type_hints = {
+        "advertiser_id": {"data_type": "text"},
+    }
+    for dimension in dimensions:
+        if dimension in KNOWN_TYPE_HINTS:
+            type_hints[dimension] = KNOWN_TYPE_HINTS[dimension]
+    for metric in metrics:
+        if metric in KNOWN_TYPE_HINTS:
+            type_hints[metric] = KNOWN_TYPE_HINTS[metric]
+
+    @dlt.resource(
+        write_disposition="merge",
+        primary_key=dimensions + ["advertiser_id"],
+        columns=type_hints,
+        parallelized=True,
+    )
+    def custom_reports(
+        datetime=(
+            dlt.sources.incremental(incremental_loading_param, start_date)
+            if is_incremental
+            else None
+        ),
+    ) -> Iterable[TDataItem]:
+        current_date = start_date.in_tz(timezone)
+
+        if datetime is not None:
+            datetime_str = datetime.last_value
+            current_date = ensure_pendulum_datetime(datetime_str).in_tz(timezone)
+
+        list_of_interval = find_intervals(
+            current_date=current_date,
+            end_date=end_date,
+            interval_days=interval_days,
+        )
+
+        for start, end in list_of_interval:
+            yield tiktok_api.fetch_pages(
+                advertiser_ids=advertiser_ids,
+                start_time=start,
+                end_time=end,
+                dimensions=dimensions,
+                metrics=metrics,
+            )
+
+    return custom_reports