ingestr 0.9.2__tar.gz → 0.9.3__tar.gz

{ingestr-0.9.2 → ingestr-0.9.3}/.gitignore

@@ -15,3 +15,4 @@ docs/.vitepress/dist
 docs/.vitepress/cache
 node_modules
 *.duckdb
+*.db

{ingestr-0.9.2 → ingestr-0.9.3}/Dockerfile

@@ -28,7 +28,7 @@ ENV VIRTUAL_ENV=/usr/local
 ADD --chmod=755 https://astral.sh/uv/install.sh /install.sh
 RUN /install.sh && rm /install.sh
 
-RUN /root/.cargo/bin/uv pip install --system --no-cache -r requirements.txt
+RUN $HOME/.local/bin/uv pip install --system --no-cache -r requirements.txt
 
 COPY . /app
 

{ingestr-0.9.2 → ingestr-0.9.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: ingestr
-Version: 0.9.2
+Version: 0.9.3
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

ingestr-0.9.3/docs/supported-sources/adjust.md

@@ -0,0 +1,84 @@
+# Adjust
+
+[Adjust](https://www.adjust.com/) is a mobile marketing analytics platform that provides solutions for measuring and optimizing campaigns, as well as protecting user data.
+
+ingestr supports Adjust as a source.
+
+## URI Format
+
+The URI format for Adjust is as follows:
+
+```plaintext
+adjust://?api_key=<api-key-here>
+```
+Parameters:
+- `api_key`: Required. The API key for the Adjust account.
+- `lookback_days`: Optional. The number of days to go back than the given start date for data. Defaults to 30 days.
+
+An API token is required to retrieve reports from the Adjust reporting API. please follow the guide to [obtain an API key](https://dev.adjust.com/en/api/rs-api/authentication/).
+
+Once you complete the guide, you should have an API key. Let's say your API key is `nr_123`, here's a sample command that will copy the data from Adjust into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'adjust://?api_key=nr_123' --source-table 'campaigns' --dest-uri duckdb:///adjust.duckdb --dest-table 'adjust.output'
+```
+
+The result of this command will be a table in the `adjust.duckdb` database.
+
+### Lookback Days
+
+Adjust data may change going back, which means you'll need to change your start date to get the latest data. The `lookback_days` parameter allows you to specify how many days to go back when calculating the start date, and takes care of automatically updating the start date and getting the past data as well. It defaults to 30 days.
+
+## Tables
+Adjust source allows ingesting data from various sources:
+
+- `campaigns`: Retrieves data for a campaign, showing the app's revenue and network costs over multiple days.
+- `creatives`: Retrieves data for a creative assest, detailing the app's revenue and network costs across multiple days.
+- `custom`: Retrieves custom data based on the dimensions and metrics specified.
+
+### Custom Table: `custom:<dimensions>:<metrics>[:<filters>]`
+
+The custom table allows you to retrieve data based on specific dimensions and metrics, and apply filters to the data.
+
+The format for the custom table is: 
+```plaintext
+custom:<dimensions>:<metrics>[:<filters>]
+```
+
+Parameters:
+- `dimensions`: A comma-separated list of [dimensions](https://dev.adjust.com/en/api/rs-api/reports#dimensions) to retrieve.
+- `metrics`: A comma-separated list of [metrics](https://dev.adjust.com/en/api/rs-api/reports#metrics) to retrieve.
+- `filters`: A comma-separated list of [filters](https://dev.adjust.com/en/api/rs-api/reports#filters) to apply to the data.
+  - Parsing the `filters` key is smart enough to handle filters that contain commas inside them.
+
+> [!WARNING]
+> Custom tables require a time-based dimension for efficient operation, such as `hour`, `day`, `week`, `month`, or `year`.
+
+## Examples
+
+Copy campaigns data from Adjust into a DuckDB database:
+```sh
+ingestr ingest \
+    --source-uri 'adjust://?api_key=nr_123' \
+    --source-table 'campaigns' \
+    --dest-uri duckdb:///adjust.duckdb \
+    --dest-table 'adjust.output'
+```
+
+Copy creatives data from Adjust into a DuckDB database:
+```sh
+ingestr ingest \
+    --source-uri 'adjust://?api_key=nr_123' \
+    --source-table 'creatives' \
+    --dest-uri duckdb:///adjust.duckdb \
+    --dest-table 'adjust.output'
+```
+
+Copy custom data from Adjust into a DuckDB database:
+```sh
+ingestr ingest \
+    --source-uri "adjust://?api_key=nr_123&lookback_days=2" \
+    --source-table "custom:hour,app,store_id,channel,os_name,country_code,campaign_network,campaign_id_network,adgroup_network, adgroup_id_network,creative_network,creative_id_network:impressions,clicks,cost,network_cost,installs,ad_revenue,all_revenue" \
+    --dest-uri duckdb:///adjust.db \
+    --dest-table "mat.example"
+```

{ingestr-0.9.2 → ingestr-0.9.3}/docs/supported-sources/kafka.md

@@ -11,8 +11,8 @@ kafka://?bootstrap_servers=localhost:9092&group_id=test_group&security_protocol=
 ```
 
 URI parameters:
-- `bootstrap_servers`: The Kafka server(s) to connect to, typically in the form of a host and port (e.g., `localhost:9092`).
-- `group_id`: The consumer group ID used for identifying the client when consuming messages.
+- `bootstrap_servers`(required): The Kafka server(s) to connect to, typically in the form of a host and port (e.g., `localhost:9092`).
+- `group_id`(required): The consumer group ID used for identifying the client when consuming messages.
 - `security_protocol`: The protocol used to communicate with brokers (e.g., `SASL_SSL` for secure communication).
 - `sasl_mechanisms`: The SASL mechanism to be used for authentication (e.g., `PLAIN`).
 - `sasl_username`: The username for SASL authentication.
@@ -29,7 +29,7 @@ Once you have your Kafka server, credentials, and group ID set up, here's a samp
 
 ```sh
 ingestr ingest \
-    --source-uri 'kafka://?bootstrap_servers=localhost:9092' \
+    --source-uri 'kafka://?bootstrap_servers=localhost:9092&group_id=test_group' \
     --source-table 'my-topic' \
     --dest-uri duckdb:///kafka.duckdb \
     --dest-table 'kafka.my_topic'

{ingestr-0.9.2 → ingestr-0.9.3}/ingestr/main.py

@@ -323,10 +323,12 @@ def ingest(
             else "Platform-specific"
         )
 
+        source_table_print = source_table.split(":")[0]
+
         print()
         print("[bold green]Initiated the pipeline with the following:[/bold green]")
         print(
-            f"[bold yellow]  Source:[/bold yellow] {factory.source_scheme} / {source_table}"
+            f"[bold yellow]  Source:[/bold yellow] {factory.source_scheme} / {source_table_print}"
         )
         print(
             f"[bold yellow]  Destination:[/bold yellow] {factory.destination_scheme} / {dest_table}"

ingestr-0.9.3/ingestr/src/adjust/__init__.py

@@ -0,0 +1,100 @@
+from typing import Optional, Sequence
+
+import dlt
+import pendulum
+from dlt.sources import DltResource
+
+from .adjust_helpers import DEFAULT_DIMENSIONS, DEFAULT_METRICS, AdjustAPI
+
+REQUIRED_CUSTOM_DIMENSIONS = [
+    "hour",
+    "day",
+    "week",
+    "month",
+    "quarter",
+    "year",
+]
+KNOWN_TYPE_HINTS = {
+    "hour": {"data_type": "timestamp"},
+    "day": {"data_type": "date"},
+    "week": {"data_type": "text"},
+    "month": {"data_type": "text"},
+    "quarter": {"data_type": "text"},
+    "year": {"data_type": "text"},
+    "campaign": {"data_type": "text"},
+    "adgroup": {"data_type": "text"},
+    "creative": {"data_type": "text"},
+    # metrics
+    "installs": {"data_type": "bigint"},
+    "clicks": {"data_type": "bigint"},
+    "cost": {"data_type": "decimal"},
+    "network_cost": {"data_type": "decimal"},
+    "impressions": {"data_type": "bigint"},
+    "ad_revenue": {"data_type": "decimal"},
+    "all_revenue": {"data_type": "decimal"},
+}
+
+
+@dlt.source(max_table_nesting=0)
+def adjust_source(
+    start_date: pendulum.DateTime,
+    end_date: pendulum.DateTime,
+    api_key: str,
+    dimensions: Optional[list[str]] = None,
+    metrics: Optional[list[str]] = None,
+    merge_key: Optional[str] = None,
+    filters: Optional[dict] = None,
+) -> Sequence[DltResource]:
+    @dlt.resource(write_disposition="merge", merge_key="day")
+    def campaigns():
+        adjust_api = AdjustAPI(api_key=api_key)
+        yield from adjust_api.fetch_report_data(
+            start_date=start_date,
+            end_date=end_date,
+            dimensions=DEFAULT_DIMENSIONS,
+            metrics=DEFAULT_METRICS,
+            filters=filters,
+        )
+
+    @dlt.resource(write_disposition="merge", merge_key="day")
+    def creatives():
+        adjust_api = AdjustAPI(api_key=api_key)
+        yield from adjust_api.fetch_report_data(
+            start_date=start_date,
+            end_date=end_date,
+            dimensions=DEFAULT_DIMENSIONS + ["adgroup", "creative"],
+            metrics=DEFAULT_METRICS,
+            filters=filters,
+        )
+
+    merge_key = merge_key
+    for dimension in REQUIRED_CUSTOM_DIMENSIONS:
+        if dimension in dimensions:
+            merge_key = dimension
+            break
+
+    type_hints = {}
+    for dimension in dimensions:
+        if dimension in KNOWN_TYPE_HINTS:
+            type_hints[dimension] = KNOWN_TYPE_HINTS[dimension]
+    for metric in metrics:
+        if metric in KNOWN_TYPE_HINTS:
+            type_hints[metric] = KNOWN_TYPE_HINTS[metric]
+
+    @dlt.resource(
+        write_disposition={"disposition": "merge", "strategy": "delete+insert"},
+        merge_key=merge_key,
+        primary_key=dimensions,
+        columns=type_hints,
+    )
+    def custom():
+        adjust_api = AdjustAPI(api_key=api_key)
+        yield from adjust_api.fetch_report_data(
+            start_date=start_date,
+            end_date=end_date,
+            dimensions=dimensions,
+            metrics=metrics,
+            filters=filters,
+        )
+
+    return campaigns, creatives, custom

ingestr-0.9.2/ingestr/src/adjust/helpers.py → ingestr-0.9.3/ingestr/src/adjust/adjust_helpers.py

@@ -1,3 +1,6 @@
+from typing import Optional
+
+import pendulum
 import requests
 from dlt.sources.helpers.requests import Client
 from requests.exceptions import HTTPError
@@ -32,33 +35,32 @@ class AdjustAPI:
 
     def fetch_report_data(
         self,
-        start_date,
-        end_date,
+        start_date: pendulum.DateTime,
+        end_date: pendulum.DateTime,
         dimensions=DEFAULT_DIMENSIONS,
         metrics=DEFAULT_METRICS,
-        utc_offset="+00:00",
-        ad_spend_mode="network",
-        attribution_source="first",
-        attribution_type="all",
-        cohort_maturity="immature",
-        reattributed="all",
-        sandbox="false",
+        filters: Optional[dict] = None,
     ):
         headers = {"Authorization": f"Bearer {self.api_key}"}
-        comma_separated_dimensions = ",".join(dimensions)
-        comma_separated_metrics = ",".join(metrics)
-        params = {
-            "date_period": f"{start_date}:{end_date}",
-            "dimensions": comma_separated_dimensions,
-            "metrics": comma_separated_metrics,
-            "utc_offset": utc_offset,
-            "ad_spend_mode": ad_spend_mode,
-            "attribution_source": attribution_source,
-            "attribution_type": attribution_type,
-            "cohort_maturity": cohort_maturity,
-            "reattributed": reattributed,
-            "sandbox": sandbox,
-        }
+        params = {}
+
+        if filters:
+            for key, value in filters.items():
+                if isinstance(value, list):
+                    params[key] = ",".join(value)
+                else:
+                    params[key] = value
+
+        params["date_period"] = (
+            f"{start_date.format('YYYY-MM-DD')}:{end_date.format('YYYY-MM-DD')}"
+        )
+        params["dimensions"] = ",".join(dimensions)
+        params["metrics"] = ",".join(metrics)
+
+        if start_date > end_date:
+            raise ValueError(
+                f"Invalid date range: Start date ({start_date}) must be earlier than end date ({end_date})."
+            )
 
         def retry_on_limit(
             response: requests.Response, exception: BaseException
@@ -80,3 +82,24 @@ class AdjustAPI:
             yield items
         else:
             raise HTTPError(f"Request failed with status code: {response.status_code}")
+
+
+def parse_filters(filters_raw: str) -> dict:
+    # Parse filter string like "key1=value1,key2=value2,value3,value4"
+    filters = {}
+    current_key = None
+
+    for item in filters_raw.split(","):
+        if "=" in item:
+            # Start of a new key-value pair
+            key, value = item.split("=")
+            filters[key] = [value]  # Always start with a list
+            current_key = key
+        elif current_key is not None:
+            # Additional value for the current key
+            filters[current_key].append(item)
+
+    # Convert single-item lists to simple values
+    filters = {k: v[0] if len(v) == 1 else v for k, v in filters.items()}
+
+    return filters

{ingestr-0.9.2 → ingestr-0.9.3}/ingestr/src/sources.py

@@ -1,15 +1,18 @@
 import base64
 import csv
 import json
-from datetime import date, datetime
+from datetime import date
 from typing import Any, Callable, Optional
 from urllib.parse import parse_qs, urlparse
 
 import dlt
+import pendulum
 from dlt.common.configuration.specs import AwsCredentials
+from dlt.common.time import ensure_pendulum_datetime
 from dlt.common.typing import TSecretStrValue
 
-from ingestr.src.adjust._init_ import adjust_source
+from ingestr.src.adjust import REQUIRED_CUSTOM_DIMENSIONS, adjust_source
+from ingestr.src.adjust.adjust_helpers import parse_filters
 from ingestr.src.airtable import airtable_source
 from ingestr.src.appsflyer._init_ import appsflyer_source
 from ingestr.src.arrow import memory_mapped_arrow
@@ -719,10 +722,10 @@ class KafkaSource:
 
 class AdjustSource:
     def handles_incrementality(self) -> bool:
-        return True
+        return False
 
     def dlt_source(self, uri: str, table: str, **kwargs):
-        if kwargs.get("incremental_key"):
+        if kwargs.get("incremental_key") and not table.startswith("custom:"):
             raise ValueError(
                 "Adjust takes care of incrementality on its own, you should not provide incremental_key"
             )
@@ -734,25 +737,62 @@ class AdjustSource:
         if not api_key:
             raise ValueError("api_key in the URI is required to connect to Adjust")
 
-        interval_start = kwargs.get("interval_start")
-        interval_end = kwargs.get("interval_end")
+        lookback_days = int(source_params.get("lookback_days", [30])[0])
 
         start_date = (
-            interval_start.strftime("%Y-%m-%d") if interval_start else "2000-01-01"
-        )
-        end_date = (
-            interval_end.strftime("%Y-%m-%d")
-            if interval_end
-            else datetime.now().strftime("%Y-%m-%d")
+            pendulum.now()
+            .replace(hour=0, minute=0, second=0, microsecond=0)
+            .subtract(days=lookback_days)
         )
+        if kwargs.get("interval_start"):
+            start_date = (
+                ensure_pendulum_datetime(str(kwargs.get("interval_start")))
+                .replace(hour=0, minute=0, second=0, microsecond=0)
+                .subtract(days=lookback_days)
+            )
 
-        Endpoint = None
-        if table in ["campaigns", "creatives"]:
-            Endpoint = table
+        end_date = pendulum.now()
+        if kwargs.get("interval_end"):
+            end_date = ensure_pendulum_datetime(str(kwargs.get("interval_end")))
+
+        dimensions = None
+        metrics = None
+        filters = []
+        if table.startswith("custom:"):
+            fields = table.split(":")
+            if len(fields) != 3 and len(fields) != 4:
+                raise ValueError(
+                    "Invalid Adjust custom table format. Expected format: custom:<dimensions>,<metrics> or custom:<dimensions>:<metrics>:<filters>"
+                )
+
+            dimensions = fields[1].split(",")
+            metrics = fields[2].split(",")
+            table = "custom"
+
+            found = False
+            for dimension in dimensions:
+                if dimension in REQUIRED_CUSTOM_DIMENSIONS:
+                    found = True
+                    break
+
+            if not found:
+                raise ValueError(
+                    f"At least one of the required dimensions is missing for custom Adjust report: {REQUIRED_CUSTOM_DIMENSIONS}"
+                )
+
+            if len(fields) == 4:
+                filters_raw = fields[3]
+                filters = parse_filters(filters_raw)
 
         return adjust_source(
-            start_date=start_date, end_date=end_date, api_key=api_key[0]
-        ).with_resources(Endpoint)
+            start_date=start_date,
+            end_date=end_date,
+            api_key=api_key[0],
+            dimensions=dimensions,
+            metrics=metrics,
+            merge_key=kwargs.get("merge_key"),
+            filters=filters,
+        ).with_resources(table)
 
 
 class AppsflyerSource:

ingestr-0.9.3/ingestr/src/version.py

@@ -0,0 +1 @@
+__version__ = "0.9.3"

ingestr-0.9.2/docs/supported-sources/adjust.md

@@ -1,30 +0,0 @@
-# Adjust
-
-[Adjust](https://www.adjust.com/) is a mobile marketing analytics platform that provides solutions for measuring and optimizing campaigns, as well as protecting user data.
-
-ingestr supports Adjust as a source.
-
-## URI Format
-
-The URI format for Adjust is as follows:
-
-```plaintext
-adjust://?api_key=<api-key-here>
-```
-
-An API token is required to retrieve reports from the Adjust reporting API. please follow the guide to [obtain a API key](https://dev.adjust.com/en/api/rs-api/authentication/).
-
-Once you complete the guide, you should have an API key. Let's say your API key is `nr_123`, here's a sample command that will copy the data from Adjust into a duckdb database:
-
-```sh
-ingestr ingest --source-uri 'adjust://?api_key=nr_123' --source-table 'campaigns' --dest-uri duckdb:///adjust.duckdb --dest-table 'adjust.output' --interval-start '2024-09-05' --interval-end '2024-09-08'
-```
-
-The result of this command will be a table in the `adjust.duckdb` database
-
-Available Source Table:
-Adjust source allows ingesting the following source into separate tables:
-
--`Campaigns`: Retrieves data for a campaign, showing the app's revenue and network costs over multiple days.
-
---`Creatives`: Retrieves data for a creative assest, detailing the app's revenue and network costs across multiple days

ingestr-0.9.2/ingestr/src/adjust/_init_.py

@@ -1,31 +0,0 @@
-from typing import Sequence
-
-import dlt
-from dlt.sources import DltResource
-
-from .helpers import DEFAULT_DIMENSIONS, AdjustAPI
-
-
-@dlt.source(max_table_nesting=0)
-def adjust_source(
-    start_date: str,
-    end_date: str,
-    api_key: str,
-) -> Sequence[DltResource]:
-    @dlt.resource(write_disposition="merge", merge_key="day")
-    def campaigns():
-        adjust_api = AdjustAPI(api_key=api_key)
-        yield from adjust_api.fetch_report_data(
-            start_date=start_date,
-            end_date=end_date,
-        )
-
-    @dlt.resource(write_disposition="merge", merge_key="day")
-    def creatives():
-        dimensions = DEFAULT_DIMENSIONS + ["adgroup", "creative"]
-        adjust_api = AdjustAPI(api_key=api_key)
-        yield from adjust_api.fetch_report_data(
-            start_date=start_date, end_date=end_date, dimensions=dimensions
-        )
-
-    return campaigns, creatives

ingestr-0.9.2/ingestr/src/version.py

@@ -1 +0,0 @@
-__version__ = "0.9.2"