ingestr 0.12.10__tar.gz → 0.12.11__tar.gz

{ingestr-0.12.10 → ingestr-0.12.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.12.10
+Version: 0.12.11
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -22,7 +22,9 @@ Requires-Dist: dlt==1.5.0
 Requires-Dist: duckdb-engine==0.13.5
 Requires-Dist: duckdb==1.1.3
 Requires-Dist: facebook-business==20.0.0
+Requires-Dist: flatten-json==0.1.14
 Requires-Dist: gcsfs==2024.10.0
+Requires-Dist: google-ads==25.1.0
 Requires-Dist: google-analytics-data==0.18.16
 Requires-Dist: google-api-python-client==2.130.0
 Requires-Dist: google-cloud-bigquery-storage==2.24.0

{ingestr-0.12.10 → ingestr-0.12.11}/docs/.vitepress/config.mjs

@@ -104,6 +104,7 @@ export default defineConfig({
               },
               { text: "Google Cloud Storage (GCS)", link: "/supported-sources/gcs.md" },
               { text: "Google Analytics", link: "/supported-sources/google_analytics.md" },
+              { text: "Google Ads", link: "/supported-sources/google-ads.md" },
               { text: "GitHub", link: "/supported-sources/github.md" },
               { text: "Google Sheets", link: "/supported-sources/gsheets.md" },
               { text: "Gorgias", link: "/supported-sources/gorgias.md" },

{ingestr-0.12.10 → ingestr-0.12.11}/docs/supported-sources/gcs.md

@@ -7,14 +7,18 @@
 The URI format for Google Cloud Storage is as follows:
 
 ```plaintext
-gs://<bucket_name>?credentials_path=/path/to/service-account.json>
+gs://?credentials_path=/path/to/service-account.json>
 ```
 
 URI parameters:
 
-- `bucket_name`: The name of the bucket
 - `credentials_path`: path to file containing your Google Cloud [Service Account](https://cloud.google.com/iam/docs/service-account-overview)
 
+The `--source-table` must be in the format:
+```
+{bucket name}/{file glob}
+```
+
 ## Setting up a GCS Integration
 
 To use Google Cloud Storage source in `ingestr`, you will need:
@@ -29,7 +33,7 @@ For more information on how to create a Service Account or it's keys, see [Creat
 Let's assume that:
 * Service account key in available in the current directory, under the filename `service_account.json`. 
 * The bucket you want to load data from is called `my-org-bucket`
-* The source file is available at `/data/latest/dump.csv`
+* The source file is available at `data/latest/dump.csv`
 * The data needs to be saved in a DuckDB database called `local.db`
 * The destination table name will be `public.latest_dump`
 
@@ -37,8 +41,8 @@ You can run the following command line to achieve this:
 
 ```sh
 ingestr ingest \
-    --source-uri "gs://my-org-bucket?credentials_path=$PWD/service_account.json" \
-    --source-table "/data/latest/dump.csv" \
+    --source-uri "gs://?credentials_path=$PWD/service_account.json" \
+    --source-table "my-org-bucket/data/latest/dump.csv" \
     --dest-uri "duckdb:///local.db" \
     --dest-table "public.latest_dump"
 ```
@@ -53,7 +57,7 @@ ingestr ingest \
 `ingestr` supports [glob](https://en.wikipedia.org/wiki/Glob_(programming)) like pattern matching for `gs` source.
 This allows for a powerful pattern matching mechanism that allows you to specify multiple files in a single `--source-table`.
 
-Below are some examples of path patterns, each path pattern is a reference from the root of the bucket:
+Below are some examples of path patterns, each path pattern is glob you can specify after the bucket name:
 
 - `**/*.csv`: Retrieves all the CSV files, regardless of how deep they are within the folder structure.
 - `*.csv`: Retrieves all the CSV files from the first level of a folder.

ingestr-0.12.11/docs/supported-sources/google-ads.md

@@ -0,0 +1,134 @@
+# Google Ads
+[Google Ads](https://ads.google.com/), formerly known as Google Adwords, is an online advertising platform developed by Google, where advertisers bid to display brief advertisements, service offerings, product listings, and videos to web users. It can place ads in the results of search engines like Google Search (the Google Search Network), mobile apps, videos, and on non-search websites.
+
+## URI format
+
+The URI format for Google Ads is as follows:
+```plaintext
+googleads://<customer_id>?credentials_path=/path/to/service-account.json&dev_token=<dev_token>
+```
+
+URI parameters:
+
+- `customer_id`: Customer ID of the Google Ads account to use.
+- `credentials_path`: path to the service account JSON file.
+- `dev_token`: [developer token](https://developers.google.com/google-ads/api/docs/get-started/dev-token) to use for accessing the account.
+
+> [!NOTE]
+> You may specify credentials using `credentials_base64` instead of `credentials_path`.
+> The value of this parameter is the base64 encoded contents of the 
+> service account json file. However, we don't recommend using this
+> parameter, unless you're integrating ingestr into another system.
+## Setting up a Google Ads integration
+
+### Prerequisites
+* A Google cloud [service account](https://cloud.google.com/iam/docs/service-account-overview)
+* A Google Ads [developer token](https://developers.google.com/google-ads/api/docs/get-started/dev-token)
+* A Google Ads account 
+
+
+### Obtaining necessary credentials
+
+You can use the [Google Cloud IAM Console](https://cloud.google.com/security/products/iam) to create a service account for ingesting data from Google Ads. Make sure to enable Google Ads API in your console.
+
+Next, you need to add your service account user to your Google Ads account. See [Google Developers Docs](https://developers.google.com/google-ads/api/docs/oauth/service-accounts) for exact steps.
+
+Finally, you need to obtain a Google Ads Developer Token. Developer token lets your app connect to the Google Ads API. Each developer token is assigned an API access level which controls the number of API calls you can make per day with as well as the environment to which you can make calls. See [Google Ads docs](https://developers.google.com/google-ads/api/docs/get-started/dev-token) for more information on how to obtain this token.
+
+You also need the 10-digit customer id of the account you're making API calls to. This is displayed in the Google Ads web interface in the form 123-456-7890. In this case, your customer id would be `1234567890`
+
+### Example
+
+Let's say we want to ingest information about campaigns (on a daily interval) and save them to a table `public.campaigns` in duckdb database called `adverts.db`.
+
+For this example, we'll assume that:
+* The service account JSON file is located in the current directory and is named `svc_account.json`
+* customer id is `1234567890`
+* the developer token is `dev-token-spec-1`
+
+You can run the following to achieve this:
+```sh
+ingestr ingest \
+  --source-uri "googleads://12345678?credentials_path=./svc_account.json&dev_token=dev-token-spec-1" \
+  --source-table "campaign_report_daily" \
+  --dest-uri "duckdb://./adverts.db" \
+  --dest-table "public.campaigns"
+```
+## Tables
+
+| Name             | Description                                                             |
+|------------------|-------------------------------------------------------------------------|
+| `account_report_daily` | Provides daily metrics aggregated at the account level. |
+| `campaign_report_daily` | Provides daily metrics aggregated at the campaign level. |
+| `ad_group_report_daily` | Provides daily metrics aggregated at the ad group level. |
+| `ad_report_daily` | Provides daily metrics aggregated at the ad level. |
+| `audience_report_daily` | Provides daily metrics aggregated at the audience level. |
+| `keyword_report_daily` | Provides daily metrics aggregated at the keyword level. |
+| `click_report_daily` | Provides daily metrics on clicks. |
+| `landing_page_report_daily` | Provides daily metrics on landing page performance. |
+| `search_keyword_report_daily` | Provides daily metrics on search keywords. |
+| `search_term_report_daily` | Provides daily metrics on search terms. |
+| `lead_form_submission_data_report_daily` | Provides daily metrics on lead form submissions. |
+| `local_services_lead_report_daily` | Provides daily metrics on local services leads. |
+| `local_services_lead_conversations_report_daily` | Provides daily metrics on local services lead conversations. |
+
+## Custom Reports
+`googleads` source supports custom reports. You can pass a custom report definition to `--source-table` and it will dynamically create a report for you. These reports are aggregated at a daily interval.
+
+The format of a custom report looks like the following:
+```
+daily:{resource_name}:{dimensions}:{metrics}
+```
+Where:
+* `{resource_name}` is a [Google Ads Resource](https://developers.google.com/google-ads/api/fields/v18/overview_query_builder#list-of-all-resources).
+* `{dimensions}` is a comma separated list of the Resource's attribute fields, or fields of [attributed resources](https://developers.google.com/google-ads/api/docs/query/overview).
+* `{metrics}` is a comma separated list of the Resource's [metrics](https://developers.google.com/google-ads/api/fields/v18/metrics). Note that the `metrics.` prefix is optional. 
+
+Notes:
+* `{dimensions}` and `{metrics}` are optional. If you don't need them, you can leave their respective segment blank.
+* `segments` are currently not supported as dimensions.
+* `segments.date` is automatically added to all custom reports.
+
+### Custom Report Example
+For this example, we will ingest data from `ad_group_ad_asset_view`.
+We want to obtain the following info:
+**dimensions**
+  * ad_group.id
+  * campagin.id
+  * customer.id
+**metrics**
+  * metrics.clicks
+  * metrics.conversions
+  * metrics.impressions
+
+To achieve this, we pass a `daily` report specification to `ingestr` source table as follows:
+```sh
+ingestr ingest \
+  --source-uri "googleads://12345678?credentials_path=./svc_account.json&dev_token=dev-token-spec-1" \
+  --source-table "daily:ad_group_ad_asset_view:ad_group.id,campaign.id,customer.id:clicks,conversions,impressions" \
+  --dest-uri "duckdb:///custom.db" \
+  --dest-table "public.report"
+```
+
+Notice the lack of `metrics.` prefix in the metrics segment. Please note that `--dest-table` is mandatory when creating
+a custom report.
+
+**Without Metrics**
+
+Here's an example of the above report, without any associated metrics:
+```sh
+ingestr ingest \
+  --source-uri "googleads://12345678?credentials_path=./svc_account.json&dev_token=dev-token-spec-1" \
+  --source-table "daily:ad_group_ad_asset_view:ad_group.id,campaign.id,customer.id:" \
+  --dest-uri "duckdb:///custom.db" \
+```
+
+**Without Dimensions**
+
+Here's an example of the above report, without any associated dimensions:
+```sh
+ingestr ingest \
+  --source-uri "googleads://12345678?credentials_path=./svc_account.json&dev_token=dev-token-spec-1" \
+  --source-table "daily:ad_group_ad_asset_view::clicks,conversions,impressions" \
+  --dest-uri "duckdb:///custom.db" \
+```

{ingestr-0.12.10 → ingestr-0.12.11}/docs/supported-sources/s3.md

@@ -9,15 +9,18 @@ ingestr supports S3 as a source.
 The URI format for S3 is as follows:
 
 ```plaintext
-s3://<bucket_name>?access_key_id=<access_key_id>&secret_access_key=<secret_access_key>
+s3://?access_key_id=<access_key_id>&secret_access_key=<secret_access_key>
 ```
 
 URI parameters:
 
-- `bucket_name`: The name of the bucket
-- `path_to_files`: The relative path from the root of the bucket. You can find this from the S3 URI. For example, if your S3 URI is `s3://mybucket/students/students_details.csv`, then your bucket name is `mybucket` and `path_to_files` is `students/students_details.csv`.
 - `access_key_id` and `secret_access_key` : Used for accessing S3 bucket.
 
+The `--source-table` must be in the format:
+```
+{bucket name}/{file glob}
+```
+
 ## Setting up a S3 Integration
 
 S3 requires an `access_key_id` and a `secret_access_key` to access the bucket. Please follow the guide on dltHub to [obtain credentials](https://dlthub.com/docs/dlt-ecosystem/verified-sources/filesystem/basic#get-credentials). Once you've completed the guide, you should have an `access_key_id` and `secret_access_key`. From the S3 URI, you can extract the `bucket_name` and `path_to_files`
@@ -26,15 +29,15 @@ For example, if your `access_key_id` is `AKC3YOW7E`, `secret_access_key` is `XCt
 
 ```sh
 ingestr ingest \
-    --source-uri 's3://my_bucket?access_key_id=AKC3YOW7E&secret_access_key=XCtkpL5B' \
-    --source-table '/students/students_details.csv' \
+    --source-uri 's3://?access_key_id=AKC3YOW7E&secret_access_key=XCtkpL5B' \
+    --source-table 'my_bucket/students/students_details.csv' \
     --dest-uri duckdb:///s3.duckdb \
     --dest-table 'dest.students_details'
 ```
 
 The result of this command will be a table in the DuckDB database in the path `s3.duckdb`.
 
-Below are some examples of path patterns, each path pattern is a reference from the root of the bucket:
+Below are some examples of path patterns, each path pattern is a glob you can specify after the bucket name:
 
 - `**/*.csv`: Retrieves all the CSV files, regardless of how deep they are within the folder structure.
 - `*.csv`: Retrieves all the CSV files from the first level of a folder.

ingestr-0.12.11/ingestr/src/blob.py

@@ -0,0 +1,49 @@
+import warnings
+from typing import Tuple, TypeAlias
+from urllib.parse import ParseResult
+
+BucketName: TypeAlias = str
+FileGlob: TypeAlias = str
+
+
+def parse_uri(uri: ParseResult, table: str) -> Tuple[BucketName, FileGlob]:
+    """
+    parse the URI of a blob storage and
+    return the bucket name and the file glob.
+
+    Supports the following Forms:
+    - uri: "gs://"
+      table: "bucket-name/file-glob"
+    - uri: gs://bucket-name/file-glob
+      table: None
+    - uri: "gs://bucket-name"
+      table: "file-glob"
+
+    The first form is the prefered method. Other forms are supported
+    for backward compatibility, but discouraged.
+    """
+
+    table = table.strip()
+    host = uri.netloc.strip()
+
+    if table == "":
+        warnings.warn(
+            f"Using the form '{uri.scheme}://bucket-name/file-glob' is deprecated and will be removed in future versions.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        return host, uri.path.lstrip("/")
+
+    if host != "":
+        warnings.warn(
+            f"Using the form '{uri.scheme}://bucket-name' is deprecated and will be removed in future versions.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        return host, table.lstrip("/")
+
+    parts = table.lstrip("/").split("/", maxsplit=1)
+    if len(parts) != 2:
+        return "", parts[0]
+
+    return parts[0], parts[1]

{ingestr-0.12.10 → ingestr-0.12.11}/ingestr/src/errors.py

@@ -8,3 +8,11 @@ class UnsupportedResourceError(Exception):
         super().__init__(
             f"Resource '{resource}' is not supported for {source} source yet, if you are interested in it please create a GitHub issue at https://github.com/bruin-data/ingestr"
         )
+
+
+class InvalidBlobTableError(Exception):
+    def __init__(self, source):
+        super().__init__(
+            f"Invalid source table for {source} "
+            "Ensure that the table is in the format {bucket-name}/{file glob}"
+        )

{ingestr-0.12.10 → ingestr-0.12.11}/ingestr/src/factory.py

@@ -27,6 +27,7 @@ from ingestr.src.sources import (
     FacebookAdsSource,
     GCSSource,
     GitHubSource,
+    GoogleAdsSource,
     GoogleAnalyticsSource,
     GoogleSheetsSource,
     GorgiasSource,
@@ -125,6 +126,7 @@ class SourceDestinationFactory:
         "asana": AsanaSource,
         "tiktok": TikTokSource,
         "googleanalytics": GoogleAnalyticsSource,
+        "googleads": GoogleAdsSource,
         "appstore": AppleAppStoreSource,
         "gs": GCSSource,
         "linkedinads": LinkedInAdsSource,

ingestr-0.12.11/ingestr/src/google_ads/__init__.py

@@ -0,0 +1,116 @@
+import json
+from datetime import date, datetime
+from typing import Any, Iterator, Optional
+
+import dlt
+import proto  # type: ignore
+from dlt.common.exceptions import MissingDependencyException
+from dlt.common.typing import TDataItem
+from dlt.sources import DltResource
+from flatten_json import flatten  # type: ignore
+from googleapiclient.discovery import Resource  # type: ignore
+
+from . import field
+from .metrics import dlt_metrics_schema
+from .predicates import date_predicate
+from .reports import BUILTIN_REPORTS, Report
+
+try:
+    from google.ads.googleads.client import GoogleAdsClient  # type: ignore
+except ImportError:
+    raise MissingDependencyException("Requests-OAuthlib", ["google-ads"])
+
+
+@dlt.source
+def google_ads(
+    client: GoogleAdsClient,
+    customer_id: str,
+    report_spec: Optional[str] = None,
+    start_date: Optional[datetime] = None,
+    end_date: Optional[datetime] = None,
+) -> Iterator[DltResource]:
+    date_range = dlt.sources.incremental(
+        "segments_date",
+        initial_value=start_date.date(),  # type: ignore
+        end_value=end_date.date() if end_date is not None else None,  # type: ignore
+        range_start="closed",
+        range_end="closed",
+    )
+    if report_spec is not None:
+        custom_report = Report().from_spec(report_spec)
+        yield dlt.resource(
+            daily_report,
+            name="daily_report",
+            write_disposition="merge",
+            primary_key=custom_report.primary_keys(),
+            columns=dlt_metrics_schema(custom_report.metrics),
+        )(client, customer_id, custom_report, date_range)
+
+    for report_name, report in BUILTIN_REPORTS.items():
+        yield dlt.resource(
+            daily_report,
+            name=report_name,
+            write_disposition="merge",
+            primary_key=report.primary_keys(),
+            columns=dlt_metrics_schema(report.metrics),
+        )(client, customer_id, report, date_range)
+
+
+def daily_report(
+    client: Resource,
+    customer_id: str,
+    report: Report,
+    date: dlt.sources.incremental[date],
+) -> Iterator[TDataItem]:
+    ga_service = client.get_service("GoogleAdsService")
+    fields = report.dimensions + report.metrics + report.segments
+    criteria = date_predicate("segments.date", date.last_value, date.end_value)  # type:ignore
+    query = f"""
+        SELECT
+            {", ".join(fields)}
+        FROM
+            {report.resource}
+        WHERE
+            {criteria}
+    """
+    if report.unfilterable is True:
+        i = query.index("WHERE", 0)
+        query = query[:i]
+
+    allowed_keys = set([field.to_column(k) for k in fields])
+    stream = ga_service.search_stream(customer_id=customer_id, query=query)
+    for batch in stream:
+        for row in batch.results:
+            data = flatten(merge_lists(to_dict(row)))
+            if "segments_date" in data:
+                data["segments_date"] = datetime.strptime(
+                    data["segments_date"], "%Y-%m-%d"
+                ).date()
+            yield {k: v for k, v in data.items() if k in allowed_keys}
+
+
+def to_dict(item: Any) -> TDataItem:
+    """
+    Processes a batch result (page of results per dimension) accordingly
+    :param batch:
+    :return:
+    """
+    return json.loads(
+        proto.Message.to_json(
+            item,
+            preserving_proto_field_name=True,
+            use_integers_for_enums=False,
+            including_default_value_fields=False,
+        )
+    )
+
+
+def merge_lists(item: dict) -> dict:
+    replacements = {}
+    for k, v in item.get("metrics", {}).items():
+        if isinstance(v, list):
+            replacements[k] = ",".join(v)
+    if len(replacements) == 0:
+        return item
+    item["metrics"].update(replacements)
+    return item

ingestr-0.12.11/ingestr/src/google_ads/field.py

@@ -0,0 +1,2 @@
+def to_column(field: str) -> str:
+    return field.replace(".", "_")