ingestr 0.14.8__tar.gz → 0.14.91__tar.gz

{ingestr-0.14.8 → ingestr-0.14.91}/.github/workflows/tests.yml

@@ -13,7 +13,24 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  check-changes:
+    runs-on: ubuntu-latest
+    outputs:
+      docs-only: ${{ steps.filter.outputs.docs-only }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dorny/paths-filter@v3
+        id: filter
+        with:
+          filters: |
+            docs-only:
+              - '!**'
+              - '**/*.md'
+              - 'docs/**'
+
   tests:
+    needs: check-changes
+    if: needs.check-changes.outputs.docs-only != 'true'
     strategy:
       matrix:
         # I tried running stuff on macOS but it was too slow and unreliable.

{ingestr-0.14.8 → ingestr-0.14.91}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.14.8
+Version: 0.14.91
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

{ingestr-0.14.8 → ingestr-0.14.91}/docs/.vitepress/config.mjs

@@ -155,6 +155,7 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
               { text: "Klaviyo", link: "/supported-sources/klaviyo.md" },
               { text: "Linear", link: "/supported-sources/linear.md" },
               { text: "LinkedIn Ads", link: "/supported-sources/linkedin_ads.md" },
+              { text: "Mailchimp", link: "/supported-sources/mailchimp.md" },
               { text: "Mixpanel", link: "/supported-sources/mixpanel.md" },
               { text: "Monday", link: "/supported-sources/monday.md" },
               { text: "Notion", link: "/supported-sources/notion.md" },

{ingestr-0.14.8 → ingestr-0.14.91}/docs/supported-sources/freshdesk.md

@@ -44,6 +44,6 @@ The Freshdesk source allows you to ingest the following tables:
 | [groups](https://developers.freshdesk.com/api/#groups)       | id | updated_at     | merge               | Retrieves agents organized based on specific criteria.  |
 | [roles](https://developers.freshdesk.com/api/#roles)       | id | updated_at     | merge               | Retrieves predefined sets of permissions that determine what actions an agent can perform.|
 | [tickets](https://developers.freshdesk.com/api/#tickets)      | id | updated_at     | merge               | Retrieves customer inquiries or issues submitted via various channels like email, chat, phone, etc.
-| [tickets:<query>](https://developers.freshdesk.com/api/#filter_tickets) | id | updated_at | merge | Executes the [Freshdesk ticket filter query](https://developers.freshdesk.com/api/#filter_tickets) while preserving incremental sync. |
+| [tickets:<query>](https://developers.freshdesk.com/api/#filter_tickets) | id | updated_at | merge | Executes the [Freshdesk ticket filter query](https://developers.freshdesk.com/api/#filter_tickets) while preserving incremental sync. Maximum results: 300 tickets. |
 
 Use these as the `--source-table` parameter in the `ingestr ingest` command.

ingestr-0.14.91/docs/supported-sources/mailchimp.md

@@ -0,0 +1,99 @@
+# Mailchimp
+
+[Mailchimp](https://mailchimp.com/) is an all-in-one marketing platform that helps businesses manage and talk to their clients, customers, and other interested parties through email marketing campaigns, automated messages, and targeted ads.
+
+ingestr supports Mailchimp as a source.
+
+## URI format
+
+The URI format for Mailchimp is as follows:
+
+```plaintext
+mailchimp://?api_key=<api-key>&server=<server-prefix>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Mailchimp API
+- `server`: The server prefix for your Mailchimp account (e.g., `us10`, `us19`)
+
+The URI is used to connect to the Mailchimp API for extracting data.
+
+## Setting up a Mailchimp Integration
+
+To get your Mailchimp API credentials:
+
+1. Log in to your Mailchimp account
+2. Navigate to **Account** → **Extras** → **API keys**
+3. Create a new API key or use an existing one
+4. Note your server prefix from your API key (the part after the dash, e.g., `us10` from `xxxxx-us10`)
+
+Once you have your credentials, here's a sample command that will copy the data from Mailchimp into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'mailchimp://?api_key=your_api_key&server=us10' --source-table 'campaigns' --dest-uri duckdb:///mailchimp.duckdb --dest-table 'mailchimp.campaigns'
+```
+
+The result of this command will be a table in the `mailchimp.duckdb` database.
+
+## Tables
+
+Mailchimp source allows ingesting the following sources into separate tables:
+
+| Table | PK | Inc Key | Inc Strategy | Details |
+| ----- | -- | ------- | ------------ | ------- |
+| [account](https://mailchimp.com/developer/marketing/api/root/) | - | - | replace | Retrieves account information including company details, account tier, and contact information. |
+| [account_exports](https://mailchimp.com/developer/marketing/api/account-exports/) | - | - | replace | Retrieves account export information. |
+| [audiences](https://mailchimp.com/developer/marketing/api/lists/) | id | date_created | merge | Retrieves audience (list) information including subscriber counts and list settings. |
+| [authorized_apps](https://mailchimp.com/developer/marketing/api/authorized-apps/) | id | - | replace | Retrieves third-party applications authorized to access your account. |
+| [automations](https://mailchimp.com/developer/marketing/api/automations/) | id | create_time | merge | Retrieves automated email workflows and their configurations. |
+| [batches](https://mailchimp.com/developer/marketing/api/batches/) | - | - | replace | Retrieves batch operation status and results. |
+| [campaign_folders](https://mailchimp.com/developer/marketing/api/campaign-folders/) | id | - | replace | Retrieves folders used to organize campaigns. |
+| [campaigns](https://mailchimp.com/developer/marketing/api/campaigns/) | id | create_time | merge | Retrieves email campaigns including their content, settings, and metadata. |
+| [chimp_chatter](https://mailchimp.com/developer/marketing/api/activity-feed/) | - | - | replace | Retrieves recent activity feed from your Mailchimp account. |
+| [connected_sites](https://mailchimp.com/developer/marketing/api/connected-sites/) | id | updated_at | merge | Retrieves websites connected to your Mailchimp account. |
+| [conversations](https://mailchimp.com/developer/marketing/api/conversations/) | id | last_message.timestamp | merge | Retrieves conversation threads from connected channels. |
+| [ecommerce_stores](https://mailchimp.com/developer/marketing/api/ecommerce-stores/) | id | updated_at | merge | Retrieves e-commerce store information including products and orders. |
+| [facebook_ads](https://mailchimp.com/developer/marketing/api/facebook-ads/) | id | updated_at | merge | Retrieves Facebook ad campaigns managed through Mailchimp. |
+| [landing_pages](https://mailchimp.com/developer/marketing/api/landing-pages/) | id | updated_at | merge | Retrieves landing pages created in Mailchimp. |
+| lists_activity | - | - | replace | Retrieves recent activity for list members. Includes `audiences_id` reference. |
+| lists_clients | - | - | replace | Retrieves email clients used by list members. Includes `audiences_id` reference. |
+| lists_growth_history | - | - | replace | Retrieves historical growth data for the list. Includes `audiences_id` reference. |
+| lists_interest_categories | - | - | replace | Retrieves interest categories (groups) for the list. Includes `audiences_id` reference. |
+| lists_locations | - | - | replace | Retrieves geographic locations of list members. Includes `audiences_id` reference. |
+| lists_merge_fields | - | - | replace | Retrieves custom merge fields defined for the list. Includes `audiences_id` reference. |
+| lists_segments | - | - | replace | Retrieves segments (filtered subsets) of the list. Includes `audiences_id` reference. |
+| [reports](https://mailchimp.com/developer/marketing/api/reports/) | id | send_time | merge | Retrieves campaign performance reports and analytics. |
+| reports_advice | - | - | replace | Retrieves feedback and suggestions for improving campaign performance. Includes `reports_id` reference. |
+| reports_domain_performance | - | - | replace | Retrieves email performance broken down by email domain. Includes `reports_id` reference. |
+| reports_locations | - | - | replace | Retrieves geographic location data for campaign opens. Includes `reports_id` reference. |
+| reports_sent_to | - | - | replace | Retrieves list of recipients who were sent the campaign. Includes `reports_id` reference. |
+| reports_sub_reports | - | - | replace | Retrieves sub-reports for A/B test campaigns. Includes `reports_id` reference. |
+| reports_unsubscribed | - | - | replace | Retrieves list of members who unsubscribed from the campaign. Includes `reports_id` reference. |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+## Examples
+
+### Ingesting Campaign Data
+
+```sh
+ingestr ingest \
+  --source-uri 'mailchimp://?api_key=your_api_key&server=us10' \
+  --source-table 'campaigns' \
+  --dest-uri duckdb:///mailchimp.duckdb \
+  --dest-table 'mailchimp.campaigns'
+```
+
+
+
+The `reports_advice` table will include a `reports_id` column that references the parent campaign report.
+
+
+The `lists_segments` table will include an `audiences_id` column that references the parent audience/list.
+
+## Notes
+
+> [!NOTE]
+> Nested resources (e.g., `reports_advice`, `lists_segments`) automatically include a reference to their parent resource through an ID column (`reports_id` or `audiences_id`).
+

ingestr-0.14.91/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.14.91"

{ingestr-0.14.8 → ingestr-0.14.91}/ingestr/src/factory.py

@@ -64,6 +64,7 @@ from ingestr.src.sources import (
     LinearSource,
     LinkedInAdsSource,
     LocalCsvSource,
+    MailchimpSource,
     MixpanelSource,
     MondaySource,
     MongoDbSource,
@@ -219,6 +220,7 @@ class SourceDestinationFactory:
         "wise": WiseSource,
         "plusvibeai": PlusVibeAISource,
         "monday": MondaySource,
+        "mailchimp": MailchimpSource,
     }
     destinations: Dict[str, Type[DestinationProtocol]] = {
         "bigquery": BigQueryDestination,

{ingestr-0.14.8 → ingestr-0.14.91}/ingestr/src/freshdesk/freshdesk_client.py

@@ -10,6 +10,8 @@ from dlt.sources.helpers import requests
 
 from ingestr.src.errors import HTTPError
 
+TICKETS_QUERY_MAX_PAGE = 10
+
 
 class FreshdeskClient:
     """
@@ -85,6 +87,8 @@ class FreshdeskClient:
         if query is not None:
             query = query.replace('"', "").strip()
 
+        is_tickets_query = query and endpoint == "tickets"
+
         while True:
             # Construct the URL for the specific endpoint
             url = f"{self.base_url}/{endpoint}"
@@ -99,7 +103,7 @@ class FreshdeskClient:
 
                 params[param_key] = start_date.to_iso8601_string()
 
-            if query and endpoint == "tickets":
+            if is_tickets_query:
                 url = f"{self.base_url}/search/tickets"
                 params = {
                     "query": f'"{query}"',
@@ -127,3 +131,7 @@ class FreshdeskClient:
                 break
             yield filtered_data
             page += 1
+
+            # https://developers.freshdesk.com/api/#filter_tickets
+            if is_tickets_query and page > TICKETS_QUERY_MAX_PAGE:
+                break

ingestr-0.14.91/ingestr/src/mailchimp/__init__.py

@@ -0,0 +1,126 @@
+"""
+Mailchimp source for data extraction via REST API.
+
+This source provides access to Mailchimp account data.
+"""
+
+from typing import Any, Iterable, Iterator
+
+import dlt
+from dlt.sources import DltResource
+
+from ingestr.src.http_client import create_client
+from ingestr.src.mailchimp.helpers import (
+    create_merge_resource,
+    create_nested_resource,
+    create_replace_resource,
+)
+from ingestr.src.mailchimp.settings import (
+    MERGE_ENDPOINTS,
+    NESTED_ENDPOINTS,
+    REPLACE_ENDPOINTS,
+)
+
+
+@dlt.source(max_table_nesting=0, name="mailchimp_source")
+def mailchimp_source(
+    api_key: str,
+    server: str,
+) -> Iterable[DltResource]:
+    """
+    Mailchimp data source.
+
+    Args:
+        api_key: Mailchimp API key for authentication
+        server: Server prefix (e.g., 'us10')
+
+    Yields:
+        DltResource: Data resources for Mailchimp data
+    """
+    base_url = f"https://{server}.api.mailchimp.com/3.0"
+    session = create_client()
+    auth = ("anystring", api_key)
+
+    @dlt.resource(
+        name="account",
+        write_disposition="replace",
+    )
+    def fetch_account() -> Iterator[dict[str, Any]]:
+        """
+        Fetch account information from Mailchimp.
+
+        Table format: account (no parameters needed)
+        """
+        response = session.get(f"{base_url}/", auth=auth)
+        response.raise_for_status()
+        data = response.json()
+        yield data
+
+    # Create resources dynamically
+    resources = [fetch_account]
+
+    # Create merge resources (with incremental loading)
+    for (
+        resource_name,
+        endpoint_path,
+        data_key,
+        primary_key,
+        incremental_key,
+    ) in MERGE_ENDPOINTS:
+        resources.append(
+            create_merge_resource(
+                base_url,
+                session,
+                auth,
+                resource_name,
+                endpoint_path,
+                data_key,
+                primary_key,
+                incremental_key,
+            )
+        )
+
+    # Create replace resources (without incremental loading)
+    for replace_endpoint in REPLACE_ENDPOINTS:
+        resource_name, endpoint_path, data_key, pk = replace_endpoint
+        resources.append(
+            create_replace_resource(
+                base_url,
+                session,
+                auth,
+                resource_name,
+                endpoint_path,
+                data_key,
+                pk,
+            )
+        )
+
+    # Create nested resources (depend on parent resources)
+    for nested_endpoint in NESTED_ENDPOINTS:
+        (
+            parent_name,
+            parent_path,
+            parent_key,
+            parent_id_field,
+            nested_name,
+            nested_path,
+            nested_key,
+            pk,
+        ) = nested_endpoint
+        resources.append(
+            create_nested_resource(
+                base_url,
+                session,
+                auth,
+                parent_name,
+                parent_path,
+                parent_key,
+                parent_id_field,
+                nested_name,
+                nested_path,
+                nested_key,
+                pk,
+            )
+        )
+
+    return tuple(resources)

ingestr-0.14.91/ingestr/src/mailchimp/helpers.py

@@ -0,0 +1,226 @@
+"""
+Helper functions for Mailchimp source.
+"""
+
+from typing import Any, Iterator
+
+import dlt
+
+
+def fetch_paginated(
+    session,
+    url: str,
+    auth: tuple,
+    data_key: str | None = None,
+) -> Iterator[dict[str, Any]]:
+    """
+    Helper function to fetch paginated data from Mailchimp API.
+
+    Args:
+        session: HTTP session
+        url: API endpoint URL
+        auth: Authentication tuple
+        data_key: Key in response containing the data array (if None, return whole response)
+
+    Yields:
+        Individual items from the paginated response
+    """
+    offset = 0
+    count = 1000  # Maximum allowed by Mailchimp
+
+    while True:
+        params = {"count": count, "offset": offset}
+        response = session.get(url, auth=auth, params=params)
+        response.raise_for_status()
+        data = response.json()
+
+        # Extract items from response
+        if data_key and data_key in data:
+            items = data[data_key]
+        elif isinstance(data, list):
+            items = data
+        else:
+            # If no data_key specified and response is dict, yield the whole response
+            yield data
+            break
+
+        if not items:
+            break
+
+        yield from items
+
+        # Check if we've received fewer items than requested (last page)
+        if len(items) < count:
+            break
+
+        offset += count
+
+
+def create_merge_resource(
+    base_url: str,
+    session,
+    auth: tuple,
+    name: str,
+    path: str,
+    key: str,
+    pk: str,
+    ik: str,
+):
+    """
+    Create a DLT resource with merge disposition for incremental loading.
+
+    Args:
+        base_url: Base API URL
+        session: HTTP session
+        auth: Authentication tuple
+        name: Resource name
+        path: API endpoint path
+        key: Data key in response
+        pk: Primary key field
+        ik: Incremental key field
+
+    Returns:
+        DLT resource function
+    """
+
+    @dlt.resource(
+        name=name,
+        write_disposition="merge",
+        primary_key=pk,
+    )
+    def fetch_data(
+        updated_at: dlt.sources.incremental[str] = dlt.sources.incremental(
+            ik, initial_value=None
+        ),
+    ) -> Iterator[dict[str, Any]]:
+        url = f"{base_url}/{path}"
+        yield from fetch_paginated(session, url, auth, data_key=key)
+
+    return fetch_data
+
+
+def create_replace_resource(
+    base_url: str,
+    session,
+    auth: tuple,
+    name: str,
+    path: str,
+    key: str,
+    pk: str | None,
+):
+    """
+    Create a DLT resource with replace disposition.
+
+    Args:
+        base_url: Base API URL
+        session: HTTP session
+        auth: Authentication tuple
+        name: Resource name
+        path: API endpoint path
+        key: Data key in response
+        pk: Primary key field (optional)
+
+    Returns:
+        DLT resource function
+    """
+
+    def fetch_data() -> Iterator[dict[str, Any]]:
+        url = f"{base_url}/{path}"
+        yield from fetch_paginated(session, url, auth, data_key=key)
+
+    # Apply the resource decorator with conditional primary_key
+    if pk is not None:
+        return dlt.resource(
+            fetch_data,
+            name=name,
+            write_disposition="replace",
+            primary_key=pk,
+        )
+    else:
+        return dlt.resource(
+            fetch_data,
+            name=name,
+            write_disposition="replace",
+        )
+
+
+def create_nested_resource(
+    base_url: str,
+    session,
+    auth: tuple,
+    parent_resource_name: str,
+    parent_path: str,
+    parent_key: str,
+    parent_id_field: str,
+    nested_name: str,
+    nested_path: str,
+    nested_key: str | None,
+    pk: str | None,
+):
+    """
+    Create a nested DLT resource that depends on a parent resource.
+
+    Args:
+        base_url: Base API URL
+        session: HTTP session
+        auth: Authentication tuple
+        parent_resource_name: Name of the parent resource
+        parent_path: Parent API endpoint path
+        parent_key: Data key in parent response
+        parent_id_field: Field name for parent ID
+        nested_name: Nested resource name
+        nested_path: Nested API endpoint path (with {id} placeholder)
+        nested_key: Data key in nested response (None to return whole response)
+        pk: Primary key field (optional)
+
+    Returns:
+        DLT resource function
+    """
+
+    def fetch_nested_data() -> Iterator[dict[str, Any]]:
+        # First, fetch parent items
+        parent_url = f"{base_url}/{parent_path}"
+        parent_items = fetch_paginated(session, parent_url, auth, data_key=parent_key)
+
+        # For each parent item, fetch nested data
+        for parent_item in parent_items:
+            parent_id = parent_item.get(parent_id_field)
+            if parent_id:
+                # Build nested URL with parent ID
+                nested_url = f"{base_url}/{nested_path.format(id=parent_id)}"
+
+                # Fetch nested data
+                response = session.get(nested_url, auth=auth)
+                response.raise_for_status()
+                data = response.json()
+
+                # Extract nested items or return whole response
+                if nested_key and nested_key in data:
+                    items = data[nested_key]
+                    if isinstance(items, list):
+                        for item in items:
+                            # Add parent reference
+                            item[f"{parent_resource_name}_id"] = parent_id
+                            yield item
+                    else:
+                        items[f"{parent_resource_name}_id"] = parent_id
+                        yield items
+                else:
+                    # Return whole response with parent reference
+                    data[f"{parent_resource_name}_id"] = parent_id
+                    yield data
+
+    # Apply the resource decorator with conditional primary_key
+    if pk is not None:
+        return dlt.resource(
+            fetch_nested_data,
+            name=nested_name,
+            write_disposition="replace",
+            primary_key=pk,
+        )
+    else:
+        return dlt.resource(
+            fetch_nested_data,
+            name=nested_name,
+            write_disposition="replace",
+        )