ingestr 0.13.34__tar.gz → 0.13.36__tar.gz

ingestr-0.13.36/.gitleaksignore

@@ -0,0 +1,4 @@
+ingestr/src/telemetry/event.py:generic-api-key:17
+ingestr/src/testdata/fakebqcredentials.json:private-key:5
+docs/supported-sources/shopify.md:generic-api-key:26
+ingestr/src/google_analytics/helpers_test.py:generic-api-key:36

{ingestr-0.13.34 → ingestr-0.13.36}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.34
+Version: 0.13.36
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -46,7 +46,7 @@ Requires-Dist: databricks-sqlalchemy==1.0.2
 Requires-Dist: dataclasses-json==0.6.7
 Requires-Dist: decorator==5.2.1
 Requires-Dist: deprecation==2.1.0
-Requires-Dist: dlt==1.9.0
+Requires-Dist: dlt==1.10.0
 Requires-Dist: dnspython==2.7.0
 Requires-Dist: duckdb-engine==0.17.0
 Requires-Dist: duckdb==1.2.1

{ingestr-0.13.34 → ingestr-0.13.36}/docs/supported-sources/airtable.md

@@ -9,12 +9,11 @@ ingestr supports Airtable as a source.
 The URI format for Airtable is as follows:
 
 ```plaintext
- airtable://?access_token=<access_token>&base_id=<base_id>
+ airtable://?access_token=<access_token>
 ```
 
 URI parameters:
 
-- `base_id`: A unique identifier for an Airtable base.
 - `access_token`: A personal access token for authentication with the Airtable API.
 
 The URI is used to connect to the Airtable API for extracting data. More details on setting up Airtable integrations can be found [here](https://airtable.com/developers/web/api).
@@ -23,15 +22,19 @@ The URI is used to connect to the Airtable API for extracting data. More details
 
 Airtable requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/airtable#setup-guide).
 
-Once you complete the guide, you should have an Access Token and Base Id. Let's say your Access Token is `patr123.abc` and Base Id is `appXYZ`, here's a sample command that will copy the data from Airtable into a DuckDB database:
+Once you complete the guide, you should have an Access Token and a Base ID. The source table you'll use for ingestr will be `<base_id>/<table_name>`.
+
+Let's say your access token is `patr123.abc` and the base ID is `appXYZ`, here's a sample command that will copy the data from Airtable into a DuckDB database:
 
 ```sh
-ingestr ingest --source-uri 'airtable://?base_id=appXYc&access_token=patr123.abc' --source-table 'employee' --dest-uri 'duckdb:///airtable.duckdb' --dest-table 'des.employee'
+ingestr ingest 
+    --source-uri 'airtable://?access_token=patr123.abc' 
+    --source-table 'appXYZ/employee' 
+    --dest-uri 'duckdb:///airtable.duckdb' 
+    --dest-table 'des.employee'
 ```
 
-The result of this command will be an `employee` table containing data from the `employee` source in the `Airtable.duckdb` database.
-
-The `source-table` can include multiple table names that share the same `base_id`, e.g. `--source-table 'employee,users'`, but this will merge all the data from the specified tables into a single destination table.
+The result of this command will be an `employee` table containing data from the `employee` source in the `airtable.duckdb` database.
 
 > [!CAUTION]
 > Airtable does not support incremental loading, which means every time you run the command, the entire table will be copied from Airtable to the destination. This can be slow for large tables.

{ingestr-0.13.34 → ingestr-0.13.36}/docs/supported-sources/athena.md

@@ -10,7 +10,6 @@ The URI format for Athena is as follows:
 
 ```plaintext
 athena://?bucket=<your-destination-bucket> \
-    query_results_path=<your-query-results-location> \
     access_key_id=<your-aws-access-key-id> \
     secret_access_key=<your-aws-secret-access-key> \
     region_name=<your-aws-region>
@@ -20,7 +19,6 @@ URI parameters:
 - `access_key_id` and `secret_access_key` (required): These are AWS credentials that will be used to authenticate with AWS services like S3 and Athena.
 - `session_token` (optional): The session token for temporary credentials.
 - `region_name` (required if there's no local profile found): The AWS region of the Athena service and S3 buckets, e.g. `eu-central-1`
-- `query_results_path` (optional): The query location path where the results of Athena queries will be saved, e.g. `dest_path` or `s3://dest_path`. If not provided, it will default to the bucket specified in the `bucket` parameter.
 - `workgroup` (optional): The name of the Athena workgroup, e.g. `my_group`
 - `profile` (optional): The name of the AWS profile to use, e.g. `my_profile`
 

{ingestr-0.13.34 → ingestr-0.13.36}/docs/supported-sources/frankfurter.md

@@ -41,7 +41,9 @@ ingestr ingest \
 ### **`--interval-end` (Optional)**
 - **Description**: The end date for fetching historical exchange rates.
 - **Value**: A date in the format `YYYY-MM-DD` (e.g., `'2025-03-28'`).
-- **Purpose**: Defines the ending point for fetching historical data. If not provided, it defaults to the value of `--interval-start`.
+- **Purpose**: Defines the end point for fetching historical data. 
+    - If `--interval-start` is provided without `--interval-end`, `--interval-end` defaults to the current date and retrieves up until the latest published data.
+    - If `--interval-end` is provided without `--interval-start`, it will be ignored and the call will retrieve the last published data.
     - For `latest` and `currencies` this parameter is ignored.
 
 ---
@@ -104,7 +106,7 @@ ingestr ingest \
 - **Primary Key**: Composite key of `date` and `currency_name`.
 - **Notes**:
   - An optional start and end date can be added via the arguments `--interval-start` and optionally `--interval-end` to define the date range (see examples below). If no start date is specified, the date will default today's date (and thus return the latest exchange rates).
-  - If a start date but no end date is specified, then the end date will default to the start date and ingestr will retrieve data for the specified start date only.
+  - If a start date but no end date is specified, then the end date will default to today's date and ingestr will retrieve data up until the latest published data.
   - Note that the [Frankfurter API](https://www.frankfurter.dev/) only publishes updates Monday-Friday. If the given date is on the weekend, the date will default to the previous Friday. The source is however implemented in ingestr in such a way as to avoid duplicating rows of data in this case (see [Incremental Loading - Replace](https://bruin-data.github.io/ingestr/getting-started/incremental-loading.html)).
 
 #### **Example Table: Handling Weekend Dates**

ingestr-0.13.36/docs/supported-sources/freshdesk.md

@@ -0,0 +1,47 @@
+# Freshdesk
+
+[Freshdesk](https://www.freshworks.com/freshdesk/) is a cloud-based customer service platform that helps businesses manages customer support via multiple channels including email,phone,websites, and social media.
+
+ingestr supports Freshdesk as a source.
+
+## URI format
+
+The URI format for Freshdesk is:
+
+```
+freshdesk://<domain>?api_key=<api_key>
+```
+
+URI parameters:
+- `api_key` :  The API token used for authentication with Freshdesk.
+- `domain`: The domain of your Freshdesk account, found in your account URL. For example, if your account URL is https://my_company.freshdesk.com/, then `my_company` is your domain.
+
+The URI is used to connect to the Freshdesk API for extracting data.
+
+## Setting up a Freshdesk integration
+
+Freshdesk requires a few steps to set up an integration, please follow dltHub [setup guide](https://dlthub.com/docs/dlt-ecosystem/verified-sources/freshdesk#setup-guide).
+
+After completing the guide,, you will have your  `api_key` and `domain`. For example, if your api_key is
+`test_key` and domain `my_company`, you can use the following command to copy data from Freshdesk into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'freshdesk://$DOMAIN?api_key=$API_KEY' --source-table 'contacts' --dest-uri duckdb:///freshdesk.duckdb --dest-table 'dest.contacts'
+```
+This command copies data from the Freshdesk source to DuckDB.
+
+<img alt="freshdesk_img" src="../media/freshdesk_ingestion.png" />
+
+## Available Tables
+The Freshdesk source allows you to ingest the following tables:
+
+| S.No. | Name      | Description                                                                               |
+| ----- | --------- | ----------------------------------------------------------------------------------------- |
+| 1.    | agents    |  Users responsible for managing and resolving customer inquiries and support tickets.     |
+| 2.    | companies |  Customer organizations or groups that agents support.                                    |
+| 3.    | contacts  |  Individuals or customers who reach out for support.                                      |
+| 4.    | groups    |  Agents organized based on specific criteria.                                             |
+| 5.    | roles     |  Predefined sets of permissions that determine what actions an agent can perform.         |
+| 6.    | tickets   |  Customer inquiries or issues submitted via various channels like email, chat, phone, etc. |
+
+Use these as the `--source-table` parameter in the `ingestr ingest` command.

{ingestr-0.13.34 → ingestr-0.13.36}/docs/supported-sources/google_analytics.md

@@ -19,31 +19,64 @@ URI parameters:
 - `credentials_path`: The path to the service account JSON file.
 - `property_id`: It is a unique number that identifies a particular property on Google Analytics. [Follow this guide](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id#what_is_my_property_id) to know more about property ID.
 
-## Setting up an Google Analytics Integration
+## Setting up a Google Analytics Integration
 Google Analytics requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/google_analytics#grab-google-service-account-credentials). Once you complete the guide, you should have an `.json` file  and `project_id`. 
 
-## Table: Custom Reports
-Custom reports allow you to retrieve data based on specific `dimensions` and  `metrics`.
+## Available Tables:
+### Custom reports 
+- `Custom reports`: allow you to retrieve data based on specific `dimensions` and  `metrics`.
 
-Custom Table Format:
+ #### Custom Table Format:
 ```
 custom:<dimensions>:<metrics>
 ```
 
-### Parameters:
-- `dimensions`(required): A comma-separated list of [dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/exploration-api-schema#dimensions) to retrieve.
-- `metrics`(required): A comma-separated list of [metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/exploration-api-schema#metrics) to retrieve.
+ #### Parameters:
+- `dimensions`(required): A comma-separated list of [dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema#dimensions) to retrieve.
+- `metrics`(required): A comma-separated list of [metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema#metrics) to retrieve.
 
-### Example
+ #### Example
 
 ```sh
 ingestr ingest \
     --source-uri "googleanalytics://?credentials_path="ingestr/src/g_analytics.json&property_id=id123" \
-    --source-table "custom:city,date:clicks,activeUsers,newUsers" \
+    --source-table "custom:date:activeUsers" \
     --dest-uri "duckdb:///analytics.duckdb" \
     --dest-table "dest.custom"
 ```
 
 This command will retrieve report and save it to the `dest.custom` table in the DuckDB database.
 
-<img alt="google_analytics_img" src="../media/googleanalytics.png" />
+<img alt="google_analytics_img" src="../media/googleanalytics.png" />
+
+
+### Realtime reports
+`Realtime reports`: allows you to retrieve data based on specific `dimensions`, `metrics`, with optional `minutes_ranges`.
+
+ #### Realtime Report Table Format:
+```
+realtime:<dimensions>:<metrics>
+
+```
+```
+realtime:<dimensions>:<metrics>:<minutes_ranges>
+```
+
+ #### Parameters:
+- `dimensions`(required): A comma-separated list of [dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/exploration-api-schema#dimensions) to retrieve.
+- `metrics`(required): A comma-separated list of [metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/exploration-api-schema#metrics) to retrieve.
+- `minutes_ranges`(optional): Allows you to specify time windows for retrieving data. You can define up to two time ranges in your query, formatted as comma-separated values (e.g., "0-5,25-29"). Each range represents minutes in the past from the current time.
+If no minute_ranges are specified, the system defaults to retrieving data from the last 30 minutes. For more information read [here](https://developers.google.com/analytics/devguides/reporting/data/v1/realtime-basics#minute_ranges)
+
+#### Example
+
+```sh
+ingestr ingest \
+    --source-uri "googleanalytics://?credentials_path="ingestr/src/g_analytics.json&property_id=id123" \
+    --source-table "realtime:streamId:activeUsers:0-4,10-29" \
+    --dest-uri "duckdb:///analytics.duckdb" \
+    --dest-table "dest.realtime"
+```
+This command will retrieve report and save it to the `dest.realtime` table in the DuckDB database.
+
+<img alt="google_analytics_realtime_report_img" src="../media/google_analytics_realtime_report.png"/>

{ingestr-0.13.34 → ingestr-0.13.36}/ingestr/src/airtable/__init__.py

@@ -9,7 +9,7 @@ import pyairtable
 from dlt.sources import DltResource
 
 
-@dlt.source
+@dlt.source(max_table_nesting=1)
 def airtable_source(
     base_id: str = dlt.config.value,
     table_names: Optional[List[str]] = dlt.config.value,
@@ -50,12 +50,13 @@ def airtable_resource(
             It starts with "app". See https://support.airtable.com/docs/finding-airtable-ids
         table (Dict[str, Any]): Metadata about an airtable, does not contain the actual records
     """
+
     primary_key_id = table["primaryFieldId"]
     primary_key_field = [
         field for field in table["fields"] if field["id"] == primary_key_id
     ][0]
     table_name: str = table["name"]
-    primary_key: List[str] = [f"fields__{primary_key_field['name']}"]
+    primary_key: List[str] = [f"fields__{primary_key_field['name']}".lower()]
     air_table = api.table(base_id, table["id"])
 
     # Table.iterate() supports rich customization options, such as chunk size, fields, cell format, timezone, locale, and view

ingestr-0.13.36/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.36"

{ingestr-0.13.34 → ingestr-0.13.36}/ingestr/src/destinations.py

@@ -235,12 +235,19 @@ class AthenaDestination:
         if not bucket.startswith("s3://"):
             bucket = f"s3://{bucket}"
 
-        query_result_path = source_params.get("query_results_path", [None])[0]
-        if query_result_path:
-            if not query_result_path.startswith("s3://"):
-                query_result_path = f"s3://{query_result_path}"
-        else:
-            query_result_path = bucket
+        bucket = bucket.rstrip("/")
+
+        dest_table = kwargs.get("dest_table", None)
+        if not dest_table:
+            raise ValueError("A destination table is required to connect to Athena.")
+
+        dest_table_fields = dest_table.split(".")
+        if len(dest_table_fields) != 2:
+            raise ValueError(
+                f"Table name must be in the format <schema>.<table>, given: {dest_table}"
+            )
+
+        query_result_path = f"{bucket}/{dest_table_fields[0]}_staging/metadata"
 
         access_key_id = source_params.get("access_key_id", [None])[0]
         secret_access_key = source_params.get("secret_access_key", [None])[0]
@@ -285,6 +292,7 @@ class AthenaDestination:
                 region_name=region_name,
             ),
             destination_name=bucket,
+            force_iceberg=True,
         )
 
     def dlt_run_params(self, uri: str, table: str, **kwargs) -> dict:

{ingestr-0.13.34 → ingestr-0.13.36}/ingestr/src/factory.py

@@ -53,6 +53,7 @@ from ingestr.src.sources import (
     StripeAnalyticsSource,
     TikTokSource,
     ZendeskSource,
+    FreshdeskSource,
 )
 
 SQL_SOURCE_SCHEMES = [
@@ -148,6 +149,7 @@ class SourceDestinationFactory:
         "kinesis": KinesisSource,
         "pipedrive": PipedriveSource,
         "frankfurter": FrankfurterSource,
+        "freshdesk": FreshdeskSource,
     }
     destinations: Dict[str, Type[DestinationProtocol]] = {
         "bigquery": BigQueryDestination,

{ingestr-0.13.34 → ingestr-0.13.36}/ingestr/src/frankfurter/__init__.py

@@ -1,4 +1,4 @@
-from typing import Any, Iterator, Optional
+from typing import Any, Iterator
 
 import dlt
 from dlt.common.pendulum import pendulum
@@ -13,25 +13,28 @@ from ingestr.src.frankfurter.helpers import get_path_with_retry
     max_table_nesting=0,
 )
 def frankfurter_source(
-    table: str,
-    start_date: Optional[TAnyDateTime] = None,
-    end_date: Optional[TAnyDateTime] = None,
+    start_date: TAnyDateTime,
+    end_date: TAnyDateTime,
 ) -> Any:
     """
     A dlt source for the frankfurter.dev API. It groups several resources (in this case frankfurter.dev API endpoints) containing
     various types of data: currencies, latest rates, historical rates.
-
-    Returns the appropriate resource based on the provided parameters.
     """
-    # Determine which resource to return based on the `table` parameter
-    if table == "currencies":
-        return currencies()
+    date_time = dlt.sources.incremental(
+
+        "date",
+        initial_value=start_date,
+        end_value=end_date,
+        range_start="closed",
+        range_end="closed",
+    )
 
-    elif table == "latest":
-        return latest()
+    return (
+        currencies(),
+        latest(),
+        exchange_rates(start_date=date_time, end_date=end_date),
 
-    elif table == "exchange_rates":
-        return exchange_rates(start_date=start_date, end_date=end_date)
+    )
 
 
 @dlt.resource(
@@ -53,13 +56,13 @@ def currencies() -> Iterator[dict]:
 
 
 @dlt.resource(
-    write_disposition="replace",
+    write_disposition="merge",
     columns={
         "date": {"data_type": "text"},
-        "currency_name": {"data_type": "text"},
+        "currency_code": {"data_type": "text"},
         "rate": {"data_type": "double"},
     },
-    primary_key=["date", "currency_name"],  # Composite primary key
+    primary_key=["date", "currency_code"],  # Composite primary key
 )
 def latest() -> Iterator[dict]:
     """
@@ -69,50 +72,54 @@ def latest() -> Iterator[dict]:
     url = "latest?"
 
     # Fetch data
-    latest_data = get_path_with_retry(url)
+    data = get_path_with_retry(url)
 
     # Extract rates and base currency
-    rates = latest_data["rates"]
+    rates = data["rates"]
 
-    # Prepare the date
-    date = pendulum.now().to_date_string()
+    date = pendulum.parse(data["date"])
 
     # Add the base currency (EUR) with a rate of 1.0
     yield {
         "date": date,
-        "currency_name": "EUR",
+        "currency_code": "EUR",
         "rate": 1.0,
     }
 
     # Add all currencies and their rates
-    for currency_name, rate in rates.items():
+    for currency_code, rate in rates.items():
         yield {
             "date": date,
-            "currency_name": currency_name,
+            "currency_code": currency_code,
             "rate": rate,
         }
 
 
 @dlt.resource(
-    write_disposition="replace",
+    write_disposition="merge",
     columns={
         "date": {"data_type": "text"},
-        "currency_name": {"data_type": "text"},
+        "currency_code": {"data_type": "text"},
         "rate": {"data_type": "double"},
     },
-    primary_key=["date", "currency_name"],  # Composite primary key
+    primary_key=("date", "currency_code"),  # Composite primary key
 )
 def exchange_rates(
-    start_date: TAnyDateTime,
     end_date: TAnyDateTime,
+    start_date: dlt.sources.incremental[TAnyDateTime] = dlt.sources.incremental("date"),
 ) -> Iterator[dict]:
     """
     Fetches exchange rates for a specified date range.
-    If only start_date is provided, fetches data for that date.
+    If only start_date is provided, fetches data until now.
     If both start_date and end_date are provided, fetches data for each day in the range.
     """
-    start_date_str = ensure_pendulum_datetime(start_date).format("YYYY-MM-DD")
-    end_date_str = ensure_pendulum_datetime(end_date).format("YYYY-MM-DD")
+    # Ensure start_date.last_value is a pendulum.DateTime object
+    start_date_obj = ensure_pendulum_datetime(start_date.last_value)  # type: ignore
+    start_date_str = start_date_obj.format("YYYY-MM-DD")
+
+    # Ensure end_date is a pendulum.DateTime object
+    end_date_obj = ensure_pendulum_datetime(end_date)
+    end_date_str = end_date_obj.format("YYYY-MM-DD")
 
     # Compose the URL
     url = f"{start_date_str}..{end_date_str}?"
@@ -121,22 +128,23 @@ def exchange_rates(
     data = get_path_with_retry(url)
 
     # Extract base currency and rates from the API response
-    base_currency = data["base"]
     rates = data["rates"]
 
     # Iterate over the rates dictionary (one entry per date)
     for date, daily_rates in rates.items():
+        formatted_date = pendulum.parse(date)
+
         # Add the base currency with a rate of 1.0
         yield {
-            "date": date,
-            "currency_name": base_currency,
+            "date": formatted_date,
+            "currency_code": "EUR",
             "rate": 1.0,
         }
 
         # Add all other currencies and their rates
-        for currency_name, rate in daily_rates.items():
+        for currency_code, rate in daily_rates.items():
             yield {
-                "date": date,
-                "currency_name": currency_name,
+                "date": formatted_date,
+                "currency_code": currency_code,
                 "rate": rate,
             }

{ingestr-0.13.34 → ingestr-0.13.36}/ingestr/src/frankfurter/helpers.py

@@ -8,7 +8,7 @@ FRANKFURTER_API_URL = "https://api.frankfurter.dev/v1/"
 
 
 def get_url_with_retry(url: str) -> StrAny:
-    r = requests.get(url)
+    r = requests.get(url, timeout=5)
     return r.json()  # type: ignore
 
 
@@ -19,7 +19,7 @@ def get_path_with_retry(path: str) -> StrAny:
 def validate_dates(start_date: datetime, end_date: datetime) -> None:
     current_date = pendulum.now()
 
-    # Check if start_date is in the future
+    # Check if start_date is in the futurep
     if start_date > current_date:
         raise ValueError("Interval-start cannot be in the future.")
 

ingestr-0.13.36/ingestr/src/freshdesk/__init__.py

@@ -0,0 +1,72 @@
+"""This source uses Freshdesk API and dlt to load data such as Agents, Companies, Tickets
+etc. to the database"""
+
+from typing import Any, Dict, Generator, Iterable, List, Optional
+
+import dlt
+from dlt.sources import DltResource
+
+from .freshdesk_client import FreshdeskClient
+from .settings import DEFAULT_ENDPOINTS
+
+
+@dlt.source()
+def freshdesk_source(
+    endpoints: Optional[List[str]] = None,
+    per_page: int = 100,
+    domain: str = dlt.secrets.value,
+    api_secret_key: str = dlt.secrets.value,
+) -> Iterable[DltResource]:
+    """
+    Retrieves data from specified Freshdesk API endpoints.
+
+    This source supports pagination and incremental data loading. It fetches data from a list of
+    specified endpoints, or defaults to predefined endpoints in 'settings.py'.
+
+    Args:
+        endpoints: A list of Freshdesk API endpoints to fetch. Deafults to 'settings.py'.
+        per_page: The number of items to fetch per page, with a maximum of 100.
+        domain: The Freshdesk domain from which to fetch the data. Defaults to 'config.toml'.
+        api_secret_key: Freshdesk API key. Defaults to 'secrets.toml'.
+
+    Yields:
+        Iterable[DltResource]: Resources with data updated after the last 'updated_at'
+        timestamp for each endpoint.
+    """
+    # Instantiate FreshdeskClient with the provided domain and API key
+    freshdesk = FreshdeskClient(api_key=api_secret_key, domain=domain)
+
+    def incremental_resource(
+        endpoint: str,
+        updated_at: Optional[Any] = dlt.sources.incremental(
+            "updated_at", initial_value="2022-01-01T00:00:00Z"
+        ),
+    ) -> Generator[Dict[Any, Any], Any, None]:
+        """
+        Fetches and yields paginated data from a specified API endpoint.
+        Each page of data is fetched based on the `updated_at` timestamp
+        to ensure incremental loading.
+        """
+
+        # Retrieve the last updated timestamp to fetch only new or updated records.
+        if updated_at is not None:
+            updated_at = updated_at.last_value
+
+        # Use the FreshdeskClient instance to fetch paginated responses
+        yield from freshdesk.paginated_response(
+            endpoint=endpoint,
+            per_page=per_page,
+            updated_at=updated_at,
+        )
+
+    # Set default endpoints if not provided
+    endpoints = endpoints or DEFAULT_ENDPOINTS
+
+    # For each endpoint, create and yield a DLT resource
+    for endpoint in endpoints:
+        yield dlt.resource(
+            incremental_resource,
+            name=endpoint,
+            write_disposition="merge",
+            primary_key="id",
+        )(endpoint=endpoint)