ingestr 0.13.86__tar.gz → 0.13.87__tar.gz

{ingestr-0.13.86 → ingestr-0.13.87}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.86
+Version: 0.13.87
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

{ingestr-0.13.86 → ingestr-0.13.87}/docs/.vitepress/config.mjs

@@ -147,6 +147,7 @@ export default defineConfig({
               { text: "Pinterest", link: "/supported-sources/pinterest.md" },
               { text: "Pipedrive", link: "/supported-sources/pipedrive.md" },
               { text: "QuickBooks", link: "/supported-sources/quickbooks.md" },
+              { text: "RevenueCat", link: "/supported-sources/revenuecat.md" },
               { text: "S3", link: "/supported-sources/s3.md" },
               { text: "Salesforce", link: "/supported-sources/salesforce.md" },
               { text: "SFTP", link: "/supported-sources/sftp.md"},

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/appstore.md

@@ -69,7 +69,7 @@ We will run `ingestr` to save this data to a [duckdb](https://duckdb.org/) datab
 
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -80,7 +80,7 @@ ingestr ingest \
 We will extend the prior example with another app with ID `67890`. To achieve this, simply add another `app_id` query parameter to the URI.
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&app_id=67890&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&app_id=67890&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -94,7 +94,7 @@ ingestr ingest \
 To begin, we will first load all data till `2025-01-01` by specifying the `--interval-end` flag. We'll assume the same credentials from our [first example](#example-loading-app-downloads-analytics)
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -105,7 +105,7 @@ ingestr ingest \
 
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -115,6 +115,14 @@ ingestr ingest \
 Notice how we didn't specify a date parameter? `ingestr` will automatically use the metadata from last load and continue loading data from that point on.
 
 ## Tables
+| Table | PK | Inc Key | Inc Strategy | Details |
+|-------|----|---------|--------------|---------|
+| `app-downloads-detailed` | [App Apple Identifier,App Name, App Version,Campaign,Date,Device,Download Type,Page Title,Page Type,Platform Version,Pre-Order,Source Info,Source Type,Territory] | processing_date | merge | App download analytics including first-time downloads, redownloads, updates, and more. |
+| `app-store-discovery-and-engagement-detailed` |  primary_key = [App Apple Identifier,App Name,Campaign,Date,Device,Engagement Type,Event,Page Title,Page Type,Platform Version,Source Info,Source Type,Territory] | processing_date | merge | App Store discovery and engagement metrics including data about user engagement with your app’s icons, product pages, in-app event pages, and other install sheets. |
+|`app-sessions-detailed` |  primary_key=[Date,App Name,App Apple Identifier,App Version,Device,Platform Version,Source Type,Source Info,Campaign,Page Type,Page Title,App Download Date,Territory] | processing_date | merge | App Session provides insights on how often people open your app, and how long they spend in your app. |
+| app-store-installation-and-deletion-detailed | [App Apple Identifier,App Download Date,App Name,App Version,Campaign,Counts,Date,Device,Download Type,Event,Page Title,Page Type,Platform Version,Source Info,Source Type,Territory,Unique Devices] | processing_date | merge | App installation and deletion metrics including device to estimate the number of times people install and delete your App Store apps. |
+| `app-store-purchases-detailed`|  [App Apple Identifier,App Download Date,App Name,Campaign,Content Apple Identifier,Content Name,Date,Device,Page Title,Page Type,Payment Method,Platform Version,Pre-Order,Purchase Type,Source Info,Source Type,Territory] | processing_date | merge | App purchase analytics including revenue, payment methods, and content details. |
+| `app-crashes-expanded`| [App Name,App Version,Build,Date,Device,Platform,Release Type,Territory] |processing_date | merge | App crash analytics including crash counts, device information, and version details. |
 
 ### `app-downloads-detailed`
 The App Downloads Report includes download data generated on the App Store. You can use this report to understand your total number of downloads, including first-time downloads, redownloads, updates, and more.

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/chess.md

@@ -29,9 +29,10 @@ The result of this command will be a table in the `chess.duckdb` database.
 ## Tables
 
 Chess source allows ingesting the following sources into separate tables:
-
-- `profiles`: Retrieves player profiles based on a list of player usernames.
-- `games`: Retrieves players' games for specified players.
-- `archives`: Retrieves the URLs to game archives for specified players.
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `profiles`        | - | –                | replace               | Retrieves player profiles based on a list of player usernames.                                |
+| `games` | - | –                | replace               | Retrieves players games for specified players.   
+| `archives`        | - | –                | replace               | Retrieves the URLs to game archives for specified players.                                      |                                                 
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.

ingestr-0.13.87/docs/supported-sources/clickup.md

@@ -0,0 +1,45 @@
+# ClickUp
+[ClickUp](https://clickup.com/) is a cloud-based productivity platform designed to help teams and individuals manage tasks, projects, and workflows in one place.
+
+ingestr supports ClickUp as a source.
+
+## URI format
+
+```
+clickup://?api_token=<token>
+```
+
+URI parameters:
+- `api_token` is a personal token used to authenticate with the ClickUp API.
+
+ClickUp requires a `api_token` to connect to the ClickUP API. For more information, read [here](https://developer.clickup.com/docs/authentication#generate-your-personal-api-token) to get credentail. 
+
+Once you've completed the guide, you should have `api_token`. Let's say your API Token  is `token_123`, here's a sample command that will copy the data from Clickup into a DuckDB database:
+
+To ingest tasks from ClickUp into a DuckDB database:
+
+```sh
+ingestr ingest \
+  --source-uri "clickup://?api_token=token_123" \
+  --source-table "tasks" \
+  --dest-uri duckdb:///clickup.duckdb \
+  --dest-table "public.tasks"
+```
+<img alt="clickup_img" src="../media/clickup_ingestion.png"/>
+
+## Tables
+
+ClickUp source allows ingesting the following resources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [user](https://developer.clickup.com/reference/getauthorizeduser)        | id | -             | merge               | The authorised user profile                       |
+| [teams](https://developer.clickup.com/reference/getauthorizedteams) | id | –                | merge               | Workspaces available to the authenticated user. In ClickUp, a "team" in the API is equivalent to a workspace, which is the highest-level container for all spaces, folders, lists, and tasks |
+| [spaces](https://developer.clickup.com/reference/getspaces)   | id | -     | merge  | Spaces available within a workspace. Spaces are major sections used to organize work, such as departments, projects, or business units. Each space can contain folders, lists, and tasks |
+| [lists](https://developer.clickup.com/reference/getlists)   | id | –  | merge               | Lists contained in each space. Lists are collections of tasks, often representing projects, sprints, or workflow stages |
+| [tasks](https://developer.clickup.com/reference/gettasks)    | id | date_updated             | merge            | Tasks belonging to each list |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> ClickUp does not support incremental loading for many endpoints, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/facebook-ads.md

@@ -2,7 +2,7 @@
 
 Facebook Ads is the advertising platform that helps users to create targeted ads on Facebook, Instagram and Messenger.
 
-ingestr supports Facebook Ads as a source.
+ingestr supports Facebook Ads as a source using [Facebook Marketing API](https://developers.facebook.com/docs/marketing-api/).
 
 ## URI format
 
@@ -41,113 +41,14 @@ The result of this command will be a table in the `facebook.duckdb` database.
 
 Facebook Ads source allows ingesting the following sources into separate tables:
 
-- `campaigns`: Retrieves campaign data with fields:
-  - `id`
-  - `updated_time`
-  - `created_time`
-  - `name`
-  - `status`
-  - `effective_status`
-  - `objective`
-  - `start_time`
-  - `stop_time`
-  - `daily_budget`
-  - `lifetime_budget`
-
-- `ad_sets`: Retrieves ad set data with fields:
-  - `id`
-  - `updated_time`
-  - `created_time`
-  - `name`
-  - `status`
-  - `effective_status`
-  - `campaign_id`
-  - `start_time`
-  - `end_time`
-  - `daily_budget`
-  - `lifetime_budget`
-  - `optimization_goal`
-  - `promoted_object`
-  - `billing_event`
-  - `bid_amount`
-  - `bid_strategy`
-  - `targeting`
-
-- `leads`: Retrieves lead data with fields:
-  - `id`
-  - `created_time`
-  - `ad_id`
-  - `ad_name`
-  - `adset_id`
-  - `adset_name`
-  - `campaign_id`
-  - `campaign_name`
-  - `form_id`
-  - `field_data`
-
-- `ads_creatives`: Retrieves ad creative data with fields:
-  - `id`
-  - `name`
-  - `status`
-  - `thumbnail_url`
-  - `object_story_spec`
-  - `effective_object_story_id`
-  - `call_to_action_type`
-  - `object_type`
-  - `template_url`
-  - `url_tags`
-  - `instagram_actor_id`
-  - `product_set_id`
-
-- `ads`: Retrieves ad data with fields:
-  - `id`
-  - `updated_time`
-  - `created_time`
-  - `name`
-  - `status`
-  - `effective_status`
-  - `adset_id`
-  - `campaign_id`
-  - `creative`
-  - `targeting`
-  - `tracking_specs`
-  - `conversion_specs`
-
-- `facebook_insights`: Retrieves insights data with fields:
-  - `campaign_id`
-  - `adset_id`
-  - `ad_id`
-  - `date_start`
-  - `date_stop`
-  - `reach`
-  - `impressions`
-  - `frequency`
-  - `clicks`
-  - `unique_clicks`
-  - `ctr`
-  - `unique_ctr`
-  - `cpc`
-  - `cpm`
-  - `cpp`
-  - `spend`
-  - `actions`
-  - `action_values`
-  - `cost_per_action_type`
-  - `website_ctr`
-  - `account_currency`
-  - `ad_click_actions`
-  - `ad_name`
-  - `adset_name`
-  - `campaign_name`
-  - `country`
-  - `dma`
-  - `full_view_impressions`
-  - `full_view_reach`
-  - `inline_link_click_ctr`
-  - `outbound_clicks`
-  - `social_spend`
-  - `conversions`
-  - `video_thruplay_watched_actions`
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `campaigns`       | id | –  |        replace     | Retrieves campaign data with `fields`: id, updated_time, created_time, name, status, effective_status, objective, start_time, stop_time, daily_budget, lifetime_budget              |
+| `ad_sets` | id | –                | replace            | Retrieves ad set data with `fields`: id, updated_time, created_time, name, status, effective_status, campaign_id, start_time, end_time, daily_budget, lifetime_budget, optimization_goal, promoted_object, billing_event, bid_amount, bid_strategy, targeting                       |
+| `ads`   | id | -     | replace  | Retrieves ad data with `fields`: id, updated_time, created_time, name, status, effective_status, adset_id, campaign_id, creative, targeting, tracking_specs, conversion_specs                          |
+| `ad_creatives`   | id | -     | replace  | Retrieves ad creative data with `fields`: id, name, status, thumbnail_url, object_story_spec, effective_object_story_id, call_to_action_type, object_type, template_url, url_tags, instagram_actor_id, product_set_id |
+| `leads`   | id | -     | replace  | Retrieves lead data with fields: id, created_time, ad_id, ad_name, adset_id, adset_name, campaign_id, campaign_name, form_id, field_data |
+| `facebook_insights`   | date_start | date_start     | merge  | Retrieves insights data with `fields`: campaign_id, adset_id, ad_id, date_start, date_stop, reach, impressions, frequency, clicks, unique_clicks, ctr, unique_ctr, cpc, cpm, cpp, spend, actions, action_values, cost_per_action_type, website_ctr, account_currency, ad_click_actions, ad_name, adset_name, campaign_name, country, dma, full_view_impressions, full_view_reach, inline_link_click_ctr, outbound_clicks, social_spend, conversions, video_thruplay_watched_actions. |
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/frankfurter.md

@@ -34,9 +34,13 @@ The result of this command will be a list of currency exchange rates from 20.03.
 
 ## Tables
 
-- `latest`: Fetches the latest exchange rates.
-- `exchange_rates`: Fetches historical exchange rates for a specified date range.
-- `currencies`: Fetches a list of the available currencies and their ISO 4217 currency code (e.g. `USD`, `EUR`, `GBP`).
+Frankfurter source allows ingesting the following sources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [currencies](https://www.frankfurter.dev/docs/#currencies)  | - | –                | replace               | Retrieves list of available currencies with ISO 4217 codes and names. |
+| [latest](https://www.frankfurter.dev/docs/#latest)    | ["date", "currency_code", "base_currency"] | –                | merge               | Fetches latest exchange rates for all currencies. |
+| [exchange_rates](https://www.frankfurter.dev/docs/#historical) | ["date", "currency_code", "base_currency"] | date    | merge               | Retrieves historical exchange rates for specified date range.|
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/freshdesk.md

@@ -33,15 +33,16 @@ This command copies data from the Freshdesk source to DuckDB.
 <img alt="freshdesk_img" src="../media/freshdesk_ingestion.png" />
 
 ## Available Tables
+
 The Freshdesk source allows you to ingest the following tables:
 
-| S.No. | Name      | Description                                                                               |
-| ----- | --------- | ----------------------------------------------------------------------------------------- |
-| 1.    | agents    |  Users responsible for managing and resolving customer inquiries and support tickets.     |
-| 2.    | companies |  Customer organizations or groups that agents support.                                    |
-| 3.    | contacts  |  Individuals or customers who reach out for support.                                      |
-| 4.    | groups    |  Agents organized based on specific criteria.                                             |
-| 5.    | roles     |  Predefined sets of permissions that determine what actions an agent can perform.         |
-| 6.    | tickets   |  Customer inquiries or issues submitted via various channels like email, chat, phone, etc. |
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [agents](https://developers.freshdesk.com/api/#agents)    | id | updated_at    | merge               | Retrieves users responsible for managing and resolving customer inquiries and support tickets |
+| [companies](https://developers.freshdesk.com/api/#companies)     | id | updated_at     | merge               | Retrieves customer organizations or groups that agents support. |
+| [contacts](https://developers.freshdesk.com/api/#contacts)     | id | updated_at     | merge               | Retrieves individuals or customers who reach out for support. |
+| [groups](https://developers.freshdesk.com/api/#groups)       | id | updated_at     | merge               | Retrieves agents organized based on specific criteria.  |
+| [roles](https://developers.freshdesk.com/api/#roles)       | id | updated_at     | merge               | Retrieves predefined sets of permissions that determine what actions an agent can perform.|
+| [tickets](https://developers.freshdesk.com/api/#tickets)      | id | updated_at     | merge               | Retrieves customer inquiries or issues submitted via various channels like email, chat, phone, etc.
 
 Use these as the `--source-table` parameter in the `ingestr ingest` command.

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/google_analytics.md

@@ -23,6 +23,13 @@ URI parameters:
 Google Analytics requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/google_analytics#grab-google-service-account-credentials). Once you complete the guide, you should have an `.json` file  and `project_id`. 
 
 ## Available Tables:
+
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [realtime](https://developers.google.com/analytics/devguides/reporting/data/v1/realtime-basics)     | ingested_at |  -    | merge               | Retrieves real-time analytics data based on specified dimensions and metrics. Format: `realtime:<dimensions>:<metrics>:<minutes_ranges>`. Supports incremental loading by ingestion timestamp. |
+| `custom` | datetime_dimension | datetime_dimension    |     merge           | Retrieves custom reports based on specified dimensions and metrics. Format: `custom:<dimensions>:<metrics>` |
+
 ### Custom reports 
 - `Custom reports`: allow you to retrieve data based on specific `dimensions` and  `metrics`.
 
@@ -79,4 +86,4 @@ ingestr ingest \
 ```
 This command will retrieve report and save it to the `dest.realtime` table in the DuckDB database.
 
-<img alt="google_analytics_realtime_report_img" src="../media/google_analytics_realtime_report.png"/>
+<img alt="google_analytics_realtime_report_img" src="../media/google_analytics_realtime_report.png"/>

ingestr-0.13.87/docs/supported-sources/gorgias.md

@@ -0,0 +1,42 @@
+# Gorgias
+[Gorgias](https://www.gorgias.com/) is a helpdesk for e-commerce merchants, providing customer service via email, social media, SMS, and live chat.
+
+ingestr supports Gorgias as a source.
+
+## URI format
+The URI format for Gorgias is as follows:
+
+```plaintext
+gorgias://<domain>?api_key=<api-key>&email=<email>
+```
+
+URI parameters:
+- `domain`: the domain of the Gorgias account without the full `gorgias.com`, e.g. `mycompany`
+- `api_key`: the integration token used for authentication with the Gorgias API
+- `email`: the email address of the user to connect to the Gorgias API
+
+The URI is used to connect to the Gorgias API for extracting data.
+
+## Examples
+```bash
+# get all the tickets that are created/updated since 2024-06-19 and write them to `gorgias.ticket_messages` table on BigQuery
+ingestr ingest --source-table 'tickets' --source-uri $GORGIAS_URI --dest-uri $BIGQUERY_URI --interval-start 2024-06-19  --dest-table 'gorgias.ticket_messages' --loader-file-format jsonl
+
+# get all the customers and write them to `gorgias.customers` table on DuckDB
+ingestr ingest --source-table 'customers' --source-uri $GORGIAS_URI --dest-uri duckdb:///gorgias.duckdb --interval-start 2024-01-01  --dest-table 'dest.customers'
+```
+
+
+
+Gorgias source allows ingesting the following sources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [customers](https://developers.gorgias.com/reference/list-customers)     | id | updated_datetime     | merge               | Customers are the users who have interacted with the support team. Each customer has a unique ID and contains information such as the name and email.  Retrieves customers lists|
+| [tickets](https://developers.gorgias.com/reference/list-tickets)  | id | updated_datetime    | merge               | Tickets are the main entity in Gorgias, representing customer inquiries. Each ticket has a unique ID and contains information such as the customer, status, and messages. Retrieves tickets lists |
+| [ticket_messages](https://developers.gorgias.com/reference/list-messages) | id | updated_datetime    | merge               | Ticket messages are the messages exchanged between the customer and the support agent in a ticket. Each message has a unique ID and contains information such as the sender, content, and timestamp. Retrieves messages lists | 
+| [satisfaction_surveys](https://developers.gorgias.com/reference/list-satisfaction-surveys) | id | updated_datetime     | merge               | Satisfaction surveys are sent to customers after a ticket is resolved to gather feedback on their experience. Each survey has a unique ID and contains information such as the rating and comments. Retrieves surveys lists.|
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+

ingestr-0.13.87/docs/supported-sources/hubspot.md

@@ -0,0 +1,52 @@
+# HubSpot
+
+[HubSpot](https://www.hubspot.com/) is a customer relationship management software that helps businesses attract visitors, connect with customers, and close deals.
+
+ingestr supports HubSpot as a source.
+
+## URI format
+
+The URI format for HubSpot is as follows:
+
+```plaintext
+hubspot://?api_key=<api-key-here>
+```
+
+URI parameters:
+
+- `api_key`: The API key is used for authentication with the HubSpot API.
+
+The URI is used to connect to the HubSpot API for extracting data.
+
+## Setting up a HubSpot Integration
+
+HubSpot requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot#setup-guide).
+
+Once you complete the guide, you should have an API key. Let's say your API key is `pat_test_12345`, here's a sample command that will copy the data from HubSpot into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'hubspot://?api_key=pat_test_12345' --source-table 'companies' --dest-uri duckdb:///hubspot.duckdb --dest-table 'companies.data'
+```
+
+The result of this command will be a table in the `hubspot.duckdb` database.
+
+## Tables
+
+HubSpot source allows ingesting the following sources into separate tables:
+ 
+Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [companies](https://developers.hubspot.com/docs/reference/api/crm/objects/companies#get-%2Fcrm%2Fv3%2Fobjects%2Fcompanies)     | - | –                | replace               | Retrieves information about organizations. |
+| [contacts](https://developers.hubspot.com/docs/reference/api/crm/objects/contacts#get-%2Fcrm%2Fv3%2Fobjects%2Fcontacts)     | - | –                | replace               | Retrieves information about visitors, potential customers, and leads. |
+| [deals](https://developers.hubspot.com/docs/reference/api/crm/objects/deals#get-%2Fcrm%2Fv3%2Fobjects%2Fdeals)       | - | –                | replace               | Retrieves deal records and tracks deal progress.|
+| [tickets](https://developers.hubspot.com/docs/reference/api/crm/objects/tickets#basic)     | - | –                | replace               | Handles requests for help from customers or users. |
+| [products](https://developers.hubspot.com/docs/reference/api/crm/objects/products#get-%2Fcrm%2Fv3%2Fobjects%2Fproducts)     | - | –                | replace               | Retrieves pricing information of products. |
+| [quotes](https://developers.hubspot.com/docs/reference/api/crm/objects/quotes#get-%2Fcrm%2Fv3%2Fobjects%2Fquotes)        | - | –                | replace               | Retrieves price proposals that salespeople can create and send to their contacts. |
+| [schemas](https://developers.hubspot.com/docs/reference/api/crm/objects/schemas#get-%2Fcrm-object-schemas%2Fv3%2Fschemas)      | id | –                | merge               | Returns all object schemas that have been defined for your account.  |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> Hubspot does not support incremental loading, which means ingestr will do a full-refresh.
+
+

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/isoc-pulse.md

@@ -37,21 +37,24 @@ This will retrieve HTTPS adoption metrics for the specified date range and store
 
 Internet Society Pulse source allows ingesting the following metrics as separate tables:
 
-| Metric | Description | Country Support | Additional Options |
-|--------|-------------|----------------|-------------------|
-| `dnssec_adoption` | DNSSEC adoption metrics for specific domains | No | Domain name |
-| `dnssec_tld_adoption` | DNSSEC adoption metrics for top-level domains | Yes | Country code |
-| `dnssec_validation` | DNSSEC validation metrics | Yes | Country code |
-| `http` | HTTP protocol metrics | No | None |
-| `http3` | HTTP/3 protocol metrics | No | None |
-| `https` | HTTPS adoption metrics | Yes | `topsites`, Country code |
-| `ipv6` | IPv6 adoption metrics | Yes | `topsites`, Country code |
-| `net_loss` | Internet disconnection metrics | Yes | Shutdown type, Country code |
-| `resilience` | Internet resilience metrics | Yes | Country code |
-| `roa` | Route Origin Authorization metrics | Yes | IP version (`4`/`6`), Country code |
-| `rov` | Route Origin Validation metrics | No | None |
-| `tls` | TLS protocol metrics | No | None |
-| `tls13` | TLS 1.3 protocol metrics | No | None |
+| Metric | Description | Country Support | Additional Options | PK | Inc Key | Inc Strategy |
+|--------|-------------|-----------------|-------------------|-----|----------|--------------|
+| `dnssec_adoption` | DNSSEC adoption metrics for specific domains | No | Domain name | date | date | merge |
+| `dnssec_tld_adoption` | DNSSEC adoption metrics for top-level domains | Yes | Country code | date | date | merge |
+| `dnssec_validation` | DNSSEC validation metrics | Yes | Country code | date | date | merge |
+| `http` | HTTP protocol metrics | No | None | date | date | merge |
+| `http3` | HTTP/3 protocol metrics | No | None | date | date | merge |
+| `https` | HTTPS adoption metrics | Yes | topsites, Country code | date | date | merge |
+| `ipv6` | IPv6 adoption metrics | Yes | topsites, Country code | date | date | merge |
+| `net_loss` | Internet disconnection metrics | Yes | Shutdown type, Country code | date | date | merge |
+| `resilience` | Internet resilience metrics | Yes | Country code | date | date | merge |
+| `roa` | Route Origin Authorization metrics | Yes | IP version (4/6), Country code | date | date | merge |
+| `rov` | Route Origin Validation metrics | No | None | date | date | merge |
+| `tls` | TLS protocol metrics | No | None | date | date | merge |
+| `tls13` | TLS 1.3 protocol metrics | No | None | date | date | merge |
+
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
 
 ## Parameter Syntax
 

ingestr-0.13.87/docs/supported-sources/klaviyo.md

@@ -0,0 +1,57 @@
+# Klaviyo
+
+[Klaviyo](https://www.klaviyo.com/) is a marketing automation platform that helps businesses build and manage smarter digital relationships with their customers by connecting through personalized email and enhancing customer loyalty.
+
+ingestr supports Klaviyo as a source.
+
+## URI format
+
+The URI format for Klaviyo is as follows:
+
+```plaintext
+klaviyo://?api_key=<api-key>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Klaviyo API.
+
+The URI is used to connect to the Klaviyo API for extracting data.
+
+```bash
+ingestr ingest --source-table 'events' --source-uri 'klaviyo://?api_key=pk_test' --dest-uri duckdb:///klaviyo.duckdb --interval-start 2022-01-01 --dest-table 'dest.events' --extract-parallelism 20
+```
+
+This command fetches all the events that are created/updated since 2022-01-01 and writes them to `dest.events` table on DuckDB, using 20 parallel threads to improve performance and efficiently handle large data .
+
+## Tables
+
+Klaviyo source allows ingesting the following sources into separate tables:
+
+## Tables
+
+Klaviyo source allows ingesting the following sources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [events](https://developers.klaviyo.com/en/reference/get_events) | id | datetime     | merge               | Retrieves all events in an account where each event represents an action taken by a profile such as a password reset or a product order. |
+| [profiles](https://developers.klaviyo.com/en/reference/get_profiles) | id | updated    | merge             |  Retrieves all profiles in an account where each profile includes details like organization, job title, email and other attributes.|
+| [campaigns](https://developers.klaviyo.com/en/reference/get_campaigns) | id | updated_at     | merge               | Retrieves all campaigns in an account where each campaign is a targeted message sent to a specific audience. |
+| [metrics](https://developers.klaviyo.com/en/reference/get_metrics) | id | updated    | merge               | Retrieves all metrics in an account where each metric represents a category of events or actions a person can take. |
+| [tags](https://developers.klaviyo.com/en/reference/get_tags) | id | –                | replace               | Retrieves all tags in an account. |
+| [coupons](https://developers.klaviyo.com/en/reference/get_coupons) | id | –                | replace               | Retrieves all coupons in an account.|
+| [catalog-variants](https://developers.klaviyo.com/en/reference/get_catalog_variants): | id | updated    | merge               | Retrieves all variants in an account. |
+| [catalog-categories](https://developers.klaviyo.com/en/reference/get_catalog_categories) | id | updated   | merge               | Retrieves all catalog categories in an account.|
+| [catalog-items](https://developers.klaviyo.com/en/reference/get_catalog_items)| id | updated    | merge               | Retrieves all catalog items in an account. |
+| [flows](https://developers.klaviyo.com/en/reference/get_flows) | id | updated     | merge               | Retrieves all flows in an account where flow is a sequence of automated actions that is triggered when a person performs a specific action. |
+| [lists](https://developers.klaviyo.com/en/reference/get_lists) | id | updated     | merge               |  Retrieves all lists in an account.|
+| [images](https://developers.klaviyo.com/en/reference/get_images) | id | updated_at     | merge               | Retrieves all images in an account. |
+| [segments](https://developers.klaviyo.com/en/reference/get_segments) | id | updated    | merge               | Retrieves all segments in an account where segment is a dynamic list that contains profiles meeting a certain set of conditions. |
+| [forms](https://developers.klaviyo.com/en/reference/get_forms) | id | updated_at    | merge               | Retrieves all forms in an account.|
+| [templates](https://developers.klaviyo.com/en/reference/get_templates) | id | updated     | merge               |  Retrieves all templates in an account. |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+
+> [!WARNING]
+> Klaviyo does not support incremental loading for many endpoints in its APIs, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.

ingestr-0.13.87/docs/supported-sources/linear.md

@@ -0,0 +1,57 @@
+# Linear
+
+[Linear](https://linear.app/) is a purpose-built tool for planning and building products that provides issue tracking and project management for teams.
+
+ingestr supports Linear as a source through [Linear's public API](https://linear.app/developers/graphql).
+
+## URI format
+
+The URI format for Linear is:
+
+```plaintext
+linear://?api_key=<api_key>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Linear API.
+
+## Example usage
+
+Assuming your API key is `lin_api_123`, you can ingest teams into DuckDB using:
+
+```bash
+ingestr ingest
+--source-uri 'linear://?api_key=lin_api_123' \
+--source-table 'teams' \
+--dest-uri duckdb:///linear.duckdb \
+--dest-table 'dest.teams'
+```
+<img alt="linear" src="../media/linear.png"/>
+
+## Tables
+Linear source allows ingesting the following tables:
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `issues`        | id | updatedAt    | merge               | Fetches all issues from your Linear workspace. |
+| `users`         | id | updatedAt     | merge               | Fetches users from your workspace. |
+| `workflow_states` | id | updatedAt    | merge               | Fetches workflow states used in your Linear workspace. |
+| `cycles`        | id | updatedAt     | merge               |  Fetches cycle information and planning data. |
+| `attachments`   | id | updatedAt    | merge               | Fetches file attachments associated with issues. |
+| `comments`      | id | updatedAt    | merge               | Fetches comments on issues and other entities. |
+| `documents`     | id | updatedAt    | merge               | Fetches documents created in Linear. |
+| `external_users` | id | updatedAt     | merge               | Fetches information about external users. |
+| `initiative`    | id | updatedAt   | merge               | Fetches initiative data for high-level planning. |
+| `integrations`  | id | updatedAt    | merge               | Fetches integration configurations. |
+| `labels`        | id | updatedAt     | merge               | Fetches labels used for categorizing issues. |
+| `project_updates` | id | updatedAt     | merge               | Fetches updates posted to projects |
+| `team_memberships` | id | updatedAt    | merge               | Fetches team membership information. |
+| `initiative_to_project` | id | updatedAt     | merge               | Fetches relationships between initiatives and projects.  |
+| `project_milestone` | id | updatedAt     | merge               | Retrieves Linear project milestones and checkpoints. |
+| `project_status` | id | updatedAt     | merge               | Fetches project status information. |
+| `projects`      | id | updatedAt    | merge               | Fetches project-level data. |
+| `teams`         | id | updatedAt     | merge               | Fetches information about the teams configured in Linear. |
+| `organization`  | id | updatedAt     | merge               | Fetches organization-level information.|
+
+
+Use these as the `--source-table` parameter in the `ingestr ingest` command.

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/linkedin_ads.md

@@ -48,8 +48,11 @@ LinkedIn Ads requires an `access_token` and `account_ids` to retrieve reports fr
 
 To find the Ad Account IDs, the ad account owner can refer to the detailed instructions provided in this [guide](https://www.linkedin.com/help/linkedin/answer/a424270/find-linkedin-ads-account-details?lang=en).
 
-## Table: Custom Reports    
-Custom reports allow you to retrieve data based on specific dimensions and metrics.
+## Table:
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |  
+|  [custom](https://learn.microsoft.com/en-us/linkedin/marketing/integrations/ads-reporting/ads-reporting?view=li-lms-2024-11&tabs=http#analytics-finder)      | [dimension, date] or [dimension, start_date, end_date]| date (daily) or start_date (monthly)    | merge               | Custom reports allow you to retrieve data based on specific dimensions and metrics. |
+
 
 Custom Table Format:
 ```
@@ -68,7 +71,7 @@ custom:<dimensions>:<metrics>
 
 Retrieve data for campaign with `account_ids` id_123 and id_456:
 ```sh
-ingestr ingest \                         
+ingestr ingest \                      
     --source-uri "linkedinads://?access_token=token_123&account_ids=id_123,id_456" \
     --source-table 'custom:campaign,date:impressions,clicks' \
     --dest-uri 'duckdb:///linkedin.duckdb' \

{ingestr-0.13.86 → ingestr-0.13.87}/docs/supported-sources/mixpanel.md

@@ -40,7 +40,9 @@ ingestr ingest \
 
 Mixpanel source allows ingesting the following tables:
 
-- `events`: Raw event data returned from the export API.
-- `profiles`: User profiles from the Engage API.
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [events](https://developer.mixpanel.com/reference/raw-event-export)        | distinct_id | time     | merge               | Retrieves events data|
+| [profiles](https://developer.mixpanel.com/reference/engage-query)     | distinct_id | last_seen     | merge               | Retrieves Mixpanel user profiles and attributes. |
 
 Use these as `--source-table` values in the `ingestr ingest` command.