ingestr 0.13.85__tar.gz → 0.13.87__tar.gz

{ingestr-0.13.85 → ingestr-0.13.87}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.85
+Version: 0.13.87
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

{ingestr-0.13.85 → ingestr-0.13.87}/docs/.vitepress/config.mjs

@@ -125,6 +125,7 @@ export default defineConfig({
                 text: "Facebook Ads",
                 link: "/supported-sources/facebook-ads.md",
               },
+              { text: "Fluxx", link: "/supported-sources/fluxx.md" },
               { text: "Frankfurter", link: "/supported-sources/frankfurter.md" },
               { text: "Freshdesk", link: "/supported-sources/freshdesk.md" },
               { text: "Trustpilot", link: "/supported-sources/trustpilot.md" },
@@ -146,6 +147,7 @@ export default defineConfig({
               { text: "Pinterest", link: "/supported-sources/pinterest.md" },
               { text: "Pipedrive", link: "/supported-sources/pipedrive.md" },
               { text: "QuickBooks", link: "/supported-sources/quickbooks.md" },
+              { text: "RevenueCat", link: "/supported-sources/revenuecat.md" },
               { text: "S3", link: "/supported-sources/s3.md" },
               { text: "Salesforce", link: "/supported-sources/salesforce.md" },
               { text: "SFTP", link: "/supported-sources/sftp.md"},

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/appstore.md

@@ -69,7 +69,7 @@ We will run `ingestr` to save this data to a [duckdb](https://duckdb.org/) datab
 
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -80,7 +80,7 @@ ingestr ingest \
 We will extend the prior example with another app with ID `67890`. To achieve this, simply add another `app_id` query parameter to the URI.
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&app_id=67890&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&app_id=67890&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -94,7 +94,7 @@ ingestr ingest \
 To begin, we will first load all data till `2025-01-01` by specifying the `--interval-end` flag. We'll assume the same credentials from our [first example](#example-loading-app-downloads-analytics)
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -105,7 +105,7 @@ ingestr ingest \
 
 ```sh
 ingestr ingest \
-    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0 \
+    --source-uri "appstore://app_id=12345&key_path=api.key&key_id=key_0&issuer_id=issue_0" \
     --source-table "app-downloads-detailed" \
     --dest-uri "duckdb:///analytics.db"  \
     --dest-table "public.app_downloads" \
@@ -115,6 +115,14 @@ ingestr ingest \
 Notice how we didn't specify a date parameter? `ingestr` will automatically use the metadata from last load and continue loading data from that point on.
 
 ## Tables
+| Table | PK | Inc Key | Inc Strategy | Details |
+|-------|----|---------|--------------|---------|
+| `app-downloads-detailed` | [App Apple Identifier,App Name, App Version,Campaign,Date,Device,Download Type,Page Title,Page Type,Platform Version,Pre-Order,Source Info,Source Type,Territory] | processing_date | merge | App download analytics including first-time downloads, redownloads, updates, and more. |
+| `app-store-discovery-and-engagement-detailed` |  primary_key = [App Apple Identifier,App Name,Campaign,Date,Device,Engagement Type,Event,Page Title,Page Type,Platform Version,Source Info,Source Type,Territory] | processing_date | merge | App Store discovery and engagement metrics including data about user engagement with your app’s icons, product pages, in-app event pages, and other install sheets. |
+|`app-sessions-detailed` |  primary_key=[Date,App Name,App Apple Identifier,App Version,Device,Platform Version,Source Type,Source Info,Campaign,Page Type,Page Title,App Download Date,Territory] | processing_date | merge | App Session provides insights on how often people open your app, and how long they spend in your app. |
+| app-store-installation-and-deletion-detailed | [App Apple Identifier,App Download Date,App Name,App Version,Campaign,Counts,Date,Device,Download Type,Event,Page Title,Page Type,Platform Version,Source Info,Source Type,Territory,Unique Devices] | processing_date | merge | App installation and deletion metrics including device to estimate the number of times people install and delete your App Store apps. |
+| `app-store-purchases-detailed`|  [App Apple Identifier,App Download Date,App Name,Campaign,Content Apple Identifier,Content Name,Date,Device,Page Title,Page Type,Payment Method,Platform Version,Pre-Order,Purchase Type,Source Info,Source Type,Territory] | processing_date | merge | App purchase analytics including revenue, payment methods, and content details. |
+| `app-crashes-expanded`| [App Name,App Version,Build,Date,Device,Platform,Release Type,Territory] |processing_date | merge | App crash analytics including crash counts, device information, and version details. |
 
 ### `app-downloads-detailed`
 The App Downloads Report includes download data generated on the App Store. You can use this report to understand your total number of downloads, including first-time downloads, redownloads, updates, and more.

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/chess.md

@@ -29,9 +29,10 @@ The result of this command will be a table in the `chess.duckdb` database.
 ## Tables
 
 Chess source allows ingesting the following sources into separate tables:
-
-- `profiles`: Retrieves player profiles based on a list of player usernames.
-- `games`: Retrieves players' games for specified players.
-- `archives`: Retrieves the URLs to game archives for specified players.
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `profiles`        | - | –                | replace               | Retrieves player profiles based on a list of player usernames.                                |
+| `games` | - | –                | replace               | Retrieves players games for specified players.   
+| `archives`        | - | –                | replace               | Retrieves the URLs to game archives for specified players.                                      |                                                 
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.

ingestr-0.13.87/docs/supported-sources/clickup.md

@@ -0,0 +1,45 @@
+# ClickUp
+[ClickUp](https://clickup.com/) is a cloud-based productivity platform designed to help teams and individuals manage tasks, projects, and workflows in one place.
+
+ingestr supports ClickUp as a source.
+
+## URI format
+
+```
+clickup://?api_token=<token>
+```
+
+URI parameters:
+- `api_token` is a personal token used to authenticate with the ClickUp API.
+
+ClickUp requires a `api_token` to connect to the ClickUP API. For more information, read [here](https://developer.clickup.com/docs/authentication#generate-your-personal-api-token) to get credentail. 
+
+Once you've completed the guide, you should have `api_token`. Let's say your API Token  is `token_123`, here's a sample command that will copy the data from Clickup into a DuckDB database:
+
+To ingest tasks from ClickUp into a DuckDB database:
+
+```sh
+ingestr ingest \
+  --source-uri "clickup://?api_token=token_123" \
+  --source-table "tasks" \
+  --dest-uri duckdb:///clickup.duckdb \
+  --dest-table "public.tasks"
+```
+<img alt="clickup_img" src="../media/clickup_ingestion.png"/>
+
+## Tables
+
+ClickUp source allows ingesting the following resources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [user](https://developer.clickup.com/reference/getauthorizeduser)        | id | -             | merge               | The authorised user profile                       |
+| [teams](https://developer.clickup.com/reference/getauthorizedteams) | id | –                | merge               | Workspaces available to the authenticated user. In ClickUp, a "team" in the API is equivalent to a workspace, which is the highest-level container for all spaces, folders, lists, and tasks |
+| [spaces](https://developer.clickup.com/reference/getspaces)   | id | -     | merge  | Spaces available within a workspace. Spaces are major sections used to organize work, such as departments, projects, or business units. Each space can contain folders, lists, and tasks |
+| [lists](https://developer.clickup.com/reference/getlists)   | id | –  | merge               | Lists contained in each space. Lists are collections of tasks, often representing projects, sprints, or workflow stages |
+| [tasks](https://developer.clickup.com/reference/gettasks)    | id | date_updated             | merge            | Tasks belonging to each list |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> ClickUp does not support incremental loading for many endpoints, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/facebook-ads.md

@@ -2,7 +2,7 @@
 
 Facebook Ads is the advertising platform that helps users to create targeted ads on Facebook, Instagram and Messenger.
 
-ingestr supports Facebook Ads as a source.
+ingestr supports Facebook Ads as a source using [Facebook Marketing API](https://developers.facebook.com/docs/marketing-api/).
 
 ## URI format
 
@@ -41,113 +41,14 @@ The result of this command will be a table in the `facebook.duckdb` database.
 
 Facebook Ads source allows ingesting the following sources into separate tables:
 
-- `campaigns`: Retrieves campaign data with fields:
-  - `id`
-  - `updated_time`
-  - `created_time`
-  - `name`
-  - `status`
-  - `effective_status`
-  - `objective`
-  - `start_time`
-  - `stop_time`
-  - `daily_budget`
-  - `lifetime_budget`
-
-- `ad_sets`: Retrieves ad set data with fields:
-  - `id`
-  - `updated_time`
-  - `created_time`
-  - `name`
-  - `status`
-  - `effective_status`
-  - `campaign_id`
-  - `start_time`
-  - `end_time`
-  - `daily_budget`
-  - `lifetime_budget`
-  - `optimization_goal`
-  - `promoted_object`
-  - `billing_event`
-  - `bid_amount`
-  - `bid_strategy`
-  - `targeting`
-
-- `leads`: Retrieves lead data with fields:
-  - `id`
-  - `created_time`
-  - `ad_id`
-  - `ad_name`
-  - `adset_id`
-  - `adset_name`
-  - `campaign_id`
-  - `campaign_name`
-  - `form_id`
-  - `field_data`
-
-- `ads_creatives`: Retrieves ad creative data with fields:
-  - `id`
-  - `name`
-  - `status`
-  - `thumbnail_url`
-  - `object_story_spec`
-  - `effective_object_story_id`
-  - `call_to_action_type`
-  - `object_type`
-  - `template_url`
-  - `url_tags`
-  - `instagram_actor_id`
-  - `product_set_id`
-
-- `ads`: Retrieves ad data with fields:
-  - `id`
-  - `updated_time`
-  - `created_time`
-  - `name`
-  - `status`
-  - `effective_status`
-  - `adset_id`
-  - `campaign_id`
-  - `creative`
-  - `targeting`
-  - `tracking_specs`
-  - `conversion_specs`
-
-- `facebook_insights`: Retrieves insights data with fields:
-  - `campaign_id`
-  - `adset_id`
-  - `ad_id`
-  - `date_start`
-  - `date_stop`
-  - `reach`
-  - `impressions`
-  - `frequency`
-  - `clicks`
-  - `unique_clicks`
-  - `ctr`
-  - `unique_ctr`
-  - `cpc`
-  - `cpm`
-  - `cpp`
-  - `spend`
-  - `actions`
-  - `action_values`
-  - `cost_per_action_type`
-  - `website_ctr`
-  - `account_currency`
-  - `ad_click_actions`
-  - `ad_name`
-  - `adset_name`
-  - `campaign_name`
-  - `country`
-  - `dma`
-  - `full_view_impressions`
-  - `full_view_reach`
-  - `inline_link_click_ctr`
-  - `outbound_clicks`
-  - `social_spend`
-  - `conversions`
-  - `video_thruplay_watched_actions`
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `campaigns`       | id | –  |        replace     | Retrieves campaign data with `fields`: id, updated_time, created_time, name, status, effective_status, objective, start_time, stop_time, daily_budget, lifetime_budget              |
+| `ad_sets` | id | –                | replace            | Retrieves ad set data with `fields`: id, updated_time, created_time, name, status, effective_status, campaign_id, start_time, end_time, daily_budget, lifetime_budget, optimization_goal, promoted_object, billing_event, bid_amount, bid_strategy, targeting                       |
+| `ads`   | id | -     | replace  | Retrieves ad data with `fields`: id, updated_time, created_time, name, status, effective_status, adset_id, campaign_id, creative, targeting, tracking_specs, conversion_specs                          |
+| `ad_creatives`   | id | -     | replace  | Retrieves ad creative data with `fields`: id, name, status, thumbnail_url, object_story_spec, effective_object_story_id, call_to_action_type, object_type, template_url, url_tags, instagram_actor_id, product_set_id |
+| `leads`   | id | -     | replace  | Retrieves lead data with fields: id, created_time, ad_id, ad_name, adset_id, adset_name, campaign_id, campaign_name, form_id, field_data |
+| `facebook_insights`   | date_start | date_start     | merge  | Retrieves insights data with `fields`: campaign_id, adset_id, ad_id, date_start, date_stop, reach, impressions, frequency, clicks, unique_clicks, ctr, unique_ctr, cpc, cpm, cpp, spend, actions, action_values, cost_per_action_type, website_ctr, account_currency, ad_click_actions, ad_name, adset_name, campaign_name, country, dma, full_view_impressions, full_view_reach, inline_link_click_ctr, outbound_clicks, social_spend, conversions, video_thruplay_watched_actions. |
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

ingestr-0.13.87/docs/supported-sources/fluxx.md

@@ -0,0 +1,116 @@
+# Fluxx
+
+[Fluxx](https://www.fluxx.io/) is a cloud-based grants management platform designed to streamline and automate the entire grantmaking process for foundations, corporations, governments, and other funding organizations.
+
+ingestr supports Fluxx as a source.
+
+## URI format
+
+The URI format for Fluxx is:
+
+```plaintext
+fluxx://<instance>?client_id=<client_id>&client_secret=<client_secret>
+```
+
+URI parameters:
+
+- `instance`: Your Fluxx instance subdomain (e.g., `mycompany.preprod` for `https://mycompany.preprod.fluxxlabs.com`)
+- `client_id`: OAuth 2.0 client ID for authentication
+- `client_secret`: OAuth 2.0 client secret for authentication
+
+## Example usage
+
+### Basic usage - all fields
+
+Assuming your instance is `myorg.preprod`, you can ingest grant requests into DuckDB using:
+
+```bash
+ingestr ingest \
+--source-uri 'fluxx://myorg.preprod?client_id=your_client_id&client_secret=your_client_secret' \
+--source-table 'grant_request' \
+--dest-uri duckdb:///fluxx.duckdb \
+--dest-table 'raw.grant_request'
+```
+
+### Custom field selection
+
+You can select specific fields to ingest using the colon syntax:
+
+```bash
+ingestr ingest \
+--source-uri 'fluxx://myorg.preprod?client_id=your_client_id&client_secret=your_client_secret' \
+--source-table 'grant_request:id,amount_requested,amount_recommended,granted' \
+--dest-uri duckdb:///fluxx.duckdb \
+--dest-table 'raw.grant_request'
+```
+
+## Tables
+
+Fluxx source currently supports the following 50 tables:
+
+### Core Resources
+- `claim_expense_datum`: Individual data entries within claim expense forms with budget category details
+- `claim_expense_row`: Specific line items or rows within claim expense forms
+- `claim_expense`: Claim expense forms and templates for financial tracking
+- `claim`: Grant claims and payment requests
+- `concept_initiative`: Concept initiatives linking programs, initiatives, and sub-programs/sub-initiatives
+- `dashboard_theme`: Dashboard theme configurations for UI customization
+- `etl_claim_expense_datum`: ETL data for claim expense items with comprehensive budget tracking details
+- `etl_grantee_budget_tracker_actual`: ETL data for actual grantee budget tracker amounts and expenses
+- `etl_grantee_budget_tracker_period_datum`: ETL data for grantee budget tracker period information with detailed financial tracking
+- `etl_relationship`: ETL data for entity relationships tracking connections between users, organizations, requests, and other entities
+- `etl_request_budget`: ETL budget data for request funding sources with comprehensive financial details
+- `etl_request_transaction_budget`: ETL budget data for request transaction funding sources including payment tracking
+- `exempt_organization`: Tax-exempt organization data including EIN, classification, and financial information
+- `geo_city`: City geographic data with coordinates and postal codes
+- `geo_county`: County geographic data with FIPS codes
+- `geo_place`: Geographic places with ancestry and location data
+- `geo_region`: Geographic regions
+- `geo_state`: State geographic data with abbreviations and FIPS codes
+- `grant_request`: Grant applications and requests (300+ fields)
+- `grantee_budget_category`: Budget category definitions used by grantees for expense tracking
+- `grantee_budget_tracker_period_datum_actual`: Actual expenses and amounts recorded for budget tracking periods
+- `grantee_budget_tracker_period_datum`: Budget data entries for specific tracking periods
+- `grantee_budget_tracker_period`: Time periods for budget tracking with start and end dates
+- `grantee_budget_tracker_row`: Individual budget line items and categories within budget trackers
+- `grantee_budget_tracker`: Budget tracking documents for grantee financial management
+- `integration_log`: Integration and system logs for tracking data processing and errors
+- `mac_model_type_dyn_financial_audit`: Dynamic financial audit models with audit tracking, compliance status, and financial variance analysis
+- `mac_model_type_dyn_mel`: Dynamic Monitoring, Evaluation & Learning (MEL) models with performance indicators, baseline tracking, and evaluation metrics
+- `mac_model_type_dyn_tool`: Dynamic tool management models for tracking deployment status, usage metrics, and tool effectiveness
+- `machine_category`: Machine category definitions for workflow state management
+- `model_attribute_value`: Model attribute values with hierarchical data and dependencies
+- `model_document_sub_type`: Document sub-type definitions and categories
+- `model_document_type`: Document type configurations including DocuSign integration and permissions
+- `model_document`: Document metadata including file information, storage details, and document relationships
+- `model_theme`: Model themes for categorization and program hierarchy organization
+- `organization`: Organizations (grantees, fiscal sponsors, etc.)
+- `population_estimate_year`: Yearly population estimates with income and demographic breakdowns
+- `population_estimate`: Population estimates by geographic area with demographic data
+- `program`: Funding programs and initiatives
+- `request_report`: Reports submitted for grants
+- `request_transaction_funding_source`: Funding source details for specific request transactions
+- `request_transaction`: Financial transactions and payments
+- `request_user`: Relationships between requests and users with roles and descriptions
+- `salesforce_authentication`: Salesforce authentication configurations with OAuth tokens, connection management, and API usage tracking
+- `sub_initiative`: Sub-initiatives for detailed planning
+- `sub_program`: Sub-programs under main programs
+- `ui_version`: User interface version information and system configuration
+- `user_organization`: Relationships between users and organizations with roles, departments, and contact details
+- `user`: User accounts and profiles
+
+### Field Selection
+
+Each resource contains numerous fields. You can:
+1. **Ingest all fields**: Use the resource name directly (e.g., `grant_request`)
+2. **Select specific fields**: Use colon syntax (e.g., `grant_request:id,name,amount_requested`)
+
+The field selection feature is particularly useful for large resources like `grant_request` which has over 300 fields.
+
+## Authentication
+
+Fluxx uses OAuth 2.0 with client credentials flow. To obtain credentials:
+
+1. Contact your Fluxx administrator to create an API client
+2. You'll receive a `client_id` and `client_secret`
+3. Note your Fluxx instance subdomain (the part before `.fluxxlabs.com`)

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/frankfurter.md

@@ -34,9 +34,13 @@ The result of this command will be a list of currency exchange rates from 20.03.
 
 ## Tables
 
-- `latest`: Fetches the latest exchange rates.
-- `exchange_rates`: Fetches historical exchange rates for a specified date range.
-- `currencies`: Fetches a list of the available currencies and their ISO 4217 currency code (e.g. `USD`, `EUR`, `GBP`).
+Frankfurter source allows ingesting the following sources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [currencies](https://www.frankfurter.dev/docs/#currencies)  | - | –                | replace               | Retrieves list of available currencies with ISO 4217 codes and names. |
+| [latest](https://www.frankfurter.dev/docs/#latest)    | ["date", "currency_code", "base_currency"] | –                | merge               | Fetches latest exchange rates for all currencies. |
+| [exchange_rates](https://www.frankfurter.dev/docs/#historical) | ["date", "currency_code", "base_currency"] | date    | merge               | Retrieves historical exchange rates for specified date range.|
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/freshdesk.md

@@ -33,15 +33,16 @@ This command copies data from the Freshdesk source to DuckDB.
 <img alt="freshdesk_img" src="../media/freshdesk_ingestion.png" />
 
 ## Available Tables
+
 The Freshdesk source allows you to ingest the following tables:
 
-| S.No. | Name      | Description                                                                               |
-| ----- | --------- | ----------------------------------------------------------------------------------------- |
-| 1.    | agents    |  Users responsible for managing and resolving customer inquiries and support tickets.     |
-| 2.    | companies |  Customer organizations or groups that agents support.                                    |
-| 3.    | contacts  |  Individuals or customers who reach out for support.                                      |
-| 4.    | groups    |  Agents organized based on specific criteria.                                             |
-| 5.    | roles     |  Predefined sets of permissions that determine what actions an agent can perform.         |
-| 6.    | tickets   |  Customer inquiries or issues submitted via various channels like email, chat, phone, etc. |
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [agents](https://developers.freshdesk.com/api/#agents)    | id | updated_at    | merge               | Retrieves users responsible for managing and resolving customer inquiries and support tickets |
+| [companies](https://developers.freshdesk.com/api/#companies)     | id | updated_at     | merge               | Retrieves customer organizations or groups that agents support. |
+| [contacts](https://developers.freshdesk.com/api/#contacts)     | id | updated_at     | merge               | Retrieves individuals or customers who reach out for support. |
+| [groups](https://developers.freshdesk.com/api/#groups)       | id | updated_at     | merge               | Retrieves agents organized based on specific criteria.  |
+| [roles](https://developers.freshdesk.com/api/#roles)       | id | updated_at     | merge               | Retrieves predefined sets of permissions that determine what actions an agent can perform.|
+| [tickets](https://developers.freshdesk.com/api/#tickets)      | id | updated_at     | merge               | Retrieves customer inquiries or issues submitted via various channels like email, chat, phone, etc.
 
 Use these as the `--source-table` parameter in the `ingestr ingest` command.

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/google_analytics.md

@@ -23,6 +23,13 @@ URI parameters:
 Google Analytics requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/google_analytics#grab-google-service-account-credentials). Once you complete the guide, you should have an `.json` file  and `project_id`. 
 
 ## Available Tables:
+
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [realtime](https://developers.google.com/analytics/devguides/reporting/data/v1/realtime-basics)     | ingested_at |  -    | merge               | Retrieves real-time analytics data based on specified dimensions and metrics. Format: `realtime:<dimensions>:<metrics>:<minutes_ranges>`. Supports incremental loading by ingestion timestamp. |
+| `custom` | datetime_dimension | datetime_dimension    |     merge           | Retrieves custom reports based on specified dimensions and metrics. Format: `custom:<dimensions>:<metrics>` |
+
 ### Custom reports 
 - `Custom reports`: allow you to retrieve data based on specific `dimensions` and  `metrics`.
 
@@ -79,4 +86,4 @@ ingestr ingest \
 ```
 This command will retrieve report and save it to the `dest.realtime` table in the DuckDB database.
 
-<img alt="google_analytics_realtime_report_img" src="../media/google_analytics_realtime_report.png"/>
+<img alt="google_analytics_realtime_report_img" src="../media/google_analytics_realtime_report.png"/>

ingestr-0.13.87/docs/supported-sources/gorgias.md

@@ -0,0 +1,42 @@
+# Gorgias
+[Gorgias](https://www.gorgias.com/) is a helpdesk for e-commerce merchants, providing customer service via email, social media, SMS, and live chat.
+
+ingestr supports Gorgias as a source.
+
+## URI format
+The URI format for Gorgias is as follows:
+
+```plaintext
+gorgias://<domain>?api_key=<api-key>&email=<email>
+```
+
+URI parameters:
+- `domain`: the domain of the Gorgias account without the full `gorgias.com`, e.g. `mycompany`
+- `api_key`: the integration token used for authentication with the Gorgias API
+- `email`: the email address of the user to connect to the Gorgias API
+
+The URI is used to connect to the Gorgias API for extracting data.
+
+## Examples
+```bash
+# get all the tickets that are created/updated since 2024-06-19 and write them to `gorgias.ticket_messages` table on BigQuery
+ingestr ingest --source-table 'tickets' --source-uri $GORGIAS_URI --dest-uri $BIGQUERY_URI --interval-start 2024-06-19  --dest-table 'gorgias.ticket_messages' --loader-file-format jsonl
+
+# get all the customers and write them to `gorgias.customers` table on DuckDB
+ingestr ingest --source-table 'customers' --source-uri $GORGIAS_URI --dest-uri duckdb:///gorgias.duckdb --interval-start 2024-01-01  --dest-table 'dest.customers'
+```
+
+
+
+Gorgias source allows ingesting the following sources into separate tables:
+
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [customers](https://developers.gorgias.com/reference/list-customers)     | id | updated_datetime     | merge               | Customers are the users who have interacted with the support team. Each customer has a unique ID and contains information such as the name and email.  Retrieves customers lists|
+| [tickets](https://developers.gorgias.com/reference/list-tickets)  | id | updated_datetime    | merge               | Tickets are the main entity in Gorgias, representing customer inquiries. Each ticket has a unique ID and contains information such as the customer, status, and messages. Retrieves tickets lists |
+| [ticket_messages](https://developers.gorgias.com/reference/list-messages) | id | updated_datetime    | merge               | Ticket messages are the messages exchanged between the customer and the support agent in a ticket. Each message has a unique ID and contains information such as the sender, content, and timestamp. Retrieves messages lists | 
+| [satisfaction_surveys](https://developers.gorgias.com/reference/list-satisfaction-surveys) | id | updated_datetime     | merge               | Satisfaction surveys are sent to customers after a ticket is resolved to gather feedback on their experience. Each survey has a unique ID and contains information such as the rating and comments. Retrieves surveys lists.|
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+

ingestr-0.13.87/docs/supported-sources/hubspot.md

@@ -0,0 +1,52 @@
+# HubSpot
+
+[HubSpot](https://www.hubspot.com/) is a customer relationship management software that helps businesses attract visitors, connect with customers, and close deals.
+
+ingestr supports HubSpot as a source.
+
+## URI format
+
+The URI format for HubSpot is as follows:
+
+```plaintext
+hubspot://?api_key=<api-key-here>
+```
+
+URI parameters:
+
+- `api_key`: The API key is used for authentication with the HubSpot API.
+
+The URI is used to connect to the HubSpot API for extracting data.
+
+## Setting up a HubSpot Integration
+
+HubSpot requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot#setup-guide).
+
+Once you complete the guide, you should have an API key. Let's say your API key is `pat_test_12345`, here's a sample command that will copy the data from HubSpot into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'hubspot://?api_key=pat_test_12345' --source-table 'companies' --dest-uri duckdb:///hubspot.duckdb --dest-table 'companies.data'
+```
+
+The result of this command will be a table in the `hubspot.duckdb` database.
+
+## Tables
+
+HubSpot source allows ingesting the following sources into separate tables:
+ 
+Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [companies](https://developers.hubspot.com/docs/reference/api/crm/objects/companies#get-%2Fcrm%2Fv3%2Fobjects%2Fcompanies)     | - | –                | replace               | Retrieves information about organizations. |
+| [contacts](https://developers.hubspot.com/docs/reference/api/crm/objects/contacts#get-%2Fcrm%2Fv3%2Fobjects%2Fcontacts)     | - | –                | replace               | Retrieves information about visitors, potential customers, and leads. |
+| [deals](https://developers.hubspot.com/docs/reference/api/crm/objects/deals#get-%2Fcrm%2Fv3%2Fobjects%2Fdeals)       | - | –                | replace               | Retrieves deal records and tracks deal progress.|
+| [tickets](https://developers.hubspot.com/docs/reference/api/crm/objects/tickets#basic)     | - | –                | replace               | Handles requests for help from customers or users. |
+| [products](https://developers.hubspot.com/docs/reference/api/crm/objects/products#get-%2Fcrm%2Fv3%2Fobjects%2Fproducts)     | - | –                | replace               | Retrieves pricing information of products. |
+| [quotes](https://developers.hubspot.com/docs/reference/api/crm/objects/quotes#get-%2Fcrm%2Fv3%2Fobjects%2Fquotes)        | - | –                | replace               | Retrieves price proposals that salespeople can create and send to their contacts. |
+| [schemas](https://developers.hubspot.com/docs/reference/api/crm/objects/schemas#get-%2Fcrm-object-schemas%2Fv3%2Fschemas)      | id | –                | merge               | Returns all object schemas that have been defined for your account.  |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> Hubspot does not support incremental loading, which means ingestr will do a full-refresh.
+
+

{ingestr-0.13.85 → ingestr-0.13.87}/docs/supported-sources/isoc-pulse.md

@@ -37,21 +37,24 @@ This will retrieve HTTPS adoption metrics for the specified date range and store
 
 Internet Society Pulse source allows ingesting the following metrics as separate tables:
 
-| Metric | Description | Country Support | Additional Options |
-|--------|-------------|----------------|-------------------|
-| `dnssec_adoption` | DNSSEC adoption metrics for specific domains | No | Domain name |
-| `dnssec_tld_adoption` | DNSSEC adoption metrics for top-level domains | Yes | Country code |
-| `dnssec_validation` | DNSSEC validation metrics | Yes | Country code |
-| `http` | HTTP protocol metrics | No | None |
-| `http3` | HTTP/3 protocol metrics | No | None |
-| `https` | HTTPS adoption metrics | Yes | `topsites`, Country code |
-| `ipv6` | IPv6 adoption metrics | Yes | `topsites`, Country code |
-| `net_loss` | Internet disconnection metrics | Yes | Shutdown type, Country code |
-| `resilience` | Internet resilience metrics | Yes | Country code |
-| `roa` | Route Origin Authorization metrics | Yes | IP version (`4`/`6`), Country code |
-| `rov` | Route Origin Validation metrics | No | None |
-| `tls` | TLS protocol metrics | No | None |
-| `tls13` | TLS 1.3 protocol metrics | No | None |
+| Metric | Description | Country Support | Additional Options | PK | Inc Key | Inc Strategy |
+|--------|-------------|-----------------|-------------------|-----|----------|--------------|
+| `dnssec_adoption` | DNSSEC adoption metrics for specific domains | No | Domain name | date | date | merge |
+| `dnssec_tld_adoption` | DNSSEC adoption metrics for top-level domains | Yes | Country code | date | date | merge |
+| `dnssec_validation` | DNSSEC validation metrics | Yes | Country code | date | date | merge |
+| `http` | HTTP protocol metrics | No | None | date | date | merge |
+| `http3` | HTTP/3 protocol metrics | No | None | date | date | merge |
+| `https` | HTTPS adoption metrics | Yes | topsites, Country code | date | date | merge |
+| `ipv6` | IPv6 adoption metrics | Yes | topsites, Country code | date | date | merge |
+| `net_loss` | Internet disconnection metrics | Yes | Shutdown type, Country code | date | date | merge |
+| `resilience` | Internet resilience metrics | Yes | Country code | date | date | merge |
+| `roa` | Route Origin Authorization metrics | Yes | IP version (4/6), Country code | date | date | merge |
+| `rov` | Route Origin Validation metrics | No | None | date | date | merge |
+| `tls` | TLS protocol metrics | No | None | date | date | merge |
+| `tls13` | TLS 1.3 protocol metrics | No | None | date | date | merge |
+
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
 
 ## Parameter Syntax