ingestr 0.13.88__tar.gz → 0.13.89__tar.gz

{ingestr-0.13.88 → ingestr-0.13.89}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.88
+Version: 0.13.89
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -42,7 +42,7 @@ Requires-Dist: confluent-kafka==2.8.0
 Requires-Dist: crate==2.0.0
 Requires-Dist: cryptography==44.0.2
 Requires-Dist: curlify==2.2.1
-Requires-Dist: databricks-sql-connector==2.9.3
+Requires-Dist: databricks-sql-connector==4.0.5
 Requires-Dist: databricks-sqlalchemy==1.0.2
 Requires-Dist: dataclasses-json==0.6.7
 Requires-Dist: decorator==5.2.1

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/adjust.md

@@ -32,12 +32,7 @@ Adjust data may change going back, which means you'll need to change your start
 ## Tables
 Adjust source allows ingesting data from various sources:
 
-- `campaigns`: Retrieves data for a campaign, showing the app's revenue and network costs over multiple days.
-- `creatives`: Retrieves data for a creative assets, detailing the app's revenue and network costs across multiple days.
-- `events`: Retrieves data for [events](https://dev.adjust.com/en/api/rs-api/events/) and event slugs.
-- `custom`: Retrieves custom data based on the dimensions and metrics specified.
-
-### Custom reports: `custom:<dimensions>:<metrics>[:<filters>]`
+#### Custom reports: `custom:<dimensions>:<metrics>[:<filters>]`
 
 The custom table allows you to retrieve data based on specific dimensions and metrics, and apply filters to the data.
 
@@ -55,7 +50,14 @@ Parameters:
 > [!WARNING]
 > Custom tables require a time-based dimension for efficient operation, such as `hour`, `day`, `week`, `month`, or `year`.
 
-## Examples
+| Table           | PK/Merge Key | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Events](https://dev.adjust.com/en/api/rs-api/events)        | id | –  |        replace     | Retrieves data for [events](https://dev.adjust.com/en/api/rs-api/events/) and event slugs.              |                                        |
+| [campaigns](https://dev.adjust.com/en/api/rs-api/reports) | day | –                | merge            | Retrieves data for a campaign, showing the app's revenue and network costs over multiple days. `Columns:` campaign, day, app, store_type, channel, country, network_cost, all_revenue_total_d0, ad_revenue_total_d0, revenue_total_d0, all_revenue_total_d1, ad_revenue_total_d1, revenue_total_d1, all_revenue_total_d3, ad_revenue_total_d3, revenue_total_d3, all_revenue_total_d7, ad_revenue_total_d7, revenue_total_d7, all_revenue_total_d14, ad_revenue_total_d14, revenue_total_d14, all_revenue_total_d21 |
+| [creatives](https://dev.adjust.com/en/api/rs-api/reports)   | day | -     | merge  | Retrieves data for a creative assets, detailing the app's revenue and network costs across multiple days. `Columns:` campaign, day, app, store_type, channel, country, adgroup, creative, network_cost, all_revenue_total_d0, ad_revenue_total_d0, revenue_total_d0, all_revenue_total_d1, ad_revenue_total_d1, revenue_total_d1, all_revenue_total_d3, ad_revenue_total_d3, revenue_total_d3, all_revenue_total_d7, ad_revenue_total_d7, revenue_total_d7, all_revenue_total_d14, ad_revenue_total_d14, revenue_total_d14, all_revenue_total_d21 |
+| `custom`   | `configurable` | -     | merge  | Retrieves custom data based on the dimensions and metrics specified.
+
+ ## Examples
 
 Copy campaigns data from Adjust into a DuckDB database:
 ```sh
@@ -82,4 +84,4 @@ ingestr ingest \
     --source-table "custom:hour,app,store_id,channel,os_name,country_code,campaign_network,campaign_id_network,adgroup_network, adgroup_id_network,creative_network,creative_id_network:impressions,clicks,cost,network_cost,installs,ad_revenue,all_revenue" \
     --dest-uri duckdb:///adjust.db \
     --dest-table "mat.example"
-```
+```

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/applovin.md

@@ -101,12 +101,12 @@ $ duckdb report.db 'select day from public.publisher_report group by 1'
 
 ## Tables
 
-| Name | Description |
-| --- | --- |
-| `publisher-report` | Provides daily metrics from the `report` end point using the report_type `publisher` |
-| `advertiser-report` | Provides daily metrics from the `report` end point using the report_type `advertiser`|
-| `advertiser-probabilistic-report` | Provides daily metrics from the `probabilisticReport` end point using the report_type `advertiser` |
-| `advertiser-ska-report` | Provides daily metrics from the `skaReport` end point using the report_type `advertiser` |
+| Name | Merge Key | Inc Key | Inc Strategy |  Details |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `publisher-report` | day | day | merge| Provides daily metrics from the `report` end point using the report_type `publisher` |
+| `advertiser-report` | day | day | merge| Provides daily metrics from the `report` end point using the report_type `advertiser`|
+| `advertiser-probabilistic-report` | day | day | merge| Provides daily metrics from the `probabilisticReport` end point using the report_type `advertiser` |
+| `advertiser-ska-report` | day | day | merge| Provides daily metrics from the `skaReport` end point using the report_type `advertiser` |
 
 ## Custom Reports
 

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/applovin_max.md

@@ -45,4 +45,8 @@ By default, `ingestr` retrieves data for the last 30 days. For a custom date ran
 <img alt="applovin_max_img" src="../media/applovin_max.png"/>
 
 ## Table
-[user_ad_revenue](https://developers.applovin.com/en/max/reporting-apis/user-level-ad-revenue-api/): Provides daily metrics from the user level ad revenue API.User-level revenue data is available eight hours after UTC day end. So, for example, data for UTC 2025-01-01 is available on UTC 2025-01-02 after 08:00.
+
+Applovin Max source allows ingesting the following sources into separate tables:
+| Table           | PK | Inc Key | Inc Strategy | Details                                                                                                                                        |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [user_ad_revenue](https://developers.applovin.com/en/max/reporting-apis/user-level-ad-revenue-api/)       | `partition_date` (extracts date-only portion from the timestamp-based date column) |        partition_date      | merge              |  Provides daily metrics from the user level ad revenue API. User-level revenue data is available eight hours after UTC day end. So, for example, data for UTC 2025-01-01 is available on UTC 2025-01-02 after 08:00.

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/appsflyer.md

@@ -34,9 +34,13 @@ The result of this command will be a table in the `appsflyer.duckdb` database.
 
 ingestr integrates with the [Master Report API](https://dev.appsflyer.com/hc/reference/master_api_get) of AppsFlyer, which allows you to retrieve data for the following tables:
 
-- `campaigns`: Retrieves data for campaigns, detailing the app's costs, loyal users, total installs, and revenue over multiple days.
-- `creatives`: Retrieves data for a creative asset, including revenue and cost.
-- `custom:<dimensions>:<metrics>`: Retrieves data for custom tables, which can be specified by the user.
+## Tables
+
+| Name | PK/Merge Key | Inc Key | Inc Strategy |  Details |
+| --------------- | ----------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [campaigns](https://dev.appsflyer.com/hc/reference/master_api_get) | install_time | install_time | merge| Retrieves data for campaigns, detailing the app's costs, loyal users, total installs, and revenue over multiple days.`columns:`  app_id, campaign, geo, install_time, average_ecpi, clicks, cohort_day_1_revenue_per_user, cohort_day_1_total_revenue_per_user, cohort_day_14_revenue_per_user, cohort_day_14_total_revenue_per_user, cohort_day_21_revenue_per_user, cohort_day_21_total_revenue_per_user, cohort_day_3_revenue_per_user, cohort_day_3_total_revenue_per_user, cohort_day_7_revenue_per_user, cohort_day_7_total_revenue_per_user, cost, impressions, installs, loyal_users, retention_day_7, revenue, roi, uninstalls |
+| [creatives](https://dev.appsflyer.com/hc/reference/master_api_get) | install_time | install_time | merge| Retrieves data for a creative asset, including revenue and cost. `columns:` geo, app_id, install_time, campaign, adset_id, adset, ad_id, impressions, clicks, installs, cost, revenue, average_ecpi, loyal_users, uninstalls, roi  |
+| `custom:<dimensions>:<metrics>` | Dynamic (dimensions + install_time) | install_time | merge| Retrieves data for custom tables, which can be specified by the user.|
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/asana.md

@@ -41,16 +41,17 @@ ingestr ingest \
 
 Asana source allows ingesting the following sources into separate tables:
 
-| **Table**    | **Description**                                                                 |
-|---------------|---------------------------------------------------------------------------------|
-| `workspaces`  | Information about people, materials, or assets required to complete a task or project successfully. |
-| `projects`    | Collections of tasks and related information.                                   |
-| `tasks`    | Tasks within a project. Only tasks that belong to a project can be ingested. Users private tasks are not ingested, for example.      |
-| `projects`    | Collections of tasks and related information.                                   |
-| `tags`        | Labels that can be attached to tasks, projects, or conversations to help categorize and organize them. |
-| `stories`     | Updates or comments that team members can add to a task or project.             |
-| `teams`       | Groups of individuals who work together to complete projects and tasks.         |
-| `users`       | Individuals who have access to the Asana platform.                              |
+| Table | Primary/Merge Key | Inc Key | Inc Strategy | Details |
+|-------|----|----------|--------------|---------|
+| `workspaces` | - | - | replace | Information about people, materials, or assets required to complete a task or project successfully. Full reload on each run. |
+| `projects` | - | - | replace | Collections of tasks and related information. Full reload on each run. |
+| `sections` | - | - | replace | Project sections and organization. Full reload on each run. |
+| `tags` | - | - | replace | Labels that can be attached to tasks, projects, or conversations. Full reload on each run. |
+| `tasks` | gid | modified_at | merge | Tasks within a project. Only tasks that belong to a project can be ingested. Uses modified_since API parameter for incremental loading. |
+| `stories` | - | - | replace | Updates or comments that team members can add to a task or project. |
+| `teams` | - | - | replace | Groups of individuals who work together to complete projects and tasks. Full reload on each run. |
+| `users` | - | - | replace | Individuals who have access to the Asana platform. Full reload on each run. |
+
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

ingestr-0.13.89/docs/supported-sources/attio.md

@@ -0,0 +1,47 @@
+# Attio
+[Attio](https://attio.com/) is an AI-native CRM platform that helps companies build, scale, and grow their business.
+
+ingestr supports Attio as a source.
+
+## URI format
+
+The URI format for Attio is as follows:
+
+```plaintext
+attio://?api_key=<api_key>
+```
+
+URI parameters:
+- `api_key`: the API key used for authentication with the Attio API
+
+## Setting up a Attio Integration
+
+You can find your Attio API key by following the guide [here](https://attio.com/help/apps/other-apps/generating-an-api-key).
+
+Let's say your `api_key` is key_123, here's a sample command that will copy the data from Attio into a DuckDB database:
+
+
+```bash
+ingestr ingest \
+--source-uri 'Attio://?api_key=key_123' \
+--source-table 'objects' \
+--dest-uri duckdb:///attio.duckdb \
+--dest-table 'dest.objects'
+```
+
+## Tables
+
+Attio source supports ingesting the following sources into separate tables:
+
+| Table | PK | Inc Key | Inc Strategy | Details |
+|-------|----|----------|--------------|---------|
+| [objects](https://docs.attio.com/rest-api/endpoint-reference/objects/list-objects) | - | - | replace | Objects are the data types used to store facts about your customers. Fetches all objects. Full reload on each run. |
+| [records:{object_api_slug}](https://docs.attio.com/rest-api/endpoint-reference/records/list-records) | - | - | replace | Fetches all records of an object. For example: `records:companies`. Full reload on each run. |
+| [lists](https://docs.attio.com/rest-api/endpoint-reference/lists/list-all-lists) | - | - | replace | Fetches all lists. Full reload on each run. |
+| [list_entries:{list_id}](https://docs.attio.com/rest-api/endpoint-reference/entries/list-entries) | - | - | replace | Lists all items in a specific list. For example: `list_entries:8abc-123-456-789d-123`. Full reload on each run. |
+| [all_list_entries:{object_api_slug}](https://docs.attio.com/rest-api/endpoint-reference/entries/list-entries) | - | - | replace | Fetches all the lists for an object, and then fetches all the entries from that list. For example: `all_list_entries:companies`. Full reload on each run. |
+
+Use this as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> Attio does not support incremental loading, which means ingestr will do a full-refresh.

ingestr-0.13.89/docs/supported-sources/fluxx.md

@@ -0,0 +1,119 @@
+# Fluxx
+
+[Fluxx](https://www.fluxx.io/) is a cloud-based grants management platform designed to streamline and automate the entire grantmaking process for foundations, corporations, governments, and other funding organizations.
+
+ingestr supports Fluxx as a source.
+
+## URI format
+
+The URI format for Fluxx is:
+
+```plaintext
+fluxx://<instance>?client_id=<client_id>&client_secret=<client_secret>
+```
+
+URI parameters:
+
+- `instance`: Your Fluxx instance subdomain (e.g., `mycompany.preprod` for `https://mycompany.preprod.fluxxlabs.com`)
+- `client_id`: OAuth 2.0 client ID for authentication
+- `client_secret`: OAuth 2.0 client secret for authentication
+
+## Example usage
+
+### Basic usage - all fields
+
+Assuming your instance is `myorg.preprod`, you can ingest grant requests into DuckDB using:
+
+```bash
+ingestr ingest \
+--source-uri 'fluxx://myorg.preprod?client_id=your_client_id&client_secret=your_client_secret' \
+--source-table 'grant_request' \
+--dest-uri duckdb:///fluxx.duckdb \
+--dest-table 'raw.grant_request'
+```
+
+### Custom field selection
+
+You can select specific fields to ingest using the colon syntax:
+
+```bash
+ingestr ingest \
+--source-uri 'fluxx://myorg.preprod?client_id=your_client_id&client_secret=your_client_secret' \
+--source-table 'grant_request:id,amount_requested,amount_recommended,granted' \
+--dest-uri duckdb:///fluxx.duckdb \
+--dest-table 'raw.grant_request'
+```
+
+## Tables
+
+Fluxx source allows ingesting the following sources into separate tables:
+
+| Table | PK | Inc Key | Inc Strategy | Details |
+| ----- | -- | ------- | ------------ | ------- |
+| `claim` | id | updated_at | merge | Grant claims and payment requests with approval status, amounts, and dates |
+| `claim_expense` | id | updated_at | merge | Claim expense forms and templates for financial tracking |
+| `claim_expense_datum` | id | updated_at | merge | Individual data entries within claim expense forms with budget category details |
+| `claim_expense_row` | id | updated_at | merge | Specific line items or rows within claim expense forms |
+| `concept_initiative` | id | updated_at | merge | Concept initiatives linking programs, initiatives, and sub-programs/sub-initiatives |
+| `dashboard_theme` | id | updated_at | merge | Dashboard theme configurations for UI customization |
+| `etl_claim_expense_datum` | id | updated_at | merge | ETL data for claim expense items with comprehensive budget tracking details |
+| `etl_grantee_budget_tracker_actual` | id | updated_at | merge | ETL data for actual grantee budget tracker amounts and expenses |
+| `etl_grantee_budget_tracker_period_datum` | id | updated_at | merge | ETL data for grantee budget tracker period information with detailed financial tracking |
+| `etl_relationship` | id | updated_at | merge | ETL data for entity relationships tracking connections between users, organizations, requests, and other entities |
+| `etl_request_budget` | id | updated_at | merge | ETL budget data for request funding sources with comprehensive financial details |
+| `etl_request_transaction_budget` | id | updated_at | merge | ETL budget data for request transaction funding sources including payment tracking |
+| `exempt_organization` | id | updated_at | merge | Tax-exempt organization data including EIN, classification, and financial information |
+| `geo_city` | id | updated_at | merge | City geographic data with coordinates and postal codes |
+| `geo_county` | id | updated_at | merge | County geographic data with FIPS codes |
+| `geo_place` | id | updated_at | merge | Geographic places with ancestry and location data |
+| `geo_region` | id | updated_at | merge | Geographic regions |
+| `geo_state` | id | updated_at | merge | State geographic data with abbreviations and FIPS codes |
+| `grant_request` | id | updated_at | merge | Grant applications and requests with comprehensive details (300+ fields) |
+| `grantee_budget_category` | id | updated_at | merge | Budget category definitions used by grantees for expense tracking |
+| `grantee_budget_tracker` | id | updated_at | merge | Budget tracking documents for grantee financial management |
+| `grantee_budget_tracker_period` | id | updated_at | merge | Time periods for budget tracking with start and end dates |
+| `grantee_budget_tracker_period_datum` | id | updated_at | merge | Budget data entries for specific tracking periods |
+| `grantee_budget_tracker_period_datum_actual` | id | updated_at | merge | Actual expenses and amounts recorded for budget tracking periods |
+| `grantee_budget_tracker_row` | id | updated_at | merge | Individual budget line items and categories within budget trackers |
+| `integration_log` | id | updated_at | merge | Integration and system logs for tracking data processing and errors |
+| `mac_model_type_dyn_financial_audit` | id | updated_at | merge | Dynamic financial audit models with audit tracking, compliance status, and financial variance analysis |
+| `mac_model_type_dyn_mel` | id | updated_at | merge | Dynamic Monitoring, Evaluation & Learning (MEL) models with performance indicators, baseline tracking, and evaluation metrics |
+| `mac_model_type_dyn_tool` | id | updated_at | merge | Dynamic tool management models for tracking deployment status, usage metrics, and tool effectiveness |
+| `machine_category` | id | updated_at | merge | Machine category definitions for workflow state management |
+| `model_attribute_value` | id | updated_at | merge | Model attribute values with hierarchical data and dependencies |
+| `model_document` | id | updated_at | merge | Document metadata including file information, storage details, and document relationships |
+| `model_document_sub_type` | id | updated_at | merge | Document sub-type definitions and categories |
+| `model_document_type` | id | updated_at | merge | Document type configurations including DocuSign integration and permissions |
+| `model_theme` | id | updated_at | merge | Model themes for categorization and program hierarchy organization |
+| `organization` | id | updated_at | merge | Organizations (grantees, fiscal sponsors, etc.) with contact information and tax details |
+| `population_estimate` | id | updated_at | merge | Population estimates by geographic area with demographic data |
+| `population_estimate_year` | id | updated_at | merge | Yearly population estimates with income and demographic breakdowns |
+| `program` | id | updated_at | merge | Funding programs and initiatives |
+| `request_report` | id | updated_at | merge | Reports submitted for grants |
+| `request_transaction` | id | updated_at | merge | Financial transactions and payments |
+| `request_transaction_funding_source` | id | updated_at | merge | Funding source details for specific request transactions |
+| `request_user` | id | updated_at | merge | Relationships between requests and users with roles and descriptions |
+| `salesforce_authentication` | id | updated_at | merge | Salesforce authentication configurations with OAuth tokens, connection management, and API usage tracking |
+| `sub_initiative` | id | updated_at | merge | Sub-initiatives for detailed planning |
+| `sub_program` | id | updated_at | merge | Sub-programs under main programs |
+| `ui_version` | id | updated_at | merge | User interface version information and system configuration |
+| `user` | id | updated_at | merge | User accounts and profiles |
+| `user_organization` | id | updated_at | merge | Relationships between users and organizations with roles, departments, and contact details |
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+### Field Selection
+
+Each resource contains numerous fields. You can:
+1. **Ingest all fields**: Use the resource name directly (e.g., `grant_request`)
+2. **Select specific fields**: Use colon syntax (e.g., `grant_request:id,name,amount_requested`)
+
+The field selection feature is particularly useful for large resources like `grant_request` which has over 300 fields.
+
+## Authentication
+
+Fluxx uses OAuth 2.0 with client credentials flow. To obtain credentials:
+
+1. Contact your Fluxx administrator to create an API client
+2. You'll receive a `client_id` and `client_secret`
+3. Note your Fluxx instance subdomain (the part before `.fluxxlabs.com`)

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/salesforce.md

@@ -7,13 +7,14 @@ Ingestr supports Salesforce as a source.
 
 The URI format for Salesforce is as follows:
 ```
-salesforce://?username=<username>&password=<password>&token=<token>
+salesforce://?username=<username>&password=<password>&token=<token>&domain=<domain>
 ```
 
 URI parameters:
 - `username` is your Salesforce account username.
 - `password` is your Salesforce account password.
 - `token` is your Salesforce security token.
+- `domain` is your Salesforce instance domain.
 
 You can obtain your security token by logging into your Salesforce account and navigating to the user settings under "Reset My Security Token."
 
@@ -27,13 +28,14 @@ Let's say:
 * Your Salesforce username is `user`.
 * Your password is `password123`.
 * Your security token is `fake_token`.
+* Your domain is `test.salesforce.com`.
 * You want to ingest `account` data from your salesforce account
 * You want to save this data in a duckdb database `sf.db` under the table `public.account`
 
 You can run the following command to achieve this:
 ```sh
 ingestr ingest \
-  --source-uri "salesforce://?username=user&password=password123&token=fake_token" \
+  --source-uri "salesforce://?username=user&password=password123&token=fake_token&domain=test.salesforce.com" \
   --source-table "account" \
   --dest-uri "duckdb:///sf.db" \
   --dest-table "public.account"
@@ -43,23 +45,23 @@ ingestr ingest \
 
 Salesforce source allows ingesting the following objects into separate tables:
 
-| **Table**                 | **Mode**   | **Description** |
-|---------------------------|-----------|----------------|
-| `user`                  | replace   | Refers to an individual who has access to a Salesforce org or instance. |
-| `user_role`             | replace   | A standard object that represents a role within the organization's hierarchy. |
-| `opportunity`          | merge     | Represents a sales opportunity for a specific account or contact. |
-| `opportunity_line_item` | merge     | Represents individual line items or products associated with an Opportunity. |
-| `opportunity_contact_role` | merge  | Represents the association between an Opportunity and a Contact. |
-| `account`               | merge     | Individual or organization that interacts with your business. |
-| `contact`               | replace   | An individual person associated with an account or organization. |
-| `lead`                  | replace   | Prospective customer/individual/org. that has shown interest in a company's products/services. |
-| `campaign`              | replace   | Marketing initiative or project designed to achieve specific goals, such as generating leads. |
-| `campaign_member`       | merge     | Association between a Contact or Lead and a Campaign. |
-| `product`               | replace   | For managing and organizing your product-related data within the Salesforce ecosystem. |
-| `pricebook`             | replace   | Used to manage product pricing and create price books. |
-| `pricebook_entry`      | replace   | Represents a specific price for a product in a price book. |
-| `task`                 | merge     | Used to track and manage various activities and tasks within the Salesforce platform. |
-| `event`                | merge     | Used to track and manage calendar-based events, such as meetings, appointments, or calls. |
+| Table | PK | Inc Key | Inc Strategy | Details |
+|-------|----|---------|--------------|---------|
+| `user` | - | - | replace | Refers to an individual who has access to a Salesforce org or instance. |
+| `user_role` | - | - | replace | A standard object that represents a role within the organization's hierarchy.|
+| `opportunity` | id | last_timestamp | merge | Represents a sales opportunity for a specific account or contact. |
+| `opportunity_line_item` | id | last_timestamp | merge | Represents individual line items or products associated with an Opportunity. |
+| `opportunity_contact_role` | id | last_timestamp | merge | Represents the association between an Opportunity and a Contact. |
+| `account` | id | last_timestamp | merge | Individual or organization that interacts with your business. |
+| `contact`  | id | - | replace | An individual person associated with an account or organization. |
+| `lead`  | id | - | replace | Prospective customer/individual/org. that has shown interest in a company's products/services. |
+| `campaign`  | id | - | replace | Marketing initiative or project designed to achieve specific goals, such as generating leads. |
+| `campaign_member`  | id | last_timestamp | merge  | Association between a Contact or Lead and a Campaign. |
+|  `product`  | id | - | replace  | For managing and organizing your product-related data within the Salesforce ecosystem. |
+|  `pricebook`   | id | - | replace  | Used to manage product pricing and create price books. |
+|  `pricebook_entry`   | id | - | replace  | Represents a specific price for a product in a price book. |
+|  `task`   | id | last_timestamp | merge | Used to track and manage various activities and tasks within the Salesforce platform.  |
+|  `event`   | id | last_timestamp | merge | Used to track and manage calendar-based events, such as meetings, appointments, or calls. |
 
 Use these as `--source-table` parameters in the `ingestr ingest` command.
 

{ingestr-0.13.88 → ingestr-0.13.89}/docs/supported-sources/stripe.md

@@ -93,45 +93,46 @@ Stripe source allows ingesting the following sources into separate tables:
 
 All endpoints support the standard async loading mode. The following endpoints are available:
 
-- `account`: Contains information about a Stripe account, including balances, payouts, and account settings.
-- `apple_pay_domain`: Represents Apple Pay domains registered with Stripe for processing Apple Pay payments.
-- `application_fee`: Records fees collected by platforms on payments processed through connected accounts.
-- `checkout_session`: Contains data about Checkout sessions created for payment processing workflows.
-- `coupon`: Stores data about discount codes or coupons that can be applied to invoices, subscriptions, or other charges.
-- `customer`: Holds information about customers, such as billing details, payment methods, and associated transactions.
-- `dispute`: Records payment disputes and chargebacks filed by customers or banks.
-- `payment_intent`: Represents payment intents tracking the lifecycle of payments from creation to completion.
-- `payment_link`: Contains information about payment links created for collecting payments.
-- `payment_method`: Stores payment method information such as cards, bank accounts, and other payment instruments.
-- `payment_method_domain`: Represents domains verified for payment method collection.
-- `payout`: Records payouts made from Stripe accounts to bank accounts or debit cards.
-- `plan`: Contains subscription plan information including pricing and billing intervals.
-- `price`: Contains pricing information for products, including currency, amount, and billing intervals.
-- `product`: Represents products that can be sold or subscribed to, including metadata and pricing information.
-- `promotion_code`: Stores data about promotion codes that customers can use to apply coupons.
-- `quote`: Contains quote information for customers, including line items and pricing.
-- `refund`: Records refunds issued for charges, including partial and full refunds.
-- `review`: Contains payment review information for payments flagged by Stripe Radar.
-- `setup_attempt`: Records attempts to set up payment methods for future payments.
-- `setup_intent`: Represents setup intents for collecting payment method information.
-- `shipping_rate`: Contains shipping rate information for orders and invoices.
-- `subscription`: Represents a customer's subscription to a recurring service, detailing billing cycles, plans, and status.
-- `subscription_item`: Contains individual items within a subscription, including quantities and pricing.
-- `subscription_schedule`: Represents scheduled changes to subscriptions over time.
-- `tax_code`: Contains tax code information for products and services.
-- `tax_id`: Stores tax ID information for customers and accounts.
-- `tax_rate`: Contains tax rate information applied to invoices and subscriptions.
-- `top_up`: Records top-ups made to Stripe accounts.
-- `transfer`: Records transfers between Stripe accounts.
-- `webhook_endpoint`: Contains webhook endpoint configurations for receiving event notifications.
-- `application_fee`: Records fees collected by platforms.
-- `balance_transaction`: Records transactions that affect the Stripe account balance, such as charges, refunds, and payouts.
-- `charge`: Returns a list of charges.
-- `credit_note`: Contains credit note information for refunds and adjustments.
-- `event`: Logs all events in the Stripe account, including customer actions, account updates, and system-generated events.
-- `invoice`: Represents invoices sent to customers, detailing line items, amounts, and payment status.
-- `invoice_item`: Contains individual line items that can be added to invoices.
-- `invoice_line_item`: Represents line items within invoices.
+| Table | PK | Inc Key | Inc Strategy | Details |
+| ----- | -- | ------- | ------------ | ------- |
+| `account` | id | created | merge | Contains information about a Stripe account, including balances, payouts, and account settings. |
+| `apple_pay_domain` | id | created | merge | Represents Apple Pay domains registered with Stripe for processing Apple Pay payments. |
+| `application_fee` | id | created | merge | Records fees collected by platforms on payments processed through connected accounts. |
+| `balance_transaction` | id | created | merge | Records transactions that affect the Stripe account balance, such as charges, refunds, and payouts. |
+| `charge` | id | created | merge | Returns a list of charges. |
+| `checkout_session` | id | created | merge | Contains data about Checkout sessions created for payment processing workflows. |
+| `coupon` | id | created | merge | Stores data about discount codes or coupons that can be applied to invoices, subscriptions, or other charges. |
+| `credit_note` | id | created | merge | Contains credit note information for refunds and adjustments. |
+| `customer` | id | created | merge | Holds information about customers, such as billing details, payment methods, and associated transactions. |
+| `dispute` | id | created | merge | Records payment disputes and chargebacks filed by customers or banks. |
+| `event` | id | created | merge | Logs all events in the Stripe account, including customer actions, account updates, and system-generated events. |
+| `invoice` | id | created | merge | Represents invoices sent to customers, detailing line items, amounts, and payment status. |
+| `invoice_item` | id | created | merge | Contains individual line items that can be added to invoices. |
+| `invoice_line_item` | id | created | merge | Represents line items within invoices. |
+| `payment_intent` | id | created | merge | Represents payment intents tracking the lifecycle of payments from creation to completion. |
+| `payment_link` | id | created | merge | Contains information about payment links created for collecting payments. |
+| `payment_method` | id | created | merge | Stores payment method information such as cards, bank accounts, and other payment instruments. |
+| `payment_method_domain` | id | created | merge | Represents domains verified for payment method collection. |
+| `payout` | id | created | merge | Records payouts made from Stripe accounts to bank accounts or debit cards. |
+| `plan` | id | created | merge | Contains subscription plan information including pricing and billing intervals. |
+| `price` | id | created | merge | Contains pricing information for products, including currency, amount, and billing intervals. |
+| `product` | id | created | merge | Represents products that can be sold or subscribed to, including metadata and pricing information. |
+| `promotion_code` | id | created | merge | Stores data about promotion codes that customers can use to apply coupons. |
+| `quote` | id | created | merge | Contains quote information for customers, including line items and pricing. |
+| `refund` | id | created | merge | Records refunds issued for charges, including partial and full refunds. |
+| `review` | id | created | merge | Contains payment review information for payments flagged by Stripe Radar. |
+| `setup_attempt` | id | created | merge | Records attempts to set up payment methods for future payments. |
+| `setup_intent` | id | created | merge | Represents setup intents for collecting payment method information. |
+| `shipping_rate` | id | created | merge | Contains shipping rate information for orders and invoices. |
+| `subscription` | id | created | merge | Represents a customer's subscription to a recurring service, detailing billing cycles, plans, and status. |
+| `subscription_item` | id | created | merge | Contains individual items within a subscription, including quantities and pricing. |
+| `subscription_schedule` | id | created | merge | Represents scheduled changes to subscriptions over time. |
+| `tax_code` | id | created | merge | Contains tax code information for products and services. |
+| `tax_id` | id | created | merge | Stores tax ID information for customers and accounts. |
+| `tax_rate` | id | created | merge | Contains tax rate information applied to invoices and subscriptions. |
+| `top_up` | id | created | merge | Records top-ups made to Stripe accounts. |
+| `transfer` | id | created | merge | Records transfers between Stripe accounts. |
+| `webhook_endpoint` | id | created | merge | Contains webhook endpoint configurations for receiving event notifications. |
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 

{ingestr-0.13.88 → ingestr-0.13.89}/ingestr/src/applovin/__init__.py

@@ -224,7 +224,7 @@ def resource(
 def custom_report_from_spec(spec: str) -> EndpointResource:
     parts = spec.split(":")
     if len(parts) != 4:
-        raise InvalidCustomReportError()
+        raise InvalidCustomReportError() 
 
     _, endpoint, report, dims = parts
     report_type = ReportType(report.strip())

{ingestr-0.13.88 → ingestr-0.13.89}/ingestr/src/asana_source/__init__.py

@@ -182,7 +182,7 @@ def tasks(
 
 @dlt.transformer(
     data_from=tasks,
-    write_disposition="append",
+    write_disposition="replace",
 )
 @dlt.defer
 def stories(

ingestr-0.13.89/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.89"

{ingestr-0.13.88 → ingestr-0.13.89}/ingestr/src/destinations.py

@@ -25,7 +25,20 @@ from ingestr.src.loader import load_dlt_file
 
 class GenericSqlDestination:
     def dlt_run_params(self, uri: str, table: str, **kwargs) -> dict:
-        table_fields = table.split(".")
+        
+        if uri.startswith("databricks://"):
+            p = urlparse(uri)
+            q = parse_qs(p.query)
+            schema = q.get("schema", [None])[0]
+            if not schema:
+                raise ValueError("Databricks requires schema in the URI.")
+            res = {
+                "dataset_name": schema,
+                "table_name": table,
+            }
+            return res
+        
+        table_fields = table.split(".") 
         if len(table_fields) != 2:
             raise ValueError("Table name must be in the format <schema>.<table>")
 
@@ -270,8 +283,30 @@ class MsSQLDestination(GenericSqlDestination):
 
 class DatabricksDestination(GenericSqlDestination):
     def dlt_dest(self, uri: str, **kwargs):
-        return dlt.destinations.databricks(credentials=uri, **kwargs)
+        p = urlparse(uri)
+        q = parse_qs(p.query)
+        access_token = p.password
+        server_hostname = p.hostname
+        http_path = q.get("http_path", [None])[0]
+        catalog = q.get("catalog", [None])[0]
+        schema = q.get("schema", [None])[0]
+        
+
+        creds = {
+            "access_token": access_token,
+            "server_hostname": server_hostname,
+            "http_path": http_path,
+            "catalog": catalog,
+            "schema": schema,
+        }
+       
+        return dlt.destinations.databricks(
+            credentials=creds,
+            **kwargs,
+        )
+    
 
+    
 
 class SynapseDestination(GenericSqlDestination):
     def dlt_dest(self, uri: str, **kwargs):

{ingestr-0.13.88 → ingestr-0.13.89}/ingestr/src/filesystem/__init__.py

@@ -37,9 +37,14 @@ def readers(
         file_glob (str, optional): The filter to apply to the files in glob format. by default lists all files in bucket_url non-recursively
     """
     filesystem_resource = filesystem(bucket_url, credentials, file_glob=file_glob)
-    filesystem_resource.apply_hints(
-        incremental=dlt.sources.incremental("modification_date"),
-    )
+
+    # NOTE: incremental support is disabled until we can figure out
+    #       how to support incremental loads per matching file, rather
+    #       than a blanket threshold.
+    #
+    # filesystem_resource.apply_hints(
+    #     incremental=dlt.sources.incremental("modification_date"),
+    # )
     return (
         filesystem_resource | dlt.transformer(name="read_csv")(_read_csv),
         filesystem_resource | dlt.transformer(name="read_jsonl")(_read_jsonl),