ingestr 0.13.50__tar.gz → 0.13.51__tar.gz

{ingestr-0.13.50 → ingestr-0.13.51}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.50
+Version: 0.13.51
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues

ingestr-0.13.51/docs/supported-sources/stripe.md

@@ -0,0 +1,91 @@
+# Stripe
+
+[Stripe](https://www.stripe.com/) is a technology company that builds economic infrastructure for the internet, providing payment processing software and APIs for e-commerce websites and mobile applications.
+
+ingestr supports Stripe as a source.
+
+## URI format
+
+The URI format for Stripe is as follows:
+
+```plaintext
+stripe://?api_key=<api-key-here>
+```
+
+URI parameters:
+
+- `api_key`: the API key used for authentication with the Stripe API
+
+The URI is used to connect to the Stripe API for extracting data. More details on setting up Stripe integrations can be found [here](https://stripe.com/docs/api).
+
+## Setting up a Stripe Integration
+
+Stripe requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/stripe#setup-guide).
+
+Once you complete the guide, you should have an API key. Let's say your API key is `sk_test_12345`, here's a sample command that will copy the data from Stripe into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'charges' --dest-uri duckdb:///stripe.duckdb --dest-table 'dest.charges'
+```
+
+The result of this command will be a table in the `stripe.duckdb` database with JSON columns.
+
+## Tables
+
+Stripe source allows ingesting the following sources into separate tables:
+
+### Regular Endpoints
+
+- `account`: Contains information about a Stripe account, including balances, payouts, and account settings.
+- `apple_pay_domain`: Represents Apple Pay domains registered with Stripe for processing Apple Pay payments.
+- `application_fee`: Records fees collected by platforms on payments processed through connected accounts.
+- `checkout_session`: Contains data about Checkout sessions created for payment processing workflows.
+- `coupon`: Stores data about discount codes or coupons that can be applied to invoices, subscriptions, or other charges.
+- `customer`: Holds information about customers, such as billing details, payment methods, and associated transactions.
+- `dispute`: Records payment disputes and chargebacks filed by customers or banks.
+- `payment_intent`: Represents payment intents tracking the lifecycle of payments from creation to completion.
+- `payment_link`: Contains information about payment links created for collecting payments.
+- `payment_method`: Stores payment method information such as cards, bank accounts, and other payment instruments.
+- `payment_method_domain`: Represents domains verified for payment method collection.
+- `payout`: Records payouts made from Stripe accounts to bank accounts or debit cards.
+- `plan`: Contains subscription plan information including pricing and billing intervals.
+- `price`: Contains pricing information for products, including currency, amount, and billing intervals.
+- `product`: Represents products that can be sold or subscribed to, including metadata and pricing information.
+- `promotion_code`: Stores data about promotion codes that customers can use to apply coupons.
+- `quote`: Contains quote information for customers, including line items and pricing.
+- `refund`: Records refunds issued for charges, including partial and full refunds.
+- `review`: Contains payment review information for payments flagged by Stripe Radar.
+- `setup_attempt`: Records attempts to set up payment methods for future payments.
+- `setup_intent`: Represents setup intents for collecting payment method information.
+- `shipping_rate`: Contains shipping rate information for orders and invoices.
+- `subscription`: Represents a customer's subscription to a recurring service, detailing billing cycles, plans, and status.
+- `subscription_item`: Contains individual items within a subscription, including quantities and pricing.
+- `subscription_schedule`: Represents scheduled changes to subscriptions over time.
+- `tax_code`: Contains tax code information for products and services.
+- `tax_id`: Stores tax ID information for customers and accounts.
+- `tax_rate`: Contains tax rate information applied to invoices and subscriptions.
+- `top_up`: Records top-ups made to Stripe accounts.
+- `transfer`: Records transfers between Stripe accounts.
+- `webhook_endpoint`: Contains webhook endpoint configurations for receiving event notifications.
+
+### Incremental Endpoints
+
+The following endpoints support incremental loading, meaning only new or updated records will be fetched:
+
+- `application_fee`: Records fees collected by platforms (incremental).
+- `balance_transaction`: Records transactions that affect the Stripe account balance, such as charges, refunds, and payouts.
+- `charge`: Returns a list of charges.
+- `credit_note`: Contains credit note information for refunds and adjustments.
+- `event`: Logs all events in the Stripe account, including customer actions, account updates, and system-generated events.
+- `invoice`: Represents invoices sent to customers, detailing line items, amounts, and payment status.
+- `invoice_item`: Contains individual line items that can be added to invoices.
+- `invoice_line_item`: Represents line items within invoices.
+- `setup_attempt`: Records attempts to set up payment methods (also available as incremental).
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> Stripe does not support incremental loading for many endpoints in its APIs, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.
+
+> [!NOTE]
+> For backward compatibility, non-underscored versions of table names (e.g., `checkoutsession`, `paymentintent`, `subscriptionitem`) are still supported but will be deprecated in future versions. Please use the underscored versions (e.g., `checkout_session`, `payment_intent`, `subscription_item`) for new integrations.

ingestr-0.13.51/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.51"

{ingestr-0.13.50 → ingestr-0.13.51}/ingestr/src/factory.py

@@ -59,6 +59,7 @@ from ingestr.src.sources import (
     StripeAnalyticsSource,
     TikTokSource,
     ZendeskSource,
+    SmartsheetSource,
 )
 
 SQL_SOURCE_SCHEMES = [
@@ -161,6 +162,7 @@ class SourceDestinationFactory:
         "elasticsearch": ElasticsearchSource,
         "attio": AttioSource,
         "solidgate": SolidgateSource,
+        "smartsheet": SmartsheetSource,
     }
     destinations: Dict[str, Type[DestinationProtocol]] = {
         "bigquery": BigQueryDestination,

ingestr-0.13.51/ingestr/src/http_client.py

@@ -0,0 +1,24 @@
+import requests
+from dlt.sources.helpers.requests import Client
+
+
+def create_client(retry_status_codes: list[int] | None = None) -> requests.Session:
+    if retry_status_codes is None:
+        retry_status_codes = [502]
+    return Client(
+        raise_for_status=False,
+        retry_condition=retry_on_status_code(retry_status_codes),
+        request_max_attempts=12,
+        request_backoff_factor=10,
+    ).session
+
+
+def retry_on_status_code(retry_status_codes: list[int]):
+    def retry_on_limit(
+        response: requests.Response | None, exception: BaseException | None
+    ) -> bool:
+        if response is None:
+            return False
+        return response.status_code in retry_status_codes
+
+    return retry_on_limit

{ingestr-0.13.50 → ingestr-0.13.51}/ingestr/src/solidgate/__init__.py

@@ -94,4 +94,34 @@ def solidgate_source(
         start_dt = dateTime.last_value
         yield solidgate_client.fetch_data(path, date_from=start_dt, date_to=end_dt)
 
-    return fetch_all_subscriptions, fetch_apm_orders, fetch_card_orders
+    @dlt.resource(
+        name="financial-entries",
+        write_disposition="merge",
+        primary_key="id",
+        columns={
+            "created_at": {"data_type": "timestamp", "partition": True},
+        },
+    )
+    def fetch_financial_entries(
+        dateTime=dlt.sources.incremental(
+            "created_at",
+            initial_value=start_date,
+            end_value=end_date,
+            range_start="closed",
+            range_end="closed",
+        ),
+    ):
+        if dateTime.end_value is None:
+            end_date = pendulum.now(tz="UTC")
+        else:
+            end_date = dateTime.end_value
+
+        start_date = dateTime.last_value
+        yield solidgate_client.fetch_financial_entry_data(start_date, end_date)
+
+    return (
+        fetch_all_subscriptions,
+        fetch_apm_orders,
+        fetch_card_orders,
+        fetch_financial_entries,
+    )

ingestr-0.13.51/ingestr/src/solidgate/helpers.py

@@ -0,0 +1,139 @@
+import base64
+import hashlib
+import hmac
+import json
+import time
+from io import StringIO
+
+import pandas as pd  # type: ignore
+import pendulum
+
+from ingestr.src.http_client import create_client
+
+
+class SolidgateClient:
+    def __init__(self, public_key, secret_key):
+        self.base_url = "https://reports.solidgate.com/api/v1"
+        self.public_key = public_key
+        self.secret_key = secret_key
+        self.client = create_client(retry_status_codes=[204])
+
+    def fetch_data(
+        self,
+        path: str,
+        date_from: pendulum.DateTime,
+        date_to: pendulum.DateTime,
+    ):
+        request_payload = {
+            "date_from": date_from.format("YYYY-MM-DD HH:mm:ss"),
+            "date_to": date_to.format("YYYY-MM-DD HH:mm:ss"),
+        }
+
+        json_string = json.dumps(request_payload)
+        signature = self.generateSignature(json_string)
+        headers = {
+            "merchant": self.public_key,
+            "Signature": signature,
+            "Content-Type": "application/json",
+        }
+
+        next_page_iterator = None
+        url = f"{self.base_url}/{path}"
+
+        while True:
+            payload = request_payload.copy()
+            if next_page_iterator:
+                payload["page_iterator"] = next_page_iterator
+
+            response = self.client.post(url, headers=headers, json=payload)
+            response.raise_for_status()
+            response_json = response.json()
+
+            if path == "subscriptions":
+                data = response_json["subscriptions"]
+                for _, value in data.items():
+                    if "updated_at" in value:
+                        value["updated_at"] = pendulum.parse(value["updated_at"])
+                    yield value
+
+            else:
+                data = response_json["orders"]
+                for value in data:
+                    if "updated_at" in value:
+                        value["updated_at"] = pendulum.parse(value["updated_at"])
+                    yield value
+
+            next_page_iterator = response_json.get("metadata", {}).get(
+                "next_page_iterator"
+            )
+            if not next_page_iterator or next_page_iterator == "None":
+                break
+
+    def fetch_financial_entry_data(
+        self, date_from: pendulum.DateTime, date_to: pendulum.DateTime
+    ):
+        request_payload = {
+            "date_from": date_from.format("YYYY-MM-DD HH:mm:ss"),
+            "date_to": date_to.format("YYYY-MM-DD HH:mm:ss"),
+        }
+
+        json_string = json.dumps(request_payload)
+        signature = self.generateSignature(json_string)
+        headers = {
+            "merchant": self.public_key,
+            "Signature": signature,
+            "Content-Type": "application/json",
+        }
+
+        url = f"{self.base_url}/finance/financial_entries"
+        post_response = self.client.post(url, headers=headers, json=request_payload)
+        post_response.raise_for_status()
+        report_url = post_response.json().get("report_url")
+        if not report_url:
+            return f"Report URL not found in the response: {post_response.json()}", 400
+
+        # Wait for 5 seconds before attempting to download the report as report may not be immediately available
+        time.sleep(5)
+
+        data = self.public_key + self.public_key
+        hmac_hash = hmac.new(
+            self.secret_key.encode("utf-8"), data.encode("utf-8"), hashlib.sha512
+        ).digest()
+        signature_get = base64.b64encode(hmac_hash.hex().encode("utf-8")).decode(
+            "utf-8"
+        )
+
+        headers_get = {
+            "merchant": self.public_key,
+            "Signature": signature_get,
+            "Content-Type": "application/json",
+        }
+
+        get_response = self.client.get(report_url, headers=headers_get)
+
+        if get_response.status_code == 200:
+            try:
+                response_json = json.loads(get_response.content)
+                if "error" in response_json:
+                    raise Exception(f"API Error: {response_json['error']['messages']}")
+            except json.JSONDecodeError:
+                try:
+                    csv_data = get_response.content.decode("utf-8")
+                    df = pd.read_csv(StringIO(csv_data))
+                    df["created_at"] = df["created_at"].apply(
+                        lambda x: pendulum.parse(x)
+                    )
+                    return df
+                except Exception as e:
+                    raise Exception(f"Error reading CSV: {e}")
+        else:
+            raise Exception(
+                f"Failed to get report. Status code: {get_response.status_code}"
+            )
+
+    def generateSignature(self, json_string):
+        data = self.public_key + json_string + self.public_key
+        hmac_hash = hmac.new(
+            self.secret_key.encode("utf-8"), data.encode("utf-8"), hashlib.sha512
+        ).digest()
+        return base64.b64encode(hmac_hash.hex().encode("utf-8")).decode("utf-8")

ingestr-0.13.51/ingestr/src/stripe_analytics/settings.py

@@ -0,0 +1,73 @@
+"""Stripe analytics source settings and constants"""
+
+# the most popular endpoints
+# Full list of the Stripe API endpoints you can find here: https://stripe.com/docs/api.
+ENDPOINTS = {
+    "account": "Account",
+    "applepaydomain": "ApplePayDomain",
+    "apple_pay_domain": "ApplePayDomain",
+    "applicationfee": "ApplicationFee",
+    "application_fee": "ApplicationFee",
+    "checkoutsession": "CheckoutSession",
+    "checkout_session": "CheckoutSession",
+    "coupon": "Coupon",
+    "customer": "Customer",
+    "dispute": "Dispute",
+    "paymentintent": "PaymentIntent",
+    "payment_intent": "PaymentIntent",
+    "paymentlink": "PaymentLink",
+    "payment_link": "PaymentLink",
+    "paymentmethod": "PaymentMethod",
+    "payment_method": "PaymentMethod",
+    "paymentmethoddomain": "PaymentMethodDomain",
+    "payment_method_domain": "PaymentMethodDomain",
+    "payout": "Payout",
+    "plan": "Plan",
+    "price": "Price",
+    "product": "Product",
+    "promotioncode": "PromotionCode",
+    "promotion_code": "PromotionCode",
+    "quote": "Quote",
+    "refund": "Refund",
+    "review": "Review",
+    "setupattempt": "SetupAttempt",
+    "setup_attempt": "SetupAttempt",
+    "setupintent": "SetupIntent",
+    "setup_intent": "SetupIntent",
+    "shippingrate": "ShippingRate",
+    "shipping_rate": "ShippingRate",
+    "subscription": "Subscription",
+    "subscriptionitem": "SubscriptionItem",
+    "subscription_item": "SubscriptionItem",
+    "subscriptionschedule": "SubscriptionSchedule",
+    "subscription_schedule": "SubscriptionSchedule",
+    "transfer": "Transfer",
+    "taxcode": "TaxCode",
+    "tax_code": "TaxCode",
+    "taxid": "TaxId",
+    "tax_id": "TaxId",
+    "taxrate": "TaxRate",
+    "tax_rate": "TaxRate",
+    "topup": "Topup",
+    "top_up": "Topup",
+    "webhookendpoint": "WebhookEndpoint",
+    "webhook_endpoint": "WebhookEndpoint",
+}
+# possible incremental endpoints
+INCREMENTAL_ENDPOINTS = {
+    "applicationfee": "ApplicationFee",
+    "application_fee": "ApplicationFee",
+    "balancetransaction": "BalanceTransaction",
+    "balance_transaction": "BalanceTransaction",
+    "charge": "Charge",
+    "creditnote": "CreditNote",
+    "credit_note": "CreditNote",
+    "event": "Event",
+    "invoice": "Invoice",
+    "invoiceitem": "InvoiceItem",
+    "invoice_item": "InvoiceItem",
+    "invoicelineitem": "InvoiceLineItem",
+    "invoice_line_item": "InvoiceLineItem",
+    "setupattempt": "SetupAttempt",
+    "setup_attempt": "SetupAttempt",
+}

ingestr-0.13.50/docs/supported-sources/stripe.md

@@ -1,51 +0,0 @@
-# Stripe
-
-[Stripe](https://www.stripe.com/) is a technology company that builds economic infrastructure for the internet, providing payment processing software and APIs for e-commerce websites and mobile applications.
-
-ingestr supports Stripe as a source.
-
-## URI format
-
-The URI format for Stripe is as follows:
-
-```plaintext
-stripe://?api_key=<api-key-here>
-```
-
-URI parameters:
-
-- `api_key`: the API key used for authentication with the Stripe API
-
-The URI is used to connect to the Stripe API for extracting data. More details on setting up Stripe integrations can be found [here](https://stripe.com/docs/api).
-
-## Setting up a Stripe Integration
-
-Stripe requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/stripe#setup-guide).
-
-Once you complete the guide, you should have an API key. Let's say your API key is `sk_test_12345`, here's a sample command that will copy the data from Stripe into a DuckDB database:
-
-```sh
-ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'charges' --dest-uri duckdb:///stripe.duckdb --dest-table 'dest.charges'
-```
-
-The result of this command will be a table in the `stripe.duckdb` database with JSON columns.
-
-## Tables
-
-Stripe source allows ingesting the following sources into separate tables:
-
-- `subscription`: Represents a customer's subscription to a recurring service, detailing billing cycles, plans, and status.
-- `account`: Contains information about a Stripe account, including balances, payouts, and account settings.
-- `coupon`: Stores data about discount codes or coupons that can be applied to invoices, subscriptions, or other charges.
-- `customer`: Holds information about customers, such as billing details, payment methods, and associated transactions.
-- `product`: Represents products that can be sold or subscribed to, including metadata and pricing information.
-- `price`: Contains pricing information for products, including currency, amount, and billing intervals.
-- `balancetransaction`: Records transactions that affect the Stripe account balance, such as charges, refunds, and payouts.
-- `invoice`: Represents invoices sent to customers, detailing line items, amounts, and payment status.
-- `event`: Logs all events in the Stripe account, including customer actions, account updates, and system-generated events.
-- `charge`: Returns a list of charges.
-
-Use these as `--source-table` parameter in the `ingestr ingest` command.
-
-> [!WARNING]
-> Stripe does not support incremental loading for many endpoints in its APIs, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.

ingestr-0.13.50/ingestr/src/buildinfo.py

@@ -1 +0,0 @@
-version = "v0.13.50"

ingestr-0.13.50/ingestr/src/http_client.py

@@ -1,18 +0,0 @@
-import requests
-from dlt.sources.helpers.requests import Client
-
-
-def create_client() -> requests.Session:
-    return Client(
-        raise_for_status=False,
-        retry_condition=retry_on_limit,
-        request_max_attempts=12,
-    ).session
-
-
-def retry_on_limit(
-    response: requests.Response | None, exception: BaseException | None
-) -> bool:
-    if response is None:
-        return False
-    return response.status_code == 502

ingestr-0.13.50/ingestr/src/solidgate/helpers.py

@@ -1,74 +0,0 @@
-import base64
-import hashlib
-import hmac
-import json
-
-import pendulum
-
-from ingestr.src.http_client import create_client
-
-
-class SolidgateClient:
-    def __init__(self, public_key, secret_key):
-        self.base_url = "https://reports.solidgate.com/api/v1"
-        self.public_key = public_key
-        self.secret_key = secret_key
-        self.client = create_client()
-
-    def fetch_data(
-        self,
-        path: str,
-        date_from: pendulum.DateTime,
-        date_to: pendulum.DateTime,
-    ):
-        request_payload = {
-            "date_from": date_from.format("YYYY-MM-DD HH:mm:ss"),
-            "date_to": date_to.format("YYYY-MM-DD HH:mm:ss"),
-        }
-
-        json_string = json.dumps(request_payload)
-        signature = self.generateSignature(json_string)
-        headers = {
-            "merchant": self.public_key,
-            "Signature": signature,
-            "Content-Type": "application/json",
-        }
-
-        next_page_iterator = None
-        url = f"{self.base_url}/{path}"
-
-        while True:
-            payload = request_payload.copy()
-            if next_page_iterator:
-                payload["page_iterator"] = next_page_iterator
-
-            response = self.client.post(url, headers=headers, json=payload)
-            response.raise_for_status()
-            response_json = response.json()
-
-            if path == "subscriptions":
-                data = response_json["subscriptions"]
-                for _, value in data.items():
-                    if "updated_at" in value:
-                        value["updated_at"] = pendulum.parse(value["updated_at"])
-                    yield value
-
-            else:
-                data = response_json["orders"]
-                for value in data:
-                    if "updated_at" in value:
-                        value["updated_at"] = pendulum.parse(value["updated_at"])
-                    yield value
-
-            next_page_iterator = response_json.get("metadata", {}).get(
-                "next_page_iterator"
-            )
-            if not next_page_iterator or next_page_iterator == "None":
-                break
-
-    def generateSignature(self, json_string):
-        data = self.public_key + json_string + self.public_key
-        hmac_hash = hmac.new(
-            self.secret_key.encode("utf-8"), data.encode("utf-8"), hashlib.sha512
-        ).digest()
-        return base64.b64encode(hmac_hash.hex().encode("utf-8")).decode("utf-8")

ingestr-0.13.50/ingestr/src/stripe_analytics/settings.py

@@ -1,26 +0,0 @@
-"""Stripe analytics source settings and constants"""
-
-# the most popular endpoints
-# Full list of the Stripe API endpoints you can find here: https://stripe.com/docs/api.
-ENDPOINTS = {
-    "subscription": "Subscription",
-    "account": "Account",
-    "coupon": "Coupon",
-    "customer": "Customer",
-    "product": "Product",
-    "price": "Price",
-    "shippingrate": "ShippingRate",
-    "dispute": "Dispute",
-    "subscriptionitem": "SubscriptionItem",
-    "checkoutsession": "CheckoutSession",
-}
-# possible incremental endpoints
-INCREMENTAL_ENDPOINTS = {
-    "event": "Event",
-    "invoice": "Invoice",
-    "balancetransaction": "BalanceTransaction",
-    "charge": "Charge",
-    "applicationfee": "ApplicationFee",
-    "setupattempt": "SetupAttempt",
-    "creditnote": "CreditNote",
-}