ingestr 0.13.56__tar.gz → 0.13.57__tar.gz

{ingestr-0.13.56 → ingestr-0.13.57}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.56
+Version: 0.13.57
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -27,7 +27,6 @@ Requires-Dist: asynch==0.2.4
 Requires-Dist: attrs==25.1.0
 Requires-Dist: backoff==2.2.1
 Requires-Dist: bcrypt==4.3.0
-Requires-Dist: beautifulsoup4==4.13.3
 Requires-Dist: boto3==1.37.1
 Requires-Dist: botocore==1.37.1
 Requires-Dist: cachetools==5.5.2
@@ -156,7 +155,6 @@ Requires-Dist: python-quickbooks==0.9.2
 Requires-Dist: pytz==2025.1
 Requires-Dist: pyyaml==6.0.2
 Requires-Dist: rauth==0.7.3
-Requires-Dist: redshift-connector==2.1.5
 Requires-Dist: requests-file==2.1.0
 Requires-Dist: requests-oauthlib==1.3.1
 Requires-Dist: requests-toolbelt==1.0.0
@@ -168,7 +166,6 @@ Requires-Dist: rsa==4.9
 Requires-Dist: rudder-sdk-python==2.1.4
 Requires-Dist: s3fs==2025.3.2
 Requires-Dist: s3transfer==0.11.3
-Requires-Dist: scramp==1.4.5
 Requires-Dist: semver==3.0.4
 Requires-Dist: setuptools==75.8.2
 Requires-Dist: shellingham==1.5.4
@@ -180,7 +177,6 @@ Requires-Dist: smmap==5.0.2
 Requires-Dist: snowflake-connector-python==3.14.0
 Requires-Dist: snowflake-sqlalchemy==1.6.1
 Requires-Dist: sortedcontainers==2.4.0
-Requires-Dist: soupsieve==2.6
 Requires-Dist: sqlalchemy-bigquery==1.12.1
 Requires-Dist: sqlalchemy-cratedb==0.42.0.dev2
 Requires-Dist: sqlalchemy-hana==2.0.0

{ingestr-0.13.56 → ingestr-0.13.57}/docs/supported-sources/kinesis.md

@@ -31,9 +31,13 @@ ingestr ingest --source-uri 'kinesis://?aws_access_key_id=id_123&aws_secret_acce
  --dest-table 'dest.results'
 ```
 
-When using Kinesis as a source, specify the `stream name` you want to read from as the `--source-table` parameter. For example, if you want to read from a Kinesis stream named "customer_events", you would use `--source-table 'customer_events'`.
+When using Kinesis as a source, specify the [StreamName] you want to read from as the `--source-table` parameter. For example, if you want to read from a Kinesis stream named "customer_events", you would use `--source-table 'customer_events'`.
+You can also use a full Kinesis [StreamARN] to address the stream in [ARN] format, like `arn:aws:kinesis:eu-central-1:842404475894:stream/customer_events`.
 
 ### Initial Load Configuration
 By default, ingestr reads from the beginning of the Kinesis stream. To start reading from a specific time, use the `interval_start` parameter.
 
 
+[ARN]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html
+[StreamARN]: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamARN
+[StreamName]: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamName

{ingestr-0.13.56 → ingestr-0.13.57}/docs/supported-sources/stripe.md

@@ -30,11 +30,68 @@ ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'ch
 
 The result of this command will be a table in the `stripe.duckdb` database with JSON columns.
 
+## Table Name Structure
+
+Stripe source supports different loading modes that can be specified using the table name structure:
+
+- `<endpoint>` - Standard async loading (default)
+- `<endpoint>:sync` - Full loading with synchronous processing
+- `<endpoint>:sync:incremental` - Incremental loading mode with synchronous processing
+
+### Loading Modes and Trade-offs
+
+#### Standard Async Loading (Default)
+**Format**: `<endpoint>` (e.g., `charges`, `subscriptions`)
+
+- **Use case**: Full data loading from all time periods
+- **Performance**: Loads data in parallel using async processing
+- **Data completeness**: Captures all historical data and updates
+- **Speed**: Slower due to comprehensive data retrieval
+- **Best for**: You want to have all updated data in your database
+
+**Example**:
+```sh
+ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'subscriptions' --dest-uri duckdb:///stripe.duckdb --dest-table 'dest.subscriptions'
+```
+
+#### Sync Loading
+**Format**: `<endpoint>:sync` (e.g., `charges:sync`, `subscriptions:sync`)
+
+- **Use case**: Full data loading from all time periods
+- **Performance**: Loads data in parallel using sync processing
+- **Data completeness**: Captures all historical data and updates
+- **Speed**: Slower due to comprehensive data retrieval, faster if you have less data
+
+#### Incremental Loading
+**Format**: `<endpoint>:sync:incremental` (e.g., `charges:sync:incremental`, `events:sync:incremental`)
+
+- **Use case**: Loading data within specific time windows
+- **Performance**: Fast, processes only data within the specified interval
+- **Data completeness**: Limited to the specified time window, does not track updates from past dates
+- **Speed**: Faster due to filtered data retrieval
+- **Processing**: Runs in synchronous mode only
+- **Best for**: Quick loads, you don't care about the updates to past data
+
+**Example**:
+```sh
+ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'charges:sync:incremental' --dest-uri duckdb:///stripe.duckdb --dest-table 'dest.charges' --interval-start '2024-01-01' --interval-end '2024-01-31'
+```
+
+### Choosing the Right Approach
+
+| Approach | Speed | Data Completeness | Use Case |
+|----------|--------|------------------|----------|
+| **Standard Async** | Faster for larger data, slower for smaller data | Complete historical data | Initial loads, full historical analysis |
+| **Sync** | Slow for larger data, faster for smaller data | Complete historical data | Initial loads, full historical analysis |
+| **Incremental** | Fastest | Time-window specific | Regular updates, recent data analysis |
+
 ## Tables
 
 Stripe source allows ingesting the following sources into separate tables:
 
-### Regular Endpoints
+### All Endpoints
+
+All endpoints support the standard async loading mode. The following endpoints are available:
 
 - `account`: Contains information about a Stripe account, including balances, payouts, and account settings.
 - `apple_pay_domain`: Represents Apple Pay domains registered with Stripe for processing Apple Pay payments.
@@ -67,12 +124,7 @@ Stripe source allows ingesting the following sources into separate tables:
 - `top_up`: Records top-ups made to Stripe accounts.
 - `transfer`: Records transfers between Stripe accounts.
 - `webhook_endpoint`: Contains webhook endpoint configurations for receiving event notifications.
-
-### Incremental Endpoints
-
-The following endpoints support incremental loading, meaning only new or updated records will be fetched:
-
-- `application_fee`: Records fees collected by platforms (incremental).
+- `application_fee`: Records fees collected by platforms.
 - `balance_transaction`: Records transactions that affect the Stripe account balance, such as charges, refunds, and payouts.
 - `charge`: Returns a list of charges.
 - `credit_note`: Contains credit note information for refunds and adjustments.
@@ -80,12 +132,14 @@ The following endpoints support incremental loading, meaning only new or updated
 - `invoice`: Represents invoices sent to customers, detailing line items, amounts, and payment status.
 - `invoice_item`: Contains individual line items that can be added to invoices.
 - `invoice_line_item`: Represents line items within invoices.
-- `setup_attempt`: Records attempts to set up payment methods (also available as incremental).
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 
+> [!TIP]
+> For time-sensitive data analysis or regular updates, use incremental loading (`:incremental`) with `--interval-start` and `--interval-end` parameters for faster processing. For comprehensive historical analysis, use standard async loading without any suffix.
+
 > [!WARNING]
-> Stripe does not support incremental loading for many endpoints in its APIs, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.
+> Incremental loading filters data based on the specified time window and does not track updates to records created outside that window. Use standard async loading if you need to capture all historical updates.
 
 > [!NOTE]
 > For backward compatibility, non-underscored versions of table names (e.g., `checkoutsession`, `paymentintent`, `subscriptionitem`) are still supported but will be deprecated in future versions. Please use the underscored versions (e.g., `checkout_session`, `payment_intent`, `subscription_item`) for new integrations.

ingestr-0.13.57/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.57"

{ingestr-0.13.56 → ingestr-0.13.57}/ingestr/src/kinesis/__init__.py

@@ -9,7 +9,7 @@ from dlt.common.time import ensure_pendulum_datetime
 from dlt.common.typing import StrStr, TAnyDateTime, TDataItem
 from dlt.common.utils import digest128
 
-from .helpers import get_shard_iterator, max_sequence_by_shard
+from .helpers import get_shard_iterator, get_stream_address, max_sequence_by_shard
 
 
 @dlt.resource(
@@ -42,7 +42,7 @@ def kinesis_stream(
         initial_at_timestamp (TAnyDateTime): An initial timestamp used to generate AT_TIMESTAMP or LATEST iterator when timestamp value is 0
         max_number_of_messages (int): Maximum number of messages to read in one run. Actual read may exceed that number by up to chunk_size. Defaults to None (no limit).
         milliseconds_behind_latest (int): The number of milliseconds behind the top of the shard to stop reading messages, defaults to 1000.
-        parse_json (bool): If True, assumes that messages are json strings, parses them and returns instead of `data` (otherwise). Defaults to False.
+        parse_json (bool): If True, assumes that messages are json strings, parses them and returns instead of `data` (otherwise). Defaults to True.
         chunk_size (int): The number of records to fetch at once. Defaults to 1000.
     Yields:
             Iterable[TDataItem]: Messages. Contain Kinesis envelope in `kinesis` and bytes data in `data` (if `parse_json` disabled)
@@ -65,7 +65,7 @@ def kinesis_stream(
     # so next time we request shards at AT_TIMESTAMP that is now
     resource_state["initial_at_timestamp"] = pendulum.now("UTC").subtract(seconds=1)
 
-    shards_list = kinesis_client.list_shards(StreamName=stream_name)
+    shards_list = kinesis_client.list_shards(**get_stream_address(stream_name))
     shards: List[StrStr] = shards_list["Shards"]
     while next_token := shards_list.get("NextToken"):
         shards_list = kinesis_client.list_shards(NextToken=next_token)

{ingestr-0.13.56 → ingestr-0.13.57}/ingestr/src/kinesis/helpers.py

@@ -2,7 +2,7 @@ from typing import Any, Sequence, Tuple
 
 import dlt
 from dlt.common import pendulum
-from dlt.common.typing import DictStrAny, StrAny, StrStr
+from dlt.common.typing import DictStrAny, DictStrStr, StrAny, StrStr
 
 
 def get_shard_iterator(
@@ -40,7 +40,7 @@ def get_shard_iterator(
         )
 
     shard_iterator: StrStr = kinesis_client.get_shard_iterator(
-        StreamName=stream_name, ShardId=shard_id, **iterator_params
+        **get_stream_address(stream_name), ShardId=shard_id, **iterator_params
     )
     return shard_iterator["ShardIterator"], iterator_params
 
@@ -63,3 +63,20 @@ def max_sequence_by_shard(values: Sequence[StrStr]) -> StrStr:
     # we compare message sequence at shard_id
     last_value[shard_id] = max(item["seq_no"], last_value.get(shard_id, ""))
     return last_value
+
+
+def get_stream_address(stream_name: str) -> DictStrStr:
+    """
+    Return address of stream, either as StreamName or StreamARN, when applicable.
+
+    Examples:
+    - customer_events
+    - arn:aws:kinesis:eu-central-1:842404475894:stream/customer_events
+
+    https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamName
+    https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamARN
+    """
+    if stream_name.startswith("arn:"):
+        return {"StreamARN": stream_name}
+    else:
+        return {"StreamName": stream_name}

{ingestr-0.13.56 → ingestr-0.13.57}/ingestr/src/sources.py

@@ -677,24 +677,33 @@ class StripeAnalyticsSource:
 
         table = table.lower()
 
-        from ingestr.src.stripe_analytics.settings import (
-            ENDPOINTS,
-            INCREMENTAL_ENDPOINTS,
-        )
+        from ingestr.src.stripe_analytics.settings import ENDPOINTS
 
-        if table in ENDPOINTS:
-            endpoint = ENDPOINTS[table]
-            from ingestr.src.stripe_analytics import stripe_source
+        endpoint = None
+        incremental = False
+        sync = False
 
-            return stripe_source(
-                endpoints=[
-                    endpoint,
-                ],
-                stripe_secret_key=api_key[0],
-            ).with_resources(endpoint)
+        table_fields = table.split(":")
+        if len(table_fields) == 1:
+            endpoint = table_fields[0]
+        elif len(table_fields) == 2:
+            endpoint = table_fields[0]
+            sync = table_fields[1] == "sync"
+        elif len(table_fields) == 3:
+            endpoint = table_fields[0]
+            sync = table_fields[1] == "sync"
+            incremental = table_fields[2] == "incremental"
+        else:
+            raise ValueError(
+                "Invalid Stripe table format. Expected: stripe:<endpoint> or stripe:<endpoint>:<sync> or stripe:<endpoint>:<sync>:<incremental>"
+            )
+
+        if incremental and not sync:
+            raise ValueError(
+                "incremental loads must be used with sync loading"
+            )
 
-        elif table in INCREMENTAL_ENDPOINTS:
-            endpoint = INCREMENTAL_ENDPOINTS[table]
+        if incremental:
             from ingestr.src.stripe_analytics import incremental_stripe_source
 
             def nullable_date(date_str: Optional[str]):
@@ -702,6 +711,7 @@ class StripeAnalyticsSource:
                     return ensure_pendulum_datetime(date_str)
                 return None
 
+            endpoint = ENDPOINTS[endpoint]
             return incremental_stripe_source(
                 endpoints=[
                     endpoint,
@@ -710,6 +720,26 @@ class StripeAnalyticsSource:
                 initial_start_date=nullable_date(kwargs.get("interval_start", None)),
                 end_date=nullable_date(kwargs.get("interval_end", None)),
             ).with_resources(endpoint)
+        else:
+            endpoint = ENDPOINTS[endpoint]
+            if sync:
+                from ingestr.src.stripe_analytics import stripe_source
+
+                return stripe_source(
+                    endpoints=[
+                        endpoint,
+                    ],
+                    stripe_secret_key=api_key[0],
+                ).with_resources(endpoint)
+            else:
+                from ingestr.src.stripe_analytics import async_stripe_source
+
+                return async_stripe_source(
+                    endpoints=[
+                        endpoint,
+                    ],
+                    stripe_secret_key=api_key[0],
+                ).with_resources(endpoint)
 
         raise ValueError(
             f"Resource '{table}' is not supported for stripe source yet, if you are interested in it please create a GitHub issue at https://github.com/bruin-data/ingestr"

{ingestr-0.13.56 → ingestr-0.13.57}/ingestr/src/stripe_analytics/__init__.py

@@ -7,7 +7,12 @@ import stripe
 from dlt.sources import DltResource
 from pendulum import DateTime
 
-from .helpers import pagination, transform_date
+from .helpers import (
+    async_parallel_pagination,
+    pagination,
+    parallel_pagination,
+    transform_date,
+)
 
 
 @dlt.source(max_table_nesting=0)
@@ -50,6 +55,86 @@ def stripe_source(
         )(endpoint)
 
 
+@dlt.source(max_table_nesting=0)
+def parallel_stripe_source(
+    endpoints: Tuple[str, ...],
+    stripe_secret_key: str = dlt.secrets.value,
+    start_date: Optional[DateTime] = None,
+    end_date: Optional[DateTime] = None,
+    max_workers: int = 12,
+) -> Iterable[DltResource]:
+    """
+    Retrieves data from the Stripe API for the specified endpoints using parallel pagination.
+
+    This source divides the date range across multiple workers to fetch data in parallel,
+    which can significantly speed up data retrieval for large date ranges.
+
+    Args:
+        endpoints (Tuple[str, ...]): A tuple of endpoint names to retrieve data from.
+        stripe_secret_key (str): The API access token for authentication. Defaults to the value in the `dlt.secrets` object.
+        start_date (Optional[DateTime]): An optional start date to limit the data retrieved. Format: datetime(YYYY, MM, DD). Required for parallel processing.
+        end_date (Optional[DateTime]): An optional end date to limit the data retrieved. Format: datetime(YYYY, MM, DD). Required for parallel processing.
+        max_workers (int): Maximum number of worker threads for parallel fetching. Defaults to 4.
+
+    Returns:
+        Iterable[DltResource]: Resources with data that was created during the period greater than or equal to 'start_date' and less than 'end_date'.
+    """
+    stripe.api_key = stripe_secret_key
+    stripe.api_version = "2022-11-15"
+
+    def parallel_stripe_resource(
+        endpoint: str,
+    ) -> Generator[Dict[Any, Any], Any, None]:
+        yield from parallel_pagination(endpoint, start_date, end_date, max_workers)
+
+    for endpoint in endpoints:
+        yield dlt.resource(
+            parallel_stripe_resource,
+            name=endpoint,
+            write_disposition="replace",
+        )(endpoint)
+
+
+@dlt.source(max_table_nesting=0)
+def async_stripe_source(
+    endpoints: Tuple[str, ...],
+    stripe_secret_key: str = dlt.secrets.value,
+    start_date: Optional[DateTime] = None,
+    end_date: Optional[DateTime] = None,
+    max_workers: int = 40,
+    rate_limit_delay: float = 0.03,
+) -> Iterable[DltResource]:
+    """
+    ULTRA-FAST async Stripe source optimized for maximum speed and throughput.
+
+    WARNING: Returns data in RANDOM ORDER for maximum performance.
+    Uses aggressive concurrency and minimal delays to maximize API throughput.
+
+    Args:
+        endpoints (Tuple[str, ...]): A tuple of endpoint names to retrieve data from.
+        stripe_secret_key (str): The API access token for authentication. Defaults to the value in the `dlt.secrets` object.
+        start_date (Optional[DateTime]): An optional start date to limit the data retrieved. Format: datetime(YYYY, MM, DD). Defaults to 2010-01-01.
+        end_date (Optional[DateTime]): An optional end date to limit the data retrieved. Format: datetime(YYYY, MM, DD). Defaults to today.
+        max_workers (int): Maximum number of concurrent async tasks. Defaults to 40 for maximum speed.
+        rate_limit_delay (float): Minimal delay between requests. Defaults to 0.03 seconds.
+
+    Returns:
+        Iterable[DltResource]: Resources with data in RANDOM ORDER (optimized for speed).
+    """
+    stripe.api_key = stripe_secret_key
+    stripe.api_version = "2022-11-15"
+
+    async def async_stripe_resource(endpoint: str):
+        yield async_parallel_pagination(endpoint, max_workers, rate_limit_delay)
+
+    for endpoint in endpoints:
+        yield dlt.resource(
+            async_stripe_resource,
+            name=endpoint,
+            write_disposition="replace",
+        )(endpoint)
+
+
 @dlt.source
 def incremental_stripe_source(
     endpoints: Tuple[str, ...],