ingestr 0.13.59__tar.gz → 0.13.61__tar.gz

{ingestr-0.13.59 → ingestr-0.13.61}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.59
+Version: 0.13.61
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -47,6 +47,7 @@ Requires-Dist: databricks-sqlalchemy==1.0.2
 Requires-Dist: dataclasses-json==0.6.7
 Requires-Dist: decorator==5.2.1
 Requires-Dist: deprecation==2.1.0
+Requires-Dist: dlt-cratedb==0.0.1
 Requires-Dist: dlt==1.10.0
 Requires-Dist: dnspython==2.7.0
 Requires-Dist: duckdb-engine==0.17.0
@@ -305,7 +306,7 @@ Pull requests are welcome. However, please open an issue first to discuss what y
     <tr>
         <td>CrateDB</td>
         <td>✅</td>
-        <td>❌</td>
+        <td>✅</td>
     </tr>
     <tr>
         <td>Databricks</td>

{ingestr-0.13.59 → ingestr-0.13.61}/README.md

@@ -90,7 +90,7 @@ Pull requests are welcome. However, please open an issue first to discuss what y
     <tr>
         <td>CrateDB</td>
         <td>✅</td>
-        <td>❌</td>
+        <td>✅</td>
     </tr>
     <tr>
         <td>Databricks</td>

{ingestr-0.13.59 → ingestr-0.13.61}/docs/supported-sources/cratedb.md

@@ -4,7 +4,7 @@
 massive amounts of data in near real-time, even with complex queries. It is
 PostgreSQL-compatible, and based on Lucene.
 
-ingestr supports CrateDB as a source database.
+ingestr supports CrateDB as a source and destination database.
 
 ## Source
 
@@ -56,6 +56,61 @@ duckdb cratedb.duckdb 'SELECT * FROM dest.summits LIMIT 5'
 
 <img alt="CrateDB_img" src="../media/cratedb-source.png" />
 
+## Destination
+
+For connecting to CrateDB as a database destination, ingestr uses the
+[dlt cratedb adapter], which is based on the [dlt postgres adapter],
+in turn using the [psycopg2] package.
+
+### URI format
+
+The URI format for CrateDB as a destination is as follows:
+```plaintext
+cratedb://<username>:<password>@<host>:<port>?sslmode=<sslmode>
+```
+> [!INFO]
+> When connecting to CrateDB on localhost, use:
+> ```plaintext
+> cratedb://crate:@localhost:5432?sslmode=disable
+> ```
+>
+> When connecting to [CrateDB Cloud], the URI looks like this:
+> ```plaintext
+> cratedb://admin:<PASSWORD>@<CLUSTERNAME>.eks1.eu-west-1.aws.cratedb.net:5432?sslmode=require
+> ```
+
+### URI parameters
+- `username` (required): The username is required to authenticate with the CrateDB server.
+- `password` (required): The password is required to authenticate the provided username.
+- `host` (required): The hostname or IP address of the CrateDB server where the database is hosted.
+- `port` (required): The TCP port number used by the CrateDB server. Mostly `5432`.
+- `sslmode` (optional): Set to one of `disable`, `allow`, `prefer`, `require`, `verify-ca`,
+  or `verify-full`, see [PostgreSQL SSL Mode Descriptions].
+
+### Example
+
+This is an example command that will import a CSV file into CrateDB,
+then display the content from CrateDB.
+
+```shell
+wget -O input.csv https://github.com/bruin-data/ingestr/raw/refs/heads/main/ingestr/testdata/create_replace.csv
+```
+```shell
+ingestr ingest \
+   --source-uri 'csv://input.csv' \
+   --source-table 'sample' \
+   --dest-uri 'cratedb://crate:@localhost:5432/?sslmode=disable' \
+   --dest-table 'doc.sample'
+```
+```shell
+uvx crash -c 'SELECT * FROM doc.sample'
+```
+
+<img alt="CrateDB_img" src="../media/cratedb-destination.png" />
+
+> [!WARNING]
+> CrateDB supports the `replace` incremental materialization strategy, but
+> currently does not support the `delete+insert`, `merge`, or `scd2` strategies.
 
 ## Appendix
 
@@ -75,6 +130,10 @@ or share relevant issue reports that help us improve interoperability. Thanks!
 
 [CrateDB]: https://github.com/crate/crate
 [CrateDB Cloud]: https://console.cratedb.cloud/
+[dlt cratedb adapter]: https://github.com/dlt-hub/dlt/pull/2733
+[dlt postgres adapter]: https://github.com/dlt-hub/dlt/tree/devel/dlt/destinations/impl/postgres
+[PostgreSQL SSL Mode Descriptions]: https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS
+[psycopg2]: https://pypi.org/project/psycopg2-binary/
 [sqlalchemy-cratedb]: https://pypi.org/project/sqlalchemy-cratedb/
 [Support for ingestr/CrateDB]: https://github.com/crate/crate-clients-tools/issues/86
 ["tool: dlt/ingestr"]: https://github.com/crate/crate/issues?q=state%3Aopen%20label%3A%22tool%3A%20dlt%2Fingestr%22

ingestr-0.13.61/docs/supported-sources/gcs.md

@@ -0,0 +1,169 @@
+# Google Cloud Storage
+
+[Google Cloud Storage](https://cloud.google.com/storage?hl=en) is an online file storage web service for storing and accessing data on Google Cloud Platform infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities. It is an Infrastructure as a Service (IaaS), comparable to Amazon S3.
+
+`ingestr` supports Google Cloud Storage as both a data source and destination.
+
+## URI format
+
+The URI format for Google Cloud Storage is as follows:
+
+```plaintext
+gs://?credentials_path=/path/to/service-account.json
+```
+
+URI parameters:
+
+- `credentials_path`: path to file containing your Google Cloud [Service Account](https://cloud.google.com/iam/docs/service-account-overview)
+- `credentials_base64`: base64-encoded service account JSON (alternative to credentials_path)
+- `layout`: Layout template (optional, destination only)
+
+The `--source-table` must be in the format:
+```
+{bucket name}/{file glob}
+```
+
+## Setting up a GCS Integration
+
+To use Google Cloud Storage source in `ingestr`, you will need:
+* A Google Cloud Project.
+* A Service Account with at least [roles/storage.objectUser](https://cloud.google.com/storage/docs/access-control/iam-roles) IAM permission for reading, or [roles/storage.objectAdmin](https://cloud.google.com/storage/docs/access-control/iam-roles) for writing to GCS.
+* A Service Account key file for the corresponding service account.
+
+For more information on how to create a Service Account or its keys, see [Create service accounts](https://cloud.google.com/iam/docs/service-accounts-create) and [Create or delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete) on Google Cloud docs.
+
+## Example: Loading data from GCS
+
+Let's assume that:
+* Service account key is available in the current directory, under the filename `service_account.json`. 
+* The bucket you want to load data from is called `my-org-bucket`
+* The source file is available at `data/latest/dump.csv`
+* The data needs to be saved in a DuckDB database called `local.db`
+* The destination table name will be `public.latest_dump`
+
+You can run the following command line to achieve this:
+
+```sh
+ingestr ingest \
+    --source-uri "gs://?credentials_path=$PWD/service_account.json" \
+    --source-table "my-org-bucket/data/latest/dump.csv" \
+    --dest-uri "duckdb:///local.db" \
+    --dest-table "public.latest_dump"
+```
+
+## Example: Uploading data to GCS
+
+For this example, we'll assume that:
+* `records.db` is a DuckDB database.
+* It has a table called `public.users`.
+* The service account key is available in the current directory.
+
+The following command demonstrates how to copy data from a local DuckDB database to GCS:
+```sh
+ingestr ingest \
+    --source-uri 'duckdb:///records.db' \
+    --source-table 'public.users' \
+    --dest-uri "gs://?credentials_path=$PWD/service_account.json" \
+    --dest-table 'my-org-bucket/records'
+```
+
+This will result in a file structure like the following:
+```
+my-org-bucket/
+└── records
+    ├── _dlt_loads
+    ├── _dlt_pipeline_state
+    ├── _dlt_version
+    └── users
+        └── <load_id>.<file_id>.parquet
+```
+
+The value of `load_id` and `file_id` is determined at runtime. The default layout creates a folder with the same table name as the source and places the data inside a parquet file. This layout is configurable using the `layout` parameter.
+
+For example, if you would like to create a parquet file with the same name as the source table (as opposed to a folder) you can set `layout` to `{table_name}.{ext}` in the command line above:
+
+```sh
+ingestr ingest \
+    --source-uri 'duckdb:///records.db' \
+    --source-table 'public.users' \
+    --dest-uri "gs://?layout={table_name}.{ext}&credentials_path=$PWD/service_account.json" \
+    --dest-table 'my-org-bucket/records'
+```
+
+Result:
+```
+my-org-bucket/
+└── records
+    ├── _dlt_loads
+    ├── _dlt_pipeline_state
+    ├── _dlt_version
+    └── users.parquet
+```
+
+List of available Layout variables is available [here](https://dlthub.com/docs/dlt-ecosystem/destinations/filesystem#available-layout-placeholders)
+
+## Supported File Formats
+`gs` source only supports loading files in the following formats:
+* `csv`: Comma Separated Values 
+* `parquet`: [Apache Parquet](https://parquet.apache.org/) storage format.
+* `jsonl`: Line delimited JSON. see [https://jsonlines.org/](https://jsonlines.org/)
+
+::: info NOTE
+When writing to GCS, only `parquet` is supported.
+:::
+## File Pattern
+`ingestr` supports [glob](https://en.wikipedia.org/wiki/Glob_(programming)) like pattern matching for `gs` source.
+This allows for a powerful pattern matching mechanism that allows you to specify multiple files in a single `--source-table`.
+
+Below are some examples of path patterns, each path pattern is glob you can specify after the bucket name:
+
+- `**/*.csv`: Retrieves all the CSV files, regardless of how deep they are within the folder structure.
+- `*.csv`: Retrieves all the CSV files from the first level of a folder.
+- `myFolder/**/*.jsonl`: Retrieves all the JSONL files from anywhere under `myFolder`.
+- `myFolder/mySubFolder/users.parquet`: Retrieves the `users.parquet` file from `mySubFolder`.
+- `employees.jsonl`: Retrieves the `employees.jsonl` file from the root level of the bucket.
+
+### Working with compressed files
+
+`ingestr` automatically detects and handles gzipped files in your GCS bucket. You can load data from compressed files with the `.gz` extension without any additional configuration.
+
+For example, to load data from a gzipped CSV file:
+
+```sh
+ingestr ingest \
+    --source-uri "gs://?credentials_path=$PWD/service_account.json" \
+    --source-table "my-org-bucket/logs/event-data.csv.gz" \
+    --dest-uri "duckdb:///compressed_data.duckdb" \
+    --dest-table "logs.events"
+```
+
+You can also use glob patterns to load multiple compressed files:
+
+```sh
+ingestr ingest \
+    --source-uri "gs://?credentials_path=$PWD/service_account.json" \
+    --source-table "my-org-bucket/logs/**/*.csv.gz" \
+    --dest-uri "duckdb:///compressed_data.duckdb" \
+    --dest-table "logs.events"
+```
+
+### File type hinting
+
+If your files are properly encoded but lack the correct file extension (CSV, JSONL, or Parquet), you can provide a file type hint to inform `ingestr` about the format of the files. This is done by appending a fragment identifier (`#format`) to the end of the path in your `--source-table` parameter.
+
+For example, if you have JSONL-formatted log files stored in GCS with a non-standard extension:
+
+```
+--source-table "my-org-bucket/logs/event-data#jsonl"
+```
+
+This tells `ingestr` to process the files as JSONL, regardless of their actual extension.
+
+Supported format hints include:
+- `#csv` - For comma-separated values files
+- `#jsonl` - For line-delimited JSON files
+- `#parquet` - For Parquet format files
+
+::: tip
+File type hinting works with `gzip` compressed files as well.
+:::

{ingestr-0.13.59 → ingestr-0.13.61}/ingestr/main.py

@@ -543,6 +543,7 @@ def ingest(
             sql_reflection_level=sql_reflection_level.value,
             sql_limit=sql_limit,
             sql_exclude_columns=sql_exclude_columns,
+            extract_parallelism=extract_parallelism,
         )
 
         resource.for_each(dlt_source, lambda x: x.add_map(cast_set_to_list))

ingestr-0.13.61/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.61"

{ingestr-0.13.59 → ingestr-0.13.61}/ingestr/src/destinations.py

@@ -1,3 +1,4 @@
+import abc
 import base64
 import csv
 import json
@@ -9,6 +10,7 @@ from urllib.parse import parse_qs, quote, urlparse
 import dlt
 import dlt.destinations.impl.filesystem.filesystem
 from dlt.common.configuration.specs import AwsCredentials
+from dlt.common.storages.configuration import FileSystemCredentials
 from dlt.destinations.impl.clickhouse.configuration import (
     ClickHouseCredentials,
 )
@@ -111,6 +113,14 @@ class BigQueryDestination:
         pass
 
 
+class CrateDBDestination(GenericSqlDestination):
+    def dlt_dest(self, uri: str, **kwargs):
+        uri = uri.replace("cratedb://", "postgres://")
+        import dlt_cratedb.impl.cratedb.factory
+
+        return dlt_cratedb.impl.cratedb.factory.cratedb(credentials=uri, **kwargs)
+
+
 class PostgresDestination(GenericSqlDestination):
     def dlt_dest(self, uri: str, **kwargs):
         return dlt.destinations.postgres(credentials=uri, **kwargs)
@@ -386,43 +396,62 @@ class ClickhouseDestination:
         pass
 
 
-class S3FSClient(dlt.destinations.impl.filesystem.filesystem.FilesystemClient):
+class BlobFSClient(dlt.destinations.impl.filesystem.filesystem.FilesystemClient):
     @property
     def dataset_path(self):
         # override to remove dataset path
         return self.bucket_path
 
 
-class S3FS(dlt.destinations.filesystem):
+class BlobFS(dlt.destinations.filesystem):
     @property
     def client_class(self):
-        return S3FSClient
+        return BlobFSClient
 
 
-class S3Destination:
+class SqliteDestination(GenericSqlDestination):
     def dlt_dest(self, uri: str, **kwargs):
-        parsed_uri = urlparse(uri)
-        params = parse_qs(parsed_uri.query)
+        return dlt.destinations.sqlalchemy(credentials=uri)
 
-        access_key_id = params.get("access_key_id", [None])[0]
-        if access_key_id is None:
-            raise MissingValueError("access_key_id", "S3")
+    def dlt_run_params(self, uri: str, table: str, **kwargs):
+        return {
+            # https://dlthub.com/docs/dlt-ecosystem/destinations/sqlalchemy#dataset-files
+            "dataset_name": "main",
+            "table_name": table,
+        }
 
-        secret_access_key = params.get("secret_access_key", [None])[0]
-        if secret_access_key is None:
-            raise MissingValueError("secret_access_key", "S3")
 
-        endpoint_url = params.get("endpoint_url", [None])[0]
-        if endpoint_url is not None:
-            parsed_endpoint = urlparse(endpoint_url)
-            if not parsed_endpoint.scheme or not parsed_endpoint.netloc:
-                raise ValueError("Invalid endpoint_url. Must be a valid URL.")
+class MySqlDestination(GenericSqlDestination):
+    def dlt_dest(self, uri: str, **kwargs):
+        return dlt.destinations.sqlalchemy(credentials=uri)
 
-        creds = AwsCredentials(
-            aws_access_key_id=access_key_id,
-            aws_secret_access_key=secret_access_key,
-            endpoint_url=endpoint_url,
-        )
+    def dlt_run_params(self, uri: str, table: str, **kwargs):
+        parsed = urlparse(uri)
+        database = parsed.path.lstrip("/")
+        if not database:
+            raise ValueError("You need to specify a database")
+        return {
+            "dataset_name": database,
+            "table_name": table,
+        }
+
+
+class BlobStorageDestination(abc.ABC):
+    @abc.abstractmethod
+    def credentials(self, params: dict) -> FileSystemCredentials:
+        """Build credentials for the blob storage destination."""
+        pass
+
+    @property
+    @abc.abstractmethod
+    def protocol(self) -> str:
+        """The protocol used for the blob storage destination."""
+        pass
+
+    def dlt_dest(self, uri: str, **kwargs):
+        parsed_uri = urlparse(uri)
+        params = parse_qs(parsed_uri.query)
+        creds = self.credentials(params)
 
         dest_table = kwargs["dest_table"]
 
@@ -442,7 +471,7 @@ class S3Destination:
         base_path = "/".join(table_parts[:-1])
 
         opts = {
-            "bucket_url": f"s3://{base_path}",
+            "bucket_url": f"{self.protocol}://{base_path}",
             "credentials": creds,
             # supresses dlt warnings about dataset name normalization.
             # we don't use dataset names in S3 so it's fine to disable this.
@@ -452,7 +481,7 @@ class S3Destination:
         if layout is not None:
             opts["layout"] = layout
 
-        return S3FS(**opts)  # type: ignore
+        return BlobFS(**opts)  # type: ignore
 
     def validate_table(self, table: str):
         table = table.strip("/ ")
@@ -470,28 +499,56 @@ class S3Destination:
         pass
 
 
-class SqliteDestination(GenericSqlDestination):
-    def dlt_dest(self, uri: str, **kwargs):
-        return dlt.destinations.sqlalchemy(credentials=uri)
+class S3Destination(BlobStorageDestination):
+    @property
+    def protocol(self) -> str:
+        return "s3"
 
-    def dlt_run_params(self, uri: str, table: str, **kwargs):
-        return {
-            # https://dlthub.com/docs/dlt-ecosystem/destinations/sqlalchemy#dataset-files
-            "dataset_name": "main",
-            "table_name": table,
-        }
+    def credentials(self, params: dict) -> FileSystemCredentials:
+        access_key_id = params.get("access_key_id", [None])[0]
+        if access_key_id is None:
+            raise MissingValueError("access_key_id", "S3")
 
+        secret_access_key = params.get("secret_access_key", [None])[0]
+        if secret_access_key is None:
+            raise MissingValueError("secret_access_key", "S3")
 
-class MySqlDestination(GenericSqlDestination):
-    def dlt_dest(self, uri: str, **kwargs):
-        return dlt.destinations.sqlalchemy(credentials=uri)
+        endpoint_url = params.get("endpoint_url", [None])[0]
+        if endpoint_url is not None:
+            parsed_endpoint = urlparse(endpoint_url)
+            if not parsed_endpoint.scheme or not parsed_endpoint.netloc:
+                raise ValueError("Invalid endpoint_url. Must be a valid URL.")
 
-    def dlt_run_params(self, uri: str, table: str, **kwargs):
-        parsed = urlparse(uri)
-        database = parsed.path.lstrip("/")
-        if not database:
-            raise ValueError("You need to specify a database")
-        return {
-            "dataset_name": database,
-            "table_name": table,
-        }
+        return AwsCredentials(
+            aws_access_key_id=access_key_id,
+            aws_secret_access_key=secret_access_key,
+            endpoint_url=endpoint_url,
+        )
+
+
+class GCSDestination(BlobStorageDestination):
+    @property
+    def protocol(self) -> str:
+        return "gs"
+
+    def credentials(self, params: dict) -> FileSystemCredentials:
+        """Builds GCS credentials from the provided parameters."""
+        credentials_path = params.get("credentials_path")
+        credentials_base64 = params.get("credentials_base64")
+        credentials_available = any(
+            map(
+                lambda x: x is not None,
+                [credentials_path, credentials_base64],
+            )
+        )
+        if credentials_available is False:
+            raise MissingValueError("credentials_path or credentials_base64", "GCS")
+
+        credentials = None
+        if credentials_path:
+            with open(credentials_path[0], "r") as f:
+                credentials = json.load(f)
+        else:
+            credentials = json.loads(base64.b64decode(credentials_base64[0]).decode())  # type: ignore
+
+        return credentials

{ingestr-0.13.59 → ingestr-0.13.61}/ingestr/src/factory.py

@@ -7,9 +7,11 @@ from ingestr.src.destinations import (
     AthenaDestination,
     BigQueryDestination,
     ClickhouseDestination,
+    CrateDBDestination,
     CsvDestination,
     DatabricksDestination,
     DuckDBDestination,
+    GCSDestination,
     MsSQLDestination,
     MySqlDestination,
     PostgresDestination,
@@ -181,6 +183,7 @@ class SourceDestinationFactory:
     }
     destinations: Dict[str, Type[DestinationProtocol]] = {
         "bigquery": BigQueryDestination,
+        "cratedb": CrateDBDestination,
         "databricks": DatabricksDestination,
         "duckdb": DuckDBDestination,
         "mssql": MsSQLDestination,
@@ -197,6 +200,7 @@ class SourceDestinationFactory:
         "clickhouse+native": ClickhouseDestination,
         "clickhouse": ClickhouseDestination,
         "s3": S3Destination,
+        "gs": GCSDestination,
         "sqlite": SqliteDestination,
         "mysql": MySqlDestination,
         "mysql+pymysql": MySqlDestination,

{ingestr-0.13.59 → ingestr-0.13.61}/ingestr/src/isoc_pulse/__init__.py

@@ -73,7 +73,7 @@ def pulse_source(
             "write_disposition": "merge",
             "primary_key": "date",
         },
-        "resources": resources, # type:ignore
+        "resources": resources,  # type:ignore
     }
     res = rest_api_resources(config)
     if metric == "net_loss":

{ingestr-0.13.59 → ingestr-0.13.61}/ingestr/src/sources.py

@@ -737,6 +737,7 @@ class StripeAnalyticsSource:
                         endpoint,
                     ],
                     stripe_secret_key=api_key[0],
+                    max_workers=kwargs.get("extract_parallelism", 4),
                 ).with_resources(endpoint)
 
         raise ValueError(
@@ -1884,7 +1885,7 @@ class GCSSource:
             endpoint = blob.parse_endpoint(path_to_file)
         except blob.UnsupportedEndpointError:
             raise ValueError(
-                "S3 Source only supports specific formats files: csv, jsonl, parquet"
+                "GCS Source only supports specific formats files: csv, jsonl, parquet"
             )
         except Exception as e:
             raise ValueError(

{ingestr-0.13.59 → ingestr-0.13.61}/ingestr/src/stripe_analytics/__init__.py

@@ -10,7 +10,6 @@ from pendulum import DateTime
 from .helpers import (
     async_parallel_pagination,
     pagination,
-    parallel_pagination,
     transform_date,
 )
 
@@ -55,53 +54,13 @@ def stripe_source(
         )(endpoint)
 
 
-@dlt.source(max_table_nesting=0)
-def parallel_stripe_source(
-    endpoints: Tuple[str, ...],
-    stripe_secret_key: str = dlt.secrets.value,
-    start_date: Optional[DateTime] = None,
-    end_date: Optional[DateTime] = None,
-    max_workers: int = 12,
-) -> Iterable[DltResource]:
-    """
-    Retrieves data from the Stripe API for the specified endpoints using parallel pagination.
-
-    This source divides the date range across multiple workers to fetch data in parallel,
-    which can significantly speed up data retrieval for large date ranges.
-
-    Args:
-        endpoints (Tuple[str, ...]): A tuple of endpoint names to retrieve data from.
-        stripe_secret_key (str): The API access token for authentication. Defaults to the value in the `dlt.secrets` object.
-        start_date (Optional[DateTime]): An optional start date to limit the data retrieved. Format: datetime(YYYY, MM, DD). Required for parallel processing.
-        end_date (Optional[DateTime]): An optional end date to limit the data retrieved. Format: datetime(YYYY, MM, DD). Required for parallel processing.
-        max_workers (int): Maximum number of worker threads for parallel fetching. Defaults to 4.
-
-    Returns:
-        Iterable[DltResource]: Resources with data that was created during the period greater than or equal to 'start_date' and less than 'end_date'.
-    """
-    stripe.api_key = stripe_secret_key
-    stripe.api_version = "2022-11-15"
-
-    def parallel_stripe_resource(
-        endpoint: str,
-    ) -> Generator[Dict[Any, Any], Any, None]:
-        yield from parallel_pagination(endpoint, start_date, end_date, max_workers)
-
-    for endpoint in endpoints:
-        yield dlt.resource(
-            parallel_stripe_resource,
-            name=endpoint,
-            write_disposition="replace",
-        )(endpoint)
-
-
 @dlt.source(max_table_nesting=0)
 def async_stripe_source(
     endpoints: Tuple[str, ...],
     stripe_secret_key: str = dlt.secrets.value,
     start_date: Optional[DateTime] = None,
     end_date: Optional[DateTime] = None,
-    max_workers: int = 40,
+    max_workers: int = 4,
     rate_limit_delay: float = 0.03,
 ) -> Iterable[DltResource]:
     """