ingestr 0.13.55__tar.gz → 0.13.57__tar.gz

{ingestr-0.13.55 → ingestr-0.13.57}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ingestr
-Version: 0.13.55
+Version: 0.13.57
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -27,7 +27,6 @@ Requires-Dist: asynch==0.2.4
 Requires-Dist: attrs==25.1.0
 Requires-Dist: backoff==2.2.1
 Requires-Dist: bcrypt==4.3.0
-Requires-Dist: beautifulsoup4==4.13.3
 Requires-Dist: boto3==1.37.1
 Requires-Dist: botocore==1.37.1
 Requires-Dist: cachetools==5.5.2
@@ -156,7 +155,6 @@ Requires-Dist: python-quickbooks==0.9.2
 Requires-Dist: pytz==2025.1
 Requires-Dist: pyyaml==6.0.2
 Requires-Dist: rauth==0.7.3
-Requires-Dist: redshift-connector==2.1.5
 Requires-Dist: requests-file==2.1.0
 Requires-Dist: requests-oauthlib==1.3.1
 Requires-Dist: requests-toolbelt==1.0.0
@@ -168,7 +166,6 @@ Requires-Dist: rsa==4.9
 Requires-Dist: rudder-sdk-python==2.1.4
 Requires-Dist: s3fs==2025.3.2
 Requires-Dist: s3transfer==0.11.3
-Requires-Dist: scramp==1.4.5
 Requires-Dist: semver==3.0.4
 Requires-Dist: setuptools==75.8.2
 Requires-Dist: shellingham==1.5.4
@@ -180,7 +177,6 @@ Requires-Dist: smmap==5.0.2
 Requires-Dist: snowflake-connector-python==3.14.0
 Requires-Dist: snowflake-sqlalchemy==1.6.1
 Requires-Dist: sortedcontainers==2.4.0
-Requires-Dist: soupsieve==2.6
 Requires-Dist: sqlalchemy-bigquery==1.12.1
 Requires-Dist: sqlalchemy-cratedb==0.42.0.dev2
 Requires-Dist: sqlalchemy-hana==2.0.0

{ingestr-0.13.55 → ingestr-0.13.57}/docs/.vitepress/config.mjs

@@ -123,6 +123,7 @@ export default defineConfig({
               },
               { text: "Frankfurter", link: "/supported-sources/frankfurter.md" },
               { text: "Freshdesk", link: "/supported-sources/freshdesk.md" },
+              { text: "Trustpilot", link: "/supported-sources/trustpilot.md" },
               { text: "Google Cloud Storage (GCS)", link: "/supported-sources/gcs.md" },
               { text: "Google Analytics", link: "/supported-sources/google_analytics.md" },
               { text: "Google Ads", link: "/supported-sources/google-ads.md" },

{ingestr-0.13.55 → ingestr-0.13.57}/docs/supported-sources/kinesis.md

@@ -31,9 +31,13 @@ ingestr ingest --source-uri 'kinesis://?aws_access_key_id=id_123&aws_secret_acce
  --dest-table 'dest.results'
 ```
 
-When using Kinesis as a source, specify the `stream name` you want to read from as the `--source-table` parameter. For example, if you want to read from a Kinesis stream named "customer_events", you would use `--source-table 'customer_events'`.
+When using Kinesis as a source, specify the [StreamName] you want to read from as the `--source-table` parameter. For example, if you want to read from a Kinesis stream named "customer_events", you would use `--source-table 'customer_events'`.
+You can also use a full Kinesis [StreamARN] to address the stream in [ARN] format, like `arn:aws:kinesis:eu-central-1:842404475894:stream/customer_events`.
 
 ### Initial Load Configuration
 By default, ingestr reads from the beginning of the Kinesis stream. To start reading from a specific time, use the `interval_start` parameter.
 
 
+[ARN]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html
+[StreamARN]: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamARN
+[StreamName]: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamName

{ingestr-0.13.55 → ingestr-0.13.57}/docs/supported-sources/s3.md

@@ -134,3 +134,49 @@ The `<file-glob-pattern>` in the `--source-table` argument allows for flexible f
 | `bucket/myFolder/**/*.jsonl`                | Retrieves all JSONL files recursively from the `myFolder` directory and its subdirectories in `s3://bucket`. |
 | `bucket/myFolder/mySubFolder/users.parquet` | Retrieves the specific `users.parquet` file from the `myFolder/mySubFolder/` path in `s3://bucket`.        |
 | `bucket/employees.jsonl`                    | Retrieves the `employees.jsonl` file located at the root level of the `s3://bucket`.                       |
+
+
+### Working with compressed files
+
+`ingestr` automatically detects and handles gzipped files in your S3 bucket. You can load data from compressed files with the `.gz` extension without any additional configuration.
+
+For example, to load data from a gzipped CSV file:
+
+```sh
+ingestr ingest \
+    --source-uri 's3://?access_key_id=AKC3YOW7E&secret_access_key=XCtkpL5B' \
+    --source-table 'my_bucket/logs/event-data.csv.gz' \
+    --dest-uri duckdb:///compressed_data.duckdb \
+    --dest-table 'logs.events'
+```
+
+You can also use glob patterns to load multiple compressed files:
+
+```sh
+ingestr ingest \
+    --source-uri 's3://?access_key_id=AKC3YOW7E&secret_access_key=XCtkpL5B' \
+    --source-table 'my_bucket/logs/**/*.csv.gz' \
+    --dest-uri duckdb:///compressed_data.duckdb \
+    --dest-table 'logs.events'
+```
+
+### File type hinting
+
+If your files are properly encoded but lack the correct file extension (CSV, JSONL, or Parquet), you can provide a file type hint to inform `ingestr` about the format of the files. This is done by appending a fragment identifier (`#format`) to the end of the path in your `--source-table` parameter.
+
+For example, if you have JSONL-formatted log files stored in S3 with a non-standard extension:
+
+```
+--source-table "my_bucket/logs/event-data#jsonl"
+```
+
+This tells `ingestr` to process the files as JSONL, regardless of their actual extension.
+
+Supported format hints include:
+- `#csv` - For comma-separated values files
+- `#jsonl` - For line-delimited JSON files
+- `#parquet` - For Parquet format files
+
+::: tip
+File type hinting works with `gzip` compressed files as well.
+:::

{ingestr-0.13.55 → ingestr-0.13.57}/docs/supported-sources/stripe.md

@@ -30,11 +30,68 @@ ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'ch
 
 The result of this command will be a table in the `stripe.duckdb` database with JSON columns.
 
+## Table Name Structure
+
+Stripe source supports different loading modes that can be specified using the table name structure:
+
+- `<endpoint>` - Standard async loading (default)
+- `<endpoint>:sync` - Full loading with synchronous processing
+- `<endpoint>:sync:incremental` - Incremental loading mode with synchronous processing
+
+### Loading Modes and Trade-offs
+
+#### Standard Async Loading (Default)
+**Format**: `<endpoint>` (e.g., `charges`, `subscriptions`)
+
+- **Use case**: Full data loading from all time periods
+- **Performance**: Loads data in parallel using async processing
+- **Data completeness**: Captures all historical data and updates
+- **Speed**: Slower due to comprehensive data retrieval
+- **Best for**: You want to have all updated data in your database
+
+**Example**:
+```sh
+ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'subscriptions' --dest-uri duckdb:///stripe.duckdb --dest-table 'dest.subscriptions'
+```
+
+#### Sync Loading
+**Format**: `<endpoint>:sync` (e.g., `charges:sync`, `subscriptions:sync`)
+
+- **Use case**: Full data loading from all time periods
+- **Performance**: Loads data in parallel using sync processing
+- **Data completeness**: Captures all historical data and updates
+- **Speed**: Slower due to comprehensive data retrieval, faster if you have less data
+
+#### Incremental Loading
+**Format**: `<endpoint>:sync:incremental` (e.g., `charges:sync:incremental`, `events:sync:incremental`)
+
+- **Use case**: Loading data within specific time windows
+- **Performance**: Fast, processes only data within the specified interval
+- **Data completeness**: Limited to the specified time window, does not track updates from past dates
+- **Speed**: Faster due to filtered data retrieval
+- **Processing**: Runs in synchronous mode only
+- **Best for**: Quick loads, you don't care about the updates to past data
+
+**Example**:
+```sh
+ingestr ingest --source-uri 'stripe://?api_key=sk_test_12345' --source-table 'charges:sync:incremental' --dest-uri duckdb:///stripe.duckdb --dest-table 'dest.charges' --interval-start '2024-01-01' --interval-end '2024-01-31'
+```
+
+### Choosing the Right Approach
+
+| Approach | Speed | Data Completeness | Use Case |
+|----------|--------|------------------|----------|
+| **Standard Async** | Faster for larger data, slower for smaller data | Complete historical data | Initial loads, full historical analysis |
+| **Sync** | Slow for larger data, faster for smaller data | Complete historical data | Initial loads, full historical analysis |
+| **Incremental** | Fastest | Time-window specific | Regular updates, recent data analysis |
+
 ## Tables
 
 Stripe source allows ingesting the following sources into separate tables:
 
-### Regular Endpoints
+### All Endpoints
+
+All endpoints support the standard async loading mode. The following endpoints are available:
 
 - `account`: Contains information about a Stripe account, including balances, payouts, and account settings.
 - `apple_pay_domain`: Represents Apple Pay domains registered with Stripe for processing Apple Pay payments.
@@ -67,12 +124,7 @@ Stripe source allows ingesting the following sources into separate tables:
 - `top_up`: Records top-ups made to Stripe accounts.
 - `transfer`: Records transfers between Stripe accounts.
 - `webhook_endpoint`: Contains webhook endpoint configurations for receiving event notifications.
-
-### Incremental Endpoints
-
-The following endpoints support incremental loading, meaning only new or updated records will be fetched:
-
-- `application_fee`: Records fees collected by platforms (incremental).
+- `application_fee`: Records fees collected by platforms.
 - `balance_transaction`: Records transactions that affect the Stripe account balance, such as charges, refunds, and payouts.
 - `charge`: Returns a list of charges.
 - `credit_note`: Contains credit note information for refunds and adjustments.
@@ -80,12 +132,14 @@ The following endpoints support incremental loading, meaning only new or updated
 - `invoice`: Represents invoices sent to customers, detailing line items, amounts, and payment status.
 - `invoice_item`: Contains individual line items that can be added to invoices.
 - `invoice_line_item`: Represents line items within invoices.
-- `setup_attempt`: Records attempts to set up payment methods (also available as incremental).
 
 Use these as `--source-table` parameter in the `ingestr ingest` command.
 
+> [!TIP]
+> For time-sensitive data analysis or regular updates, use incremental loading (`:incremental`) with `--interval-start` and `--interval-end` parameters for faster processing. For comprehensive historical analysis, use standard async loading without any suffix.
+
 > [!WARNING]
-> Stripe does not support incremental loading for many endpoints in its APIs, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.
+> Incremental loading filters data based on the specified time window and does not track updates to records created outside that window. Use standard async loading if you need to capture all historical updates.
 
 > [!NOTE]
 > For backward compatibility, non-underscored versions of table names (e.g., `checkoutsession`, `paymentintent`, `subscriptionitem`) are still supported but will be deprecated in future versions. Please use the underscored versions (e.g., `checkout_session`, `payment_intent`, `subscription_item`) for new integrations.

ingestr-0.13.57/docs/supported-sources/trustpilot.md

@@ -0,0 +1,35 @@
+# Trustpilot
+
+[Trustpilot](https://www.trustpilot.com/) provides a platform for collecting and
+sharing customer reviews.
+
+ingestr supports Trustpilot as a source.
+
+## URI format
+
+The URI format for Trustpilot is:
+
+```
+trustpilot://<business_unit_id>?api_key=<api_key>
+```
+
+URI parameters:
+- `api_key`: Your Trustpilot API key.
+- `business_unit_id`: Identifier of the business unit whose reviews you want to fetch.
+
+## Example usage
+
+Assuming your `business_unit_id` is `123` and your API key is `key_abc`, you can ingest reviews into DuckDB using:
+
+```bash
+ingestr ingest --source-uri 'trustpilot://123?api_key=key_abc' --source-table 'reviews' --dest-uri duckdb:///trustpilot.duckdb --dest-table 'dest.reviews'
+```
+
+## Tables
+
+Currently the Trustpilot source exposes the following table:
+
+| Name    | Description                                 |
+| ------- | ------------------------------------------- |
+| reviews | Customer reviews for the specified business |
+

{ingestr-0.13.55 → ingestr-0.13.57}/ingestr/src/blob.py

@@ -6,6 +6,10 @@ BucketName: TypeAlias = str
 FileGlob: TypeAlias = str
 
 
+class UnsupportedEndpointError(Exception):
+    pass
+
+
 def parse_uri(uri: ParseResult, table: str) -> Tuple[BucketName, FileGlob]:
     """
     parse the URI of a blob storage and
@@ -50,3 +54,23 @@ def parse_uri(uri: ParseResult, table: str) -> Tuple[BucketName, FileGlob]:
         return "", parts[0]
 
     return parts[0], parts[1]
+
+
+def parse_endpoint(path: str) -> str:
+    """
+    Parse the endpoint kind from the URI.
+
+    kind is a file format. one of [csv, jsonl, parquet]
+    """
+    file_extension = path.split(".")[-1]
+    if file_extension == "gz":
+        file_extension = path.split(".")[-2]
+    if file_extension == "csv":
+        endpoint = "read_csv"
+    elif file_extension == "jsonl":
+        endpoint = "read_jsonl"
+    elif file_extension == "parquet":
+        endpoint = "read_parquet"
+    else:
+        raise UnsupportedEndpointError(f"Unsupported file format: {file_extension}")
+    return endpoint

ingestr-0.13.57/ingestr/src/buildinfo.py

@@ -0,0 +1 @@
+version = "v0.13.57"

{ingestr-0.13.55 → ingestr-0.13.57}/ingestr/src/factory.py

@@ -64,6 +64,7 @@ from ingestr.src.sources import (
     SqlSource,
     StripeAnalyticsSource,
     TikTokSource,
+    TrustpilotSource,
     ZendeskSource,
 )
 
@@ -165,6 +166,7 @@ class SourceDestinationFactory:
         "pipedrive": PipedriveSource,
         "frankfurter": FrankfurterSource,
         "freshdesk": FreshdeskSource,
+        "trustpilot": TrustpilotSource,
         "phantombuster": PhantombusterSource,
         "elasticsearch": ElasticsearchSource,
         "attio": AttioSource,

{ingestr-0.13.55 → ingestr-0.13.57}/ingestr/src/kinesis/__init__.py

@@ -9,7 +9,7 @@ from dlt.common.time import ensure_pendulum_datetime
 from dlt.common.typing import StrStr, TAnyDateTime, TDataItem
 from dlt.common.utils import digest128
 
-from .helpers import get_shard_iterator, max_sequence_by_shard
+from .helpers import get_shard_iterator, get_stream_address, max_sequence_by_shard
 
 
 @dlt.resource(
@@ -42,7 +42,7 @@ def kinesis_stream(
         initial_at_timestamp (TAnyDateTime): An initial timestamp used to generate AT_TIMESTAMP or LATEST iterator when timestamp value is 0
         max_number_of_messages (int): Maximum number of messages to read in one run. Actual read may exceed that number by up to chunk_size. Defaults to None (no limit).
         milliseconds_behind_latest (int): The number of milliseconds behind the top of the shard to stop reading messages, defaults to 1000.
-        parse_json (bool): If True, assumes that messages are json strings, parses them and returns instead of `data` (otherwise). Defaults to False.
+        parse_json (bool): If True, assumes that messages are json strings, parses them and returns instead of `data` (otherwise). Defaults to True.
         chunk_size (int): The number of records to fetch at once. Defaults to 1000.
     Yields:
             Iterable[TDataItem]: Messages. Contain Kinesis envelope in `kinesis` and bytes data in `data` (if `parse_json` disabled)
@@ -65,7 +65,7 @@ def kinesis_stream(
     # so next time we request shards at AT_TIMESTAMP that is now
     resource_state["initial_at_timestamp"] = pendulum.now("UTC").subtract(seconds=1)
 
-    shards_list = kinesis_client.list_shards(StreamName=stream_name)
+    shards_list = kinesis_client.list_shards(**get_stream_address(stream_name))
     shards: List[StrStr] = shards_list["Shards"]
     while next_token := shards_list.get("NextToken"):
         shards_list = kinesis_client.list_shards(NextToken=next_token)

{ingestr-0.13.55 → ingestr-0.13.57}/ingestr/src/kinesis/helpers.py

@@ -2,7 +2,7 @@ from typing import Any, Sequence, Tuple
 
 import dlt
 from dlt.common import pendulum
-from dlt.common.typing import DictStrAny, StrAny, StrStr
+from dlt.common.typing import DictStrAny, DictStrStr, StrAny, StrStr
 
 
 def get_shard_iterator(
@@ -40,7 +40,7 @@ def get_shard_iterator(
         )
 
     shard_iterator: StrStr = kinesis_client.get_shard_iterator(
-        StreamName=stream_name, ShardId=shard_id, **iterator_params
+        **get_stream_address(stream_name), ShardId=shard_id, **iterator_params
     )
     return shard_iterator["ShardIterator"], iterator_params
 
@@ -63,3 +63,20 @@ def max_sequence_by_shard(values: Sequence[StrStr]) -> StrStr:
     # we compare message sequence at shard_id
     last_value[shard_id] = max(item["seq_no"], last_value.get(shard_id, ""))
     return last_value
+
+
+def get_stream_address(stream_name: str) -> DictStrStr:
+    """
+    Return address of stream, either as StreamName or StreamARN, when applicable.
+
+    Examples:
+    - customer_events
+    - arn:aws:kinesis:eu-central-1:842404475894:stream/customer_events
+
+    https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamName
+    https://docs.aws.amazon.com/kinesis/latest/APIReference/API_StreamDescription.html#Streams-Type-StreamDescription-StreamARN
+    """
+    if stream_name.startswith("arn:"):
+        return {"StreamARN": stream_name}
+    else:
+        return {"StreamName": stream_name}

{ingestr-0.13.55 → ingestr-0.13.57}/ingestr/src/sources.py

@@ -677,24 +677,33 @@ class StripeAnalyticsSource:
 
         table = table.lower()
 
-        from ingestr.src.stripe_analytics.settings import (
-            ENDPOINTS,
-            INCREMENTAL_ENDPOINTS,
-        )
+        from ingestr.src.stripe_analytics.settings import ENDPOINTS
 
-        if table in ENDPOINTS:
-            endpoint = ENDPOINTS[table]
-            from ingestr.src.stripe_analytics import stripe_source
+        endpoint = None
+        incremental = False
+        sync = False
 
-            return stripe_source(
-                endpoints=[
-                    endpoint,
-                ],
-                stripe_secret_key=api_key[0],
-            ).with_resources(endpoint)
+        table_fields = table.split(":")
+        if len(table_fields) == 1:
+            endpoint = table_fields[0]
+        elif len(table_fields) == 2:
+            endpoint = table_fields[0]
+            sync = table_fields[1] == "sync"
+        elif len(table_fields) == 3:
+            endpoint = table_fields[0]
+            sync = table_fields[1] == "sync"
+            incremental = table_fields[2] == "incremental"
+        else:
+            raise ValueError(
+                "Invalid Stripe table format. Expected: stripe:<endpoint> or stripe:<endpoint>:<sync> or stripe:<endpoint>:<sync>:<incremental>"
+            )
+
+        if incremental and not sync:
+            raise ValueError(
+                "incremental loads must be used with sync loading"
+            )
 
-        elif table in INCREMENTAL_ENDPOINTS:
-            endpoint = INCREMENTAL_ENDPOINTS[table]
+        if incremental:
             from ingestr.src.stripe_analytics import incremental_stripe_source
 
             def nullable_date(date_str: Optional[str]):
@@ -702,6 +711,7 @@ class StripeAnalyticsSource:
                     return ensure_pendulum_datetime(date_str)
                 return None
 
+            endpoint = ENDPOINTS[endpoint]
             return incremental_stripe_source(
                 endpoints=[
                     endpoint,
@@ -710,6 +720,26 @@ class StripeAnalyticsSource:
                 initial_start_date=nullable_date(kwargs.get("interval_start", None)),
                 end_date=nullable_date(kwargs.get("interval_end", None)),
             ).with_resources(endpoint)
+        else:
+            endpoint = ENDPOINTS[endpoint]
+            if sync:
+                from ingestr.src.stripe_analytics import stripe_source
+
+                return stripe_source(
+                    endpoints=[
+                        endpoint,
+                    ],
+                    stripe_secret_key=api_key[0],
+                ).with_resources(endpoint)
+            else:
+                from ingestr.src.stripe_analytics import async_stripe_source
+
+                return async_stripe_source(
+                    endpoints=[
+                        endpoint,
+                    ],
+                    stripe_secret_key=api_key[0],
+                ).with_resources(endpoint)
 
         raise ValueError(
             f"Resource '{table}' is not supported for stripe source yet, if you are interested in it please create a GitHub issue at https://github.com/bruin-data/ingestr"
@@ -1362,17 +1392,25 @@ class S3Source:
             secret=secret_access_key[0],
         )
 
-        file_extension = path_to_file.split(".")[-1]
-        if file_extension == "csv":
-            endpoint = "read_csv"
-        elif file_extension == "jsonl":
-            endpoint = "read_jsonl"
-        elif file_extension == "parquet":
-            endpoint = "read_parquet"
+        endpoint: Optional[str] = None
+        if "#" in table:
+            _, endpoint = table.split("#")
+            if endpoint not in ["csv", "jsonl", "parquet"]:
+                raise ValueError(
+                    "S3 Source only supports specific formats files: csv, jsonl, parquet"
+                )
+            endpoint = f"read_{endpoint}"
         else:
-            raise ValueError(
-                "S3 Source only supports specific formats files: csv, jsonl, parquet"
-            )
+            try:
+                endpoint = blob.parse_endpoint(path_to_file)
+            except blob.UnsupportedEndpointError:
+                raise ValueError(
+                    "S3 Source only supports specific formats files: csv, jsonl, parquet"
+                )
+            except Exception as e:
+                raise ValueError(
+                    f"Failed to parse endpoint from path: {path_to_file}"
+                ) from e
 
         from ingestr.src.filesystem import readers
 
@@ -1844,17 +1882,16 @@ class GCSSource:
             token=credentials,
         )
 
-        file_extension = path_to_file.split(".")[-1]
-        if file_extension == "csv":
-            endpoint = "read_csv"
-        elif file_extension == "jsonl":
-            endpoint = "read_jsonl"
-        elif file_extension == "parquet":
-            endpoint = "read_parquet"
-        else:
+        try:
+            endpoint = blob.parse_endpoint(path_to_file)
+        except blob.UnsupportedEndpointError:
             raise ValueError(
-                "GCS Source only supports specific formats files: csv, jsonl, parquet"
+                "S3 Source only supports specific formats files: csv, jsonl, parquet"
             )
+        except Exception as e:
+            raise ValueError(
+                f"Failed to parse endpoint from path: {path_to_file}"
+            ) from e
 
         from ingestr.src.filesystem import readers
 
@@ -2392,6 +2429,47 @@ class FreshdeskSource:
         ).with_resources(table)
 
 
+class TrustpilotSource:
+    # trustpilot://<business_unit_id>?api_key=<api_key>
+    def handles_incrementality(self) -> bool:
+        return True
+
+    def dlt_source(self, uri: str, table: str, **kwargs):
+        parsed_uri = urlparse(uri)
+        business_unit_id = parsed_uri.netloc
+        params = parse_qs(parsed_uri.query)
+
+        if not business_unit_id:
+            raise MissingValueError("business_unit_id", "Trustpilot")
+
+        api_key = params.get("api_key")
+        if api_key is None:
+            raise MissingValueError("api_key", "Trustpilot")
+
+        start_date = kwargs.get("interval_start")
+        if start_date is None:
+            start_date = ensure_pendulum_datetime("2000-01-01").in_tz("UTC").isoformat()
+        else:
+            start_date = ensure_pendulum_datetime(start_date).in_tz("UTC").isoformat()
+
+        end_date = kwargs.get("interval_end")
+
+        if end_date is not None:
+            end_date = ensure_pendulum_datetime(end_date).in_tz("UTC").isoformat()
+
+        if table not in ["reviews"]:
+            raise UnsupportedResourceError(table, "Trustpilot")
+
+        from ingestr.src.trustpilot import trustpilot_source
+
+        return trustpilot_source(
+            business_unit_id=business_unit_id,
+            api_key=api_key[0],
+            start_date=start_date,
+            end_date=end_date,
+        ).with_resources(table)
+
+
 class PhantombusterSource:
     def handles_incrementality(self) -> bool:
         return True
@@ -2622,18 +2700,15 @@ class SFTPSource:
         else:
             file_glob = f"/{table}"
 
-        file_extension = table.split(".")[-1].lower()
-        endpoint: str
-        if file_extension == "csv":
-            endpoint = "read_csv"
-        elif file_extension == "jsonl":
-            endpoint = "read_jsonl"
-        elif file_extension == "parquet":
-            endpoint = "read_parquet"
-        else:
+        try:
+            endpoint = blob.parse_endpoint(table)
+        except blob.UnsupportedEndpointError:
             raise ValueError(
-                "FTPServer Source only supports specific file formats: csv, jsonl, parquet."
+                "SFTP Source only supports specific formats files: csv, jsonl, parquet"
             )
+        except Exception as e:
+            raise ValueError(f"Failed to parse endpoint from path: {table}") from e
+
         from ingestr.src.filesystem import readers
 
         dlt_source_resource = readers(bucket_url, fs, file_glob)