ingestr 0.7.6__tar.gz → 0.7.8__tar.gz

{ingestr-0.7.6 → ingestr-0.7.8}/.gitignore

@@ -13,4 +13,5 @@ pipeline_data
 dist
 docs/.vitepress/dist
 docs/.vitepress/cache
-node_modules
+node_modules
+*.duckdb

{ingestr-0.7.6 → ingestr-0.7.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: ingestr
-Version: 0.7.6
+Version: 0.7.8
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -14,17 +14,20 @@ Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Database
 Requires-Python: >=3.9
+Requires-Dist: confluent-kafka>=2.3.0
 Requires-Dist: cx-oracle==8.3.0
 Requires-Dist: databricks-sql-connector==2.9.3
 Requires-Dist: dlt==0.5.1
 Requires-Dist: duckdb-engine==0.11.5
 Requires-Dist: duckdb==0.10.2
+Requires-Dist: facebook-business==20.0.0
 Requires-Dist: google-api-python-client==2.130.0
 Requires-Dist: google-cloud-bigquery-storage==2.24.0
 Requires-Dist: mysql-connector-python==9.0.0
 Requires-Dist: pendulum==3.0.0
 Requires-Dist: psycopg2-binary==2.9.9
 Requires-Dist: py-machineid==0.5.1
+Requires-Dist: pyairtable==2.3.3
 Requires-Dist: pymongo==4.6.3
 Requires-Dist: pymysql==1.1.0
 Requires-Dist: pyodbc==5.1.0
@@ -55,7 +58,7 @@ Description-Content-Type: text/markdown
   </a>
 </div>
 
------
+---
 
 Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
 
@@ -65,8 +68,8 @@ Ingestr is a command-line application that allows you to ingest data from any so
 
 ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
 
-
 ## Installation
+
 ```
 pip install ingestr
 ```
@@ -84,15 +87,17 @@ ingestr ingest \
 That's it.
 
 This command will:
+
 - get the table `public.some_data` from the Postgres instance.
 - upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
 
 ## Documentation
+
 You can see the full documentation [here](https://bruin-data.github.io/ingestr/getting-started/quickstart.html).
 
 ## Community
-Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
 
+Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
 
 ## Supported Sources & Destinations
 
@@ -173,10 +178,20 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
     <tr>
         <td colspan="3" style='text-align:center;'><strong>Platforms</strong></td>
     </tr>
+    <tr>
+        <td>Airtable</td>
+        <td>✅</td>
+        <td>-</td>
+    </tr>
     <tr>
         <td>Chess.com</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Facebook Ads</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Gorgias</td>
@@ -192,6 +207,11 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
         <td>HubSpot</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Klaviyo</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Notion</td>
@@ -202,6 +222,11 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
         <td>Shopify</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Slack</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Stripe</td>
@@ -213,4 +238,5 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
 More to come soon!
 
 ## Acknowledgements
-This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.
+
+This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.

{ingestr-0.7.6 → ingestr-0.7.8}/README.md

@@ -10,7 +10,7 @@
   </a>
 </div>
 
------
+---
 
 Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
 
@@ -20,8 +20,8 @@ Ingestr is a command-line application that allows you to ingest data from any so
 
 ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
 
-
 ## Installation
+
 ```
 pip install ingestr
 ```
@@ -39,15 +39,17 @@ ingestr ingest \
 That's it.
 
 This command will:
+
 - get the table `public.some_data` from the Postgres instance.
 - upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
 
 ## Documentation
+
 You can see the full documentation [here](https://bruin-data.github.io/ingestr/getting-started/quickstart.html).
 
 ## Community
-Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
 
+Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
 
 ## Supported Sources & Destinations
 
@@ -128,10 +130,20 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
     <tr>
         <td colspan="3" style='text-align:center;'><strong>Platforms</strong></td>
     </tr>
+    <tr>
+        <td>Airtable</td>
+        <td>✅</td>
+        <td>-</td>
+    </tr>
     <tr>
         <td>Chess.com</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Facebook Ads</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Gorgias</td>
@@ -147,6 +159,11 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
         <td>HubSpot</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Klaviyo</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Notion</td>
@@ -157,6 +174,11 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
         <td>Shopify</td>
         <td>✅</td>
         <td>-</td>
+    </tr>
+     <tr>
+        <td>Slack</td>
+        <td>✅</td>
+        <td>-</td>
     </tr>
     <tr>
         <td>Stripe</td>
@@ -168,4 +190,5 @@ Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shar
 More to come soon!
 
 ## Acknowledgements
-This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.
+
+This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.

{ingestr-0.7.6 → ingestr-0.7.8}/docs/.vitepress/config.mjs

@@ -64,6 +64,7 @@ export default defineConfig({
                 text: "Google BigQuery",
                 link: "/supported-sources/bigquery.md",
               },
+              { text: "Kafka", link: "/supported-sources/kafka.md" },
               { text: "Local CSV Files", link: "/supported-sources/csv.md" },
               {
                 text: "Microsoft SQL Server",
@@ -83,12 +84,19 @@ export default defineConfig({
             text: "Platforms",
             collapsed: false,
             items: [
+              { text: "Airtable", link: "/supported-sources/airtable.md" },
               { text: "Chess.com", link: "/supported-sources/chess.md" },
+              {
+                text: "Facebook Ads",
+                link: "/supported-sources/facebook-ads.md",
+              },
               { text: "Google Sheets", link: "/supported-sources/gsheets.md" },
               { text: "Gorgias", link: "/supported-sources/gorgias.md" },
               { text: "HubSpot", link: "/supported-sources/hubspot.md" },
+              { text: "Klaviyo", link: "/supported-sources/klaviyo.md" },
               { text: "Notion", link: "/supported-sources/notion.md" },
               { text: "Shopify", link: "/supported-sources/shopify.md" },
+              { text: "Slack", link: "/supported-sources/slack.md" },
               { text: "Stripe", link: "/supported-sources/stripe.md" },
             ],
           },

ingestr-0.7.8/docs/supported-sources/airtable.md

@@ -0,0 +1,37 @@
+# Airtable
+
+[Airtable](https://airtable.com/) is a cloud-based platform that combines spreadsheet and database functionalities, designed for data management and collaboration.
+
+ingestr supports Airtable as a source.
+
+## URI Format
+
+The URI format for Airtable is as follows:
+
+```plaintext
+ airtable://?access_token=<access_token>&base_id=<base_id>
+```
+
+URI parameters:
+
+- `base_id`: A unique identifier for an Airtable base.
+- `access_token`: A personal access token for authentication with the Airtable API.
+
+The URI is used to connect to the Airtable API for extracting data. More details on setting up Airtable integrations can be found [here](https://airtable.com/developers/web/api).
+
+## Setting up a Airtable Integration
+
+Airtable requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/airtable#setup-guide).
+
+Once you complete the guide, you should have an Access Token and Base Id. Let's say your Access Token is `patr123.abc` and Base Id is `appXYZ`, here's a sample command that will copy the data from Airtable into a duckdb database:
+
+```sh
+ingestr ingest --source-uri 'airtable://?base_id=appXYc&access_token=patr123.abc' --source-table 'employee' --dest-uri 'duckdb:///airtable.duckdb' --dest-table 'des.employee'
+```
+
+The result of this command will be an `employee` table containing data from the `employee` source in the `Airtable.duckdb` database.
+
+The `source-table` can include multiple table names that share the `same base_id` (e.g.--source-table 'employee,users') but this will merge all the data from the specified tables into a single destination table.
+
+> [!CAUTION]
+> Airtable does not support incremental loading, which means every time you run the command, the entire table will be copied from Airtable to the destination. This can be slow for large tables.

ingestr-0.7.8/docs/supported-sources/facebook-ads.md

@@ -0,0 +1,51 @@
+# Facebook Ads
+
+Facebook Ads is the advertising platform that helps users to create targeted ads on Facebook, Instagram and Messenger.
+
+ingestr supports Facebook Ads as a source.
+
+## URI Format
+
+The URI format for Facebook Ads is as follows:
+
+```plaintext
+facebookads://?access_token=<access_token>&account_id=<account_id>
+```
+
+URI parameters:
+
+- `access_token` is associated with Business Facebook App.
+- `account_id` is associated with Ad manager.
+
+Both are used for authentication with Facebook Ads API.
+
+The URI is used to connect to Facebook Ads API for extracting data.
+
+## Setting up a Facebook Ads Integration
+
+Facebook Ads requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/facebook_ads#setup-guide).
+
+Once you complete the guide, you should have an Access_Token and Account ID . Let's say your access_token is `abcdef` and account_id is `1234` , here's a sample command that will copy the data from Facebook Ads into a duckdb database:
+
+```sh
+ingestr ingest \
+  --source-uri 'facebookads://?access_token=easdyh&account_id=1234' \
+  --source-table 'campaigns' \
+  --dest-uri 'duckdb:///facebook.duckdb' \
+  --dest-table 'dest.campaigns'
+```
+
+The result of this command will be a table in the `facebook.duckdb` database.
+
+## Available Tables
+
+Facebook Ads source allows ingesting the following sources into separate tables:
+
+- `campaigns`: Retrieves all DEFAULT_CAMPAIGN_FIELDS.
+- `ad_sets`: Retrieves all DEFAULT_ADSET_FIELDS.
+- `leads`: Retrieves all DEFAULT_LEAD_FIELDS.
+- `ads_creatives`: Retrieves all DEFAULT_ADCREATIVE_FIELDS.
+- `ads`: Retrieves all DEFAULT_ADS_FIELDS.
+- `facebook_insights`: Retrieves all DEFAULT_INSIGHTS_FIELDS.
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.

ingestr-0.7.8/docs/supported-sources/kafka.md

@@ -0,0 +1,38 @@
+# Apache Kafka
+[Apache Kafka](https://kafka.apache.org/) is a distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
+
+ingestr supports Apache Kafka as a source.
+
+## URI Format
+The URI format for Apache Kafka is as follows:
+
+```plaintext
+kafka://?bootstrap_servers=localhost:9092&group_id=test_group&security_protocol=SASL_SSL&sasl_mechanisms=PLAIN&sasl_username=example_username&sasl_password=example_secret&batch_size=1000&batch_timeout=3
+```
+
+URI parameters:
+- `bootstrap_servers`: The Kafka server(s) to connect to, typically in the form of a host and port (e.g., `localhost:9092`).
+- `group_id`: The consumer group ID used for identifying the client when consuming messages.
+- `security_protocol`: The protocol used to communicate with brokers (e.g., `SASL_SSL` for secure communication).
+- `sasl_mechanisms`: The SASL mechanism to be used for authentication (e.g., `PLAIN`).
+- `sasl_username`: The username for SASL authentication.
+- `sasl_password`: The password for SASL authentication.
+- `batch_size`: The number of messages to fetch in a single batch, defaults to 3000.
+- `batch_timeout`: The maximum time to wait for messages, defaults to 3 seconds.
+
+The URI is used to connect to the Kafka brokers for ingesting messages.
+
+### Group ID
+The group ID is used to identify the consumer group that reads messages from a topic. Kafka uses the group ID to manage consumer offsets and assign partitions to consumers, which means that the group ID is the key to reading messages from the correct partition and position in the topic.
+
+Once you have your Kafka server, credentials, and group ID set up, here's a sample command to ingest messages from a Kafka topic into a duckdb database:
+
+```sh
+ingestr ingest \
+    --source-uri 'kafka://?bootstrap_servers=localhost:9092' \
+    --source-table 'my-topic' \
+    --dest-uri duckdb:///kafka.duckdb \
+    --dest-table 'kafka.my_topic'
+```
+
+The result of this command will be a table in the `kafka.duckdb` database with JSON columns.

ingestr-0.7.8/docs/supported-sources/klaviyo.md

@@ -0,0 +1,64 @@
+# Klaviyo
+
+[Klaviyo](https://www.klaviyo.com/) is a marketing automation platform that helps businesses build and manage smarter digital relationships with their customers by connecting through personalized email and enhancing customer loyality.
+
+ingestr supports Klaviyo as a source.
+
+## URI Format
+
+The URI format for Klaviyo is as follows:
+
+```plaintext
+klaviyo://?api_key=<api-key>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Klaviyo API.
+
+The URI is used to connect to the Klaviyo API for extracting data.
+
+```bash
+ingestr ingest --source-table 'events' --source-uri 'klaviyo://?api_key=pk_test' --dest-uri duckdb:///klaviyo.duckdb --interval-start 2022-01-01 --dest-table 'klaviyo.events' --extract-parallelism 20
+```
+
+This command fethes all the events that are created/updated since 2022-01-01 and writes them to `klaviyo.events` table on DuckDB, using 20 parallel threads to improve performance and efficiently handle large data .
+
+## Available Tables
+
+Klaviyo source allows ingesting the following sources into separate tables:
+
+[events](https://developers.klaviyo.com/en/reference/events_api_overview): Retrieves all events in an account where each event represents an action taken by a profile such as a password reset or a product order.
+
+[profiles](https://developers.klaviyo.com/en/reference/profiles_api_overview): Retrieves all profiles in an account where each profile includes details like organization, job title, email and other attributes.
+
+[campaigns](https://developers.klaviyo.com/en/reference/campaigns_api_overview): Retrieves all campaigns in an account where each campaign is a targeted message sent to a specific audience.
+
+[metrics](https://developers.klaviyo.com/en/reference/metrics_api_overview): Retrieves all metrics in an account where each metric represents a category of events or actions a person can take.
+
+[tags](https://developers.klaviyo.com/en/reference/get_tags): Retrieves all tags in an account.
+
+[coupons](https://developers.klaviyo.com/en/reference/get_coupons): Retrieves all coupons in an account.
+
+[catalog-variants](https://developers.klaviyo.com/en/reference/get_catalog_variants): Retrieves all variants in an account.
+
+[catalog-categories](https://developers.klaviyo.com/en/reference/get_catalog_categories): Retrieves all catalog categories in an account.
+
+[catalog-items](https://developers.klaviyo.com/en/reference/get_catalog_items): Retrieves all catalog items in an account.
+
+[flows](https://developers.klaviyo.com/en/reference/get_flows): Retrieves all flows in an account where flow is a sequence of automated actions that is triggered when a person performs a specific action.
+
+[lists](https://developers.klaviyo.com/en/reference/get_lists): Retrieves all lists in an account.
+
+[images](https://developers.klaviyo.com/en/reference/get_images): Retrieves all images in an account..
+
+[segments](https://developers.klaviyo.com/en/reference/get_segments): Retrieves all segments in an account where segment is a dynamic list that contains profiles meeting a certain set of conditions.
+
+[forms](https://developers.klaviyo.com/en/reference/get_forms): Retrieves all forms in an account.
+
+[templates](https://developers.klaviyo.com/en/reference/get_templates): Retrieves all templates in an account.
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.
+
+> [!WARNING]
+> Klaviyo does not support incremental loading for many endpoints in its APIs, which means ingestr will load endpoints incrementally if they support it, and do a full-refresh if not.

ingestr-0.7.8/docs/supported-sources/slack.md

@@ -0,0 +1,42 @@
+# Slack
+
+[Slack](https://www.Slack.com/) is a messaging platform for teams and organizations where they can collaborate, share ideas and information.
+
+ingestr supports Slack as a source.
+
+## URI Format
+
+The URI format for Slack is as follows:
+
+```plaintext
+slack://?api_key=<api-key-here>
+```
+
+URI parameters:
+
+- `api_key`: The API key used for authentication with the Slack API.
+
+The URI is used to connect to the Slack API for extracting data.
+
+## Setting up a Slack Integration
+
+Slack requires a few steps to set up an integration, please follow the guide dltHub [has built here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/Slack#setup-guide).
+
+Once you complete the guide, you should have an API key with the necessary permissions as mentioned in the guide. Let's say your API key is axb-test-564. Here's a sample command that will copy the data from Slack into a DuckDB database:
+
+```sh
+ingestr ingest --source-uri 'slack://?api_key=axb-test-564' --source-table 'channels' --dest-uri duckdb:///slack.duckdb --dest-table 'dest.channels'
+```
+
+The result of this command will be a table in the `slack.duckdb` database.
+
+## Available Tables
+
+Slack source allows ingesting the following sources into separate tables:
+
+- `channels`: Retrieves information about all the channels.
+- `users`: Retrieves information about all the users.
+- `messages:chan1,chan2`: Retrieves messages from specified channels, where chan1 and chan2 represent user-defined channels (e.g: general, memes). Multiple channels can be listed.
+- `access_logs`: Retrieves all the access logs.
+
+Use these as `--source-table` parameter in the `ingestr ingest` command.

{ingestr-0.7.6 → ingestr-0.7.8}/ingestr/main.py

@@ -1,7 +1,7 @@
 import hashlib
+import tempfile
 from datetime import datetime
 from enum import Enum
-import tempfile
 from typing import Optional
 
 import dlt
@@ -244,6 +244,13 @@ def ingest(
             envvar="PIPELINES_DIR",
         ),
     ] = None,  # type: ignore
+    extract_parallelism: Annotated[
+        Optional[int],
+        typer.Option(
+            help="The number of parallel jobs to run for extracting data from the source, only applicable for certain sources",
+            envvar="EXTRACT_PARALLELISM",
+        ),
+    ] = 5,  # type: ignore
 ):
     track(
         "command_triggered",
@@ -253,6 +260,8 @@ def ingest(
     )
 
     dlt.config["data_writer.file_max_items"] = loader_file_size
+    dlt.config["extract.workers"] = extract_parallelism
+    dlt.config["extract.max_parallel_items"] = extract_parallelism
     if schema_naming != SchemaNaming.default:
         dlt.config["schema.naming"] = schema_naming.value
 
@@ -413,6 +422,7 @@ def ingest(
         # remove the pipelines_dir folder if it was created by ingestr
         if is_pipelines_dir_temp:
             import shutil
+
             shutil.rmtree(pipelines_dir)
 
         print(

ingestr-0.7.8/ingestr/src/.gitignore

@@ -0,0 +1,10 @@
+# ignore secrets, virtual environments and typical python compilation artifacts
+secrets.toml
+# ignore basic python artifacts
+.env
+**/__pycache__/
+**/*.py[cod]
+**/*$py.class
+# ignore duckdb
+*.duckdb
+*.wal

ingestr-0.7.8/ingestr/src/airtable/__init__.py

@@ -0,0 +1,69 @@
+"""Source that loads tables form Airtable.
+Supports whitelisting of tables or loading of all tables from a specified base.
+"""
+
+from typing import Any, Dict, Iterable, Iterator, List, Optional
+
+import dlt
+import pyairtable
+from dlt.sources import DltResource
+
+
+@dlt.source
+def airtable_source(
+    base_id: str = dlt.config.value,
+    table_names: Optional[List[str]] = dlt.config.value,
+    access_token: str = dlt.secrets.value,
+) -> Iterable[DltResource]:
+    """
+    Represents tables for a single Airtable base.
+    Args:
+        base_id (str): The id of the base. Obtain it e.g. from the URL in your webbrowser.
+            It starts with "app". See https://support.airtable.com/docs/finding-airtable-ids
+        table_names (Optional[List[str]]): A list of table IDs or table names to load.
+            Unless specified otherwise, all tables in the schema are loaded.
+            Names are freely user-defined. IDs start with "tbl". See https://support.airtable.com/docs/finding-airtable-ids
+        access_token (str): The personal access token.
+            See https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens#personal-access-tokens-basic-actions
+    """
+    api = pyairtable.Api(access_token)
+    all_tables_url = api.build_url(f"meta/bases/{base_id}/tables")
+    tables = api.request(method="GET", url=all_tables_url).get("tables")
+    for t in tables:
+        if table_names:
+            if t.get("id") in table_names or t.get("name") in table_names:
+                yield airtable_resource(api, base_id, t)
+        else:
+            yield airtable_resource(api, base_id, t)
+
+
+def airtable_resource(
+    api: pyairtable.Api,
+    base_id: str,
+    table: Dict[str, Any],
+) -> DltResource:
+    """
+    Represents a single airtable.
+    Args:
+        api (pyairtable.Api): The API connection object
+        base_id (str): The id of the base. Obtain it e.g. from the URL in your webbrowser.
+            It starts with "app". See https://support.airtable.com/docs/finding-airtable-ids
+        table (Dict[str, Any]): Metadata about an airtable, does not contain the actual records
+    """
+    primary_key_id = table["primaryFieldId"]
+    primary_key_field = [
+        field for field in table["fields"] if field["id"] == primary_key_id
+    ][0]
+    table_name: str = table["name"]
+    primary_key: List[str] = [primary_key_field["name"]]
+    air_table = api.table(base_id, table["id"])
+
+    # Table.iterate() supports rich customization options, such as chunk size, fields, cell format, timezone, locale, and view
+    air_table_generator: Iterator[List[Any]] = air_table.iterate()
+
+    return dlt.resource(
+        air_table_generator,
+        name=table_name,
+        primary_key=primary_key,
+        write_disposition="replace",
+    )