ingestr 0.0.2__tar.gz → 0.0.4__tar.gz

{ingestr-0.0.2 → ingestr-0.0.4}/.gitignore

@@ -10,4 +10,7 @@ venv
 .pytest_cache
 .mypy_cache
 pipeline_data
-dist
+dist
+docs/.vitepress/dist
+docs/.vitepress/cache
+node_modules

{ingestr-0.0.2 → ingestr-0.0.4}/Makefile

@@ -18,7 +18,7 @@ test: venv
 	. venv/bin/activate; $(MAKE) test-ci
 
 test-specific: venv
-	. venv/bin/activate; pytest -rP -vv --tb=short --cov=ingestr --no-cov-on-fail -k $(test)
+	. venv/bin/activate; pytest -rP -vv --tb=short --cov=ingestr --no-cov-on-fail --capture=no -k $(test)
 
 lint-ci:
 	ruff ingestr --fix && ruff format ingestr

{ingestr-0.0.2 → ingestr-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ingestr
-Version: 0.0.2
+Version: 0.0.4
 Summary: ingestr is a command-line application that ingests data from various sources and stores them in any database.
 Project-URL: Homepage, https://github.com/bruin-data/ingestr
 Project-URL: Issues, https://github.com/bruin-data/ingestr/issues
@@ -16,11 +16,14 @@ Classifier: Topic :: Database
 Requires-Python: >=3.9
 Requires-Dist: databricks-sql-connector==2.9.3
 Requires-Dist: dlt==0.4.3
+Requires-Dist: duckdb-engine==0.11.1
 Requires-Dist: duckdb==0.9.2
+Requires-Dist: google-cloud-bigquery-storage
 Requires-Dist: pendulum==3.0.0
 Requires-Dist: psycopg2==2.9.9
 Requires-Dist: pyodbc==5.1.0
 Requires-Dist: rich==13.7.0
+Requires-Dist: rudder-sdk-python==2.0.2
 Requires-Dist: snowflake-sqlalchemy==1.5.1
 Requires-Dist: sqlalchemy-bigquery==1.9.0
 Requires-Dist: sqlalchemy2-stubs==0.0.2a38
@@ -32,18 +35,19 @@ Description-Content-Type: text/markdown
 <div align="center">
     <img src="./resources/ingestr.svg" width="500" />
     <p>Ingest & copy data from any source to any destination without any code</p>
+    <img src="./resources/demo.gif" width="500" />
 </div>
 
+
 -----
 
 Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
 
-- ✨ copy data from your Postges / Mongo / BigQuery or any other source into any destination
-- ➕ incremental loading
+- ✨ copy data from your database into any destination
+- ➕ incremental loading: `append`, `merge` or `delete+insert`
 - 🐍 single-command installation
-- 💅 Docker image for easy installation & usage
 
-ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the magic.
+ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
 
 
 ## Installation
@@ -67,6 +71,10 @@ This command will:
 - get the table `public.some_data` from the Postgres instance.
 - upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
 
+## Documentation
+You can see the full documentation [here](https://bruindata.github.com/ingestr).
+
+
 ## Supported Sources & Destinations
 
 | Database             | Source | Destination |
@@ -79,4 +87,9 @@ This command will:
 | DuckDB               | ✅      | ✅         |
 | Microsoft SQL Server | ✅      | ✅         |
 | SQLite               | ✅      | ❌         |
-| MySQL                | ✅      | ❌         |
+| MySQL                | ✅      | ❌         |
+
+More to come soon!
+
+## Acknowledgements
+This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.

{ingestr-0.0.2 → ingestr-0.0.4}/README.md

@@ -1,18 +1,19 @@
 <div align="center">
     <img src="./resources/ingestr.svg" width="500" />
     <p>Ingest & copy data from any source to any destination without any code</p>
+    <img src="./resources/demo.gif" width="500" />
 </div>
 
+
 -----
 
 Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
 
-- ✨ copy data from your Postges / Mongo / BigQuery or any other source into any destination
-- ➕ incremental loading
+- ✨ copy data from your database into any destination
+- ➕ incremental loading: `append`, `merge` or `delete+insert`
 - 🐍 single-command installation
-- 💅 Docker image for easy installation & usage
 
-ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the magic.
+ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
 
 
 ## Installation
@@ -36,6 +37,10 @@ This command will:
 - get the table `public.some_data` from the Postgres instance.
 - upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
 
+## Documentation
+You can see the full documentation [here](https://bruindata.github.com/ingestr).
+
+
 ## Supported Sources & Destinations
 
 | Database             | Source | Destination |
@@ -48,4 +53,9 @@ This command will:
 | DuckDB               | ✅      | ✅         |
 | Microsoft SQL Server | ✅      | ✅         |
 | SQLite               | ✅      | ❌         |
-| MySQL                | ✅      | ❌         |
+| MySQL                | ✅      | ❌         |
+
+More to come soon!
+
+## Acknowledgements
+This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.

ingestr-0.0.4/docs/.vitepress/config.mjs

@@ -0,0 +1,49 @@
+import { defineConfig } from "vitepress";
+
+// https://vitepress.dev/reference/site-config
+export default defineConfig({
+  title: "ingestr",
+  description: "Ingest & copy data between any source and any destination",
+  themeConfig: {
+    // https://vitepress.dev/reference/default-theme-config
+    nav: [
+      { text: "Home", link: "/" },
+      { text: "Getting started", link: "/getting-started/quickstart.md" },
+    ],
+
+    sidebar: [
+      {
+        text: "Getting started",
+        items: [
+          { text: "Quickstart", link: "/getting-started/quickstart.md" },
+          { text: "Core Concepts", link: "/getting-started/core-concepts.md" },
+          { text: "Incremental Loading", link: "/getting-started/incremental-loading.md" },
+        ],
+      },
+      {
+        text: "Commands",
+        items: [
+          { text: "ingest", link: "/commands/ingest.md" },
+          { text: "example-uris", link: "/commands/example-uris.md" },
+        ],
+      },
+      {
+        text: "Sources & Destinations",
+        items: [
+          { text: "Overview", link: "/supported-sources/overview.md" },
+          { text: "Postgres", link: "/supported-sources/postgres.md" },
+          { text: "Google BigQuery", link: "/supported-sources/bigquery.md" },
+          { text: "Snowflake", link: "/supported-sources/snowflake.md" },
+          { text: "AWS Redshift", link: "/supported-sources/redshift.md" },
+          { text: "Databricks", link: "/supported-sources/databricks.md" },
+          { text: "DuckDB", link: "/supported-sources/duckdb.md" },
+          { text: "Microsoft SQL Server", link: "/supported-sources/mssql.md" },
+          { text: "SQLite", link: "/supported-sources/sqlite.md" },
+          { text: "MySQL", link: "/supported-sources/mysql.md" },
+        ],
+      },
+    ],
+
+    socialLinks: [{ icon: "github", link: "https://github.com/bruin-data/ingestr" }],
+  },
+});

ingestr-0.0.4/docs/api-examples.md

@@ -0,0 +1,49 @@
+---
+outline: deep
+---
+
+# Runtime API Examples
+
+This page demonstrates usage of some of the runtime APIs provided by VitePress.
+
+The main `useData()` API can be used to access site, theme, and page data for the current page. It works in both `.md` and `.vue` files:
+
+```md
+<script setup>
+import { useData } from 'vitepress'
+
+const { theme, page, frontmatter } = useData()
+</script>
+
+## Results
+
+### Theme Data
+<pre>{{ theme }}</pre>
+
+### Page Data
+<pre>{{ page }}</pre>
+
+### Page Frontmatter
+<pre>{{ frontmatter }}</pre>
+```
+
+<script setup>
+import { useData } from 'vitepress'
+
+const { site, theme, page, frontmatter } = useData()
+</script>
+
+## Results
+
+### Theme Data
+<pre>{{ theme }}</pre>
+
+### Page Data
+<pre>{{ page }}</pre>
+
+### Page Frontmatter
+<pre>{{ frontmatter }}</pre>
+
+## More
+
+Check out the documentation for the [full list of runtime APIs](https://vitepress.dev/reference/runtime-api#usedata).

ingestr-0.0.4/docs/commands/example-uris.md

@@ -0,0 +1,5 @@
+# `ingestr example-uris`
+
+This command is supposed to serve as a guide for the user to understand the various URI formats that are supported by the `ingestr` tool. The command will provide a list of supported sources and destinations, along with the URI format for each of them.
+
+For the detailed documentation, please refer to the [Sources & Destinations](../supported-sources/overview.md) section.

ingestr-0.0.4/docs/commands/ingest.md

@@ -0,0 +1,44 @@
+# `ingestr ingest`
+
+The `ingest` command is a core functionality of the `ingestr` tool, allowing users to transfer data from a source to a destination with optional support for incremental updates.
+
+## Example
+
+The following example demonstrates how to use the `ingest` command to transfer data from a source to a destination.
+
+```bash
+ingestr ingest 
+   --source-uri '<your-source-uri-here>'
+   --source-table '<your-schema>.<your-table>'
+   --dest-uri '<your-destination-uri-here>'
+```
+
+## Required Options
+
+- `--source-uri TEXT`: Specifies the URI of the data source. This parameter is required.
+- `--dest-uri TEXT`: Specifies the URI of the destination where data will be ingested. This parameter is required.
+- `--source-table TEXT`: Defines the source table to fetch data from. This parameter is required.
+
+## Optional Options
+
+- `--dest-table TEXT`: Designates the destination table to save the data. If not specified, defaults to the value of `--source-table`.
+- `--incremental-key TEXT`: Identifies the key used for incremental data strategies. Defaults to `None`.
+- `--incremental-strategy TEXT`: Defines the strategy for incremental updates. Options include `replace`, `append`, `delete+insert`, or `merge`. The default strategy is `replace`.
+- `--interval-start`: Sets the start of the interval for the incremental key. Defaults to `None`.
+- `--interval-end`: Sets the end of the interval for the incremental key. Defaults to `None`.
+- `--primary-key TEXT`: Specifies the primary key for the merge operation. Defaults to `None`.
+
+The `interval-start` and `interval-end` options support various datetime formats, here are some examples:
+- `%Y-%m-%d`: `2023-01-31`
+- `%Y-%m-%dT%H:%M:%S`: `2023-01-31T15:00:00`
+- `%Y-%m-%dT%H:%M:%S%z`: `2023-01-31T15:00:00+00:00`
+- `%Y-%m-%dT%H:%M:%S.%f`: `2023-01-31T15:00:00.000123`
+- `%Y-%m-%dT%H:%M:%S.%f%z`: `2023-01-31T15:00:00.000123+00:00`
+
+> [!INFO]
+> For the details around the incremental key and the various strategies, please refer to the [Incremental Loading](../getting-started/incremental-loading.md) section.
+
+## General Options
+
+- `--help`: Displays the help message and exits the command.
+

ingestr-0.0.4/docs/getting-started/core-concepts.md

@@ -0,0 +1,41 @@
+---
+outline: deep
+---
+
+# Core Concepts
+ingestr has a few simple concepts that you should understand before you start using it.
+
+## Source & Destination URIs
+The source and destination are the two main components of ingestr. The source is the place from where you want to ingest the data, hence the name "source", and the destination is the place where you want to store the data.
+
+The sources and destinations are identified with [URIs](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier). A URI is a simple string that contains the credentials used to connect to the source or destination.
+
+Here's an example URI for a Postgres database:
+```
+postgresql://admin:admin@localhost:8837/web?sslmode=disable
+```
+
+The URI is composed of the following parts:
+- `postgresql`: the name of the database
+- `admin:admin`: the username and password
+- `localhost:8837`: the host and port
+- `web`: the database name
+- `sslmode=disable`: the query parameters
+
+ingestr can connect to any source or destination using this structure across all databases.
+
+> [!NOTE]
+> ingestr uses [dlt](https://github.com/dlt-hub/dlt) & [SQLAlchemy](https://www.sqlalchemy.org/) libraries internally, which means you can get connection URIs by following their documentation as well, they are supposed to work right away in ingestr.
+
+## Source & Destination Tables
+The source and destination tables are the tables from the source and destination databases, respectively. The source table is the table from where you want to ingest the data from, and the destination table is the table where you want to store the data.
+
+ingestr uses the `--source-table` and `--dest-table` flags to specify the source and destination tables, respectively. The `--dest-table` is optional, if you don't specify it, ingestr will use the same table name as the source table.
+
+
+## Incremental Loading
+ingestr supports incremental loading, which means you can choose to append, merge or delete+insert data into the destination table. Incremental loading allows you to ingest only the new rows from the source table into the destination table, which means that you don't have to ingest the entire table every time you run ingestr.
+
+Incremental loading requires various identifiers in the source table to understand what has changed when, so that the new rows can be ingested into the destination table. Read more in the [Incremental Loading](/getting-started/incremental-loading.md) section.
+
+

ingestr-0.0.4/docs/getting-started/incremental-loading.md

@@ -0,0 +1,228 @@
+---
+outline: deep
+---
+
+# Incremental Loading
+ingestr supports incremental loading, which means you can choose to append, merge or delete+insert data into the destination table. Incremental loading allows you to ingest only the new rows from the source table into the destination table, which means that you don't have to ingest the entire table every time you run ingestr.
+
+Before you use incremental loading, you should understand 3 important keys:
+- `primary_key`: the column(s) that uniquely identify a row in the table, if you give a primary key for an ingestion the resulting rows will be deduplicated based on the primary key, a.k.a there will only be one row for each primary key in the destination.
+- `incremental_key`: the column that will be used to determine the new rows, if you give an incremental key for an ingestion the resulting rows will be filtered based on the incremental key, a.k.a only the new rows will be ingested.
+  - A good example of an incremental key is a timestamp column, where you only want to ingest the rows that are newer than the last ingestion, e.g. `created_at` or `updated_at`.
+- `strategy`: the strategy to use for incremental loading, the available strategies are:
+  - `replace`: replace the existing destination table with the source directly, this is the default strategy and the simplest one.
+    - This strategy is not recommended for large tables, as it will replace the entire table and can be slow.
+  - `append`: simply append the new rows to the destination table.
+  - `merge`: merge the new rows with the existing rows in the destination table, insert the new ones and update the existing ones with the new values.
+  - `delete+insert`: delete the existing rows in the destination table that match the incremental key and then insert the new rows.
+
+
+
+## Replace
+Replace is the default strategy, and it simply replaces the entire destination table with the source table.
+
+The following example below will replace the entire `my_schema.some_data` table in BigQuery with the `my_schema.some_data` table in Postgres.
+```bash
+ingestr \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'my_schema.some_data' \
+    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
+```
+
+Here's how the replace strategy works:
+- The source table is downloaded.
+- The source table is uploaded to the destination, replacing the destination table.
+
+
+> [!CAUTION]
+> This strategy will delete the entire destination table and replace it with the source table, use with caution.
+
+## Append
+Append will simply append the new rows from the source table to the destination table. By default, it will append all the rows, in order to use it as an incremental strategy, you should provide an `incremental_key`.
+
+The following example below will append the new rows from the `my_schema.some_data` table in Postgres to the `my_schema.some_data` table in BigQuery, only where there's a new table.
+```bash
+ingestr \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'my_schema.some_data' \
+    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
+    --incremental-strategy append
+    --incremental-key updated_at
+```
+
+### Example
+
+Let's assume you had the following source table:
+| id | name | updated_at |
+|----|------|------------|
+| 1  | John | 2021-01-01 |
+| 2  | Jane | 2021-01-01 |
+
+#### First Ingestion
+The first time you run the command, it will ingest all the rows into the destination table. Here's how your destination looks like now:
+
+| id | name | updated_at |
+|----|------|------------|
+| 1  | John | 2021-01-01 |
+| 2  | Jane | 2021-01-01 |
+
+#### Second Ingestion, no new data
+When there's no new data in the source table, the destination table will remain the same.
+
+#### Third Ingestion, new data
+Let's say John changed his name to Johnny, and Jane's `updated_at` was updated to `2021-01-02`, e.g. your source:
+| id | name   | updated_at |
+|----|--------|------------|
+| 1  | Johnny | 2021-01-02 |
+| 2  | Jane   | 2021-01-01 |
+
+
+When you run the command again, it will only ingest the new rows into the destination table. Here's how your destination looks like now:
+| id | name   | updated_at |
+|----|--------|------------|
+| 1  | John   | 2021-01-01 |
+| 2  | Jane   | 2021-01-01 |
+| 1  | Johnny | 2021-01-02 |
+
+**Notice the last row in the table:** it's the new row that was ingested from the source table.
+
+The behavior is the same if there were new rows in the source table, they would be appended to the destination table if they have `updated_at` that is **later than the latest record** in the destination table.
+
+> [!TIP]
+> The `append` strategy allows you to keep a version history of your data, as it will keep appending the new rows to the destination table. You can use it to build [SCD Type 2](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) tables, for example.
+
+
+## Merge
+Merge will merge the new rows with the existing rows in the destination table, insert the new ones and update the existing ones with the new values. By default, it will merge all the rows, in order to use it as an incremental strategy, you should provide an `incremental_key` as well as a `primary_key` to find the right rows to update.
+
+The following example below will merge the new rows from the `my_schema.some_data` table in Postgres to the `my_schema.some_data` table in BigQuery, only where there's a new table.
+```bash
+ingestr \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'my_schema.some_data' \
+    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
+    --incremental-strategy merge
+    --incremental-key updated_at
+    --primary-key id
+```
+
+Here's how the merge strategy works:
+- If the row with the `primary_key` exists in the destination table, it will be updated with the new values from the source table.
+- If the row with the `primary_key` doesn't exist in the destination table, it will be inserted into the destination table.
+- If the row with the `primary_key` exists in the destination table but not in the source table, it will remain in the destination table.
+- If the row with the `primary_key` doesn't exist in the destination table but exists in the source table, it will be inserted into the destination table.
+
+### Example
+
+Let's assume you had the following source table:
+| id | name | updated_at |
+|----|------|------------|
+| 1  | John | 2021-01-01 |
+| 2  | Jane | 2021-01-01 |
+
+#### First Ingestion
+The first time you run the command, it will ingest all the rows into the destination table. Here's how your destination looks like now:
+
+| id | name | updated_at |
+|----|------|------------|
+| 1  | John | 2021-01-01 |
+| 2  | Jane | 2021-01-01 |
+
+#### Second Ingestion, no new data
+When there's no new data in the source table, the destination table will remain the same.
+
+#### Third Ingestion, new data
+Let's say John changed his name to Johnny, e.g. your source:
+| id | name   | updated_at |
+|----|--------|------------|
+| 1  | Johnny | 2021-01-02 |
+| 2  | Jane   | 2021-01-01 |
+    
+When you run the command again, it will merge the new rows into the destination table. Here's how your destination looks like now:
+| id | name   | updated_at |
+|----|--------|------------|
+| 1  | Johnny | 2021-01-02 |
+| 2  | Jane   | 2021-01-01 |
+
+**Notice the first row in the table:** it's the updated row that was ingested from the source table.
+
+The behavior is the same if there were new rows in the source table, they would be merged into the destination table if they have `updated_at` that is **later than the latest record** in the destination table.
+
+> [!TIP]
+> The `merge` strategy is different from the `append` strategy, as it will update the existing rows in the destination table with the new values from the source table. It's useful when you want to keep the latest version of your data in the destination table.
+
+> [!CAUTION]
+> For the cases where there's a primary key match, the `merge` strategy will **update** the existing rows in the destination table with the new values from the source table. Use with caution, as it can lead to data loss if not used properly, as well as data processing charges if your data warehouse charges for updates.
+
+## Delete+Insert
+Delete+Insert will delete the existing rows in the destination table that match the `incremental_key` and then insert the new rows from the source table. By default, it will delete and insert all the rows, in order to use it as an incremental strategy, you should provide an `incremental_key`.
+
+The following example below will delete the existing rows in the `my_schema.some_data` table in BigQuery that match the `updated_at` and then insert the new rows from the `my_schema.some_data` table in Postgres.
+```bash
+ingestr \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'my_schema.some_data' \
+    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
+    --incremental-strategy delete+insert
+    --incremental-key updated_at
+```
+
+Here's how the delete+insert strategy works:
+- The new rows from the source table will be inserted into a staging table in the destination database.
+- The existing rows in the destination table that match the `incremental_key` will be deleted.
+- The new rows from the staging table will be inserted into the destination table.
+
+A few important notes about the `delete+insert` strategy: 
+- it does not guarantee the order of the rows in the destination table, as it will delete and insert the rows in the destination table.
+- it does not deduplicate the rows in the destination table, as it will delete and insert the rows in the destination table, which means you may have multiple rows with the same `incremental_key` in the destination table.
+
+### Example
+Let's assume you had the following source table:
+| id | name | updated_at |
+|----|------|------------|
+| 1  | John | 2021-01-01 |
+| 2  | Jane | 2021-01-01 |
+
+#### First Ingestion
+The first time you run the command, it will ingest all the rows into the destination table. Here's how your destination looks like now:
+| id | name | updated_at |
+|----|------|------------|
+| 1  | John | 2021-01-01 |
+| 2  | Jane | 2021-01-01 |
+
+#### Second Ingestion, no new data
+Even when there's no new data in the source table, the rows from the source table will be inserted into a staging table in the destination database, and then the existing rows in the destination table that match the `incremental_key` will be deleted, and then the new rows from the staging table will be inserted into the destination table. The destination table will remain the same for the case of this example.
+> [!CAUTION]
+> If you had rows in the destination table that does not exist in the source table, they will be deleted from the destination table.
+
+#### Third Ingestion, new data
+Let's say John changed his name to Johnny, e.g. your source:
+| id | name   | updated_at |
+|----|--------|------------|
+| 1  | Johnny | 2021-01-02 |
+| 2  | Jane   | 2021-01-01 |
+
+When you run the command again, it will delete the existing rows in the destination table that match the `incremental_key` and then insert the new rows from the source table. Here's how your destination looks like now:
+| id | name   | updated_at |
+|----|--------|------------|
+| 1  | Johnny | 2021-01-02 |
+| 2  | Jane   | 2021-01-01 |
+
+**Notice the first row in the table:** it's the updated row that was ingested from the source table.
+
+The behavior is the same if there were new rows in the source table, they would be deleted and inserted into the destination table if they have `updated_at` that is **later than the latest record** in the destination table.
+
+> [!TIP]
+> The `delete+insert` strategy is useful when you want to keep the destination table clean, as it will delete the existing rows in the destination table that match the `incremental_key` and then insert the new rows from the source table. `delete+insert` strategy also allows you to do backfills on the data, e.g. going back to a past date and ingesting the data again.
+
+
+## Conclusion
+Incremental loading is a powerful feature that allows you to ingest only the new rows from the source table into the destination table. It's useful when you want to keep the destination table up-to-date with the source table, as well as when you want to keep a version history of your data in the destination table. However, there are a few things to keep in mind when using incremental loading:
+
+- If you can and your data is not huge, use the `replace` strategy, as it's the simplest strategy and it will replace the entire destination table with the source table, which will always give you a clean exact replica of the source table.
+- If you want to keep a version history of your data, use the `append` strategy, as it will keep appending the new rows to the destination table, which will give you a version history of your data.
+- If you want to keep the latest version of your data in the destination table and your table has a natural primary key, such as a user ID, use the `merge` strategy, as it will update the existing rows in the destination table with the new values from the source table.
+- If you want to keep the destination table clean and you want to do backfills on the data, use the `delete+insert` strategy, as it will delete the existing rows in the destination table that match the `incremental_key` and then insert the new rows from the source table.
+
+> [!TIP]
+> Even though the document tries to explain, there's no better learning than trying it yourself. You can use the [Quickstart](/getting-started/quickstart.md) to try the incremental loading strategies yourself.

ingestr-0.0.4/docs/getting-started/quickstart.md

@@ -0,0 +1,39 @@
+---
+outline: deep
+---
+
+# Quickstart
+ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
+
+- ✨ copy data from your Postges / Mongo / BigQuery or any other source into any destination
+- ➕ incremental loading: `append`, `merge` or `delete+insert`
+- 🐍 single-command installation
+
+ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the magic.
+
+
+## Installation
+```
+pip install ingestr
+```
+
+## Quickstart
+
+```bash
+ingestr \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'public.some_data' \
+    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
+    --dest-table 'ingestr.some_data'
+```
+
+That's it.
+
+This command will:
+- get the table `public.some_data` from the Postgres instance.
+- upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
+
+
+## Supported Sources & Destinations
+
+See the [Supported Sources & Destinations](/supported-sources/overview.md) page for a list of all supported sources and destinations. More to come soon!

ingestr-0.0.4/docs/index.md

@@ -0,0 +1,25 @@
+---
+# https://vitepress.dev/reference/default-theme-home-page
+layout: home
+
+hero:
+  name: "ingestr"
+  text: Copy data between any source and any destination
+  tagline: "ingestr is a command-line application that allows ingesting or copying data from any source into any destination database."
+  actions:
+    - theme: brand
+      text: Getting Started
+      link: /getting-started/quickstart.md
+    - theme: alt
+      text: GitHub
+      link: https://github.com/bruin-data/ingestr
+
+features:
+  - title: Feature A
+    details: Lorem ipsum dolor sit amet, consectetur adipiscing elit
+  - title: Feature B
+    details: Lorem ipsum dolor sit amet, consectetur adipiscing elit
+  - title: Feature C
+    details: Lorem ipsum dolor sit amet, consectetur adipiscing elit
+---
+