airbyte-source-faker 4.0.0__tar.gz

airbyte-source-faker-4.0.0/PKG-INFO

@@ -0,0 +1,7 @@
+Metadata-Version: 2.1
+Name: airbyte-source-faker
+Version: 4.0.0
+Summary: Source implementation for fake but realistic looking data.
+Author: Airbyte
+Author-email: contact@airbyte.io
+Provides-Extra: tests

airbyte-source-faker-4.0.0/README.md

@@ -0,0 +1,168 @@
+# Faker Source
+
+This is the repository for the Faker source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/faker).
+
+## Local development
+
+### Prerequisites
+
+**To iterate on this connector, make sure to complete this prerequisites section.**
+
+#### Minimum Python version required `= 3.9.0`
+
+#### Build & Activate Virtual Environment and install dependencies
+
+From this connector directory, create a virtual environment:
+
+```
+python -m venv .venv
+```
+
+This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
+development environment of choice. To activate it from the terminal, run:
+
+```
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+
+Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
+used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
+If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
+should work as you expect.
+
+#### Building via Gradle
+
+From the Airbyte repository root, run:
+
+```
+./gradlew :airbyte-integrations:connectors:source-faker:build
+```
+
+#### Create credentials
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/faker)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_faker/spec.json` file.
+Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+See `integration_tests/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source faker test creds`
+and place them into `secrets/config.json`.
+
+### Locally running the connector
+
+```
+python main.py spec
+python main.py check --config secrets/config.json
+python main.py discover --config secrets/config.json
+python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+```
+
+### Locally running the connector docker image
+
+#### Build
+
+First, make sure you build the latest Docker image:
+
+```
+docker build . -t airbyte/source-faker:dev
+```
+
+You can also build the connector image via Gradle:
+
+```
+./gradlew :airbyte-integrations:connectors:source-faker:airbyteDocker
+```
+
+When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
+the Dockerfile.
+
+#### Run
+
+Then run any of the connector commands as follows:
+
+```
+docker run --rm airbyte/source-faker:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-faker:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-faker:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-faker:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+## Testing
+
+Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
+First install test dependencies into your virtual environment:
+
+```
+pip install ".[tests]"
+```
+
+### Unit Tests
+
+To run unit tests locally, from the connector directory run:
+
+```
+python -m pytest unit_tests
+```
+
+### Integration Tests
+
+There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
+
+#### Custom Integration tests
+
+Place custom tests inside `integration_tests/` folder, then, from the connector root, run
+
+```
+python -m pytest integration_tests
+```
+
+#### Acceptance Tests
+
+Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+To run your integration tests with acceptance tests, from the connector root, run
+
+```
+python -m pytest integration_tests -p integration_tests.acceptance
+```
+
+To run your integration tests with docker
+
+### Using gradle to run tests
+
+All commands should be run from airbyte project root.
+To run unit tests:
+
+```
+./gradlew :airbyte-integrations:connectors:source-faker:unitTest
+```
+
+To run acceptance and custom integration tests:
+
+```
+./gradlew :airbyte-integrations:connectors:source-faker:integrationTest
+```
+
+## Dependency Management
+
+All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+We split dependencies between two groups, dependencies that are:
+
+- required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+- required for the testing need to go to `TEST_REQUIREMENTS` list
+
+### Publishing a new version of the connector
+
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+
+1. Make sure your changes are passing unit and integration tests.
+1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
+1. Create a Pull Request.
+1. Pat yourself on the back for being an awesome contributor.
+1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+
+The end

airbyte-source-faker-4.0.0/airbyte_source_faker.egg-info/PKG-INFO

@@ -0,0 +1,7 @@
+Metadata-Version: 2.1
+Name: airbyte-source-faker
+Version: 4.0.0
+Summary: Source implementation for fake but realistic looking data.
+Author: Airbyte
+Author-email: contact@airbyte.io
+Provides-Extra: tests

airbyte-source-faker-4.0.0/airbyte_source_faker.egg-info/SOURCES.txt

@@ -0,0 +1,28 @@
+README.md
+setup.cfg
+setup.py
+airbyte_source_faker.egg-info/PKG-INFO
+airbyte_source_faker.egg-info/SOURCES.txt
+airbyte_source_faker.egg-info/dependency_links.txt
+airbyte_source_faker.egg-info/requires.txt
+airbyte_source_faker.egg-info/top_level.txt
+integration_tests/__init__.py
+integration_tests/abnormal_state.json
+integration_tests/acceptance.py
+integration_tests/catalog.json
+integration_tests/configured_catalog.json
+integration_tests/invalid_config.json
+integration_tests/sample_config.json
+integration_tests/sample_state.json
+source_faker/__init__.py
+source_faker/airbyte_message_with_cached_json.py
+source_faker/purchase_generator.py
+source_faker/source.py
+source_faker/spec.json
+source_faker/streams.py
+source_faker/user_generator.py
+source_faker/utils.py
+source_faker/record_data/products.json
+source_faker/schemas/products.json
+source_faker/schemas/purchases.json
+source_faker/schemas/users.json

airbyte-source-faker-4.0.0/airbyte_source_faker.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

airbyte-source-faker-4.0.0/airbyte_source_faker.egg-info/requires.txt

@@ -0,0 +1,6 @@
+airbyte-cdk~=0.2
+mimesis==6.1.1
+
+[tests]
+pytest~=6.2
+connector-acceptance-test

airbyte-source-faker-4.0.0/airbyte_source_faker.egg-info/top_level.txt

@@ -0,0 +1,2 @@
+integration_tests
+source_faker

airbyte-source-faker-4.0.0/integration_tests/__init__.py

@@ -0,0 +1,3 @@
+#
+# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
+#

airbyte-source-faker-4.0.0/integration_tests/abnormal_state.json

@@ -0,0 +1,35 @@
+[
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": {
+        "updated_at": 11
+      },
+      "stream_descriptor": {
+        "name": "users"
+      }
+    }
+  },
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": {
+        "updated_at": 11
+      },
+      "stream_descriptor": {
+        "name": "purchases"
+      }
+    }
+  },
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": {
+        "updated_at": 101
+      },
+      "stream_descriptor": {
+        "name": "products"
+      }
+    }
+  }
+]

airbyte-source-faker-4.0.0/integration_tests/acceptance.py

@@ -0,0 +1,14 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+import pytest
+
+pytest_plugins = ("connector_acceptance_test.plugin",)
+
+
+@pytest.fixture(scope="session", autouse=True)
+def connector_setup():
+    """This fixture is a placeholder for external resources that acceptance test might require."""
+    yield

airbyte-source-faker-4.0.0/integration_tests/catalog.json

@@ -0,0 +1,40 @@
+{
+  "streams": [
+    {
+      "name": "users",
+      "json_schema": {
+        "$schema": "http://json-schema.org/draft-07/schema#",
+        "type": "object",
+        "properties": {
+          "id": { "type": "number" },
+          "created_at": {
+            "type": "string",
+            "format": "date-time",
+            "airbyte_type": "timestamp_with_timezone"
+          },
+          "updated_at": {
+            "type": "string",
+            "format": "date-time",
+            "airbyte_type": "timestamp_with_timezone"
+          },
+          "name": { "type": "string" },
+          "title": { "type": "string" },
+          "age": { "type": "integer" },
+          "email": { "type": "string" },
+          "telephone": { "type": "string" },
+          "gender": { "type": "string" },
+          "language": { "type": "string" },
+          "academic_degree": { "type": "string" },
+          "nationality": { "type": "string" },
+          "occupation": { "type": "string" },
+          "height": { "type": "string" },
+          "blood_type": { "type": "string" },
+          "weight": { "type": "integer" }
+        }
+      },
+      "supported_sync_modes": ["incremental", "full_refresh"],
+      "source_defined_cursor": true,
+      "default_cursor_field": ["created_at"]
+    }
+  ]
+}

airbyte-source-faker-4.0.0/integration_tests/configured_catalog.json

@@ -0,0 +1,115 @@
+{
+  "streams": [
+    {
+      "stream": {
+        "name": "users",
+        "json_schema": {
+          "$schema": "http://json-schema.org/draft-07/schema#",
+          "type": "object",
+          "properties": {
+            "id": { "type": "number" },
+            "created_at": {
+              "type": "string",
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "updated_at": {
+              "type": "string",
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "name": { "type": "string" },
+            "title": { "type": "string" },
+            "age": { "type": "integer" },
+            "email": { "type": "string" },
+            "telephone": { "type": "string" },
+            "gender": { "type": "string" },
+            "language": { "type": "string" },
+            "academic_degree": { "type": "string" },
+            "nationality": { "type": "string" },
+            "occupation": { "type": "string" },
+            "height": { "type": "string" },
+            "blood_type": { "type": "string" },
+            "weight": { "type": "integer" }
+          }
+        },
+        "supported_sync_modes": ["incremental", "full_refresh"],
+        "source_defined_cursor": true,
+        "default_cursor_field": ["created_at"]
+      },
+      "sync_mode": "incremental",
+      "destination_sync_mode": "overwrite"
+    },
+    {
+      "stream": {
+        "name": "purchases",
+        "json_schema": {
+          "properties": {
+            "id": { "type": "number" },
+            "user_id": { "type": "number" },
+            "product_id": { "type": "number" },
+            "created_at": {
+              "type": "string",
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "updated_at": {
+              "type": "string",
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "added_to_cart_at": {
+              "type": ["null", "string"],
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "purchased_at": {
+              "type": ["null", "string"],
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "returned_at": {
+              "type": ["null", "string"],
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            }
+          }
+        },
+        "supported_sync_modes": ["incremental", "full_refresh"],
+        "source_defined_cursor": true,
+        "default_cursor_field": ["created_at"]
+      },
+      "sync_mode": "incremental",
+      "destination_sync_mode": "overwrite"
+    },
+    {
+      "stream": {
+        "name": "products",
+        "json_schema": {
+          "properties": {
+            "id": { "type": "number" },
+            "make": { "type": "string" },
+            "model": { "type": "string" },
+            "year": { "type": "number" },
+            "price": { "type": "number" },
+            "created_at": {
+              "type": "string",
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            },
+            "updated_at": {
+              "type": "string",
+              "format": "date-time",
+              "airbyte_type": "timestamp_with_timezone"
+            }
+          }
+        },
+        "supported_sync_modes": ["incremental", "full_refresh"],
+        "source_defined_cursor": true,
+        "default_cursor_field": ["created_at"]
+      },
+      "sync_mode": "incremental",
+      "destination_sync_mode": "overwrite"
+    }
+  ]
+}

airbyte-source-faker-4.0.0/integration_tests/invalid_config.json

@@ -0,0 +1,3 @@
+{
+  "count": -1
+}

airbyte-source-faker-4.0.0/integration_tests/sample_config.json

@@ -0,0 +1,6 @@
+{
+  "count": 10,
+  "seed": 0,
+  "parallelism": 1,
+  "always_updated": false
+}

airbyte-source-faker-4.0.0/integration_tests/sample_state.json

@@ -0,0 +1,24 @@
+[
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": {
+        "id": 0
+      },
+      "stream_descriptor": {
+        "name": "users"
+      }
+    }
+  },
+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": {
+        "user_id": 0
+      },
+      "stream_descriptor": {
+        "name": "purchases"
+      }
+    }
+  }
+]

airbyte-source-faker-4.0.0/setup.cfg

@@ -0,0 +1,10 @@
+[metadata]
+name = airbyte-source-faker
+version = 4.0.0
+author = Airbyte
+author_email = contact@airbyte.io
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

airbyte-source-faker-4.0.0/setup.py

@@ -0,0 +1,23 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from setuptools import find_packages, setup
+
+MAIN_REQUIREMENTS = ["airbyte-cdk~=0.2", "mimesis==6.1.1"]
+
+TEST_REQUIREMENTS = [
+    "pytest~=6.2",
+    "connector-acceptance-test",
+]
+
+setup(
+    description="Source implementation for fake but realistic looking data.",
+    packages=find_packages(),
+    install_requires=MAIN_REQUIREMENTS,
+    package_data={"": ["*.json", "schemas/*.json", "record_data/*.json"]},
+    extras_require={
+        "tests": TEST_REQUIREMENTS,
+    },
+)

airbyte-source-faker-4.0.0/source_faker/__init__.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
+#
+
+
+from .source import SourceFaker
+
+__all__ = ["SourceFaker"]

airbyte-source-faker-4.0.0/source_faker/airbyte_message_with_cached_json.py

@@ -0,0 +1,22 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+from airbyte_cdk.models import AirbyteMessage
+
+
+class AirbyteMessageWithCachedJSON(AirbyteMessage):
+    """
+    I a monkeypatch to AirbyteMessage which pre-renders the JSON-representation of the object upon initialization.
+    This allows the JSON to be calculated in the process that builds the object rather than the main process.
+
+    Note: We can't use @cache here because the LRU cache is not serializable when passed to child workers.
+    """
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self._json = self.json(exclude_unset=True)
+        self.json = self.get_json
+
+    def get_json(self, **kwargs):
+        return self._json

airbyte-source-faker-4.0.0/source_faker/purchase_generator.py

@@ -0,0 +1,106 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+import datetime
+from multiprocessing import current_process
+from typing import Dict, List
+
+from airbyte_cdk.models import AirbyteRecordMessage, Type
+from mimesis import Datetime, Numeric
+
+from .airbyte_message_with_cached_json import AirbyteMessageWithCachedJSON
+from .utils import format_airbyte_time, now_millis
+
+
+class PurchaseGenerator:
+    def __init__(self, stream_name: str, seed: int) -> None:
+        self.stream_name = stream_name
+        self.seed = seed
+
+    def prepare(self):
+        """
+        Note: the instances of the mimesis generators need to be global.
+        Yes, they *should* be able to be instance variables on this class, which should only instantiated once-per-worker, but that's not quite the case:
+        * relying only on prepare as a pool initializer fails because we are calling the parent process's method, not the fork
+        * Calling prepare() as part of generate() (perhaps checking if self.person is set) and then `print(self, current_process()._identity, current_process().pid)` reveals multiple object IDs in the same process, resetting the internal random counters
+        """
+
+        seed_with_offset = self.seed
+        if self.seed is not None and len(current_process()._identity) > 0:
+            seed_with_offset = self.seed + current_process()._identity[0]
+
+        global dt
+        global numeric
+
+        dt = Datetime(seed=seed_with_offset)
+        numeric = Numeric(seed=seed_with_offset)
+
+    def random_date_in_range(
+        self, start_date: datetime.datetime, end_date: datetime.datetime = datetime.datetime.now()
+    ) -> datetime.datetime:
+        time_between_dates = end_date - start_date
+        days_between_dates = time_between_dates.days
+        if days_between_dates < 2:
+            days_between_dates = 2
+        random_number_of_days = numeric.integer_number(0, days_between_dates)
+        random_date = start_date + datetime.timedelta(days=random_number_of_days)
+        return random_date
+
+    def generate(self, user_id: int) -> List[Dict]:
+        """
+        Because we are doing this work in parallel processes, we need a deterministic way to know what a purchase's ID should be given on the input of a user_id.
+        tldr; Every 10 user_ids produce 10 purchases.  User ID x5 has no purchases, User ID mod x7 has 2, and everyone else has 1
+        """
+
+        purchases: List[Dict] = []
+        last_user_id_digit = int(repr(user_id)[-1])
+        purchase_count = 1
+        id_offset = 0
+        if last_user_id_digit - 1 == 5:
+            purchase_count = 0
+        elif last_user_id_digit - 1 == 6:
+            id_offset = 1
+        elif last_user_id_digit - 1 == 7:
+            id_offset = 1
+            purchase_count = 2
+
+        total_products = 100
+        i = 0
+
+        while purchase_count > 0:
+            id = user_id + i + 1 - id_offset
+            time_a = dt.datetime()
+            time_b = dt.datetime()
+            updated_at = format_airbyte_time(datetime.datetime.now())
+            created_at = time_a if time_a <= time_b else time_b
+            product_id = numeric.integer_number(1, total_products)
+            added_to_cart_at = self.random_date_in_range(created_at)
+            purchased_at = (
+                self.random_date_in_range(added_to_cart_at)
+                if added_to_cart_at is not None and numeric.integer_number(1, 100) <= 70
+                else None
+            )  # 70% likely to purchase the item in the cart
+            returned_at = (
+                self.random_date_in_range(purchased_at) if purchased_at is not None and numeric.integer_number(1, 100) <= 15 else None
+            )  # 15% likely to return the item
+
+            purchase = {
+                "id": id,
+                "product_id": product_id,
+                "user_id": user_id + 1,
+                "created_at": created_at,
+                "updated_at": updated_at,
+                "added_to_cart_at": format_airbyte_time(added_to_cart_at) if added_to_cart_at is not None else None,
+                "purchased_at": format_airbyte_time(purchased_at) if purchased_at is not None else None,
+                "returned_at": format_airbyte_time(returned_at) if returned_at is not None else None,
+            }
+
+            record = AirbyteRecordMessage(stream=self.stream_name, data=purchase, emitted_at=now_millis())
+            message = AirbyteMessageWithCachedJSON(type=Type.RECORD, record=record)
+            purchases.append(message)
+
+            purchase_count = purchase_count - 1
+            i += 1
+
+        return purchases