airbyte-source-firebase-realtime-database 0.1.0__tar.gz

airbyte-source-firebase-realtime-database-0.1.0/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.1
+Name: airbyte-source-firebase-realtime-database
+Version: 0.1.0
+Summary: Source implementation for Firebase Realtime Database.
+Author: Airbyte
+Author-email: contact@airbyte.io
+Description-Content-Type: text/markdown
+Requires-Dist: airbyte-cdk
+Requires-Dist: firebase-admin
+Provides-Extra: tests
+Requires-Dist: requests-mock~=1.9.3; extra == "tests"
+Requires-Dist: pytest-mock~=3.6.1; extra == "tests"
+Requires-Dist: pytest~=6.1; extra == "tests"
+Requires-Dist: source-acceptance-test; extra == "tests"
+
+# Firebase Realtime Database Source
+
+This is the repository for the Firebase Realtime Database source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database).
+
+
+**To iterate on this connector, make sure to complete this prerequisites section.**
+
+
+From this connector directory, create a virtual environment:
+```
+python -m venv .venv
+```
+
+This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
+development environment of choice. To activate it from the terminal, run:
+```
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+
+Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
+used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
+If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
+should work as you expect.
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_firebase_realtime_database/spec.yaml` file.
+Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+See `integration_tests/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source firebase-realtime-database test creds`
+and place them into `secrets/config.json`.
+
+```
+python main.py spec
+python main.py check --config secrets/config.json
+python main.py discover --config secrets/config.json
+python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+```
+
+
+
+**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
+```bash
+airbyte-ci connectors --name=source-firebase-realtime-database build
+```
+
+An image will be built with the tag `airbyte/source-firebase-realtime-database:dev`.
+
+**Via `docker build`:**
+```bash
+docker build -t airbyte/source-firebase-realtime-database:dev .
+```
+
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-firebase-realtime-database:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-firebase-realtime-database:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+```bash
+airbyte-ci connectors --name=source-firebase-realtime-database test
+```
+
+Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+We split dependencies between two groups, dependencies that are:
+* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+* required for the testing need to go to `TEST_REQUIREMENTS` list
+
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-firebase-realtime-database test`
+2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/firebase-realtime-database.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

airbyte-source-firebase-realtime-database-0.1.0/README.md

@@ -0,0 +1,99 @@
+# Firebase Realtime Database Source
+
+This is the repository for the Firebase Realtime Database source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database).
+
+## Local development
+
+### Prerequisites
+**To iterate on this connector, make sure to complete this prerequisites section.**
+
+#### Minimum Python version required `= 3.9.0`
+
+#### Build & Activate Virtual Environment and install dependencies
+From this connector directory, create a virtual environment:
+```
+python -m venv .venv
+```
+
+This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
+development environment of choice. To activate it from the terminal, run:
+```
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+
+Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
+used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
+If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
+should work as you expect.
+
+#### Create credentials
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_firebase_realtime_database/spec.yaml` file.
+Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+See `integration_tests/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source firebase-realtime-database test creds`
+and place them into `secrets/config.json`.
+
+### Locally running the connector
+```
+python main.py spec
+python main.py check --config secrets/config.json
+python main.py discover --config secrets/config.json
+python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+```
+
+### Locally running the connector docker image
+
+
+#### Build
+**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
+```bash
+airbyte-ci connectors --name=source-firebase-realtime-database build
+```
+
+An image will be built with the tag `airbyte/source-firebase-realtime-database:dev`.
+
+**Via `docker build`:**
+```bash
+docker build -t airbyte/source-firebase-realtime-database:dev .
+```
+
+#### Run
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-firebase-realtime-database:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-firebase-realtime-database:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+## Testing
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+```bash
+airbyte-ci connectors --name=source-firebase-realtime-database test
+```
+
+### Customizing acceptance Tests
+Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+## Dependency Management
+All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+We split dependencies between two groups, dependencies that are:
+* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+* required for the testing need to go to `TEST_REQUIREMENTS` list
+
+### Publishing a new version of the connector
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-firebase-realtime-database test`
+2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/firebase-realtime-database.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+

airbyte-source-firebase-realtime-database-0.1.0/airbyte_source_firebase_realtime_database.egg-info/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.1
+Name: airbyte-source-firebase-realtime-database
+Version: 0.1.0
+Summary: Source implementation for Firebase Realtime Database.
+Author: Airbyte
+Author-email: contact@airbyte.io
+Description-Content-Type: text/markdown
+Requires-Dist: airbyte-cdk
+Requires-Dist: firebase-admin
+Provides-Extra: tests
+Requires-Dist: requests-mock~=1.9.3; extra == "tests"
+Requires-Dist: pytest-mock~=3.6.1; extra == "tests"
+Requires-Dist: pytest~=6.1; extra == "tests"
+Requires-Dist: source-acceptance-test; extra == "tests"
+
+# Firebase Realtime Database Source
+
+This is the repository for the Firebase Realtime Database source connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database).
+
+
+**To iterate on this connector, make sure to complete this prerequisites section.**
+
+
+From this connector directory, create a virtual environment:
+```
+python -m venv .venv
+```
+
+This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
+development environment of choice. To activate it from the terminal, run:
+```
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+
+Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
+used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
+If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
+should work as you expect.
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_firebase_realtime_database/spec.yaml` file.
+Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+See `integration_tests/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source firebase-realtime-database test creds`
+and place them into `secrets/config.json`.
+
+```
+python main.py spec
+python main.py check --config secrets/config.json
+python main.py discover --config secrets/config.json
+python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+```
+
+
+
+**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
+```bash
+airbyte-ci connectors --name=source-firebase-realtime-database build
+```
+
+An image will be built with the tag `airbyte/source-firebase-realtime-database:dev`.
+
+**Via `docker build`:**
+```bash
+docker build -t airbyte/source-firebase-realtime-database:dev .
+```
+
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/source-firebase-realtime-database:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-firebase-realtime-database:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+```bash
+airbyte-ci connectors --name=source-firebase-realtime-database test
+```
+
+Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+We split dependencies between two groups, dependencies that are:
+* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+* required for the testing need to go to `TEST_REQUIREMENTS` list
+
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-firebase-realtime-database test`
+2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/firebase-realtime-database.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

airbyte-source-firebase-realtime-database-0.1.0/airbyte_source_firebase_realtime_database.egg-info/SOURCES.txt

@@ -0,0 +1,21 @@
+README.md
+setup.cfg
+setup.py
+airbyte_source_firebase_realtime_database.egg-info/PKG-INFO
+airbyte_source_firebase_realtime_database.egg-info/SOURCES.txt
+airbyte_source_firebase_realtime_database.egg-info/dependency_links.txt
+airbyte_source_firebase_realtime_database.egg-info/entry_points.txt
+airbyte_source_firebase_realtime_database.egg-info/requires.txt
+airbyte_source_firebase_realtime_database.egg-info/top_level.txt
+integration_tests/__init__.py
+integration_tests/acceptance.py
+integration_tests/catalog.json
+integration_tests/configured_catalog.json
+integration_tests/invalid_config.json
+integration_tests/records.json
+integration_tests/sample_config.json
+source_firebase_realtime_database/__init__.py
+source_firebase_realtime_database/firebase_rtdb.py
+source_firebase_realtime_database/run.py
+source_firebase_realtime_database/source.py
+source_firebase_realtime_database/spec.yaml

airbyte-source-firebase-realtime-database-0.1.0/airbyte_source_firebase_realtime_database.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

airbyte-source-firebase-realtime-database-0.1.0/airbyte_source_firebase_realtime_database.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+source-firebase-realtime-database = source_firebase_realtime_database.run:run

airbyte-source-firebase-realtime-database-0.1.0/airbyte_source_firebase_realtime_database.egg-info/requires.txt

@@ -0,0 +1,8 @@
+airbyte-cdk
+firebase-admin
+
+[tests]
+requests-mock~=1.9.3
+pytest-mock~=3.6.1
+pytest~=6.1
+source-acceptance-test

airbyte-source-firebase-realtime-database-0.1.0/airbyte_source_firebase_realtime_database.egg-info/top_level.txt

@@ -0,0 +1,2 @@
+integration_tests
+source_firebase_realtime_database

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/__init__.py

@@ -0,0 +1,3 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/acceptance.py

@@ -0,0 +1,66 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+import json
+import os
+import tarfile
+from pathlib import Path
+
+import docker
+import pytest
+
+pytest_plugins = ("source_acceptance_test.plugin",)
+
+
+def create_setup_container(client):
+    integration_tests_dir = Path(os.path.dirname(__file__))
+    secrets_dir = integration_tests_dir.parent / "secrets"
+
+    with open(secrets_dir / "config.json") as f:
+        secrets = json.load(f)
+        database_name = secrets["database_name"]
+
+    container = client.containers.create(
+        image="gcr.io/google.com/cloudsdktool/google-cloud-cli",
+        command="/bin/sh",
+        auto_remove=True,
+        detach=True,
+        tty=True,
+        environment={"DB_NAME": database_name},
+    )
+
+    # The setup container is a docker in docker container.
+    # It is necessary to mount volumes by specifying the file path of the host
+    # that runs the container that runs the setup container, but there is no way to specify that path,
+    # so we should copy the files into the container.
+    archive_path = "./setup_files.tar"
+    try:
+        with tarfile.open(archive_path, "w") as f:
+            f.add(integration_tests_dir / "setup_data.sh", "setup_data.sh")
+            f.add(integration_tests_dir / "teardown.sh", "teardown.sh")
+            f.add(integration_tests_dir / "records.json", "records.json")
+            f.add(secrets_dir / "firebase-admin.json", "firebase-admin.json")
+
+        with open(archive_path, "rb") as f:
+            data = f.read()
+    finally:
+        os.remove(archive_path)
+
+    container.put_archive("/", data)
+
+    container.start()
+
+    return container
+
+
+@pytest.fixture(scope="session", autouse=True)
+def connector_setup():
+    client = docker.from_env()
+    container = create_setup_container(client)
+    container.exec_run("/setup_data.sh")
+
+    yield
+
+    container.exec_run("/teardown.sh")
+    container.stop()

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/catalog.json

@@ -0,0 +1,20 @@
+{
+  "streams": [
+    {
+      "name": "users",
+      "supported_sync_modes": ["full_refresh"],
+      "json_schema": {
+        "$schema": "http://json-schema.org/draft-07/schema#",
+        "type": "object",
+        "properties": {
+          "key": {
+            "type": "string"
+          },
+          "value": {
+            "type": "string"
+          }
+        }
+      }
+    }
+  ]
+}

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/configured_catalog.json

@@ -0,0 +1,24 @@
+{
+  "streams": [
+    {
+      "sync_mode": "full_refresh",
+      "destination_sync_mode": "overwrite",
+      "stream": {
+        "name": "users",
+        "supported_sync_modes": ["full_refresh"],
+        "json_schema": {
+          "$schema": "http://json-schema.org/draft-07/schema#",
+          "type": "object",
+          "properties": {
+            "key": {
+              "type": "string"
+            },
+            "value": {
+              "type": "string"
+            }
+          }
+        }
+      }
+    }
+  ]
+}

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/invalid_config.json

@@ -0,0 +1,6 @@
+{
+  "database_name": "invalid_database",
+  "google_application_credentials": "{\"type\":\"service_account\",\"project_id\":\"invalid\",\"private_key_id\":\"invalid_key_id\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\ninvalid_private_key_string\\n-----END PRIVATE KEY-----\\n\",\"client_email\":\"invalid-admin@invalid-project.iam.gserviceaccount.com\",\"client_id\":\"invalid_id\",\"auth_uri\":\"https://accounts.google.com/o/oauth2/auth\",\"token_uri\":\"https://oauth2.googleapis.com/token\",\"auth_provider_x509_cert_url\":\"https://www.googleapis.com/oauth2/v1/certs\",\"client_x509_cert_url\":\"https://www.googleapis.com/robot/v1/metadata/x509/invalid-admin%40invalid-project.iam.gserviceaccount.com\"}",
+  "path": "invalid_path",
+  "buffer_size": 2
+}

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/records.json

@@ -0,0 +1,6 @@
+{
+  "a": { "a": 1 },
+  "aa": { "aa": 2 },
+  "b": { "b": 3 },
+  "c": { "c": 4 }
+}

airbyte-source-firebase-realtime-database-0.1.0/integration_tests/sample_config.json

@@ -0,0 +1,6 @@
+{
+  "database_name": "your-database-name",
+  "google_application_credentials": "your service_account's credentials JSON",
+  "path": "path/to/node",
+  "buffer_size": 10000
+}

airbyte-source-firebase-realtime-database-0.1.0/setup.cfg

@@ -0,0 +1,96 @@
+[metadata]
+name = airbyte-source-firebase-realtime-database
+version = 0.1.0
+author = Airbyte
+author_email = contact@airbyte.io
+long_description = # Firebase Realtime Database Source
+	
+	This is the repository for the Firebase Realtime Database source connector, written in Python.
+	For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database).
+	
+	
+	**To iterate on this connector, make sure to complete this prerequisites section.**
+	
+	
+	From this connector directory, create a virtual environment:
+	```
+	python -m venv .venv
+	```
+	
+	This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
+	development environment of choice. To activate it from the terminal, run:
+	```
+	source .venv/bin/activate
+	pip install -r requirements.txt
+	```
+	If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+	
+	Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
+	used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
+	If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
+	should work as you expect.
+	
+	**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/firebase-realtime-database)
+	to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_firebase_realtime_database/spec.yaml` file.
+	Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+	See `integration_tests/sample_config.json` for a sample config file.
+	
+	**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source firebase-realtime-database test creds`
+	and place them into `secrets/config.json`.
+	
+	```
+	python main.py spec
+	python main.py check --config secrets/config.json
+	python main.py discover --config secrets/config.json
+	python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+	```
+	
+	
+	
+	**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
+	```bash
+	airbyte-ci connectors --name=source-firebase-realtime-database build
+	```
+	
+	An image will be built with the tag `airbyte/source-firebase-realtime-database:dev`.
+	
+	**Via `docker build`:**
+	```bash
+	docker build -t airbyte/source-firebase-realtime-database:dev .
+	```
+	
+	Then run any of the connector commands as follows:
+	```
+	docker run --rm airbyte/source-firebase-realtime-database:dev spec
+	docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev check --config /secrets/config.json
+	docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-firebase-realtime-database:dev discover --config /secrets/config.json
+	docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-firebase-realtime-database:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+	```
+	
+	You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+	```bash
+	airbyte-ci connectors --name=source-firebase-realtime-database test
+	```
+	
+	Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+	If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+	
+	All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+	We split dependencies between two groups, dependencies that are:
+	* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+	* required for the testing need to go to `TEST_REQUIREMENTS` list
+	
+	You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+	1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-firebase-realtime-database test`
+	2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
+	3. Make sure the `metadata.yaml` content is up to date.
+	4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/firebase-realtime-database.md`).
+	5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+	6. Pat yourself on the back for being an awesome contributor.
+	7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+long_description_content_type = text/markdown
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

airbyte-source-firebase-realtime-database-0.1.0/setup.py

@@ -0,0 +1,45 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from setuptools import find_packages, setup
+
+MAIN_REQUIREMENTS = [
+    "airbyte-cdk",
+    "firebase-admin",
+]
+
+TEST_REQUIREMENTS = [
+    "requests-mock~=1.9.3",
+    "pytest-mock~=3.6.1",
+    "pytest~=6.1",
+    "source-acceptance-test",
+]
+
+setup(
+    entry_points={
+        "console_scripts": [
+            "source-firebase-realtime-database=source_firebase_realtime_database.run:run",
+        ],
+    },
+    description="Source implementation for Firebase Realtime Database.",
+    packages=find_packages(),
+    install_requires=MAIN_REQUIREMENTS,
+    package_data={
+        "": [
+            # Include yaml files in the package (if any)
+            "*.yml",
+            "*.yaml",
+            # Include all json files in the package, up to 4 levels deep
+            "*.json",
+            "*/*.json",
+            "*/*/*.json",
+            "*/*/*/*.json",
+            "*/*/*/*/*.json",
+        ]
+    },
+    extras_require={
+        "tests": TEST_REQUIREMENTS,
+    },
+)

airbyte-source-firebase-realtime-database-0.1.0/source_firebase_realtime_database/__init__.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+
+from .source import SourceFirebaseRealtimeDatabase
+
+__all__ = ["SourceFirebaseRealtimeDatabase"]

airbyte-source-firebase-realtime-database-0.1.0/source_firebase_realtime_database/firebase_rtdb.py

@@ -0,0 +1,85 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+import json
+
+import firebase_admin
+from firebase_admin import credentials, db
+
+
+class Client:
+    def __init__(self, path="", buffer_size=10000):
+        self._path = path
+        self._buffer_size = buffer_size
+
+    def initialize(self, database_name, google_application_credentials):
+        database_url = f"https://{database_name}.firebaseio.com"
+        sa_key = json.loads(google_application_credentials)
+
+        cred = credentials.Certificate(sa_key)
+
+        firebase_admin.initialize_app(
+            cred,
+            {
+                "databaseURL": database_url,
+            },
+        )
+        self._ref = db.reference(self._path)
+
+    def check_connection(self):
+        self._ref.get(shallow=True)
+
+    def fetch_records(self, start_key=None):
+        if start_key:
+            return self._ref.order_by_key().start_at(start_key).limit_to_first(self._buffer_size).get()
+        else:
+            return self._ref.order_by_key().limit_to_first(self._buffer_size).get()
+
+    def extract(self):
+        return Records(self)
+
+    def set_records(self, records):
+        self._ref.set(records)
+
+    def delete_records(self):
+        self._ref.delete()
+
+
+class Records:
+    def __init__(self, client):
+        self._client = client
+
+    def __iter__(self):
+        def _gen():
+            records = self._client.fetch_records()
+            if records is None or len(records) == 0:
+                return
+
+            for k, v in records.items():
+                last_key = k
+                data = {"key": k, "value": json.dumps(v)}
+
+                yield data
+
+            # fetch data start at last_key inclusive
+            while records := self._client.fetch_records(last_key):
+                num_records = len(records)
+
+                records_iter = iter(records.items())
+                first_key, first_value = next(records_iter)
+                if first_key == last_key:
+                    if num_records == 1:
+                        return
+                else:
+                    last_key = first_key
+                    data = {"key": first_key, "value": json.dumps(first_value)}
+                    yield data
+
+                for k, v in records_iter:
+                    last_key = k
+                    data = {"key": k, "value": json.dumps(v)}
+
+                    yield data
+
+        return _gen()

airbyte-source-firebase-realtime-database-0.1.0/source_firebase_realtime_database/run.py

@@ -0,0 +1,14 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+import sys
+
+from airbyte_cdk.entrypoint import launch
+from source_firebase_realtime_database import SourceFirebaseRealtimeDatabase
+
+
+def run():
+    source = SourceFirebaseRealtimeDatabase()
+    launch(source, sys.argv[1:])

airbyte-source-firebase-realtime-database-0.1.0/source_firebase_realtime_database/source.py

@@ -0,0 +1,135 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+import json
+from datetime import datetime
+from typing import Dict, Generator
+
+from airbyte_cdk.logger import AirbyteLogger
+from airbyte_cdk.models import (
+    AirbyteCatalog,
+    AirbyteConnectionStatus,
+    AirbyteMessage,
+    AirbyteRecordMessage,
+    AirbyteStream,
+    ConfiguredAirbyteCatalog,
+    Status,
+    Type,
+)
+from airbyte_cdk.sources import Source
+
+from .firebase_rtdb import Client
+
+
+class SourceFirebaseRealtimeDatabase(Source):
+    DEFAULT_BUFFER_SIZE = 10000
+    DEFAULT_PATH = ""
+
+    @staticmethod
+    def stream_name_from(config):
+        path = config.get("path", SourceFirebaseRealtimeDatabase.DEFAULT_PATH)
+        node_name = path.rstrip("/").split("/")[-1]
+        if not node_name:
+            node_name = config["database_name"]
+
+        return node_name.replace("-", "_")
+
+    def check(self, logger: AirbyteLogger, config: json) -> AirbyteConnectionStatus:
+        """
+        Tests if the input configuration can be used to successfully connect to Firebase Realtime Database
+
+        :param logger: Logging object to display debug/info/error to the logs
+            (logs will not be accessible via airbyte UI if they are not passed to this logger)
+        :param config: Json object containing the configuration of this source, content of this json is as specified in
+        the properties of the spec.yaml file
+
+        :return: AirbyteConnectionStatus indicating a Success or Failure
+        """
+        try:
+            database_name = config["database_name"]
+            google_application_credentials = config["google_application_credentials"]
+
+            client = Client()
+            client.initialize(database_name, google_application_credentials)
+
+            # get root-node's keys to check connectivity
+            client.check_connection()
+
+            return AirbyteConnectionStatus(status=Status.SUCCEEDED)
+        except Exception as e:
+            return AirbyteConnectionStatus(status=Status.FAILED, message=f"An exception occurred: {str(e)}")
+
+    def discover(self, logger: AirbyteLogger, config: json) -> AirbyteCatalog:
+        """
+        Returns an AirbyteCatalog representing the available streams and fields in the users's connection to Firebase.
+        This connector returns only one stream that is specified by `path` in the config.
+
+        :param logger: Logging object to display debug/info/error to the logs
+            (logs will not be accessible via airbyte UI if they are not passed to this logger)
+        :param config: Json object containing the configuration of this source, content of this json is as specified in
+        the properties of the spec.yaml file
+
+        :return: AirbyteCatalog is an object describing a list of all available streams in this source.
+            A stream is an AirbyteStream object that includes:
+            - its stream name (or table name in the case of Postgres)
+            - json_schema providing the specifications of expected schema for this stream (a list of columns described
+            by their names and types)
+        """
+        streams = []
+
+        json_schema = {
+            "$schema": "http://json-schema.org/draft-07/schema#",
+            "type": "object",
+            "properties": {
+                "key": {"type": "string"},
+                "value": {"type": "string"},
+            },
+        }
+
+        stream_name = self.stream_name_from(config)
+
+        streams.append(AirbyteStream(name=stream_name, json_schema=json_schema, supported_sync_modes=["full_refresh"]))
+
+        return AirbyteCatalog(streams=streams)
+
+    def read(
+        self, logger: AirbyteLogger, config: json, catalog: ConfiguredAirbyteCatalog, state: Dict[str, any]
+    ) -> Generator[AirbyteMessage, None, None]:
+        """
+        Returns a generator of the AirbyteMessages generated by reading the source with the given configuration,
+        catalog, and state.
+
+        :param logger: Logging object to display debug/info/error to the logs
+            (logs will not be accessible via airbyte UI if they are not passed to this logger)
+        :param config: Json object containing the configuration of this source, content of this json is as specified in
+            the properties of the spec.yaml file
+        :param catalog: The input catalog is a ConfiguredAirbyteCatalog which is almost the same as AirbyteCatalog
+            returned by discover(), but
+        in addition, it's been configured in the UI! For each particular stream and field, there may have been provided
+        with extra modifications such as: filtering streams and/or columns out, renaming some entities, etc
+        :param state: When a Airbyte reads data from a source, it might need to keep a checkpoint cursor to resume
+            replication in the future from that saved checkpoint.
+            This is the object that is provided with state from previous runs and avoid replicating the entire set of
+            data everytime.
+
+        :return: A generator that produces a stream of AirbyteRecordMessage contained in AirbyteMessage object.
+        """
+
+        stream = catalog.streams[0].stream
+        stream_name = stream.name
+
+        buffer_size = config.get("buffer_size", self.DEFAULT_BUFFER_SIZE)
+        path = config.get("path", self.DEFAULT_PATH)
+        database_name = config["database_name"]
+        google_application_credentials = config["google_application_credentials"]
+
+        client = Client(path, buffer_size)
+        client.initialize(database_name, google_application_credentials)
+
+        for data in client.extract():
+            yield AirbyteMessage(
+                type=Type.RECORD,
+                record=AirbyteRecordMessage(stream=stream_name, data=data, emitted_at=int(datetime.now().timestamp()) * 1000),
+            )

airbyte-source-firebase-realtime-database-0.1.0/source_firebase_realtime_database/spec.yaml

@@ -0,0 +1,27 @@
+documentationUrl: https://docs.airbyte.io/integrations/sources/firebase-realtime-database
+supported_sync_modes: ["full_refresh"]
+connectionSpecification:
+  $schema: http://json-schema.org/draft-07/schema#
+  title: Firebase Realtime Database Spec
+  type: object
+  required:
+    - database_name
+    - google_application_credentials
+  properties:
+    database_name:
+      title: Database Name
+      type: string
+      description: Database name (This will be part of the url pointing to the database, https://<database_name>.firebaseio.com/)
+    google_application_credentials:
+      title: Service Account Key JSON
+      type: string
+      description: Cert credentials in JSON format of Service Account with Firebase Realtime Database Viewer role. (see, https://firebase.google.com/docs/projects/iam/roles-predefined-product#realtime-database)
+      airbyte_secret: true
+    path:
+      title: Node Path
+      type: string
+      description: Path to a node in the Firebase realtime database
+    buffer_size:
+      title: Buffer Size
+      type: number
+      description: Number of records to fetch at once