bizon 0.1.2__py3-none-any.whl → 0.3.0__py3-none-any.whl

bizon-0.3.0.dist-info/METADATA

@@ -0,0 +1,323 @@
+Metadata-Version: 2.4
+Name: bizon
+Version: 0.3.0
+Summary: Extract and load your data reliably from API Clients with native fault-tolerant and checkpointing mechanism.
+Author-email: Antoine Balliet <antoine.balliet@gmail.com>, Anas El Mhamdi <anas.elmhamdi@gmail.com>
+License-File: LICENSE
+Requires-Python: <3.13,>=3.9
+Requires-Dist: backoff>=2.2.1
+Requires-Dist: click>=8.1.7
+Requires-Dist: dpath>=2.2.0
+Requires-Dist: google-cloud-storage>=2.17.0
+Requires-Dist: loguru>=0.7.2
+Requires-Dist: orjson>=3.10.16
+Requires-Dist: pendulum>=3.0.0
+Requires-Dist: polars>=1.16.0
+Requires-Dist: pyarrow>=16.1.0
+Requires-Dist: pydantic-extra-types>=2.9.0
+Requires-Dist: pydantic>=2.8.2
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: pytz>=2024.2
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: requests>=2.28.2
+Requires-Dist: simplejson>=3.20.1
+Requires-Dist: sqlalchemy>=2.0.32
+Requires-Dist: tenacity>=9.0.0
+Provides-Extra: bigquery
+Requires-Dist: google-cloud-bigquery-storage>=2.25.0; extra == 'bigquery'
+Requires-Dist: google-cloud-bigquery>=3.25.0; extra == 'bigquery'
+Requires-Dist: protobuf>=4.24.0; extra == 'bigquery'
+Requires-Dist: sqlalchemy-bigquery>=1.11.0; extra == 'bigquery'
+Provides-Extra: datadog
+Requires-Dist: datadog>=0.50.2; extra == 'datadog'
+Requires-Dist: ddtrace>=3.10.0; extra == 'datadog'
+Provides-Extra: gsheets
+Requires-Dist: gspread>=6.1.2; extra == 'gsheets'
+Provides-Extra: kafka
+Requires-Dist: avro>=1.12.0; extra == 'kafka'
+Requires-Dist: confluent-kafka>=2.6.0; extra == 'kafka'
+Requires-Dist: fastavro>=1.9.7; extra == 'kafka'
+Requires-Dist: kafka-python>=2.0.2; extra == 'kafka'
+Provides-Extra: postgres
+Requires-Dist: psycopg2-binary>=2.9.9; extra == 'postgres'
+Provides-Extra: rabbitmq
+Requires-Dist: pika>=1.3.2; extra == 'rabbitmq'
+Description-Content-Type: text/markdown
+
+# bizon ⚡️
+Extract and load your largest data streams with a framework you can trust for billion records.
+
+## Features
+- **Natively fault-tolerant**: Bizon uses a checkpointing mechanism to keep track of the progress and recover from the last checkpoint.
+
+- **High throughput**: Bizon is designed to handle high throughput and can process billions of records.
+
+- **Queue system agnostic**: Bizon is agnostic of the queuing system, you can use any queuing system among Python Queue, RabbitMQ, Kafka or Redpanda. Thanks to the `bizon.engine.queue.Queue` interface, adapters can be written for any queuing system.
+
+- **Pipeline metrics**: Bizon provides exhaustive pipeline metrics and implement Datadog & OpenTelemetry for tracing. You can monitor:
+    - ETAs for completion
+    - Number of records processed
+    - Completion percentage
+    - Latency Source <> Destination
+
+- **Lightweight & lean**: Bizon is lightweight, minimal codebase and only uses few dependencies:
+    - `requests` for HTTP requests
+    - `pyyaml` for configuration
+    - `sqlalchemy` for database / warehouse connections
+    - `polars` for memory efficient data buffering and vectorized processing
+    - `pyarrow` for Parquet file format
+
+## Installation
+
+### For Users
+```bash
+pip install bizon
+
+# With optional dependencies
+pip install bizon[postgres]    # PostgreSQL backend
+pip install bizon[kafka]       # Kafka queue
+pip install bizon[bigquery]    # BigQuery backend/destination
+pip install bizon[rabbitmq]    # RabbitMQ queue
+```
+
+### For Development
+```bash
+# Install uv (if not already installed)
+pip install uv
+
+# Clone and install
+git clone https://github.com/bizon-data/bizon-core.git
+cd bizon-core
+uv sync --all-extras --all-groups
+
+# Run tests
+uv run pytest tests/
+
+# Format code
+uv run ruff format .
+uv run ruff check --fix .
+```
+
+## Usage
+
+### List available sources and streams
+```bash
+bizon source list
+bizon stream list <source_name>
+```
+
+### Create a pipeline
+
+Create a file named `config.yml` in your working directory with the following content:
+
+```yaml
+name: demo-creatures-pipeline
+
+source:
+  name: dummy
+  stream: creatures
+  authentication:
+    type: api_key
+    params:
+      token: dummy_key
+
+destination:
+  name: logger
+  config:
+    dummy: dummy
+```
+
+Run the pipeline with the following command:
+
+```bash
+bizon run config.yml
+```
+## Backend configuration
+
+Backend is the interface used by Bizon to store its state. It can be configured in the `backend` section of the configuration file. The following backends are supported:
+- `sqlite`: In-memory SQLite database, useful for testing and development.
+- `bigquery`: Google BigQuery backend, perfect for light setup & production.
+- `postgres`: PostgreSQL backend, for production use and frequent cursor updates.
+
+## Queue configuration
+
+Queue is the interface used by Bizon to exchange data between `Source` and `Destination`. It can be configured in the `queue` section of the configuration file. The following queues are supported:
+- `python_queue`: Python Queue, useful for testing and development.
+- `rabbitmq`: RabbitMQ, for production use and high throughput.
+- `kafka`: Apache Kafka, for production use and high throughput and strong persistence.
+
+## Runner configuration
+
+Runner is the interface used by Bizon to run the pipeline. It can be configured in the `runner` section of the configuration file. The following runners are supported:
+- `thread` (asynchronous)
+- `process` (asynchronous)
+- `stream` (synchronous)
+
+## Sync Modes
+
+Bizon supports three sync modes:
+- `full_refresh`: Re-syncs all data from scratch on each run
+- `incremental`: Syncs only new/updated data since the last successful run
+- `stream`: Continuous streaming mode for real-time data (e.g., Kafka)
+
+### Incremental Sync
+
+Incremental sync fetches only new or updated records since the last successful run, using an **append-only** strategy.
+
+#### Configuration
+
+```yaml
+source:
+  name: your_source
+  stream: your_stream
+  sync_mode: incremental
+  cursor_field: updated_at  # The timestamp field to filter records by
+```
+
+#### How It Works
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                        INCREMENTAL SYNC FLOW                        │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  1. Producer checks for last successful job                         │
+│     └─> Backend.get_last_successful_stream_job()                    │
+│                                                                     │
+│  2. If found, creates SourceIncrementalState:                       │
+│     └─> last_run = previous_job.created_at                          │
+│     └─> cursor_field = config.cursor_field (e.g., "updated_at")     │
+│                                                                     │
+│  3. Calls source.get_records_after(source_state, pagination)        │
+│     └─> Source filters: WHERE cursor_field > last_run               │
+│                                                                     │
+│  4. Records written to temp table: {table}_incremental              │
+│                                                                     │
+│  5. finalize() appends temp table to main table                     │
+│     └─> INSERT INTO main_table SELECT * FROM temp_table             │
+│     └─> Deletes temp table                                          │
+│                                                                     │
+│  FIRST RUN: No previous job → falls back to get() (full refresh)    │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+#### Configuration Options
+
+| Option | Required | Description | Example |
+|--------|----------|-------------|---------|
+| `sync_mode` | Yes | Set to `incremental` | `incremental` |
+| `cursor_field` | Yes | Timestamp field to filter by | `updated_at`, `last_edited_time`, `modified_at` |
+
+#### Supported Sources
+
+Sources must implement `get_records_after()` to support incremental sync:
+
+| Source | Cursor Field | Notes |
+|--------|--------------|-------|
+| `notion` | `last_edited_time` | Supports `pages`, `databases`, `blocks`, `blocks_markdown` streams |
+| (others) | Varies | Check source docs or implement `get_records_after()` |
+
+#### Supported Destinations
+
+Destinations must implement `finalize()` with incremental logic:
+
+| Destination | Support | Notes |
+|-------------|---------|-------|
+| `bigquery` | ✅ | Append-only via temp table |
+| `bigquery_streaming_v2` | ✅ | Append-only via temp table |
+| `file` | ✅ | Appends to existing file |
+| `logger` | ✅ | Logs completion |
+
+#### Example: Notion Incremental Sync
+
+```yaml
+name: notion_incremental_sync
+
+source:
+  name: notion
+  stream: blocks_markdown
+  sync_mode: incremental
+  cursor_field: last_edited_time
+  authentication:
+    type: api_key
+    params:
+      token: secret_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+  database_ids:
+    - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+
+  # Optional: filter which pages to sync
+  database_filters:
+    "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx":
+      property: "Status"
+      select:
+        equals: "Published"
+
+destination:
+  name: bigquery
+  config:
+    project_id: my-gcp-project
+    dataset_id: notion_data
+    dataset_location: US
+
+engine:
+  backend:
+    type: bigquery
+    database: my-gcp-project
+    schema: bizon_backend
+    syncCursorInDBEvery: 2
+```
+
+#### First Run Behavior
+
+On the first incremental run (no previous successful job):
+- Falls back to `get()` method (full refresh behavior)
+- All data is fetched and loaded
+- Job is marked as successful
+- Subsequent runs use `get_records_after()` with `last_run` timestamp
+
+## Start syncing your data 🚀
+
+### Quick setup without any dependencies ✌️
+
+Queue configuration can be set to `python_queue` and backend configuration to `sqlite`.
+This will allow you to test the pipeline without any external dependencies.
+
+
+### Local Kafka setup
+
+To test the pipeline with Kafka, you can use `docker compose` to setup Kafka or Redpanda locally.
+
+**Kafka**
+```bash
+docker compose --file ./scripts/kafka-compose.yml up # Kafka
+docker compose --file ./scripts/redpanda-compose.yml up # Redpanda
+```
+
+In your YAML configuration, set the `queue` configuration to Kafka under `engine`:
+```yaml
+engine:
+  queue:
+    type: kafka
+    config:
+      queue:
+        bootstrap_server: localhost:9092 # Kafka:9092 & Redpanda: 19092
+```
+
+**RabbitMQ**
+```bash
+docker compose --file ./scripts/rabbitmq-compose.yml up
+```
+
+In your YAML configuration, set the `queue` configuration to Kafka under `engine`:
+
+```yaml
+engine:
+  queue:
+    type: rabbitmq
+    config:
+      queue:
+        host: localhost
+        queue_name: bizon
+```

bizon-0.3.0.dist-info/RECORD

@@ -0,0 +1,142 @@
+bizon/__main__.py,sha256=6GV4zEg2wC8UBnmESrr71ZmpWo4cNWlrYE3PQuKwFHA,69
+bizon/utils.py,sha256=HXaPiyxpWKoy3XN5vSYOve1ezlFeOYin3aFqTjcabUQ,81
+bizon/alerting/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/alerting/alerts.py,sha256=i3c5Y7WcyzPHqilkP8CrujwXVd058Lb0W7XsjB1Ef7w,675
+bizon/alerting/models.py,sha256=kWTeoT7dDC6UrkybU1sRAIXNAE0Wwipf8W7dEEhMZM0,553
+bizon/alerting/slack/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/alerting/slack/config.py,sha256=D_CAtzXu-O41OVoyrlYRlBxnGN9rNbnohYQ9CmBYt_E,84
+bizon/alerting/slack/handler.py,sha256=0m6IUSkxqDMlpDWslkImQX74ScD8wIx3YtrhfpYNGUA,1620
+bizon/cli/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/cli/main.py,sha256=pQnphPLznllBVzifyIrn3es0U9E1VGSzeEv_COzq9FI,3364
+bizon/cli/utils.py,sha256=aZ47YjFfifHkW95bAVzWfEQD3ZnxGSMT32bkRLmc5-c,953
+bizon/common/models.py,sha256=eL_Ii0CkeJFIjak1CKrB74mbC3OkmWP2uI27ynlYgkQ,10070
+bizon/common/errors/backoff.py,sha256=z7RkQt1Npdh0sfD3hBDaiWQKe4iqS6ewvT1Q4Fds5aU,508
+bizon/common/errors/errors.py,sha256=mrYx1uE2kOuR2pEaB7ztK1l2m0E4V-_-hxq-DuILerY,682
+bizon/connectors/destinations/bigquery/config/bigquery.example.yml,sha256=sy5-Piew00BlcjX5CFayFVrUq9G_vFYWXDmpWi9beTY,1263
+bizon/connectors/destinations/bigquery/config/bigquery_incremental.example.yml,sha256=z0pz4W1x0dlsoAjorYR2DxMjkzTvIWn9tigqtOR8PUY,1076
+bizon/connectors/destinations/bigquery/src/config.py,sha256=q55zR_9V5-ZZmOmSK7fDOHSzzYhoT-fwlppDzX4he9U,4000
+bizon/connectors/destinations/bigquery/src/destination.py,sha256=awS3dZsSKqLTVnhBKuP_9rXSt3IpGv3c4WjZOCwqu9o,9888
+bizon/connectors/destinations/bigquery_streaming/config/bigquery_streaming.example.yml,sha256=rF0mQ5IaOe6oqsbVy6q0innn7SXsOoBdBvIN8BTwPVc,1869
+bizon/connectors/destinations/bigquery_streaming/src/config.py,sha256=LdBKEqHPaGll8PW6c6q_lH7PJvsGdtv2BCrtB-TukTA,1898
+bizon/connectors/destinations/bigquery_streaming/src/destination.py,sha256=Uyne57NoT-z9uk7Yi4EgOUFYQ4QlvXDLFxgZC5KyCFE,14222
+bizon/connectors/destinations/bigquery_streaming_v2/config/bigquery_streaming_v2.example.yml,sha256=hIQXlXtiBT8DgMVAs0x_h-19xoLkjHr-Ko7oSn8jnc0,2023
+bizon/connectors/destinations/bigquery_streaming_v2/src/config.py,sha256=cdHST5Vx1VQbLsIVsPkoEtOJKmbA35XjsKzj6fZ5DHw,1907
+bizon/connectors/destinations/bigquery_streaming_v2/src/destination.py,sha256=5aXEsbzyWKzS2F1pFMZ8pdbJaXmdGTaIrwgl2cd1IbU,19026
+bizon/connectors/destinations/bigquery_streaming_v2/src/proto_utils.py,sha256=aWYVzMPMTgsdDapYniu8h6Tf2Pty4fDisT_33d9yEJ4,3692
+bizon/connectors/destinations/file/config/file.example.yml,sha256=sMeX92hTrTQUrLmQgQFsq5OdG5Dk3BbpDo0NhRbBahI,986
+bizon/connectors/destinations/file/config/file_incremental.example.yml,sha256=Xh5KwWiQRuq_MnMgOCHiHqIwHjOjXbwQlVlVcKdXARA,620
+bizon/connectors/destinations/file/src/config.py,sha256=dU64aFe7J63aBGh6Os8mXl2kvECj3s4pPC7H3EmOvb8,585
+bizon/connectors/destinations/file/src/destination.py,sha256=RQEL0Z5l409S319fAJyvW8cDblUCVAxPhALJVhjQKDM,4253
+bizon/connectors/destinations/logger/config/logger.example.yml,sha256=KtQRmqqFeziJtBZ7vzrXGQLdTgWZNjxx2sdFXpIgIp4,672
+bizon/connectors/destinations/logger/config/logger_incremental.example.yml,sha256=rwTLlXib-Jo3b4-_NcFv2ShdPC73WEpiiX3apP3sKg0,541
+bizon/connectors/destinations/logger/src/config.py,sha256=vIV_G0k9c8DPcDxU6CGvEOL2zAEvAmKZcx3RV0eRi7A,426
+bizon/connectors/destinations/logger/src/destination.py,sha256=YUC_lAN5nrcrNAN90hnalKFAKX49KTDlJwdLfwTaC0U,2007
+bizon/connectors/sources/cycle/config/cycle.example.yml,sha256=UDiqOa-8ZsykmNT625kxq9tyXOj_gKe9CFwg9r_8SYk,230
+bizon/connectors/sources/cycle/src/source.py,sha256=6sXMneq59XZAT5oJseM9k6sGJaoQw4NDp8FTtg8lPhk,4213
+bizon/connectors/sources/cycle/tests/cycle_customers.py,sha256=A48S20LxIC0A74JLoFn4NTHNTgBWV_5stTFtF1Gfk2c,271
+bizon/connectors/sources/dummy/config/dummy.example.yml,sha256=Wvn8v644u1aKvRRsPzPPML4kYmWk0ZhMGrCPFoSbVZQ,331
+bizon/connectors/sources/dummy/src/fake_api.py,sha256=5EEETp9INOmDJ57FNXm6wnsRtofQmR1xYp8hUKjn0Es,3067
+bizon/connectors/sources/dummy/src/source.py,sha256=IAfNdd2CX8QzprAwLLVDGT3dd5a-bFa5BkkuRUTWGTc,4078
+bizon/connectors/sources/dummy/tests/dummy_pipeline.py,sha256=V9EKvugFXm3aNQVip-kMzlmvjlW6sZNGldxW1WdRa-E,397
+bizon/connectors/sources/dummy/tests/dummy_pipeline_bigquery_backend.py,sha256=SO_x_IH9iY1e09NHJlS1-CON0SI21fxGFqljSz4EmM4,542
+bizon/connectors/sources/dummy/tests/dummy_pipeline_kafka.py,sha256=zKhE7dhd6C4E_7VMepR8nSsGnesTJWXwlak-Dj01GD0,473
+bizon/connectors/sources/dummy/tests/dummy_pipeline_rabbitmq.py,sha256=A6KmSEOlWMK0UuIZhZxWO8y-z8h19rtWEV8DuhKX8Xs,686
+bizon/connectors/sources/dummy/tests/dummy_pipeline_unnest.py,sha256=64ZhjXUyZHzaIZTL_DO425yN6SlpB1vxzfhziE_pSEw,510
+bizon/connectors/sources/dummy/tests/dummy_pipeline_write_data_bigquery.py,sha256=oTV9HmOzjs3B2WwTPzDlIT_VkoYuXxy1BtSkeTGkQqw,686
+bizon/connectors/sources/dummy/tests/dummy_pipeline_write_data_bigquery_through_kafka.py,sha256=PFUhDuFw1Q1AMNMsnXPQxoqHIWf_wHEL1hLQodYlLcQ,596
+bizon/connectors/sources/gsheets/config/default_auth.example.yml,sha256=KOBp6MfO4uJwpwEYW0tJ4X5ctVwwdur9poJB4Ohba6s,348
+bizon/connectors/sources/gsheets/config/service_account.example.yml,sha256=XxVUnk9gGWc3lDb8CnzTHjTu8xz4Asyr5tXzY6qLvPg,1081
+bizon/connectors/sources/gsheets/config/service_account_incremental.example.yml,sha256=WGvAtw4aOwSMWrSZW0tHaRncZnGbI6gd4LJk1aHIP_c,1765
+bizon/connectors/sources/gsheets/src/source.py,sha256=xNF5FR9QLTM4kCiZ2eKZ5CZWNhLw6tyLaJZbliNzYnY,5675
+bizon/connectors/sources/gsheets/tests/gsheets_pipeline.py,sha256=lNSM3kZTd4W_-ajGIO3mdp8qGdEbnmWqsMm5pRiS0cw,181
+bizon/connectors/sources/hubspot/config/api_key.example.yml,sha256=VDTRloE5caqAdGdXgvsJZ6nQT46JHzX_YboxeGbpP18,389
+bizon/connectors/sources/hubspot/config/api_key_incremental.example.yml,sha256=g4SBeVEXSr3tCgy5VjgZPWkhnuvEZ0jl5nPNn3u05Jc,920
+bizon/connectors/sources/hubspot/config/oauth.example.yml,sha256=YqBtj1IxIsdM9E85_4eVWl6mPiHsQNoQn41EzCqORy0,499
+bizon/connectors/sources/hubspot/src/hubspot_base.py,sha256=THo8ImrPrIxeTuFcBMRJYwaDMstIfLIGjrQLE2cqqsU,3424
+bizon/connectors/sources/hubspot/src/hubspot_objects.py,sha256=ykqvxaFihv0e0A3-gGDmentp1KCGCoYvvDwZ3CcHzNg,6301
+bizon/connectors/sources/hubspot/src/models/hs_object.py,sha256=IuuMB_54kiEAZRDvl3AOVdktyZsLeYDRB2qoOVd1TZg,1234
+bizon/connectors/sources/hubspot/tests/hubspot_pipeline.py,sha256=e6dCF5_MHMySkeiF6kKrSAuCa_48J22-ZeSCZSjrfUI,216
+bizon/connectors/sources/kafka/config/kafka.example.yml,sha256=taIj3QUL3jynQCpO-YlDtt6nGvQp8hOzGkS0_RJFUQU,933
+bizon/connectors/sources/kafka/config/kafka_debezium.example.yml,sha256=lqFbNbSAFRh1Q8a9xQ_7nhhvVsyVIZM4amGEvCqHXQE,2254
+bizon/connectors/sources/kafka/config/kafka_streams.example.yml,sha256=aGiGMDMG6l0RUhrSVYl9djw1yy6T--2QOfwesuNhF1g,3368
+bizon/connectors/sources/kafka/src/callback.py,sha256=NgP9PLquHbVagz6E9VJK5Vx-kK8K2l80MhoeenbhOXY,484
+bizon/connectors/sources/kafka/src/config.py,sha256=Plb-d59PmQSie7vxAIPpJrwbpbA7pT6Xua0joR86nKI,2764
+bizon/connectors/sources/kafka/src/decode.py,sha256=RhPjazRQHb72D9iBhb763Nje7SH9t_6EKaFW-BtGVpM,2800
+bizon/connectors/sources/kafka/src/source.py,sha256=0Hv6viyVZGAd4azhQnqCteyHuwsbbDL4rSGEjMCff9E,19722
+bizon/connectors/sources/kafka/tests/kafka_pipeline.py,sha256=9LaCqXJIEx2ye3dkWq0YK_bPX7d4fCX_OcDOJCk34WE,206
+bizon/connectors/sources/notion/config/api_key.example.yml,sha256=TagqOqaho4u_G5ZP4L8je89Y4G_NvCo8s4Wf9e8yVH8,1061
+bizon/connectors/sources/notion/config/api_key_incremental.example.yml,sha256=52uQJo-SrqFny00zIVbA86qVq3asYHMFALqBcdmPmc8,1499
+bizon/connectors/sources/notion/src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/connectors/sources/notion/src/config.py,sha256=L-FZWijUa-aWK9VenWGsl6mv40i4ww46FacjYoX9gXo,1886
+bizon/connectors/sources/notion/src/source.py,sha256=aViwfLuBzsNGZHwU4-z-xI40cROJTvx7Tlkw3ApF3q8,66217
+bizon/connectors/sources/notion/tests/notion_pipeline.py,sha256=lyiD9b5uUF3oih8vY4gk7QXnfySGSawnbrBuSdTLym8,200
+bizon/connectors/sources/notion/tests/test_notion.py,sha256=-G0DbTLDS2Gc_Bx8xR2VXnY89vW64s1-puwPc9x2N7A,4029
+bizon/connectors/sources/periscope/config/periscope_charts.example.yml,sha256=9OgFDB7vguiNz2F2fmRqDNV8S_ddO9ncN5hgW9MhME4,350
+bizon/connectors/sources/periscope/config/periscope_dashboards.example.yml,sha256=mQey_04cfCzNpm-tYICDRUmPOFTZGSaTYI2OpK604d0,364
+bizon/connectors/sources/periscope/src/source.py,sha256=WePeQLn5hkPArhzgJ5gBCRu6VXbTpvOImUk7NN4mi0c,12270
+bizon/connectors/sources/periscope/tests/periscope_pipeline_charts.py,sha256=mU0JtfhS1KmWsS3iovGhGxK7iPVWiYzjBM_QfRL3ZQI,275
+bizon/connectors/sources/periscope/tests/periscope_pipeline_dashboard.py,sha256=VmkixgCMLTqNPXYXPYqcK7ncHeeR96h68IXPcuC70iA,290
+bizon/connectors/sources/pokeapi/config/pokeapi_pokemon_to_json.example.yml,sha256=H3FpBsyft5qSvuyurh-hhaASpq7GnehNeVAOOn31Txk,283
+bizon/connectors/sources/pokeapi/config/pokeapi_pokemon_to_logger.example.yml,sha256=wuJUob6QF1jwm5pnfNcFfXeO01HCcu-TEoZCeuYl_lo,134
+bizon/connectors/sources/pokeapi/src/source.py,sha256=OeDUXKciu7p5tBJ2LeA6oseMd-gqqpxk50g7DXVaODA,2537
+bizon/connectors/sources/sana_ai/config/sana.example.yml,sha256=sWOGxKMG9Cjgn11rNDCjcBL_YZdnUYvBLU7P9zfS3DE,802
+bizon/connectors/sources/sana_ai/src/source.py,sha256=ZAfv_OwkPXuwJF7KFe6eYQ0rmadTdot20X3L_gmTrs8,3212
+bizon/destination/buffer.py,sha256=tCwlm0iGo0RM5EeTnOiKI5Pz_JfHsJv6U3oIA9Va8_0,3155
+bizon/destination/config.py,sha256=dv_zHrFUs4X4paCglaKdSbBLfZRzfaNEQKUSVv0j8tU,3033
+bizon/destination/destination.py,sha256=qkEhnfCxI8JmCf43fyyFVpYuGiY6oc3pkVl5o2RaQeI,14614
+bizon/destination/models.py,sha256=_LUnkbbD_9XauYrNTthh9VmbYwWsVgPHF90FX6vmSjg,1278
+bizon/engine/config.py,sha256=tqc3K2i99BH_SSRRzfRpzSXoqux929Q6eg2oRsfjjcY,2038
+bizon/engine/engine.py,sha256=Xl8Bghx0kMlvST8S3uwxrcZ6cvqy525Kv94BOvAvr8U,1705
+bizon/engine/backend/backend.py,sha256=4J6uLKVh0hIPkOYO5XEg4ia8mPlOS13C3hSrIJ171VA,5849
+bizon/engine/backend/config.py,sha256=GhRzPWRGMaO-UJJRXkaqN_nlkFOCW6UOovwZLHLXrA8,900
+bizon/engine/backend/models.py,sha256=ECImDNji7u9eSkkjiw8sYq80l17vDF7MplTFvCpMgqA,5101
+bizon/engine/backend/adapters/sqlalchemy/backend.py,sha256=ipJ7eY_iiqjrvtq4NS39C5lH8VShMjXDAaApgTHJtpY,15435
+bizon/engine/backend/adapters/sqlalchemy/config.py,sha256=CeTWncVK27Y6lEKMVCF5RxD8Illhx2IQqqFkGrf0WKA,1845
+bizon/engine/pipeline/consumer.py,sha256=DtCR3mG791h35poYJdXjL9geNO-GWPKl_YC0zPsF5qI,3207
+bizon/engine/pipeline/models.py,sha256=qOra2MJGN6-PuouKpKuZRjutnQmzom0mgWDFZ16LcM8,405
+bizon/engine/pipeline/producer.py,sha256=XV2fR6CNMRlbYwqTl9mlqy6nkG37ODyh2aiiTZ371VM,11995
+bizon/engine/queue/config.py,sha256=0XwiQSB2OKTs-rODCSZqT5txNZzGOic2-PvODbcSrGg,1267
+bizon/engine/queue/queue.py,sha256=Y9uj31d-ZgW2f0F02iccp_o-m-RoMm_jR61NkLdMQ2M,3461
+bizon/engine/queue/adapters/kafka/config.py,sha256=ndNEXRT-nIgyWgoqlNXFhmlN206v87GobXIW9Z0zrSA,1085
+bizon/engine/queue/adapters/kafka/consumer.py,sha256=JPpvp3u5NXGvKRo60-ihRI4DlZxGJB1qH5Qw91XT8qc,2576
+bizon/engine/queue/adapters/kafka/queue.py,sha256=4baF7ns7LH3rQOjLjLSw2P7Ey4omRaVrxs_0atYyeSo,2000
+bizon/engine/queue/adapters/python_queue/config.py,sha256=_pyiIm1_PUjBo3WhKTATQcT2gazk-iHv0SdzNoGTeow,961
+bizon/engine/queue/adapters/python_queue/consumer.py,sha256=uVXOnS6tjhZG5NI_IU_qWg6-Uk3qjFJaMvDBDjQXjZg,1817
+bizon/engine/queue/adapters/python_queue/queue.py,sha256=jwKeqvJ4wzPbXIY1JnVrqBemRQ52A2tLzybwyJk4TL8,2670
+bizon/engine/queue/adapters/rabbitmq/config.py,sha256=9N_7WREvNjJgcNTC3Y2kHII-iId2MZa3ssHHks6PyAs,987
+bizon/engine/queue/adapters/rabbitmq/consumer.py,sha256=WlNHEqd5wZc31cwNLYYf_7YYPo2B3jYHTiyG5pW3Bj4,2037
+bizon/engine/queue/adapters/rabbitmq/queue.py,sha256=O0DfzNKqzKfUZCTOnEcbaHY_W67WbZkKDSeZAkh4Nf0,1744
+bizon/engine/runner/config.py,sha256=Clxa8ToeisaBH5BveZz06jSJeq8QRJTH-a_oXBIWxgw,2455
+bizon/engine/runner/runner.py,sha256=P5GJigzsbPQp7NKrX1pHqXdqmxiXHNtlrx3DB7vOA3Y,11067
+bizon/engine/runner/adapters/process.py,sha256=D1KJxK85GNvi4bGQ_YMoV44psWZvhNvqdMu-OfhNO5o,3049
+bizon/engine/runner/adapters/streaming.py,sha256=PPdhwA5oTNXH7NXB5-uye2asT4L0Zhfrx_wZFDDCsq4,8300
+bizon/engine/runner/adapters/thread.py,sha256=jQ2jzSe7zlZoPP_g46Oh70UN6RqJkGOMXC01SYg9s2o,3790
+bizon/monitoring/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/monitoring/config.py,sha256=wRZRfW4ejjFDSG6swWj7kIULlaNYUPdbIKblQC9lzsk,1112
+bizon/monitoring/monitor.py,sha256=aJ3JEgD-f-HydGFAmprwL8YCA0WumDeXpm2P-uqp4wM,2627
+bizon/monitoring/datadog/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/monitoring/datadog/monitor.py,sha256=YSdyMVEIjkDyp91_mGED_kx8j76MbQyQGkGJCijpAJ0,6152
+bizon/monitoring/noop/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+bizon/monitoring/noop/monitor.py,sha256=Pu7Qt9SpUG1UvC8aWysgtoDY-t5tnKd4FlUXAC4MjbI,1066
+bizon/source/callback.py,sha256=lfTwU_bzJwR0q5sbiKoK8uedQ-dhfHzoYkPVqm8b_Ho,602
+bizon/source/config.py,sha256=JyZbKjlU0xhiyuuIGJYJPGUl9JxS4xyGeCyHoHgHHos,2473
+bizon/source/cursor.py,sha256=Wjh9eNEiHV5P9YnjS5bdS2ahyFc0gPm9QLQtD-QjQCI,4089
+bizon/source/discover.py,sha256=h9IVqtAQsTH-XxR-UkAFgNvEphLP2LgataQCCuHbGrk,11174
+bizon/source/models.py,sha256=CHPKvO9chRi85WPDfLYy9vWnPsua8LTwYvjjN7Dj2uA,1837
+bizon/source/session.py,sha256=klbCv0g6sm6ac-pzM50eAJSP8DdQ9DOegHgjpmKKUrI,1978
+bizon/source/source.py,sha256=k_fHOOvam5ixZ9oPuQzUa9Kq3jVvv2HY7ghrCo-0o3I,4342
+bizon/source/auth/builder.py,sha256=hc4zBNj31LZc-QqgIyx1VQEYTm9Xv81vY5pJiwQroJo,860
+bizon/source/auth/config.py,sha256=2jjcBLP95XsCkfKxdUei4X2yHI2WX92lJb8D8Txw86g,750
+bizon/source/auth/authenticators/abstract_oauth.py,sha256=T4iv4IDeQgRn1d_0ODJeGA23PlieSpaD8w5zgLXPFI8,5573
+bizon/source/auth/authenticators/abstract_token.py,sha256=GYM4srti2VLYVuAvozv6AdqXrIzXw0HROzhvVq5YhCo,926
+bizon/source/auth/authenticators/basic.py,sha256=xMD9g1PCN-xBmGRT8R1zCiSrkqA_OpOb_pU54SgZoUk,1284
+bizon/source/auth/authenticators/cookies.py,sha256=mqNp6TGEfyam7ou-s-908Jh8R0ZeX89ftbeExDWg1oY,829
+bizon/source/auth/authenticators/oauth.py,sha256=tY_UZsWTy4FkifqJ7-smPaD61gg1dMJizO9_iSqTt5o,3670
+bizon/source/auth/authenticators/token.py,sha256=P6SKRAarAEv28YiWp8hQLSKAV7twNlyNTGRr9sxlx58,956
+bizon/transform/config.py,sha256=Q9F7jlsuaXK8OYrO5qcdk8lxXTDoIgzoVMhhHW3igEw,213
+bizon/transform/transform.py,sha256=Ufla8YFx9C9WEiN0ppmZS1a86Sk0PgggqC-8DIvDeAQ,1414
+bizon-0.3.0.dist-info/METADATA,sha256=oX7OZjHhKAVvQ8UiRS0ksqu3C65t2kOp2mAfXoEBdJY,11159
+bizon-0.3.0.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+bizon-0.3.0.dist-info/entry_points.txt,sha256=hHZPN-V6JwwhSYWNCKVu3WNxekuhXtIAaz_zdwO7NDo,45
+bizon-0.3.0.dist-info/licenses/LICENSE,sha256=OXLcl0T2SZ8Pmy2_dmlvKuetivmyPd5m1q-Gyd-zaYY,35149
+bizon-0.3.0.dist-info/RECORD,,

{bizon-0.1.2.dist-info → bizon-0.3.0.dist-info}/WHEEL

@@ -1,4 +1,4 @@
 Wheel-Version: 1.0
-Generator: poetry-core 2.1.3
+Generator: hatchling 1.28.0
 Root-Is-Purelib: true
 Tag: py3-none-any

bizon-0.3.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+bizon = bizon.cli.main:cli

bizon-0.1.2.dist-info/METADATA

@@ -1,179 +0,0 @@
-Metadata-Version: 2.3
-Name: bizon
-Version: 0.1.2
-Summary: Extract and load your data reliably from API Clients with native fault-tolerant and checkpointing mechanism.
-Author: Antoine Balliet
-Author-email: antoine.balliet@gmail.com
-Requires-Python: >=3.9,<3.13
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Provides-Extra: bigquery
-Provides-Extra: datadog
-Provides-Extra: gsheets
-Provides-Extra: kafka
-Provides-Extra: postgres
-Provides-Extra: rabbitmq
-Requires-Dist: avro (>=1.12.0,<2.0.0) ; extra == "kafka"
-Requires-Dist: backoff (>=2.2.1,<3.0.0)
-Requires-Dist: click (>=8.1.7,<9.0.0)
-Requires-Dist: confluent-kafka (>=2.6.0,<3.0.0) ; extra == "kafka"
-Requires-Dist: datadog (>=0.50.2,<0.51.0) ; extra == "datadog"
-Requires-Dist: ddtrace (>=3.10.0,<4.0.0) ; extra == "datadog"
-Requires-Dist: dpath (>=2.2.0,<3.0.0)
-Requires-Dist: fastavro (>=1.9.7,<2.0.0) ; extra == "kafka"
-Requires-Dist: google-cloud-bigquery (>=3.25.0,<4.0.0) ; extra == "bigquery"
-Requires-Dist: google-cloud-bigquery-storage (>=2.25.0,<3.0.0) ; extra == "bigquery"
-Requires-Dist: google-cloud-storage (>=2.17.0,<3.0.0)
-Requires-Dist: gspread (>=6.1.2,<7.0.0) ; extra == "gsheets"
-Requires-Dist: kafka-python (>=2.0.2,<3.0.0) ; extra == "kafka"
-Requires-Dist: loguru (>=0.7.2,<0.8.0)
-Requires-Dist: orjson (>=3.10.16,<4.0.0)
-Requires-Dist: pendulum (>=3.0.0,<4.0.0)
-Requires-Dist: pika (>=1.3.2,<2.0.0) ; extra == "rabbitmq"
-Requires-Dist: polars (>=1.16.0,<2.0.0)
-Requires-Dist: protobuf (>=4.24.0,<5.0.0) ; extra == "bigquery"
-Requires-Dist: psycopg2-binary (>=2.9.9,<3.0.0) ; extra == "postgres"
-Requires-Dist: pyarrow (>=16.1.0,<17.0.0)
-Requires-Dist: pydantic (>=2.8.2,<3.0.0)
-Requires-Dist: pydantic-extra-types (>=2.9.0,<3.0.0)
-Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
-Requires-Dist: pytz (>=2024.2,<2025.0)
-Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
-Requires-Dist: requests (>=2.28.2,<3.0.0)
-Requires-Dist: simplejson (>=3.20.1,<4.0.0)
-Requires-Dist: sqlalchemy (>=2.0.32,<3.0.0)
-Requires-Dist: sqlalchemy-bigquery (>=1.11.0,<2.0.0) ; extra == "bigquery"
-Requires-Dist: tenacity (>=9.0.0,<10.0.0)
-Description-Content-Type: text/markdown
-
-# bizon ⚡️
-Extract and load your largest data streams with a framework you can trust for billion records.
-
-## Features
-- **Natively fault-tolerant**: Bizon uses a checkpointing mechanism to keep track of the progress and recover from the last checkpoint.
-
-- **High throughput**: Bizon is designed to handle high throughput and can process billions of records.
-
-- **Queue system agnostic**: Bizon is agnostic of the queuing system, you can use any queuing system among Python Queue, RabbitMQ, Kafka or Redpanda. Thanks to the `bizon.engine.queue.Queue` interface, adapters can be written for any queuing system.
-
-- **Pipeline metrics**: Bizon provides exhaustive pipeline metrics and implement Datadog & OpenTelemetry for tracing. You can monitor:
-    - ETAs for completion
-    - Number of records processed
-    - Completion percentage
-    - Latency Source <> Destination
-
-- **Lightweight & lean**: Bizon is lightweight, minimal codebase and only uses few dependencies:
-    - `requests` for HTTP requests
-    - `pyyaml` for configuration
-    - `sqlalchemy` for database / warehouse connections
-    - `polars` for memory efficient data buffering and vectorized processing
-    - `pyarrow` for Parquet file format
-
-## Installation
-```bash
-pip install bizon
-```
-
-## Usage
-
-### List available sources and streams
-```bash
-bizon source list
-bizon stream list <source_name>
-```
-
-### Create a pipeline
-
-Create a file named `config.yml` in your working directory with the following content:
-
-```yaml
-name: demo-creatures-pipeline
-
-source:
-  name: dummy
-  stream: creatures
-  authentication:
-    type: api_key
-    params:
-      token: dummy_key
-
-destination:
-  name: logger
-  config:
-    dummy: dummy
-```
-
-Run the pipeline with the following command:
-
-```bash
-bizon run config.yml
-```
-## Backend configuration
-
-Backend is the interface used by Bizon to store its state. It can be configured in the `backend` section of the configuration file. The following backends are supported:
-- `sqlite`: In-memory SQLite database, useful for testing and development.
-- `bigquery`: Google BigQuery backend, perfect for light setup & production.
-- `postgres`: PostgreSQL backend, for production use and frequent cursor updates.
-
-## Queue configuration
-
-Queue is the interface used by Bizon to exchange data between `Source` and `Destination`. It can be configured in the `queue` section of the configuration file. The following queues are supported:
-- `python_queue`: Python Queue, useful for testing and development.
-- `rabbitmq`: RabbitMQ, for production use and high throughput.
-- `kafka`: Apache Kafka, for production use and high throughput and strong persistence.
-
-## Runner configuration
-
-Runner is the interface used by Bizon to run the pipeline. It can be configured in the `runner` section of the configuration file. The following runners are supported:
-- `thread` (asynchronous)
-- `process` (asynchronous)
-- `stream` (synchronous)
-
-## Start syncing your data 🚀
-
-### Quick setup without any dependencies ✌️
-
-Queue configuration can be set to `python_queue` and backend configuration to `sqlite`.
-This will allow you to test the pipeline without any external dependencies.
-
-
-### Local Kafka setup
-
-To test the pipeline with Kafka, you can use `docker compose` to setup Kafka or Redpanda locally.
-
-**Kafka**
-```bash
-docker compose --file ./scripts/kafka-compose.yml up # Kafka
-docker compose --file ./scripts/redpanda-compose.yml up # Redpanda
-```
-
-In your YAML configuration, set the `queue` configuration to Kafka under `engine`:
-```yaml
-engine:
-  queue:
-    type: kafka
-    config:
-      queue:
-        bootstrap_server: localhost:9092 # Kafka:9092 & Redpanda: 19092
-```
-
-**RabbitMQ**
-```bash
-docker compose --file ./scripts/rabbitmq-compose.yml up
-```
-
-In your YAML configuration, set the `queue` configuration to Kafka under `engine`:
-
-```yaml
-engine:
-  queue:
-    type: rabbitmq
-    config:
-      queue:
-        host: localhost
-        queue_name: bizon
-```
-