matrixone-python-sdk 0.1.10__tar.gz → 0.1.11__tar.gz

{matrixone_python_sdk-0.1.10/matrixone_python_sdk.egg-info → matrixone_python_sdk-0.1.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: matrixone-python-sdk
-Version: 0.1.10
+Version: 0.1.11
 Summary: A comprehensive Python SDK for MatrixOne database operations with vector search, fulltext search, and advanced features
 Home-page: https://github.com/matrixorigin/matrixone
 Author: MatrixOne Team
@@ -106,6 +106,7 @@ A comprehensive, high-level Python SDK for MatrixOne that provides SQLAlchemy-li
   - Export to local files or external stages (``stage://`` protocol)
   - Support for raw SQL, SQLAlchemy, and MatrixOne queries
   - Transaction-aware exports for consistency
+- 🔁 **Change Data Capture (CDC)**: Create, monitor, and control CDC tasks directly from Python (`client.cdc`, `async_client.cdc`)
 - 📸 **Snapshot Management**: Create and manage database snapshots at multiple levels
 - ⏰ **Point-in-Time Recovery**: PITR functionality for precise data recovery
 - 🔄 **Table Cloning**: Clone databases and tables efficiently
@@ -348,6 +349,152 @@ client.export.to_csv_stage('s3_stage', 'backup2.csv', stmt)
 client.disconnect()
 ```
 
+### CDC Task Management
+
+Create and control Change Data Capture (CDC) tasks directly from the Python SDK:
+
+```python
+from matrixone import Client, build_mysql_uri
+
+client = Client()
+client.connect(database='test')
+
+source_uri = build_mysql_uri('127.0.0.1', 6001, user='admin', password='111', account='acct')
+mysql_sink = build_mysql_uri('192.168.1.100', 3306, user='root', password='111')
+
+# Database-level replication (automatically ensures Level='database')
+client.cdc.create_database_task(
+    task_name='replicate_sales',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    source_database='sales',
+    sink_database='sales_archive',
+    options={'Frequency': '30m'}
+)
+
+# Table-level replication helper (list of tuples/dicts/strings)
+client.cdc.create_table_task(
+    task_name='replicate_key_tables',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    table_mappings=[('sales', 'orders', 'sales_archive', 'orders_backup'), 'sales.customers:sales_archive.customers']
+)
+
+# Pause/resume lifecycle
+client.cdc.pause('replicate_sales')
+client.cdc.resume('replicate_sales')
+
+# Inspect status and per-table watermarks
+tasks = client.cdc.list()
+watermarks = client.cdc.list_watermarks('replicate_sales')
+
+# Drop task when no longer needed
+client.cdc.drop('replicate_sales')
+client.cdc.drop('replicate_key_tables')
+
+client.disconnect()
+```
+
+> Async usage: call the same helpers via `await async_client.cdc.create_database_task(...)` / `create_table_task(...)` or inside `async with client.session()`.
+
+### CDC API Overview
+
+The :class:`matrixone.cdc.CDCManager` class provides high-level helpers for all CDC lifecycle operations. Typical workflows are:
+
+```python
+from matrixone import Client
+from matrixone.cdc import CDCManager
+
+client = Client()
+client.connect(host='127.0.0.1', port=6001, user='root', password='111', database='test')
+
+cdc = CDCManager(client)
+
+# Create a table-level task
+task = cdc.create_table_task(
+    task_name='orders_sync',
+    source_uri='mysql://sys#root:111@127.0.0.1:6001',
+    sink_type='matrixone',
+    sink_uri='mysql://sys#root:111@127.0.0.1:6001',
+    table_mappings=[('sales', 'orders', 'backup', 'orders')],
+    options={'Frequency': '1h', 'NoFull': True},
+)
+
+# Pause and resume
+cdc.pause(task.task_name)
+cdc.resume(task.task_name)
+
+# Inspect metadata
+info = cdc.get(task.task_name)
+print(info.state)
+
+# Clean up
+cdc.drop(task.task_name)
+client.disconnect()
+```
+
+Key helper methods:
+
+* `create()` – create a CDC task from a raw mapping string.
+* `create_database_task()` – convenience wrapper for database-level replication.
+* `create_table_task()` – convenience wrapper for table-level replication using Python data structures.
+* `pause()`/`resume()`/`restart()` – control CDC task state.
+* `list()`/`get()` – retrieve CDC task metadata as :class:`matrixone.cdc.CDCTaskInfo` objects.
+* `list_watermarks()` – read per-table watermarks as :class:`matrixone.cdc.CDCWatermarkInfo`.
+* `list_failing_tasks()` – return tasks whose `err_msg` field is populated.
+* `list_stuck_tasks()` – highlight running tasks that have per-table errors.
+* `list_late_table_watermarks()` – detect tables whose watermarks lag behind expected thresholds (supports custom per-task/per-table overrides).
+
+Refer to :class:`matrixone.cdc.CDCManager` docstrings or ``examples/example_31_cdc_operations.py`` for a comprehensive lifecycle example.
+
+```python
+from matrixone import Client, build_mysql_uri
+
+client = Client()
+client.connect(database='test')
+
+source_uri = build_mysql_uri('127.0.0.1', 6001, user='admin', password='111', account='acct')
+mysql_sink = build_mysql_uri('192.168.1.100', 3306, user='root', password='111')
+
+# Database-level replication (automatically ensures Level='database')
+client.cdc.create_database_task(
+    task_name='replicate_sales',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    source_database='sales',
+    sink_database='sales_archive',
+    options={'Frequency': '30m'}
+)
+
+# Table-level replication helper (list of tuples/dicts/strings)
+client.cdc.create_table_task(
+    task_name='replicate_key_tables',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    table_mappings=[('sales', 'orders', 'sales_archive', 'orders_backup'), 'sales.customers:sales_archive.customers']
+)
+
+# Pause/resume lifecycle
+client.cdc.pause('replicate_sales')
+client.cdc.resume('replicate_sales')
+
+# Inspect status and per-table watermarks
+tasks = client.cdc.list()
+watermarks = client.cdc.list_watermarks('replicate_sales')
+
+# Drop task when no longer needed
+client.cdc.drop('replicate_sales')
+client.cdc.drop('replicate_key_tables')
+
+client.disconnect()
+```
+
+> Async usage: call the same helpers via `await async_client.cdc.create_database_task(...)` / `create_table_task(...)` or inside `async with client.session()`.
+
 ### Wrapping Existing Sessions (For Legacy Code)
 
 If you have existing SQLAlchemy code, wrap your sessions to add MatrixOne features:
@@ -585,6 +732,26 @@ mo-diag -d test -c "sql SELECT COUNT(*) FROM my_table"
 mo-diag -d test -c "flush_table my_table"
 ```
 
+#### CDC Task Shortcuts
+
+CDC management is integrated with the CLI. The interactive commands (`cdc_tasks`,
+`cdc_task`, `cdc_create`, `cdc_drop`) now have non-interactive equivalents
+under `mo-diag cdc`, making automation straightforward.
+
+```bash
+# Summaries (mirrors cdc_tasks)
+mo-diag cdc show
+
+# Inspect a task and tighten watermark thresholds
+mo-diag cdc show nightly_sync --details --threshold=5m
+
+# Start the guided creator in table mode
+mo-diag cdc create --table-level
+
+# Drop a task without interactive prompts
+mo-diag cdc drop nightly_sync --force
+```
+
 ### Available Commands
 
 #### Index Management

{matrixone_python_sdk-0.1.10 → matrixone_python_sdk-0.1.11}/README.md

@@ -824,6 +824,26 @@ mo-diag -d test -c "sql SELECT COUNT(*) FROM my_table"
 mo-diag -d test -c "flush_table my_table"
 ```
 
+#### CDC Task Shortcuts
+
+Manage Change Data Capture directly from the shell or scripts. The interactive
+commands (`cdc_tasks`, `cdc_task`, `cdc_create`, `cdc_drop`) now have matching
+non-interactive helpers via `mo-diag cdc`.
+
+```bash
+# Summaries (same as running cdc_tasks inside the shell)
+mo-diag cdc show
+
+# Inspect a task with watermark threshold control
+mo-diag cdc show nightly_sync --details --threshold=5m
+
+# Launch the guided creator with a forced mode
+mo-diag cdc create --table-level
+
+# Drop a task with explicit confirmation override
+mo-diag cdc drop nightly_sync --force
+```
+
 ### Available Commands
 
 #### Index Management

{matrixone_python_sdk-0.1.10 → matrixone_python_sdk-0.1.11}/README_USER.md

@@ -38,6 +38,7 @@ A comprehensive, high-level Python SDK for MatrixOne that provides SQLAlchemy-li
   - Export to local files or external stages (``stage://`` protocol)
   - Support for raw SQL, SQLAlchemy, and MatrixOne queries
   - Transaction-aware exports for consistency
+- 🔁 **Change Data Capture (CDC)**: Create, monitor, and control CDC tasks directly from Python (`client.cdc`, `async_client.cdc`)
 - 📸 **Snapshot Management**: Create and manage database snapshots at multiple levels
 - ⏰ **Point-in-Time Recovery**: PITR functionality for precise data recovery
 - 🔄 **Table Cloning**: Clone databases and tables efficiently
@@ -280,6 +281,152 @@ client.export.to_csv_stage('s3_stage', 'backup2.csv', stmt)
 client.disconnect()
 ```
 
+### CDC Task Management
+
+Create and control Change Data Capture (CDC) tasks directly from the Python SDK:
+
+```python
+from matrixone import Client, build_mysql_uri
+
+client = Client()
+client.connect(database='test')
+
+source_uri = build_mysql_uri('127.0.0.1', 6001, user='admin', password='111', account='acct')
+mysql_sink = build_mysql_uri('192.168.1.100', 3306, user='root', password='111')
+
+# Database-level replication (automatically ensures Level='database')
+client.cdc.create_database_task(
+    task_name='replicate_sales',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    source_database='sales',
+    sink_database='sales_archive',
+    options={'Frequency': '30m'}
+)
+
+# Table-level replication helper (list of tuples/dicts/strings)
+client.cdc.create_table_task(
+    task_name='replicate_key_tables',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    table_mappings=[('sales', 'orders', 'sales_archive', 'orders_backup'), 'sales.customers:sales_archive.customers']
+)
+
+# Pause/resume lifecycle
+client.cdc.pause('replicate_sales')
+client.cdc.resume('replicate_sales')
+
+# Inspect status and per-table watermarks
+tasks = client.cdc.list()
+watermarks = client.cdc.list_watermarks('replicate_sales')
+
+# Drop task when no longer needed
+client.cdc.drop('replicate_sales')
+client.cdc.drop('replicate_key_tables')
+
+client.disconnect()
+```
+
+> Async usage: call the same helpers via `await async_client.cdc.create_database_task(...)` / `create_table_task(...)` or inside `async with client.session()`.
+
+### CDC API Overview
+
+The :class:`matrixone.cdc.CDCManager` class provides high-level helpers for all CDC lifecycle operations. Typical workflows are:
+
+```python
+from matrixone import Client
+from matrixone.cdc import CDCManager
+
+client = Client()
+client.connect(host='127.0.0.1', port=6001, user='root', password='111', database='test')
+
+cdc = CDCManager(client)
+
+# Create a table-level task
+task = cdc.create_table_task(
+    task_name='orders_sync',
+    source_uri='mysql://sys#root:111@127.0.0.1:6001',
+    sink_type='matrixone',
+    sink_uri='mysql://sys#root:111@127.0.0.1:6001',
+    table_mappings=[('sales', 'orders', 'backup', 'orders')],
+    options={'Frequency': '1h', 'NoFull': True},
+)
+
+# Pause and resume
+cdc.pause(task.task_name)
+cdc.resume(task.task_name)
+
+# Inspect metadata
+info = cdc.get(task.task_name)
+print(info.state)
+
+# Clean up
+cdc.drop(task.task_name)
+client.disconnect()
+```
+
+Key helper methods:
+
+* `create()` – create a CDC task from a raw mapping string.
+* `create_database_task()` – convenience wrapper for database-level replication.
+* `create_table_task()` – convenience wrapper for table-level replication using Python data structures.
+* `pause()`/`resume()`/`restart()` – control CDC task state.
+* `list()`/`get()` – retrieve CDC task metadata as :class:`matrixone.cdc.CDCTaskInfo` objects.
+* `list_watermarks()` – read per-table watermarks as :class:`matrixone.cdc.CDCWatermarkInfo`.
+* `list_failing_tasks()` – return tasks whose `err_msg` field is populated.
+* `list_stuck_tasks()` – highlight running tasks that have per-table errors.
+* `list_late_table_watermarks()` – detect tables whose watermarks lag behind expected thresholds (supports custom per-task/per-table overrides).
+
+Refer to :class:`matrixone.cdc.CDCManager` docstrings or ``examples/example_31_cdc_operations.py`` for a comprehensive lifecycle example.
+
+```python
+from matrixone import Client, build_mysql_uri
+
+client = Client()
+client.connect(database='test')
+
+source_uri = build_mysql_uri('127.0.0.1', 6001, user='admin', password='111', account='acct')
+mysql_sink = build_mysql_uri('192.168.1.100', 3306, user='root', password='111')
+
+# Database-level replication (automatically ensures Level='database')
+client.cdc.create_database_task(
+    task_name='replicate_sales',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    source_database='sales',
+    sink_database='sales_archive',
+    options={'Frequency': '30m'}
+)
+
+# Table-level replication helper (list of tuples/dicts/strings)
+client.cdc.create_table_task(
+    task_name='replicate_key_tables',
+    source_uri=source_uri,
+    sink_type='mysql',
+    sink_uri=mysql_sink,
+    table_mappings=[('sales', 'orders', 'sales_archive', 'orders_backup'), 'sales.customers:sales_archive.customers']
+)
+
+# Pause/resume lifecycle
+client.cdc.pause('replicate_sales')
+client.cdc.resume('replicate_sales')
+
+# Inspect status and per-table watermarks
+tasks = client.cdc.list()
+watermarks = client.cdc.list_watermarks('replicate_sales')
+
+# Drop task when no longer needed
+client.cdc.drop('replicate_sales')
+client.cdc.drop('replicate_key_tables')
+
+client.disconnect()
+```
+
+> Async usage: call the same helpers via `await async_client.cdc.create_database_task(...)` / `create_table_task(...)` or inside `async with client.session()`.
+
 ### Wrapping Existing Sessions (For Legacy Code)
 
 If you have existing SQLAlchemy code, wrap your sessions to add MatrixOne features:
@@ -517,6 +664,26 @@ mo-diag -d test -c "sql SELECT COUNT(*) FROM my_table"
 mo-diag -d test -c "flush_table my_table"
 ```
 
+#### CDC Task Shortcuts
+
+CDC management is integrated with the CLI. The interactive commands (`cdc_tasks`,
+`cdc_task`, `cdc_create`, `cdc_drop`) now have non-interactive equivalents
+under `mo-diag cdc`, making automation straightforward.
+
+```bash
+# Summaries (mirrors cdc_tasks)
+mo-diag cdc show
+
+# Inspect a task and tighten watermark thresholds
+mo-diag cdc show nightly_sync --details --threshold=5m
+
+# Start the guided creator in table mode
+mo-diag cdc create --table-level
+
+# Drop a task without interactive prompts
+mo-diag cdc drop nightly_sync --force
+```
+
 ### Available Commands
 
 #### Index Management