datatoolpack 0.10.0__tar.gz → 0.11.0__tar.gz

{datatoolpack-0.10.0 → datatoolpack-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datatoolpack
-Version: 0.10.0
+Version: 0.11.0
 Summary: Official Python SDK for the AutoData ML data preparation pipeline API
 Home-page: https://autodata.datatoolpack.com
 Author: AutoData Team
@@ -62,7 +62,7 @@ with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
     result = client.process(
         file_path="data.csv",
         target_columns=["price"],
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 ```
@@ -162,7 +162,7 @@ client.close()
 result = client.process(
     file_path="data.csv",           # Path to input file (CSV, XLSX, Parquet, ...)
     target_columns=["price"],       # y-column(s) for ML
-    output_rows=20000,              # Target row count in output
+    output_rows=10000,              # Target row count in the synthetic output (default 10000)
     tools={                         # Toggle pipeline steps (all optional)
         "anomaly": False,           # Anomaly detection (off by default)
         "dtc": True,                # Data Type Conversion
@@ -199,7 +199,7 @@ result = client.process(
         {"name": "dsg.csv", "url": "/download/.../dsg.csv", "size": 2097152, "description": "..."},
         {"name": "dsm_train.csv", ...},
     ],
-    "row_count": 20000,
+    "row_count": 10000,
     "duration_seconds": 42.1,
 }
 ```
@@ -289,6 +289,72 @@ client.download_file("/api/v1/download/abc123.../dsg.csv", "dsg.csv")
 
 ---
 
+## Feature Selection (F4)
+
+Rank a completed session's columns by predictive importance — **without re-running the pipeline**. Handy for trimming wide datasets before training.
+
+### `client.recommend_features(session_id, target_columns, methods, top_k)`
+
+```python
+ranking = client.recommend_features(
+    session_id="abc123...",            # a completed session you own
+    target_columns=["price"],          # rank features against these target(s)
+    methods=["mutual_information"],    # one or more methods (see below); default ["mutual_information"]
+    top_k=20,                          # number of top features to return (default 20)
+)
+# {
+#   "ranking":    [{"feature": "sqft", "score": 0.81}, ...],   # combined across methods
+#   "per_method": {"mutual_information": [...], "shap_importance": [...]},
+#   "warnings":   [...],
+# }
+```
+
+**Available methods:** `variance_threshold`, `correlation_pruning`, `mutual_information`, `rfe`, `shap_importance`, `pca`. Pass several to blend their rankings, e.g. `methods=["mutual_information", "shap_importance"]`. (SHAP-based ranking uses a fast LightGBM model on the server; it degrades gracefully to `mutual_information` if those libraries are unavailable.)
+
+---
+
+## Inference & Retraining (v0.11.0+)
+
+Apply a **completed** session's fitted transforms to new data — the missing
+half of the ML lifecycle: train once, then score fresh production rows with
+*exactly* the same encoding, imputation and scaling.
+
+### `client.infer(...)` — Score new data
+
+Lightweight pipeline (Anomaly Detection → DTC → MDH → CDS). Outputs
+`features.csv` (+ `y_columns.csv` if targets are present in the new file).
+
+```python
+result = client.infer(
+    original_session_id="5d3bc0f0-...",   # a completed training session you own
+    file_path="new_rows.csv",             # columns must match the training data
+    download_path="./scored",             # optional: save outputs locally
+    return_dataframes=True,               # optional: get pandas DataFrames back
+)
+features_df = result["dataframes"]["features"]
+```
+
+### `client.retrain(...)` — Refresh a training set with new rows
+
+Full pipeline including Split → CDS → DSM → DSG; produces
+`retraining_output.csv` (features + targets combined).
+
+```python
+result = client.retrain(
+    original_session_id="5d3bc0f0-...",
+    file_path="new_rows.csv",
+    run_dsg=True, output_rows=20000,      # synthetic augmentation target
+    download_path="./retrained",
+)
+```
+
+Both calls are **synchronous** (the response arrives when processing ends;
+default per-request timeout is ≥ 900 s — raise `timeout=` for very large
+files), require the original session to be **completed and owned by you**,
+and are free of credit charges in v1.
+
+---
+
 ## API Keys & Usage
 
 ### `client.list_keys()` — List API keys
@@ -336,6 +402,8 @@ Read data directly from databases and cloud storage instead of uploading files.
 | `kafka`       | Apache Kafka                       | `bootstrap_servers`, `topic`, `group_id`            |
 | `kinesis`     | Amazon Kinesis                     | `stream_name`, `region`, `access_key_id`, `secret_access_key` |
 
+> **More connectors:** AutoData supports 30+ source types. Beyond the common ones above you can also pass `oracle`, `redshift`, `cassandra`, `dynamodb`, `elasticsearch`, `clickhouse`, `timescaledb`, `synapse`, `azure_blob`, `adls`, `sap_hana`, `mqtt`, `opcua`, `influxdb`, `pi_web`, `salesforce`, `hubspot`, `stripe`, `netsuite`, `google_sheets`, `ga4`, `sharepoint`, and `http`/`rest` as `connector_type`. See **Dashboard → Documentation → Connector Data Sources** for each type's required `secrets` fields.
+
 ### Quick connector example
 
 ```python
@@ -363,7 +431,7 @@ with AutoDataClient(api_key="dtpk_...") as client:
         table="orders",
         target_columns=["price"],
         secrets={"connection_string": "postgresql://user:pass@host/db"},
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 ```
@@ -452,7 +520,7 @@ result = client.process_from_connector(
     table="orders",
     target_columns=["price"],
     secrets={"connection_string": "postgresql://..."},
-    output_rows=20000,
+    output_rows=10000,
     incremental=True,          # Only fetch new rows since last sync
     type_casts={"age": "int"}, # Override column types before processing
     wait=True,
@@ -474,7 +542,7 @@ client.write_output(
     if_exists="replace",       # "replace", "append", "fail", or "merge"
     merge_keys=["id"],         # column names for merge matching (when if_exists='merge')
 )
-# {"success": True, "rows_written": 20000, "table": "ml_prepared_data"}
+# {"success": True, "rows_written": 10000, "table": "ml_prepared_data"}
 ```
 
 ---
@@ -639,6 +707,65 @@ client.delete_scheduled_run(run_id="run-id")
 
 ---
 
+## Triggers (F7)
+
+Auto-start a pipeline when an external event fires — an inbound webhook, a connector watermark advancing, another pipeline completing, or a quality alert. Triggers are feature-flagged on the server (`FEATURE_FLAG_TRIGGERS`); if the flag is off, these endpoints return `404`.
+
+### `client.create_trigger(name, trigger_type, config, target_pipeline_config, enabled)`
+
+```python
+trigger = client.create_trigger(
+    name="Reprocess on new S3 batch",
+    trigger_type="inbound_webhook",   # inbound_webhook | watermark_advance | pipeline_completion | quality_alert
+    config={},                         # type-specific (see table below)
+    target_pipeline_config={           # the pipeline to run when the trigger fires
+        "source_type": "s3",
+        "credential_id": "cred-uuid",
+        "table": "incoming/sales.csv",
+        "target_columns": ["revenue"],
+        "output_rows": 10000,
+    },
+    enabled=True,
+)
+# For inbound_webhook the secret is returned ONLY once — save it now:
+print(trigger["webhook_secret"], trigger["inbound_url"])
+```
+
+**Trigger types & their `config`:**
+
+| `trigger_type`        | `config` keys |
+|-----------------------|---------------|
+| `inbound_webhook`     | optional `{"hmac_key": "..."}` — require HMAC-SHA256-signed requests |
+| `watermark_advance`   | `{"credential_id", "table", "watermark_column", "min_advance"}` |
+| `pipeline_completion` | `{}` — chain off another pipeline finishing |
+| `quality_alert`       | `{}` — fire when a quality-alert rule trips |
+
+```python
+# List / fetch / update / delete
+triggers = client.list_triggers()
+t        = client.get_trigger(trigger_id)
+client.update_trigger(trigger_id, enabled=False)   # name | enabled | config | target_pipeline_config
+client.delete_trigger(trigger_id)
+
+# Dry-run: validate config + target WITHOUT firing
+report = client.test_trigger(trigger_id)
+# {"would_fire": True, "checks": {...}}
+```
+
+**Firing an inbound webhook** from your own system:
+
+```bash
+curl -X POST "https://autodata.datatoolpack.com/trigger/inbound/<webhook_secret>" \
+     -H "Content-Type: application/json" \
+     -d '{"any": "payload"}'
+# If you set an hmac_key, also sign the raw body:
+#   X-Signature: sha256=<hex HMAC-SHA256 of the request body>
+```
+
+Inbound firing is rate-limited and HMAC-verified server-side; repeated failures auto-disable the trigger.
+
+---
+
 ## Folder Listeners
 
 Automatically trigger pipelines when new files appear in cloud storage:
@@ -819,6 +946,11 @@ with AutoDataClient() as client:
 | `list_keys()` | List all API keys for the account |
 | `get_usage()` | Get credit usage statistics |
 
+### Feature Selection
+| Method | Description |
+|--------|-------------|
+| `recommend_features(session_id, target_columns, ...)` | Rank a completed session's features (F4) |
+
 ### Connectors
 | Method | Description |
 |--------|-------------|
@@ -866,6 +998,16 @@ with AutoDataClient() as client:
 | `toggle_scheduled_run(run_id)` | Enable/disable a scheduled run |
 | `run_scheduled_now(run_id)` | Trigger a scheduled run immediately |
 
+### Triggers
+| Method | Description |
+|--------|-------------|
+| `list_triggers()` | List your triggers |
+| `create_trigger(name, trigger_type, ...)` | Create an event trigger (F7) |
+| `get_trigger(trigger_id)` | Fetch one trigger |
+| `update_trigger(trigger_id, ...)` | Update a trigger |
+| `delete_trigger(trigger_id)` | Delete a trigger |
+| `test_trigger(trigger_id)` | Dry-run a trigger without firing |
+
 ### Folder Listeners
 | Method | Description |
 |--------|-------------|

{datatoolpack-0.10.0 → datatoolpack-0.11.0}/README.md

@@ -25,7 +25,7 @@ with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
     result = client.process(
         file_path="data.csv",
         target_columns=["price"],
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 ```
@@ -125,7 +125,7 @@ client.close()
 result = client.process(
     file_path="data.csv",           # Path to input file (CSV, XLSX, Parquet, ...)
     target_columns=["price"],       # y-column(s) for ML
-    output_rows=20000,              # Target row count in output
+    output_rows=10000,              # Target row count in the synthetic output (default 10000)
     tools={                         # Toggle pipeline steps (all optional)
         "anomaly": False,           # Anomaly detection (off by default)
         "dtc": True,                # Data Type Conversion
@@ -162,7 +162,7 @@ result = client.process(
         {"name": "dsg.csv", "url": "/download/.../dsg.csv", "size": 2097152, "description": "..."},
         {"name": "dsm_train.csv", ...},
     ],
-    "row_count": 20000,
+    "row_count": 10000,
     "duration_seconds": 42.1,
 }
 ```
@@ -252,6 +252,72 @@ client.download_file("/api/v1/download/abc123.../dsg.csv", "dsg.csv")
 
 ---
 
+## Feature Selection (F4)
+
+Rank a completed session's columns by predictive importance — **without re-running the pipeline**. Handy for trimming wide datasets before training.
+
+### `client.recommend_features(session_id, target_columns, methods, top_k)`
+
+```python
+ranking = client.recommend_features(
+    session_id="abc123...",            # a completed session you own
+    target_columns=["price"],          # rank features against these target(s)
+    methods=["mutual_information"],    # one or more methods (see below); default ["mutual_information"]
+    top_k=20,                          # number of top features to return (default 20)
+)
+# {
+#   "ranking":    [{"feature": "sqft", "score": 0.81}, ...],   # combined across methods
+#   "per_method": {"mutual_information": [...], "shap_importance": [...]},
+#   "warnings":   [...],
+# }
+```
+
+**Available methods:** `variance_threshold`, `correlation_pruning`, `mutual_information`, `rfe`, `shap_importance`, `pca`. Pass several to blend their rankings, e.g. `methods=["mutual_information", "shap_importance"]`. (SHAP-based ranking uses a fast LightGBM model on the server; it degrades gracefully to `mutual_information` if those libraries are unavailable.)
+
+---
+
+## Inference & Retraining (v0.11.0+)
+
+Apply a **completed** session's fitted transforms to new data — the missing
+half of the ML lifecycle: train once, then score fresh production rows with
+*exactly* the same encoding, imputation and scaling.
+
+### `client.infer(...)` — Score new data
+
+Lightweight pipeline (Anomaly Detection → DTC → MDH → CDS). Outputs
+`features.csv` (+ `y_columns.csv` if targets are present in the new file).
+
+```python
+result = client.infer(
+    original_session_id="5d3bc0f0-...",   # a completed training session you own
+    file_path="new_rows.csv",             # columns must match the training data
+    download_path="./scored",             # optional: save outputs locally
+    return_dataframes=True,               # optional: get pandas DataFrames back
+)
+features_df = result["dataframes"]["features"]
+```
+
+### `client.retrain(...)` — Refresh a training set with new rows
+
+Full pipeline including Split → CDS → DSM → DSG; produces
+`retraining_output.csv` (features + targets combined).
+
+```python
+result = client.retrain(
+    original_session_id="5d3bc0f0-...",
+    file_path="new_rows.csv",
+    run_dsg=True, output_rows=20000,      # synthetic augmentation target
+    download_path="./retrained",
+)
+```
+
+Both calls are **synchronous** (the response arrives when processing ends;
+default per-request timeout is ≥ 900 s — raise `timeout=` for very large
+files), require the original session to be **completed and owned by you**,
+and are free of credit charges in v1.
+
+---
+
 ## API Keys & Usage
 
 ### `client.list_keys()` — List API keys
@@ -299,6 +365,8 @@ Read data directly from databases and cloud storage instead of uploading files.
 | `kafka`       | Apache Kafka                       | `bootstrap_servers`, `topic`, `group_id`            |
 | `kinesis`     | Amazon Kinesis                     | `stream_name`, `region`, `access_key_id`, `secret_access_key` |
 
+> **More connectors:** AutoData supports 30+ source types. Beyond the common ones above you can also pass `oracle`, `redshift`, `cassandra`, `dynamodb`, `elasticsearch`, `clickhouse`, `timescaledb`, `synapse`, `azure_blob`, `adls`, `sap_hana`, `mqtt`, `opcua`, `influxdb`, `pi_web`, `salesforce`, `hubspot`, `stripe`, `netsuite`, `google_sheets`, `ga4`, `sharepoint`, and `http`/`rest` as `connector_type`. See **Dashboard → Documentation → Connector Data Sources** for each type's required `secrets` fields.
+
 ### Quick connector example
 
 ```python
@@ -326,7 +394,7 @@ with AutoDataClient(api_key="dtpk_...") as client:
         table="orders",
         target_columns=["price"],
         secrets={"connection_string": "postgresql://user:pass@host/db"},
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 ```
@@ -415,7 +483,7 @@ result = client.process_from_connector(
     table="orders",
     target_columns=["price"],
     secrets={"connection_string": "postgresql://..."},
-    output_rows=20000,
+    output_rows=10000,
     incremental=True,          # Only fetch new rows since last sync
     type_casts={"age": "int"}, # Override column types before processing
     wait=True,
@@ -437,7 +505,7 @@ client.write_output(
     if_exists="replace",       # "replace", "append", "fail", or "merge"
     merge_keys=["id"],         # column names for merge matching (when if_exists='merge')
 )
-# {"success": True, "rows_written": 20000, "table": "ml_prepared_data"}
+# {"success": True, "rows_written": 10000, "table": "ml_prepared_data"}
 ```
 
 ---
@@ -602,6 +670,65 @@ client.delete_scheduled_run(run_id="run-id")
 
 ---
 
+## Triggers (F7)
+
+Auto-start a pipeline when an external event fires — an inbound webhook, a connector watermark advancing, another pipeline completing, or a quality alert. Triggers are feature-flagged on the server (`FEATURE_FLAG_TRIGGERS`); if the flag is off, these endpoints return `404`.
+
+### `client.create_trigger(name, trigger_type, config, target_pipeline_config, enabled)`
+
+```python
+trigger = client.create_trigger(
+    name="Reprocess on new S3 batch",
+    trigger_type="inbound_webhook",   # inbound_webhook | watermark_advance | pipeline_completion | quality_alert
+    config={},                         # type-specific (see table below)
+    target_pipeline_config={           # the pipeline to run when the trigger fires
+        "source_type": "s3",
+        "credential_id": "cred-uuid",
+        "table": "incoming/sales.csv",
+        "target_columns": ["revenue"],
+        "output_rows": 10000,
+    },
+    enabled=True,
+)
+# For inbound_webhook the secret is returned ONLY once — save it now:
+print(trigger["webhook_secret"], trigger["inbound_url"])
+```
+
+**Trigger types & their `config`:**
+
+| `trigger_type`        | `config` keys |
+|-----------------------|---------------|
+| `inbound_webhook`     | optional `{"hmac_key": "..."}` — require HMAC-SHA256-signed requests |
+| `watermark_advance`   | `{"credential_id", "table", "watermark_column", "min_advance"}` |
+| `pipeline_completion` | `{}` — chain off another pipeline finishing |
+| `quality_alert`       | `{}` — fire when a quality-alert rule trips |
+
+```python
+# List / fetch / update / delete
+triggers = client.list_triggers()
+t        = client.get_trigger(trigger_id)
+client.update_trigger(trigger_id, enabled=False)   # name | enabled | config | target_pipeline_config
+client.delete_trigger(trigger_id)
+
+# Dry-run: validate config + target WITHOUT firing
+report = client.test_trigger(trigger_id)
+# {"would_fire": True, "checks": {...}}
+```
+
+**Firing an inbound webhook** from your own system:
+
+```bash
+curl -X POST "https://autodata.datatoolpack.com/trigger/inbound/<webhook_secret>" \
+     -H "Content-Type: application/json" \
+     -d '{"any": "payload"}'
+# If you set an hmac_key, also sign the raw body:
+#   X-Signature: sha256=<hex HMAC-SHA256 of the request body>
+```
+
+Inbound firing is rate-limited and HMAC-verified server-side; repeated failures auto-disable the trigger.
+
+---
+
 ## Folder Listeners
 
 Automatically trigger pipelines when new files appear in cloud storage:
@@ -782,6 +909,11 @@ with AutoDataClient() as client:
 | `list_keys()` | List all API keys for the account |
 | `get_usage()` | Get credit usage statistics |
 
+### Feature Selection
+| Method | Description |
+|--------|-------------|
+| `recommend_features(session_id, target_columns, ...)` | Rank a completed session's features (F4) |
+
 ### Connectors
 | Method | Description |
 |--------|-------------|
@@ -829,6 +961,16 @@ with AutoDataClient() as client:
 | `toggle_scheduled_run(run_id)` | Enable/disable a scheduled run |
 | `run_scheduled_now(run_id)` | Trigger a scheduled run immediately |
 
+### Triggers
+| Method | Description |
+|--------|-------------|
+| `list_triggers()` | List your triggers |
+| `create_trigger(name, trigger_type, ...)` | Create an event trigger (F7) |
+| `get_trigger(trigger_id)` | Fetch one trigger |
+| `update_trigger(trigger_id, ...)` | Update a trigger |
+| `delete_trigger(trigger_id)` | Delete a trigger |
+| `test_trigger(trigger_id)` | Dry-run a trigger without firing |
+
 ### Folder Listeners
 | Method | Description |
 |--------|-------------|

{datatoolpack-0.10.0 → datatoolpack-0.11.0}/datatoolpack/__init__.py

@@ -1,4 +1,4 @@
 from .client import AutoDataClient, AutoDataError
 
-__version__ = "0.10.0"
+__version__ = "0.11.0"
 __all__ = ["AutoDataClient", "AutoDataError", "__version__"]

{datatoolpack-0.10.0 → datatoolpack-0.11.0}/datatoolpack/client.py

@@ -12,7 +12,7 @@ Usage::
     result = client.process(
         file_path="data.csv",
         target_columns=["price"],
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 """
@@ -789,7 +789,8 @@ class AutoDataClient:
                 Object storage / data lakes / files:
                   ``s3``, ``gcs``, ``azure_blob``, ``adls_gen2`` (Azure
                   Data Lake Storage Gen2), ``delta`` (Delta Lake),
-                  ``google_sheets``.
+                  ``google_sheets``, ``google_drive``, ``dropbox``,
+                  ``onedrive``, ``box``.
 
                 Streaming & IoT / time-series:
                   ``kafka``, ``kinesis``, ``mqtt``, ``opcua`` (OPC-UA),
@@ -975,7 +976,7 @@ class AutoDataClient:
             "excluded_columns": adv.get("excluded_columns", []),
             "text_mode": adv.get("text_mode", 0),
             "text_cleaning": adv.get("text_cleaning", True),
-            "mdh_mode": adv.get("mdh_mode", 1),
+            "mdh_mode": adv.get("mdh_mode", 0),
             "zscore_limit": adv.get("zscore_limit", 3.0),
             "similarity_p": adv.get("similarity_p", 95),
             "dsg_mode": adv.get("dsg_mode", "copula"),
@@ -1307,6 +1308,150 @@ class AutoDataClient:
         self._raise_for_error(r)
         return r.json()
 
+    # ------------------------------------------------------------------
+    # Inference & retraining — apply a COMPLETED session to new data
+    # ------------------------------------------------------------------
+
+    def infer(
+        self,
+        original_session_id: str,
+        file_path: str,
+        enable_anomaly_detection: Optional[bool] = None,
+        download_path: Optional[str] = None,
+        return_dataframes: bool = False,
+        timeout: Optional[float] = None,
+    ) -> Dict:
+        """Score a new CSV with a completed session's fitted transforms.
+
+        Runs the lightweight inference pipeline server-side (Anomaly
+        Detection → DTC → MDH → CDS) using the artifacts the original
+        training run stored, so new rows get *exactly* the same encoding,
+        imputation and scaling as the training data. Synchronous — the call
+        returns when the server finishes; raise ``timeout`` for big files.
+
+        Args:
+            original_session_id: A COMPLETED pipeline session owned by the
+                                 caller (the ``session_id`` returned by
+                                 :meth:`process` / :meth:`process_from_connector`).
+            file_path:           Local CSV with new rows to score. Column
+                                 names must match the original training data.
+            enable_anomaly_detection: Override the original session's anomaly
+                                 setting; ``None`` inherits it.
+            download_path:       Directory to save ``features.csv`` /
+                                 ``y_columns.csv`` into (skipped when None).
+            return_dataframes:   Also load the CSV outputs into a
+                                 ``dataframes`` dict (requires pandas).
+            timeout:             Per-request timeout in seconds
+                                 (default ``max(client timeout, 900)``).
+
+        Returns:
+            Server result dict: ``status``, ``new_session_id``,
+            ``output_files`` — plus ``files`` (local paths) when downloaded
+            and ``dataframes`` when requested.
+        """
+        extra: Dict[str, str] = {}
+        if enable_anomaly_detection is not None:
+            extra["enableAnomalyDetection"] = (
+                "true" if enable_anomaly_detection else "false")
+        return self._apply_session(
+            "/inference", original_session_id, file_path, extra,
+            download_path, return_dataframes, timeout)
+
+    def retrain(
+        self,
+        original_session_id: str,
+        file_path: str,
+        run_dsm: bool = True,
+        run_dsg: bool = True,
+        output_rows: Optional[int] = None,
+        enable_anomaly_detection: Optional[bool] = None,
+        download_path: Optional[str] = None,
+        return_dataframes: bool = False,
+        timeout: Optional[float] = None,
+    ) -> Dict:
+        """Re-run the FULL pipeline on new data with a completed session's config.
+
+        Unlike :meth:`infer` this includes the downstream stages
+        (… → Split → CDS → DSM → DSG) and produces ``retraining_output.csv``
+        (features + targets combined) — use it to refresh a training set with
+        newly arrived rows. Synchronous, like :meth:`infer`.
+
+        Args:
+            original_session_id: A COMPLETED session owned by the caller.
+            file_path:           Local CSV with the new training rows.
+            run_dsm:             Run the data-splitting stage (default True).
+            run_dsg:             Run synthetic generation (default True).
+            output_rows:         DSG target row count (None = server default).
+            enable_anomaly_detection: Override; ``None`` inherits the original.
+            download_path:       Directory to save the output files into.
+            return_dataframes:   Also load CSV outputs as DataFrames.
+            timeout:             Per-request timeout (default ≥ 900s).
+
+        Returns:
+            Server result dict: ``status``, ``new_session_id``,
+            ``output_files`` [+ ``files`` / ``dataframes``].
+        """
+        extra = {
+            "run_dsm": "true" if run_dsm else "false",
+            "run_dsg": "true" if run_dsg else "false",
+        }
+        if output_rows is not None:
+            extra["output_sample_size"] = str(output_rows)
+        if enable_anomaly_detection is not None:
+            extra["enableAnomalyDetection"] = (
+                "true" if enable_anomaly_detection else "false")
+        return self._apply_session(
+            "/retraining", original_session_id, file_path, extra,
+            download_path, return_dataframes, timeout)
+
+    def _apply_session(
+        self,
+        endpoint: str,
+        original_session_id: str,
+        file_path: str,
+        extra_form: Dict[str, str],
+        download_path: Optional[str],
+        return_dataframes: bool,
+        timeout: Optional[float],
+    ) -> Dict:
+        """Shared upload/response handling for :meth:`infer` / :meth:`retrain`."""
+        data = {"original_session_id": original_session_id}
+        data.update(extra_form)
+        effective_timeout = timeout if timeout is not None else max(self.timeout, 900)
+        with open(file_path, "rb") as f:
+            r = self._request(
+                "POST",
+                self._url(endpoint),
+                files={"file": (os.path.basename(file_path), f, "text/csv")},
+                data=data,
+                timeout=effective_timeout,
+            )
+        self._raise_for_error(r)
+        result = r.json()
+
+        new_session_id = result.get("new_session_id")
+        outputs = result.get("output_files") or []
+        if new_session_id and (download_path or return_dataframes):
+            import tempfile
+            target_dir = download_path or tempfile.mkdtemp(prefix="autodata_infer_")
+            files: Dict[str, str] = {}
+            for out in outputs:
+                name = out.get("name")
+                if not name:
+                    continue
+                local = os.path.join(target_dir, name)
+                self.download_file(
+                    self._url(f"/download/{new_session_id}/{name}"), local)
+                files[name] = local
+            result["files"] = files
+            if return_dataframes:
+                pd = self._require_pandas()
+                result["dataframes"] = {
+                    os.path.splitext(name)[0]: pd.read_csv(path)
+                    for name, path in files.items() if name.endswith(".csv")
+                }
+        return result
+
     def apply_mapping(
         self,
         session_id: str,

{datatoolpack-0.10.0 → datatoolpack-0.11.0}/datatoolpack.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datatoolpack
-Version: 0.10.0
+Version: 0.11.0
 Summary: Official Python SDK for the AutoData ML data preparation pipeline API
 Home-page: https://autodata.datatoolpack.com
 Author: AutoData Team
@@ -62,7 +62,7 @@ with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
     result = client.process(
         file_path="data.csv",
         target_columns=["price"],
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 ```
@@ -162,7 +162,7 @@ client.close()
 result = client.process(
     file_path="data.csv",           # Path to input file (CSV, XLSX, Parquet, ...)
     target_columns=["price"],       # y-column(s) for ML
-    output_rows=20000,              # Target row count in output
+    output_rows=10000,              # Target row count in the synthetic output (default 10000)
     tools={                         # Toggle pipeline steps (all optional)
         "anomaly": False,           # Anomaly detection (off by default)
         "dtc": True,                # Data Type Conversion
@@ -199,7 +199,7 @@ result = client.process(
         {"name": "dsg.csv", "url": "/download/.../dsg.csv", "size": 2097152, "description": "..."},
         {"name": "dsm_train.csv", ...},
     ],
-    "row_count": 20000,
+    "row_count": 10000,
     "duration_seconds": 42.1,
 }
 ```
@@ -289,6 +289,72 @@ client.download_file("/api/v1/download/abc123.../dsg.csv", "dsg.csv")
 
 ---
 
+## Feature Selection (F4)
+
+Rank a completed session's columns by predictive importance — **without re-running the pipeline**. Handy for trimming wide datasets before training.
+
+### `client.recommend_features(session_id, target_columns, methods, top_k)`
+
+```python
+ranking = client.recommend_features(
+    session_id="abc123...",            # a completed session you own
+    target_columns=["price"],          # rank features against these target(s)
+    methods=["mutual_information"],    # one or more methods (see below); default ["mutual_information"]
+    top_k=20,                          # number of top features to return (default 20)
+)
+# {
+#   "ranking":    [{"feature": "sqft", "score": 0.81}, ...],   # combined across methods
+#   "per_method": {"mutual_information": [...], "shap_importance": [...]},
+#   "warnings":   [...],
+# }
+```
+
+**Available methods:** `variance_threshold`, `correlation_pruning`, `mutual_information`, `rfe`, `shap_importance`, `pca`. Pass several to blend their rankings, e.g. `methods=["mutual_information", "shap_importance"]`. (SHAP-based ranking uses a fast LightGBM model on the server; it degrades gracefully to `mutual_information` if those libraries are unavailable.)
+
+---
+
+## Inference & Retraining (v0.11.0+)
+
+Apply a **completed** session's fitted transforms to new data — the missing
+half of the ML lifecycle: train once, then score fresh production rows with
+*exactly* the same encoding, imputation and scaling.
+
+### `client.infer(...)` — Score new data
+
+Lightweight pipeline (Anomaly Detection → DTC → MDH → CDS). Outputs
+`features.csv` (+ `y_columns.csv` if targets are present in the new file).
+
+```python
+result = client.infer(
+    original_session_id="5d3bc0f0-...",   # a completed training session you own
+    file_path="new_rows.csv",             # columns must match the training data
+    download_path="./scored",             # optional: save outputs locally
+    return_dataframes=True,               # optional: get pandas DataFrames back
+)
+features_df = result["dataframes"]["features"]
+```
+
+### `client.retrain(...)` — Refresh a training set with new rows
+
+Full pipeline including Split → CDS → DSM → DSG; produces
+`retraining_output.csv` (features + targets combined).
+
+```python
+result = client.retrain(
+    original_session_id="5d3bc0f0-...",
+    file_path="new_rows.csv",
+    run_dsg=True, output_rows=20000,      # synthetic augmentation target
+    download_path="./retrained",
+)
+```
+
+Both calls are **synchronous** (the response arrives when processing ends;
+default per-request timeout is ≥ 900 s — raise `timeout=` for very large
+files), require the original session to be **completed and owned by you**,
+and are free of credit charges in v1.
+
+---
+
 ## API Keys & Usage
 
 ### `client.list_keys()` — List API keys
@@ -336,6 +402,8 @@ Read data directly from databases and cloud storage instead of uploading files.
 | `kafka`       | Apache Kafka                       | `bootstrap_servers`, `topic`, `group_id`            |
 | `kinesis`     | Amazon Kinesis                     | `stream_name`, `region`, `access_key_id`, `secret_access_key` |
 
+> **More connectors:** AutoData supports 30+ source types. Beyond the common ones above you can also pass `oracle`, `redshift`, `cassandra`, `dynamodb`, `elasticsearch`, `clickhouse`, `timescaledb`, `synapse`, `azure_blob`, `adls`, `sap_hana`, `mqtt`, `opcua`, `influxdb`, `pi_web`, `salesforce`, `hubspot`, `stripe`, `netsuite`, `google_sheets`, `ga4`, `sharepoint`, and `http`/`rest` as `connector_type`. See **Dashboard → Documentation → Connector Data Sources** for each type's required `secrets` fields.
+
 ### Quick connector example
 
 ```python
@@ -363,7 +431,7 @@ with AutoDataClient(api_key="dtpk_...") as client:
         table="orders",
         target_columns=["price"],
         secrets={"connection_string": "postgresql://user:pass@host/db"},
-        output_rows=20000,
+        output_rows=10000,
     )
     print(result["files"])
 ```
@@ -452,7 +520,7 @@ result = client.process_from_connector(
     table="orders",
     target_columns=["price"],
     secrets={"connection_string": "postgresql://..."},
-    output_rows=20000,
+    output_rows=10000,
     incremental=True,          # Only fetch new rows since last sync
     type_casts={"age": "int"}, # Override column types before processing
     wait=True,
@@ -474,7 +542,7 @@ client.write_output(
     if_exists="replace",       # "replace", "append", "fail", or "merge"
     merge_keys=["id"],         # column names for merge matching (when if_exists='merge')
 )
-# {"success": True, "rows_written": 20000, "table": "ml_prepared_data"}
+# {"success": True, "rows_written": 10000, "table": "ml_prepared_data"}
 ```
 
 ---
@@ -639,6 +707,65 @@ client.delete_scheduled_run(run_id="run-id")
 
 ---
 
+## Triggers (F7)
+
+Auto-start a pipeline when an external event fires — an inbound webhook, a connector watermark advancing, another pipeline completing, or a quality alert. Triggers are feature-flagged on the server (`FEATURE_FLAG_TRIGGERS`); if the flag is off, these endpoints return `404`.
+
+### `client.create_trigger(name, trigger_type, config, target_pipeline_config, enabled)`
+
+```python
+trigger = client.create_trigger(
+    name="Reprocess on new S3 batch",
+    trigger_type="inbound_webhook",   # inbound_webhook | watermark_advance | pipeline_completion | quality_alert
+    config={},                         # type-specific (see table below)
+    target_pipeline_config={           # the pipeline to run when the trigger fires
+        "source_type": "s3",
+        "credential_id": "cred-uuid",
+        "table": "incoming/sales.csv",
+        "target_columns": ["revenue"],
+        "output_rows": 10000,
+    },
+    enabled=True,
+)
+# For inbound_webhook the secret is returned ONLY once — save it now:
+print(trigger["webhook_secret"], trigger["inbound_url"])
+```
+
+**Trigger types & their `config`:**
+
+| `trigger_type`        | `config` keys |
+|-----------------------|---------------|
+| `inbound_webhook`     | optional `{"hmac_key": "..."}` — require HMAC-SHA256-signed requests |
+| `watermark_advance`   | `{"credential_id", "table", "watermark_column", "min_advance"}` |
+| `pipeline_completion` | `{}` — chain off another pipeline finishing |
+| `quality_alert`       | `{}` — fire when a quality-alert rule trips |
+
+```python
+# List / fetch / update / delete
+triggers = client.list_triggers()
+t        = client.get_trigger(trigger_id)
+client.update_trigger(trigger_id, enabled=False)   # name | enabled | config | target_pipeline_config
+client.delete_trigger(trigger_id)
+
+# Dry-run: validate config + target WITHOUT firing
+report = client.test_trigger(trigger_id)
+# {"would_fire": True, "checks": {...}}
+```
+
+**Firing an inbound webhook** from your own system:
+
+```bash
+curl -X POST "https://autodata.datatoolpack.com/trigger/inbound/<webhook_secret>" \
+     -H "Content-Type: application/json" \
+     -d '{"any": "payload"}'
+# If you set an hmac_key, also sign the raw body:
+#   X-Signature: sha256=<hex HMAC-SHA256 of the request body>
+```
+
+Inbound firing is rate-limited and HMAC-verified server-side; repeated failures auto-disable the trigger.
+
+---
+
 ## Folder Listeners
 
 Automatically trigger pipelines when new files appear in cloud storage:
@@ -819,6 +946,11 @@ with AutoDataClient() as client:
 | `list_keys()` | List all API keys for the account |
 | `get_usage()` | Get credit usage statistics |
 
+### Feature Selection
+| Method | Description |
+|--------|-------------|
+| `recommend_features(session_id, target_columns, ...)` | Rank a completed session's features (F4) |
+
 ### Connectors
 | Method | Description |
 |--------|-------------|
@@ -866,6 +998,16 @@ with AutoDataClient() as client:
 | `toggle_scheduled_run(run_id)` | Enable/disable a scheduled run |
 | `run_scheduled_now(run_id)` | Trigger a scheduled run immediately |
 
+### Triggers
+| Method | Description |
+|--------|-------------|
+| `list_triggers()` | List your triggers |
+| `create_trigger(name, trigger_type, ...)` | Create an event trigger (F7) |
+| `get_trigger(trigger_id)` | Fetch one trigger |
+| `update_trigger(trigger_id, ...)` | Update a trigger |
+| `delete_trigger(trigger_id)` | Delete a trigger |
+| `test_trigger(trigger_id)` | Dry-run a trigger without firing |
+
 ### Folder Listeners
 | Method | Description |
 |--------|-------------|

{datatoolpack-0.10.0 → datatoolpack-0.11.0}/setup.py

@@ -7,7 +7,7 @@ with open(os.path.join(here, "README.md"), encoding="utf-8") as f:
 
 setup(
     name="datatoolpack",
-    version="0.10.0",
+    version="0.11.0",
     description="Official Python SDK for the AutoData ML data preparation pipeline API",
     long_description=long_description,
     long_description_content_type="text/markdown",