pam-python 0.2.2__tar.gz → 0.2.4__tar.gz

{pam_python-0.2.2 → pam_python-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pam-python
-Version: 0.2.2
+Version: 0.2.4
 Summary: Pam Python Library
 Author-email: Narongrit Kanhanoi <narongrit@pams.ai>
 Project-URL: Homepage, https://github.com/heart/pam-python
@@ -122,13 +122,13 @@ The runtime calls your service in two main phases.
 2. `on_data_input`
 
 - Called when CDP sends input files
-- `req.input_files` contains ordered Parquet files
+- `req.input_files` is the ordered positional data-source contract configured in PAM
 - `req.file_format` is the typed format sent by PAM
 - Should also return quickly (use a thread if needed)
 
 When your service is done:
 
-- Upload ordinary CDP result rows through `ResultBatchUploader` or `_upload_result(...)`
+- Upload ordinary CDP result rows through `ResultBatchUploader`
 - Publish managed reports through `self.reports`; do not build report JSON manually
 - Call `self._exit()` to signal completion
 
@@ -151,11 +151,11 @@ Notes:
 ---
 
 **Uploading Results in Batches**
-Polars templates pass `LazyFrame` results directly to `_upload_result(...)`, which
-streams them to the CSV upload boundary. Do not collect or convert to Pandas first.
-
-For eager Polars or Pandas DataFrames, use the batch uploader to handle chunking
-and flushing automatically.
+Polars `LazyFrame`/`DataFrame` and Pandas `DataFrame` results all pass through
+`ResultBatchUploader`. It streams lazy results in bounded chunks, buffers incomplete
+batches across calls, writes complete batches as CSV, and uploads them through PAM.
+The framework does not prescribe page-local/global computation or intermediate
+storage; submit rows when your business result is ready.
 
 Recommended usage:
 
@@ -165,6 +165,7 @@ from pam.result_batch_uploader import ResultBatchUploader
 batch_uploader = ResultBatchUploader(self, batch_size=50000)
 batch_uploader.upload(df, name="main")
 batch_uploader.flush()
+batch_uploader.wait_for_uploader()
 status = batch_uploader.get_status()
 ```
 
@@ -173,9 +174,12 @@ Notes:
 - `request.runtime_parameters["batch_size"]` overrides the constructor default when valid.
 - `name` separates result streams and enforces one stable column schema per stream.
 - `options` may be passed to `upload(...)` and are forwarded to `_upload_result`.
-- Complete batches upload immediately; `flush()` sends the remaining rows.
-- `get_status()` returns buffered rows, uploaded rows, and uploaded batch counts per stream.
-- `ResultBatchUploader` intentionally rejects Polars `LazyFrame`; upload it directly.
+- Uploads run in a bounded background queue so processing can continue while PAM receives CSV batches.
+- `flush()` closes input and queues the final remainder; it does not wait for network completion.
+- Always call `wait_for_uploader()` after `flush()` and before `_exit()`.
+- Failed uploads retry with exponential backoff, then log/record the failed batch and continue.
+- `get_status()` includes buffered, uploaded, failed, and retried counts per stream.
+- Polars `LazyFrame` is supported without collecting the complete result at once.
 
 ---
 
@@ -304,6 +308,5 @@ After `pam init` and one service:
 - Implement your logic in `functions.py`.
 - Wire it into `on_start` and `on_data_input` in your service class.
 - Use the temp utilities to write intermediate files.
-- Keep Polars transformations lazy and upload the resulting `LazyFrame` directly.
-- Use `ResultBatchUploader` only for eager Polars/Pandas frames that need row batching.
+- Keep Polars transformations lazy and submit final results through `ResultBatchUploader`.
 - Read `REPORTS.md` before implementing a managed report.

{pam_python-0.2.2 → pam_python-0.2.4}/README.md

@@ -102,13 +102,13 @@ The runtime calls your service in two main phases.
 2. `on_data_input`
 
 - Called when CDP sends input files
-- `req.input_files` contains ordered Parquet files
+- `req.input_files` is the ordered positional data-source contract configured in PAM
 - `req.file_format` is the typed format sent by PAM
 - Should also return quickly (use a thread if needed)
 
 When your service is done:
 
-- Upload ordinary CDP result rows through `ResultBatchUploader` or `_upload_result(...)`
+- Upload ordinary CDP result rows through `ResultBatchUploader`
 - Publish managed reports through `self.reports`; do not build report JSON manually
 - Call `self._exit()` to signal completion
 
@@ -131,11 +131,11 @@ Notes:
 ---
 
 **Uploading Results in Batches**
-Polars templates pass `LazyFrame` results directly to `_upload_result(...)`, which
-streams them to the CSV upload boundary. Do not collect or convert to Pandas first.
-
-For eager Polars or Pandas DataFrames, use the batch uploader to handle chunking
-and flushing automatically.
+Polars `LazyFrame`/`DataFrame` and Pandas `DataFrame` results all pass through
+`ResultBatchUploader`. It streams lazy results in bounded chunks, buffers incomplete
+batches across calls, writes complete batches as CSV, and uploads them through PAM.
+The framework does not prescribe page-local/global computation or intermediate
+storage; submit rows when your business result is ready.
 
 Recommended usage:
 
@@ -145,6 +145,7 @@ from pam.result_batch_uploader import ResultBatchUploader
 batch_uploader = ResultBatchUploader(self, batch_size=50000)
 batch_uploader.upload(df, name="main")
 batch_uploader.flush()
+batch_uploader.wait_for_uploader()
 status = batch_uploader.get_status()
 ```
 
@@ -153,9 +154,12 @@ Notes:
 - `request.runtime_parameters["batch_size"]` overrides the constructor default when valid.
 - `name` separates result streams and enforces one stable column schema per stream.
 - `options` may be passed to `upload(...)` and are forwarded to `_upload_result`.
-- Complete batches upload immediately; `flush()` sends the remaining rows.
-- `get_status()` returns buffered rows, uploaded rows, and uploaded batch counts per stream.
-- `ResultBatchUploader` intentionally rejects Polars `LazyFrame`; upload it directly.
+- Uploads run in a bounded background queue so processing can continue while PAM receives CSV batches.
+- `flush()` closes input and queues the final remainder; it does not wait for network completion.
+- Always call `wait_for_uploader()` after `flush()` and before `_exit()`.
+- Failed uploads retry with exponential backoff, then log/record the failed batch and continue.
+- `get_status()` includes buffered, uploaded, failed, and retried counts per stream.
+- Polars `LazyFrame` is supported without collecting the complete result at once.
 
 ---
 
@@ -284,6 +288,5 @@ After `pam init` and one service:
 - Implement your logic in `functions.py`.
 - Wire it into `on_start` and `on_data_input` in your service class.
 - Use the temp utilities to write intermediate files.
-- Keep Polars transformations lazy and upload the resulting `LazyFrame` directly.
-- Use `ResultBatchUploader` only for eager Polars/Pandas frames that need row batching.
+- Keep Polars transformations lazy and submit final results through `ResultBatchUploader`.
 - Read `REPORTS.md` before implementing a managed report.

{pam_python-0.2.2 → pam_python-0.2.4}/pam/__init__.py

@@ -1,3 +1,3 @@
 """PAM Data Plugin framework."""
 
-__version__ = "0.2.2"
+__version__ = "0.2.4"

{pam_python-0.2.2 → pam_python-0.2.4}/pam/cli.py

@@ -55,7 +55,7 @@ SERVICE_TEMPLATES = {
 }
 
 ENGINE_DEPENDENCIES = {
-    "polars": ["polars>=1.0.0"],
+    "polars": ["polars>=1.43.0"],
     "pandas": ["pandas>=2.2.3", "pyarrow>=19.0.1"],
 }
 

pam_python-0.2.4/pam/result_batch_uploader.py

@@ -0,0 +1,412 @@
+"""Asynchronous, bounded result batching for PAM CSV uploads."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import queue
+import threading
+import time
+from typing import Any, Dict, Optional, Tuple
+
+from pam.utils import log
+
+
+DEFAULT_BATCH_SIZE = 50000
+DEFAULT_QUEUE_SIZE = 4
+DEFAULT_MAX_RETRIES = 3
+DEFAULT_RETRY_DELAY_SECONDS = 2.0
+DEFAULT_RETRY_MAX_DELAY_SECONDS = 30.0
+_FLUSH = object()
+
+
+@dataclass
+class _ResultInput:
+    dataframe: Any
+    name: str
+    options: Optional[dict]
+
+
+@dataclass
+class _ResultStream:
+    engine: str
+    columns: Tuple[object, ...]
+    options: Optional[dict]
+    buffer: Any
+    uploaded_rows: int = 0
+    uploaded_batches: int = 0
+    failed_rows: int = 0
+    failed_batches: int = 0
+    retried_uploads: int = 0
+
+
+class ResultBatchUploader:
+    """Queue result frames and upload bounded CSV batches in one worker.
+
+    Plugin computation and intermediate storage are outside this class. Calls to
+    ``upload`` submit final result frames without waiting for network I/O. A
+    bounded queue applies backpressure when producers outrun the uploader.
+
+    ``flush`` closes input and places a FIFO completion marker. The worker first
+    consumes every earlier result, uploads all complete batches, then uploads all
+    remaining rows. ``wait_for_uploader`` is the completion barrier that must
+    return before a service exits.
+    """
+
+    def __init__(self, service, batch_size: int = DEFAULT_BATCH_SIZE):
+        self._service = service
+        self.batch_size = self._resolve_positive_int("batch_size", batch_size)
+        self.queue_size = self._resolve_positive_int(
+            "upload_queue_size", DEFAULT_QUEUE_SIZE
+        )
+        self.max_retries = self._resolve_non_negative_int(
+            "upload_max_retries", DEFAULT_MAX_RETRIES
+        )
+        self.retry_delay_seconds = self._resolve_non_negative_float(
+            "upload_retry_delay_seconds", DEFAULT_RETRY_DELAY_SECONDS
+        )
+        self.retry_max_delay_seconds = self._resolve_non_negative_float(
+            "upload_retry_max_delay_seconds", DEFAULT_RETRY_MAX_DELAY_SECONDS
+        )
+
+        self._streams: Dict[str, _ResultStream] = {}
+        self._queue: queue.Queue = queue.Queue(maxsize=self.queue_size)
+        self._submission_lock = threading.Lock()
+        self._state_lock = threading.RLock()
+        self._completion = threading.Event()
+        self._accepting = True
+        self._flush_requested = False
+        self._fatal_error: Optional[BaseException] = None
+        self._state = "open"
+
+        self._worker = threading.Thread(
+            target=self._run_worker,
+            name=f"{self.__class__.__name__}-worker",
+            daemon=True,
+        )
+        self._worker.start()
+
+    def upload(
+        self,
+        dataframe: Any,
+        name: str = "default",
+        options: Optional[dict] = None,
+    ) -> None:
+        """Submit a final result without waiting for CSV/network upload."""
+        stream_name = self._normalize_name(name)
+        normalized_options = self._normalize_options(options)
+        submitted = self._clone_submitted_dataframe(dataframe)
+
+        with self._submission_lock:
+            with self._state_lock:
+                if not self._accepting:
+                    raise RuntimeError("ResultBatchUploader no longer accepts results")
+                if self._fatal_error is not None:
+                    raise RuntimeError("ResultBatchUploader worker failed") from self._fatal_error
+            self._queue.put(_ResultInput(submitted, stream_name, normalized_options))
+
+    def flush(self) -> None:
+        """Close input and enqueue one terminal flush marker without blocking."""
+        with self._submission_lock:
+            with self._state_lock:
+                if self._flush_requested:
+                    return
+                self._accepting = False
+                self._flush_requested = True
+                self._state = "flush_requested"
+            self._queue.put(_FLUSH)
+
+    def wait_for_uploader(self) -> None:
+        """Wait for all queued/retried uploads and the final remainder."""
+        with self._state_lock:
+            if not self._flush_requested:
+                raise RuntimeError("flush() must be called before wait_for_uploader()")
+        self._completion.wait()
+        with self._state_lock:
+            if self._fatal_error is not None:
+                raise RuntimeError("ResultBatchUploader worker failed") from self._fatal_error
+
+    def get_status(self) -> Dict[str, dict]:
+        with self._state_lock:
+            return {
+                name: {
+                    "buffered_rows": len(stream.buffer),
+                    "uploaded_rows": stream.uploaded_rows,
+                    "uploaded_batches": stream.uploaded_batches,
+                    "failed_rows": stream.failed_rows,
+                    "failed_batches": stream.failed_batches,
+                    "retried_uploads": stream.retried_uploads,
+                }
+                for name, stream in self._streams.items()
+            }
+
+    @property
+    def state(self) -> str:
+        with self._state_lock:
+            return self._state
+
+    def _run_worker(self) -> None:
+        try:
+            while True:
+                item = self._queue.get()
+                try:
+                    if item is _FLUSH:
+                        if self._fatal_error is None:
+                            self._flush_all_streams()
+                        with self._state_lock:
+                            self._state = (
+                                "failed" if self._fatal_error is not None else "completed"
+                            )
+                        return
+                    if self._fatal_error is None:
+                        self._consume_input(item)
+                except BaseException as exc:  # keep worker alive until the flush barrier
+                    self._record_fatal_error(exc)
+                finally:
+                    self._queue.task_done()
+        finally:
+            with self._state_lock:
+                self._accepting = False
+                if self._state not in {"completed", "failed"}:
+                    self._state = "failed"
+            self._completion.set()
+
+    def _consume_input(self, item: _ResultInput) -> None:
+        dataframe = item.dataframe
+        if self._is_polars_lazyframe(dataframe):
+            for batch in dataframe.collect_batches(
+                chunk_size=self.batch_size,
+                maintain_order=True,
+                engine="streaming",
+            ):
+                self._consume_eager(batch, item.name, item.options)
+            return
+        self._consume_eager(dataframe, item.name, item.options)
+
+    def _consume_eager(
+        self,
+        dataframe: Any,
+        name: str,
+        options: Optional[dict],
+    ) -> None:
+        engine, normalized = self._normalize_eager_dataframe(dataframe)
+        batches = []
+        with self._state_lock:
+            stream = self._get_or_create_stream(name, engine, normalized, options)
+            if len(normalized) == 0:
+                return
+
+            remaining = normalized
+            if len(stream.buffer) > 0:
+                rows_needed = self.batch_size - len(stream.buffer)
+                stream.buffer = self._concat(
+                    engine,
+                    stream.buffer,
+                    self._slice(engine, remaining, 0, rows_needed),
+                )
+                remaining = self._slice(
+                    engine,
+                    remaining,
+                    rows_needed,
+                    len(remaining) - rows_needed,
+                )
+                if len(stream.buffer) == self.batch_size:
+                    batches.append(self._clone(engine, stream.buffer))
+                    stream.buffer = self._empty_frame(engine, stream.buffer)
+
+            while len(remaining) >= self.batch_size:
+                batches.append(self._slice(engine, remaining, 0, self.batch_size))
+                remaining = self._slice(
+                    engine,
+                    remaining,
+                    self.batch_size,
+                    len(remaining) - self.batch_size,
+                )
+
+            if len(remaining) > 0:
+                stream.buffer = self._clone(engine, remaining)
+
+        for batch in batches:
+            self._upload_batch(name, stream, batch)
+
+    def _flush_all_streams(self) -> None:
+        pending = []
+        with self._state_lock:
+            for name, stream in self._streams.items():
+                if len(stream.buffer) == 0:
+                    continue
+                pending.append((name, stream, self._clone(stream.engine, stream.buffer)))
+                stream.buffer = self._empty_frame(stream.engine, stream.buffer)
+        for name, stream, batch in pending:
+            self._upload_batch(name, stream, batch)
+
+    def _upload_batch(self, name: str, stream: _ResultStream, dataframe: Any) -> None:
+        attempts = self.max_retries + 1
+        for attempt in range(1, attempts + 1):
+            try:
+                self._service._upload_result(
+                    self._clone(stream.engine, dataframe),
+                    stream.options,
+                )
+                with self._state_lock:
+                    stream.uploaded_rows += len(dataframe)
+                    stream.uploaded_batches += 1
+                return
+            except Exception as exc:  # network/upload failures are retryable
+                if attempt < attempts:
+                    with self._state_lock:
+                        stream.retried_uploads += 1
+                    delay = min(
+                        self.retry_delay_seconds * (2 ** (attempt - 1)),
+                        self.retry_max_delay_seconds,
+                    )
+                    log(
+                        f"Result upload retry stream={name} rows={len(dataframe)} "
+                        f"attempt={attempt + 1}/{attempts} delay={delay}s error={exc}",
+                        level="WARNING",
+                    )
+                    if delay > 0:
+                        time.sleep(delay)
+                    continue
+
+                with self._state_lock:
+                    stream.failed_rows += len(dataframe)
+                    stream.failed_batches += 1
+                log(
+                    f"Result upload skipped after retries stream={name} "
+                    f"rows={len(dataframe)} attempts={attempts} error={exc}",
+                    level="ERROR",
+                )
+
+    def _record_fatal_error(self, exc: BaseException) -> None:
+        with self._state_lock:
+            if self._fatal_error is None:
+                self._fatal_error = exc
+                self._accepting = False
+        log(f"Result uploader worker failed: {exc}", level="ERROR")
+
+    def _get_or_create_stream(
+        self,
+        name: str,
+        engine: str,
+        dataframe: Any,
+        options: Optional[dict],
+    ) -> _ResultStream:
+        columns = tuple(dataframe.columns)
+        stream = self._streams.get(name)
+        if stream is None:
+            stream = _ResultStream(
+                engine=engine,
+                columns=columns,
+                options=options,
+                buffer=self._empty_frame(engine, dataframe),
+            )
+            self._streams[name] = stream
+            return stream
+        if stream.engine != engine:
+            raise ValueError(f"Result stream '{name}' dataframe engine changed")
+        if stream.columns != columns:
+            raise ValueError(
+                f"Result stream '{name}' columns changed from "
+                f"{list(stream.columns)!r} to {list(columns)!r}"
+            )
+        if stream.options != options:
+            raise ValueError(f"Result stream '{name}' upload options changed")
+        return stream
+
+    def _runtime_parameters(self) -> dict:
+        return getattr(self._service.request, "runtime_parameters", {})
+
+    def _resolve_positive_int(self, key: str, fallback: int) -> int:
+        try:
+            value = int(self._runtime_parameters().get(key, fallback))
+        except (TypeError, ValueError):
+            return fallback
+        return value if value > 0 else fallback
+
+    def _resolve_non_negative_int(self, key: str, fallback: int) -> int:
+        try:
+            value = int(self._runtime_parameters().get(key, fallback))
+        except (TypeError, ValueError):
+            return fallback
+        return value if value >= 0 else fallback
+
+    def _resolve_non_negative_float(self, key: str, fallback: float) -> float:
+        try:
+            value = float(self._runtime_parameters().get(key, fallback))
+        except (TypeError, ValueError):
+            return fallback
+        return value if value >= 0 else fallback
+
+    @classmethod
+    def _clone_submitted_dataframe(cls, dataframe: Any):
+        if cls._is_polars_lazyframe(dataframe):
+            return dataframe.clone()
+        _, normalized = cls._normalize_eager_dataframe(dataframe)
+        return normalized
+
+    @staticmethod
+    def _normalize_eager_dataframe(dataframe: Any) -> tuple[str, Any]:
+        module_name = type(dataframe).__module__.split(".", 1)[0]
+        class_name = type(dataframe).__name__
+        if module_name == "polars" and class_name == "DataFrame":
+            return "polars", dataframe.clone()
+        if module_name == "pandas" and class_name == "DataFrame":
+            return "pandas", dataframe.reset_index(drop=True).copy()
+        raise TypeError(
+            "dataframe must be a Polars LazyFrame, Polars DataFrame, "
+            "or Pandas DataFrame"
+        )
+
+    @staticmethod
+    def _is_polars_lazyframe(dataframe: Any) -> bool:
+        return (
+            type(dataframe).__module__.split(".", 1)[0] == "polars"
+            and type(dataframe).__name__ == "LazyFrame"
+        )
+
+    @staticmethod
+    def _clone(engine: str, dataframe: Any):
+        if engine == "polars":
+            return dataframe.clone()
+        return dataframe.reset_index(drop=True).copy()
+
+    @staticmethod
+    def _empty_frame(engine: str, dataframe: Any):
+        if engine == "polars":
+            return dataframe.head(0)
+        return dataframe.iloc[0:0].copy()
+
+    @staticmethod
+    def _slice(engine: str, dataframe: Any, offset: int, length: int):
+        length = max(0, length)
+        if engine == "polars":
+            return dataframe.slice(offset, length)
+        return dataframe.iloc[offset:offset + length].reset_index(drop=True)
+
+    @staticmethod
+    def _concat(engine: str, left: Any, right: Any):
+        if len(left) == 0:
+            return ResultBatchUploader._clone(engine, right)
+        if len(right) == 0:
+            return ResultBatchUploader._clone(engine, left)
+        if engine == "polars":
+            import polars as pl
+
+            return pl.concat([left, right], how="vertical")
+        import pandas as pd
+
+        return pd.concat([left, right], ignore_index=True)
+
+    @staticmethod
+    def _normalize_name(name: str) -> str:
+        if not isinstance(name, str) or not name.strip():
+            raise ValueError("name must be a non-empty string")
+        return name.strip()
+
+    @staticmethod
+    def _normalize_options(options: Optional[dict]) -> Optional[dict]:
+        if options is None:
+            return None
+        if not isinstance(options, dict):
+            raise ValueError("options must be a dictionary or None")
+        return dict(options)

{pam_python-0.2.2 → pam_python-0.2.4}/pam/task_manager.py

@@ -305,7 +305,10 @@ class TaskManager(ITaskManager):
 
     def service_upload_result(self, service: Service, file_path, options=None):
         """
-        Uploads a result file asynchronously and logs the response.
+        Uploads one result file synchronously.
+
+        ResultBatchUploader owns background concurrency and invokes this method
+        from its worker, so failures must propagate for retry handling.
         """
         endpoint = service.request.response_api
         payload = None
@@ -318,25 +321,21 @@ class TaskManager(ITaskManager):
                 is_priority = options.get("is_priority")
                 payload["is_priority"] = str(is_priority).lower() if isinstance(is_priority, bool) else is_priority
 
-        def handle_upload_response(response):
-            """Logs the response after the upload completes."""
-            if response is None:
-                log(f"Response from upload to {endpoint}: None")
-                return
-            try:
-                response_data = response.json()
-            except ValueError:
-                response_data = response.text
-            log(f"Response from upload to {endpoint}: {response_data}")
-
-        def upload_wrapper():
-            """Wrapper for the upload to handle response logging."""
-            response = self.api.http_upload(endpoint, file_path, payload)
-            handle_upload_response(response)
-
         log(f"Uploading Result to: {endpoint}")
-        http_thread = threading.Thread(target=upload_wrapper)
-        http_thread.start()
+        response = self.api.http_upload(endpoint, file_path, payload)
+        if response is None:
+            raise RuntimeError(f"No response from result upload to {endpoint}")
+        if not response.ok:
+            raise RuntimeError(
+                f"Result upload failed status={response.status_code} "
+                f"endpoint={endpoint}"
+            )
+        try:
+            response_data = response.json()
+        except ValueError:
+            response_data = response.text
+        log(f"Response from upload to {endpoint}: {response_data}")
+        return response
 
     def service_upload_report(self, service: Service, file_path):
         """