pam-python 0.1.32__tar.gz → 0.1.34__tar.gz

{pam_python-0.1.32 → pam_python-0.1.34}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pam-python
-Version: 0.1.32
+Version: 0.1.34
 Summary: Pam Python Library
 Home-page: https://github.com/heart/pam-python
 Author: Narongrit Kanhanoi

{pam_python-0.1.32 → pam_python-0.1.34}/pam/service.py

@@ -80,6 +80,12 @@ class Service:
         if not isinstance(report_json, dict):
             raise ValueError("report_json must be a dictionary")
 
+        try:
+            pretty_report_json = json.dumps(report_json, indent=2, ensure_ascii=False, sort_keys=True)
+        except Exception:
+            pretty_report_json = str(report_json)
+        log(f"{self.request.service_name}: report_json\n{pretty_report_json}")
+
         adapter_id = get_adapter_id(self.request.response_api)
         service_name = self.request.service_name
         token = self.request.token
@@ -129,6 +135,11 @@ class Service:
         :param sqlite_file: Path to the SQLite file to upload.
         :return: The uploaded file name if successful, or empty string if failed.
         """
+        dry_run = self.request.runtime_parameters.get("dry_run", "false")
+        if dry_run is not None and dry_run.lower() == "true":
+            log("Dry run mode will not upload sqlite.")
+            return sqlite_file
+
         return self.task_manager.service_upload_sqlite(self, file_name, is_shared, sqlite_file)
 
     def _exit(self) -> None:

pam_python-0.1.34/pam/templates/init/AGENT.md

@@ -0,0 +1,419 @@
+# AGENT
+
+This document defines the **data plugin runtime specification** and execution contract so developers and agents can create services and understand runtime behavior inside the CDP environment.
+
+This is not a tutorial.
+This document defines behavioral rules, lifecycle expectations, and safety constraints enforced by the platform.
+
+---
+
+# 1. Runtime Overview
+
+## Project Overview
+
+1. This project is a data plugin that will be connected to the main system in production.
+2. The data plugin processes data from CDP and is created as separate services by domain. For example, if we process RFM, we create a new service dedicated to that domain.
+
+CDP spawns a service instance when a job starts and terminates it after the job completes. This design supports parallel execution. For example, two runs will spawn two separate instances. The only risk is shared local files (e.g., sqlite) which can cause conflicts.
+
+The lifecycle is wired in `rfm_segment/Rfm_segmentSvc.py` (example service name).
+
+---
+
+# 2. Service Creation Contract
+
+## Create a New Service
+
+1. Create a service with:
+   `pam new service [service name]`
+2. The system creates a new folder named `[service name]` with sample code and basic templates.
+3. Example:
+   `pam new service rfm_segment`
+   This generates `rfm_segment/Rfm_segmentSvc.py`, which is the starting point of the service.
+4. `rfm_segment/service.yaml` registers the service automatically.
+   The value `endpoint: /Rfm_segmentSvc` means CDP will call:
+   `http://ip-of-this-k8s-pod/services/Rfm_segmentSvc`
+
+**IMPORTANT**
+You MUST activate the virtual environment before running `pam` commands.
+
+---
+
+## Service scaffolding (MUST use pam generator)
+
+### Rules
+
+- Do NOT create new files or folders manually for service scaffolding.
+- Always generate the service by running the `pam` generator command via a Python virtualenv that has `pam` installed.
+- Only edit files AFTER the generator has created the full template.
+- If the generator cannot be executed, STOP and report the exact error. Do NOT hand-roll templates.
+
+### Why (important)
+
+The `pam` CLI is a Python console_script pointing to `pam.cli:main` (source: `pam/cli.py`).
+Do NOT rely on `activate` because the agent may execute commands in separate processes.
+
+### Virtualenv discovery
+
+Prefer an existing venv in the repo:
+
+- `.venv/`
+- `venv/`
+- otherwise: any directory containing `pyvenv.cfg` at its root
+
+### Run generator (preferred order)
+
+Once a venv is found, run ONE of the following:
+
+#### 1. Run through venv python as a module (preferred)
+
+- macOS/Linux:
+  - `<VENV>/bin/python -m pam.cli new service <service_name>`
+
+- Windows:
+  - `<VENV>\Scripts\python -m pam.cli new service <service_name>`
+
+#### 2. If the above fails, run the console_script shim created by pip (still must be inside the venv)
+
+- macOS/Linux:
+  - `<VENV>/bin/pam new service <service_name>`
+
+- Windows:
+  - `<VENV>\Scripts\pam new service <service_name>`
+
+### Hard stop condition
+
+- If no venv is found OR `pam` cannot be imported/executed from the venv:
+  - STOP and report the error.
+  - Do NOT create the template manually.
+
+### Verification step (required)
+
+- After running the generator, list generated files and confirm the expected structure exists before editing anything.
+
+---
+
+## Code Organization Guidelines
+
+- Separate each lifecycle step into clear, dedicated functions. Add comments at critical points so humans can review and audit later.
+- Avoid nested functions unless absolutely necessary.
+- Put service logic in `[service_name]/functions.py`, then import and use it from the service class to keep the overall code easy to read.
+
+---
+
+# 3. Lifecycle Execution Contract
+
+## on_start
+
+`on_start` runs at startup. Use it to read runtime parameters from CDP and initialize work.
+
+Example:
+
+```python
+def on_start(self):
+    log("on_start")
+
+    some_param = self.request.runtime_parameters.get("some_param", "")
+    self._request_data()
+```
+
+`runtime_parameters` are configured by CDP to control service behavior without code changes (e.g., `period_day="7"`).
+
+Notes
+
+- All values are strings. Cast with defaults to avoid errors.
+- `dry_run` is the only built-in parameter. If missing, it runs for real. If `dry_run="true"`, the service runs fully but does not upload results to CDP.
+- You do not need to check `dry_run` manually. Always call `_upload_result`; the system handles it.
+
+**Important:** `on_start` must return quickly because CDP calls via HTTP. If it hangs, the client may time out and the job will be terminated. If work is long-running, move it to a thread and return from `on_start` first. This applies to all lifecycle functions.
+
+---
+
+## Request Data from CDP
+
+Call `self._request_data()` to ask CDP for input data. CDP returns up to 5 CSV files (each file is a separate event).
+
+- File order is guaranteed per agreement, so you can index by position.
+- Column names and record counts are defined by CDP configuration.
+
+CSV example:
+
+```csv
+_id,id,data1,data2,data3
+xx,yy,zz,nn,mm
+```
+
+Column meaning
+
+- `_id` = event id for dedup/processing checks. Often unused and can be dropped.
+- `id` = customer id used by CDP as a key.
+- Other columns depend on the agreement.
+
+Date format is always:
+`"02/08/2026, 18:23:54"`
+Timezone may vary by customer data (usually Bangkok time).
+You may sometimes be asked for the exact time zone if it is logically important for the program.
+
+---
+
+## on_data_input
+
+When CDP finishes collecting data, it calls `on_data_input`.
+
+Example:
+
+```python
+def on_data_input(self, req: RequestCommand):
+    log(f"on_data_input req.is_end = {req.is_end}")
+
+    # RUN IN A SEPARATED THREAD TO PREVENT HTTP REQUEST BLOCKING
+    thread = threading.Thread(
+        target=lambda: self.__run_process_data_in_thread(req),
+        daemon=True,
+    )
+    thread.start()
+
+
+def __run_process_data_in_thread(self, req: RequestCommand):
+    dataframe = self.__process_data(req.input_files)
+    self._upload_result(dataframe)
+
+    if not req.is_end:
+        self._request_data(req.next)
+    else:
+        self._exit()
+```
+
+Why a thread: to end the HTTP request quickly and avoid timeouts.
+
+`req.input_files` is the ordered list of CSV paths.
+
+---
+
+## Upload Results to CDP
+
+After processing, call `_upload_result` with a dataframe. Example output:
+
+```csv
+id,data_x,rfm
+xxx,20,hero
+```
+
+- `id` is required for CDP matching.
+- Other columns are defined by agreement.
+
+Sometimes you may include an `event` column to create per-user events:
+
+```csv
+id,data_x,event
+xxx,20,purchase
+yyy,50,refund
+```
+
+If `event` is missing, CDP uses the default event configured in the system.
+
+---
+
+## Pagination
+
+```python
+if not req.is_end:
+    self._request_data(req.next)
+else:
+    self._exit()
+```
+
+- `req.is_end` indicates whether this is the last page.
+- If not end, call `self._request_data(req.next)` for the next page.
+- If end, call `self._exit()` to terminate the instance.
+
+CDP will not send the next page until you call `_request_data`, so the flow is serialized and no two threads will handle the same data unless you explicitly design it that way.
+
+---
+
+## Paging Model (Customer-based Pagination Contract)
+
+CDP paging is **customer-based**, not row-based.
+
+When requesting data page by page:
+
+- CDP collects customers in batches of approximately **10,000 customers per page**.
+- All events belonging to those customers are merged into **a single CSV file**.
+- The number of rows is **unknown** and depends on how many events each customer has.
+
+This means:
+
+- Each page represents a logical customer group, not a fixed number of records.
+- A page may contain a small or very large number of rows depending on behavior density.
+
+### Execution Implications
+
+Because paging is based on customers:
+
+- In many cases, a service can process **one page (≈10,000 customers)** independently.
+- If the logic allows, you may process and upload results immediately because the dataset is self-contained.
+
+However, upload behavior must balance two constraints:
+
+- Avoid uploading too frequently (too many small uploads).
+- Avoid generating very large files that exceed safe batch size.
+
+### Recommended Strategy
+
+Example approach:
+
+- If each page represents ~10,000 customers:
+  - You may accumulate multiple pages locally until reaching the configured `batch_size`.
+  - Then upload results in a single batch to reduce upload overhead.
+
+Conversely:
+
+- If processing local files, sqlite, or duckdb produces a very large result in one pass:
+  - You MUST split output according to `batch_size`.
+  - Do NOT upload extremely large files in one call.
+
+### Key Principle
+
+Paging size (≈10,000 customers) and upload batch size (`batch_size`) are **independent controls**:
+
+- Paging controls **input grouping**.
+- Batch size controls **output safety**.
+
+Services must design processing flow with both limits in mind.
+
+---
+
+## on_terminate
+
+Before termination, the system calls:
+
+```python
+def on_terminate(self):
+    log("on_terminate")
+```
+
+Use it for cleanup.
+
+---
+
+# 4. Persist State Between Runs (sqlite/duckdb)
+
+Services can be killed and restarted at any time. If you need state across runs, use sqlite or duckdb via the provided API.
+
+Example (sqlite):
+
+```python
+SQLITE_FILE_NAME = "mydata.sqlite"
+
+sqlite_file = self._request_sqlite(file_name=SQLITE_FILE_NAME, is_shared=False)
+if sqlite_file is not None:
+    pass
+
+self._upload_sqlite(
+    file_name=SQLITE_FILE_NAME,
+    is_shared=False,
+    sqlite_file="/local/path/to/upload/sqlite.sqlite",
+)
+```
+
+Notes
+
+- `_request_sqlite` downloads from the server and is automatically scoped to this plugin. It will not mix with other plugins even if the name matches.
+- The call is synchronous and returns the local path.
+- If the return value is `None`, no file exists on the server; you can create a new one.
+- `_upload_sqlite` always overwrites the previous file.
+
+`is_shared` behavior
+
+- `False` (recommended): isolate per service. Multiple instances get their own copies.
+- `True`: shared storage. If another plugin updates the file, this plugin can download the latest version.
+
+Note: `file_name` acts like a server key, while `sqlite_file` is the local file path.
+
+---
+
+# 5. Batch upload policy (DataFrame batching)
+
+## Goal
+
+When uploading large DataFrames, always upload in batches to avoid memory/network spikes.
+Batch size MUST be configurable via `request.runtime_parameters["batch_size"]`.
+
+## Required behavior
+
+- Read batch size from `self.request.runtime_parameters.get("batch_size", "50000")`
+- Convert to int safely:
+  - If parsing fails (TypeError/ValueError), fallback to 50000
+
+- Validate:
+  - If batch_size <= 0, fallback to 50000
+
+- Upload logic:
+  - If total_rows == 0: upload once (empty DF) and return
+  - Otherwise, iterate `start in range(0, total_rows, batch_size)` and upload slices:
+    - `batch_df = result_df.iloc[start : start + batch_size]`
+    - `_upload_result(batch_df)`
+
+## Reference implementation pattern
+
+Use the same pattern as `Rfm_segmentSvc.__get_batch_size()` and `Rfm_segmentSvc.__upload_in_batches()`.
+
+## Notes
+
+- Do NOT hardcode a fixed batch size inside upload loops.
+- Do NOT read environment variables for batch sizing unless explicitly required; use `runtime_parameters` only.
+
+---
+
+# 6. Memory / RAM safety policy (Data Plugin)
+
+## Goal
+
+Avoid high RAM usage and OOM by preventing creation of very large DataFrames.
+Prefer streaming/chunked/incremental processing whenever possible.
+
+## Hard rules
+
+- Do NOT load entire large CSVs into a single DataFrame if it can be avoided.
+- Do NOT build large intermediate DataFrames by wide merges/joins/concat unless necessary.
+- Do NOT create duplicate full-size DataFrames (e.g., repeated `.copy()`, chained transforms that materialize multiple full frames).
+
+## Preferred approaches (in order)
+
+### 1. Chunked ingestion
+
+- Use `pandas.read_csv(..., chunksize=N)` and process each chunk incrementally.
+- Write/insert each chunk into the DB (DuckDB/SQLite) immediately.
+- Aggregate results at the DB layer, not in pandas, when feasible.
+
+### 2. DB-first computation
+
+- Persist raw rows into DuckDB early.
+- Perform joins, filters, grouping, segmentation rules in SQL inside DuckDB.
+- Only pull final result set into pandas at the end (and still batch-upload it).
+
+### 3. Reduce payload early
+
+- Select only needed columns at read time (`usecols=[...]`).
+- Use explicit dtypes to reduce memory (`string`, `Int32`, `Float32`, categories where appropriate).
+- Filter rows as early as possible (refunds, invalid rows, null timestamps).
+
+### 4. Avoid wide merges in pandas
+
+- If a merge can be done in SQL, do it in DuckDB instead of pandas.
+- If merging in pandas is unavoidable, keep only necessary columns from both sides before merge.
+
+## Output upload policy
+
+- Upload results in batches using `runtime_parameters["batch_size"]` (see Batch upload policy).
+- If the final result can be generated in streaming form, avoid materializing a full `result_df`.
+
+## When a full DataFrame is allowed
+
+- Only when the expected row count is small enough, or when chunking/DB-first is impractical.
+- If unsure, default to chunking + DB-first.
+
+## Required implementation mindset
+
+- Treat pandas as a convenience layer, not the primary compute engine for large datasets.
+- Prefer "incremental + durable" steps (chunk -> DB insert -> SQL compute -> batched upload).

{pam_python-0.1.32 → pam_python-0.1.34}/pam_python.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pam-python
-Version: 0.1.32
+Version: 0.1.34
 Summary: Pam Python Library
 Home-page: https://github.com/heart/pam-python
 Author: Narongrit Kanhanoi

{pam_python-0.1.32 → pam_python-0.1.34}/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
 
 setup(
     name="pam-python",
-    version="0.1.32",
+    version="0.1.34",
     author="Narongrit Kanhanoi",
     author_email="narongrit@pams.ai",
     description="Pam Python Library",

pam_python-0.1.32/pam/templates/init/AGENT.md

@@ -1,197 +0,0 @@
-# AGENT
-
-This document explains the data plugin service lifecycle so developers can create new services and understand runtime behavior inside the CDP environment.
-
-## Project Overview
-
-1. This project is a data plugin that will be connected to the main system in production.
-2. The data plugin processes data from CDP and is created as separate services by domain. For example, if we process RFM, we create a new service dedicated to that domain.
-
-## Create a New Service
-
-1. Create a service with:
-   `pam new service [service name]`
-2. The system creates a new folder named `[service name]` with sample code and basic templates.
-3. Example:
-   `pam new service rfm_segment`
-   This generates `rfm_segment/Rfm_segmentSvc.py`, which is the starting point of the service.
-4. `rfm_segment/service.yaml` registers the service automatically.
-   The value `endpoint: /Rfm_segmentSvc` means CDP will call:
-   `http://ip-of-this-k8s-pod/services/Rfm_segmentSvc`
-
-### Code Organization Guidelines
-
-- Separate each lifecycle step into clear, dedicated functions. Add comments at critical points so humans can review and audit later.
-- Avoid nested functions unless absolutely necessary.
-- Put service logic in `[service_name]/functions.py`, then import and use it from the service class to keep the overall code easy to read.
-
-## Service Lifecycle
-
-CDP spawns a service instance when a job starts and terminates it after the job completes. This design supports parallel execution. For example, two runs will spawn two separate instances. The only risk is shared local files (e.g., sqlite) which can cause conflicts.
-
-The lifecycle is wired in `rfm_segment/Rfm_segmentSvc.py` (example service name).
-
-### on_start
-
-`on_start` runs at startup. Use it to read runtime parameters from CDP and initialize work.
-
-Example:
-
-```python
-def on_start(self):
-    log("on_start")
-
-    some_param = self.request.runtime_parameters.get("some_param", "")
-    self._request_data()
-```
-
-`runtime_parameters` are configured by CDP to control service behavior without code changes (e.g., `period_day="7"`).
-
-Notes
-
-- All values are strings. Cast with defaults to avoid errors.
-- `dry_run` is the only built-in parameter. If missing, it runs for real. If `dry_run="true"`, the service runs fully but does not upload results to CDP.
-- You do not need to check `dry_run` manually. Always call `_upload_result`; the system handles it.
-
-**Important:** `on_start` must return quickly because CDP calls via HTTP. If it hangs, the client may time out and the job will be terminated. If work is long-running, move it to a thread and return from `on_start` first. This applies to all lifecycle functions.
-
-### Request Data from CDP
-
-Call `self._request_data()` to ask CDP for input data. CDP returns up to 5 CSV files (each file is a separate event).
-
-- File order is guaranteed per agreement, so you can index by position.
-- Column names and record counts are defined by CDP configuration.
-
-CSV example:
-
-```csv
-_id,id,data1,data2,data3
-xx,yy,zz,nn,mm
-```
-
-Column meaning
-
-- `_id` = event id for dedup/processing checks. Often unused and can be dropped.
-- `id` = customer id used by CDP as a key.
-- Other columns depend on the agreement.
-
-Date format is always:
-`"02/08/2026, 18:23:54"`
-Timezone may vary by customer data (usually Bangkok time).
-You may sometimes be asked for the exact time zone if it is logically important for the program.
-
-### on_data_input
-
-When CDP finishes collecting data, it calls `on_data_input`.
-
-Example:
-
-```python
-def on_data_input(self, req: RequestCommand):
-    log(f"on_data_input req.is_end = {req.is_end}")
-
-    # RUN IN A SEPARATED THREAD TO PREVENT HTTP REQUEST BLOCKING
-    thread = threading.Thread(
-        target=lambda: self.__run_process_data_in_thread(req),
-        daemon=True,
-    )
-    thread.start()
-
-
-def __run_process_data_in_thread(self, req: RequestCommand):
-    dataframe = self.__process_data(req.input_files)
-    self._upload_result(dataframe)
-
-    if not req.is_end:
-        self._request_data(req.next)
-    else:
-        self._exit()
-```
-
-Why a thread: to end the HTTP request quickly and avoid timeouts.
-
-`req.input_files` is the ordered list of CSV paths.
-
-### Upload Results to CDP
-
-After processing, call `_upload_result` with a dataframe. Example output:
-
-```csv
-id,data_x,rfm
-xxx,20,hero
-```
-
-- `id` is required for CDP matching.
-- Other columns are defined by agreement.
-
-Sometimes you may include an `event` column to create per-user events:
-
-```csv
-id,data_x,event
-xxx,20,purchase
-yyy,50,refund
-```
-
-If `event` is missing, CDP uses the default event configured in the system.
-
-### Pagination
-
-In the example:
-
-```python
-if not req.is_end:
-    self._request_data(req.next)
-else:
-    self._exit()
-```
-
-- `req.is_end` indicates whether this is the last page.
-- If not end, call `self._request_data(req.next)` for the next page.
-- If end, call `self._exit()` to terminate the instance.
-
-CDP will not send the next page until you call `_request_data`, so the flow is serialized and no two threads will handle the same data unless you explicitly design it that way.
-
-### on_terminate
-
-Before termination, the system calls:
-
-```python
-def on_terminate(self):
-    log("on_terminate")
-```
-
-Use it for cleanup.
-
-## Persist State Between Runs (sqlite/duckdb)
-
-Services can be killed and restarted at any time. If you need state across runs, use sqlite or duckdb via the provided API.
-
-Example (sqlite):
-
-```python
-SQLITE_FILE_NAME = "mydata.sqlite"
-
-sqlite_file = self._request_sqlite(file_name=SQLITE_FILE_NAME, is_shared=False)
-if sqlite_file is not None:
-    pass
-
-self._upload_sqlite(
-    file_name=SQLITE_FILE_NAME,
-    is_shared=False,
-    sqlite_file="/local/path/to/upload/sqlite.sqlite",
-)
-```
-
-Notes
-
-- `_request_sqlite` downloads from the server and is automatically scoped to this plugin. It will not mix with other plugins even if the name matches.
-- The call is synchronous and returns the local path.
-- If the return value is `None`, no file exists on the server; you can create a new one.
-- `_upload_sqlite` always overwrites the previous file.
-
-`is_shared` behavior
-
-- `False` (recommended): isolate per service. Multiple instances get their own copies.
-- `True`: shared storage. If another plugin updates the file, this plugin can download the latest version.
-
-Note: `file_name` acts like a server key, while `sqlite_file` is the local file path.