pam-python 0.1.34__tar.gz → 0.1.36__tar.gz

pam_python-0.1.36/PKG-INFO

@@ -0,0 +1,257 @@
+Metadata-Version: 2.4
+Name: pam-python
+Version: 0.1.36
+Summary: Pam Python Library
+Home-page: https://github.com/heart/pam-python
+Author: Narongrit Kanhanoi
+Author-email: narongrit@pams.ai
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: License :: Other/Proprietary License
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Customer Service
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: setuptools>=70.0.0
+Requires-Dist: Flask>=3.0.2
+Requires-Dist: aiohttp>=3.11.11
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: Faker>=33.1.0
+Requires-Dist: dask>=2024.12.1
+Requires-Dist: dask-expr>=1.1.21
+Requires-Dist: requests>=2.32.3
+Requires-Dist: gunicorn>=23.0.0
+Requires-Dist: pyarrow>=19.0.1
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# pam-python-data-plugin-framework
+
+This repository provides the `pam` CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.
+
+This README is a practical, step-by-step guide you can follow to create and run a real service.
+
+**What you get**
+
+- CLI to initialize a project and scaffold services
+- Service lifecycle contract (start, data input, upload, exit)
+- Temp file and SQLite helpers
+- A monitoring loop for service timeouts and periodic cleanup
+
+---
+
+**Table of Contents**
+
+1. Prerequisites
+2. Install
+3. Initialize a Project
+4. Create a Service
+5. Understand the Lifecycle
+6. Using Temp Files Correctly
+7. Running the Server
+8. Testing a Service
+9. Configuration
+10. Project Structure
+11. Troubleshooting
+
+---
+
+**Prerequisites**
+
+- Python 3.8+ recommended
+- `pip` and a working virtual environment
+
+---
+
+**Install**
+Create a project folder and a virtual environment.
+
+```bash
+mkdir my_data_plugin
+cd my_data_plugin
+python3 -m venv venv
+source venv/bin/activate
+```
+
+Install the framework:
+
+```bash
+pip install pam-python
+```
+
+---
+
+**Initialize a Project**
+This creates a runnable project with templates (including `AGENT.md`).
+
+```bash
+pam init
+```
+
+When `requirements.txt` already exists, you will be prompted to choose how to proceed:
+
+- overwrite
+- keep
+- merge
+
+---
+
+**Create a Service**
+Generate a service scaffold. Do not hand-create service templates.
+
+```bash
+pam new service rfm_segment
+```
+
+This creates a new folder (e.g. `rfm_segment/`) with:
+
+- a service class (`RfmSegmentSvc.py`)
+- `functions.py` for your logic
+- `service.yaml` for registration
+- a test file
+
+---
+
+**Understand the Lifecycle**
+The runtime calls your service in two main phases.
+
+1. `on_start`
+
+- Called once at the beginning
+- Read parameters from `self.request.runtime_parameters`
+- Should return quickly (start a thread for long work)
+
+2. `on_data_input`
+
+- Called when CDP sends input files
+- `req.input_files` contains ordered CSV files
+- Should also return quickly (use a thread if needed)
+
+When your service is done:
+
+- Call `self._upload_result(...)` or `self._upload_report(...)`
+- Call `self._exit()` to signal completion
+
+---
+
+**Using Temp Files Correctly**
+Temp storage is managed by the framework. Do not delete temp files manually.
+
+Standard helpers:
+
+- `TempfileUtils.get_temp_path_for_service(self, self.service_name)`
+- `TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)`
+
+Notes:
+
+- `get_temp_path_for_service(...)` returns a directory path without a trailing slash.
+- The temp path includes date/service/token in this structure:
+  `TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>`
+
+---
+
+**Uploading Results in Batches**
+If your service produces too many rows (or too few per event), use the batch uploader to handle chunking and flushing automatically.
+
+Recommended usage:
+
+```python
+from pam.result_batch_uploader import ResultBatchUploader
+
+batch_uploader = ResultBatchUploader(self, batch_size=50000)
+batch_uploader.upload(df, "data-name")
+batch_uploader.flush()
+status = batch_uploader.get_status()
+```
+
+Notes:
+
+- `name` separates different result streams (A/B) to avoid schema conflicts.
+- `flush()` uploads any remaining rows that are below the batch size.
+
+---
+
+**Running the Server**
+The generated `main.py` runs the Flask server.
+
+```bash
+python main.py
+```
+
+By default it binds to `0.0.0.0:8000`. You can override with:
+
+```bash
+export SERVER_HOST=0.0.0.0
+export SERVER_PORT=8000
+```
+
+---
+
+**Testing a Service**
+Run unit tests for a service:
+
+```bash
+pam test rfm_segment
+```
+
+If you write custom tests, place them in the service folder and name them `test_<service>.py`.
+
+---
+
+**Configuration**
+Environment variables you can set:
+
+- `SERVER_HOST`
+- `SERVER_PORT`
+- `TEMP_BASE_PATH` (default `/app/data`)
+- `TEMP_DATASOURCE_PATH` (default `/app/data/data_sources`)
+- `TEMP_CLEAN_DAYS` (default `10`)
+- `TEMP_CLEAN_INTERVAL_HOURS` (default `6`, set empty to disable periodic cleanup)
+
+---
+
+**Project Structure**
+After `pam init` and one service:
+
+```
+.
+├── main.py
+├── AGENT.md
+├── Dockerfile
+├── rfm_segment/
+│   ├── RfmSegmentSvc.py
+│   ├── functions.py
+│   ├── service.yaml
+│   └── test_rfm_segment.py
+├── requirements.txt
+└── run_unit_test.sh
+```
+
+---
+
+**Troubleshooting**
+
+- If `pam` command is missing, ensure your virtualenv is activated.
+- If `pam new service` fails, confirm the service name is provided.
+- If temp cleanup is too frequent or too slow, adjust `TEMP_CLEAN_INTERVAL_HOURS` and `TEMP_CLEAN_DAYS`.
+
+---
+
+**Next Steps**
+
+- Implement your logic in `functions.py`.
+- Wire it into `on_start` and `on_data_input` in your service class.
+- Use the temp utilities to write intermediate files.
+- Use `_upload_result` to return output to CDP.

pam_python-0.1.36/README.md

@@ -0,0 +1,218 @@
+# pam-python-data-plugin-framework
+
+This repository provides the `pam` CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.
+
+This README is a practical, step-by-step guide you can follow to create and run a real service.
+
+**What you get**
+
+- CLI to initialize a project and scaffold services
+- Service lifecycle contract (start, data input, upload, exit)
+- Temp file and SQLite helpers
+- A monitoring loop for service timeouts and periodic cleanup
+
+---
+
+**Table of Contents**
+
+1. Prerequisites
+2. Install
+3. Initialize a Project
+4. Create a Service
+5. Understand the Lifecycle
+6. Using Temp Files Correctly
+7. Running the Server
+8. Testing a Service
+9. Configuration
+10. Project Structure
+11. Troubleshooting
+
+---
+
+**Prerequisites**
+
+- Python 3.8+ recommended
+- `pip` and a working virtual environment
+
+---
+
+**Install**
+Create a project folder and a virtual environment.
+
+```bash
+mkdir my_data_plugin
+cd my_data_plugin
+python3 -m venv venv
+source venv/bin/activate
+```
+
+Install the framework:
+
+```bash
+pip install pam-python
+```
+
+---
+
+**Initialize a Project**
+This creates a runnable project with templates (including `AGENT.md`).
+
+```bash
+pam init
+```
+
+When `requirements.txt` already exists, you will be prompted to choose how to proceed:
+
+- overwrite
+- keep
+- merge
+
+---
+
+**Create a Service**
+Generate a service scaffold. Do not hand-create service templates.
+
+```bash
+pam new service rfm_segment
+```
+
+This creates a new folder (e.g. `rfm_segment/`) with:
+
+- a service class (`RfmSegmentSvc.py`)
+- `functions.py` for your logic
+- `service.yaml` for registration
+- a test file
+
+---
+
+**Understand the Lifecycle**
+The runtime calls your service in two main phases.
+
+1. `on_start`
+
+- Called once at the beginning
+- Read parameters from `self.request.runtime_parameters`
+- Should return quickly (start a thread for long work)
+
+2. `on_data_input`
+
+- Called when CDP sends input files
+- `req.input_files` contains ordered CSV files
+- Should also return quickly (use a thread if needed)
+
+When your service is done:
+
+- Call `self._upload_result(...)` or `self._upload_report(...)`
+- Call `self._exit()` to signal completion
+
+---
+
+**Using Temp Files Correctly**
+Temp storage is managed by the framework. Do not delete temp files manually.
+
+Standard helpers:
+
+- `TempfileUtils.get_temp_path_for_service(self, self.service_name)`
+- `TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)`
+
+Notes:
+
+- `get_temp_path_for_service(...)` returns a directory path without a trailing slash.
+- The temp path includes date/service/token in this structure:
+  `TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>`
+
+---
+
+**Uploading Results in Batches**
+If your service produces too many rows (or too few per event), use the batch uploader to handle chunking and flushing automatically.
+
+Recommended usage:
+
+```python
+from pam.result_batch_uploader import ResultBatchUploader
+
+batch_uploader = ResultBatchUploader(self, batch_size=50000)
+batch_uploader.upload(df, "data-name")
+batch_uploader.flush()
+status = batch_uploader.get_status()
+```
+
+Notes:
+
+- `name` separates different result streams (A/B) to avoid schema conflicts.
+- `flush()` uploads any remaining rows that are below the batch size.
+
+---
+
+**Running the Server**
+The generated `main.py` runs the Flask server.
+
+```bash
+python main.py
+```
+
+By default it binds to `0.0.0.0:8000`. You can override with:
+
+```bash
+export SERVER_HOST=0.0.0.0
+export SERVER_PORT=8000
+```
+
+---
+
+**Testing a Service**
+Run unit tests for a service:
+
+```bash
+pam test rfm_segment
+```
+
+If you write custom tests, place them in the service folder and name them `test_<service>.py`.
+
+---
+
+**Configuration**
+Environment variables you can set:
+
+- `SERVER_HOST`
+- `SERVER_PORT`
+- `TEMP_BASE_PATH` (default `/app/data`)
+- `TEMP_DATASOURCE_PATH` (default `/app/data/data_sources`)
+- `TEMP_CLEAN_DAYS` (default `10`)
+- `TEMP_CLEAN_INTERVAL_HOURS` (default `6`, set empty to disable periodic cleanup)
+
+---
+
+**Project Structure**
+After `pam init` and one service:
+
+```
+.
+├── main.py
+├── AGENT.md
+├── Dockerfile
+├── rfm_segment/
+│   ├── RfmSegmentSvc.py
+│   ├── functions.py
+│   ├── service.yaml
+│   └── test_rfm_segment.py
+├── requirements.txt
+└── run_unit_test.sh
+```
+
+---
+
+**Troubleshooting**
+
+- If `pam` command is missing, ensure your virtualenv is activated.
+- If `pam new service` fails, confirm the service name is provided.
+- If temp cleanup is too frequent or too slow, adjust `TEMP_CLEAN_INTERVAL_HOURS` and `TEMP_CLEAN_DAYS`.
+
+---
+
+**Next Steps**
+
+- Implement your logic in `functions.py`.
+- Wire it into `on_start` and `on_data_input` in your service class.
+- Use the temp utilities to write intermediate files.
+- Use `_upload_result` to return output to CDP.

{pam_python-0.1.34 → pam_python-0.1.36}/pam/api.py

@@ -66,12 +66,15 @@ class API:
         """
         headers = {'Content-Type': 'application/json'}
         try:
-            response = self.session.post(url, json=data, timeout=60, headers=headers)
+            response = self.session.post(
+                url, json=data, timeout=60, headers=headers, stream=True
+            )
             response.raise_for_status()
 
-            # Save response content to file
             with open(output_path, "wb") as f:
-                f.write(response.content)
+                for chunk in response.iter_content(chunk_size=1024 * 1024):
+                    if chunk:
+                        f.write(chunk)
 
             log(f"SQLite file saved to {output_path}")
             return True

{pam_python-0.1.34 → pam_python-0.1.36}/pam/cli.py

@@ -16,12 +16,20 @@ def main():
     if cmd == "init":
         init_project()
     elif cmd == "new":
+        if len(args) < 2:
+            print("Usage: pam new service <service_name>")
+            return
         create_type = args[1]
         if create_type == "service":
+            if len(args) < 3:
+                print("Usage: pam new service <service_name>")
+                return
             name = args[2]
             create_service(name)
+        else:
+            print(f"Unknown new target: {create_type}")
     else:
-        print(f"Unknown command: {args.command}")
+        print(f"Unknown command: {cmd}")
 
 
 def to_pascal_case(input_string: str) -> str:
@@ -99,14 +107,75 @@ def init_project():
     cpy("init/run_unit_test.ps1", "run_unit_test.ps1")
     cpy("init/AGENT.md", "AGENT.md")
 
-    if not os.path.exists("requirements.txt"):
-        open("requirements.txt", 'a', encoding='utf-8').close()
-
-    with open("requirements.txt", "w", encoding='utf-8') as f:
-        subprocess.run(["pip", "freeze"], stdout=f, check=True)
+    requirements_path = "requirements.txt"
+    has_requirements = os.path.exists(requirements_path)
+
+    mode = "overwrite"
+    if has_requirements:
+        print("requirements.txt already exists. Choose how to proceed:")
+        print("1) overwrite")
+        print("2) keep")
+        print("3) merge")
+        selection = input("Select [1-3] (default: 1): ").strip()
+        if selection == "2":
+            mode = "keep"
+        elif selection == "3":
+            mode = "merge"
+
+    if mode == "keep":
+        print("Keeping existing requirements.txt")
+    else:
+        freeze = subprocess.run(
+            ["pip", "freeze"],
+            check=True,
+            text=True,
+            capture_output=True
+        ).stdout.splitlines()
+
+        if mode == "overwrite" or not has_requirements:
+            with open(requirements_path, "w", encoding="utf-8") as f:
+                f.write("\n".join(freeze) + ("\n" if freeze else ""))
+        elif mode == "merge":
+            with open(requirements_path, "r", encoding="utf-8") as f:
+                existing_lines = f.read().splitlines()
+            merged_lines = _merge_requirements(existing_lines, freeze)
+            with open(requirements_path, "w", encoding="utf-8") as f:
+                f.write("\n".join(merged_lines) + ("\n" if merged_lines else ""))
 
     if not os.path.exists("__init__.py"):
         open("__init__.py", 'a', encoding='utf-8').close()
 
     print("--- Welcome to PAM ---\n")
     print("To create a new servive run\n`pam new service <service_name>`\n\n")
+
+
+def _parse_requirement_name(line: str) -> str:
+    line = line.strip()
+    if not line or line.startswith("#"):
+        return ""
+    if line.startswith("-"):
+        return line
+    # Normalize common requirement formats like pkg==1.2, pkg>=1.2, pkg[extra]==1.2
+    splitters = ["==", ">=", "<=", "~=", ">", "<"]
+    for splitter in splitters:
+        if splitter in line:
+            return line.split(splitter, 1)[0].strip()
+    return line
+
+
+def _merge_requirements(existing_lines, frozen_lines):
+    """
+    Merge frozen requirements into existing lines while keeping existing entries.
+    """
+    existing_keys = {_parse_requirement_name(line) for line in existing_lines}
+    existing_keys.discard("")
+
+    merged = list(existing_lines)
+    for line in frozen_lines:
+        key = _parse_requirement_name(line)
+        if not key:
+            continue
+        if key not in existing_keys:
+            merged.append(line)
+            existing_keys.add(key)
+    return merged

{pam_python-0.1.34 → pam_python-0.1.36}/pam/interface_task_manager.py

@@ -22,11 +22,11 @@ class ITaskManager(ABC):
         pass
 
     @abstractmethod
-    def service_request_sqlite(self, service: Service, file_name: str = "", is_shared: bool = False) -> str:
+    def service_request_sqlite(self, service: Service, file_name: str = "", is_shared: bool = False):
         pass
 
     @abstractmethod
-    def service_upload_sqlite(self, service: Service, file_name: str = "", is_shared: bool = False, sqlite_file: str = "") -> str:
+    def service_upload_sqlite(self, service: Service, file_name: str = "", is_shared: bool = False, sqlite_file: str = ""):
         pass
 
     @abstractmethod

{pam_python-0.1.34 → pam_python-0.1.36}/pam/models/request_command.py

@@ -19,6 +19,7 @@ Methods:
 
 """
 import os
+import shutil
 from pathlib import Path
 from dataclasses import dataclass
 import zipfile
@@ -69,6 +70,8 @@ class RequestCommand:
         """
         Extract data from a Flask request object based on its content type.
         """
+        if request.content_type is None:
+            return {}
         if request.content_type == 'application/json':
             return request.get_json() or {}
         elif request.content_type.startswith('multipart/form-data'):
@@ -93,12 +96,11 @@ class RequestCommand:
             log("Invalid file type uploaded.")
             return [], "Invalid file type"
 
-        temp_dir = TempfileUtils.get_temp_file_name(service_name, token, "dataset_")
+        temp_dir = TempfileUtils.get_temp_subdir(service_name, token, "dataset_")
         zip_file_path = f"{temp_dir}.zip"
         zip_file.save(zip_file_path)
 
         extract_dir = Path(temp_dir)
-        extract_dir.mkdir(parents=True, exist_ok=True)
 
         try:
             with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
@@ -109,7 +111,7 @@ class RequestCommand:
                 unwanted_path = extract_dir / unwanted
                 if unwanted_path.exists():
                     if unwanted_path.is_dir():
-                        os.rmdir(unwanted_path)
+                        shutil.rmtree(unwanted_path, ignore_errors=True)
                     else:
                         unwanted_path.unlink()
 

{pam_python-0.1.34 → pam_python-0.1.36}/pam/server.py

@@ -12,7 +12,6 @@ from pam.models.request_command import RequestCommand
 from pam.task_manager import TaskManager
 from pam.temp_file_utils import TempfileUtils
 from threading import Lock
-from pam.utils import log
 
 class Server:
     """
@@ -23,9 +22,10 @@ class Server:
     servicePool = {}
     _lock = Lock()
 
-    def __init__(self, app: Flask):
+    def __init__(self, app: Flask, temp_clean_days: int = None):
         self.task_manager = TaskManager(self)
         self.app = app
+        self.temp_clean_days = self._resolve_temp_clean_days(temp_clean_days)
         self.register_service()
         self.task_manager.start_service_monitoring_schedul()
 
@@ -36,7 +36,7 @@ class Server:
         @app.route('/service/<service_name>', methods=['POST'])
         def run(service_name):
             log(f"[run] service = {service_name}")
-            TempfileUtils.clean_temp()
+            TempfileUtils.clean_temp(self.temp_clean_days)
             log(f"[run] service = {service_name} after clean_temp")
             req, error_request = RequestCommand.parse(request, service_name)
             if error_request:
@@ -119,6 +119,15 @@ class Server:
         data = jsonify(json)
         return make_response(data, code)
 
+    @staticmethod
+    def _resolve_temp_clean_days(temp_clean_days: int):
+        if temp_clean_days is not None:
+            return temp_clean_days
+        try:
+            return int(os.getenv("TEMP_CLEAN_DAYS", "10"))
+        except ValueError:
+            return 10
+
     def response_ok(self, json: dict, code: int = 200):
         """
         Create a response object with 200 OK.