lam-cli 0.1.7__tar.gz → 1.0.0__tar.gz

lam_cli-1.0.0/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## [0.2.0] - 2024-01-20
+
+### Added
+- Support for handling multiple JSON objects from JQ output
+- Comprehensive logging throughout JQ execution
+- New test suite for JQ engine
+- Better error handling and reporting
+
+### Changed
+- JQ output processing now properly handles arrays of objects
+- Improved logging verbosity and clarity
+- Updated documentation structure
+
+### Fixed
+- Issue with JQ output not being properly parsed when multiple objects are returned 

lam_cli-1.0.0/DOCS.md

@@ -0,0 +1,174 @@
+# LAM
+LAM is a data transformation tool designed for Laminar's API integration platform.
+
+## Overview
+LAM enables you to write efficient transformations for your API data using either JavaScript (Bun) or Python. It's designed to be secure, fast, and easy to integrate into your Laminar workflows.
+
+## Features
+- **Dual Engine Support**: Choose between JavaScript (Bun runtime) for fast execution or Python for complex data processing
+- **Built-in Libraries**: Access lodash and date-fns in JavaScript, comprehensive Python standard library modules
+- **Security**: Runs in sandboxed environments with strict resource limits and security restrictions
+- **Performance**: Uses Bun runtime for JavaScript and sandboxed Python interpreter
+- **Monitoring**: Built-in execution statistics and error tracking
+
+## Execution Environments
+
+### Bun JavaScript Runtime (js)
+**Configuration**:
+- **Engine**: Bun
+- **Timeout**: 5 seconds
+- **Execution**: Isolated with `--no-fetch --smol --silent` flags
+- **Storage**: No localStorage/sessionStorage support
+- **Modules**: Shared node_modules directory
+
+**Available Libraries**:
+- **lodash** (^4.17.21): Utility library for array/object manipulation, data transformations (Global: `_`)
+- **date-fns** (^2.30.0): Modern date utility library with `format`, `parseISO` functions
+
+**Transform Function Signature**:
+```js
+(input) => { /* transform logic */ return result; }
+```
+
+### Python Interpreter with Sandboxing (py)
+**Configuration**:
+- **Engine**: Python interpreter
+- **Timeout**: 5 seconds
+- **Memory Limit**: 100MB
+- **CPU Limit**: 5 seconds (RLIMIT_CPU)
+- **Virtual Memory**: 100MB (RLIMIT_AS)
+- **Execution**: Isolated with `-I` flag (ignores environment/site packages)
+
+**Security Restrictions**:
+- **Blocked Modules**: subprocess, sys, os, shutil, pathlib, importlib, builtins, _thread, ctypes, socket, pickle, multiprocessing
+- **Blocked Functions**: __import__, eval, exec, globals, locals, getattr, setattr, delattr, compile, open
+- **Blocked Patterns**: __subclasses__, dunder attributes access
+
+**Available Standard Library Modules**:
+- **json**: JSON encoder and decoder
+- **datetime**: Date and time handling
+- **time**: Time-related functions
+- **math**: Mathematical functions and constants
+- **statistics**: Statistical functions (mean, median, mode, standard deviation)
+- **collections**: Counter, defaultdict, OrderedDict, deque
+- **itertools**: Efficient looping, combinations, permutations
+- **functools**: reduce, partial, lru_cache
+- **re**: Regular expression operations
+- **copy**: Shallow and deep copy operations
+- **decimal**: Precise decimal calculations
+- **csv**: CSV file reading and writing
+- **io**: StringIO, BytesIO for in-memory files
+- **dataclasses**: Data classes for storing data
+- **typing**: Type hints support
+- **enum**: Support for enumerations
+- **random**: Random number generation
+- **uuid**: UUID generation
+- **hashlib**: Secure hash and message digest algorithms
+- **base64**: Base64 encoding and decoding
+- **urllib**: URL handling modules
+- **urllib.parse**: URL parsing utilities
+- **html**: HTML processing utilities
+- **xml**: XML processing
+- **xml.etree**: XML ElementTree API
+- **xml.etree.ElementTree**: XML parsing and creation
+- **string**: String constants and classes
+- **textwrap**: Text wrapping and filling
+- **operator**: Standard operators as functions
+- **bisect**: Array bisection algorithm
+- **heapq**: Heap queue algorithm
+- **array**: Efficient arrays of numeric values
+- **unicodedata**: Unicode character database
+- **locale**: Internationalization services
+- **calendar**: Calendar-related functions
+- **zoneinfo**: Time zone support (Python 3.9+)
+- **struct**: Pack and unpack binary data
+- **binascii**: Binary/ASCII conversions
+- **codecs**: Encode and decode data
+- **difflib**: Sequence comparison utilities
+- **pprint**: Pretty-printer for data structures
+- **reprlib**: Alternate repr() implementation
+- **abc**: Abstract base classes
+- **contextlib**: Context management utilities
+- **secrets**: Cryptographically secure random numbers
+- **fractions**: Rational numbers
+- **numbers**: Numeric abstract base classes
+
+**Safe Built-in Functions**:
+`abs`, `all`, `any`, `bool`, `chr`, `dict`, `divmod`, `enumerate`, `filter`, `float`, `format`, `frozenset`, `hash`, `hex`, `int`, `isinstance`, `issubclass`, `iter`, `len`, `list`, `map`, `max`, `min`, `next`, `oct`, `ord`, `pow`, `print`, `range`, `repr`, `reversed`, `round`, `set`, `slice`, `sorted`, `str`, `sum`, `tuple`, `type`, `zip`
+
+**Transform Function Signature**:
+```py
+def transform(input_data):
+    # transform logic
+    return result
+```
+
+## Examples
+
+### JavaScript (Bun) Transformations
+Perfect for fast data manipulation with familiar syntax:
+
+```javascript
+(input) => {
+    // Use lodash for data manipulation
+    const processed = _.map(input.data, item => ({
+        id: item.id,
+        formattedDate: format(parseISO(item.date), 'MMM dd, yyyy'),
+        value: item.value * 2
+    }));
+
+    return {
+        processed,
+        summary: {
+            total: _.sumBy(processed, 'value'),
+            count: processed.length
+        }
+    };
+}
+```
+
+### Python Transformations
+Ideal for complex data processing and statistical analysis:
+
+```python
+def transform(input_data):
+    import statistics
+    from collections import Counter
+    
+    # Process numerical data
+    values = [item["value"] for item in input_data["data"] if "value" in item]
+    
+    return {
+        "statistics": {
+            "mean": statistics.mean(values) if values else 0,
+            "median": statistics.median(values) if values else 0,
+            "count": len(values)
+        },
+        "frequency": dict(Counter(item["category"] for item in input_data["data"])),
+        "processedAt": datetime.now().isoformat()
+    }
+```
+
+## Integration with Laminar
+LAM is designed to work seamlessly with Laminar's integration platform:
+
+1. **Flows**: Add data transformations to your API flows
+2. **Automation**: Schedule and automate data processing
+3. **Monitoring**: Track execution statistics and errors
+
+## Getting Started
+
+### Using LAM in Laminar
+1. Create a new flow in [Laminar](https://app.laminar.run)
+2. Add a transformation step
+3. Choose your engine (JavaScript or Python)
+4. Write your transformation function
+5. Deploy and monitor
+
+## Resources
+- [Laminar Documentation](https://docs.laminar.run)
+- [Sign up for Laminar](https://app.laminar.run)
+
+## Support
+Get help with LAM:
+- [Contact Support](mailto:connect@laminar.run)

lam_cli-1.0.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include DOCS.md
+include README.md
+include LICENSE
+include CHANGELOG.md
+include requirements.txt 

lam_cli-1.0.0/PKG-INFO

@@ -0,0 +1,229 @@
+Metadata-Version: 2.4
+Name: lam-cli
+Version: 1.0.0
+Summary: Secure data transformation tool supporting JQ and JavaScript (Bun)
+Home-page: https://github.com/laminar-run/lam
+Author: Laminar Run, Inc.
+Author-email: connect@laminar.run
+License: GPLv3
+Project-URL: Documentation, https://docs.laminar.run
+Project-URL: Source, https://github.com/laminar-run/lam
+Project-URL: Issue Tracker, https://github.com/laminar-run/lam/issues
+Keywords: laminar,api,integration,transformation,json,jq,javascript,bun
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
+Classifier: Topic :: Software Development :: Build Tools
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: backoff>=2.2.1
+Requires-Dist: certifi>=2024.12.14
+Requires-Dist: charset-normalizer>=3.3.2
+Requires-Dist: click>=8.1.7
+Requires-Dist: idna>=3.7
+Requires-Dist: logtail-python>=0.2.2
+Requires-Dist: monotonic>=1.6
+Requires-Dist: msgpack>=1.0.8
+Requires-Dist: posthog>=3.4.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: python-dateutil>=2.8.2
+Requires-Dist: requests>=2.32.3
+Requires-Dist: six>=1.16.0
+Requires-Dist: urllib3>=2.2.2
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# LAM
+LAM is a data transformation tool designed for Laminar's API integration platform.
+
+## Overview
+LAM enables you to write efficient transformations for your API data using either JavaScript (Bun) or Python. It's designed to be secure, fast, and easy to integrate into your Laminar workflows.
+
+## Features
+- **Dual Engine Support**: Choose between JavaScript (Bun runtime) for fast execution or Python for complex data processing
+- **Built-in Libraries**: Access lodash and date-fns in JavaScript, comprehensive Python standard library modules
+- **Security**: Runs in sandboxed environments with strict resource limits and security restrictions
+- **Performance**: Uses Bun runtime for JavaScript and sandboxed Python interpreter
+- **Monitoring**: Built-in execution statistics and error tracking
+
+## Execution Environments
+
+### Bun JavaScript Runtime (js)
+**Configuration**:
+- **Engine**: Bun
+- **Timeout**: 5 seconds
+- **Execution**: Isolated with `--no-fetch --smol --silent` flags
+- **Storage**: No localStorage/sessionStorage support
+- **Modules**: Shared node_modules directory
+
+**Available Libraries**:
+- **lodash** (^4.17.21): Utility library for array/object manipulation, data transformations (Global: `_`)
+- **date-fns** (^2.30.0): Modern date utility library with `format`, `parseISO` functions
+
+**Transform Function Signature**:
+```js
+(input) => { /* transform logic */ return result; }
+```
+
+### Python Interpreter with Sandboxing (py)
+**Configuration**:
+- **Engine**: Python interpreter
+- **Timeout**: 5 seconds
+- **Memory Limit**: 100MB
+- **CPU Limit**: 5 seconds (RLIMIT_CPU)
+- **Virtual Memory**: 100MB (RLIMIT_AS)
+- **Execution**: Isolated with `-I` flag (ignores environment/site packages)
+
+**Security Restrictions**:
+- **Blocked Modules**: subprocess, sys, os, shutil, pathlib, importlib, builtins, _thread, ctypes, socket, pickle, multiprocessing
+- **Blocked Functions**: __import__, eval, exec, globals, locals, getattr, setattr, delattr, compile, open
+- **Blocked Patterns**: __subclasses__, dunder attributes access
+
+**Available Standard Library Modules**:
+- **json**: JSON encoder and decoder
+- **datetime**: Date and time handling
+- **time**: Time-related functions
+- **math**: Mathematical functions and constants
+- **statistics**: Statistical functions (mean, median, mode, standard deviation)
+- **collections**: Counter, defaultdict, OrderedDict, deque
+- **itertools**: Efficient looping, combinations, permutations
+- **functools**: reduce, partial, lru_cache
+- **re**: Regular expression operations
+- **copy**: Shallow and deep copy operations
+- **decimal**: Precise decimal calculations
+- **csv**: CSV file reading and writing
+- **io**: StringIO, BytesIO for in-memory files
+- **dataclasses**: Data classes for storing data
+- **typing**: Type hints support
+- **enum**: Support for enumerations
+- **random**: Random number generation
+- **uuid**: UUID generation
+- **hashlib**: Secure hash and message digest algorithms
+- **base64**: Base64 encoding and decoding
+- **urllib**: URL handling modules
+- **urllib.parse**: URL parsing utilities
+- **html**: HTML processing utilities
+- **xml**: XML processing
+- **xml.etree**: XML ElementTree API
+- **xml.etree.ElementTree**: XML parsing and creation
+- **string**: String constants and classes
+- **textwrap**: Text wrapping and filling
+- **operator**: Standard operators as functions
+- **bisect**: Array bisection algorithm
+- **heapq**: Heap queue algorithm
+- **array**: Efficient arrays of numeric values
+- **unicodedata**: Unicode character database
+- **locale**: Internationalization services
+- **calendar**: Calendar-related functions
+- **zoneinfo**: Time zone support (Python 3.9+)
+- **struct**: Pack and unpack binary data
+- **binascii**: Binary/ASCII conversions
+- **codecs**: Encode and decode data
+- **difflib**: Sequence comparison utilities
+- **pprint**: Pretty-printer for data structures
+- **reprlib**: Alternate repr() implementation
+- **abc**: Abstract base classes
+- **contextlib**: Context management utilities
+- **secrets**: Cryptographically secure random numbers
+- **fractions**: Rational numbers
+- **numbers**: Numeric abstract base classes
+
+**Safe Built-in Functions**:
+`abs`, `all`, `any`, `bool`, `chr`, `dict`, `divmod`, `enumerate`, `filter`, `float`, `format`, `frozenset`, `hash`, `hex`, `int`, `isinstance`, `issubclass`, `iter`, `len`, `list`, `map`, `max`, `min`, `next`, `oct`, `ord`, `pow`, `print`, `range`, `repr`, `reversed`, `round`, `set`, `slice`, `sorted`, `str`, `sum`, `tuple`, `type`, `zip`
+
+**Transform Function Signature**:
+```py
+def transform(input_data):
+    # transform logic
+    return result
+```
+
+## Examples
+
+### JavaScript (Bun) Transformations
+Perfect for fast data manipulation with familiar syntax:
+
+```javascript
+(input) => {
+    // Use lodash for data manipulation
+    const processed = _.map(input.data, item => ({
+        id: item.id,
+        formattedDate: format(parseISO(item.date), 'MMM dd, yyyy'),
+        value: item.value * 2
+    }));
+
+    return {
+        processed,
+        summary: {
+            total: _.sumBy(processed, 'value'),
+            count: processed.length
+        }
+    };
+}
+```
+
+### Python Transformations
+Ideal for complex data processing and statistical analysis:
+
+```python
+def transform(input_data):
+    import statistics
+    from collections import Counter
+    
+    # Process numerical data
+    values = [item["value"] for item in input_data["data"] if "value" in item]
+    
+    return {
+        "statistics": {
+            "mean": statistics.mean(values) if values else 0,
+            "median": statistics.median(values) if values else 0,
+            "count": len(values)
+        },
+        "frequency": dict(Counter(item["category"] for item in input_data["data"])),
+        "processedAt": datetime.now().isoformat()
+    }
+```
+
+## Integration with Laminar
+LAM is designed to work seamlessly with Laminar's integration platform:
+
+1. **Flows**: Add data transformations to your API flows
+2. **Automation**: Schedule and automate data processing
+3. **Monitoring**: Track execution statistics and errors
+
+## Getting Started
+
+### Using LAM in Laminar
+1. Create a new flow in [Laminar](https://app.laminar.run)
+2. Add a transformation step
+3. Choose your engine (JavaScript or Python)
+4. Write your transformation function
+5. Deploy and monitor
+
+## Resources
+- [Laminar Documentation](https://docs.laminar.run)
+- [Sign up for Laminar](https://app.laminar.run)
+
+## Support
+Get help with LAM:
+- [Contact Support](mailto:connect@laminar.run)

{lam_cli-0.1.7 → lam_cli-1.0.0}/README.md

@@ -116,6 +116,9 @@ make test-js-edge-cases
 
 # Run single test
 make test-single TEST=test/js/example.js DATA=test/data/input.json
+
+# Or
+python -m lam.lam run --language js test/js/simple.js test/data/simple.json
 ```
 
 ## Releases

lam_cli-1.0.0/lam/core.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+
+import logging
+import os
+import shutil
+import tempfile
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+import psutil
+from logtail import LogtailHandler
+
+
+def setup_logging():
+    """Configure logging with UTC timezone and handlers"""
+    # Configure logging with UTC timezone
+    logging.Formatter.converter = lambda *args: datetime.now(timezone.utc).timetuple()
+    
+    # Set up root logger configuration
+    logging.basicConfig(
+        level=logging.DEBUG, 
+        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+        force=True  # Override any existing configuration
+    )
+    
+    # Suppress noisy loggers
+    logging.getLogger('urllib3').setLevel(logging.WARNING)
+    
+    # Add LogTail handler to root logger
+    root_logger = logging.getLogger()
+    handler = LogtailHandler(source_token="TYz3WrrvC8ehYjXdAEGGyiDp")
+    root_logger.addHandler(handler)
+    
+    return logging.getLogger(__name__)
+
+
+class LAMError(Exception):
+    """Base exception for LAM errors"""
+    pass
+
+
+class UserError(LAMError):
+    """Errors caused by user input"""
+    pass
+
+
+class SystemError(LAMError):
+    """Errors caused by system issues"""
+    pass
+
+
+class ResourceLimitError(LAMError):
+    """Errors caused by resource limits"""
+    pass
+
+
+class ProcessingError(Exception):
+    """Custom exception for processing errors"""
+    pass
+
+
+def check_resource_limits(modules_dir: Optional[Path] = None) -> None:
+    """Check system resource availability"""
+    disk = shutil.disk_usage(tempfile.gettempdir())
+    if disk.free < 100 * 1024 * 1024:  # 100MB minimum
+        raise ResourceLimitError("Insufficient disk space")
+    
+    if modules_dir and modules_dir.exists():
+        modules_size = sum(
+            os.path.getsize(os.path.join(dirpath, filename))
+            for dirpath, _, filenames in os.walk(modules_dir)
+            for filename in filenames
+        )
+        if modules_size > 500 * 1024 * 1024:
+            shutil.rmtree(modules_dir)
+            modules_dir.mkdir(exist_ok=True)
+
+
+class Stats:
+    """Track execution statistics"""
+    def __init__(self):
+        self.start_time = datetime.now()
+        self.memory_start = self.get_memory_usage()
+    
+    def get_memory_usage(self):
+        process = psutil.Process()
+        return process.memory_info().rss
+    
+    def finalize(self):
+        return {
+            'duration_ms': (datetime.now() - self.start_time).total_seconds() * 1000,
+            'memory_used_mb': (self.get_memory_usage() - self.memory_start) / (1024 * 1024),
+            'timestamp': datetime.now().isoformat()
+        } 

lam_cli-1.0.0/lam/engines/__init__.py

@@ -0,0 +1,23 @@
+from .base import Engine, EngineType
+from .javascript import BunEngine
+from .jq import JQEngine
+from .python import PythonEngine
+
+
+def get_engine(engine_type: str, workspace_id: str, flow_id: str, execution_id: str) -> Engine:
+    """Factory function to get the appropriate execution engine"""
+    engines = {
+        EngineType.JQ.value: JQEngine,
+        EngineType.JAVASCRIPT.value: BunEngine,
+        EngineType.PYTHON.value: PythonEngine
+    }
+    
+    engine_class = engines.get(engine_type)
+    if not engine_class:
+        raise ValueError(f"Unsupported engine type: {engine_type}")
+    
+    engine = engine_class(workspace_id, flow_id, execution_id)
+    if not engine.validate_environment():
+        raise EnvironmentError(f"Required dependencies not found for {engine_type}")
+    
+    return engine 

lam_cli-1.0.0/lam/engines/base.py

@@ -0,0 +1,56 @@
+import logging
+import os
+import socket
+from datetime import datetime
+from enum import Enum
+from typing import Any, Dict, Optional, Tuple, Union
+
+from posthog import Posthog
+
+# Initialize analytics
+posthog = Posthog(project_api_key='phc_wfeHFG0p5yZIdBpjVYy00o5x1HbEpggdMzIuFYgNPSK', 
+                  host='https://app.posthog.com')
+
+logger = logging.getLogger(__name__)
+
+
+class EngineType(Enum):
+    JQ = "jq"
+    JAVASCRIPT = "js"
+    PYTHON = "py"
+
+
+class Engine:
+    """Base class for execution engines"""
+    def __init__(self, workspace_id: str, flow_id: str, execution_id: str):
+        self.workspace_id = workspace_id
+        self.flow_id = flow_id
+        self.execution_id = execution_id
+        self.timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+
+    def get_log_file(self) -> str:
+        return f"lam_run_{self.workspace_id}_{self.flow_id}_{self.execution_id}_{self.timestamp}.log"
+
+    def get_result_file(self) -> str:
+        return f"lam_result_{self.workspace_id}_{self.flow_id}_{self.execution_id}_{self.timestamp}.json"
+
+    def track_event(self, event_name: str, properties: Dict[str, Any]) -> None:
+        """Track events with PostHog"""
+        try:
+            distinct_id = f"{os.getuid()}_{socket.gethostname()}_{self.workspace_id}_{self.flow_id}"
+            properties |= {
+                'workspace_id': self.workspace_id,
+                'flow_id': self.flow_id,
+                'engine': self.__class__.__name__,
+            }
+            posthog.capture(distinct_id=distinct_id, event=event_name, properties=properties)
+        except Exception as e:
+            logger.error(f"Error tracking event: {e}")
+
+    def validate_environment(self) -> bool:
+        """Validate that the engine can run in this environment"""
+        raise NotImplementedError
+
+    def execute(self, program_file: str, input_data: str) -> Tuple[Union[Dict, str], Optional[str]]:
+        """Execute the program with input data"""
+        raise NotImplementedError