nui-lambda-shared-utils 1.1.1__tar.gz → 1.1.4__tar.gz

{nui_lambda_shared_utils-1.1.1 → nui_lambda_shared_utils-1.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nui-lambda-shared-utils
-Version: 1.1.1
+Version: 1.1.4
 Summary: Enterprise-grade utilities for AWS Lambda functions with Slack, Elasticsearch, and monitoring integrations
 Home-page: https://github.com/nuimarkets/nui-lambda-shared-utils
 Author: NUI Markets

{nui_lambda_shared_utils-1.1.1 → nui_lambda_shared_utils-1.1.4}/docs/README.md

@@ -17,8 +17,9 @@ Welcome to the comprehensive documentation for `nui-lambda-shared-utils`.
 Component-specific guides for major features:
 
 - **[AWS Powertools Integration](guides/powertools-integration.md)** - Standardized logging, metrics, and error handling
+- **[Lambda Context Helpers](guides/lambda-utilities.md)** - Environment info extraction for logging and metrics
 - **[Slack Integration](guides/slack-integration.md)** - Messaging, formatting, and file uploads
-- Elasticsearch Operations (planned)
+- **[Elasticsearch Integration](guides/elasticsearch-integration.md)** - Search, bulk indexing, health checks
 - Database Connections (planned)
 - Error Handling Patterns (planned)
 - CloudWatch Metrics (planned)
@@ -32,8 +33,9 @@ Command-line utilities included with the package:
 
 ### 📋 Reference
 
-API reference and detailed component documentation (planned):
+API reference and detailed component documentation:
 
+- **[Shared Types & Data Structures](guides/shared-types.md)** - Core types, interfaces, and response structures
 - Client APIs (planned)
 - Utility Functions (planned)
 - Configuration Options (planned)
@@ -117,14 +119,17 @@ When contributing to documentation:
 - Main documentation (this README)
 - Getting started guides (installation, configuration, quickstart)
 - AWS Powertools integration guide (guides/powertools-integration.md)
+- Lambda context helpers guide (guides/lambda-utilities.md)
 - Slack integration guide (guides/slack-integration.md)
+- Elasticsearch integration guide (guides/elasticsearch-integration.md)
+- Shared types reference (guides/shared-types.md)
 - CLI tools guide (guides/cli-tools.md)
 - Testing guide (development/testing.md)
 - Configuration templates (Slack account names, channel setup)
 
 ### 🚧 Planned
 
-- Component-specific guides (Elasticsearch, Database, Metrics, Error Handling)
+- Component-specific guides (Database, Metrics, Error Handling)
 - API reference documentation
 - Advanced topics (AWS infrastructure, Lambda integration)
 - Troubleshooting guide

nui_lambda_shared_utils-1.1.4/docs/guides/elasticsearch-integration.md

@@ -0,0 +1,375 @@
+# Elasticsearch Integration Guide
+
+Comprehensive guide for using Elasticsearch utilities in `nui-lambda-shared-utils`.
+
+**Last Updated**: 2026-01-30
+
+## Overview
+
+The package provides Elasticsearch integration through:
+
+- **ElasticsearchClient** - Search, aggregations, bulk indexing, health checks
+- **ElasticsearchQueryBuilder** - Fluent query construction
+
+## Quick Start
+
+```python
+import nui_lambda_shared_utils as nui
+
+# Configure Elasticsearch credentials
+nui.configure(es_credentials_secret="prod/elasticsearch-credentials")
+
+# Create client and search
+es = nui.ElasticsearchClient()
+results = es.search(
+    index="logs-*",
+    body={"query": {"match": {"level": "error"}}}
+)
+```
+
+## Installation
+
+Install with the elasticsearch extra:
+
+```bash
+pip install nui-lambda-shared-utils[elasticsearch]
+```
+
+## Configuration
+
+### AWS Secrets Manager Setup
+
+Create a secret with your Elasticsearch credentials:
+
+```bash
+aws secretsmanager create-secret \
+  --name "elasticsearch-credentials" \
+  --description "Elasticsearch credentials" \
+  --secret-string '{"username":"elastic","password":"YOUR_PASSWORD"}'
+```
+
+**Secret Format:**
+
+```json
+{
+  "username": "elastic",
+  "password": "YOUR_PASSWORD"
+}
+```
+
+### Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `ES_HOST` | Elasticsearch host | `localhost:9200` |
+| `ES_CREDENTIALS_SECRET` | Secret name for credentials | `elasticsearch-credentials` |
+
+### Programmatic Configuration
+
+```python
+import nui_lambda_shared_utils as nui
+
+# Configure at startup
+nui.configure(
+    es_host="elasticsearch.example.com:9200",
+    es_credentials_secret="prod/es-creds"
+)
+
+# Or pass directly to client
+client = nui.ElasticsearchClient(
+    host="elasticsearch.example.com:9200",
+    secret_name="prod/es-creds"
+)
+```
+
+## Basic Operations
+
+### Search
+
+```python
+from nui_lambda_shared_utils import ElasticsearchClient
+
+es = ElasticsearchClient()
+
+# Simple search
+results = es.search(
+    index="logs-*",
+    body={"query": {"match_all": {}}},
+    size=100
+)
+
+for doc in results:
+    print(doc["message"])
+```
+
+### Aggregations
+
+```python
+# Get aggregation results
+aggs = es.aggregate(
+    index="logs-*",
+    body={
+        "query": {"range": {"@timestamp": {"gte": "now-1h"}}},
+        "aggs": {
+            "by_level": {"terms": {"field": "level.keyword"}},
+            "avg_duration": {"avg": {"field": "duration"}}
+        }
+    }
+)
+
+print(aggs["by_level"]["buckets"])
+```
+
+### Count Documents
+
+```python
+# Count all documents
+total = es.count(index="logs-*")
+
+# Count with query
+errors = es.count(
+    index="logs-*",
+    body={"query": {"term": {"level": "error"}}}
+)
+```
+
+## Bulk Indexing
+
+### streaming_bulk Method
+
+For efficient bulk indexing of documents, use the `streaming_bulk` method:
+
+```python
+from nui_lambda_shared_utils import ElasticsearchClient
+
+es = ElasticsearchClient()
+
+def generate_documents():
+    """Yield documents to index."""
+    for item in items:
+        yield {
+            "_index": "my-index",
+            "_source": {
+                "field1": item.field1,
+                "field2": item.field2,
+                "@timestamp": item.timestamp.isoformat()
+            }
+        }
+
+# Index documents with automatic error handling
+success, failed = es.streaming_bulk(
+    actions=generate_documents(),
+    chunk_size=100,
+    max_retries=2
+)
+
+print(f"Indexed {success} documents, {failed} failures")
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `actions` | Iterator[Dict] | required | Iterator yielding action dictionaries |
+| `chunk_size` | int | 100 | Documents per batch |
+| `max_retries` | int | 2 | Retries for failed documents |
+| `raise_on_error` | bool | False | Raise exception on first error |
+
+### Action Dictionary Format
+
+Each yielded action should contain:
+
+```python
+{
+    "_index": "target-index-name",
+    "_source": {
+        # Your document fields
+    },
+    # Optional fields:
+    "_id": "custom-id",  # Auto-generated if not provided
+    "_op_type": "index"  # "index", "create", "update", "delete"
+}
+```
+
+### Error Handling
+
+By default, `streaming_bulk` logs errors but continues processing:
+
+```python
+# Default: log errors, continue processing
+success, failed = es.streaming_bulk(generate_docs())
+if failed > 0:
+    logger.warning(f"{failed} documents failed to index")
+
+# Strict mode: raise on first error
+try:
+    success, failed = es.streaming_bulk(
+        generate_docs(),
+        raise_on_error=True
+    )
+except Exception as e:
+    logger.error(f"Bulk indexing failed: {e}")
+```
+
+### Integration with Log Processors
+
+The `streaming_bulk` method pairs well with log extraction utilities:
+
+```python
+from nui_lambda_shared_utils import ElasticsearchClient
+
+es = ElasticsearchClient()
+
+def extract_logs_from_kinesis(records):
+    """Extract logs from Kinesis records."""
+    for record in records:
+        # Your log extraction logic
+        yield {
+            "_index": f"logs-{service_name}-{date}",
+            "_source": log_data
+        }
+
+def handler(event, context):
+    success, failed = es.streaming_bulk(
+        actions=extract_logs_from_kinesis(event["Records"]),
+        chunk_size=200
+    )
+    return {"indexed": success, "failed": failed}
+```
+
+## Service Statistics
+
+Get comprehensive service statistics:
+
+```python
+stats = es.get_service_stats(
+    service="order",
+    hours=24,
+    index_prefix="logs"
+)
+
+print(f"Total requests: {stats['total_count']}")
+print(f"Error rate: {stats['error_rate']:.2f}%")
+print(f"P95 latency: {stats['p95_response_time']}ms")
+```
+
+## Recent Errors
+
+Retrieve recent error logs:
+
+```python
+errors = es.get_recent_errors(
+    service="auth",
+    hours=1,
+    limit=10
+)
+
+for error in errors:
+    print(f"{error['@timestamp']}: {error['message']}")
+```
+
+## Health Checks
+
+### Cluster Health
+
+```python
+# Get cluster info
+info = es.get_cluster_info()
+print(f"Cluster: {info['cluster_name']}, Status: {info['cluster_status']}")
+
+# Check health (raises on failure)
+health = es.check_health()
+```
+
+### Index Information
+
+```python
+indices = es.get_indices_info(pattern="logs-*")
+for idx in indices:
+    print(f"{idx['index']}: {idx['docs.count']} docs, {idx['store.size']}")
+```
+
+## Query Builder
+
+For complex queries, use the query builder:
+
+```python
+from nui_lambda_shared_utils import ElasticsearchQueryBuilder
+
+query = (
+    ElasticsearchQueryBuilder()
+    .must_match("service", "order")
+    .must_range("@timestamp", gte="now-1h")
+    .filter_term("level", "error")
+    .sort("@timestamp", "desc")
+    .build()
+)
+
+results = es.search(index="logs-*", body=query)
+```
+
+## Lambda Handler Pattern
+
+Recommended pattern for Lambda functions:
+
+```python
+import os
+import nui_lambda_shared_utils as nui
+from nui_lambda_shared_utils import (
+    ElasticsearchClient,
+    get_powertools_logger,
+    powertools_handler
+)
+
+# Configure once at module level
+nui.configure(
+    es_host=os.environ.get("ES_HOST"),
+    es_credentials_secret=os.environ.get("ES_CREDENTIALS_SECRET")
+)
+
+logger = get_powertools_logger("my-service")
+
+
+@powertools_handler(service_name="my-service")
+def handler(event, context):
+    es = ElasticsearchClient()
+
+    # Your indexing logic
+    success, failed = es.streaming_bulk(
+        actions=process_records(event["Records"])
+    )
+
+    logger.info(f"Indexed {success} documents, {failed} failures")
+    return {"statusCode": 200, "body": f"Processed {success} documents"}
+```
+
+## Troubleshooting
+
+### Connection Issues
+
+```python
+# Test connectivity
+try:
+    es = ElasticsearchClient()
+    info = es.get_cluster_info()
+    print(f"Connected to: {info['cluster_name']}")
+except Exception as e:
+    print(f"Connection failed: {e}")
+```
+
+### Common Errors
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `AuthenticationException` | Invalid credentials | Check secret values |
+| `ConnectionError` | Host unreachable | Verify `ES_HOST` and network |
+| `TransportError(403)` | Missing permissions | Check index permissions |
+
+### Debug Logging
+
+Enable debug logging for troubleshooting:
+
+```python
+import logging
+logging.getLogger("elasticsearch").setLevel(logging.DEBUG)
+```

nui_lambda_shared_utils-1.1.4/docs/guides/lambda-utilities.md

@@ -0,0 +1,274 @@
+# Lambda Context Helpers Guide
+
+This guide covers the Lambda context helpers provided by `nui-lambda-shared-utils`. These utilities provide standardized environment info extraction for logging context, metric dimensions, and conditional behavior based on execution environment.
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Function Reference](#function-reference)
+- [Usage Patterns](#usage-patterns)
+- [Environment Detection](#environment-detection)
+- [Related Documentation](#related-documentation)
+
+## Installation
+
+No additional dependencies required. Lambda helpers use only Python standard library:
+
+```bash
+# Base package includes lambda_helpers
+pip install nui-lambda-shared-utils
+```
+
+## Quick Start
+
+```python
+from nui_lambda_shared_utils import get_lambda_environment_info
+
+def handler(event, context):
+    env_info = get_lambda_environment_info()
+
+    # Use for conditional behavior
+    if env_info["is_local"]:
+        print("Running locally")
+    else:
+        print(f"Running in Lambda: {env_info['function_name']}")
+
+    return {"statusCode": 200}
+```
+
+## Function Reference
+
+### `get_lambda_environment_info()`
+
+Extracts standard Lambda environment info from environment variables.
+
+**Signature:**
+
+```python
+def get_lambda_environment_info() -> Dict[str, str | bool]:
+    """
+    Extract standard Lambda environment info.
+
+    Returns:
+        Dict with keys:
+            - environment: "prod" | "dev" | "staging" | "unknown"
+            - aws_region: AWS region (e.g., "ap-southeast-2")
+            - function_name: Lambda function name
+            - function_version: Lambda function version (e.g., "$LATEST")
+            - memory_limit: Memory limit in MB (e.g., "512")
+            - is_local: True if running outside Lambda environment
+    """
+```
+
+**Return Value Example:**
+
+```python
+{
+    "environment": "prod",
+    "aws_region": "ap-southeast-2",
+    "function_name": "order-processor",
+    "function_version": "$LATEST",
+    "memory_limit": "512",
+    "is_local": False
+}
+```
+
+**Environment Variable Sources:**
+
+| Return Key | Primary Env Var | Fallback | Default |
+|------------|-----------------|----------|---------|
+| `environment` | `ENVIRONMENT` | `ENV`, `STAGE` | `"unknown"` |
+| `aws_region` | `AWS_REGION` | `AWS_DEFAULT_REGION` | `""` |
+| `function_name` | `AWS_LAMBDA_FUNCTION_NAME` | - | `""` |
+| `function_version` | `AWS_LAMBDA_FUNCTION_VERSION` | - | `""` |
+| `memory_limit` | `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` | - | `""` |
+| `is_local` | (absence of `AWS_LAMBDA_RUNTIME_API`) | - | `True` |
+
+**Environment Normalization:**
+
+The `environment` value is normalized for consistency:
+
+- `"production"`, `"prd"` → `"prod"`
+- `"development"` → `"dev"`
+- All values are lowercased
+
+## Usage Patterns
+
+### With Powertools Logger
+
+Add environment context to structured logs:
+
+```python
+from nui_lambda_shared_utils import get_powertools_logger, get_lambda_environment_info
+
+logger = get_powertools_logger("my-service")
+
+def handler(event, context):
+    env_info = get_lambda_environment_info()
+
+    logger.info(
+        "Processing event",
+        extra={
+            **env_info,
+            "event_type": event.get("type")
+        }
+    )
+
+    return {"statusCode": 200}
+```
+
+### With CloudWatch Metrics
+
+Add standard dimensions to metrics:
+
+```python
+from nui_lambda_shared_utils import MetricsPublisher, get_lambda_environment_info
+
+def handler(event, context):
+    env_info = get_lambda_environment_info()
+
+    metrics = MetricsPublisher(namespace="MyService")
+    metrics.add_dimension("Environment", env_info["environment"])
+    metrics.add_dimension("FunctionName", env_info["function_name"])
+    metrics.add_dimension("Region", env_info["aws_region"])
+
+    # Publish metrics with consistent dimensions
+    metrics.put_metric("ProcessedEvents", 1, "Count")
+
+    return {"statusCode": 200}
+```
+
+### Conditional Behavior
+
+Execute different code paths based on environment:
+
+```python
+from nui_lambda_shared_utils import get_lambda_environment_info
+
+def handler(event, context):
+    env_info = get_lambda_environment_info()
+
+    # Use different configuration for local vs Lambda
+    if env_info["is_local"]:
+        config = load_local_config()
+    else:
+        config = load_lambda_config()
+
+    # Environment-specific behavior
+    if env_info["environment"] == "prod":
+        send_to_production_queue(event)
+    else:
+        send_to_dev_queue(event)
+
+    return {"statusCode": 200}
+```
+
+### Error Context Enhancement
+
+Include environment info in error reports:
+
+```python
+from nui_lambda_shared_utils import (
+    get_powertools_logger,
+    get_lambda_environment_info,
+    SlackClient
+)
+
+logger = get_powertools_logger("my-service")
+
+def handler(event, context):
+    env_info = get_lambda_environment_info()
+
+    try:
+        process_event(event)
+    except Exception as e:
+        logger.exception(
+            "Handler failed",
+            extra={
+                **env_info,
+                "error_type": type(e).__name__,
+                "error_message": str(e)
+            }
+        )
+
+        # Include environment context in Slack alerts
+        slack = SlackClient()
+        slack.send_message(
+            channel="#errors",
+            text=(
+                f"*Error in {env_info['function_name']}*\n"
+                f"Environment: {env_info['environment']}\n"
+                f"Region: {env_info['aws_region']}\n"
+                f"Error: {str(e)}"
+            )
+        )
+        raise
+
+    return {"statusCode": 200}
+```
+
+## Environment Detection
+
+### How `is_local` Works
+
+The `is_local` flag detects whether code is running in a Lambda environment:
+
+```python
+# Detection logic
+is_local = os.getenv("AWS_LAMBDA_RUNTIME_API") is None
+```
+
+**Detection Rules:**
+
+| `AWS_LAMBDA_RUNTIME_API` | Result | Scenario |
+|--------------------------|--------|----------|
+| Not set | `is_local = True` | Local dev, unit tests |
+| Set to any value | `is_local = False` | Lambda runtime |
+
+**Note:** SAM Local sets `AWS_LAMBDA_RUNTIME_API` during `sam local invoke`. Use `AWS_SAM_LOCAL` to detect SAM specifically if needed.
+
+### Testing Considerations
+
+For unit tests, mock environment variables:
+
+```python
+import pytest
+
+def test_handler_local_behavior(monkeypatch):
+    """Test handler behavior when running locally"""
+    monkeypatch.delenv("AWS_LAMBDA_RUNTIME_API", raising=False)
+
+    from nui_lambda_shared_utils import get_lambda_environment_info
+
+    env_info = get_lambda_environment_info()
+    assert env_info["is_local"] is True
+
+
+def test_handler_lambda_behavior(monkeypatch):
+    """Test handler behavior when running in Lambda"""
+    monkeypatch.setenv("AWS_LAMBDA_RUNTIME_API", "127.0.0.1:9001")
+    monkeypatch.setenv("ENVIRONMENT", "prod")
+    monkeypatch.setenv("AWS_REGION", "ap-southeast-2")
+    monkeypatch.setenv("AWS_LAMBDA_FUNCTION_NAME", "my-function")
+
+    from nui_lambda_shared_utils import get_lambda_environment_info
+
+    env_info = get_lambda_environment_info()
+    assert env_info["is_local"] is False
+    assert env_info["environment"] == "prod"
+    assert env_info["function_name"] == "my-function"
+```
+
+## Related Documentation
+
+- [Powertools Integration Guide](powertools-integration.md) - Logger and handler decorators
+- [Quick Start Guide](../getting-started/quickstart.md) - Package usage patterns
+- [CloudWatch Metrics](../getting-started/quickstart.md#cloudwatch-metrics) - Metrics publishing
+
+## Support
+
+For issues or questions:
+
+- [GitHub Issues](https://github.com/nuimarkets/nui-lambda-shared-utils/issues)
+- [Package Documentation](https://github.com/nuimarkets/nui-lambda-shared-utils)