nui-python-shared-utils 2.4.1__tar.gz → 2.4.2__tar.gz

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/AGENTS.md

@@ -194,6 +194,11 @@ Clients resolve credentials in this order (first match wins):
 2. **Environment variables** — per-client patterns, skips Secrets Manager
 3. **AWS Secrets Manager** — default behavior
 
+The `get_*_credentials` helpers in `secrets_helper.py` follow tiers 2 and 3 through the same
+per-service env resolver and secret normaliser the clients use. Elasticsearch host:
+`host` argument > `ES_HOST`/`ELASTICSEARCH_HOST` > `configure(es_host=...)` > secret `host` >
+`localhost:9200`.
+
 ### Environment Variables
 
 Runtime configuration:

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/CHANGELOG.md

@@ -10,11 +10,39 @@ that contained them.
 
 ### Changed
 
+- **BREAKING**: `get_database_credentials` and `get_slack_credentials` check environment
+  variables before Secrets Manager, matching `DatabaseClient`, `SlackClient` and
+  `get_elasticsearch_credentials`. With `DB_HOST` and `DB_PASSWORD` set, or `SLACK_BOT_TOKEN`
+  set, they return the env values (MySQL defaults for unset `DB_PORT`, `DB_USERNAME`,
+  `DB_DATABASE`) and make no AWS call, even when a `secret_name` is passed, and they no longer
+  raise for a missing secret name in that case. A caller that reads a second database or
+  workspace through `get_database_credentials("<other secret>")` or
+  `get_slack_credentials("<other secret>")` while the `DB_*` or `SLACK_BOT_TOKEN` env vars
+  describe the first gets the first; read that secret with `get_secret` and map its fields.
+- `ElasticsearchClient` reads the `host` field of its resolved credentials. Host order on the
+  client and `get_elasticsearch_credentials` is: `host` argument (client only), `ES_HOST` or
+  `ELASTICSEARCH_HOST`, a host set with `configure(es_host=...)`, the credentials' `host`, then
+  `localhost:9200`. A client built from a secret that carries a host, with neither `ES_HOST` nor
+  `configure(es_host=...)` set, connects to that host instead of `localhost:9200`. The helper
+  appends `:9200` to any host that is not a URL and names no port, including one that starts
+  with `http` (`httpd-es`), as the client does.
+- `ElasticsearchClient` and `SlackClient` add the normalised fields the helpers return to
+  credentials fetched from Secrets Manager (`host`/`username`/`password`, and
+  `bot_token`/`webhook_url`), keeping the secret's other keys. A legacy `token` field is used
+  when `bot_token` is missing, null or empty; the helper used to keep a null or empty
+  `bot_token`. `SlackClient.credentials` from env vars always carries a `webhook_url` key,
+  `None` when `SLACK_WEBHOOK_URL` is unset.
+
 - **BREAKING**: require Python 3.10 or newer in both the main package and the legacy
   redirect package. Python 3.9 is no longer supported or included in CI test matrices.
+- Powertools docstring and guide examples use generic service and namespace names
+  (`order-processor`, `Ecommerce/Orders`) instead of internal repo names. Docs only, no
+  behaviour change.
 
 ### Fixed
 
+- `PostgreSQLClient` given a secret with no `port` or `database` fills PostgreSQL defaults
+  (`5432`, `postgres`) instead of the MySQL ones (`3306`, `app`).
 - Use canonical IANA names `America/New_York` and `America/Los_Angeles` for US timezone
   constants and option labels, so importing timezone utilities and Slack formatting works
   on systems without legacy timezone links.

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nui-python-shared-utils
-Version: 2.4.1
+Version: 2.4.2
 Summary: Shared Python utilities for AWS Lambda, CLI tools, and agents with Slack, Elasticsearch, and monitoring integrations
 Home-page: https://github.com/nuimarkets/nui-python-shared-utils
 Author: NUI Markets
@@ -221,7 +221,7 @@ from nui_shared_utils import get_secret, get_slack_credentials
 api_keys = get_secret("my-service/api-keys")
 
 # Specialized getters with normalized field names
-slack_creds = get_slack_credentials()  # Uses configured secret name
+slack_creds = get_slack_credentials()  # SLACK_BOT_TOKEN if set, else the configured secret
 ```
 
 **[→ See full secrets management guide](docs/getting-started/configuration.md#aws-secrets-manager)**

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/README.md

@@ -121,7 +121,7 @@ from nui_shared_utils import get_secret, get_slack_credentials
 api_keys = get_secret("my-service/api-keys")
 
 # Specialized getters with normalized field names
-slack_creds = get_slack_credentials()  # Uses configured secret name
+slack_creds = get_slack_credentials()  # SLACK_BOT_TOKEN if set, else the configured secret
 ```
 
 **[→ See full secrets management guide](docs/getting-started/configuration.md#aws-secrets-manager)**

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/docs/decisions/ADR-004-credential-resolution-precedence.md

@@ -24,14 +24,19 @@ The function-based helper historically undercut this. `get_elasticsearch_credent
 (`secrets_helper.py`) called Secrets Manager first even when `ES_PASSWORD` was set, so CLI and
 local-dev paths paid an avoidable AWS call. Commit `0d3f8e8` (PR #23) added the env-var check
 ahead of the store lookup, aligning the helper with the precedence the clients already enforced.
+The database and Slack helpers kept calling the store first until they were routed through the
+clients' own env resolvers, and the Elasticsearch client ignored a secret's `host` field that the
+helper honoured.
 
 ## Decision
 
 Credentials resolve in a fixed order, first match wins, owned by `BaseClient._resolve_credentials`
 in `base_client.py`:
 
-1. **Explicit `credentials` dict** passed to the constructor. Returned as-is; nothing else is
-   consulted. The caller owns these values and the package does not merge or second-guess them.
+1. **Explicit `credentials` dict** passed to the constructor. Returned as-is; no other credential
+   source is consulted. The caller owns these values and the package does not merge or
+   second-guess them. An Elasticsearch `host` key in the dict is the one exception: it follows
+   the host order below.
 2. **Per-client environment variables**, via each client's `_resolve_credentials_from_env` override
    (Elasticsearch reads `ES_PASSWORD`/`ES_PASS` in `es_client.py`; database requires
    `DB_HOST` + `DB_PASSWORD` in `db_client.py`; Slack requires `SLACK_BOT_TOKEN` in
@@ -42,9 +47,19 @@ in `base_client.py`:
 
 A later tier fires only when every earlier tier is absent. An AWS call happens only when neither
 an explicit dict nor env vars resolve, so local-dev, CLI, and test paths that supply explicit or
-environment credentials make **zero** Secrets Manager calls. The direct function helper mirrors
-the tail of this ladder for callers that bypass the client classes:
-`get_elasticsearch_credentials` checks env vars, then the store.
+environment credentials make **zero** Secrets Manager calls. The direct function helpers mirror
+the tail of this ladder for callers that bypass the client classes: `get_elasticsearch_credentials`,
+`get_database_credentials` and `get_slack_credentials` each check the same env vars as their
+client, then the store. Client hook and helper call the same per-service functions in
+`secrets_helper.py` (an env resolver and a secret normaliser per service, with per-engine defaults
+for MySQL and PostgreSQL), so the two paths cannot drift apart.
+
+The Elasticsearch host follows its own order on both paths: the client's `host` argument, then
+`ES_HOST` (or `ELASTICSEARCH_HOST`), then the `host` field of the resolved credentials (the
+secret's, or an explicit dict's), then `localhost:9200`. A host set with `configure(es_host=...)`
+ranks between the env var and the credentials' `host`, because a caller that set it chose it; an
+`ES_HOST` the config picked up from the environment at startup does not. To pin the host of one
+client whatever the environment holds, pass its `host` argument.
 
 ## Alternatives considered
 

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/docs/getting-started/configuration.md

@@ -18,6 +18,18 @@ All clients (Slack, Elasticsearch, Database) resolve credentials in this order:
 2. **Credential environment variables** - Per-client env vars, skips Secrets Manager
 3. **AWS Secrets Manager** - Default behavior (unchanged)
 
+The helper functions `get_elasticsearch_credentials`, `get_database_credentials` and
+`get_slack_credentials` follow the same order from step 2: when the client's env vars are set
+they return those values without calling Secrets Manager. `get_database_credentials` applies the
+MySQL defaults; `PostgreSQLClient` applies the PostgreSQL ones on both the env and the secret path.
+
+The Elasticsearch host resolves as: the client's `host` argument, then `ES_HOST` (or
+`ELASTICSEARCH_HOST`), then a host set with `nui.configure(es_host=...)`, then the secret's
+`host` field, then the built-in default
+(`localhost:9200`). The client and `get_elasticsearch_credentials` use the same order. A `host`
+key in an explicit `credentials` dict ranks where the secret's does, so to pin one client's host
+pass `host=` instead.
+
 ## Environment Variables
 
 ### Core Configuration
@@ -163,7 +175,8 @@ The package expects AWS secrets in specific JSON formats:
 
 **Optional Fields:**
 
-- `host` - Can override ES_HOST if provided
+- `host` - Used when the credentials come from this secret (no `ES_PASSWORD` or `ES_PASS`) and
+  no `host=` argument, `ES_HOST`, `ELASTICSEARCH_HOST` or `configure(es_host=...)` names one
 - `username` - Defaults to "elastic" if not provided
 
 #### Database Credentials
@@ -178,7 +191,8 @@ The package expects AWS secrets in specific JSON formats:
 }
 ```
 
-**All fields are required** for database connections.
+`host`, `username` and `password` are required. A missing `port` or `database` takes the
+engine's default: `3306`/`app` for MySQL, `5432`/`postgres` for `PostgreSQLClient`.
 
 #### Slack Credentials
 
@@ -274,6 +288,8 @@ slack = nui.SlackClient(secret_name="custom-slack-secret")
 es_creds = nui.get_elasticsearch_credentials("custom-es-secret")
 db_creds = nui.get_database_credentials("custom-db-secret")
 slack_creds = nui.get_slack_credentials("custom-slack-secret")
+# These three return the env values instead, ignoring the secret name, when the
+# client's credential env vars are set (ES_PASSWORD, DB_HOST + DB_PASSWORD, SLACK_BOT_TOKEN).
 
 # Generic secret retrieval
 api_keys = nui.get_secret("my-service/api-keys")

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/docs/guides/powertools-integration.md

@@ -80,7 +80,7 @@ def get_powertools_logger(
     Create AWS Powertools Logger with Elasticsearch-compatible formatting.
 
     Args:
-        service_name: Service identifier (e.g., "nui-tender-analyser")
+        service_name: Service identifier (e.g., "order-processor")
         level: Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
         local_dev_colors: Enable coloredlogs for local development
 
@@ -177,7 +177,7 @@ def powertools_handler(
 
     Args:
         service_name: Service identifier for logging and metrics dimensions
-        metrics_namespace: CloudWatch namespace (e.g., "NUI/TenderAnalyser")
+        metrics_namespace: CloudWatch namespace (e.g., "Ecommerce/Orders")
         slack_alert_channel: Slack channel for error alerts (e.g., "#alerts")
         slack_account_names: AWS account ID to display-name mapping for alerts
         slack_account_names_config: YAML file containing account-name defaults

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/nui_shared_utils/config.py

@@ -34,7 +34,9 @@ class Config:
             slack_credentials_secret: AWS secret name for Slack credentials
             aws_region: AWS region for secrets/services
         """
-        # Elasticsearch settings
+        # Elasticsearch settings. An es_host passed here outranks a host carried in a
+        # credentials secret; the env-derived and built-in values do not.
+        self._es_host_explicit = es_host is not None
         if es_host is not None:
             self.es_host = es_host
         else:

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/nui_shared_utils/db_client.py

@@ -2,7 +2,6 @@
 Refactored Database client using BaseClient for DRY code patterns.
 """
 
-import os
 import time
 import logging
 from typing import Dict, List, Optional, Any
@@ -11,7 +10,7 @@ import pymysql
 
 from .base_client import BaseClient, ServiceHealthMixin
 from .utils import handle_client_errors, safe_close_connection
-from .secrets_helper import get_database_credentials
+from .secrets_helper import get_database_credentials, _db_credentials_from_env, _normalise_db_secret
 
 # Optional PostgreSQL support
 try:
@@ -27,47 +26,6 @@ log = logging.getLogger(__name__)
 _connection_pool = {}
 
 
-def _clean_expired_connections(pool_key: str, pool_recycle: int) -> None:
-    """
-    Clean expired connections from the pool.
-
-    Args:
-        pool_key: Pool identifier
-        pool_recycle: Maximum connection age in seconds (0 disables recycling)
-    """
-    if not pool_recycle or pool_recycle <= 0:
-        return
-    
-    if pool_key not in _connection_pool:
-        return
-    
-    current_time = time.time()
-    pool_entries = _connection_pool[pool_key]
-    
-    # Find expired connections
-    expired_connections = []
-    fresh_connections = []
-    
-    for entry in pool_entries:
-        age = current_time - entry["timestamp"]
-        if age > pool_recycle:
-            expired_connections.append(entry["connection"])
-        else:
-            fresh_connections.append(entry)
-    
-    # Close expired connections
-    for conn in expired_connections:
-        safe_close_connection(conn)
-    
-    # Update pool with only fresh connections
-    _connection_pool[pool_key] = fresh_connections
-    
-    if expired_connections:
-        log.debug(
-            f"Cleaned {len(expired_connections)} expired connections from pool {pool_key}"
-        )
-
-
 def get_pool_stats() -> Dict[str, Any]:
     """
     Get current connection pool statistics.
@@ -158,26 +116,8 @@ class DatabaseClient(BaseClient, ServiceHealthMixin):
         return None
 
     def _resolve_credentials_from_env(self) -> Optional[Dict[str, Any]]:
-        """Resolve MySQL credentials from environment variables.
-
-        Requires both DB_HOST and DB_PASSWORD to trigger.
-        """
-        host = os.environ.get("DB_HOST")
-        password = os.environ.get("DB_PASSWORD")
-        if not host or not password:
-            return None
-        port_str = os.environ.get("DB_PORT", "3306")
-        try:
-            port = int(port_str)
-        except ValueError:
-            raise ValueError(f"DB_PORT must be an integer, got: {port_str!r}")
-        return {
-            "host": host,
-            "port": port,
-            "username": os.environ.get("DB_USERNAME", "root"),
-            "password": password,
-            "database": os.environ.get("DB_DATABASE", "app"),
-        }
+        """Resolve MySQL credentials from DB_* env vars (DB_HOST + DB_PASSWORD required)."""
+        return _db_credentials_from_env("mysql")
 
     def _fetch_credentials_from_sm(self, secret_name: Optional[str]) -> Dict[str, Any]:
         """Override to use get_database_credentials for normalized field names."""
@@ -511,29 +451,11 @@ class PostgreSQLClient(BaseClient, ServiceHealthMixin):
         return None
 
     def _resolve_credentials_from_env(self) -> Optional[Dict[str, Any]]:
-        """Resolve PostgreSQL credentials from environment variables.
-
-        Requires both DB_HOST and DB_PASSWORD to trigger.
-        """
-        host = os.environ.get("DB_HOST")
-        password = os.environ.get("DB_PASSWORD")
-        if not host or not password:
-            return None
-        port_str = os.environ.get("DB_PORT", "5432")
-        try:
-            port = int(port_str)
-        except ValueError:
-            raise ValueError(f"DB_PORT must be an integer, got: {port_str!r}")
-        return {
-            "host": host,
-            "port": port,
-            "username": os.environ.get("DB_USERNAME", "postgres"),
-            "password": password,
-            "database": os.environ.get("DB_DATABASE", "postgres"),
-        }
+        """Resolve PostgreSQL credentials from DB_* env vars (DB_HOST + DB_PASSWORD required)."""
+        return _db_credentials_from_env("postgresql")
 
     def _fetch_credentials_from_sm(self, secret_name: Optional[str]) -> Dict[str, Any]:
-        """Override with auth-specific credential handling."""
+        """Fetch from Secrets Manager, preferring the auth_* keys when requested and present."""
         from .secrets_helper import get_secret
 
         resolved_secret_name = self._resolve_secret_name(secret_name)
@@ -548,8 +470,7 @@ class PostgreSQLClient(BaseClient, ServiceHealthMixin):
                 "password": raw_creds.get("auth_password"),
                 "database": raw_creds.get("auth_database", "auth-service-db"),
             }
-        else:
-            return get_database_credentials(resolved_secret_name)
+        return _normalise_db_secret(raw_creds, "postgresql")
 
     @contextmanager
     def get_connection(self, database: Optional[str] = None):

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/nui_shared_utils/es_client.py

@@ -2,7 +2,6 @@
 Refactored Elasticsearch client using BaseClient for DRY code patterns.
 """
 
-import os
 import logging
 from datetime import datetime, timedelta
 from typing import Dict, Iterator, List, Optional, Any, Tuple
@@ -11,7 +10,8 @@ from elasticsearch import Elasticsearch
 from elasticsearch.helpers import streaming_bulk as es_streaming_bulk
 
 from .base_client import BaseClient, ServiceHealthMixin
-from .utils import handle_client_errors, resolve_config_value
+from .utils import handle_client_errors
+from .secrets_helper import _es_credentials_from_env, _es_host_with_port, _normalise_es_secret, _resolve_es_host
 
 log = logging.getLogger(__name__)
 
@@ -55,29 +55,27 @@ class ElasticsearchClient(BaseClient, ServiceHealthMixin):
         Accepts both conventions: ES_PASSWORD/ES_USERNAME and ES_PASS/ES_USER.
         Password is required to trigger. Username defaults to "elastic".
         """
-        password = os.environ.get("ES_PASSWORD") or os.environ.get("ES_PASS")
-        if not password:
-            return None
-        return {
-            "username": os.environ.get("ES_USERNAME") or os.environ.get("ES_USER") or "elastic",
-            "password": password,
-        }
+        return _es_credentials_from_env()
+
+    def _fetch_credentials_from_sm(self, secret_name: Optional[str]) -> Dict[str, Any]:
+        """Fetch from Secrets Manager; normalise host, username, password and keep any other keys."""
+        raw = super()._fetch_credentials_from_sm(secret_name)
+        return {**raw, **_normalise_es_secret(raw)}
 
     def _create_service_client(self) -> Elasticsearch:
         """Create Elasticsearch client with resolved configuration."""
-        # Resolve host using utility
-        host = resolve_config_value(
-            self._host_override,
-            ["ES_HOST", "ELASTICSEARCH_HOST"],
-            getattr(self.config, "es_host", "localhost:9200")
-        )
+        # Host precedence: constructor arg > ES_HOST/ELASTICSEARCH_HOST > configure(es_host=...)
+        # > credentials "host" (e.g. from the secret) > default. Shared with get_elasticsearch_credentials.
+        if self._host_override is not None:
+            host = self._host_override
+        else:
+            host = _resolve_es_host(self.credentials.get("host"))
+        host = _es_host_with_port(host)
 
         parsed = urlparse(host)
         if parsed.scheme and parsed.netloc:  # Valid URL with scheme://netloc
             es_url = host
         else:
-            if ":" not in host:
-                host = f"{host}:9200"
             scheme = self.client_config.get("scheme", "http")
             es_url = f"{scheme}://{host}"
 

{nui_python_shared_utils-2.4.1 → nui_python_shared_utils-2.4.2}/nui_shared_utils/powertools_helpers.py

@@ -75,7 +75,7 @@ def get_powertools_logger(
     and enforces UTC timezone for consistency with log aggregation systems.
 
     Args:
-        service_name: Service identifier (e.g., "nui-tender-analyser", "connect-email-ingest")
+        service_name: Service identifier (e.g., "order-processor", "email-ingest")
         level: Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL). Default: INFO
         local_dev_colors: Enable coloredlogs for local development. Default: True
 
@@ -182,7 +182,7 @@ def powertools_handler(
 
     Args:
         service_name: Service identifier for logging and metrics dimensions
-        metrics_namespace: CloudWatch namespace for metrics (e.g., "NUI/TenderAnalyser").
+        metrics_namespace: CloudWatch namespace for metrics (e.g., "Ecommerce/Orders").
                           If None, metrics publishing is disabled.
         slack_alert_channel: Slack channel for error alerts (e.g., "#alerts", "#errors").
                             If None, Slack alerting is disabled.