libsrg 4.4.0__tar.gz

libsrg-4.4.0/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) [year] [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

libsrg-4.4.0/PKG-INFO

@@ -0,0 +1,102 @@
+Metadata-Version: 2.1
+Name: libsrg
+Version: 4.4.0
+Summary: Utility lib logging, statistics, subprocesses
+Author: Steve Goncalo
+Author-email: steven@goncalo.us
+Requires-Python: >=3.9,<4.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Utilities
+Requires-Dist: jinja2 (>=3.1.3,<4.0.0)
+Description-Content-Type: text/markdown
+
+## Name
+libsrg -- count calls to python logging at each logging severity level
+
+## Description
+libsrg provides the following classes:
+* libsrg.LoggingCounter
+  * extends logging.Handler with count of logs at each severity level
+  * counts on output, so logs suppressed by current logging level do not get counted
+  * atexit hook used to print statistics on program exit
+* libsrg.LoggingAppBase
+  * application base class with LoggingCounter installed
+  * argparse.ArgumentParser
+    * --verbose sets logging level to DEBUG
+    * --logfile <file> directs logging to a file
+  * Derived classes can add application specific options to ArgumentParser
+* libsrg.Runner
+  * run command line in a subprocess
+  * capture stdout, stderr
+  * try catch block for exceptions
+  * exceptions can optionally be raised back to calling program
+
+I write a lot of smallish command line utilities for internal use, and factored out some common 
+repetitive boilerplate code into this package. Published for my own installation convenience, 
+but might be useful to others as well. 
+
+## Example output
+
+Library uses atexit hook to summarize logging activity counts and total execution time. 
+Counts are also available to the program at runtime.
+```
+2022-02-22 12:46:22,568 INFO     (libsrg.LoggingCounter:61) __log_atexit 
+Logging Summary:
+Logging at Level INFO       occurred         13 times
+Logging at Level DEBUG      occurred          1 times
+Logging at Level WARNING    occurred          1 times
+Logging at Level ERROR      occurred          2 times
+Logging at Level CRITICAL   occurred          1 times
+Elapsed time was 0.087 seconds
+```
+
+## Internal notes on updating
+
+* /GNAS/PROJ/PycharmProjects/libsrg/publish.bash
+```
+#! /bin/bash
+
+cd /GNAS/PROJ/PycharmProjects/libsrg
+rm -rf dist
+python3 -m build --wheel
+python3 -m twine check dist/*
+python3 -m twine upload dist/*
+```
+
+## Installation from pypi via pip
+* pip3 install libsrg
+* pip3 install libsrg --update
+
+## Installation from Gitlab or local repository
+* pip3 install git+ssh://git@gitlab.com/SRG_gitlab/libsrg.git
+* pip3 install /GNAS/PROJ/PycharmProjects/libsrg
+
+## Testing
+
+Pytest performs its own logging setup before calling any user supplied tests, so it
+doesn't work well testing these classes logging setup. Ad-hoc test code is supplied
+at the end of the classes and executes if the class is loaded as __main__.
+
+
+## Roadmap
+My intention is to keep this library small. It is not expected to evolve into an all-purpose framework. 
+Output is geared towards developers, not end users. 
+
+## License
+MIT
+
+## Project status
+Core functionality appears solid, but naming and packaging may be fluid for a while.
+
+This is my first attempt at publishing to pypi, so there may be surprises. 
+
+I will bump version to 1.x when out of beta.

libsrg-4.4.0/README.md

@@ -0,0 +1,81 @@
+## Name
+libsrg -- count calls to python logging at each logging severity level
+
+## Description
+libsrg provides the following classes:
+* libsrg.LoggingCounter
+  * extends logging.Handler with count of logs at each severity level
+  * counts on output, so logs suppressed by current logging level do not get counted
+  * atexit hook used to print statistics on program exit
+* libsrg.LoggingAppBase
+  * application base class with LoggingCounter installed
+  * argparse.ArgumentParser
+    * --verbose sets logging level to DEBUG
+    * --logfile <file> directs logging to a file
+  * Derived classes can add application specific options to ArgumentParser
+* libsrg.Runner
+  * run command line in a subprocess
+  * capture stdout, stderr
+  * try catch block for exceptions
+  * exceptions can optionally be raised back to calling program
+
+I write a lot of smallish command line utilities for internal use, and factored out some common 
+repetitive boilerplate code into this package. Published for my own installation convenience, 
+but might be useful to others as well. 
+
+## Example output
+
+Library uses atexit hook to summarize logging activity counts and total execution time. 
+Counts are also available to the program at runtime.
+```
+2022-02-22 12:46:22,568 INFO     (libsrg.LoggingCounter:61) __log_atexit 
+Logging Summary:
+Logging at Level INFO       occurred         13 times
+Logging at Level DEBUG      occurred          1 times
+Logging at Level WARNING    occurred          1 times
+Logging at Level ERROR      occurred          2 times
+Logging at Level CRITICAL   occurred          1 times
+Elapsed time was 0.087 seconds
+```
+
+## Internal notes on updating
+
+* /GNAS/PROJ/PycharmProjects/libsrg/publish.bash
+```
+#! /bin/bash
+
+cd /GNAS/PROJ/PycharmProjects/libsrg
+rm -rf dist
+python3 -m build --wheel
+python3 -m twine check dist/*
+python3 -m twine upload dist/*
+```
+
+## Installation from pypi via pip
+* pip3 install libsrg
+* pip3 install libsrg --update
+
+## Installation from Gitlab or local repository
+* pip3 install git+ssh://git@gitlab.com/SRG_gitlab/libsrg.git
+* pip3 install /GNAS/PROJ/PycharmProjects/libsrg
+
+## Testing
+
+Pytest performs its own logging setup before calling any user supplied tests, so it
+doesn't work well testing these classes logging setup. Ad-hoc test code is supplied
+at the end of the classes and executes if the class is loaded as __main__.
+
+
+## Roadmap
+My intention is to keep this library small. It is not expected to evolve into an all-purpose framework. 
+Output is geared towards developers, not end users. 
+
+## License
+MIT
+
+## Project status
+Core functionality appears solid, but naming and packaging may be fluid for a while.
+
+This is my first attempt at publishing to pypi, so there may be surprises. 
+
+I will bump version to 1.x when out of beta.

libsrg-4.4.0/libsrg/AppTemplate.py

@@ -0,0 +1,37 @@
+import logging
+
+from libsrg.LoggingAppBase import LoggingAppBase
+
+"""
+This module is a sample application template for libsrg application logging
+"""
+
+
+class Other:
+    def __init__(self, n):
+        self.logger = logging.getLogger(self.__class__.__name__)
+        self.logger.info(f"Created Other {n}!")
+
+
+class SampleApp(LoggingAppBase):
+
+    def __init__(self):
+        super().__init__()  # super defines self.logger
+        self.logger.info("before adding args")
+        # setup any program specific command line arguments
+        self.parser.add_argument('--zap', help="Zap something", dest='zap', action='store_true', default=False)
+        self.parser.add_argument('--zip', help="Zip something", dest='zip', action='store_true', default=False)
+        # invoke the parser
+        self.perform_parse()
+        #
+        self.logger.info(f"after parsing {self.args}")
+
+    @classmethod
+    def demo(cls):
+        _ = SampleApp()
+        _ = Other(1)
+        _ = Other(2)
+
+
+if __name__ == '__main__':
+    SampleApp.demo()

libsrg-4.4.0/libsrg/Config.py

@@ -0,0 +1,266 @@
+import configparser
+import json
+import os
+import re
+from collections import ChainMap
+from pathlib import Path
+from typing import Any, Self, Optional, Tuple, TypeAlias
+
+from jinja2 import Template
+
+FileName: TypeAlias = str | bytes | os.PathLike
+ConfigSource: TypeAlias = ChainMap[str, Any] | dict[str, Any] | FileName
+
+
+class Config(ChainMap):
+    """
+    The Config class extends the ChainMap class to provide access to config files.
+
+    From outside, a Config looks like a series of python dictionaries which get searched in
+    order for a given configuation item,
+
+    Jinja argument processing can be applied on a per item basis using the get_item function,
+    Jinja processing is NOT applied as files are read in from the config file. Other items in the config
+    can be referenced in jinja templates.
+
+    The Config class maintains one special Config instance which contains secrets. These can be usernames,
+    passwords, api keys, or anything else that should not normally be kept in publicly accessible
+    configuration files. One file of secrets can be read from a secure location and referenced by items
+    in the normal config files.
+
+    """
+    _secret_config: Self | None = None
+
+    @classmethod
+    def set_secrets(cls, *args: ConfigSource) -> None:
+        """
+        Sets the secret config held by this class.
+        :param args: zero or more Config, ChainMap, dict[str, Any], Path
+        :return: None
+        """
+        cls._secret_config = cls(*args)
+
+    def __init__(self, *args: ConfigSource):
+        """
+        Constructs a new Config object.
+        :param args: zero or more Config, ChainMap, dict[str, Any], Path
+        """
+        args2 = self.process_args(*args)
+        super().__init__(*args2)
+
+    @classmethod
+    def process_args(cls, *args: ConfigSource) -> list[dict[str, Any]]:
+        """
+        Processes a mixed list of Config like inputs to a list of dictionaries.
+        * Paths or str filenames are opened and loaded from disk.
+        * Config or ChainMap inputs are split into their internal maps (preserving order)
+        :param args: zero or more Config, ChainMap, dict[str, Any], Path
+        :return: an ordered list of dict objects
+        """
+        pending: list[Any] = list(args)
+        out: list[dict[str, Any]] = []
+        while len(pending) > 0:
+            candidate = pending.pop(0)
+
+            # for ChainMap/Config inputs, add the individual maps
+            if isinstance(candidate, ChainMap):
+                maps = candidate.maps.copy()  # dont consume the source!
+                while len(maps) > 0:
+                    amap = maps.pop(-1)
+                    pending.insert(0, amap)
+                continue
+
+            # assume string argument is a pathname
+            if isinstance(candidate, str):
+                candidate = Path(candidate)
+
+            # open path arguments
+            if isinstance(candidate, Path):
+                with open(candidate, "r") as f:
+                    text = f.read()
+                # noinspection PyBroadException
+                candidate = cls.text_to_dict(text)
+
+            # only valid choice is dict here
+            if isinstance(candidate, dict):
+                out.append(candidate)
+            else:
+                raise TypeError(f"{type(candidate)} unexpected at {candidate}")
+        return out
+
+    @classmethod
+    def text_to_dict(cls, text: str) -> dict[str, Any]:
+        """
+        Converts a string into a dict. Content type is determined using a few heuristics.
+        * convert from json if a line starting with "{" is found
+        * convert from yaml if a line starting with "---" is found  TODO #36
+        * convert from ini/configparser if a line starting with "[" is found
+          * note first level in dict will be sections, not items
+        * convert from env/bash name=value  otherwise
+        :param text:
+        :return: ict[str, Any]
+        """
+        text = cls.strip_comments(text)
+        json_pat = re.compile(r"^\s*\{", re.MULTILINE)
+        if json_pat.match(text):
+            out = json.loads(text)
+            return out
+        ini_pat = re.compile(r"^\s*\[", re.MULTILINE)
+        if ini_pat.match(text):
+            cp = configparser.ConfigParser()
+            cp.read_string(text)
+            # reformat ini data as native dict of dicts
+            out = {name: {k: v for k, v in cp[name].items()} for name in cp.sections()}
+            return out
+        # fallback to flat env like file w/o [inisection]
+        cp = configparser.ConfigParser()
+        cp.read_string("[qqq]\n" + text)
+        out = {k: v for k, v in cp["qqq"].items()}
+        return out
+
+    @classmethod
+    def text_to_config(cls, text: str) -> Self:
+        return cls(cls.text_to_dict(text))
+
+    @classmethod
+    def strip_comments(cls, text: str, markers: Tuple[str] = None) -> str:
+        """
+        Strips line comments starting with markers
+        :param markers: tuple of line markers, defaults to ("#", ";", "//")
+        :param text: input text
+        :return: output text with line comments removed
+        """
+        if markers is None:
+            markers = ("#", ";", "//")
+        lines = text.split("\n")
+        new_lines = [line for line in lines if not line.strip(" \t").startswith(markers)]
+        return "\n".join(new_lines)
+
+    def get_item(self, *item_names: str, default=None, allow_none=False,
+                 secrets: bool = False, jinja=True) -> Any:
+        """
+        get an argument from self.merged_args
+        :param item_names: one or more alternate names, first found used
+        :param default: if none of the names are found return this value
+        :param allow_none: set True to suppress KeyError if returning None
+        :param jinja: enable jinja processing on item read from config
+        :param secrets: enable secrets in config chaining
+        :return: first item found or None if allow_none is set
+
+        This getter is designed to support gradual switchover from legacy names to newer names.
+
+        As a special case, if an Exception type is placed in the chain of adaption data, it will get raised.
+        This is intended to facilitate marking required parameters in default config data.
+        """
+        val = default
+        config = self
+        if secrets and self._secret_config is not None:
+            config = ChainMap(config, self._secret_config)
+        for name in item_names:
+            if name in config:
+                val = config[name]
+                break
+        if val is None and not allow_none:
+            raise KeyError(f"keys not found: {item_names}")
+        if isinstance(val, Exception):
+            raise val
+        if jinja or secrets:
+            # config already has secrets if enabled
+            val = self.apply_templates_to_strings(val, config=config, secrets=False)
+        return val
+
+    def apply_templates_to_strings(self, data: Any, config: Optional[Self] = None,
+                                   secrets: bool = False) -> Any:
+        """
+        apply templates to strings in data (applies to str or list of str, else return unchanged
+          * if data is a str, apply templates
+          * if data is a list containing ONLY str, apply recursively for one level only
+          * anything else returns unmodified
+        :param data: data to be processed
+        :param config: defaults to self, if caller has already appended secrets, that can be passed here
+        :param secrets: append secrets before templates
+        :return: result
+        """
+        if config is None:
+            config = self
+        if secrets:
+            config = ChainMap(config, self._secret_config)
+        if isinstance(data, str):
+            template = Template(data)
+            val = template.render(config)
+            return val
+        if isinstance(data, list):
+            # only apply to list of str, not list of Any
+            if all([isinstance(x, str) for x in data]):
+                return [self.apply_templates_to_strings(x, config=config, secrets=False) for x in data]
+        return data
+
+    def set_item_if_missing(self, key: str, val: Any) -> None:
+        """
+        Convenience method for set_item which prevents overwriting existing item value
+        :param key: name of item
+        :param val: value of item
+        :return: None
+        """
+        self.set_item(key, val, overwrite=False)
+
+    def set_item(self, key: str, value: Any, overwrite=True, ) -> None:
+        """
+        Set one item in config
+        :param key: item name
+        :param value: item value
+        :param overwrite: Allow overwriting existing item (defaults True)
+        :return: None
+        """
+        if key not in self or overwrite:
+            self[key] = value
+
+    def set_items(self, overwrite: bool = True, **key_value_pairs) -> None:
+        """Set one or more items in config using key/value pairs
+        :param overwrite: Allow overwriting existing item (defaults True)
+        :param key_value_pairs: key/value pairs to be set
+        :return: None
+        """
+        for key, val in key_value_pairs.items():
+            self.set_item(key, val, overwrite=overwrite)
+
+    def find_item_depth(self, item: str) -> Tuple[Optional[int], Any]:
+        """Looks for item in maps
+        :param item: item name
+        :return: (depth, value) if found tuple with depth and value, else (None,None)
+        """
+        for depth, dct in enumerate(self.maps):
+            if item in dct:
+                return depth, dct[item]
+        return None, None
+
+    def to_list(self) -> list[tuple[str, Any, int]]:
+        """Finds and alphabetizes all config items in config
+        :return: Ordered list of tuples (name, value, depth)
+        """
+        return self.config_to_list(self)
+
+    @classmethod
+    def config_to_list(cls, config: Self) -> list[tuple[str, Any, int]]:
+        """Finds and alphabetizes all config items in config
+        :param config: Config instance
+        :return: Ordered list of tuples (name, value, depth)
+        """
+        names = list(config.keys())
+        names.sort()
+        lst = []
+        for name in names:
+            depth, val = config.find_item_depth(name)
+            lst.append((name, val, depth))
+        return lst
+
+    def to_json_str(self, **kwargs) -> str:
+        return json.dumps(self.to_flat_dict(), **kwargs)
+
+    def to_json_file(self, file: FileName, **kwargs):
+        txt = self.to_json_str(**kwargs)
+        with open(file, 'w') as f:
+            f.write(txt)
+
+    def to_flat_dict(self, ) -> dict[str, Any]:
+        return dict(self)

libsrg-4.4.0/libsrg/ElapsedTime.py

@@ -0,0 +1,66 @@
+"""
+ElapsedTime is a class to measure elapsed time between start and stop events.
+It can be operated directly using start and stop methods, or it can
+act as a context manager in a "with" statement block.
+"""
+import logging
+from datetime import timedelta
+from time import time
+
+
+class ElapsedTime:
+    """Constructs ElapsedTime object, started"""
+
+    def __init__(self, name=None):
+        self.logger = logging.getLogger(self.__class__.__name__)
+        if name:
+            self.name = name
+        else:
+            self.name = repr(self)
+        self._starttime = 0
+        self._endtime = 0
+        self._elapsed = 0
+        self.start()
+
+    def elapsed_asc(self) -> str:
+        td = timedelta(seconds=self._elapsed)
+        return str(td)
+
+    def __str__(self):
+        return f"ET({self.name!r},{self.elapsed_asc()},{self._elapsed})"
+
+    """records start time and zeros elapsed
+    if called more than once, last call wipes any previous data"""
+
+    def start(self):
+        self._starttime = time()
+        self._endtime = self._starttime
+        self._elapsed = 0
+
+    """records stop time, computes and returns elapsed
+    if called more than once, each call records time since last start"""
+
+    def stop(self) -> float:
+        self._endtime = time()
+        self._elapsed = self._endtime - self._starttime
+        return self._elapsed
+
+    """returns elapsed time at last stop, but does not perform a stop"""
+
+    def elapsed(self) -> float:
+        return self._elapsed
+
+    """returns time since last start, but does not perform a stop"""
+
+    def current(self) -> float:
+        return time() - self._starttime
+
+    """translates enter "with" statement into start command"""
+
+    def __enter__(self):
+        self.start()
+
+    """translates exit "with" statement into stop command"""
+
+    def __exit__(self, exc_type, exc_value, exc_tb):
+        self.stop()