pybasemkit 0.2.0__tar.gz → 0.2.2__tar.gz

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/.github/workflows/upload-to-pypi.yml

@@ -19,7 +19,8 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install hatch
+        # work around https://github.com/pypa/hatch/issues/2198
+        pip install 'virtualenv<20.26.0' hatch
     - name: Build and publish
       run: |
         hatch build

pybasemkit-0.2.2/AGENTS.md

@@ -0,0 +1,284 @@
+# AGENTS.md — pybasemkit
+
+Coding agent instructions for the `pybasemkit` repository.
+Package installs as `basemkit`; PyPI name is `pybasemkit`.
+
+---
+
+## Project Overview
+
+- **Python:** `>=3.10` (CI runs 3.12; classifiers cover 3.10–3.13)
+- **Build backend:** `hatchling` (configured entirely in `pyproject.toml`)
+- **Runtime deps:** `dacite`, `dataclasses-json`, `PyYAML`, `shutup`
+- **Test deps (optional):** `pytest`, `green`, `tox`
+- **Dev deps (optional):** `black`, `isort`
+
+---
+
+## Build & Install
+
+```bash
+# Install from source (editable not required; plain install is the norm)
+pip install .
+
+# Install with test extras
+pip install ".[test]"
+
+# Install with dev extras
+pip install ".[dev]"
+
+# Build wheel + sdist
+hatch build
+```
+
+---
+
+## Formatting
+
+Two tools are used together. Always run both before committing:
+
+```bash
+# Format all source and test files (isort then black)
+scripts/blackisort
+```
+
+Under the hood this runs:
+```bash
+isort basemkit/*.py
+black basemkit/*.py
+isort tests/*.py
+black tests/*.py
+```
+
+**Line length is 120 characters** (configured in `[tool.black]` in `pyproject.toml`).
+
+No other linters (flake8, ruff, pylint, mypy) are configured.
+
+---
+
+## Running Tests
+
+The default runner is `unittest discover`. All test classes inherit from
+`basemkit.basetest.Basetest` (which subclasses `unittest.TestCase`).
+`pytest` also works because it collects unittest-style classes.
+
+```bash
+# Run the full test suite (default)
+python3 -m unittest discover
+
+# Using the project script (same as above)
+scripts/test
+
+# Using green (colorful output, serial)
+scripts/test -g
+# or:
+green tests/ -s 1
+
+# Using tox
+scripts/test -t
+# or:
+tox -e py
+
+# Using pytest
+python -m pytest tests/
+```
+
+### Running a Single Test
+
+```bash
+# Single test module
+python -m unittest tests.test_yamlable
+
+# Single test class
+python -m unittest tests.test_yamlable.TestYamlAble
+
+# Single test method  ← most common for targeted debugging
+python -m unittest tests.test_yamlable.TestYamlAble.test_to_yaml
+
+# With pytest
+python -m pytest tests/test_yamlable.py
+python -m pytest tests/test_yamlable.py::TestYamlAble::test_to_yaml
+
+# With green
+green tests/test_yamlable.py -s 1
+```
+
+---
+
+## Code Style
+
+### Imports
+
+Follow PEP 8 import order, sorted by `isort` (default profile):
+
+1. Standard library
+2. Third-party packages
+3. Local `basemkit.*` imports
+
+```python
+import sys
+import traceback
+from argparse import ArgumentParser, Namespace
+from typing import Any, Dict, List, Optional, Tuple, Type, TypeVar
+
+import yaml
+from dacite import from_dict
+
+from basemkit.yamlable import YamlAble
+```
+
+Do **not** use `from __future__ import annotations`. Use `from typing import ...` imports for all type hints.
+
+### Type Annotations
+
+Every function parameter and return type must be annotated.
+
+```python
+def load_from_yaml_file(cls: Type[T], filename: str) -> T: ...
+def run(self, cmd: str, text: bool = True, debug: bool = False) -> subprocess.CompletedProcess: ...
+def get_level_summary(self, level: str, limit: int = 7) -> Tuple[int, str]: ...
+```
+
+- Use `Optional[X]` (not `X | None`) — keeps 3.10 compatibility explicit
+- Use `Union[str, Path]` (not `str | Path`)
+- Use `TypeVar` for generic patterns: `T = TypeVar("T")`
+
+### Naming Conventions
+
+| Category | Convention | Examples |
+|---|---|---|
+| Classes | `PascalCase` | `YamlAble`, `BaseCmd`, `Basetest`, `ShellResult` |
+| Functions / methods | `snake_case` | `to_yaml`, `load_from_yaml_file`, `handle_exception` |
+| Instance / local variables | `snake_case` | `self.shell_path`, `self.do_log`, `result` |
+| Module-level constants | `UPPER_SNAKE_CASE` | `BLUE`, `RED`, `GREEN`, `END_COLOR` |
+| Private / internal | `_leading_underscore` | `_yaml_setup`, `_yaml_dumper`, `_run` |
+| TypeVars | Single capital | `T = TypeVar("T")` |
+
+Avoid introducing new `camelCase` method names; those that exist are legacy compatibility shims.
+
+### Docstrings
+
+Use **Google-style** docstrings (configured in `mkdocs.yml` as `docstring_style: google`).
+
+```python
+def to_yaml(
+    self,
+    ignore_none: bool = True,
+    sort_keys: bool = False,
+) -> str:
+    """
+    Convert this dataclass object to a YAML string.
+
+    Args:
+        ignore_none: Omit attributes whose value is None.
+        sort_keys: Sort dictionary keys in the output.
+
+    Returns:
+        YAML string representation of the dataclass.
+
+    Raises:
+        ValueError: If the object is not a dataclass instance.
+    """
+```
+
+- One-liner docstrings are acceptable for trivial methods.
+- Module-level docstrings use the format:
+
+```python
+"""
+Created on YYYY-MM-DD
+
+@author: wf
+"""
+```
+
+### Error Handling
+
+```python
+# Preferred: catch specific exceptions; use debug flag for traceback
+try:
+    result = self.shell.run(command, debug=self.debug)
+except Exception as ex:
+    self.handle_exception(f"command '{command}'", ex)
+
+# Optional imports that may not be installed: guard with try/except ImportError
+try:
+    import pydevd
+except ImportError:
+    print("Error: 'pydevd' is required for remote debugging.", file=sys.stderr)
+    return
+
+# Silently ignore cleanup errors
+try:
+    os.unlink(tmp_path)
+except Exception:
+    pass
+
+# Top-level entry points: catch BaseException and map to exit codes
+try:
+    args = self.parse_args(argv)
+    self.handle_args(args)
+except BaseException as e:
+    exit_code = self.handle_exception(e)
+```
+
+### Module Organization
+
+1. Module docstring (creation date + author)
+2. Imports (stdlib → third-party → local)
+3. Module-level constants
+4. Class definitions (one primary class per module)
+5. No `if __name__ == "__main__"` guard except in `basetest.py`
+
+---
+
+## Writing Tests
+
+All test classes must subclass `Basetest`, not `unittest.TestCase` directly.
+
+```python
+from basemkit.basetest import Basetest
+
+class TestMyFeature(Basetest):
+    """Tests for my_feature."""
+
+    def setUp(self, debug=False, profile=True):
+        Basetest.setUp(self, debug=debug, profile=profile)
+        # additional setup here
+
+    def test_something(self):
+        # use self.debug for conditional diagnostic prints
+        result = my_function()
+        self.assertEqual(result, expected)
+```
+
+Key `Basetest` utilities:
+
+```python
+Basetest.inPublicCI()   # True when running under GitHub Actions / Travis / Jenkins
+Basetest.isUser("wf")  # True if the current user matches
+```
+
+`Basetest.setUp` creates a `Profiler` that automatically times each test and prints
+elapsed time in `tearDown`.
+
+---
+
+## CI / CD
+
+- **CI:** `.github/workflows/build.yml` — runs `scripts/install && scripts/test` on every push/PR to `main` (Python 3.12, ubuntu-latest). Env var `GHACTIONS=ACTIVE` is set.
+- **CD:** `.github/workflows/upload-to-pypi.yml` — runs `hatch build` and publishes to PyPI via OIDC trusted publishing on GitHub release creation.
+- No coverage reporting, no linting step, and no multi-version matrix in CI currently.
+
+---
+
+## Repository Layout
+
+```
+basemkit/           # Main package (import as "basemkit")
+tests/              # unittest-style tests; filenames match test_<module>.py
+scripts/            # Shell scripts: blackisort, test, install, doc, release
+docs/               # MkDocs source (material theme + mkdocstrings)
+pyproject.toml      # All project metadata, build config, and tool settings
+mkdocs.yml          # Documentation site config
+```

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pybasemkit
-Version: 0.2.0
+Version: 0.2.2
 Summary: Python base module kit: YAML/JSON I/O, structured logging, CLI tooling, shell execution, and pydevd remote debug support.
 Project-URL: Home, https://github.com/WolfgangFahl/pybasemkit
 Project-URL: Documentation, https://wiki.bitplan.com/index.php/pybasemkit

pybasemkit-0.2.2/basemkit/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.2"

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/basemkit/remotedebug.py

@@ -19,6 +19,7 @@ https://github.com/fabioz/PyDev.Debugger/blob/main/pydevd_file_utils.py
 
 @author: wf
 """
+
 import os
 import socket
 import sys
@@ -26,6 +27,7 @@ from argparse import Namespace
 from dataclasses import dataclass, field
 from typing import List, Tuple
 
+
 @dataclass
 class PathMapping:
     """
@@ -58,18 +60,21 @@ class PathMapping:
         the source code sits on your physical laptop.
 
     """
-    remote: str # eclipse / vscode/ pycharm
+
+    remote: str  # eclipse / vscode/ pycharm
     local: str  # python execution environment path to be remotely debugged
 
+
 @dataclass
 class PathMappings:
     """
     Represents a collection of path mappings.
     """
+
     mappings: List[PathMapping] = field(default_factory=list)
 
     @classmethod
-    def from_args(cls, remote_str: str, local_str: str) -> 'PathMappings':
+    def from_args(cls, remote_str: str, local_str: str) -> "PathMappings":
         """
         Parses comma-separated remote
         and local path strings into a PathMappings object.
@@ -81,16 +86,17 @@ class PathMappings:
             raise ValueError("debugRemotePath and debugLocalPath must have the same number of entries")
 
         mapping_list = [PathMapping(remote=r, local=l) for r, l in zip(remote_paths, local_paths)]
-        path_mappings= cls(mappings=mapping_list)
+        path_mappings = cls(mappings=mapping_list)
         return path_mappings
 
     def as_tuple_list(self) -> List[Tuple[str, str]]:
         """
         Returns the mappings in the format required by pydevd (list of tuples).
         """
-        tuple_list= [(m.remote, m.local) for m in self.mappings]
+        tuple_list = [(m.remote, m.local) for m in self.mappings]
         return tuple_list
 
+
 class RemoteDebugSetup:
     """
     Handles initialization and connection for pydevd remote debugging.
@@ -104,7 +110,7 @@ class RemoteDebugSetup:
             args (Namespace): Parsed CLI arguments containing debug flags.
         """
         self.args = args
-        self.path_mappings=None
+        self.path_mappings = None
 
     def get_path_mappings(self):
         """
@@ -120,7 +126,6 @@ class RemoteDebugSetup:
         if remote_path and local_path:
             self.path_mappings = PathMappings.from_args(remote_path, local_path)
 
-
     def log(self, msg: str):
         """
         Print debug message to stderr if debug mode is enabled.
@@ -180,7 +185,7 @@ class RemoteDebugSetup:
             # Check if this is the bad call (single tuple with comma-separated strings)
             if len(paths) == 1 and isinstance(paths[0], tuple):
                 remote, local = paths[0]
-                if ',' in remote and ',' in local:
+                if "," in remote and "," in local:
                     self.log("IGNORING bad setup_client_server_paths call with comma-separated strings")
                     self.log(f"remote='{remote}'")
                     self.log(f"local='{local}'")
@@ -197,7 +202,7 @@ class RemoteDebugSetup:
         pydevd_file_utils.setup_client_server_paths = fixed_setup
 
         # https://github.com/fabioz/PyDev.Debugger/blob/main/pydevd_file_utils.py
-        tuple_list=self.path_mappings.as_tuple_list()
+        tuple_list = self.path_mappings.as_tuple_list()
         pydevd_file_utils.setup_client_server_paths(tuple_list)
 
     def print_debug_info(self):
@@ -214,4 +219,4 @@ class RemoteDebugSetup:
         if self.path_mappings:
             for m in self.path_mappings.mappings:
                 marker = "✅" if os.path.exists(m.local) else "❌"
-                self.log(f"PATH MAP: Remote (IDE)='{m.remote}' <-> Local='{m.local}' {marker}")
+                self.log(f"PATH MAP: Remote (IDE)='{m.remote}' <-> Local='{m.local}' {marker}")

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/basemkit/shell.py

@@ -11,7 +11,7 @@ import sys
 import threading
 from argparse import Namespace
 from pathlib import Path
-from typing import Dict, List
+from typing import Callable, Dict, List, Optional
 
 
 class ShellResult:
@@ -38,13 +38,22 @@ class ShellResult:
 class StreamTee:
     """
     Tees a single input stream to both a mirror and a capture buffer.
+    An optional per-line callback is invoked for each line after buffering.
     """
 
-    def __init__(self, source, mirror, buffer, tee=True):
+    def __init__(
+        self,
+        source,
+        mirror,
+        buffer,
+        tee=True,
+        callback: Optional[Callable[[str], None]] = None,
+    ):
         self.source = source
         self.mirror = mirror
         self.buffer = buffer
         self.tee = tee
+        self.callback = callback
         self.thread = threading.Thread(target=self._run, daemon=True)
 
     def _run(self):
@@ -53,6 +62,8 @@ class StreamTee:
                 self.mirror.write(line)
                 self.mirror.flush()
             self.buffer.write(line)
+            if self.callback:
+                self.callback(line)
         self.source.close()
 
     def start(self):
@@ -92,13 +103,32 @@ class StdTee:
     """
     Manages teeing for both stdout and stderr using StreamTee instances.
     Captures output in instance variables.
+    Optional per-line callbacks are invoked for each stdout/stderr line.
     """
 
-    def __init__(self, process, tee=True):
+    def __init__(
+        self,
+        process,
+        tee=True,
+        stdout_callback: Optional[Callable[[str], None]] = None,
+        stderr_callback: Optional[Callable[[str], None]] = None,
+    ):
         self.stdout_buffer = io.StringIO()
         self.stderr_buffer = io.StringIO()
-        self.out_tee = StreamTee(process.stdout, sys.stdout, self.stdout_buffer, tee)
-        self.err_tee = StreamTee(process.stderr, sys.stderr, self.stderr_buffer, tee)
+        self.out_tee = StreamTee(
+            process.stdout,
+            sys.stdout,
+            self.stdout_buffer,
+            tee,
+            callback=stdout_callback,
+        )
+        self.err_tee = StreamTee(
+            process.stderr,
+            sys.stderr,
+            self.stderr_buffer,
+            tee,
+            callback=stderr_callback,
+        )
 
     def start(self):
         self.out_tee.start()
@@ -109,12 +139,23 @@ class StdTee:
         self.err_tee.join()
 
     @classmethod
-    def run(cls, process, tee=True):
+    def run(
+        cls,
+        process,
+        tee=True,
+        stdout_callback: Optional[Callable[[str], None]] = None,
+        stderr_callback: Optional[Callable[[str], None]] = None,
+    ):
         """
         Run teeing and capture for the given process.
         Returns a StdTee instance with stdout/stderr captured.
         """
-        std_tee = cls(process, tee=tee)
+        std_tee = cls(
+            process,
+            tee=tee,
+            stdout_callback=stdout_callback,
+            stderr_callback=stderr_callback,
+        )
         std_tee.start()
         std_tee.join()
         return std_tee
@@ -181,15 +222,30 @@ class Shell:
         shell = cls(profile=profile)
         return shell
 
-    def run(self, cmd: str, text: bool = True, debug: bool = False, tee: bool = False) -> subprocess.CompletedProcess:
+    def run(
+        self,
+        cmd: str,
+        text: bool = True,
+        encoding: str = "utf-8",
+        errors: str = "replace",
+        debug: bool = False,
+        tee: bool = False,
+        stdout_callback: Optional[Callable[[str], None]] = None,
+        stderr_callback: Optional[Callable[[str], None]] = None,
+    ) -> subprocess.CompletedProcess:
         """
         Run command with profile, always capturing output and optionally teeing it.
 
         Args:
             cmd: Command to run
             text: Text mode for subprocess I/O
+            encoding: Character encoding for subprocess I/O (default: utf-8)
+            errors: Error handling for decoding (default: replace — bad bytes become U+FFFD
+                instead of raising UnicodeDecodeError; pass 'strict' to restore old behaviour)
             debug: Print the command to be run
             tee: If True, also print output live while capturing
+            stdout_callback: Optional callable invoked with each stdout line as it is produced
+            stderr_callback: Optional callable invoked with each stderr line as it is produced
 
         Returns:
             subprocess.CompletedProcess
@@ -204,9 +260,16 @@ class Shell:
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
             text=text,
+            encoding=encoding,
+            errors=errors,
         )
 
-        std_tee = StdTee.run(popen_process, tee=tee)
+        std_tee = StdTee.run(
+            popen_process,
+            tee=tee,
+            stdout_callback=stdout_callback,
+            stderr_callback=stderr_callback,
+        )
         returncode = popen_process.wait()
 
         process = subprocess.CompletedProcess(

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/basemkit/yamlable.py

@@ -40,7 +40,7 @@ from collections.abc import Iterable, Mapping
 from dataclasses import asdict, dataclass, is_dataclass
 from datetime import date, datetime
 from pathlib import Path
-from typing import Any, Generic, TextIO, Type, TypeVar, Union
+from typing import Any, Generic, Optional, TextIO, Tuple, Type, TypeVar, Union
 
 import yaml
 from dacite import from_dict
@@ -107,6 +107,40 @@ class YamlAble(Generic[T]):
             self._yaml_dumper.add_representer(type(None), self.represent_none)
             self._yaml_dumper.add_representer(str, self.represent_literal)
 
+    @staticmethod
+    def _split_yaml_header(text: str) -> Tuple[str, str]:
+        """
+        Split raw YAML text into a leading comment block and the data body.
+
+        Scans lines from the top; a line belongs to the header if it starts
+        with '#' or is blank (blank lines between comment lines are kept).
+        Scanning stops at the first line that is neither a comment nor blank.
+        Trailing blank lines are trimmed from the header and prepended to the
+        body so that the body remains valid standalone YAML.
+
+        Args:
+            text: Raw YAML file content.
+
+        Returns:
+            A tuple (header, body) where header is the extracted comment block
+            (including a trailing newline) and body is the remainder.
+            If no leading comments are found, header is '' and body is text.
+        """
+        lines = text.splitlines(keepends=True)
+        split_idx = 0
+        for i, line in enumerate(lines):
+            stripped = line.strip()
+            if stripped.startswith("#") or stripped == "":
+                split_idx = i + 1
+            else:
+                break
+        # Trim trailing blank lines from the header, move them to the body
+        while split_idx > 0 and lines[split_idx - 1].strip() == "":
+            split_idx -= 1
+        header = "".join(lines[:split_idx])
+        body = "".join(lines[split_idx:])
+        return header, body
+
     def represent_none(self, _, __) -> yaml.Node:
         """
         Custom representer for ignoring None values in the YAML output.
@@ -169,33 +203,42 @@ class YamlAble(Generic[T]):
         return instance
 
     @classmethod
-    def load_from_yaml_stream(cls: Type[T], stream: TextIO) -> T:
+    def load_from_yaml_stream(cls: Type[T], stream: TextIO, with_header_comment: bool = False) -> T:
         """
         Loads a dataclass instance from a YAML stream.
 
         Args:
             stream (TextIO): The input stream containing YAML data.
+            with_header_comment: If True, extract any leading comment block from the
+                raw text and store it on the instance as ``_yaml_header`` so it can
+                be re-emitted by :meth:`save_to_yaml_stream`.
 
         Returns:
             T: An instance of the dataclass.
         """
         yaml_str: str = stream.read()
+        if with_header_comment:
+            header, yaml_str = cls._split_yaml_header(yaml_str)
         instance: T = cls.from_yaml(yaml_str)
+        if with_header_comment:
+            instance._yaml_header = header if header else None
         return instance
 
     @classmethod
-    def load_from_yaml_file(cls: Type[T], filename: str) -> T:
+    def load_from_yaml_file(cls: Type[T], filename: str, with_header_comment: bool = False) -> T:
         """
         Loads a dataclass instance from a YAML file.
 
         Args:
             filename (str): The path to the YAML file.
+            with_header_comment: If True, preserve any leading comment block found in
+                the file; see :meth:`load_from_yaml_stream` for details.
 
         Returns:
             T: An instance of the dataclass.
         """
         with open(filename, "r") as file:
-            return cls.load_from_yaml_stream(file)
+            return cls.load_from_yaml_stream(file, with_header_comment=with_header_comment)
 
     @classmethod
     def load_from_yaml_url(cls: Type[T], url: str) -> T:
@@ -212,26 +255,35 @@ class YamlAble(Generic[T]):
         instance: T = cls.from_yaml(yaml_str)
         return instance
 
-    def save_to_yaml_stream(self, file: TextIO):
+    def save_to_yaml_stream(self, file: TextIO, with_header_comment: bool = False):
         """
         Saves the current dataclass instance to the given YAML stream.
 
         Args:
             file (TextIO): The stream to which YAML content will be saved.
+            with_header_comment: If True and ``self._yaml_header`` is set (populated
+                by a previous :meth:`load_from_yaml_stream` call with the same flag),
+                the comment block is written before the YAML body.  If no header is
+                stored the flag is silently a no-op.
         """
         yaml_content: str = self.to_yaml()
+        header: Optional[str] = getattr(self, "_yaml_header", None)
+        if with_header_comment and header:
+            file.write(header)
+            file.write("\n")
         file.write(yaml_content)
 
-    def save_to_yaml_file(self, filename: str):
+    def save_to_yaml_file(self, filename: str, with_header_comment: bool = False):
         """
         Saves the current dataclass instance to a YAML file.
 
         Args:
             filename (str): The path where the YAML file will be saved.
+            with_header_comment: If True, re-emit any leading comment block that was
+                captured during loading; see :meth:`save_to_yaml_stream` for details.
         """
-
         with open(filename, "w", encoding="utf-8") as file:
-            self.save_to_yaml_stream(file)
+            self.save_to_yaml_stream(file, with_header_comment=with_header_comment)
 
     @classmethod
     def load_from_json_file(cls: Type[T], filename: Union[str, Path]) -> T:

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/tests/test_remotedebug.py

@@ -7,8 +7,7 @@ Created on 2025-112-29
 from argparse import Namespace
 
 from basemkit.basetest import Basetest
-
-from basemkit.remotedebug import RemoteDebugSetup, PathMappings, PathMapping
+from basemkit.remotedebug import PathMapping, PathMappings, RemoteDebugSetup
 
 
 class TestRemoteDebugSetup(Basetest):
@@ -28,10 +27,10 @@ class TestRemoteDebugSetup(Basetest):
             debugPort=5678,
             debugRemotePath="/remote/app",
             debugLocalPath="/local/app",
-            debug=True # Enable internal logging
+            debug=True,  # Enable internal logging
         )
 
-        debug_setup=RemoteDebugSetup(args=args)
+        debug_setup = RemoteDebugSetup(args=args)
         debug_setup.get_path_mappings()
         debug_setup.print_debug_info()
 
@@ -54,24 +53,20 @@ class TestRemoteDebugSetup(Basetest):
         self.assertEqual("/app/lib", pm.mappings[1].remote)
         self.assertEqual("C:\\Users\\Lib", pm.mappings[1].local)
 
-
     def test_path_mappings_mismatch(self):
         """
         Test that unequal list lengths raise an error.
         """
         remote_str = "/app/src"
-        local_str = "C:\\src,C:\\lib" # Two locals, one remote
+        local_str = "C:\\src,C:\\lib"  # Two locals, one remote
 
         with self.assertRaises(ValueError):
             PathMappings.from_args(remote_str, local_str)
 
-
     def test_tuple_conversion(self):
         """
         Test conversion to list of tuples for pydevd.
         """
-        pm = PathMappings(mappings=[
-            PathMapping(remote="/r", local="/l")
-        ])
+        pm = PathMappings(mappings=[PathMapping(remote="/r", local="/l")])
         expected = [("/r", "/l")]
         self.assertEqual(expected, pm.as_tuple_list())

pybasemkit-0.2.2/tests/test_shell.py

@@ -0,0 +1,78 @@
+"""
+Created on 2025-05-14
+
+@author: wf
+"""
+
+from typing import List
+
+from basemkit.basetest import Basetest
+from basemkit.shell import Shell
+
+
+class TestShell(Basetest):
+    """
+    test shell commands
+    """
+
+    def setUp(self, debug=False, profile=True):
+        Basetest.setUp(self, debug=debug, profile=profile)
+
+    def testShell(self):
+        """
+        test the shell handling
+        """
+        shell = Shell()
+        for cmd, expected in [
+            # ("pwd", "test"),
+            # ("which git", "git"),
+            ("echo $PATH", "bin"),
+            # ("docker ps", "CONTAINER ID"),
+            # ("which soffice", "soffice"),
+        ]:
+            p = shell.run(cmd, tee=self.debug)
+            if self.debug:
+                print(p)
+                print(p.stdout)
+            self.assertEqual(0, p.returncode)
+            self.assertIn(expected, p.stdout)
+
+    def test_encoding_errors(self):
+        """
+        Test that non-UTF-8 bytes in subprocess output do not raise UnicodeDecodeError.
+        With errors='replace' (the new default) bad bytes are replaced by U+FFFD.
+        """
+        shell = Shell()
+        # printf emits a raw 0xdf byte which is invalid UTF-8 (it is valid Latin-1: ß)
+        p = shell.run("printf '\\xdf'")
+        if self.debug:
+            print(repr(p.stdout))
+        # Must not raise; returncode must be 0
+        self.assertEqual(0, p.returncode)
+        # The replacement character U+FFFD must appear in the decoded output
+        self.assertIn("\ufffd", p.stdout)
+
+    def test_callbacks(self):
+        """
+        Test that stdout_callback and stderr_callback are invoked for each line.
+        """
+        shell = Shell()
+        stdout_lines: List[str] = []
+        stderr_lines: List[str] = []
+
+        p = shell.run(
+            "printf 'line1\\nline2\\nline3\\n' && printf 'err1\\nerr2\\n' >&2",
+            stdout_callback=lambda line: stdout_lines.append(line),
+            stderr_callback=lambda line: stderr_lines.append(line),
+        )
+        if self.debug:
+            print("stdout_lines:", stdout_lines)
+            print("stderr_lines:", stderr_lines)
+
+        self.assertEqual(0, p.returncode)
+        # Callbacks receive lines including the trailing newline from readline()
+        self.assertEqual(["line1\n", "line2\n", "line3\n"], stdout_lines)
+        self.assertEqual(["err1\n", "err2\n"], stderr_lines)
+        # Captured buffers must also be consistent
+        self.assertEqual("line1\nline2\nline3\n", p.stdout)
+        self.assertEqual("err1\nerr2\n", p.stderr)

{pybasemkit-0.2.0 → pybasemkit-0.2.2}/tests/test_yamlable.py

@@ -131,6 +131,72 @@ class TestYamlAble(Basetest):
             # Clean up the temp file
             os.remove(temp_file.name)
 
+    def test_header_comment_preserved(self) -> None:
+        """
+        Test that a leading YAML comment block survives a full file round-trip
+        when with_header_comment=True is used on both load and save.
+        """
+        yaml_with_header = (
+            "# This is a project config file\n"
+            "# Generated by the system — do not edit manually\n"
+            "\n"
+            "name: Example\n"
+            "id: 123\n"
+            "flag: true\n"
+        )
+        with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, encoding="utf-8") as f:
+            src_path = f.name
+            f.write(yaml_with_header)
+
+        try:
+            # Load with header preservation
+            loaded = MockDataClass.load_from_yaml_file(src_path, with_header_comment=True)
+            if self.debug:
+                print(f"_yaml_header: {repr(loaded._yaml_header)}")
+            self.assertIsNotNone(loaded._yaml_header, "_yaml_header should be set")
+            self.assertIn("project config file", loaded._yaml_header)
+            self.assertIn("do not edit manually", loaded._yaml_header)
+            self.assertEqual(loaded.name, "Example")
+            self.assertEqual(loaded.id, 123)
+
+            # Modify a field and save back with header
+            loaded.id = 999
+            with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, encoding="utf-8") as f:
+                dst_path = f.name
+            loaded.save_to_yaml_file(dst_path, with_header_comment=True)
+
+            with open(dst_path, "r", encoding="utf-8") as f:
+                result = f.read()
+            if self.debug:
+                print(result)
+
+            # Header must be at the top
+            self.assertTrue(
+                result.startswith("# This is a project config file"),
+                "Header must be first",
+            )
+            self.assertIn("do not edit manually", result)
+            # Modified data must be present
+            self.assertIn("id: 999", result)
+            self.assertIn("name: Example", result)
+
+            # Saving without flag must NOT include the header
+            with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False, encoding="utf-8") as f:
+                no_header_path = f.name
+            loaded.save_to_yaml_file(no_header_path, with_header_comment=False)
+            with open(no_header_path, "r", encoding="utf-8") as f:
+                no_header_result = f.read()
+            self.assertFalse(
+                no_header_result.startswith("#"),
+                "Header must not appear when with_header_comment=False",
+            )
+        finally:
+            for path in [src_path, dst_path, no_header_path]:
+                try:
+                    os.remove(path)
+                except Exception:
+                    pass
+
     def test_load_with_none_optional_field(self) -> None:
         """
         Test that loading YAML with a None value for an Optional[str] field works without error.

pybasemkit-0.2.2/yamable.md

@@ -0,0 +1,324 @@
+# yamable — YamlAble & lod_storable
+
+Module: `basemkit/yamlable.py`
+
+Provides YAML and JSON serialization/deserialization for Python dataclasses via
+the `YamlAble` base class and the `@lod_storable` decorator.
+
+---
+
+## Overview
+
+`YamlAble` is a generic mixin that adds YAML and JSON I/O to any `@dataclass`.
+It handles:
+
+- Serializing to YAML with block-scalar strings and optional omission of `None`
+  values and underscore-prefixed attributes
+- Deserializing from YAML strings, streams, files, and URLs
+- Serializing to / deserializing from JSON (delegating to `dataclasses-json`)
+- Recursive filtering of `None`, empty collections, and private attributes
+  before serialization
+
+The `@lod_storable` decorator is a one-shot convenience that applies
+`@dataclass`, `@dataclass_json`, **and** `YamlAble` inheritance to a plain
+class with a single annotation.
+
+---
+
+## Quick Start
+
+### Using `@lod_storable` (recommended)
+
+```python
+from typing import Optional
+from basemkit.yamlable import lod_storable
+
+@lod_storable
+class Person:
+    name: str
+    age: int
+    email: Optional[str] = None
+
+# Serialize
+p = Person(name="Alice", age=30)
+print(p.to_yaml())
+# name: Alice
+# age: 30
+
+# Deserialize
+p2 = Person.from_yaml("name: Bob\nage: 25\n")
+print(p2.name)  # Bob
+```
+
+### Using `YamlAble` directly
+
+When you need explicit control over the MRO or already have `@dataclass` and
+`@dataclass_json` applied:
+
+```python
+from dataclasses import dataclass
+from dataclasses_json import dataclass_json
+from basemkit.yamlable import YamlAble
+
+@dataclass_json
+@dataclass
+class Config(YamlAble):
+    host: str = "localhost"
+    port: int = 8080
+```
+
+---
+
+## `@lod_storable` Decorator
+
+```python
+def lod_storable(cls):
+```
+
+Transforms a plain class into a fully capable storable dataclass by:
+
+1. Applying `@dataclass` — adds `__init__`, `__repr__`, `__eq__`, etc.
+2. Applying `@dataclass_json` — adds `from_json` / `to_json` / `from_dict` /
+   `to_dict`
+3. Creating an inner `LoDStorable` class that inherits from both `YamlAble`
+   and the decorated class, then restoring `__name__`, `__doc__`, and
+   `__module__` so the class identity is transparent to serializers and
+   module lookups.
+
+The name *LoDStorable* stands for **List-of-Dicts Storable** — the pattern
+used throughout pyLoDStorage for tabular in-memory data.
+
+---
+
+## `YamlAble` Class
+
+```python
+class YamlAble(Generic[T]):
+```
+
+### YAML Serialization
+
+#### `to_yaml`
+
+```python
+def to_yaml(
+    self,
+    ignore_none: bool = True,
+    ignore_underscore: bool = True,
+    allow_unicode: bool = True,
+    sort_keys: bool = False,
+) -> str:
+```
+
+Converts the dataclass instance to a YAML string.
+
+| Parameter | Default | Effect |
+|---|---|---|
+| `ignore_none` | `True` | Omit keys whose value is `None` |
+| `ignore_underscore` | `True` | Omit keys whose name starts with `_` |
+| `allow_unicode` | `True` | Emit unicode characters unescaped |
+| `sort_keys` | `False` | Alphabetically sort mapping keys |
+
+Multi-line strings are automatically rendered in **block scalar style** (`|`),
+preserving newlines readably.
+
+```python
+obj.to_yaml()
+# description: |-
+#   First line
+#   Second line
+# name: Example
+```
+
+#### `save_to_yaml_stream` / `save_to_yaml_file`
+
+```python
+def save_to_yaml_stream(self, file: TextIO) -> None:
+def save_to_yaml_file(self, filename: str) -> None:
+```
+
+Write YAML output to an open stream or a file path (UTF-8).
+
+```python
+obj.save_to_yaml_file("/tmp/config.yaml")
+```
+
+---
+
+### YAML Deserialization
+
+All three class methods return a fully constructed instance of the calling
+class (`cls`).
+
+#### `from_yaml`
+
+```python
+@classmethod
+def from_yaml(cls: Type[T], yaml_str: str) -> T:
+```
+
+Parses a YAML string and reconstructs the dataclass via `dacite.from_dict`.
+Handles `null` values in YAML correctly for `Optional` fields.
+
+```python
+instance = MyClass.from_yaml(yaml_string)
+```
+
+#### `load_from_yaml_stream`
+
+```python
+@classmethod
+def load_from_yaml_stream(cls: Type[T], stream: TextIO) -> T:
+```
+
+Reads the entire stream and delegates to `from_yaml`.
+
+#### `load_from_yaml_file`
+
+```python
+@classmethod
+def load_from_yaml_file(cls: Type[T], filename: str) -> T:
+```
+
+Opens `filename` in text mode and delegates to `load_from_yaml_stream`.
+
+```python
+config = Config.load_from_yaml_file("config.yaml")
+```
+
+#### `load_from_yaml_url`
+
+```python
+@classmethod
+def load_from_yaml_url(cls: Type[T], url: str) -> T:
+```
+
+Fetches the URL with `urllib.request` and delegates to `from_yaml`.
+Raises `Exception` if the HTTP status is not 200.
+
+---
+
+### JSON Serialization / Deserialization
+
+JSON support is provided by `dataclasses-json` and is available on any class
+decorated with `@lod_storable` (or `@dataclass_json` directly). `YamlAble`
+adds file and URL convenience wrappers.
+
+#### `save_to_json_file`
+
+```python
+def save_to_json_file(self, filename: str, **kwargs: Any) -> None:
+```
+
+Serializes to JSON and writes to `filename` (UTF-8). Extra `**kwargs` are
+forwarded to `to_json()`.
+
+#### `load_from_json_file`
+
+```python
+@classmethod
+def load_from_json_file(cls: Type[T], filename: Union[str, Path]) -> T:
+```
+
+Reads a JSON file and reconstructs the instance via `from_json`.
+
+#### `load_from_json_url`
+
+```python
+@classmethod
+def load_from_json_url(cls: Type[T], url: str) -> T:
+```
+
+Fetches JSON from a URL and reconstructs the instance.
+
+---
+
+### Filtering Helper
+
+#### `remove_ignored_values`
+
+```python
+@classmethod
+def remove_ignored_values(
+    cls,
+    value: Any,
+    ignore_none: bool = True,
+    ignore_underscore: bool = False,
+    ignore_empty: bool = True,
+) -> Any:
+```
+
+Recursively walks a dict / list structure and removes entries that match the
+active ignore flags. Called internally by `to_yaml` but also usable standalone.
+
+| Flag | Default | Removes |
+|---|---|---|
+| `ignore_none` | `True` | Keys with `None` values |
+| `ignore_underscore` | `False` | Keys whose name starts with `_` |
+| `ignore_empty` | `True` | Empty dicts, lists, sets, tuples |
+
+Strings and bytes are treated as scalars, not iterables, and are never
+removed by `ignore_empty`.
+
+---
+
+### `from_dict2`
+
+```python
+@classmethod
+def from_dict2(cls: Type[T], data: dict) -> T:
+```
+
+Alternative deserializer using `dacite.from_dict` instead of
+`dataclasses-json`. Returns `None` if `data` is falsy. Useful when
+`dataclasses-json` is not available or when `dacite`'s strict type coercion
+is preferred.
+
+---
+
+## `DateConvert` Helper
+
+```python
+class DateConvert:
+    @classmethod
+    def iso_date_to_datetime(cls, iso_date: str) -> date:
+```
+
+Converts an ISO 8601 date string (`"YYYY-MM-DD"`) to a `datetime.date`
+object. Returns `None` if `iso_date` is falsy. Intended as a `dacite`
+type-hook for fields typed as `date`.
+
+---
+
+## Internal Representers
+
+These are set up automatically by `_yaml_setup()` on the first call to
+`to_yaml()` and should not need to be called directly.
+
+| Method | Purpose |
+|---|---|
+| `represent_none` | Renders `None` as an empty YAML scalar (`""`) rather than `null` |
+| `represent_literal` | Renders strings containing `\n` in block scalar style (`\|`) |
+
+---
+
+## Dependencies
+
+| Package | Role |
+|---|---|
+| `PyYAML` | YAML parsing and emission |
+| `dacite` | Strict dict-to-dataclass construction (`from_dict2`) |
+| `dataclasses-json` | JSON serialization (`from_json`, `to_json`, `from_dict`) |
+
+---
+
+## Notes
+
+- `YamlAble` requires the instance to be a dataclass (`is_dataclass(self)` is
+  asserted in `_yaml_setup`). A `ValueError` is raised otherwise.
+- The YAML dumper is cached on `self._yaml_dumper` after the first setup call;
+  custom representers are registered only once per instance.
+- URL loading uses the stdlib `urllib.request` — no `requests` dependency.
+- `lod_storable` preserves `__name__`, `__qualname__`, `__doc__`, and
+  `__module__` so `pickle`, `dacite`, and `dataclasses-json` can resolve the
+  class correctly.

pybasemkit-0.2.0/basemkit/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.2.0"

pybasemkit-0.2.0/tests/test_shell.py

@@ -1,36 +0,0 @@
-"""
-Created on 2025-05-14
-
-@author: wf
-"""
-
-from basemkit.basetest import Basetest
-from basemkit.shell import Shell
-
-
-class TestShell(Basetest):
-    """
-    test shell commands
-    """
-
-    def setUp(self, debug=True, profile=True):
-        Basetest.setUp(self, debug=debug, profile=profile)
-
-    def testShell(self):
-        """
-        test the shell handling
-        """
-        shell = Shell()
-        for cmd, expected in [
-            # ("pwd", "test"),
-            # ("which git", "git"),
-            ("echo $PATH", "bin"),
-            # ("docker ps", "CONTAINER ID"),
-            # ("which soffice", "soffice"),
-        ]:
-            p = shell.run(cmd, tee=self.debug)
-            if self.debug:
-                print(p)
-                print(p.stdout)
-            self.assertEqual(0, p.returncode)
-            self.assertIn(expected, p.stdout)