runem 0.0.27__tar.gz → 0.0.29__tar.gz

{runem-0.0.27 → runem-0.0.29}/HISTORY.md

@@ -4,6 +4,184 @@ Changelog
 
 (unreleased)
 ------------
+- Merge pull request #40 from lursight/fix/time_saved. [Frank Harrison]
+
+  Fix/time saved
+- Feat(prettier-bars): clarifies that total is user-space time. [Frank
+  Harrison]
+
+  ... not wall-clock or system-time
+- Feat(prettier-bars): distiguishes the wall-clock bars. [Frank
+  Harrison]
+
+  ... from the total/sum and sub-job bars, so that it's slightly easier to
+  see where the time is being really spent.
+- Fix(time-saved): clarifies which measurement is the wall-clock time
+  for the entire run. [Frank Harrison]
+- Fix(time-saved): add message about how long we _would_ have waited
+  without runem. [Frank Harrison]
+- Fix(time-saved): renames all variable associated with timing reports.
+  [Frank Harrison]
+
+  This just makes someting which can become intractable/confusing a lot
+  easier to follow.
+- Fix(time-saved): check that time-saved is reported correctly. [Frank
+  Harrison]
+
+  Here we add a test first and then fix the missing math to calculate the
+  time-saved by using runem. We broke this in the previous feature for
+  rendering the tree slightly more elegantly.
+- Feat(hide-single-leafs): only show the job when it has a single child.
+  [Frank Harrison]
+
+  We would get duplicated information for jobs which had single
+  run_command invocations. This only shows sub-tasks/jobs if there are
+  more than one sub-tasks meaning the output looks a lot nicer & clearer.
+- Chore(deps): adds setuptools as a explicit dep. [Frank Harrison]
+
+  ... otherwise we get the following error (more often in python 3.12),
+  perhaps due to setuptools being removed from distros?:
+
+  ```text
+  Traceback (most recent call last):
+    File "/var/www/mydir/virtualenvs/dev/bin/pip", line 5, in <module>
+      from pkg_resources import load_entry_point
+  ImportError: No module named pkg_resources
+  ```
+- Merge pull request #39 from lursight/feat/time_all_run_command_calls.
+  [Frank Harrison]
+
+  Feat/time all run command calls
+- Feat(pretty-tree): refactors out the phase-job report generator.
+  [Frank Harrison]
+
+  This is just to make pylint happy.
+- Feat(pretty-tree): makes the report tree neater. [Frank Harrison]
+- Feat(time-all-sub-tasks): re-raise errors for context in ci/cd. [Frank
+  Harrison]
+
+  In github ci/cd we were hitting the asserts but had no context of where
+  they're raised from or why. This should fix that if they still occur.
+- Feat(time-all-sub-tasks): adds a test to test the time-recording
+  functions. [Frank Harrison]
+- Feat(time-all-sub-tasks): adds all run_command times to report output.
+  [Frank Harrison]
+- Chore(type): uses a type-alias instead of manual type. [Frank
+  Harrison]
+- Merge pull request #37 from
+  lursight/dependabot/github_actions/actions/upload-artifact-4. [Frank
+  Harrison]
+
+  chore(deps): bump actions/upload-artifact from 3 to 4
+- Chore(deps): bump actions/upload-artifact from 3 to 4.
+  [dependabot[bot]]
+
+  Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
+  - [Release notes](https://github.com/actions/upload-artifact/releases)
+  - [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)
+
+  ---
+  updated-dependencies:
+  - dependency-name: actions/upload-artifact
+    dependency-type: direct:production
+    update-type: version-update:semver-major
+  ...
+- Merge pull request #36 from lursight/chore/more_python_versions_in_ci.
+  [Frank Harrison]
+
+  Chore/more python versions in ci
+- Chore(github-ci): workaround for python 3.12 setuptools issue. [Frank
+  Harrison]
+
+  This fies what looks like an issue with pytest hooks running ci/cd (and
+  on local machine) where we get:
+    ModuleNotFoundError: No module named 'pkg_resources'
+- Chore(github-ci): updates the python `--help` tests for 3.11 (and
+  later) [Frank Harrison]
+- Chore(github-ci): test python 3.9, 3.11 and 3.12 in ci. [Frank
+  Harrison]
+
+  We don't bother with 3.10 because we test 3.9 and boundaries for
+  features are on the 3.11 version.
+
+  We don't bother with earlier than 3.9, even though 3.8 is the earliest
+  officially supported version.
+
+
+0.0.28 (2024-03-01)
+-------------------
+- Release: version 0.0.28 🚀 [Frank Harrison]
+- Merge pull request #35 from lursight/fix/aliases_not_setting_options.
+  [Frank Harrison]
+
+  Fix/aliases not setting options
+- Switches the API so 'read-only' Options are the prominent type. [Frank
+  Harrison]
+
+  ... and Writable are the recessive type, making it easier for API users to ensure they're using the correct type
+- Uses OptionsReadOnly, stopping accidentally overwritting options by
+  individual jobs. [Frank Harrison]
+- Adds a read-only version of the Options dict. [Frank Harrison]
+- Ports Options type to an InformativeDict. [Frank Harrison]
+
+  This shows you what options are available in the options dict if you
+  look up a value that doesn't exist.
+- Updates logging for clarity. [Frank Harrison]
+- Merge pull request #34 from lursight/chore/github-actions. [Frank
+  Harrison]
+
+  Chore/GitHub actions
+- Chore(github-ci): knocks out windows ci/cd for now. [Frank Harrison]
+- Chore(github-ci): sets the windows ci to use utf-8. [Frank Harrison]
+- Chore(github-ci): HACK ignore coverage in tested code. [Frank
+  Harrison]
+
+  It's not cler why the ci/cd thinks this isn't being hit, it is locally
+  and I can't figure out why it wouldn't be on the ci/cd, perhaps it's
+  down to the multithreaded pytest runs, but that would be
+  non-deterministic. This will probably need looking into at some point.
+- Chore(ci-coverage): updates the _get_jobs_matching test case. [Frank
+  Harrison]
+
+  Making it more explicit and easier to follow
+- Chore(ci-coverage): fixes the help-test false-positive. [Frank
+  Harrison]
+
+  We were always writing the help"
+- Chore(ci-coverage): adds test for
+  test_run_command_basic_call_verbose_with_cwd. [Frank Harrison]
+- Chore(ci-coverage): ignores un-hit 'communicate' mock function. [Frank
+  Harrison]
+- Chore(github-ci): upload reports on failures. [Frank Harrison]
+- Chore(github-ci): revert back to using the makefile for the redundant
+  checks. [Frank Harrison]
+- Chore(github-ci): installs yarn deps for ci job. [Frank Harrison]
+- Chore(github-ci): makes the log output less verbose by using --no-
+  spinner. [Frank Harrison]
+- Chore(github-ci): moves the redundancy checks first to own job. [Frank
+  Harrison]
+- Chore(github-ci): runs two basic checks for redunancy during ci.
+  [Frank Harrison]
+
+  This should help catch errors where runem has been broken and returns a
+  false-positive when run against itself.
+- Chore(github-ci): runs runem against itself. [Frank Harrison]
+
+  This could have drawbacks later so we will think about adding some
+  redundancy to the ci checks.
+- Merge pull request #33 from lursight/fix/mutiline_stdout. [Frank
+  Harrison]
+
+  fix(stdout-parsing): ensures that trailing newlines are handled
+- Fix(stdout-parsing): ensures that trailing newlines are handled.
+  [Frank Harrison]
+
+  There will be a slight performance cost to this, hopefully not too much.
+
+
+0.0.27 (2024-02-26)
+-------------------
+- Release: version 0.0.27 🚀 [Frank Harrison]
 - Merge pull request #32 from lursight/feat/min-version_check. [Frank
   Harrison]
 

{runem-0.0.27 → runem-0.0.29}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: runem
-Version: 0.0.27
+Version: 0.0.29
 Summary: Awesome runem created by lursight
 Home-page: https://github.com/lursight/runem/
 Author: lursight
@@ -26,6 +26,7 @@ Requires-Dist: pytest-cov==4.1.0; extra == "test"
 Requires-Dist: pytest-profiling==1.7.0; extra == "test"
 Requires-Dist: pytest-xdist==3.3.1; extra == "test"
 Requires-Dist: pytest==7.4.3; extra == "test"
+Requires-Dist: setuptools; extra == "test"
 Requires-Dist: termplotlib==0.3.9; extra == "test"
 Requires-Dist: types-PyYAML==6.0.12.12; extra == "test"
 Requires-Dist: requests-mock==1.10.0; extra == "test"
@@ -400,12 +401,12 @@ runem: reports:
 runem: runem: 8.820488s
 runem: ├runem.pre-build: 0.019031s
 runem: ├runem.run-phases: 8.801317s
-runem: ├pre-run (total): 0.00498s
+runem: ├pre-run (user-time): 0.00498s
 runem: │├pre-run.install python requirements: 2.6e-05s
 runem: │├pre-run.ls -alh runem: 0.004954s
-runem: ├edit (total): 0.557559s
+runem: ├edit (user-time): 0.557559s
 runem: │├edit.reformat py: 0.557559s
-runem: ├analysis (total): 21.526145s
+runem: ├analysis (user-time): 21.526145s
 runem: │├analysis.pylint py: 7.457029s
 runem: │├analysis.flake8 py: 0.693754s
 runem: │├analysis.mypy py: 1.071956s
@@ -430,12 +431,12 @@ runem: reports:
 runem                                  [14.174612]  ███████████████▋
 ├runem.pre-build                       [ 0.025858]
 ├runem.run-phases                      [14.148587]  ███████████████▋
-├pre-run (total)                       [ 0.005825]
+├pre-run (user-time)                   [ 0.005825]
 │├pre-run.install python requirements  [ 0.000028]
 │├pre-run.ls -alh runem                [ 0.005797]
-├edit (total)                          [ 0.579153]  ▋
+├edit (user-time)                      [ 0.579153]  ▋
 │├edit.reformat py                     [ 0.579153]  ▋
-├analysis (total)                      [36.231034]  ████████████████████████████████████████
+├analysis (user-time)                  [36.231034]  ████████████████████████████████████████
 │├analysis.pylint py                   [12.738303]  ██████████████▏
 │├analysis.flake8 py                   [ 0.798575]  ▉
 │├analysis.mypy py                     [ 0.335984]  ▍

{runem-0.0.27 → runem-0.0.29}/README.md

@@ -367,12 +367,12 @@ runem: reports:
 runem: runem: 8.820488s
 runem: ├runem.pre-build: 0.019031s
 runem: ├runem.run-phases: 8.801317s
-runem: ├pre-run (total): 0.00498s
+runem: ├pre-run (user-time): 0.00498s
 runem: │├pre-run.install python requirements: 2.6e-05s
 runem: │├pre-run.ls -alh runem: 0.004954s
-runem: ├edit (total): 0.557559s
+runem: ├edit (user-time): 0.557559s
 runem: │├edit.reformat py: 0.557559s
-runem: ├analysis (total): 21.526145s
+runem: ├analysis (user-time): 21.526145s
 runem: │├analysis.pylint py: 7.457029s
 runem: │├analysis.flake8 py: 0.693754s
 runem: │├analysis.mypy py: 1.071956s
@@ -397,12 +397,12 @@ runem: reports:
 runem                                  [14.174612]  ███████████████▋
 ├runem.pre-build                       [ 0.025858]
 ├runem.run-phases                      [14.148587]  ███████████████▋
-├pre-run (total)                       [ 0.005825]
+├pre-run (user-time)                   [ 0.005825]
 │├pre-run.install python requirements  [ 0.000028]
 │├pre-run.ls -alh runem                [ 0.005797]
-├edit (total)                          [ 0.579153]  ▋
+├edit (user-time)                      [ 0.579153]  ▋
 │├edit.reformat py                     [ 0.579153]  ▋
-├analysis (total)                      [36.231034]  ████████████████████████████████████████
+├analysis (user-time)                  [36.231034]  ████████████████████████████████████████
 │├analysis.pylint py                   [12.738303]  ██████████████▏
 │├analysis.flake8 py                   [ 0.798575]  ▉
 │├analysis.mypy py                     [ 0.335984]  ▍

runem-0.0.29/runem/VERSION

@@ -0,0 +1 @@
+0.0.29

{runem-0.0.27 → runem-0.0.29}/runem/cli/initialise_options.py

@@ -1,21 +1,22 @@
 import argparse
 
 from runem.config_metadata import ConfigMetadata
-from runem.types import Options
+from runem.informative_dict import InformativeDict
+from runem.types import OptionsWritable
 
 
 def initialise_options(
     config_metadata: ConfigMetadata,
     args: argparse.Namespace,
-) -> Options:
+) -> OptionsWritable:
     """Initialises and returns the set of options to use for this run.
 
     Returns the options dictionary
     """
 
-    options: Options = {
-        option["name"]: option["default"] for option in config_metadata.options_config
-    }
+    options: OptionsWritable = InformativeDict(
+        {option["name"]: option["default"] for option in config_metadata.options_config}
+    )
     if config_metadata.options_config and args.overrides_on:  # pragma: no branch
         for option_name in args.overrides_on:  # pragma: no branch
             options[option_name] = True

{runem-0.0.27 → runem-0.0.29}/runem/command_line.py

@@ -5,9 +5,10 @@ import sys
 import typing
 
 from runem.config_metadata import ConfigMetadata
+from runem.informative_dict import InformativeDict
 from runem.log import log
 from runem.runem_version import get_runem_version
-from runem.types import JobNames, OptionConfig, Options
+from runem.types import JobNames, OptionConfig, OptionsWritable
 from runem.utils import printable_set
 
 
@@ -175,7 +176,7 @@ def parse_args(
         # cleanly exit
         sys.exit(0)
 
-    options: Options = initialise_options(config_metadata, args)
+    options: OptionsWritable = initialise_options(config_metadata, args)
 
     if not _validate_filters(config_metadata, args):
         sys.exit(1)
@@ -245,15 +246,15 @@ def _validate_filters(
 def initialise_options(
     config_metadata: ConfigMetadata,
     args: argparse.Namespace,
-) -> Options:
+) -> OptionsWritable:
     """Initialises and returns the set of options to use for this run.
 
     Returns the options dictionary
     """
 
-    options: Options = {
-        option["name"]: option["default"] for option in config_metadata.options_config
-    }
+    options: OptionsWritable = InformativeDict(
+        {option["name"]: option["default"] for option in config_metadata.options_config}
+    )
     if config_metadata.options_config and args.overrides_on:  # pragma: no branch
         for option_name in args.overrides_on:  # pragma: no branch
             options[option_name] = True

{runem-0.0.27 → runem-0.0.29}/runem/config_metadata.py

@@ -1,12 +1,13 @@
 import argparse
 import pathlib
 
+from runem.informative_dict import InformativeDict
 from runem.types import (
     JobNames,
     JobPhases,
     JobTags,
     OptionConfigs,
-    Options,
+    OptionsWritable,
     OrderedPhases,
     PhaseGroupedJobs,
     TagFileFilters,
@@ -24,7 +25,7 @@ class ConfigMetadata:
     all_job_phases: JobPhases  # the set of job-phases (should be subset of 'phases')
     all_job_tags: JobTags  # the set of job-tags (used for filtering)
 
-    options: Options  # the final configured options to pass to jobs
+    options: OptionsWritable  # the final configured options to pass to jobs
 
     args: argparse.Namespace  # the raw cli args, probably missing information
     jobs_to_run: JobNames  # superset of job-name candidates to run, from cli+config
@@ -52,7 +53,7 @@ class ConfigMetadata:
         self.all_job_phases = all_job_phases
         self.all_job_tags = all_job_tags
 
-        self.options = {}  # will be defined after cli argument parsing
+        self.options = InformativeDict()  # shows useful errors on bad-option lookups
 
         self.args = (
             argparse.Namespace()
@@ -69,7 +70,7 @@ class ConfigMetadata:
         phases_to_run: JobPhases,
         tags_to_run: JobTags,
         tags_to_avoid: JobTags,
-        options: Options,
+        options: OptionsWritable,
     ) -> None:
         self.options = options
         self.args = args

runem-0.0.29/runem/informative_dict.py

@@ -0,0 +1,42 @@
+import typing
+
+# Define type variables for key and value to be used in the custom dictionary
+K = typing.TypeVar("K")
+V = typing.TypeVar("V")
+
+
+class InformativeDict(typing.Dict[K, V], typing.Generic[K, V]):
+    """A dictionary type that prints out the available keys."""
+
+    def __getitem__(self, key: K) -> V:
+        """Attempt to retrieve an item, raising a detailed exception if the key is not
+        found."""
+        try:
+            return super().__getitem__(key)
+        except KeyError:
+            available_keys: typing.Iterable[str] = (str(k) for k in self.keys())
+            raise KeyError(
+                f"Key '{key}' not found. Available keys: {', '.join(available_keys)}"
+            ) from None
+
+
+class ReadOnlyInformativeDict(InformativeDict[K, V], typing.Generic[K, V]):
+    """A read-only variant of the above."""
+
+    def __setitem__(self, key: K, value: V) -> None:
+        raise NotImplementedError("This dictionary is read-only")
+
+    def __delitem__(self, key: K) -> None:
+        raise NotImplementedError("This dictionary is read-only")
+
+    def pop(self, *args: typing.Any, **kwargs: typing.Any) -> V:
+        raise NotImplementedError("This dictionary is read-only")
+
+    def popitem(self) -> typing.Tuple[K, V]:
+        raise NotImplementedError("This dictionary is read-only")
+
+    def clear(self) -> None:
+        raise NotImplementedError("This dictionary is read-only")
+
+    def update(self, *args: typing.Any, **kwargs: typing.Any) -> None:
+        raise NotImplementedError("This dictionary is read-only")

{runem-0.0.27 → runem-0.0.29}/runem/job_execute.py

@@ -7,24 +7,34 @@ from datetime import timedelta
 from timeit import default_timer as timer
 
 from runem.config_metadata import ConfigMetadata
+from runem.informative_dict import ReadOnlyInformativeDict
 from runem.job import Job
 from runem.job_wrapper import get_job_wrapper
 from runem.log import log
-from runem.types import FilePathListLookup, JobConfig, JobFunction, JobReturn, JobTags
+from runem.types import (
+    FilePathListLookup,
+    JobConfig,
+    JobFunction,
+    JobReturn,
+    JobTags,
+    JobTiming,
+    TimingEntries,
+    TimingEntry,
+)
 
 
 def job_execute_inner(
     job_config: JobConfig,
     config_metadata: ConfigMetadata,
     file_lists: FilePathListLookup,
-) -> typing.Tuple[typing.Tuple[str, timedelta], JobReturn]:
+) -> typing.Tuple[JobTiming, JobReturn]:
     """Wrapper for running a job inside a sub-process.
 
     Returns the time information and any reports the job generated
     """
     label = Job.get_job_name(job_config)
     if config_metadata.args.verbose:
-        log(f"START: {label}")
+        log(f"START: '{label}'")
     root_path: pathlib.Path = config_metadata.cfg_filepath.parent
     function: JobFunction
     job_tags: typing.Optional[JobTags] = Job.get_job_tags(job_config)
@@ -37,7 +47,19 @@ def job_execute_inner(
     if not file_list:
         # no files to work on
         log(f"WARNING: skipping job '{label}', no files for job")
-        return (f"{label}: no files!", timedelta(0)), None
+        return {
+            "job": (f"{label}: no files!", timedelta(0)),
+            "commands": [],
+        }, None
+
+    sub_command_timings: TimingEntries = []
+
+    def _record_sub_job_time(label: str, timing: timedelta) -> None:
+        """Record timing information for sub-commands/tasks, atomically.
+
+        For example inside of run_command() calls
+        """
+        sub_command_timings.append((label, timing))
 
     if (
         "ctx" in job_config
@@ -53,29 +75,38 @@ def job_execute_inner(
     start = timer()
     func_signature = inspect.signature(function)
     if config_metadata.args.verbose:
-        log(f"job: running {Job.get_job_name(job_config)}")
+        log(f"job: running: '{Job.get_job_name(job_config)}'")
     reports: JobReturn
-    if "args" in func_signature.parameters:
-        reports = function(  # type: ignore  # FIXME: which function do we have?
-            config_metadata.args, config_metadata.options, file_list
-        )
-    else:
-        reports = function(
-            options=config_metadata.options,  # type: ignore
-            file_list=file_list,
-            procs=config_metadata.args.procs,
-            root_path=root_path,
-            verbose=config_metadata.args.verbose,
-            # unpack useful data points from the job_config
-            label=Job.get_job_name(job_config),
-            job=job_config,
-        )
+    try:
+        if "args" in func_signature.parameters:
+            reports = function(  # type: ignore  # FIXME: which function do we have?
+                config_metadata.args, config_metadata.options, file_list
+            )
+        else:
+            reports = function(
+                options=ReadOnlyInformativeDict(config_metadata.options),  # type: ignore
+                file_list=file_list,
+                procs=config_metadata.args.procs,
+                root_path=root_path,
+                verbose=config_metadata.args.verbose,
+                # unpack useful data points from the job_config
+                label=Job.get_job_name(job_config),
+                job=job_config,
+                record_sub_job_time=_record_sub_job_time,
+            )
+    except BaseException:  # pylint: disable=broad-exception-caught
+        # log that we hit an error on this job and re-raise
+        log(decorate=False)
+        log(f"job: ERROR: job '{Job.get_job_name(job_config)}' failed to complete!")
+        # re-raise
+        raise
+
     end = timer()
     time_taken: timedelta = timedelta(seconds=end - start)
     if config_metadata.args.verbose:
-        log(f"DONE: {label}: {time_taken}")
-    timing_data = (label, time_taken)
-    return (timing_data, reports)
+        log(f"job: DONE: '{label}': {time_taken}")
+    this_job_timing_data: TimingEntry = (label, time_taken)
+    return ({"job": this_job_timing_data, "commands": sub_command_timings}, reports)
 
 
 def job_execute(
@@ -83,7 +114,7 @@ def job_execute(
     running_jobs: typing.Dict[str, str],
     config_metadata: ConfigMetadata,
     file_lists: FilePathListLookup,
-) -> typing.Tuple[typing.Tuple[str, timedelta], JobReturn]:
+) -> typing.Tuple[JobTiming, JobReturn]:
     """Thin-wrapper around job_execute_inner needed for mocking in tests.
 
     Needed for faster tests.

{runem-0.0.27 → runem-0.0.29}/runem/job_filter.py

@@ -65,6 +65,10 @@ def _get_jobs_matching(
     filtered_jobs: PhaseGroupedJobs,
     verbose: bool,
 ) -> None:
+    """Via filtered_jobs, filters 'jobs' that match the given phase and and tags.
+
+    Warns if the job-name isn't found in list of valid job-names.
+    """
     phase_jobs: typing.List[JobConfig] = jobs[phase]
 
     job: JobConfig
@@ -74,7 +78,10 @@ def _get_jobs_matching(
 
         job_name: str = Job.get_job_name(job)
         if job_name not in job_names:
-            if verbose:
+            # test test_get_jobs_matching_when_job_not_in_valid_job_names should
+            # cover the follow in Ci but does not for some reason I don't have
+            # time to look in to. /FH
+            if verbose:  # pragma: FIXME: add code coverage
                 log(
                     (
                         f"not running job '{job_name}' because it isn't in the "
@@ -118,7 +125,10 @@ def filter_jobs(  # noqa: C901
     filtered_jobs: PhaseGroupedJobs = defaultdict(list)
     for phase in config_metadata.phases:
         if phase not in phases_to_run:
-            if verbose:
+            # test test_get_jobs_matching_when_job_not_in_valid_job_names should
+            # cover the follow in Ci but does not for some reason I don't have
+            # time to look in to. /FH
+            if verbose:  # pragma: FIXME: add code coverage
                 log(f"skipping phase '{phase}'")
             continue
         _get_jobs_matching(
@@ -131,7 +141,10 @@ def filter_jobs(  # noqa: C901
             verbose=verbose,
         )
         if len(filtered_jobs[phase]) == 0:
-            if verbose:
+            # test test_get_jobs_matching_when_job_not_in_valid_job_names should
+            # cover the follow in Ci but does not for some reason I don't have
+            # time to look in to. /FH
+            if verbose:  # pragma: FIXME: add code coverage
                 log(f"No jobs for phase '{phase}' tags {printable_set(tags_to_run)}")
             continue