ParUtils 1.1.9__py3-none-any.whl → 1.2.1__py3-none-any.whl

{ParUtils-1.1.9.dist-info → ParUtils-1.2.1.dist-info}/METADATA

@@ -1,110 +1,110 @@
-Metadata-Version: 2.1
-Name: ParUtils
-Version: 1.1.9
-Summary: This package contains a bunch of Python utils developed for Support, Test and Automation IT Engineers
-Home-page: https://github.com/paularnaud2/ParUtils
-Author: Paul ARNAUD
-Author-email: paularnaud2@gmail.com
-Project-URL: Bug Tracker, https://github.com/paularnaud2/ParUtils/issues
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.6
-Description-Content-Type: text/markdown
-License-File: LICENSE
-
-# ParUtils
- 
-This package is a light version of ParTools, requiring no third-party dependencies. It includes a reworked logging feature, as well as string, and file handling features.
-You can mainly use them for:
- 
-- Logging information in a file
-- Loading / saving text files
-- Listing files
-- Loading / saving csv files (lighter but more performant than built in csv module)
-- Creating hash strings, random string, comparing strings with wildcard char
-- Comparing files and lists
-- Listing and removing duplicates from a list
- 
-## QuickStart
-  
-    pip install parutils
- 
-You can start by testing the logger with the following code:
-
-    import parutils as u
-
-    u.Logger()  # initializes a log file
-    u.log('Hello World')  # logs something in the console and in the log file
-
-You should then see something like this in the console:
- 
-    Log file initialised (c:\Dev\ParUtils\log\20221213_072132.txt)
-    CWD: c:\Dev\ParUtils
-    Python interpreter path: C:\Python\python.exe
-    Python version: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
-    ParUtils version: 1.0.8
-
-    07:21:32 - Hello World
-
-## Example of useful functions
- 
-ParUtils provides generic functions meant to be reused by external packages. In this section, a few of these functions are listed. For an exhaustive list, you can check out  __parutils/\_\_init\_\_.py__.
-
-Manipulating files
-- `save_list`: saves a list into a text file
-- `load_txt`: loads a text file into a string or a list
-- `load_csv`: loads a csv file into a list of lists
-- `save_csv`: saves a list of lists into a csv file
-- `list_files`: lists files in a given directory (possibility to recurse subdirectories)
-
-Manipulating strings
-- `like`: behaves as the LIKE of Oracle SQL (you can match strings with wildcard character '\*'). It returns a re.match object giving you access to the matched wildcards strings.  
-Example: ``m = like('Hello World', 'He\*o w\*d')``, ``m.group(1)`` => 'll'
-- `like_list` / `like_dict`: apply the `like` function directly to lists or dictionaries (see doc).
-- `big_number`: converts a potentially big number into a readable string.  
-Example: ``big_number(10000000)`` => '10 000 000'.
-- `get_duration_string`: outputs a string representing the time elapsed since the input ``start_time``.  
-Example: ``get_duration_string(0, end_time=200)`` => '3 minutes and 20 seconds'.
-- `hash512`: creates a non randomised hash string from a string.
-- `gen_random_string`: generates a random string.
-
-Data quality features:
-- `diff_list`: compares two lists
-- `file_match`: compare two files
-- `find_dup_list`: finds duplicates in a list
-- `del_dup_list`: removes duplicates from a list
-
- 
-### Logging with parutils
- 
-The `log` function and the `Logger` class are directly available from the parutils package. So you can do:
-
-    import parutils as u
-
-    u.Logger()
-    u.log('Hello World')
-
-Note: if you want the `log` function to actually write in a log file, you have to create a ``Logger`` object before using it, otherwise it will just print out the log info in the console.
-
-The relevant parameters such a the log directory or the log format can be specified when initializing the ``Logger`` object. The default ``log_format`` is ``'%H:%M:%S -'``, and a default log line looks like:
- 
-    19:45:04 - This line has been generated by the parutils.log function
-
-
-Note that the default constants for the logging sub package are stored in __parutils.logging.const__. So for example, if you want to overwrite the default value for the logging directory, you can do:
-
-    import parutils as u
-
-    u.logging.const.DEFAULT_DIR = '<my_custom_dir>'
- 
-The `step_log` function allows you to log some information only when the input ``counter`` is a multiple of the input ``step``. Thus, `step_log` is to be used in loops to __track the progress of long processes__ such as reading or writing millions of lines in a file. The ``what`` input expects a description of what is being counted. It's default value is ``'lines written'``.  
-In order to correctly measure the elapsed time for the first log line, the ``step_log`` function has to be initialised by running ``init_sl_timer()``.  
-So for example, if you input ``step=500`` and don't input any ``what`` value, you should get something like this:
- 
-    19:45:04 - 500 lines written in 3 ms. 500 lines written in total.
-    19:45:04 - 500 lines written in 2 ms. 1 000 lines written in total.
-    19:45:04 - 500 lines written in 2 ms. 1 500 lines written in total.
- 
-Checkout the __test_logging.py__ file in __tests/logging__ for simple examples of use.
- 
+Metadata-Version: 2.2
+Name: ParUtils
+Version: 1.2.1
+Summary: This package contains a bunch of Python utils developed for Support, Test and Automation IT Engineers
+Home-page: https://github.com/paularnaud2/ParUtils
+Author: Paul ARNAUD
+Author-email: paularnaud2@gmail.com
+Project-URL: Bug Tracker, https://github.com/paularnaud2/ParUtils/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# ParUtils
+ 
+This package is a light version of ParTools, requiring no third-party dependencies. It includes a reworked logging feature, as well as string, and file handling features.
+You can mainly use them for:
+ 
+- Logging information in a file
+- Loading / saving text files
+- Listing files
+- Loading / saving csv files (lighter but more performant than built in csv module)
+- Creating hash strings, random string, comparing strings with wildcard char
+- Comparing files and lists
+- Listing and removing duplicates from a list
+ 
+## QuickStart
+  
+    pip install parutils
+ 
+You can start by testing the logger with the following code:
+
+    import parutils as u
+
+    u.Logger()  # initializes a log file
+    u.log('Hello World')  # logs something in the console and in the log file
+
+You should then see something like this in the console:
+ 
+    Log file initialised (c:\Dev\ParUtils\log\20221213_072132.txt)
+    CWD: c:\Dev\ParUtils
+    Python interpreter path: C:\Python\python.exe
+    Python version: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
+    ParUtils version: 1.0.8
+
+    07:21:32 - Hello World
+
+## Example of useful functions
+ 
+ParUtils provides generic functions meant to be reused by external packages. In this section, a few of these functions are listed. For an exhaustive list, you can check out  __parutils/\_\_init\_\_.py__.
+
+Manipulating files
+- `save_list`: saves a list into a text file
+- `load_txt`: loads a text file into a string or a list
+- `load_csv`: loads a csv file into a list of lists
+- `save_csv`: saves a list of lists into a csv file
+- `list_files`: lists files in a given directory (possibility to recurse subdirectories)
+
+Manipulating strings
+- `like`: behaves as the LIKE of Oracle SQL (you can match strings with wildcard character '\*'). It returns a re.match object giving you access to the matched wildcards strings.  
+Example: ``m = like('Hello World', 'He\*o w\*d')``, ``m.group(1)`` => 'll'
+- `like_list` / `like_dict`: apply the `like` function directly to lists or dictionaries (see doc).
+- `big_number`: converts a potentially big number into a readable string.  
+Example: ``big_number(10000000)`` => '10 000 000'.
+- `get_duration_string`: outputs a string representing the time elapsed since the input ``start_time``.  
+Example: ``get_duration_string(0, end_time=200)`` => '3 minutes and 20 seconds'.
+- `hash512`: creates a non randomised hash string from a string.
+- `gen_random_string`: generates a random string.
+
+Data quality features:
+- `diff_list`: compares two lists
+- `file_match`: compare two files
+- `find_dup_list`: finds duplicates in a list
+- `del_dup_list`: removes duplicates from a list
+
+ 
+### Logging with parutils
+ 
+The `log` function and the `Logger` class are directly available from the parutils package. So you can do:
+
+    import parutils as u
+
+    u.Logger()
+    u.log('Hello World')
+
+Note: if you want the `log` function to actually write in a log file, you have to create a ``Logger`` object before using it, otherwise it will just print out the log info in the console.
+
+The relevant parameters such a the log directory or the log format can be specified when initializing the ``Logger`` object. The default ``log_format`` is ``'%H:%M:%S -'``, and a default log line looks like:
+ 
+    19:45:04 - This line has been generated by the parutils.log function
+
+
+Note that the default constants for the logging sub package are stored in __parutils.logging.const__. So for example, if you want to overwrite the default value for the logging directory, you can do:
+
+    import parutils as u
+
+    u.logging.const.DEFAULT_DIR = '<my_custom_dir>'
+ 
+The `step_log` function allows you to log some information only when the input ``counter`` is a multiple of the input ``step``. Thus, `step_log` is to be used in loops to __track the progress of long processes__ such as reading or writing millions of lines in a file. The ``what`` input expects a description of what is being counted. It's default value is ``'lines written'``.  
+In order to correctly measure the elapsed time for the first log line, the ``step_log`` function has to be initialised by running ``init_sl_timer()``.  
+So for example, if you input ``step=500`` and don't input any ``what`` value, you should get something like this:
+ 
+    19:45:04 - 500 lines written in 3 ms. 500 lines written in total.
+    19:45:04 - 500 lines written in 2 ms. 1 000 lines written in total.
+    19:45:04 - 500 lines written in 2 ms. 1 500 lines written in total.
+ 
+Checkout the __test_logging.py__ file in __tests/logging__ for simple examples of use.
+ 

{ParUtils-1.1.9.dist-info → ParUtils-1.2.1.dist-info}/RECORD

@@ -1,5 +1,5 @@
 parutils/__init__.py,sha256=EaFtUmU0kYgnjvAFwPmoKt7I5zn9bxY02R7TUtcE6M4,1434
-parutils/changelog.py,sha256=cDppejgTUKwpHW9xL5NJX-J91hJnyZKQScbY9vVeZlU,1347
+parutils/changelog.py,sha256=3uz3n01a5Vt_JhNq6OfcTKPICGesQBhPcUaD5FeWSLk,1547
 parutils/csvl.py,sha256=UBqw0lyhTNqFLSNIE-JM7TAQYCRgHftYQmHi21ru8-A,2874
 parutils/dq.py,sha256=kdvzOo5FLCumXagY05kGSseYlbR-zvjjmsMtGVliVc8,2302
 parutils/file.py,sha256=-OPfMhTjHDqubZLWJquY-FOv3W_k9GWoMq_rNgtdU9Y,2881
@@ -9,15 +9,15 @@ parutils/strg.py,sha256=_8NQJ1iY58Z_xACTDVcyonvRHDRuy9Ai3Kv3CvVXRh4,5563
 parutils/testing.py,sha256=kXrMz8Kze4mZejaePmV_OIca4iuNcG_dhGVHCgVuF-k,610
 parutils/wrap.py,sha256=PqMyKodgEjWDXoIVZZXRaBSdkGFL5OWmVJttl2Crrg8,421
 parutils/logging/__init__.py,sha256=rSNpgjuYer-Hhn6zKzKwKSn_KfcajEXlk6cJnPC9eJU,461
-parutils/logging/cl.py,sha256=L40SkNdrdAT-7hfXAcFPHdB73PVqAhKVlMGC7Awtx1w,2079
-parutils/logging/const.py,sha256=YvIi9AROVeQHLS0hFrL96dWjW6jh4DO9r_ghZ4G-adA,115
+parutils/logging/cl.py,sha256=NCLuxLHvqRTypZlAgwnegkByLTnTirKntnwguJHBelA,2086
+parutils/logging/const.py,sha256=L2OppBDAZYKUTIjsLPt6hTUh8UWH_1mX9YaGWlwzpj4,135
 parutils/logging/core.py,sha256=vBwZnpM2GRNSSABPdzEM9SRpqr_YNqdPlG3T3lNPSiY,843
 parutils/logging/g.py,sha256=ZSrgZw63kwxIW66A7-9_iYeDt4AstNZ_XXQgK8MglyQ,47
-parutils/logging/logger.py,sha256=bBExUDHcqbaL4-7XktyAcJg5Pr0Bt52rLOpRYfiZqGA,2907
-parutils/logging/main.py,sha256=Z-ooY43PUk36nwVB_D9c9ZF7HUGngrFyVS8yBUVsTXA,1908
+parutils/logging/logger.py,sha256=XELvfvK4fB8DtbRnvlkHUnKgFnobAKGNnnkDwvUA9SM,3828
+parutils/logging/main.py,sha256=arUvgB0YTZpidt4wl2GZsR4bRUj_rrSYMrTnCMKt4gA,2127
 parutils/logging/sl.py,sha256=3-sj_o33cZmOqeFxlTl5HyHOvSAhn9glYcc-BmTUpZc,1164
-ParUtils-1.1.9.dist-info/LICENSE,sha256=6kbiFSfobTZ7beWiKnHpN902HgBx-Jzgcme0SvKqhKY,1091
-ParUtils-1.1.9.dist-info/METADATA,sha256=wHEdosiDfPzz7PzwJB4LyTqQF8KiqkPbikZBsvmU4RI,5175
-ParUtils-1.1.9.dist-info/WHEEL,sha256=G16H4A3IeoQmnOrYV4ueZGKSjhipXx8zc8nu9FGlvMA,92
-ParUtils-1.1.9.dist-info/top_level.txt,sha256=1MDobUcroeYEvdupZCAFvA5hJjm7LSDUV5A4jHySNis,9
-ParUtils-1.1.9.dist-info/RECORD,,
+ParUtils-1.2.1.dist-info/LICENSE,sha256=6kbiFSfobTZ7beWiKnHpN902HgBx-Jzgcme0SvKqhKY,1091
+ParUtils-1.2.1.dist-info/METADATA,sha256=wwlE_iL727342kRMkKB6dzuLDlqN0rLoyWIROkrqc7U,5285
+ParUtils-1.2.1.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
+ParUtils-1.2.1.dist-info/top_level.txt,sha256=1MDobUcroeYEvdupZCAFvA5hJjm7LSDUV5A4jHySNis,9
+ParUtils-1.2.1.dist-info/RECORD,,

{ParUtils-1.1.9.dist-info → ParUtils-1.2.1.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: bdist_wheel (0.37.1)
+Generator: setuptools (75.8.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

parutils/changelog.py

@@ -1,6 +1,13 @@
-__VERSION__ = '1.1.9'
-# check_log - name appears when check_log ok
-# u.g.logs
+__VERSION__ = '1.2.1'
+# log_dict + log_array - only one log_print -> better perfs
+# Improved check_log logs - name appears when nok
+
+# __VERSION__ = '1.2.0'
+# # logging - error handling
+
+# __VERSION__ = '1.1.9'
+# # check_log - name appears when check_log ok
+# # u.g.logs
 
 # __VERSION__ = '1.1.8'
 # # check_log - in_list optional

parutils/logging/cl.py

@@ -65,8 +65,8 @@ def check_warn(n_w, max_warn, name):
     if n_w == 0:
         log(f'check_log{s} ok')
     elif n_w <= max_warn:
-        log(f'check_log ended with {n_w} warnings')
+        log(f'check_log{s} ended with {n_w} warnings')
     else:
-        s = f'check_log ko, too many warnings ({n_w} warnings)'
+        s = f'check_log{s} nok, too many warnings ({n_w} warnings)'
         log(s)
         raise Exception(s)

parutils/logging/const.py

@@ -2,3 +2,4 @@ DEFAULT_LEVEL = 0
 DEFAULT_DIR = 'log'
 DEFAULT_LOG_FORMAT = '%H:%M:%S - '
 DEFAULT_FILE_FORMAT = '%Y%m%d_%H%M%S'
+MAX_ERR_COUNT = 10

parutils/logging/logger.py

@@ -29,6 +29,8 @@ class Logger:
             return
 
         self.logs = []
+        self.buffer = ''
+        self.err_count = 0
         self.level = level if level else const.DEFAULT_LEVEL
         self.log_format = log_format if log_format else const.DEFAULT_LOG_FORMAT
         self.file_write = file_write
@@ -50,7 +52,7 @@ class Logger:
         s = (f"Log file initialised ({self.log_path})\n"
              f"CWD: {os.getcwd()}\n"
              f"Python interpreter path: {sys.executable}\n"
-             f"Python version: {sys.version }\n"
+             f"Python version: {sys.version}\n"
              f"ParUtils version: {u.__VERSION__}\n")
         self.log_print(s)
         g.logger = self
@@ -70,7 +72,7 @@ class Logger:
         s = f"{fdate}{msg}"
         self.log_print(s, c_out=c_out)
 
-    def log_print(self, *args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char='    '):
+    def log_print(self, *args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char='    ', str_out=False):
         if self.level < level:
             return
 
@@ -83,16 +85,37 @@ class Logger:
         if dashes > 0:
             s = u.extend_str(s, '-', dashes)
 
+        if str_out:
+            return s + '\n'
         with lock:
-            if c_out:
-                print(s)
-            self._write_log(s)
+            self._write_log(s, c_out)
 
-    def _write_log(self, str_in):
+    def _write_log(self, str_in, c_out):
         s = str(str_in)
-        u.g.logs.append(s)
-        self.logs.append(s)
         if not self.file_write:
+            self._append_and_print(s, c_out)
             return
-        with open(self.log_path, 'a', encoding='utf-8') as in_file:
-            in_file.write(s + '\n')
+        try:
+            with open(self.log_path, 'a', encoding='utf-8') as in_file:
+                in_file.write(self.buffer + s + '\n')
+            self.buffer = ''
+            self.err_count = 0
+        except Exception as e:
+            s = self._handle_e(str_in, e)
+        self._append_and_print(s, c_out)
+
+    def _append_and_print(self, s, c_out):
+        u.g.logs.append(s)
+        self.logs.append(s)
+        if c_out:
+            print(s)
+
+    def _handle_e(self, str_in, e):
+        s = f"Warning: the following message couldn't be logged because of {e}: {u.truncate(str_in, 256)}"
+        self.buffer += s + '\n'
+        self.err_count += 1
+        if self.err_count > const.MAX_ERR_COUNT:
+            s += f"\nThe number of logging errors in a row reached the maximum set limit of {const.MAX_ERR_COUNT}. Disabling file_write."
+            self.buffer += s + '\n'
+            self.file_write = False
+        return s

parutils/logging/main.py

@@ -12,7 +12,7 @@ def log(*args, level=0, c_out=True):
 
 
 @logger_methode
-def log_print(*args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char='    '):
+def log_print(*args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char='    ', str_out=False):
     """Prints something in the current log file (log_path)
 
     - level: log level. Current log level is the attribute level of the current logger.
@@ -33,17 +33,23 @@ def log_input(str_in):
 
 
 def log_array(array, nb_tab=0, tab_char='    '):
+    out = ''
     for elt in array:
-        log_print(elt, nb_tab=nb_tab, tab_char=tab_char)
+        out += log_print(elt, nb_tab=nb_tab, tab_char=tab_char, str_out=True)
+    log_print(out)
 
 
-def log_dict(d, nb_tab=0, depth=0, tab_char='    '):
+def log_dict(d, nb_tab=0, depth=0, tab_char='    ', str_out=False):
+    out = ''
     for key in d:
         if isinstance(d[key], dict) and depth > 0:
-            log_print(f'{key}:', nb_tab=nb_tab, tab_char=tab_char)
-            log_dict(d[key], nb_tab + 1, depth - 1, tab_char=tab_char)
+            out += log_print(f'{key}:', nb_tab=nb_tab, tab_char=tab_char, str_out=True)
+            out += log_dict(d[key], nb_tab + 1, depth - 1, tab_char=tab_char, str_out=True)
         else:
-            log_print(f'{key}: {d[key]}', nb_tab=nb_tab, tab_char=tab_char)
+            out += log_print(f'{key}: {d[key]}', nb_tab=nb_tab, tab_char=tab_char, str_out=True)
+    if str_out:
+        return out
+    log_print(out)
 
 
 def log_example(list_in, what="duplicates", n_print=5):