ParUtils 1.1.9__py3-none-any.whl → 1.2.1__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,110 +1,110 @@
1
- Metadata-Version: 2.1
2
- Name: ParUtils
3
- Version: 1.1.9
4
- Summary: This package contains a bunch of Python utils developed for Support, Test and Automation IT Engineers
5
- Home-page: https://github.com/paularnaud2/ParUtils
6
- Author: Paul ARNAUD
7
- Author-email: paularnaud2@gmail.com
8
- Project-URL: Bug Tracker, https://github.com/paularnaud2/ParUtils/issues
9
- Classifier: Programming Language :: Python :: 3
10
- Classifier: License :: OSI Approved :: MIT License
11
- Classifier: Operating System :: OS Independent
12
- Requires-Python: >=3.6
13
- Description-Content-Type: text/markdown
14
- License-File: LICENSE
15
-
16
- # ParUtils
17
-
18
- This package is a light version of ParTools, requiring no third-party dependencies. It includes a reworked logging feature, as well as string, and file handling features.
19
- You can mainly use them for:
20
-
21
- - Logging information in a file
22
- - Loading / saving text files
23
- - Listing files
24
- - Loading / saving csv files (lighter but more performant than built in csv module)
25
- - Creating hash strings, random string, comparing strings with wildcard char
26
- - Comparing files and lists
27
- - Listing and removing duplicates from a list
28
-
29
- ## QuickStart
30
-
31
- pip install parutils
32
-
33
- You can start by testing the logger with the following code:
34
-
35
- import parutils as u
36
-
37
- u.Logger() # initializes a log file
38
- u.log('Hello World') # logs something in the console and in the log file
39
-
40
- You should then see something like this in the console:
41
-
42
- Log file initialised (c:\Dev\ParUtils\log\20221213_072132.txt)
43
- CWD: c:\Dev\ParUtils
44
- Python interpreter path: C:\Python\python.exe
45
- Python version: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
46
- ParUtils version: 1.0.8
47
-
48
- 07:21:32 - Hello World
49
-
50
- ## Example of useful functions
51
-
52
- ParUtils provides generic functions meant to be reused by external packages. In this section, a few of these functions are listed. For an exhaustive list, you can check out __parutils/\_\_init\_\_.py__.
53
-
54
- Manipulating files
55
- - `save_list`: saves a list into a text file
56
- - `load_txt`: loads a text file into a string or a list
57
- - `load_csv`: loads a csv file into a list of lists
58
- - `save_csv`: saves a list of lists into a csv file
59
- - `list_files`: lists files in a given directory (possibility to recurse subdirectories)
60
-
61
- Manipulating strings
62
- - `like`: behaves as the LIKE of Oracle SQL (you can match strings with wildcard character '\*'). It returns a re.match object giving you access to the matched wildcards strings.
63
- Example: ``m = like('Hello World', 'He\*o w\*d')``, ``m.group(1)`` => 'll'
64
- - `like_list` / `like_dict`: apply the `like` function directly to lists or dictionaries (see doc).
65
- - `big_number`: converts a potentially big number into a readable string.
66
- Example: ``big_number(10000000)`` => '10 000 000'.
67
- - `get_duration_string`: outputs a string representing the time elapsed since the input ``start_time``.
68
- Example: ``get_duration_string(0, end_time=200)`` => '3 minutes and 20 seconds'.
69
- - `hash512`: creates a non randomised hash string from a string.
70
- - `gen_random_string`: generates a random string.
71
-
72
- Data quality features:
73
- - `diff_list`: compares two lists
74
- - `file_match`: compare two files
75
- - `find_dup_list`: finds duplicates in a list
76
- - `del_dup_list`: removes duplicates from a list
77
-
78
-
79
- ### Logging with parutils
80
-
81
- The `log` function and the `Logger` class are directly available from the parutils package. So you can do:
82
-
83
- import parutils as u
84
-
85
- u.Logger()
86
- u.log('Hello World')
87
-
88
- Note: if you want the `log` function to actually write in a log file, you have to create a ``Logger`` object before using it, otherwise it will just print out the log info in the console.
89
-
90
- The relevant parameters such a the log directory or the log format can be specified when initializing the ``Logger`` object. The default ``log_format`` is ``'%H:%M:%S -'``, and a default log line looks like:
91
-
92
- 19:45:04 - This line has been generated by the parutils.log function
93
-
94
-
95
- Note that the default constants for the logging sub package are stored in __parutils.logging.const__. So for example, if you want to overwrite the default value for the logging directory, you can do:
96
-
97
- import parutils as u
98
-
99
- u.logging.const.DEFAULT_DIR = '<my_custom_dir>'
100
-
101
- The `step_log` function allows you to log some information only when the input ``counter`` is a multiple of the input ``step``. Thus, `step_log` is to be used in loops to __track the progress of long processes__ such as reading or writing millions of lines in a file. The ``what`` input expects a description of what is being counted. It's default value is ``'lines written'``.
102
- In order to correctly measure the elapsed time for the first log line, the ``step_log`` function has to be initialised by running ``init_sl_timer()``.
103
- So for example, if you input ``step=500`` and don't input any ``what`` value, you should get something like this:
104
-
105
- 19:45:04 - 500 lines written in 3 ms. 500 lines written in total.
106
- 19:45:04 - 500 lines written in 2 ms. 1 000 lines written in total.
107
- 19:45:04 - 500 lines written in 2 ms. 1 500 lines written in total.
108
-
109
- Checkout the __test_logging.py__ file in __tests/logging__ for simple examples of use.
110
-
1
+ Metadata-Version: 2.2
2
+ Name: ParUtils
3
+ Version: 1.2.1
4
+ Summary: This package contains a bunch of Python utils developed for Support, Test and Automation IT Engineers
5
+ Home-page: https://github.com/paularnaud2/ParUtils
6
+ Author: Paul ARNAUD
7
+ Author-email: paularnaud2@gmail.com
8
+ Project-URL: Bug Tracker, https://github.com/paularnaud2/ParUtils/issues
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Operating System :: OS Independent
12
+ Requires-Python: >=3.6
13
+ Description-Content-Type: text/markdown
14
+ License-File: LICENSE
15
+
16
+ # ParUtils
17
+
18
+ This package is a light version of ParTools, requiring no third-party dependencies. It includes a reworked logging feature, as well as string, and file handling features.
19
+ You can mainly use them for:
20
+
21
+ - Logging information in a file
22
+ - Loading / saving text files
23
+ - Listing files
24
+ - Loading / saving csv files (lighter but more performant than built in csv module)
25
+ - Creating hash strings, random string, comparing strings with wildcard char
26
+ - Comparing files and lists
27
+ - Listing and removing duplicates from a list
28
+
29
+ ## QuickStart
30
+
31
+ pip install parutils
32
+
33
+ You can start by testing the logger with the following code:
34
+
35
+ import parutils as u
36
+
37
+ u.Logger() # initializes a log file
38
+ u.log('Hello World') # logs something in the console and in the log file
39
+
40
+ You should then see something like this in the console:
41
+
42
+ Log file initialised (c:\Dev\ParUtils\log\20221213_072132.txt)
43
+ CWD: c:\Dev\ParUtils
44
+ Python interpreter path: C:\Python\python.exe
45
+ Python version: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
46
+ ParUtils version: 1.0.8
47
+
48
+ 07:21:32 - Hello World
49
+
50
+ ## Example of useful functions
51
+
52
+ ParUtils provides generic functions meant to be reused by external packages. In this section, a few of these functions are listed. For an exhaustive list, you can check out __parutils/\_\_init\_\_.py__.
53
+
54
+ Manipulating files
55
+ - `save_list`: saves a list into a text file
56
+ - `load_txt`: loads a text file into a string or a list
57
+ - `load_csv`: loads a csv file into a list of lists
58
+ - `save_csv`: saves a list of lists into a csv file
59
+ - `list_files`: lists files in a given directory (possibility to recurse subdirectories)
60
+
61
+ Manipulating strings
62
+ - `like`: behaves as the LIKE of Oracle SQL (you can match strings with wildcard character '\*'). It returns a re.match object giving you access to the matched wildcards strings.
63
+ Example: ``m = like('Hello World', 'He\*o w\*d')``, ``m.group(1)`` => 'll'
64
+ - `like_list` / `like_dict`: apply the `like` function directly to lists or dictionaries (see doc).
65
+ - `big_number`: converts a potentially big number into a readable string.
66
+ Example: ``big_number(10000000)`` => '10 000 000'.
67
+ - `get_duration_string`: outputs a string representing the time elapsed since the input ``start_time``.
68
+ Example: ``get_duration_string(0, end_time=200)`` => '3 minutes and 20 seconds'.
69
+ - `hash512`: creates a non randomised hash string from a string.
70
+ - `gen_random_string`: generates a random string.
71
+
72
+ Data quality features:
73
+ - `diff_list`: compares two lists
74
+ - `file_match`: compare two files
75
+ - `find_dup_list`: finds duplicates in a list
76
+ - `del_dup_list`: removes duplicates from a list
77
+
78
+
79
+ ### Logging with parutils
80
+
81
+ The `log` function and the `Logger` class are directly available from the parutils package. So you can do:
82
+
83
+ import parutils as u
84
+
85
+ u.Logger()
86
+ u.log('Hello World')
87
+
88
+ Note: if you want the `log` function to actually write in a log file, you have to create a ``Logger`` object before using it, otherwise it will just print out the log info in the console.
89
+
90
+ The relevant parameters such a the log directory or the log format can be specified when initializing the ``Logger`` object. The default ``log_format`` is ``'%H:%M:%S -'``, and a default log line looks like:
91
+
92
+ 19:45:04 - This line has been generated by the parutils.log function
93
+
94
+
95
+ Note that the default constants for the logging sub package are stored in __parutils.logging.const__. So for example, if you want to overwrite the default value for the logging directory, you can do:
96
+
97
+ import parutils as u
98
+
99
+ u.logging.const.DEFAULT_DIR = '<my_custom_dir>'
100
+
101
+ The `step_log` function allows you to log some information only when the input ``counter`` is a multiple of the input ``step``. Thus, `step_log` is to be used in loops to __track the progress of long processes__ such as reading or writing millions of lines in a file. The ``what`` input expects a description of what is being counted. It's default value is ``'lines written'``.
102
+ In order to correctly measure the elapsed time for the first log line, the ``step_log`` function has to be initialised by running ``init_sl_timer()``.
103
+ So for example, if you input ``step=500`` and don't input any ``what`` value, you should get something like this:
104
+
105
+ 19:45:04 - 500 lines written in 3 ms. 500 lines written in total.
106
+ 19:45:04 - 500 lines written in 2 ms. 1 000 lines written in total.
107
+ 19:45:04 - 500 lines written in 2 ms. 1 500 lines written in total.
108
+
109
+ Checkout the __test_logging.py__ file in __tests/logging__ for simple examples of use.
110
+
@@ -1,5 +1,5 @@
1
1
  parutils/__init__.py,sha256=EaFtUmU0kYgnjvAFwPmoKt7I5zn9bxY02R7TUtcE6M4,1434
2
- parutils/changelog.py,sha256=cDppejgTUKwpHW9xL5NJX-J91hJnyZKQScbY9vVeZlU,1347
2
+ parutils/changelog.py,sha256=3uz3n01a5Vt_JhNq6OfcTKPICGesQBhPcUaD5FeWSLk,1547
3
3
  parutils/csvl.py,sha256=UBqw0lyhTNqFLSNIE-JM7TAQYCRgHftYQmHi21ru8-A,2874
4
4
  parutils/dq.py,sha256=kdvzOo5FLCumXagY05kGSseYlbR-zvjjmsMtGVliVc8,2302
5
5
  parutils/file.py,sha256=-OPfMhTjHDqubZLWJquY-FOv3W_k9GWoMq_rNgtdU9Y,2881
@@ -9,15 +9,15 @@ parutils/strg.py,sha256=_8NQJ1iY58Z_xACTDVcyonvRHDRuy9Ai3Kv3CvVXRh4,5563
9
9
  parutils/testing.py,sha256=kXrMz8Kze4mZejaePmV_OIca4iuNcG_dhGVHCgVuF-k,610
10
10
  parutils/wrap.py,sha256=PqMyKodgEjWDXoIVZZXRaBSdkGFL5OWmVJttl2Crrg8,421
11
11
  parutils/logging/__init__.py,sha256=rSNpgjuYer-Hhn6zKzKwKSn_KfcajEXlk6cJnPC9eJU,461
12
- parutils/logging/cl.py,sha256=L40SkNdrdAT-7hfXAcFPHdB73PVqAhKVlMGC7Awtx1w,2079
13
- parutils/logging/const.py,sha256=YvIi9AROVeQHLS0hFrL96dWjW6jh4DO9r_ghZ4G-adA,115
12
+ parutils/logging/cl.py,sha256=NCLuxLHvqRTypZlAgwnegkByLTnTirKntnwguJHBelA,2086
13
+ parutils/logging/const.py,sha256=L2OppBDAZYKUTIjsLPt6hTUh8UWH_1mX9YaGWlwzpj4,135
14
14
  parutils/logging/core.py,sha256=vBwZnpM2GRNSSABPdzEM9SRpqr_YNqdPlG3T3lNPSiY,843
15
15
  parutils/logging/g.py,sha256=ZSrgZw63kwxIW66A7-9_iYeDt4AstNZ_XXQgK8MglyQ,47
16
- parutils/logging/logger.py,sha256=bBExUDHcqbaL4-7XktyAcJg5Pr0Bt52rLOpRYfiZqGA,2907
17
- parutils/logging/main.py,sha256=Z-ooY43PUk36nwVB_D9c9ZF7HUGngrFyVS8yBUVsTXA,1908
16
+ parutils/logging/logger.py,sha256=XELvfvK4fB8DtbRnvlkHUnKgFnobAKGNnnkDwvUA9SM,3828
17
+ parutils/logging/main.py,sha256=arUvgB0YTZpidt4wl2GZsR4bRUj_rrSYMrTnCMKt4gA,2127
18
18
  parutils/logging/sl.py,sha256=3-sj_o33cZmOqeFxlTl5HyHOvSAhn9glYcc-BmTUpZc,1164
19
- ParUtils-1.1.9.dist-info/LICENSE,sha256=6kbiFSfobTZ7beWiKnHpN902HgBx-Jzgcme0SvKqhKY,1091
20
- ParUtils-1.1.9.dist-info/METADATA,sha256=wHEdosiDfPzz7PzwJB4LyTqQF8KiqkPbikZBsvmU4RI,5175
21
- ParUtils-1.1.9.dist-info/WHEEL,sha256=G16H4A3IeoQmnOrYV4ueZGKSjhipXx8zc8nu9FGlvMA,92
22
- ParUtils-1.1.9.dist-info/top_level.txt,sha256=1MDobUcroeYEvdupZCAFvA5hJjm7LSDUV5A4jHySNis,9
23
- ParUtils-1.1.9.dist-info/RECORD,,
19
+ ParUtils-1.2.1.dist-info/LICENSE,sha256=6kbiFSfobTZ7beWiKnHpN902HgBx-Jzgcme0SvKqhKY,1091
20
+ ParUtils-1.2.1.dist-info/METADATA,sha256=wwlE_iL727342kRMkKB6dzuLDlqN0rLoyWIROkrqc7U,5285
21
+ ParUtils-1.2.1.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
22
+ ParUtils-1.2.1.dist-info/top_level.txt,sha256=1MDobUcroeYEvdupZCAFvA5hJjm7LSDUV5A4jHySNis,9
23
+ ParUtils-1.2.1.dist-info/RECORD,,
@@ -1,5 +1,5 @@
1
1
  Wheel-Version: 1.0
2
- Generator: bdist_wheel (0.37.1)
2
+ Generator: setuptools (75.8.0)
3
3
  Root-Is-Purelib: true
4
4
  Tag: py3-none-any
5
5
 
parutils/changelog.py CHANGED
@@ -1,6 +1,13 @@
1
- __VERSION__ = '1.1.9'
2
- # check_log - name appears when check_log ok
3
- # u.g.logs
1
+ __VERSION__ = '1.2.1'
2
+ # log_dict + log_array - only one log_print -> better perfs
3
+ # Improved check_log logs - name appears when nok
4
+
5
+ # __VERSION__ = '1.2.0'
6
+ # # logging - error handling
7
+
8
+ # __VERSION__ = '1.1.9'
9
+ # # check_log - name appears when check_log ok
10
+ # # u.g.logs
4
11
 
5
12
  # __VERSION__ = '1.1.8'
6
13
  # # check_log - in_list optional
parutils/logging/cl.py CHANGED
@@ -65,8 +65,8 @@ def check_warn(n_w, max_warn, name):
65
65
  if n_w == 0:
66
66
  log(f'check_log{s} ok')
67
67
  elif n_w <= max_warn:
68
- log(f'check_log ended with {n_w} warnings')
68
+ log(f'check_log{s} ended with {n_w} warnings')
69
69
  else:
70
- s = f'check_log ko, too many warnings ({n_w} warnings)'
70
+ s = f'check_log{s} nok, too many warnings ({n_w} warnings)'
71
71
  log(s)
72
72
  raise Exception(s)
parutils/logging/const.py CHANGED
@@ -2,3 +2,4 @@ DEFAULT_LEVEL = 0
2
2
  DEFAULT_DIR = 'log'
3
3
  DEFAULT_LOG_FORMAT = '%H:%M:%S - '
4
4
  DEFAULT_FILE_FORMAT = '%Y%m%d_%H%M%S'
5
+ MAX_ERR_COUNT = 10
@@ -29,6 +29,8 @@ class Logger:
29
29
  return
30
30
 
31
31
  self.logs = []
32
+ self.buffer = ''
33
+ self.err_count = 0
32
34
  self.level = level if level else const.DEFAULT_LEVEL
33
35
  self.log_format = log_format if log_format else const.DEFAULT_LOG_FORMAT
34
36
  self.file_write = file_write
@@ -50,7 +52,7 @@ class Logger:
50
52
  s = (f"Log file initialised ({self.log_path})\n"
51
53
  f"CWD: {os.getcwd()}\n"
52
54
  f"Python interpreter path: {sys.executable}\n"
53
- f"Python version: {sys.version }\n"
55
+ f"Python version: {sys.version}\n"
54
56
  f"ParUtils version: {u.__VERSION__}\n")
55
57
  self.log_print(s)
56
58
  g.logger = self
@@ -70,7 +72,7 @@ class Logger:
70
72
  s = f"{fdate}{msg}"
71
73
  self.log_print(s, c_out=c_out)
72
74
 
73
- def log_print(self, *args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char=' '):
75
+ def log_print(self, *args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char=' ', str_out=False):
74
76
  if self.level < level:
75
77
  return
76
78
 
@@ -83,16 +85,37 @@ class Logger:
83
85
  if dashes > 0:
84
86
  s = u.extend_str(s, '-', dashes)
85
87
 
88
+ if str_out:
89
+ return s + '\n'
86
90
  with lock:
87
- if c_out:
88
- print(s)
89
- self._write_log(s)
91
+ self._write_log(s, c_out)
90
92
 
91
- def _write_log(self, str_in):
93
+ def _write_log(self, str_in, c_out):
92
94
  s = str(str_in)
93
- u.g.logs.append(s)
94
- self.logs.append(s)
95
95
  if not self.file_write:
96
+ self._append_and_print(s, c_out)
96
97
  return
97
- with open(self.log_path, 'a', encoding='utf-8') as in_file:
98
- in_file.write(s + '\n')
98
+ try:
99
+ with open(self.log_path, 'a', encoding='utf-8') as in_file:
100
+ in_file.write(self.buffer + s + '\n')
101
+ self.buffer = ''
102
+ self.err_count = 0
103
+ except Exception as e:
104
+ s = self._handle_e(str_in, e)
105
+ self._append_and_print(s, c_out)
106
+
107
+ def _append_and_print(self, s, c_out):
108
+ u.g.logs.append(s)
109
+ self.logs.append(s)
110
+ if c_out:
111
+ print(s)
112
+
113
+ def _handle_e(self, str_in, e):
114
+ s = f"Warning: the following message couldn't be logged because of {e}: {u.truncate(str_in, 256)}"
115
+ self.buffer += s + '\n'
116
+ self.err_count += 1
117
+ if self.err_count > const.MAX_ERR_COUNT:
118
+ s += f"\nThe number of logging errors in a row reached the maximum set limit of {const.MAX_ERR_COUNT}. Disabling file_write."
119
+ self.buffer += s + '\n'
120
+ self.file_write = False
121
+ return s
parutils/logging/main.py CHANGED
@@ -12,7 +12,7 @@ def log(*args, level=0, c_out=True):
12
12
 
13
13
 
14
14
  @logger_methode
15
- def log_print(*args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char=' '):
15
+ def log_print(*args, level=0, c_out=True, nb_tab=0, dashes=0, tab_char=' ', str_out=False):
16
16
  """Prints something in the current log file (log_path)
17
17
 
18
18
  - level: log level. Current log level is the attribute level of the current logger.
@@ -33,17 +33,23 @@ def log_input(str_in):
33
33
 
34
34
 
35
35
  def log_array(array, nb_tab=0, tab_char=' '):
36
+ out = ''
36
37
  for elt in array:
37
- log_print(elt, nb_tab=nb_tab, tab_char=tab_char)
38
+ out += log_print(elt, nb_tab=nb_tab, tab_char=tab_char, str_out=True)
39
+ log_print(out)
38
40
 
39
41
 
40
- def log_dict(d, nb_tab=0, depth=0, tab_char=' '):
42
+ def log_dict(d, nb_tab=0, depth=0, tab_char=' ', str_out=False):
43
+ out = ''
41
44
  for key in d:
42
45
  if isinstance(d[key], dict) and depth > 0:
43
- log_print(f'{key}:', nb_tab=nb_tab, tab_char=tab_char)
44
- log_dict(d[key], nb_tab + 1, depth - 1, tab_char=tab_char)
46
+ out += log_print(f'{key}:', nb_tab=nb_tab, tab_char=tab_char, str_out=True)
47
+ out += log_dict(d[key], nb_tab + 1, depth - 1, tab_char=tab_char, str_out=True)
45
48
  else:
46
- log_print(f'{key}: {d[key]}', nb_tab=nb_tab, tab_char=tab_char)
49
+ out += log_print(f'{key}: {d[key]}', nb_tab=nb_tab, tab_char=tab_char, str_out=True)
50
+ if str_out:
51
+ return out
52
+ log_print(out)
47
53
 
48
54
 
49
55
  def log_example(list_in, what="duplicates", n_print=5):