hashio 0.3.0__tar.gz → 0.3.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: hashio
3
- Version: 0.3.0
3
+ Version: 0.3.2
4
4
  Summary: Custom file and directory checksum tool
5
5
  Home-page: http://github.com/rsgalloway/hashio
6
6
  Author: Ryan Galloway
@@ -33,6 +33,7 @@ Custom file and directory checksum and verification tool.
33
33
  - multiple hash algos: c4, crc32, md5, sha256, sha512, xxh64
34
34
  - recursively runs checksums on files in directory trees
35
35
  - ignores predefined file name patterns
36
+ - collects important file stat metadata
36
37
  - caches results for better performance
37
38
 
38
39
  ## Installation
@@ -59,6 +60,18 @@ results to a hash.json file:
59
60
  $ hashio <DIR>
60
61
  ```
61
62
 
63
+ #### Quick usage with uvx
64
+
65
+ You can run `hashio` instantly using uvx:
66
+
67
+ ```bash
68
+ $ uvx hashio <DIR>
69
+ ```
70
+
71
+ This downloads and runs `hashio` in a temporary, isolated environment — no
72
+ installs, no virtualenvs, no cleanup needed. Perfect for quick hash verification
73
+ tasks on large directories.
74
+
62
75
  #### Ignorable files
63
76
 
64
77
  Note that files matching patterns defined in `config.IGNORABLE` will be skipped,
@@ -80,10 +93,17 @@ $ hashio --verify hash.json
80
93
 
81
94
  #### Portability
82
95
 
83
- To make a portable hash file, use `--start` to make the paths relative:
96
+ To make a portable hash file, use `-or` to make the paths relative to the
97
+ `hash.json` file :
84
98
 
85
99
  ```bash
86
- $ hashio <DIR> --start <START> -o hash.json
100
+ $ hashio <DIR> -or hash.json
101
+ ```
102
+
103
+ or use `--start` to make them relative to the `<START>` value
104
+
105
+ ```bash
106
+ $ hashio <DIR> -o hash.json --start <START>
87
107
  ```
88
108
 
89
109
  To verify the data in the hash file, run hashio from the parent dir of the data,
@@ -115,6 +135,35 @@ $ vi debug.env # make edits
115
135
  $ ./debug.env -- hashio
116
136
  ```
117
137
 
138
+ ## Metadata
139
+
140
+ By default `hashio` collects the following file metadata:
141
+
142
+ | Key | Value |
143
+ |-------|-------------|
144
+ | name | file name |
145
+ | atime | file access time (st_atime) |
146
+ | ctime | file creattion time (st_ctime) |
147
+ | mtime | file modify time (st_mtime) |
148
+ | ino | file inode (st_ino) |
149
+ | dev | filesystem device (st_dev) |
150
+ | size | file size in bytes |
151
+ | type | path type - (f)ile or (d)irectory |
152
+
153
+ To walk a directory and collect metadata without peforming file checksums, use
154
+ the "null" hash algo:
155
+
156
+ ```bash
157
+ $ hashio <DIR> -a null
158
+ ```
159
+
160
+ To make "null" the default, update `${HASHIO_ALGO}` in the environment or the
161
+ `hashio.env` file.
162
+
163
+ ```bash
164
+ $ export HASHIO_ALGO=null
165
+ ```
166
+
118
167
  ## Python API
119
168
 
120
169
  Generate a `hash.json` file for a given path (Default is the current working
@@ -8,6 +8,7 @@ Custom file and directory checksum and verification tool.
8
8
  - multiple hash algos: c4, crc32, md5, sha256, sha512, xxh64
9
9
  - recursively runs checksums on files in directory trees
10
10
  - ignores predefined file name patterns
11
+ - collects important file stat metadata
11
12
  - caches results for better performance
12
13
 
13
14
  ## Installation
@@ -34,6 +35,18 @@ results to a hash.json file:
34
35
  $ hashio <DIR>
35
36
  ```
36
37
 
38
+ #### Quick usage with uvx
39
+
40
+ You can run `hashio` instantly using uvx:
41
+
42
+ ```bash
43
+ $ uvx hashio <DIR>
44
+ ```
45
+
46
+ This downloads and runs `hashio` in a temporary, isolated environment — no
47
+ installs, no virtualenvs, no cleanup needed. Perfect for quick hash verification
48
+ tasks on large directories.
49
+
37
50
  #### Ignorable files
38
51
 
39
52
  Note that files matching patterns defined in `config.IGNORABLE` will be skipped,
@@ -55,10 +68,17 @@ $ hashio --verify hash.json
55
68
 
56
69
  #### Portability
57
70
 
58
- To make a portable hash file, use `--start` to make the paths relative:
71
+ To make a portable hash file, use `-or` to make the paths relative to the
72
+ `hash.json` file :
59
73
 
60
74
  ```bash
61
- $ hashio <DIR> --start <START> -o hash.json
75
+ $ hashio <DIR> -or hash.json
76
+ ```
77
+
78
+ or use `--start` to make them relative to the `<START>` value
79
+
80
+ ```bash
81
+ $ hashio <DIR> -o hash.json --start <START>
62
82
  ```
63
83
 
64
84
  To verify the data in the hash file, run hashio from the parent dir of the data,
@@ -90,6 +110,35 @@ $ vi debug.env # make edits
90
110
  $ ./debug.env -- hashio
91
111
  ```
92
112
 
113
+ ## Metadata
114
+
115
+ By default `hashio` collects the following file metadata:
116
+
117
+ | Key | Value |
118
+ |-------|-------------|
119
+ | name | file name |
120
+ | atime | file access time (st_atime) |
121
+ | ctime | file creattion time (st_ctime) |
122
+ | mtime | file modify time (st_mtime) |
123
+ | ino | file inode (st_ino) |
124
+ | dev | filesystem device (st_dev) |
125
+ | size | file size in bytes |
126
+ | type | path type - (f)ile or (d)irectory |
127
+
128
+ To walk a directory and collect metadata without peforming file checksums, use
129
+ the "null" hash algo:
130
+
131
+ ```bash
132
+ $ hashio <DIR> -a null
133
+ ```
134
+
135
+ To make "null" the default, update `${HASHIO_ALGO}` in the environment or the
136
+ `hashio.env` file.
137
+
138
+ ```bash
139
+ $ export HASHIO_ALGO=null
140
+ ```
141
+
93
142
  ## Python API
94
143
 
95
144
  Generate a `hash.json` file for a given path (Default is the current working
@@ -35,7 +35,7 @@ Custom file and directory checksum tool.
35
35
 
36
36
  __author__ = "ryan@rsgalloway.com"
37
37
  __prog__ = "hashio"
38
- __version__ = "0.3.0"
38
+ __version__ = "0.3.2"
39
39
 
40
40
  try:
41
41
  import envstack
@@ -34,9 +34,11 @@ Contains command line interface for hashio.
34
34
  """
35
35
 
36
36
  import argparse
37
- import multiprocessing
38
37
  import os
39
38
  import sys
39
+ import threading
40
+ import time
41
+ from tqdm import tqdm
40
42
 
41
43
  from hashio import __version__, config, utils
42
44
  from hashio.encoder import get_encoder_class, verify_caches, verify_checksums
@@ -44,9 +46,29 @@ from hashio.logger import logger
44
46
  from hashio.worker import HashWorker
45
47
 
46
48
 
49
+ def normalize_args(argv):
50
+ """Rewrite `-or value` to `-o value -r value`."""
51
+ new_argv = []
52
+ skip = False
53
+ for i, arg in enumerate(argv):
54
+ if skip:
55
+ skip = False
56
+ continue
57
+ if arg == "-or" and i + 1 < len(argv):
58
+ val = argv[i + 1]
59
+ new_argv.extend(["-o", val, "-r", os.path.dirname(val) or "."])
60
+ skip = True
61
+ else:
62
+ new_argv.append(arg)
63
+ return new_argv
64
+
65
+
47
66
  def parse_args():
48
67
  """Parse command line arguments."""
49
68
 
69
+ # normalize arguments to handle -or value as -o value -r value
70
+ argv = normalize_args(sys.argv[1:])
71
+
50
72
  parser = argparse.ArgumentParser(
51
73
  description=__doc__, formatter_class=argparse.RawTextHelpFormatter
52
74
  )
@@ -67,6 +89,7 @@ def parse_args():
67
89
  default=config.CACHE_FILENAME,
68
90
  )
69
91
  parser.add_argument(
92
+ "-p",
70
93
  "--procs",
71
94
  type=int,
72
95
  metavar="PROCS",
@@ -74,7 +97,7 @@ def parse_args():
74
97
  default=config.MAX_PROCS,
75
98
  )
76
99
  parser.add_argument(
77
- "-s",
100
+ "-r",
78
101
  "--start",
79
102
  type=str,
80
103
  metavar="START",
@@ -114,41 +137,33 @@ def parse_args():
114
137
  help="verify checksums from a previously created hash file",
115
138
  )
116
139
 
117
- args = parser.parse_args()
140
+ args = parser.parse_args(argv)
118
141
  return args
119
142
 
120
143
 
121
- def watch_progress(worker: HashWorker):
122
- """Watch the progress of the worker and update a progress bar.
144
+ def start_progress_thread(worker, update_interval=0.2):
145
+ """Start a thread to monitor worker.progress.value and update tqdm.
123
146
 
124
- :param worker: HashWorker instance to monitor
147
+ :param worker: The HashWorker instance to monitor.
148
+ :param update_interval: How often to update the progress bar (in seconds).
149
+ :return: The thread that updates the progress bar.
125
150
  """
151
+ pbar = tqdm(desc="hashing files", unit="file", dynamic_ncols=True)
126
152
 
127
- import time
128
- from tqdm import tqdm
153
+ def watch():
154
+ while not worker.is_done():
155
+ pbar.n = worker.progress_count()
156
+ pbar.refresh()
157
+ time.sleep(update_interval)
129
158
 
130
- pbar = tqdm(
131
- desc="hashing files",
132
- disable=(worker.verbose or not sys.stdout.isatty()),
133
- unit="file",
134
- )
135
- last = 0
136
-
137
- while not worker.done.is_set():
138
- with worker.progress.get_lock():
139
- current = worker.progress.value
140
- delta = current - last
141
- if delta:
142
- pbar.update(delta)
143
- last = current
144
- time.sleep(0.2)
159
+ # final update to catch any remaining progress
160
+ pbar.n = worker.progress_count()
161
+ pbar.refresh()
162
+ pbar.close()
145
163
 
146
- # final update in case we missed some
147
- with worker.progress.get_lock():
148
- final = worker.progress.value
149
-
150
- pbar.update(final - last)
151
- pbar.close()
164
+ thread = threading.Thread(target=watch, daemon=True)
165
+ thread.start()
166
+ return thread
152
167
 
153
168
 
154
169
  def main():
@@ -204,15 +219,16 @@ def main():
204
219
  )
205
220
 
206
221
  try:
207
- watcher = multiprocessing.Process(target=watch_progress, args=(worker,))
208
- watcher.start()
222
+ if sys.stdout.isatty():
223
+ progress_thread = start_progress_thread(worker)
209
224
  worker.run()
210
- watcher.join()
225
+ if sys.stdout.isatty():
226
+ progress_thread.join()
211
227
 
212
228
  except KeyboardInterrupt:
213
- print("stopping...")
214
229
  worker.stop()
215
- return 2
230
+ print("\nstopping...")
231
+ sys.exit(0)
216
232
 
217
233
  finally:
218
234
  logger.debug("done in %s seconds", worker.total_time)
@@ -160,8 +160,35 @@ def checksum_gen(
160
160
  yield (algo, value, path)
161
161
 
162
162
 
163
+ class NullChecksum(object):
164
+ """A no-op checksum class that does nothing."""
165
+
166
+ def update(self, data: bytes):
167
+ """No-op update method."""
168
+ return self # for chaining
169
+
170
+ def digest(self):
171
+ """Return an empty byte string."""
172
+ return b""
173
+
174
+ def hexdigest(self):
175
+ """Return an empty hex string."""
176
+ return ""
177
+
178
+ def copy(self):
179
+ """Return a new instance of NullChecksum."""
180
+ return NullChecksum()
181
+
182
+ def reset(self):
183
+ """No-op reset method."""
184
+ return self
185
+
186
+
163
187
  class CRC32(object):
164
- """Simple implementation of CRC32 checksum class."""
188
+ """
189
+ Simple implementation of CRC32 checksum class. Used by the
190
+ CRC32Encoder class.
191
+ """
165
192
 
166
193
  def __init__(self):
167
194
  self._initial = 0
@@ -221,6 +248,16 @@ class Encoder(object):
221
248
  self.hash.update(data)
222
249
 
223
250
 
251
+ class NullEncoder(Encoder):
252
+ """Null Encoder class."""
253
+
254
+ name = "null"
255
+
256
+ def __init__(self):
257
+ super(NullEncoder, self).__init__()
258
+ self.hash = NullChecksum()
259
+
260
+
224
261
  class MD5Encoder(Encoder):
225
262
  """MD5 Encoder class."""
226
263
 
@@ -65,11 +65,11 @@ def writer_process(
65
65
  :param flush_interval: time interval in seconds to flush data
66
66
  :param batch_size: number of items to collect before flushing
67
67
  """
68
+ from queue import Empty
69
+
68
70
  buffer = []
69
71
  last_flush = time.time()
70
72
 
71
- from queue import Empty
72
-
73
73
  # ensure the exporter is open
74
74
  while True:
75
75
  try:
@@ -79,6 +79,8 @@ def writer_process(
79
79
  buffer.append(item)
80
80
  except Empty:
81
81
  pass
82
+ except (KeyboardInterrupt, EOFError):
83
+ break # clean exit
82
84
  except Exception as e:
83
85
  logger.error("write error: %s", e)
84
86
 
@@ -98,7 +100,6 @@ class HashWorker:
98
100
 
99
101
  >>> w = HashWorker(path, outfile="hash.json")
100
102
  >>> w.run()
101
- >>> pprint(w.results)
102
103
  """
103
104
 
104
105
  def __init__(
@@ -122,6 +123,7 @@ class HashWorker:
122
123
  :param verbose: if True, print verbose output
123
124
  """
124
125
  self.path = path
126
+ self.algo = algo
125
127
  self.outfile = outfile
126
128
  self.procs = procs
127
129
  self.force = force
@@ -132,7 +134,6 @@ class HashWorker:
132
134
  self.verbose = verbose
133
135
  self.progress = Value("i", 0) # shared int for progress
134
136
  self.start = start or os.path.relpath(path)
135
- self.encoder = get_encoder_class(algo)()
136
137
  self.exporter = get_exporter_class(os.path.splitext(outfile)[1])(outfile)
137
138
  self.queue = Queue() # task queue
138
139
  self.result_queue = Queue() # write queue
@@ -185,14 +186,17 @@ class HashWorker:
185
186
 
186
187
  :param path: file path
187
188
  """
188
- value = checksum_file(path, self.encoder)
189
+ encoder = get_encoder_class(self.algo)()
190
+ value = checksum_file(path, encoder)
189
191
 
190
192
  # normalize path to be relative to the start directory
191
193
  npath = normalize_path(path, start=self.start)
192
194
 
193
195
  # get metadata for the file
194
196
  metadata = get_metadata(path)
195
- metadata.update({self.encoder.name: value})
197
+
198
+ # add the checksum value to metadata
199
+ metadata.update({self.algo: value})
196
200
 
197
201
  # print progress to stdout
198
202
  if self.verbose:
@@ -229,11 +233,25 @@ class HashWorker:
229
233
  self.done.set()
230
234
 
231
235
  def stop(self):
232
- """Stops worker."""
233
- logger.info("stopping %s", str(self))
234
- self.pool.terminate()
236
+ """Stops the worker and cleans up resources."""
237
+ if self.pool:
238
+ self.pool.terminate()
239
+ self.pool.join()
240
+ if self.writer:
241
+ self.writer.terminate()
242
+ self.writer.join()
235
243
  self.exporter.close()
236
244
  self.total_time = time.time() - self.start_time
245
+ self.done.set()
246
+ logger.debug("stopping %s", multiprocessing.current_process())
247
+
248
+ def progress_count(self):
249
+ """Returns the current progress count."""
250
+ return self.progress.value
251
+
252
+ def is_done(self):
253
+ """Checks if the worker has completed its tasks."""
254
+ return self.done.is_set()
237
255
 
238
256
  @staticmethod
239
257
  def main(worker):
@@ -251,6 +269,8 @@ class HashWorker:
251
269
  worker.do_hash(path)
252
270
  except queue.Empty:
253
271
  break
272
+ except (KeyboardInterrupt, EOFError):
273
+ break # clean exit
254
274
  except Exception as err:
255
275
  logger.error(err)
256
276
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: hashio
3
- Version: 0.3.0
3
+ Version: 0.3.2
4
4
  Summary: Custom file and directory checksum tool
5
5
  Home-page: http://github.com/rsgalloway/hashio
6
6
  Author: Ryan Galloway
@@ -33,6 +33,7 @@ Custom file and directory checksum and verification tool.
33
33
  - multiple hash algos: c4, crc32, md5, sha256, sha512, xxh64
34
34
  - recursively runs checksums on files in directory trees
35
35
  - ignores predefined file name patterns
36
+ - collects important file stat metadata
36
37
  - caches results for better performance
37
38
 
38
39
  ## Installation
@@ -59,6 +60,18 @@ results to a hash.json file:
59
60
  $ hashio <DIR>
60
61
  ```
61
62
 
63
+ #### Quick usage with uvx
64
+
65
+ You can run `hashio` instantly using uvx:
66
+
67
+ ```bash
68
+ $ uvx hashio <DIR>
69
+ ```
70
+
71
+ This downloads and runs `hashio` in a temporary, isolated environment — no
72
+ installs, no virtualenvs, no cleanup needed. Perfect for quick hash verification
73
+ tasks on large directories.
74
+
62
75
  #### Ignorable files
63
76
 
64
77
  Note that files matching patterns defined in `config.IGNORABLE` will be skipped,
@@ -80,10 +93,17 @@ $ hashio --verify hash.json
80
93
 
81
94
  #### Portability
82
95
 
83
- To make a portable hash file, use `--start` to make the paths relative:
96
+ To make a portable hash file, use `-or` to make the paths relative to the
97
+ `hash.json` file :
84
98
 
85
99
  ```bash
86
- $ hashio <DIR> --start <START> -o hash.json
100
+ $ hashio <DIR> -or hash.json
101
+ ```
102
+
103
+ or use `--start` to make them relative to the `<START>` value
104
+
105
+ ```bash
106
+ $ hashio <DIR> -o hash.json --start <START>
87
107
  ```
88
108
 
89
109
  To verify the data in the hash file, run hashio from the parent dir of the data,
@@ -115,6 +135,35 @@ $ vi debug.env # make edits
115
135
  $ ./debug.env -- hashio
116
136
  ```
117
137
 
138
+ ## Metadata
139
+
140
+ By default `hashio` collects the following file metadata:
141
+
142
+ | Key | Value |
143
+ |-------|-------------|
144
+ | name | file name |
145
+ | atime | file access time (st_atime) |
146
+ | ctime | file creattion time (st_ctime) |
147
+ | mtime | file modify time (st_mtime) |
148
+ | ino | file inode (st_ino) |
149
+ | dev | filesystem device (st_dev) |
150
+ | size | file size in bytes |
151
+ | type | path type - (f)ile or (d)irectory |
152
+
153
+ To walk a directory and collect metadata without peforming file checksums, use
154
+ the "null" hash algo:
155
+
156
+ ```bash
157
+ $ hashio <DIR> -a null
158
+ ```
159
+
160
+ To make "null" the default, update `${HASHIO_ALGO}` in the environment or the
161
+ `hashio.env` file.
162
+
163
+ ```bash
164
+ $ export HASHIO_ALGO=null
165
+ ```
166
+
118
167
  ## Python API
119
168
 
120
169
  Generate a `hash.json` file for a given path (Default is the current working
@@ -40,7 +40,7 @@ with open(os.path.join(here, "README.md")) as f:
40
40
 
41
41
  setup(
42
42
  name="hashio",
43
- version="0.3.0",
43
+ version="0.3.2",
44
44
  description="Custom file and directory checksum tool",
45
45
  long_description=long_description,
46
46
  long_description_content_type="text/markdown",
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes