vete 0.6.7 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +146 -0
- data/lib/vete.rb +2 -2
- data/test/example.rb +9 -7
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: b13f06ca7363e892403446d521839c18bbdd16ca832a9af2fdab9ed093703135
|
4
|
+
data.tar.gz: d17273c1de3f1b6ea4455ea9293d009d2353408121b87e8f5586ee3532bbca3c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c5a9083c1193e9338bf943df433eb8933d0247ef587338511d163a595403379f7cd06449f6e2eff57c374984a7fd4dc8d5dc5b27ccc9c1fa1cb426792c10c9de
|
7
|
+
data.tar.gz: aa56000561c727f918553b6ccf436cd3394decccf5473e8f6c1b4a754e24acab4e421116da13c509be61b9913499be0fc25d8799c85a0809fb5565830b31f653
|
data/README.md
CHANGED
@@ -4,8 +4,154 @@ Ruby CLI to spawn processes to get work done
|
|
4
4
|
|
5
5
|
The phrase "¡véte!" in Spanish means, basically, "Get out!". This tool helps to clear out work in a hurry, using a simple approach of spawning a set number of concurrent processes to handle each job. Jobs are defined as files in a directory, so there is no need for a database or any other complexity.
|
6
6
|
|
7
|
+
### Summary
|
8
|
+
|
9
|
+
To use `vete`, there are three steps:
|
10
|
+
|
11
|
+
1. Define a method called `setup` which sets up a context for each task
|
12
|
+
2. Define a method called `perform(task)` which is invoked for each task
|
13
|
+
3. At the end of your script, trigger everything with `require "vete"`
|
14
|
+
|
15
|
+
When your script executes, the `setup` method is called once. Its purpose is to
|
16
|
+
initialize a context that all subsequent tasks will inherit. It also is where new
|
17
|
+
tasks are defined or prior failed tasks can be prepared to be retried. Instance
|
18
|
+
variables and other context defined in the `setup` method is available to each task.
|
19
|
+
|
20
|
+
Once the `setup` method has been called, a configurable number of worker processes
|
21
|
+
will be spawned in parallel. Each worker will immediately call `perform(task)`. Since
|
22
|
+
each process inherits the context defined by the `setup` method, memory is efficiently
|
23
|
+
shared. As tasks are executed, a progress bar will indicate the overall completion status.
|
24
|
+
|
7
25
|
### Example
|
8
26
|
|
9
27
|
Running the `test/example.rb` script with 10 workers:
|
10
28
|
|
11
29
|

|
30
|
+
|
31
|
+
Here is the code for the above:
|
32
|
+
|
33
|
+
```ruby
|
34
|
+
#!/usr/bin/env ruby
|
35
|
+
|
36
|
+
def setup
|
37
|
+
vete_retry or begin # retry prior failed tasks, or
|
38
|
+
vete_init # initialize the main task directory structure
|
39
|
+
100.times {|i| vete_todo(i + 1) } # create 100 new tasks
|
40
|
+
end
|
41
|
+
@time = Time.now # instance variables are visible to each task
|
42
|
+
end
|
43
|
+
|
44
|
+
def perform(task)
|
45
|
+
sleep rand # simulate some work performed
|
46
|
+
secs = Time.now - @time # do something with @time (defined in setup)
|
47
|
+
exit 1 if rand < 0.03 # simulate a 3% chance of failure
|
48
|
+
end
|
49
|
+
|
50
|
+
require "vete"
|
51
|
+
```
|
52
|
+
|
53
|
+
### Inner workings
|
54
|
+
|
55
|
+
```
|
56
|
+
.vete/
|
57
|
+
├── died/
|
58
|
+
├── done/
|
59
|
+
└── todo/
|
60
|
+
```
|
61
|
+
|
62
|
+
The above directory structure is used by `vete` to define tasks and to process
|
63
|
+
their lifecycle. Tasks are defined as files in the `.vete/todo` directory. For example,
|
64
|
+
if we needed to pull down a report for four days in April 2023, we may define these
|
65
|
+
four tasks as follows:
|
66
|
+
|
67
|
+
```
|
68
|
+
.vete/
|
69
|
+
├── died/
|
70
|
+
├── done/
|
71
|
+
└── todo/
|
72
|
+
│ ├── 20230410
|
73
|
+
│ ├── 20230411
|
74
|
+
│ ├── 20230412
|
75
|
+
│ └── 20230413
|
76
|
+
```
|
77
|
+
|
78
|
+
This file structure can be defined in the `setup` method, or you could choose to
|
79
|
+
manually create the files any other way.
|
80
|
+
|
81
|
+
When `vete` is launched by the `require "vete"` line in the script, it will call
|
82
|
+
the `setup` script (if it is defined). Then, it will look for files in the `.vete/todo`
|
83
|
+
directory. The desired number of worker processes is then launched in parallel, each
|
84
|
+
time calling `perform(task)` with `task` being the full pathname of the next file in the
|
85
|
+
`todo` directory.
|
86
|
+
|
87
|
+
If `perform(task)` executes without any error, then the file for that task will be moved
|
88
|
+
to the `done` directory. If errors occur, the file is moved to the `died` directory.
|
89
|
+
Suppose that three of the tasks above successfully completed, but one failed. This would
|
90
|
+
yield the following file structure:
|
91
|
+
|
92
|
+
```
|
93
|
+
.vete/
|
94
|
+
├── died/
|
95
|
+
│ ├── 20230412
|
96
|
+
├── done/
|
97
|
+
│ ├── 20230410
|
98
|
+
│ ├── 20230411
|
99
|
+
│ └── 20230413
|
100
|
+
└── todo/
|
101
|
+
```
|
102
|
+
|
103
|
+
### Flexible tasks
|
104
|
+
|
105
|
+
Note that any filename can be used and the files can be either empty (with the filename
|
106
|
+
being used to indicate the nature of the task), or the files can contain data (such as
|
107
|
+
JSON or anything else). The `perform` method is free to do whatever is needed to process
|
108
|
+
the task and since it's running in it's own process, there is no concern for traditional
|
109
|
+
thread concurrency issues, etc.
|
110
|
+
|
111
|
+
As an example, here is another valid set of tasks that may contain JSON payloads that
|
112
|
+
are needed when processing each task.
|
113
|
+
|
114
|
+
```
|
115
|
+
.vete/
|
116
|
+
├── died/
|
117
|
+
├── done/
|
118
|
+
└── todo/
|
119
|
+
│ ├── amazon.json
|
120
|
+
│ ├── apple.json
|
121
|
+
│ ├── facebook.json
|
122
|
+
│ └── google.json
|
123
|
+
```
|
124
|
+
|
125
|
+
### Additional tips
|
126
|
+
|
127
|
+
A command line utility (simply called `vete`) can be used to launch a script that
|
128
|
+
defines the `perform(task)` method and, optionally, the `setup` method. You can also
|
129
|
+
run `vete -r` to remove the entire `.vete` directory.
|
130
|
+
|
131
|
+
Running `vete -h` provides some additional help:
|
132
|
+
|
133
|
+
```shell
|
134
|
+
$ vete -h
|
135
|
+
|
136
|
+
usage: vete [options]
|
137
|
+
-b, --bar <width> Progress bar width, in characters
|
138
|
+
-c, --char <character> Character to use for progress bar
|
139
|
+
-d, --delay <mode> Delay mode (rand, task, numeric)
|
140
|
+
-h, --help Show help and command usage
|
141
|
+
-r, --reset Remove directory used for job processing and quit
|
142
|
+
-v, --version Show version number
|
143
|
+
-w, --workers <count> Set the number of workers (default is 1)
|
144
|
+
```
|
145
|
+
|
146
|
+
Running a `vete` enabled script (ie - one that contains `require "vete"` as the last
|
147
|
+
line of the file) will automatically extend the `vete` command line utility. As a result,
|
148
|
+
you can run your `vete` enabled script directly and pass any of the above command line
|
149
|
+
options, as follows:
|
150
|
+
|
151
|
+
```shell
|
152
|
+
test/example.rb -w 10
|
153
|
+
```
|
154
|
+
|
155
|
+
This will run the `example.rb` file (which creates 100 tasks) and it will spawn 10
|
156
|
+
concurrent processes to perform the work. See the screencast at the top of this file
|
157
|
+
to see how this works.
|
data/lib/vete.rb
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
# vete - Ruby CLI to spawn processes to get work done
|
3
3
|
#
|
4
4
|
# Author: Steve Shreeve (steve.shreeve@gmail.com)
|
5
|
-
# Date:
|
5
|
+
# Date: July 1, 2023
|
6
6
|
# ============================================================================
|
7
7
|
|
8
8
|
STDOUT.sync = true
|
@@ -19,7 +19,7 @@ trap("INT" ) { print clear + go; abort "\n" }
|
|
19
19
|
trap("WINCH") { print clear or draw if @pid == Process.pid }
|
20
20
|
|
21
21
|
OptionParser.new.instance_eval do
|
22
|
-
@version = "0.
|
22
|
+
@version = "1.0.0"
|
23
23
|
@banner = "usage: #{program_name} [options]"
|
24
24
|
|
25
25
|
on "-b", "--bar <width>" , "Progress bar width, in characters", Integer
|
data/test/example.rb
CHANGED
@@ -1,15 +1,17 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
3
|
def setup
|
4
|
-
|
5
|
-
|
6
|
-
|
4
|
+
vete_retry or begin # retry prior failed tasks, or
|
5
|
+
vete_init # initialize the main task directory structure
|
6
|
+
100.times {|i| vete_todo(i + 1) } # create 100 new tasks
|
7
|
+
end
|
8
|
+
@time = Time.now # instance variables are visible to each task
|
7
9
|
end
|
8
10
|
|
9
11
|
def perform(task)
|
10
|
-
sleep rand
|
11
|
-
secs = Time.now - @time
|
12
|
-
exit 1 if rand < 0.03
|
12
|
+
sleep rand # simulate some work performed
|
13
|
+
secs = Time.now - @time # do something with @time (defined in setup)
|
14
|
+
exit 1 if rand < 0.03 # simulate a 3% chance of failure
|
13
15
|
end
|
14
16
|
|
15
|
-
|
17
|
+
require "vete"
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: vete
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Steve Shreeve
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-07-01 00:00:00.000000000 Z
|
12
12
|
dependencies: []
|
13
13
|
description: Ruby CLI to spawn processes to get work done
|
14
14
|
email: steve.shreeve@gmail.com
|