vete 0.6.7 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +146 -0
  3. data/lib/vete.rb +2 -2
  4. data/test/example.rb +9 -7
  5. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 96b64fc68cb1332add0e65d46131d363ac1723c9b70f0f814382c29efeecb65f
4
- data.tar.gz: 2601b593a3ca14b98576cde84de8dc40abd79cb3a6243784099ec8e4298baf7d
3
+ metadata.gz: b13f06ca7363e892403446d521839c18bbdd16ca832a9af2fdab9ed093703135
4
+ data.tar.gz: d17273c1de3f1b6ea4455ea9293d009d2353408121b87e8f5586ee3532bbca3c
5
5
  SHA512:
6
- metadata.gz: fb245248250c901de9ca611e4aa424fb7376fae97a96a71f448879cda58852632c6d7f98437ec271431ffa23cb8fc66fbcd09254850192480e3fe8d16032c7f7
7
- data.tar.gz: c9e88e46d546b8ea63ef92f71b2684db2c114e1303cac1380a7e43a574d3bd45fb79a01ba53f8909d1dbef8e893506b7f00dfc5be1f40549f5fd62ae01788faa
6
+ metadata.gz: c5a9083c1193e9338bf943df433eb8933d0247ef587338511d163a595403379f7cd06449f6e2eff57c374984a7fd4dc8d5dc5b27ccc9c1fa1cb426792c10c9de
7
+ data.tar.gz: aa56000561c727f918553b6ccf436cd3394decccf5473e8f6c1b4a754e24acab4e421116da13c509be61b9913499be0fc25d8799c85a0809fb5565830b31f653
data/README.md CHANGED
@@ -4,8 +4,154 @@ Ruby CLI to spawn processes to get work done
4
4
 
5
5
  The phrase "¡véte!" in Spanish means, basically, "Get out!". This tool helps to clear out work in a hurry, using a simple approach of spawning a set number of concurrent processes to handle each job. Jobs are defined as files in a directory, so there is no need for a database or any other complexity.
6
6
 
7
+ ### Summary
8
+
9
+ To use `vete`, there are three steps:
10
+
11
+ 1. Define a method called `setup` which sets up a context for each task
12
+ 2. Define a method called `perform(task)` which is invoked for each task
13
+ 3. At the end of your script, trigger everything with `require "vete"`
14
+
15
+ When your script executes, the `setup` method is called once. Its purpose is to
16
+ initialize a context that all subsequent tasks will inherit. It also is where new
17
+ tasks are defined or prior failed tasks can be prepared to be retried. Instance
18
+ variables and other context defined in the `setup` method is available to each task.
19
+
20
+ Once the `setup` method has been called, a configurable number of worker processes
21
+ will be spawned in parallel. Each worker will immediately call `perform(task)`. Since
22
+ each process inherits the context defined by the `setup` method, memory is efficiently
23
+ shared. As tasks are executed, a progress bar will indicate the overall completion status.
24
+
7
25
  ### Example
8
26
 
9
27
  Running the `test/example.rb` script with 10 workers:
10
28
 
11
29
  ![Example](https://raw.githubusercontent.com/shreeve/vete/main/test/vete.gif)
30
+
31
+ Here is the code for the above:
32
+
33
+ ```ruby
34
+ #!/usr/bin/env ruby
35
+
36
+ def setup
37
+ vete_retry or begin # retry prior failed tasks, or
38
+ vete_init # initialize the main task directory structure
39
+ 100.times {|i| vete_todo(i + 1) } # create 100 new tasks
40
+ end
41
+ @time = Time.now # instance variables are visible to each task
42
+ end
43
+
44
+ def perform(task)
45
+ sleep rand # simulate some work performed
46
+ secs = Time.now - @time # do something with @time (defined in setup)
47
+ exit 1 if rand < 0.03 # simulate a 3% chance of failure
48
+ end
49
+
50
+ require "vete"
51
+ ```
52
+
53
+ ### Inner workings
54
+
55
+ ```
56
+ .vete/
57
+ ├── died/
58
+ ├── done/
59
+ └── todo/
60
+ ```
61
+
62
+ The above directory structure is used by `vete` to define tasks and to process
63
+ their lifecycle. Tasks are defined as files in the `.vete/todo` directory. For example,
64
+ if we needed to pull down a report for four days in April 2023, we may define these
65
+ four tasks as follows:
66
+
67
+ ```
68
+ .vete/
69
+ ├── died/
70
+ ├── done/
71
+ └── todo/
72
+ │   ├── 20230410
73
+ │   ├── 20230411
74
+ │   ├── 20230412
75
+ │   └── 20230413
76
+ ```
77
+
78
+ This file structure can be defined in the `setup` method, or you could choose to
79
+ manually create the files any other way.
80
+
81
+ When `vete` is launched by the `require "vete"` line in the script, it will call
82
+ the `setup` script (if it is defined). Then, it will look for files in the `.vete/todo`
83
+ directory. The desired number of worker processes is then launched in parallel, each
84
+ time calling `perform(task)` with `task` being the full pathname of the next file in the
85
+ `todo` directory.
86
+
87
+ If `perform(task)` executes without any error, then the file for that task will be moved
88
+ to the `done` directory. If errors occur, the file is moved to the `died` directory.
89
+ Suppose that three of the tasks above successfully completed, but one failed. This would
90
+ yield the following file structure:
91
+
92
+ ```
93
+ .vete/
94
+ ├── died/
95
+ │   ├── 20230412
96
+ ├── done/
97
+ │   ├── 20230410
98
+ │   ├── 20230411
99
+ │   └── 20230413
100
+ └── todo/
101
+ ```
102
+
103
+ ### Flexible tasks
104
+
105
+ Note that any filename can be used and the files can be either empty (with the filename
106
+ being used to indicate the nature of the task), or the files can contain data (such as
107
+ JSON or anything else). The `perform` method is free to do whatever is needed to process
108
+ the task and since it's running in it's own process, there is no concern for traditional
109
+ thread concurrency issues, etc.
110
+
111
+ As an example, here is another valid set of tasks that may contain JSON payloads that
112
+ are needed when processing each task.
113
+
114
+ ```
115
+ .vete/
116
+ ├── died/
117
+ ├── done/
118
+ └── todo/
119
+ │   ├── amazon.json
120
+ │   ├── apple.json
121
+ │   ├── facebook.json
122
+ │   └── google.json
123
+ ```
124
+
125
+ ### Additional tips
126
+
127
+ A command line utility (simply called `vete`) can be used to launch a script that
128
+ defines the `perform(task)` method and, optionally, the `setup` method. You can also
129
+ run `vete -r` to remove the entire `.vete` directory.
130
+
131
+ Running `vete -h` provides some additional help:
132
+
133
+ ```shell
134
+ $ vete -h
135
+
136
+ usage: vete [options]
137
+ -b, --bar <width> Progress bar width, in characters
138
+ -c, --char <character> Character to use for progress bar
139
+ -d, --delay <mode> Delay mode (rand, task, numeric)
140
+ -h, --help Show help and command usage
141
+ -r, --reset Remove directory used for job processing and quit
142
+ -v, --version Show version number
143
+ -w, --workers <count> Set the number of workers (default is 1)
144
+ ```
145
+
146
+ Running a `vete` enabled script (ie - one that contains `require "vete"` as the last
147
+ line of the file) will automatically extend the `vete` command line utility. As a result,
148
+ you can run your `vete` enabled script directly and pass any of the above command line
149
+ options, as follows:
150
+
151
+ ```shell
152
+ test/example.rb -w 10
153
+ ```
154
+
155
+ This will run the `example.rb` file (which creates 100 tasks) and it will spawn 10
156
+ concurrent processes to perform the work. See the screencast at the top of this file
157
+ to see how this works.
data/lib/vete.rb CHANGED
@@ -2,7 +2,7 @@
2
2
  # vete - Ruby CLI to spawn processes to get work done
3
3
  #
4
4
  # Author: Steve Shreeve (steve.shreeve@gmail.com)
5
- # Date: June 29, 2023
5
+ # Date: July 1, 2023
6
6
  # ============================================================================
7
7
 
8
8
  STDOUT.sync = true
@@ -19,7 +19,7 @@ trap("INT" ) { print clear + go; abort "\n" }
19
19
  trap("WINCH") { print clear or draw if @pid == Process.pid }
20
20
 
21
21
  OptionParser.new.instance_eval do
22
- @version = "0.6.7"
22
+ @version = "1.0.0"
23
23
  @banner = "usage: #{program_name} [options]"
24
24
 
25
25
  on "-b", "--bar <width>" , "Progress bar width, in characters", Integer
data/test/example.rb CHANGED
@@ -1,15 +1,17 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
3
  def setup
4
- vete_init
5
- 100.times {|i| vete_todo(i + 1) }
6
- @time = Time.now
4
+ vete_retry or begin # retry prior failed tasks, or
5
+ vete_init # initialize the main task directory structure
6
+ 100.times {|i| vete_todo(i + 1) } # create 100 new tasks
7
+ end
8
+ @time = Time.now # instance variables are visible to each task
7
9
  end
8
10
 
9
11
  def perform(task)
10
- sleep rand
11
- secs = Time.now - @time
12
- exit 1 if rand < 0.03
12
+ sleep rand # simulate some work performed
13
+ secs = Time.now - @time # do something with @time (defined in setup)
14
+ exit 1 if rand < 0.03 # simulate a 3% chance of failure
13
15
  end
14
16
 
15
- require_relative "../lib/vete"
17
+ require "vete"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: vete
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.7
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Steve Shreeve
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-06-29 00:00:00.000000000 Z
11
+ date: 2023-07-01 00:00:00.000000000 Z
12
12
  dependencies: []
13
13
  description: Ruby CLI to spawn processes to get work done
14
14
  email: steve.shreeve@gmail.com