vete 0.6.7 → 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (5) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +146 -0
  3. data/lib/vete.rb +2 -2
  4. data/test/example.rb +9 -7
  5. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 96b64fc68cb1332add0e65d46131d363ac1723c9b70f0f814382c29efeecb65f
4
- data.tar.gz: 2601b593a3ca14b98576cde84de8dc40abd79cb3a6243784099ec8e4298baf7d
3
+ metadata.gz: b13f06ca7363e892403446d521839c18bbdd16ca832a9af2fdab9ed093703135
4
+ data.tar.gz: d17273c1de3f1b6ea4455ea9293d009d2353408121b87e8f5586ee3532bbca3c
5
5
  SHA512:
6
- metadata.gz: fb245248250c901de9ca611e4aa424fb7376fae97a96a71f448879cda58852632c6d7f98437ec271431ffa23cb8fc66fbcd09254850192480e3fe8d16032c7f7
7
- data.tar.gz: c9e88e46d546b8ea63ef92f71b2684db2c114e1303cac1380a7e43a574d3bd45fb79a01ba53f8909d1dbef8e893506b7f00dfc5be1f40549f5fd62ae01788faa
6
+ metadata.gz: c5a9083c1193e9338bf943df433eb8933d0247ef587338511d163a595403379f7cd06449f6e2eff57c374984a7fd4dc8d5dc5b27ccc9c1fa1cb426792c10c9de
7
+ data.tar.gz: aa56000561c727f918553b6ccf436cd3394decccf5473e8f6c1b4a754e24acab4e421116da13c509be61b9913499be0fc25d8799c85a0809fb5565830b31f653
data/README.md CHANGED
@@ -4,8 +4,154 @@ Ruby CLI to spawn processes to get work done
4
4
 
5
5
  The phrase "¡véte!" in Spanish means, basically, "Get out!". This tool helps to clear out work in a hurry, using a simple approach of spawning a set number of concurrent processes to handle each job. Jobs are defined as files in a directory, so there is no need for a database or any other complexity.
6
6
 
7
+ ### Summary
8
+
9
+ To use `vete`, there are three steps:
10
+
11
+ 1. Define a method called `setup` which sets up a context for each task
12
+ 2. Define a method called `perform(task)` which is invoked for each task
13
+ 3. At the end of your script, trigger everything with `require "vete"`
14
+
15
+ When your script executes, the `setup` method is called once. Its purpose is to
16
+ initialize a context that all subsequent tasks will inherit. It also is where new
17
+ tasks are defined or prior failed tasks can be prepared to be retried. Instance
18
+ variables and other context defined in the `setup` method is available to each task.
19
+
20
+ Once the `setup` method has been called, a configurable number of worker processes
21
+ will be spawned in parallel. Each worker will immediately call `perform(task)`. Since
22
+ each process inherits the context defined by the `setup` method, memory is efficiently
23
+ shared. As tasks are executed, a progress bar will indicate the overall completion status.
24
+
7
25
  ### Example
8
26
 
9
27
  Running the `test/example.rb` script with 10 workers:
10
28
 
11
29
  ![Example](https://raw.githubusercontent.com/shreeve/vete/main/test/vete.gif)
30
+
31
+ Here is the code for the above:
32
+
33
+ ```ruby
34
+ #!/usr/bin/env ruby
35
+
36
+ def setup
37
+ vete_retry or begin # retry prior failed tasks, or
38
+ vete_init # initialize the main task directory structure
39
+ 100.times {|i| vete_todo(i + 1) } # create 100 new tasks
40
+ end
41
+ @time = Time.now # instance variables are visible to each task
42
+ end
43
+
44
+ def perform(task)
45
+ sleep rand # simulate some work performed
46
+ secs = Time.now - @time # do something with @time (defined in setup)
47
+ exit 1 if rand < 0.03 # simulate a 3% chance of failure
48
+ end
49
+
50
+ require "vete"
51
+ ```
52
+
53
+ ### Inner workings
54
+
55
+ ```
56
+ .vete/
57
+ ├── died/
58
+ ├── done/
59
+ └── todo/
60
+ ```
61
+
62
+ The above directory structure is used by `vete` to define tasks and to process
63
+ their lifecycle. Tasks are defined as files in the `.vete/todo` directory. For example,
64
+ if we needed to pull down a report for four days in April 2023, we may define these
65
+ four tasks as follows:
66
+
67
+ ```
68
+ .vete/
69
+ ├── died/
70
+ ├── done/
71
+ └── todo/
72
+ │   ├── 20230410
73
+ │   ├── 20230411
74
+ │   ├── 20230412
75
+ │   └── 20230413
76
+ ```
77
+
78
+ This file structure can be defined in the `setup` method, or you could choose to
79
+ manually create the files any other way.
80
+
81
+ When `vete` is launched by the `require "vete"` line in the script, it will call
82
+ the `setup` script (if it is defined). Then, it will look for files in the `.vete/todo`
83
+ directory. The desired number of worker processes is then launched in parallel, each
84
+ time calling `perform(task)` with `task` being the full pathname of the next file in the
85
+ `todo` directory.
86
+
87
+ If `perform(task)` executes without any error, then the file for that task will be moved
88
+ to the `done` directory. If errors occur, the file is moved to the `died` directory.
89
+ Suppose that three of the tasks above successfully completed, but one failed. This would
90
+ yield the following file structure:
91
+
92
+ ```
93
+ .vete/
94
+ ├── died/
95
+ │   ├── 20230412
96
+ ├── done/
97
+ │   ├── 20230410
98
+ │   ├── 20230411
99
+ │   └── 20230413
100
+ └── todo/
101
+ ```
102
+
103
+ ### Flexible tasks
104
+
105
+ Note that any filename can be used and the files can be either empty (with the filename
106
+ being used to indicate the nature of the task), or the files can contain data (such as
107
+ JSON or anything else). The `perform` method is free to do whatever is needed to process
108
+ the task and since it's running in it's own process, there is no concern for traditional
109
+ thread concurrency issues, etc.
110
+
111
+ As an example, here is another valid set of tasks that may contain JSON payloads that
112
+ are needed when processing each task.
113
+
114
+ ```
115
+ .vete/
116
+ ├── died/
117
+ ├── done/
118
+ └── todo/
119
+ │   ├── amazon.json
120
+ │   ├── apple.json
121
+ │   ├── facebook.json
122
+ │   └── google.json
123
+ ```
124
+
125
+ ### Additional tips
126
+
127
+ A command line utility (simply called `vete`) can be used to launch a script that
128
+ defines the `perform(task)` method and, optionally, the `setup` method. You can also
129
+ run `vete -r` to remove the entire `.vete` directory.
130
+
131
+ Running `vete -h` provides some additional help:
132
+
133
+ ```shell
134
+ $ vete -h
135
+
136
+ usage: vete [options]
137
+ -b, --bar <width> Progress bar width, in characters
138
+ -c, --char <character> Character to use for progress bar
139
+ -d, --delay <mode> Delay mode (rand, task, numeric)
140
+ -h, --help Show help and command usage
141
+ -r, --reset Remove directory used for job processing and quit
142
+ -v, --version Show version number
143
+ -w, --workers <count> Set the number of workers (default is 1)
144
+ ```
145
+
146
+ Running a `vete` enabled script (ie - one that contains `require "vete"` as the last
147
+ line of the file) will automatically extend the `vete` command line utility. As a result,
148
+ you can run your `vete` enabled script directly and pass any of the above command line
149
+ options, as follows:
150
+
151
+ ```shell
152
+ test/example.rb -w 10
153
+ ```
154
+
155
+ This will run the `example.rb` file (which creates 100 tasks) and it will spawn 10
156
+ concurrent processes to perform the work. See the screencast at the top of this file
157
+ to see how this works.
data/lib/vete.rb CHANGED
@@ -2,7 +2,7 @@
2
2
  # vete - Ruby CLI to spawn processes to get work done
3
3
  #
4
4
  # Author: Steve Shreeve (steve.shreeve@gmail.com)
5
- # Date: June 29, 2023
5
+ # Date: July 1, 2023
6
6
  # ============================================================================
7
7
 
8
8
  STDOUT.sync = true
@@ -19,7 +19,7 @@ trap("INT" ) { print clear + go; abort "\n" }
19
19
  trap("WINCH") { print clear or draw if @pid == Process.pid }
20
20
 
21
21
  OptionParser.new.instance_eval do
22
- @version = "0.6.7"
22
+ @version = "1.0.0"
23
23
  @banner = "usage: #{program_name} [options]"
24
24
 
25
25
  on "-b", "--bar <width>" , "Progress bar width, in characters", Integer
data/test/example.rb CHANGED
@@ -1,15 +1,17 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
3
  def setup
4
- vete_init
5
- 100.times {|i| vete_todo(i + 1) }
6
- @time = Time.now
4
+ vete_retry or begin # retry prior failed tasks, or
5
+ vete_init # initialize the main task directory structure
6
+ 100.times {|i| vete_todo(i + 1) } # create 100 new tasks
7
+ end
8
+ @time = Time.now # instance variables are visible to each task
7
9
  end
8
10
 
9
11
  def perform(task)
10
- sleep rand
11
- secs = Time.now - @time
12
- exit 1 if rand < 0.03
12
+ sleep rand # simulate some work performed
13
+ secs = Time.now - @time # do something with @time (defined in setup)
14
+ exit 1 if rand < 0.03 # simulate a 3% chance of failure
13
15
  end
14
16
 
15
- require_relative "../lib/vete"
17
+ require "vete"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: vete
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.7
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Steve Shreeve
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-06-29 00:00:00.000000000 Z
11
+ date: 2023-07-01 00:00:00.000000000 Z
12
12
  dependencies: []
13
13
  description: Ruby CLI to spawn processes to get work done
14
14
  email: steve.shreeve@gmail.com