vete 0.6.8 → 1.0.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (5) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +146 -0
  3. data/lib/vete.rb +4 -7
  4. data/test/example.rb +7 -7
  5. metadata +1 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6f5835e7087158f60f5a7942503ed0228b740438c071350496bb15dfe870f19b
4
- data.tar.gz: 3e83bcb77d36963b471b66773799d04976440bf46b95d1664448ca95a0590eaa
3
+ metadata.gz: 17a743a15d0924c27b9d12b013832b086a4fbd1b7ef1e4fce78fc2d6a144a3c9
4
+ data.tar.gz: a406a4f2dbd83b256fb2403421c9abcbc4c3a3c49a7a39348630521fc6c2fa17
5
5
  SHA512:
6
- metadata.gz: fab1349a4aff1b601172f465b9f1a4b03ae2cc6d3f77c9f62390c174134f62ba51a8722eab052217800fa3070aae9d646011353c9d8a73d2335857ddd69520e5
7
- data.tar.gz: a5198676ce76cdcc22553d8531ae0b3ea61aac18639cf08b4083a9e2d4c4d98ce9b16eb56541cf2905f6c9d799f255d9c472d5312e8450c627acb65f9f17e231
6
+ metadata.gz: 440ea4d9a82ddbe830da97150a1752ca0f3624955db89cf9c3782b1546641cc19ba1bd8c165eaf435520ce105d87f9f65b5877917bba48dda48d762d3c35760e
7
+ data.tar.gz: 1cc41d58721becaf76e6980527e801658431af3ca0a3100bccc53a92ea9d33ce2a23afb3873fd17f3a6eec4533247d23d16476191401bf54361e9d773e3a0b3d
data/README.md CHANGED
@@ -4,8 +4,154 @@ Ruby CLI to spawn processes to get work done
4
4
 
5
5
  The phrase "¡véte!" in Spanish means, basically, "Get out!". This tool helps to clear out work in a hurry, using a simple approach of spawning a set number of concurrent processes to handle each job. Jobs are defined as files in a directory, so there is no need for a database or any other complexity.
6
6
 
7
+ ### Summary
8
+
9
+ To use `vete`, there are three steps:
10
+
11
+ 1. Define a method called `setup` which sets up a context for each task
12
+ 2. Define a method called `perform(task)` which is invoked for each task
13
+ 3. At the end of your script, trigger everything with `require "vete"`
14
+
15
+ When your script executes, the `setup` method is called once. Its purpose is to
16
+ initialize a context that all subsequent tasks will inherit. It also is where new
17
+ tasks are defined or prior failed tasks can be prepared to be retried. Instance
18
+ variables and other context defined in the `setup` method is available to each task.
19
+
20
+ Once the `setup` method has been called, a configurable number of worker processes
21
+ will be spawned in parallel. Each worker will immediately call `perform(task)`. Since
22
+ each process inherits the context defined by the `setup` method, memory is efficiently
23
+ shared. As tasks are executed, a progress bar will indicate the overall completion status.
24
+
7
25
  ### Example
8
26
 
9
27
  Running the `test/example.rb` script with 10 workers:
10
28
 
11
29
  ![Example](https://raw.githubusercontent.com/shreeve/vete/main/test/vete.gif)
30
+
31
+ Here is the code for the above:
32
+
33
+ ```ruby
34
+ #!/usr/bin/env ruby
35
+
36
+ def setup
37
+ vete_retry or begin # retry prior failed tasks, or
38
+ vete_init # initialize the main task directory structure
39
+ 100.times {|i| vete_todo(i + 1) } # create 100 new tasks
40
+ end
41
+ @time = Time.now # instance variables are visible to each task
42
+ end
43
+
44
+ def perform(task)
45
+ sleep rand # simulate some work performed
46
+ secs = Time.now - @time # do something with @time (defined in setup)
47
+ exit 1 if rand < 0.03 # simulate a 3% chance of failure
48
+ end
49
+
50
+ require "vete"
51
+ ```
52
+
53
+ ### Inner workings
54
+
55
+ ```
56
+ .vete/
57
+ ├── died/
58
+ ├── done/
59
+ └── todo/
60
+ ```
61
+
62
+ The above directory structure is used by `vete` to define tasks and to process
63
+ their lifecycle. Tasks are defined as files in the `.vete/todo` directory. For example,
64
+ if we needed to pull down a report for four days in April 2023, we may define these
65
+ four tasks as follows:
66
+
67
+ ```
68
+ .vete/
69
+ ├── died/
70
+ ├── done/
71
+ └── todo/
72
+ │   ├── 20230410
73
+ │   ├── 20230411
74
+ │   ├── 20230412
75
+ │   └── 20230413
76
+ ```
77
+
78
+ This file structure can be defined in the `setup` method, or you could choose to
79
+ manually create the files any other way.
80
+
81
+ When `vete` is launched by the `require "vete"` line in the script, it will call
82
+ the `setup` script (if it is defined). Then, it will look for files in the `.vete/todo`
83
+ directory. The desired number of worker processes is then launched in parallel, each
84
+ time calling `perform(task)` with `task` being the full pathname of the next file in the
85
+ `todo` directory.
86
+
87
+ If `perform(task)` executes without any error, then the file for that task will be moved
88
+ to the `done` directory. If errors occur, the file is moved to the `died` directory.
89
+ Suppose that three of the tasks above successfully completed, but one failed. This would
90
+ yield the following file structure:
91
+
92
+ ```
93
+ .vete/
94
+ ├── died/
95
+ │   ├── 20230412
96
+ ├── done/
97
+ │   ├── 20230410
98
+ │   ├── 20230411
99
+ │   └── 20230413
100
+ └── todo/
101
+ ```
102
+
103
+ ### Flexible tasks
104
+
105
+ Note that any filename can be used and the files can be either empty (with the filename
106
+ being used to indicate the nature of the task), or the files can contain data (such as
107
+ JSON or anything else). The `perform` method is free to do whatever is needed to process
108
+ the task and since it's running in it's own process, there is no concern for traditional
109
+ thread concurrency issues, etc.
110
+
111
+ As an example, here is another valid set of tasks that may contain JSON payloads that
112
+ are needed when processing each task.
113
+
114
+ ```
115
+ .vete/
116
+ ├── died/
117
+ ├── done/
118
+ └── todo/
119
+ │   ├── amazon.json
120
+ │   ├── apple.json
121
+ │   ├── facebook.json
122
+ │   └── google.json
123
+ ```
124
+
125
+ ### Additional tips
126
+
127
+ A command line utility (simply called `vete`) can be used to launch a script that
128
+ defines the `perform(task)` method and, optionally, the `setup` method. You can also
129
+ run `vete -r` to remove the entire `.vete` directory.
130
+
131
+ Running `vete -h` provides some additional help:
132
+
133
+ ```text
134
+ $ vete -h
135
+
136
+ usage: vete [options]
137
+ -b, --bar <width> Progress bar width, in characters
138
+ -c, --char <character> Character to use for progress bar
139
+ -d, --delay <mode> Delay mode (rand, task, numeric)
140
+ -h, --help Show help and command usage
141
+ -r, --reset Remove directory used for job processing and quit
142
+ -v, --version Show version number
143
+ -w, --workers <count> Set the number of workers (default is 1)
144
+ ```
145
+
146
+ Running a `vete` enabled script (ie - one that contains `require "vete"` as the last
147
+ line of the file) will automatically extend the `vete` command line utility. As a result,
148
+ you can run your `vete` enabled script directly and pass any of the above command line
149
+ options, as follows:
150
+
151
+ ```shell
152
+ test/example.rb -w 10
153
+ ```
154
+
155
+ This will run the `example.rb` file (which creates 100 tasks) and it will spawn 10
156
+ concurrent processes to perform the work. See the screencast at the top of this file
157
+ to see how this works.
data/lib/vete.rb CHANGED
@@ -2,7 +2,7 @@
2
2
  # vete - Ruby CLI to spawn processes to get work done
3
3
  #
4
4
  # Author: Steve Shreeve (steve.shreeve@gmail.com)
5
- # Date: June 29, 2023
5
+ # Date: July 1, 2023
6
6
  # ============================================================================
7
7
 
8
8
  STDOUT.sync = true
@@ -19,7 +19,7 @@ trap("INT" ) { print clear + go; abort "\n" }
19
19
  trap("WINCH") { print clear or draw if @pid == Process.pid }
20
20
 
21
21
  OptionParser.new.instance_eval do
22
- @version = "0.6.8"
22
+ @version = "1.0.1"
23
23
  @banner = "usage: #{program_name} [options]"
24
24
 
25
25
  on "-b", "--bar <width>" , "Progress bar width, in characters", Integer
@@ -157,11 +157,8 @@ end
157
157
  @mtx = Mutex.new
158
158
  @que = Thread::Queue.new; @work.times {|slot| @que << (slot + 1) }
159
159
 
160
- begin
161
- setup if defined?(setup)
162
-
163
- list = Dir[File.join(@todo, "*")]
164
-
160
+ defined?(setup ) and setup
161
+ defined?(perform) and list = Dir[File.join(@todo, "*")] and !list.empty? and begin
165
162
  live = 0
166
163
  done = 0
167
164
  died = 0
data/test/example.rb CHANGED
@@ -1,17 +1,17 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
3
  def setup
4
- vete_retry or begin
5
- vete_init
6
- 100.times {|i| vete_todo(i + 1) }
4
+ vete_retry or begin # retry prior failed tasks, or
5
+ vete_init # initialize the main task directory structure
6
+ 100.times {|i| vete_todo(i + 1) } # create 100 new tasks
7
7
  end
8
- @time = Time.now
8
+ @time = Time.now # instance variables are visible to each task
9
9
  end
10
10
 
11
11
  def perform(task)
12
- sleep rand
13
- secs = Time.now - @time # @time defined in setup
14
- exit 1 if rand < 0.08 # 8% chance of failure
12
+ sleep rand # simulate some work performed
13
+ secs = Time.now - @time # do something with @time (defined in setup)
14
+ exit 1 if rand < 0.03 # simulate a 3% chance of failure
15
15
  end
16
16
 
17
17
  require "vete"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: vete
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.8
4
+ version: 1.0.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Steve Shreeve