vete 0.6.8 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6f5835e7087158f60f5a7942503ed0228b740438c071350496bb15dfe870f19b
-  data.tar.gz: 3e83bcb77d36963b471b66773799d04976440bf46b95d1664448ca95a0590eaa
+  metadata.gz: b13f06ca7363e892403446d521839c18bbdd16ca832a9af2fdab9ed093703135
+  data.tar.gz: d17273c1de3f1b6ea4455ea9293d009d2353408121b87e8f5586ee3532bbca3c
 SHA512:
-  metadata.gz: fab1349a4aff1b601172f465b9f1a4b03ae2cc6d3f77c9f62390c174134f62ba51a8722eab052217800fa3070aae9d646011353c9d8a73d2335857ddd69520e5
-  data.tar.gz: a5198676ce76cdcc22553d8531ae0b3ea61aac18639cf08b4083a9e2d4c4d98ce9b16eb56541cf2905f6c9d799f255d9c472d5312e8450c627acb65f9f17e231
+  metadata.gz: c5a9083c1193e9338bf943df433eb8933d0247ef587338511d163a595403379f7cd06449f6e2eff57c374984a7fd4dc8d5dc5b27ccc9c1fa1cb426792c10c9de
+  data.tar.gz: aa56000561c727f918553b6ccf436cd3394decccf5473e8f6c1b4a754e24acab4e421116da13c509be61b9913499be0fc25d8799c85a0809fb5565830b31f653

data/README.md

@@ -4,8 +4,154 @@ Ruby CLI to spawn processes to get work done
 
 The phrase "¡véte!" in Spanish means, basically, "Get out!". This tool helps to clear out work in a hurry, using a simple approach of spawning a set number of concurrent processes to handle each job. Jobs are defined as files in a directory, so there is no need for a database or any other complexity.
 
+### Summary
+
+To use `vete`, there are three steps:
+
+1. Define a method called `setup` which sets up a context for each task
+2. Define a method called `perform(task)` which is invoked for each task
+3. At the end of your script, trigger everything with `require "vete"`
+
+When your script executes, the `setup` method is called once. Its purpose is to
+initialize a context that all subsequent tasks will inherit. It also is where new
+tasks are defined or prior failed tasks can be prepared to be retried. Instance
+variables and other context defined in the `setup` method is available to each task.
+
+Once the `setup` method has been called, a configurable number of worker processes
+will be spawned in parallel. Each worker will immediately call `perform(task)`. Since
+each process inherits the context defined by the `setup` method, memory is efficiently
+shared. As tasks are executed, a progress bar will indicate the overall completion status.
+
 ### Example
 
 Running the `test/example.rb` script with 10 workers:
 
 ![Example](https://raw.githubusercontent.com/shreeve/vete/main/test/vete.gif)
+
+Here is the code for the above:
+
+```ruby
+#!/usr/bin/env ruby
+
+def setup
+  vete_retry or begin # retry prior failed tasks, or
+    vete_init # initialize the main task directory structure
+    100.times {|i| vete_todo(i + 1) } # create 100 new tasks
+  end
+  @time = Time.now # instance variables are visible to each task
+end
+
+def perform(task)
+  sleep rand # simulate some work performed
+  secs = Time.now - @time # do something with @time (defined in setup)
+  exit 1 if rand < 0.03 # simulate a 3% chance of failure
+end
+
+require "vete"
+```
+
+### Inner workings
+
+```
+  .vete/
+  ├── died/
+  ├── done/
+  └── todo/
+```
+
+The above directory structure is used by `vete` to define tasks and to process
+their lifecycle. Tasks are defined as files in the `.vete/todo` directory. For example,
+if we needed to pull down a report for four days in April 2023, we may define these
+four tasks as follows:
+
+```
+  .vete/
+  ├── died/
+  ├── done/
+  └── todo/
+  │   ├── 20230410
+  │   ├── 20230411
+  │   ├── 20230412
+  │   └── 20230413
+```
+
+This file structure can be defined in the `setup` method, or you could choose to
+manually create the files any other way.
+
+When `vete` is launched by the `require "vete"` line in the script, it will call
+the `setup` script (if it is defined). Then, it will look for files in the `.vete/todo`
+directory. The desired number of worker processes is then launched in parallel, each
+time calling `perform(task)` with `task` being the full pathname of the next file in the
+`todo` directory.
+
+If `perform(task)` executes without any error, then the file for that task will be moved
+to the `done` directory. If errors occur, the file is moved to the `died` directory.
+Suppose that three of the tasks above successfully completed, but one failed. This would
+yield the following file structure:
+
+```
+  .vete/
+  ├── died/
+  │   ├── 20230412
+  ├── done/
+  │   ├── 20230410
+  │   ├── 20230411
+  │   └── 20230413
+  └── todo/
+```
+
+### Flexible tasks
+
+Note that any filename can be used and the files can be either empty (with the filename
+being used to indicate the nature of the task), or the files can contain data (such as
+JSON or anything else). The `perform` method is free to do whatever is needed to process
+the task and since it's running in it's own process, there is no concern for traditional
+thread concurrency issues, etc.
+
+As an example, here is another valid set of tasks that may contain JSON payloads that
+are needed when processing each task.
+
+```
+  .vete/
+  ├── died/
+  ├── done/
+  └── todo/
+  │   ├── amazon.json
+  │   ├── apple.json
+  │   ├── facebook.json
+  │   └── google.json
+```
+
+### Additional tips
+
+A command line utility (simply called `vete`) can be used to launch a script that
+defines the `perform(task)` method and, optionally, the `setup` method. You can also
+run `vete -r` to remove the entire `.vete` directory.
+
+Running `vete -h` provides some additional help:
+
+```shell
+$ vete -h
+
+usage: vete [options]
+    -b, --bar <width>                Progress bar width, in characters
+    -c, --char <character>           Character to use for progress bar
+    -d, --delay <mode>               Delay mode (rand, task, numeric)
+    -h, --help                       Show help and command usage
+    -r, --reset                      Remove directory used for job processing and quit
+    -v, --version                    Show version number
+    -w, --workers <count>            Set the number of workers (default is 1)
+```
+
+Running a `vete` enabled script (ie - one that contains `require "vete"` as the last
+line of the file) will automatically extend the `vete` command line utility. As a result,
+you can run your `vete` enabled script directly and pass any of the above command line
+options, as follows:
+
+```shell
+test/example.rb -w 10
+```
+
+This will run the `example.rb` file (which creates 100 tasks) and it will spawn 10
+concurrent processes to perform the work. See the screencast at the top of this file
+to see how this works.

data/lib/vete.rb

@@ -2,7 +2,7 @@
 # vete - Ruby CLI to spawn processes to get work done
 #
 # Author: Steve Shreeve (steve.shreeve@gmail.com)
-#   Date: June 29, 2023
+#   Date: July 1, 2023
 # ============================================================================
 
 STDOUT.sync = true
@@ -19,7 +19,7 @@ trap("INT"  ) { print clear + go; abort "\n" }
 trap("WINCH") { print clear or draw if @pid == Process.pid }
 
 OptionParser.new.instance_eval do
-  @version = "0.6.8"
+  @version = "1.0.0"
   @banner  = "usage: #{program_name} [options]"
 
   on "-b", "--bar <width>"            , "Progress bar width, in characters", Integer

data/test/example.rb

@@ -1,17 +1,17 @@
 #!/usr/bin/env ruby
 
 def setup
-  vete_retry or begin
-    vete_init
-    100.times {|i| vete_todo(i + 1) }
+  vete_retry or begin # retry prior failed tasks, or
+    vete_init # initialize the main task directory structure
+    100.times {|i| vete_todo(i + 1) } # create 100 new tasks
   end
-  @time = Time.now
+  @time = Time.now # instance variables are visible to each task
 end
 
 def perform(task)
-  sleep rand
-  secs = Time.now - @time # @time defined in setup
-  exit 1 if rand < 0.08 # 8% chance of failure
+  sleep rand # simulate some work performed
+  secs = Time.now - @time # do something with @time (defined in setup)
+  exit 1 if rand < 0.03 # simulate a 3% chance of failure
 end
 
 require "vete"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: vete
 version: !ruby/object:Gem::Version
-  version: 0.6.8
+  version: 1.0.0
 platform: ruby
 authors:
 - Steve Shreeve