thimble 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1fbffaeb62da2fbbb2c0cfa474aa2411b0bc32e32736af38f20e83890921c3af
-  data.tar.gz: 78ce3f45be02b118af41229ecd82ede236e3b8817cb9f7dfe742aad6171a560f
+  metadata.gz: 9d5dba5ce3a256f520a9c75924f886fe3d94fd5e64369528570ef57d7b2e226a
+  data.tar.gz: d99df41c042a2360e4f8c36f8f9b00d4923174abc71b78ce6c35d06fa966cfff
 SHA512:
-  metadata.gz: 8df7d0f0444515437e2d70038d399f85d3732456b299c2ab92f128abc5b7cfbde4f9ec71221c2b1a0c6e76dd9dcfefe91358d27582086ed5135ea9dc4974062a
-  data.tar.gz: 61f615dd222a89da1624f96ea35ca308db5fd2137e59395753e4db2dab6c66917d224667d158525b64574eee18c06156960f99e70b5263dce2746536e257ad56
+  metadata.gz: a9d708451d9b0fe90a133401f4f999930067abd4af81856ad039851f8d1c0bfc44be8da8c99fcfc19eb121927490b0b1283512c8be2ff445c63fcdbd6d80f64d
+  data.tar.gz: e8026d10b4f8d7265c282d59acce8c5e8186aff83b76b75f93f54ad7cceba2f24d428941629d14a3210b467381b531ac31c8b82c2a3ee2f9c6f237ae422ceb15

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Andrew Kovanda
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,126 @@
+# Thimble
+Thimble is a Ruby gem for parallelism and concurrency. It lets you choose threads (good for IO) or processes (good for CPU) and build pipelines using stages backed by a thread-safe queue.
+
+---
+
+## Installation
+Add this line to your application's Gemfile:
+
+```
+gem 'thimble'
+```
+
+And then execute:
+
+```
+bundle install
+```
+
+Or install it yourself as:
+
+```
+gem install thimble
+```
+
+## Supported Ruby and platforms
+- Ruby >= 3.0
+- MRI: threads are limited by the GVL for CPU-bound work. Use `worker_type: :fork` for CPU-bound pipelines.
+- JRuby/TruffleRuby: threads can run in parallel; `:thread` often suffices.
+- Windows: `fork` is not available. Use `worker_type: :thread`.
+
+## Quick start
+
+Example 1: parallel map using forked processes (CPU-bound)
+```
+require 'thimble'
+
+manager = Thimble::Manager.new(max_workers: 5, batch_size: 5, queue_size: 10, worker_type: :fork)
+thimble = Thimble::Thimble.new((1..100).to_a, manager)
+results = thimble.map { |x| x * 1000 }
+# results is a Thimble::ThimbleQueue; consume it as needed
+p results.to_a
+```
+
+Example 2: feed an intermediate queue from a threaded stage (IO-bound)
+```
+require 'thimble'
+# We create a queue to store intermediate work
+queue = Thimble::ThimbleQueue.new(3, 'stage 2')
+# Our array of data
+ary = (1..10).to_a
+# A separate thread worker who will be processing the intermediate queue
+thread = Thimble::Thimble.async do
+  queue.each { |x| puts "I did work on #{x}!"; sleep 1 }
+end
+# Our Thimble, plus its manager. Note we are using Thread in this example.
+thim = Thimble::Thimble.new(ary, Thimble::Manager.new(batch_size: 1, worker_type: :thread))
+# We in parallel push data to the Thimble Queue
+thim.map { |e| queue.push(e); sleep 0.1; puts "I pushed #{e} to the queue!" }
+# The queue is closed (no more work can come in)
+queue.close
+# join the thread
+thread.join
+```
+
+Manager quick reference
+```
+m = Thimble::Manager.new(max_workers: 10, batch_size: 100, worker_type: :fork)
+Thimble::Thimble.new(array, m)
+```
+- max_workers: how many workers can run at the same time
+- batch_size: how many items to send to each worker (tune for workload)
+- worker_type: :thread or :fork
+
+The same Manager can be shared across Thimble instances to coordinate concurrency limits.
+
+All thimbles require an explicit manager.
+
+---
+
+## ThimbleQueue
+ThimbleQueue is the queue underpinning Thimble. Taking from it is destructive. It is thread-safe for multi-thread producers/consumers.
+
+```
+q = Thimble::ThimbleQueue.new(10, 'name')
+q.push(1)
+q.close
+q.each { |x| puts x }
+# => 1
+```
+If you do not close the queue, consumers will wait for more data. Creating a Thimble creates a "closed" input queue; transformations create a new queue.
+
+---
+
+## Caveats and best practices
+These are common pitfalls and how Thimble helps you avoid them:
+
+- MRI GVL and workload choice
+  - Threads do not run CPU-bound Ruby in parallel on MRI. Use `worker_type: :fork` for CPU-bound tasks; `:thread` shines for IO-bound tasks.
+- Platform differences
+  - `fork` is Unix-only. On Windows, use `:thread`.
+- Forking and safety
+  - Thimble forks child workers before creating additional threads inside children. Children trap HUP and exit cleanly; the parent detaches workers to avoid zombies.
+  - Recreate external resources in children (DB connections, sockets, clients). Don’t share them across a fork.
+- Memory and copy-on-write
+  - Each process has its own heap and GC. Batching reduces IPC overhead. Freeze large constants to improve CoW where possible.
+- Backpressure
+  - ThimbleQueue is bounded; tune `queue_size` to avoid unbounded growth.
+- Shutdown
+  - ThimbleQueue supports `close` and `close(true)` for immediate close. Avoid closing from multiple places.
+- Error propagation
+  - Exceptions in workers are propagated back through results. For `:thread`, thread exceptions are surfaced; for `:fork`, exceptions are marshaled back and re-raised when consumed.
+- Signal handling
+  - The main process receives signals; Thimble sends HUP to child workers when their results are consumed.
+- Ordering
+  - Parallel stages may reorder results. If you need original order, attach sequence numbers to items and reorder at the end.
+- Tuning
+  - Start with `max_workers` ~ number of cores for CPU-bound, higher for IO-bound. Adjust `batch_size` to minimize overhead without starving workers.
+
+---
+
+## Development
+- Run tests: `bundle exec rspec`
+- Linting: consider adding RuboCop (`rubocop`)
+- Releasing: bump `Thimble::VERSION` in `lib/thimble/version.rb`, tag and push, then build and push the gem
+
+Contributions welcome! Please open issues and PRs.

data/lib/thimble/version.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module Thimble
+  VERSION = '0.3.0'
+end
+

data/lib/thimble.rb

@@ -3,6 +3,7 @@
 require_relative 'manager'
 require_relative 'thimble_queue'
 require_relative 'queue_item'
+require_relative 'thimble/version'
 require 'io/wait'
 require 'ostruct'
 
@@ -100,10 +101,15 @@ module Thimble
       if @manager.worker_type == :fork
         if tuple.reader.ready?
           piped_result = tuple.reader.read
+          tuple.reader.close unless tuple.reader.closed?
           loadedResult = Marshal.load(piped_result)
           loadedResult.each { |r| raise r if r.class <= Exception }
           push_result(loadedResult)
-          Process.kill('HUP', tuple.pid)
+          begin
+            Process.kill('HUP', tuple.pid)
+          rescue Errno::ESRCH
+            # Process already exited; nothing to do
+          end
           @manager.rem_worker(tuple)
         end
       elsif tuple.done == true

metadata

@@ -1,15 +1,42 @@
 --- !ruby/object:Gem::Specification
 name: thimble
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Andrew Kovanda
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-06-29 00:00:00.000000000 Z
-dependencies: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: ostruct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Thimble is a ruby gem for parallelism and concurrency. It allows you
   to decide if you want to use separate processes, or if you want to use threads in
   ruby. It allows you to create stages with a thread safe queue, and break apart large
@@ -19,15 +46,22 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- LICENSE
+- README.md
 - lib/manager.rb
 - lib/queue_item.rb
 - lib/thimble.rb
+- lib/thimble/version.rb
 - lib/thimble_queue.rb
 homepage: https://github.com/akovanda/thimble
 licenses:
 - MIT
-metadata: {}
-post_install_message:
+metadata:
+  source_code_uri: https://github.com/akovanda/thimble
+  bug_tracker_uri: https://github.com/akovanda/thimble/issues
+  changelog_uri: https://github.com/akovanda/thimble/releases
+  documentation_uri: https://github.com/akovanda/thimble#readme
+  rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths:
 - lib
@@ -35,15 +69,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 3.0.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.7
-signing_key:
+rubygems_version: 3.6.7
 specification_version: 4
 summary: Concurrency and Parallelism gem that uses blocks to move data
 test_files: []