specwrk 0.19.2 → 0.19.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: befb2589fe879bbe901360402374acb906359ff133c877a9169bcb0f928d1833
-  data.tar.gz: 30c49a123a026d5a4af2da4fadf01cfc92637b095328c83c660d5a3ee7c26d43
+  metadata.gz: ed4bee81f0d4eef75556d4987bfa36101083bd1ee124770857f76ea2827ad036
+  data.tar.gz: e33ddadc340adf10fc4d66bd36598ad3322b27e5cb37dd9e0f32c0d74c65c305
 SHA512:
-  metadata.gz: 9e997305868b8cca5ce07ea2285462ed26b01079ccebe09617749ae76c42380787b68ed693af0df4aa6cf1e196b941c5449ae7034cf21614b16650c73de4fc6b
-  data.tar.gz: 21bb6a0af32806ed771f0e49ac0295e87ac6f57afb6259b5d74e09971adc5be23bdb1a5bcfb6762578f3f85b0387478d23767f1068e1fc53dd0dd2a63e84bbd3
+  metadata.gz: 2ce4eeb7ca55eae597425b10eb3a1236ae8b04fa6bae13f3564d761a88eb138671c1ffae85cf89f7944fb64c463f9ff96b261e9b650df3dbf34675237d91df7d
+  data.tar.gz: 81b3f08f0870b9d7af23476ad2282793e2891c42fa110d18e5f5ae1b567c843eaa945b125fc16d4af72a1b124f5f3e8d5bd214bd8e7804222f6f7c6a1d613649

data/CHANGELOG.md

@@ -1,7 +1,23 @@
 # Changelog
 
 ## Unreleased  
-[Compare](https://github.com/danielwestendorf/specwrk/compare/v0.19.1...main)  
+[Compare](https://github.com/danielwestendorf/specwrk/compare/v0.19.4...main)  
+
+## v0.19.4 — 2026-08-23  
+[Compare](https://github.com/danielwestendorf/specwrk/compare/v0.19.3...v0.19.4)  
+- Return HTTP 423 on lock contention and retry clients with jitter — [#189](https://github.com/danielwestendorf/specwrk/pull/189) by [@danielwestendorf](https://github.com/danielwestendorf)  
+- Document the gem and Docker release process — [#190](https://github.com/danielwestendorf/specwrk/pull/190) by [@danielwestendorf](https://github.com/danielwestendorf)  
+
+## v0.19.3 — 2026-07-26  
+[Compare](https://github.com/danielwestendorf/specwrk/compare/v0.19.2...v0.19.3)  
+- Retry transient file store read descriptor errors — [@danielwestendorf](https://github.com/danielwestendorf)  
+- Complete omitted worker processing examples as failures — [@danielwestendorf](https://github.com/danielwestendorf), [@benjaminwood](https://github.com/benjaminwood)  
+
+
+## v0.19.2 — 2025-12-09  
+[Compare](https://github.com/danielwestendorf/specwrk/compare/v0.19.1...v0.19.2)  
+- Optimize msgpack usage by using one packer/unpacker — [#180](https://github.com/danielwestendorf/specwrk/pull/180) by [@danielwestendorf](https://github.com/danielwestendorf)  
+- Standard style to v3.4 — [#179](https://github.com/danielwestendorf/specwrk/pull/179) by [@danielwestendorf](https://github.com/danielwestendorf)  
 - Add msgpack to the docker server — [#178](https://github.com/danielwestendorf/specwrk/pull/178) by [@danielwestendorf](https://github.com/danielwestendorf)  
 
 ## v0.19.1 — 2025-12-08  

data/README.md

@@ -292,6 +292,8 @@ specwrk is different because it:
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/danielwestendorf/specwrk.
 
+Maintainers can follow [RELEASING.md](RELEASING.md) to publish the gem and Docker image.
+
 ## License
 
 The gem is available as open source under the terms of the [LGPLv3 License](http://www.gnu.org/licenses/lgpl-3.0.html).

data/RELEASING.md

@@ -0,0 +1,126 @@
+# Releasing Specwrk
+
+Releases are cut from `main` in two stages:
+
+1. `gem bump` updates the version and creates the release commit.
+2. Bundler's `rake release` task tags the commit, pushes Git, and publishes the gem.
+
+The Docker image is published separately after the gem release.
+
+## One-time setup
+
+Install `gem-release`, which provides the `gem bump` command. It is intentionally
+not a development dependency because it is only needed by maintainers cutting a
+release.
+
+```sh
+gem install gem-release
+```
+
+Configure credentials for RubyGems and Docker Hub:
+
+```sh
+gem signin
+docker login docker.io
+```
+
+The Docker release also requires a buildx builder capable of building `linux/arm64`
+and `linux/amd64` images. Check the active builder with:
+
+```sh
+docker buildx inspect --bootstrap
+```
+
+## Cut the gem release
+
+Start from an up-to-date, clean `main` branch. This is important because
+`rake release` pushes the current branch.
+
+```sh
+git switch main
+git pull --ff-only origin main
+git status --short
+```
+
+Bump the version without committing. Use `patch`, `minor`, `major`, or an
+explicit version number:
+
+```sh
+gem bump --version patch --no-commit
+```
+
+This updates `lib/specwrk/version.rb` while leaving the release changes
+uncommitted. Update `CHANGELOG.md` using the new version:
+
+1. Change the Unreleased comparison link to start at the new version.
+2. Add a dated section for the new version.
+3. Move the Unreleased entries into that section.
+
+Stage the version, changelog, and any release documentation, then create the
+version commit:
+
+```sh
+git add lib/specwrk/version.rb CHANGELOG.md RELEASING.md
+git commit -m "Bump specwrk to VERSION"
+```
+
+Replace `VERSION` with the new version, for example `0.19.4`. Confirm that the
+commit contains the version and changelog updates:
+
+```sh
+git show --stat --oneline HEAD
+git status --short
+```
+
+Run the full checks:
+
+```sh
+bundle exec rake
+```
+
+Publish the release:
+
+```sh
+bundle exec rake release
+```
+
+The release task requires a clean tracked worktree and then:
+
+1. Builds `pkg/specwrk-VERSION.gem`.
+2. Creates the annotated `vVERSION` tag.
+3. Pushes `main` and the tag to `origin`.
+4. Pushes the gem to RubyGems.org.
+
+Use `gem bump` only to update the version file; use `bundle exec rake release`,
+not `gem release`, for publishing this project.
+
+If the RubyGems push fails after the tag was pushed, fix the authentication or
+network problem and rerun `bundle exec rake release`. The task recognizes the
+existing tag and retries the gem publication.
+
+## Build and publish the Docker image
+
+After the gem release succeeds, run:
+
+```sh
+scripts/build-server
+```
+
+The script reads the version from `bundle exec exe/specwrk --version`, builds the
+gem from the current checkout, and runs a multi-platform `docker buildx build`
+with `--push`. It publishes both of these tags:
+
+- `docker.io/danielwestendorf/specwrk-server:VERSION`
+- `docker.io/danielwestendorf/specwrk-server:latest`
+
+It builds images for both `linux/arm64` and `linux/amd64`. The temporary gem in
+the repository root is removed after a successful build.
+
+Verify the published manifest with:
+
+```sh
+docker buildx imagetools inspect docker.io/danielwestendorf/specwrk-server:VERSION
+```
+
+Replace `VERSION` with the released version without the `v` prefix, for example
+`0.19.4`.

data/lib/specwrk/cli.rb

@@ -67,7 +67,7 @@ module Specwrk
 
       def worker_env_for(idx)
         {
-          "TEST_ENV_NUMBER" => idx.to_s,
+          "TEST_ENV_NUMBER" => (idx == 1) ? "" : idx.to_s,
           "SPECWRK_FORKED" => idx.to_s,
           "SPECWRK_ID" => "#{ENV.fetch("SPECWRK_ID", "specwrk-worker")}-#{idx}"
         }

data/lib/specwrk/client.rb

@@ -12,6 +12,8 @@ Specwrk.net_http = Net::HTTP
 
 module Specwrk
   class Client
+    LockedResponseError = Class.new(StandardError)
+
     def self.connect?
       http = build_http
       http.start
@@ -158,10 +160,14 @@ module Specwrk
         @last_request_at = Time.now
         @http.request(request).tap do |response|
           @retry_count = 0
+          raise LockedResponseError if response.code == "423"
 
           @worker_status = response["x-specwrk-status"].to_i if response["x-specwrk-status"]
         end
       end
+    rescue LockedResponseError
+      sleep rand
+      retry
     rescue Net::ReadTimeout, Net::WriteTimeout => e
       @retry_count ||= 0
 

data/lib/specwrk/store/base.rb

@@ -4,6 +4,8 @@ require "uri"
 
 module Specwrk
   class Store
+    LockUnavailableError = Class.new(StandardError)
+
     class << self
       def with_lock(uri, key)
         adapter_klass(uri).with_lock(uri, key) { yield }

data/lib/specwrk/store/file_adapter.rb

@@ -10,6 +10,9 @@ require "specwrk/store/base_adapter"
 module Specwrk
   class Store
     class FileAdapter < BaseAdapter
+      READ_ATTEMPTS = 3
+      READ_RETRY_DELAY = 0.01
+
       @work_queue = Queue.new
       @threads = []
 
@@ -21,11 +24,13 @@ module Specwrk
 
           lock_file = File.open(lock_file_path, "a")
 
-          Thread.pass until lock_file.flock(File::LOCK_EX)
+          locked = lock_file.flock(File::LOCK_EX | File::LOCK_NB)
+          raise LockUnavailableError unless locked
 
           yield
         ensure
-          lock_file.flock(File::LOCK_UN)
+          lock_file&.flock(File::LOCK_UN) if locked
+          lock_file&.close
         end
 
         def schedule_work(&blk)
@@ -91,18 +96,21 @@ module Specwrk
 
         read_keys.each do |key|
           self.class.schedule_work do
-            result_queue.push([key.to_s, read(key)])
+            result_queue.push([:ok, key.to_s, read(key)])
+          rescue => e
+            result_queue.push([:error, key.to_s, e])
           end
         end
 
-        Thread.pass until result_queue.length == read_keys.length
+        raw_results = read_keys.length.times.map { result_queue.pop }
+        error = raw_results.find { |status, _key, _result| status == :error }
+        raise error.last if error
 
         results = {}
-        until result_queue.empty?
-          result = result_queue.pop
-          next if result.last.nil?
+        raw_results.each do |_status, key, content|
+          next if content.nil?
 
-          results[result.first] = self.class.serializer.load(result.last)
+          results[key] = self.class.serializer.load(content)
         end
 
         read_keys.map { |key| [key.to_s, results[key.to_s]] if results.key?(key.to_s) }.compact.to_h # respect order requested in the returned hash
@@ -140,7 +148,17 @@ module Specwrk
 
       def read(key)
         filename = filename_for_key key
-        File.binread(filename)
+        attempts = 0
+
+        begin
+          attempts += 1
+          File.binread(filename)
+        rescue Errno::EBADF
+          raise if attempts >= READ_ATTEMPTS
+
+          sleep(READ_RETRY_DELAY * attempts)
+          retry
+        end
       rescue Errno::ENOENT
         nil
       end

data/lib/specwrk/store/serializer.rb

@@ -36,14 +36,14 @@ module Specwrk
 
         def dump(value)
           packer.tap(&:clear)
-            .tap { it.write(value) }
+            .tap { |msgpack| msgpack.write(value) }
             .then(&:to_s)
         end
 
         def load(payload)
           unpacker.tap(&:reset)
-            .tap { it.feed(payload) }
-            .then { it.read }
+            .tap { |msgpack| msgpack.feed(payload) }
+            .then { |msgpack| msgpack.read }
         end
 
         def packer

data/lib/specwrk/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Specwrk
-  VERSION = "0.19.2"
+  VERSION = "0.19.4"
 end

data/lib/specwrk/web/endpoints/base.rb

@@ -30,6 +30,8 @@ module Specwrk
           with_response.tap do |response|
             response[1]["x-specwrk-status"] = worker_status.to_s
           end
+        rescue Store::LockUnavailableError
+          locked
         end
 
         def with_response
@@ -56,6 +58,14 @@ module Specwrk
           end
         end
 
+        def locked
+          if request.head?
+            [423, {}, []]
+          else
+            [423, {"content-type" => "text/plain"}, ["Locked. Try again later."]]
+          end
+        end
+
         def payload
           return unless request.content_type&.start_with?("application/json")
           return unless request.post? || request.put? || request.delete?

data/lib/specwrk/web/endpoints/complete_and_pop.rb

@@ -7,12 +7,14 @@ module Specwrk
     module Endpoints
       class CompleteAndPop < Popable
         EXAMPLE_STATUSES = %w[passed failed pending]
+        SYNTHETIC_FAILURE_CLASS = "Specwrk::WorkerExecutionError"
 
         def with_response
           retry_examples # pre-calculate before lock
+          omitted_processing_examples # pre-calculate before lock
 
           with_lock do
-            processing.delete(*(completed_examples.keys + retry_examples.keys))
+            processing.delete(*(completed_examples.keys + retry_examples.keys).map(&:to_s))
             pending.merge!(retry_examples)
           end
 
@@ -26,24 +28,30 @@ module Specwrk
 
         private
 
-        def all_examples
-          @all_examples ||= payload.map { |example| [example[:id], example] if processing_examples[example[:id]] }.compact.to_h
+        def reported_processing_examples
+          @reported_processing_examples ||= payload.map do |example|
+            [example[:id], example] if processing_examples_for_payload[example[:id]]
+          end.compact.to_h
         end
 
-        def processing_examples
-          @processing_examples ||= processing.multi_read(*payload.map { |example| example[:id] })
+        def processing_examples_for_payload
+          @processing_examples_for_payload ||= processing.multi_read(*payload.map { |example| example[:id] })
         end
 
         def completed_examples
-          @completed_examples ||= all_examples.map do |id, example|
-            next if retry_example?(example)
+          @completed_examples ||= begin
+            reported_completed_examples = reported_processing_examples.map do |id, example|
+              next if retry_example?(example)
 
-            [id, example]
-          end.compact.to_h
+              [id, example]
+            end.compact.to_h
+
+            reported_completed_examples.merge(unreported_completed_examples)
+          end
         end
 
         def retry_examples
-          @retry_examples ||= all_examples.map do |id, example|
+          @retry_examples ||= reported_processing_examples.map do |id, example|
             next unless retry_example?(example)
 
             [id, example]
@@ -66,7 +74,48 @@ module Specwrk
         end
 
         def all_example_failure_counts
-          @all_example_failure_counts ||= failure_counts.multi_read(*all_examples.keys)
+          @all_example_failure_counts ||= failure_counts.multi_read(*reported_processing_examples.keys)
+        end
+
+        def omitted_processing_examples
+          @omitted_processing_examples ||= processing.to_h.select do |_id, example|
+            example[:worker_id].to_s == worker_id && !payload_example_ids.include?(example[:id].to_s)
+          end
+        end
+
+        def unreported_completed_examples
+          @unreported_completed_examples ||= omitted_processing_examples.transform_values { |example| synthetic_failure_for(example) }
+        end
+
+        def payload_example_ids
+          @payload_example_ids ||= payload.map { |example| example[:id].to_s }.uniq
+        end
+
+        def synthetic_failure_for(example)
+          finished_at = Time.now
+
+          {
+            id: example[:id],
+            full_description: "Specwrk worker execution failed before completing #{example[:id]}",
+            status: "failed",
+            file_path: example[:file_path],
+            line_number: line_number_for(example),
+            started_at: finished_at.iso8601(6),
+            finished_at: finished_at.iso8601(6),
+            run_time: 0.0,
+            exception: {
+              class: SYNTHETIC_FAILURE_CLASS,
+              message: "Worker #{worker_id} claimed this example but did not submit a completion result. Check worker stderr/stdout for the underlying RSpec output.",
+              backtrace: []
+            }
+          }
+        end
+
+        def line_number_for(example)
+          return example[:line_number] if example[:line_number]
+
+          match = example[:id].to_s.match(/:(\d+)\z/)
+          match[1].to_i if match
         end
 
         def completed_examples_status_counts
@@ -87,7 +136,9 @@ module Specwrk
         end
 
         def run_time_data
-          @run_time_data ||= payload.map { |example| [example[:id], example[:run_time]] }.to_h
+          @run_time_data ||= payload.map { |example| [example[:id], example[:run_time]] }.to_h.merge(
+            unreported_completed_examples.transform_values { |example| example[:run_time] }
+          )
         end
       end
     end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: specwrk
 version: !ruby/object:Gem::Version
-  version: 0.19.2
+  version: 0.19.4
 platform: ruby
 authors:
 - Daniel Westendorf
@@ -204,6 +204,7 @@ files:
 - CHANGELOG.md
 - LICENSE
 - README.md
+- RELEASING.md
 - Rakefile
 - Specwrk.watchfile.rb
 - config.ru
@@ -274,7 +275,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.9
+rubygems_version: 4.0.10
 specification_version: 4
 summary: Parallel rspec test runner from a queue of pending jobs.
 test_files: []