pipeline_machines 0.1.4-x86_64-linux

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e4b980ba128e51a91bd4effaf26aae4584a52f4cf2dacc8d3494eab5250fda84
+  data.tar.gz: ff87d4042f1ed86a71b2207ed678dcb8dc80119c80e3f1b63599bb48f0c719d0
+SHA512:
+  metadata.gz: 2d8faa0c17e0bef8607b50b43bafb92803b02b2759f8658bc13d75f445d42f420ba605466d8b139ce0e69c4c9ce9c7d5b2d0e4fbd759f0b9a405534edb241845
+  data.tar.gz: a154716a50a82bf1c338c8f0f6d6a3350891bcef96b142ef20bff35a2ebefcf9481d18af260bdd8dfeb9b1f9614e10fc8d8c0b0b103eaf3e6478940e6aeab987

data/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+## [0.1.4](https://github.com/seuros/pipeline_machines/compare/pipeline_machines/v0.1.3...pipeline_machines/v0.1.4) (2026-11-06)
+
+
+### Bug Fixes
+
+* extract has_queued lane check ([8c9510f](https://github.com/seuros/pipeline_machines/commit/8c9510fbd5987c2d2f9d4c653ad7aff983e30955))
+
+## [0.1.3](https://github.com/seuros/pipeline_machines/compare/pipeline_machines/v0.1.2...pipeline_machines/v0.1.3) (2026-11-06)
+
+
+### Bug Fixes
+
+* crates.io keywords and categories for the core ([f308575](https://github.com/seuros/pipeline_machines/commit/f30857529c4ac61c145128710bb916770699f760))
+
+## [0.1.2](https://github.com/seuros/pipeline_machines/compare/pipeline_machines/v0.1.1...pipeline_machines/v0.1.2) (2026-11-06)
+
+
+### Bug Fixes
+
+* keep test sources out of the packaged gem and crate ([cd46e0d](https://github.com/seuros/pipeline_machines/commit/cd46e0d8adb4868270b1278fb7cd24c001c89c72))
+
+## [0.1.1](https://github.com/seuros/pipeline_machines/compare/pipeline_machines-v0.1.0...pipeline_machines/v0.1.1) (2026-11-06)
+
+
+### Features
+
+* initial extraction ([5e3f8da](https://github.com/seuros/pipeline_machines/commit/5e3f8dadd4cb7159e5fcbdf64a0c8d2de110c680))
+
+
+### Bug Fixes
+
+* name the gemspec block spec so the release workflow detects the extension ([4a86b80](https://github.com/seuros/pipeline_machines/commit/4a86b80992c296d0157d5488daea23c1295665c4))

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abdelkader Boudih
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,250 @@
+# PipelineMachines
+
+> A job you cancelled is still on the ship until someone signs for it. 🛤️
+
+A stage coordinator for Ruby. You submit work into one of four weighted lanes, pull it out in fair order, report what happened, and collect the receipt. Fan a job out across several lanes and join on all of them. The stage keeps count of every slot, including the ones holding work you told to stop but that hasn't stopped yet.
+
+## Quick Start
+
+```bash
+gem 'pipeline_machines'
+```
+
+```ruby
+# Four lanes: transcode, thumbnail, captions, notify. Thumbnails get double share.
+stage = PipelineMachines::Stage.new([1, 2, 1, 1])
+
+# One upload, three lanes at once.
+jobs = stage.fan_out([0, 1, 2], upload.id)
+
+# The worker loop. dispatch hands back [job, payload, lane].
+while (job, payload, lane = stage.dispatch)
+  run_lane(lane, payload)
+  stage.complete(job)
+end
+
+stage.join(jobs)                     # => :ready
+jobs.each { |job| stage.reap(job) }  # collect receipts, free the slots
+```
+
+## A Message to the Resistance
+
+So you built the upload pipeline. A job comes in, you push three tasks onto a queue, three workers pick them up, the video gets transcoded, thumbnailed, and captioned. Then product asked for cancel.
+
+You added a `cancelled` flag. Workers check it before starting. Great. Except the transcode worker checked it four seconds ago and is now thirty seconds into an ffmpeg call that doesn't care about your flag. The user sees "cancelled", uploads a replacement, and now two transcodes of the same video are fighting over one CPU. Meanwhile your concurrency limit says there's a free slot, because the cancelled job was removed from the queue the moment the flag flipped.
+
+The queue was honest about what was *waiting*. It lied about what was *running*. A slot isn't free when you decide a job should stop. It's free when the job has actually stopped and you've collected the outcome.
+
+This gem keeps that count. Queued work can be cancelled outright. Running work stays on the books until the worker confirms it quit, or finishes anyway. Nothing frees a slot except reaping the receipt.
+
+## Chapter 1: Stardate 2025.186, the AMS Welsh
+
+> **AMS Welsh**, NCC-4212, *Welsh*-class stage carrier.
+> Freight is not gone merely because it left the waiting deck.
+
+The carrier doesn't run any of the work. It holds the manifest: which berths are charged, which descriptors are on the waiting deck, which are on the work deck, and which outcomes are still unsigned. A berth stays charged until the receipt is collected. Not when the job is dispatched, not when it's cancelled. Collected.
+
+Somewhere on the bridge, a junior officer proposes:
+
+```ruby
+def cancel(job)
+  queue.delete(job)
+  job.update(status: :cancelled)   # the worker will notice. probably.
+end
+```
+
+"That," says the deck officer, "is how a berth gets charged twice."
+
+## The Flight Plan
+
+Every job is in exactly one of these states. Only reaping frees its slot.
+
+```mermaid
+stateDiagram-v2
+    [*] --> queued: submit / fan_out
+    queued --> running: dispatch
+    queued --> cancelled: cancel
+    running --> complete: complete
+    running --> cancelling: cancel (worker not yet stopped)
+    cancelling --> cancelled: confirm_cancel
+    cancelling --> complete: complete (finished anyway)
+    complete --> [*]: reap
+    cancelled --> [*]: reap
+```
+
+And the upload that got cancelled mid-transcode:
+
+```mermaid
+sequenceDiagram
+    participant You
+    participant Stage
+    participant W as Worker
+
+    You->>Stage: submit(lane 0, upload 42)
+    W->>Stage: dispatch
+    Stage-->>W: job, 42, lane 0
+    Note over W: ffmpeg starts, 30 seconds to go
+
+    You->>Stage: cancel(job)
+    Stage-->>You: true
+    You->>Stage: join([job])
+    Stage-->>You: pending
+    Note over Stage: slot still charged
+
+    Note over W: ffmpeg killed
+    W->>Stage: confirm_cancel(job)
+    Stage-->>W: true
+    You->>Stage: reap(job)
+    Stage-->>You: [42, cancelled]
+    Note over Stage: slot free
+```
+
+## How the Deck Works
+
+### Submit and dispatch
+
+Lanes are numbered 0 to 3. Each lane has a weight you choose at construction. Dispatch walks the lanes in weighted round robin, first in first out within a lane.
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 2, 1, 1])
+stage.lanes      # => 4
+stage.capacity   # => 64
+
+stage.submit(0, 100)
+stage.submit(0, 101)
+stage.submit(1, 200)
+stage.submit(1, 201)
+stage.submit(2, 300)
+
+job, payload, lane = stage.dispatch
+[payload, lane]   # => [100, 0]
+```
+
+Lane 1 has weight 2, so it gets two turns per round. With the queue above, dispatch order is lane 0, lane 1, lane 1, lane 2, then lane 0 again. Payloads are Integers: ids, pointers into your own tables, whatever names the work. The stage never sees the work itself.
+
+A lane that doesn't exist, or a full stage, returns `nil` from `submit`. It never raises for that.
+
+### Fan out and join
+
+One payload, several lanes, admitted atomically. Either every lane gets a job or none does.
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 1, 1, 1])
+jobs = stage.fan_out([3, 1], 9)
+jobs.size             # => 2
+stage.join(jobs)      # => :pending
+
+2.times do
+  job, = stage.dispatch
+  stage.complete(job)
+end
+stage.join(jobs)      # => :ready
+```
+
+`join` answers for the whole group. `:ready` when every job completed, `:cancelled` when any was cancelled, `:pending` while any is still queued, running, or waiting on a cancel confirmation, and `:unknown` if any handle has been reaped or belongs to another stage. Lanes in a fan-out must be distinct, and asking for a lane that doesn't exist returns `nil` without admitting anything.
+
+### Cancel, and what cancel actually means
+
+Cancel a queued job and it's done. Terminal. Reap it whenever you like.
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 1, 1, 1])
+job = stage.submit(2, 77)
+
+stage.cancel(job)     # => true
+stage.join([job])     # => :cancelled
+stage.reap(job)       # => [77, :cancelled]
+stage.dispatch        # => nil, nothing left
+```
+
+Cancel a *running* job and you've only asked. The slot stays charged until the worker says it stopped.
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 1, 1, 1])
+stage.submit(0, 5)
+job, = stage.dispatch
+
+stage.cancel(job)           # => true, request noted
+stage.reap(job)             # => nil, not terminal yet
+stage.join([job])           # => :pending
+
+stage.confirm_cancel(job)   # => true, worker has stopped
+stage.reap(job)             # => [5, :cancelled]
+```
+
+If the worker finishes before it sees the request, `complete` wins and the cancel is moot:
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 1, 1, 1])
+stage.submit(0, 1)
+job, = stage.dispatch
+
+stage.cancel(job)           # => true
+stage.complete(job)         # => true, it finished anyway
+stage.confirm_cancel(job)   # => false, nothing to confirm
+stage.reap(job)             # => [1, :complete]
+```
+
+### Reap, or the slot never comes back
+
+The stage holds 64 jobs. Not 64 queued, not 64 running. 64 *unreaped*. A completed job you never reap is a slot you never get back.
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 1, 1, 1])
+jobs = 64.times.map { |i| stage.submit(0, i) }
+stage.submit(0, 65)         # => nil, full
+
+job, = stage.dispatch
+stage.complete(job)
+stage.submit(0, 65)         # => nil, still full. complete isn't reap.
+
+stage.reap(job)             # => [0, :complete]
+stage.submit(0, 65)         # now admitted
+```
+
+`reap` only returns for terminal jobs: `[payload, :complete]` or `[payload, :cancelled]`. Anything else is `nil`. After reaping, the handle is dead. Every call with it returns `false` or `nil`, and `join` reports `:unknown`.
+
+### Drain
+
+Shutting down a worker pool means two things: stop taking work, and know when the last job is gone.
+
+```ruby
+stage = PipelineMachines::Stage.new([1, 1, 1, 1])
+job = stage.submit(0, 5)
+
+stage.begin_drain
+stage.submit(0, 6)          # => nil, closed
+stage.drained?              # => false, one job still charged
+
+work, = stage.dispatch      # draining still dispatches what's queued
+stage.complete(work)
+stage.drained?              # => false, completed but not reaped
+stage.reap(job)
+stage.drained?              # => true
+```
+
+## The Rules of the Carrier
+
+- **Four lanes, sixty-four slots.** Weights are an Array of exactly four Integers in `1..2**32-1`. Anything else raises `TypeError` or `ArgumentError`.
+- **Lanes and payloads are Integers** in `0..2**64-1`. Floats and strings raise `TypeError`. Negative or too large raises `RangeError`.
+- **Bad lanes and full stages return `nil`.** Only malformed input raises.
+- **Handles belong to one stage.** A job from another stage, or one already reaped, mutates nothing: `cancel`, `complete`, and `confirm_cancel` return `false`, `reap` returns `nil`, `join` says `:unknown`.
+- **`join` takes up to 64 handles.** An empty Array is `:ready`.
+- **Frozen means frozen.** Every mutation raises `FrozenError`. `join` and `drained?` still answer.
+
+## What You Think You're Doing vs Reality
+
+### You Think: "I cancelled it, so the slot is free."
+### Reality: You cancelled it, so the slot is spoken for by a job nobody is watching.
+
+Removing work from a queue is instant. Stopping work that has already started is a negotiation with a process that may be deep inside a syscall. Between those two moments the resources are still in use, and any capacity count that says otherwise will admit one job too many, every time, right when the system is under the most pressure.
+
+## What's Yours
+
+The stage counts. It doesn't run, schedule, persist, or time anything out. You run the worker loop and call `dispatch` when you have a free worker. You decide what a payload means. You tell the stage `complete` or `confirm_cancel` when the work really stopped, and you `reap` every job, every time, or the slots drain away. Across a restart, the manifest is gone, so write down what you dispatched before you dispatch it.
+
+## License
+
+MIT License. See [LICENSE.txt](LICENSE.txt) for details.
+
+**Remember: The transcode you cancelled at 14:02 is still burning a core at 14:03. The stage knows. Your queue didn't.**

data/ext/pipeline_machines_native/core/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+## [0.2.3](https://github.com/seuros/pipeline_machines/compare/pipeline-machines-v0.2.2...pipeline-machines-v0.2.3) (2026-11-06)
+
+
+### Bug Fixes
+
+* extract has_queued lane check ([8c9510f](https://github.com/seuros/pipeline_machines/commit/8c9510fbd5987c2d2f9d4c653ad7aff983e30955))
+
+## [0.2.2](https://github.com/seuros/pipeline_machines/compare/pipeline-machines-v0.2.1...pipeline-machines-v0.2.2) (2026-11-06)
+
+
+### Bug Fixes
+
+* crates.io keywords and categories for the core ([f308575](https://github.com/seuros/pipeline_machines/commit/f30857529c4ac61c145128710bb916770699f760))
+
+## [0.2.1](https://github.com/seuros/pipeline_machines/compare/pipeline-machines-v0.2.0...pipeline-machines-v0.2.1) (2026-11-06)
+
+
+### Bug Fixes
+
+* keep test sources out of the packaged gem and crate ([cd46e0d](https://github.com/seuros/pipeline_machines/commit/cd46e0d8adb4868270b1278fb7cd24c001c89c72))
+
+## [0.2.0](https://github.com/seuros/pipeline_machines/compare/pipeline-machines-v0.1.0...pipeline-machines-v0.2.0) (2026-11-06)
+
+
+### Features
+
+* initial extraction ([5e3f8da](https://github.com/seuros/pipeline_machines/commit/5e3f8dadd4cb7159e5fcbdf64a0c8d2de110c680))

data/ext/pipeline_machines_native/core/README.md

@@ -0,0 +1,38 @@
+# pipeline-machines
+
+> **AMS Welsh**, NCC-4212, *Welsh*-class stage carrier.
+
+Engine-room flight manual: account for bounded cargo stages and unresolved flights.
+
+Allocation-free `no_std` coordinator for fixed lanes and shared bounded job slots.
+Copyable payloads should be command IDs or small descriptors; adapters own actual
+resources and execute work. WRR selects ready lanes, FIFO within each lane.
+External eligibility masks can incorporate arbitration or deadline policy.
+
+Flow credits bound **all unreaped jobs**, including running and terminal jobs.
+Fanout admission is atomic across distinct lanes. Fan-in reports readiness over
+unreaped handles, not an output aggregation or distributed transaction.
+
+Queued cancellation is immediate. Running cancellation becomes `Cancelling` and
+keeps its capacity until the executor confirms cancellation or reports completion.
+This never assumes an unresolved external effect has disappeared. Reap only
+terminal jobs to return credit; generation checks reject stale handles.
+Handle namespaces are scoped to one stage lifetime.
+
+Draining refuses submissions while allowing dispatch and completion. No async
+runtime, multi-stage graph orchestration, automatic propagation, result payloads,
+durability, or Ruby binding is implemented.
+
+```rust
+use core::num::NonZeroU32;
+use pipeline_machines::{Stage, State};
+# fn example() -> Option<()> {
+let mut stage = Stage::<u64, 1, 4>::new([NonZeroU32::MIN])?;
+let id = stage.submit(0, 42)?;
+assert_eq!(stage.dispatch(&[true])?.id, id);
+assert!(stage.complete(id));
+assert_eq!(stage.reap(id), Some((42, State::Complete)));
+# Some(())
+# }
+# assert_eq!(example(), Some(()));
+```

data/ext/pipeline_machines_native/extconf.rb

@@ -0,0 +1,2 @@
+# frozen_string_literal: true
+require_relative 'ffi/extconf'

data/lib/pipeline_machines/version.rb

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+module PipelineMachines
+  VERSION = '0.1.4'
+end

data/lib/pipeline_machines.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+require_relative 'pipeline_machines/version'
+require 'pipeline_machines_native/pipeline_machines_native'

data/sig/pipeline_machines.rbs

@@ -0,0 +1,20 @@
+module PipelineMachines
+  VERSION: String
+  class Job
+  end
+  class Stage
+    def self.new: (Array[Integer] weights) -> Stage
+    def capacity: () -> Integer
+    def lanes: () -> Integer
+    def submit: (Integer lane, Integer payload) -> Job?
+    def fan_out: (Array[Integer] lanes, Integer payload) -> Array[Job]?
+    def dispatch: () -> [Job, Integer, Integer]?
+    def cancel: (Job) -> bool
+    def complete: (Job) -> bool
+    def confirm_cancel: (Job) -> bool
+    def reap: (Job) -> [Integer, Symbol]?
+    def join: (Array[Job]) -> Symbol
+    def begin_drain: () -> nil
+    def drained?: () -> bool
+  end
+end

metadata

@@ -0,0 +1,60 @@
+--- !ruby/object:Gem::Specification
+name: pipeline_machines
+version: !ruby/object:Gem::Version
+  version: 0.1.4
+platform: x86_64-linux
+authors:
+- Abdelkader Boudih
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2026-11-06 00:00:00.000000000 Z
+dependencies: []
+description: Local unstable stage policy model, not a durable pipeline runtime.
+email:
+- oss@seuros.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- ext/pipeline_machines_native/core/CHANGELOG.md
+- ext/pipeline_machines_native/core/README.md
+- ext/pipeline_machines_native/extconf.rb
+- lib/pipeline_machines.rb
+- lib/pipeline_machines/version.rb
+- lib/pipeline_machines_native/pipeline_machines_native.so
+- sig/pipeline_machines.rbs
+homepage: https://github.com/seuros/pipeline_machines
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/seuros/pipeline_machines
+  rubygems_mfa_required: 'true'
+  cargo_crate_name: pipeline_machines_native
+  cargo_manifest_path: ext/pipeline_machines_native/ffi/Cargo.toml
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '4.0'
+  - - "<"
+    - !ruby/object:Gem::Version
+      version: 4.1.dev
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.23
+signing_key: 
+specification_version: 4
+summary: Initial native bounded stage lifecycle and fanout decisions
+test_files: []