med_pipe 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 254b6d0a324f400418ad4ead68cc2741f779e77a1a18f01b0e865737d783358b
-  data.tar.gz: ab801a44a97d8ff5a9f6bba5ca023bd0dc1712707d20b7d3754981f6c7528228
+  metadata.gz: e033daa9e892bde3d031d927cf70e7ea1164edcef333eea69c0cdb71d17124d2
+  data.tar.gz: 1253ceea7ed2d9021c2e620a52addfe813fd80bda18f2a55920e18b9f7134623
 SHA512:
-  metadata.gz: 8d94037e54f43df01d53f95057e21597660afa3c4eedb24d107ed2ffeb5d7d5c823f4ed9f0ea28a9267bd26d58be6fc22e7dbc6eb918ddb059bbead9db85f418
-  data.tar.gz: 0c744341829cb8f970acbd26da6e3b6708534b18e2ad0a34cc8641b7ba48d313a6f788de486827c37b7ffd0238d33227f76130e40ddc5753159b53f12af3dee9
+  metadata.gz: bbf20fedd6d3d99d1789da72fd6e7555357882a4acf17f6f18e97d23aa739e4b028beecbf149a9ac28017cefb1a92079ec7bb3c9f78e39a4cb8311496a051feb
+  data.tar.gz: bcd4881d759ccb04b2c1f9bac09f758534098b5524ff266952d0ccd5642c0c96536e94a3a8e95c5246f724bc9f9a3ae65d42b69550828116eb3c2c3e8655e219

data/README.md

@@ -1,39 +1,56 @@
-# MedPipe <sup>BETA</sup>
-100万 ~ 数10億程度のデータを処理するための仕組みを提供する Rails エンジンです。
+# MedPipe
+![test_badge](https://github.com/medpeer-dev/med_pipe/actions/workflows/test.yml/badge.svg)
+
+A Rails engine that provides mechanisms for processing datasets ranging from 1 million to several billion records.
 
 ## Concept
+
+![MedPipeConcept](https://github.com/user-attachments/assets/69ef986b-33cc-478c-830f-78d24ff6c9f4)
+
 ### MedPipe::Pipeline
-apply で後述する PipelineTask を登録し、run で順番に実行します。
+Register PipelineTask through 'apply' method and execute them sequentially using 'run'.
 
 ### MedPipe::PipelineTask
-Pipeline に登録する処理の単位です。  
-DB からの読み込みや、S3 へのアップロード等やることを分割してタスク化します。  
-大量データを扱う際には Enumerable::Lazy を使うことで分割して処理をすることができます。  
-call を実装する必要があります
+This is the basic unit of processing registered in the pipeline.  
+Tasks are divided into specific operations such as reading from DB or uploading to S3.  
+When handling large datasets, Enumerable::Lazy can be used to process data in chunks.  
+You need to implement the 'call' method:
 
-```.rb
+```ruby
 @param context [Hash] Stores data during pipeline execution
 @param prev_result [Object] The result of the previous task
 def call(context, prev_result)
-  yield 次のTaskに渡すデータ
+  yield "data_to_pass_to_next_task"
 end
 ```
 
 ### MedPipe::PipelinePlan
-Pipeline の状態、オプション、結果を保存するためのモデルです。  
-Task で使うためのオプションを渡す方法は PipelinePlan から取得するか、contextで伝搬するかの二択です。
+A model for storing pipeline state, options, and results.  
+There are two ways to pass options for tasks: either retrieve from PipelinePlan or propagate through context.
 
 ### MedPipe::PipelineGroup
-一つのジョブで実行する Plan をまとめるためのモデルです。  
-実行中に parallel_limit を 0 にすることで中断することができます。
+A model for grouping plans.  
+Execution can be interrupted by setting parallel_limit to 0 during runtime.
 
 ## Usage
 
-1. Reader, Uploader 等の PipelineTask を作成 [Samples](https://github.com/medpeer-dev/med_pipe/tree/main/spec/dummy/app/models/pipeline_task)
-2. PipelineRunner を作成 [Sample](https://github.com/medpeer-dev/med_pipe/blob/main/spec/dummy/app/models/sample_pipeline_runner.rb)
-3. Pipeline を並列実行するためのジョブを作成 [Sample](https://github.com/medpeer-dev/med_pipe/blob/main/spec/dummy/app/jobs/sample_execute_pipeline_job.rb)
-4. PipelinePlan を登録するコードを記述
-5. 実行
+1. Create PipelineTask such as Reader, Uploader, etc. [Samples](https://github.com/medpeer-dev/med_pipe/tree/main/spec/dummy/app/models/pipeline_task)
+2. Create PipelineRunner [Sample](https://github.com/medpeer-dev/med_pipe/blob/main/spec/dummy/app/models/sample_pipeline_runner.rb)
+3. Create a job for parallel Pipeline execution [Sample](https://github.com/medpeer-dev/med_pipe/blob/main/spec/dummy/app/jobs/sample_execute_pipeline_job.rb)
+4. Write code to register PipelinePlan
+5. Execute like this:
+
+```ruby
+# add plan
+pipeline_group = MedPipe::PipelineGroup.create!(parallel_limit: 10)
+date_range = Date.new(2024, 6, 1)..Date.new(2024, 6, 30)
+date_range.each do |date|
+  pipeline_group.pipeline_plans.status_waiting.create!(name: 'point_events', output_unit: :daily, target_date: date)
+end
+
+# execute
+ExecutePipelineJob.perform_later(pipeline_group.id)
+```
 
 ## Installation
 Add this line to your application's Gemfile:
@@ -42,7 +59,7 @@ Add this line to your application's Gemfile:
 gem "med_pipe"
 ```
 
-### migrationファイルの追加
+### Adding migration files
 
 ```shell
 $ rails med_pipe:install:migrations

data/app/models/med_pipe/pipeline_plan.rb

@@ -9,18 +9,16 @@ class MedPipe::PipelinePlan < MedPipe::ApplicationRecord
   validates :output_unit, presence: true
   validates :status, presence: true
 
-  # TODO: Rails6記法のため、Rails8に上げる際に定義の仕方を変える
-  # https://zenn.dev/kanazawa/articles/8bc1fcbba3ef1d#enum%E3%81%AE%E5%AE%9A%E7%BE%A9%E6%96%B9%E6%B3%95%E3%81%8C%E5%A4%89%E3%82%8F%E3%82%8B
-  enum status: {
+  enum :status, {
     waiting: "waiting",
     enqueued: "enqueued",
     running: "running",
     finished: "finished",
     failed: "failed"
-  }, _prefix: true
+  }, prefix: true, default: :waiting
 
-  enum output_unit: {
+  enum :output_unit, {
     daily: "daily",
     all: "all"
-  }, _prefix: true
+  }, prefix: true
 end

data/lib/med_pipe/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module MedPipe
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: med_pipe
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - mpg-taichi-sato
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-11-28 00:00:00.000000000 Z
+date: 2024-11-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -16,20 +16,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 6.1.7
-    - - "<"
-      - !ruby/object:Gem::Version
-        version: '8.0'
+        version: 7.2.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 6.1.7
-    - - "<"
-      - !ruby/object:Gem::Version
-        version: '8.0'
+        version: 7.2.0
 description: Provides a system for processing data ranging from 1 million to several
   billion records
 email: