pdi 1.0.0.pre.alpha.1 → 2.0.0.pre.alpha

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e653f6aed9fd26cce60e4f84cfdf662e0be847f40828142dd28fcc6118150cc7
4
- data.tar.gz: 6ab7d8da5c628dc867d0607bcb3a3db6a5b4c030f07d91adc3f47516a2b3350f
3
+ metadata.gz: 3adae07a9c634b7075a87a47c810c0d4e3c90137ee85a08e01e09038baf990e1
4
+ data.tar.gz: 9f7ecf0e06ace02def29a7e710ea87d5ed5e2af82282a1dba935fa3628116533
5
5
  SHA512:
6
- metadata.gz: 2e2688a153fea7a256e80a9e682b6488e71923d2752e32f37c23a39c7899f33faf2ffd99c91b5770e4e4279dd0af0ea9f3bdd6d52057b9780d6f798f7a44fedc
7
- data.tar.gz: 66f618cc1203d313091cd849403501c127907a719f2b6503792cffe606f02062d0f2f31e179a45395e4460c47245de5461c1b78496208ad5ddf68437f7889558
6
+ metadata.gz: 5104d4d97f12a70e7f2d492dee6f8f870c388fea8091bc26d60fd87efb826331479a5efa851de7cbeb278d48aad9368d12cbad7bbb675b9cd9a20baee50f087e
7
+ data.tar.gz: 3ff6caa7e64806a1f03f918b7a6692e25c7493aa6280c0f9edabc99dd937e8890d76f3e8e0f060584964b993791c990621dc3cd8c908f91b5e81a25ac3e9f5ff
@@ -1,4 +1,4 @@
1
- Metrics/LineLength:
1
+ Layout/LineLength:
2
2
  Max: 100
3
3
  Exclude:
4
4
  - pdi.gemspec
@@ -0,0 +1,19 @@
1
+ # 2.0.0 (May 7th, 2020)
2
+
3
+ Breaking Changes:
4
+
5
+ * Standard error and output have been combined into one stream (out). Early feedback indicated that reading both at the same time was preferable.
6
+
7
+ Enhancements:
8
+
9
+ * Added optional `timeout_in_seconds` argument to Pdi::Spoon#initialize.
10
+
11
+ # 1.0.1 (February 19th, 2020)
12
+
13
+ Fixes:
14
+
15
+ * It now properly calls kitchen for jobs (it was previously calling pan.)
16
+
17
+ # 1.0.0 (February 19th, 2020)
18
+
19
+ Initial release.
data/README.md CHANGED
@@ -1,4 +1,4 @@
1
- # Pdi
1
+ # PDI
2
2
 
3
3
  ---
4
4
 
@@ -36,17 +36,21 @@ Pull Requests are welcome for:
36
36
 
37
37
  ## Examples
38
38
 
39
- All examples assume Pdi has been installed to your home directory: `~/data-integration`.
39
+ All examples assume PDI has been installed to your home directory: `~/data-integration`.
40
40
 
41
41
  ### Creating a Spoon Instance
42
42
 
43
- `Pdi::Spoon` is the common interface you will use when interacting with Pdi. It will use Pan and Kitchen for executing Spoon commands.
43
+ `Pdi::Spoon` is the common interface you will use when interacting with PDI. It will use Pan and Kitchen for executing Spoon commands.
44
44
 
45
45
  ```ruby
46
- spoon = Pdi::Spoon.new(dir: `~/data-integration`)
46
+ spoon = Pdi::Spoon.new(dir: '~/data-integration')
47
47
  ```
48
48
 
49
- Note: You can also override the names of the scripts using the `kitchen` and `pan` constructor keyword arguments. The defaults are `kitchen.sh` and `pan.sh`, respectively.
49
+ Notes:
50
+
51
+ * You can also override the names of the scripts using the `kitchen` and `pan` constructor keyword arguments. The defaults are `kitchen.sh` and `pan.sh`, respectively.
52
+ * For other command line arguments that are not supported first-class in the Options objects below you can utilize the `args` argument when instantiating a `Spoon` instance.
53
+ * Another optional argument is `timeout_in_seconds`. If set it will ensure the sub-process runs within a given window. If it times out the sub-process will be terminated and a Timeout::Error will be raised.
50
54
 
51
55
  ### Executing a Job/Transformation
52
56
 
@@ -71,7 +75,7 @@ result = spoon.run(options)
71
75
 
72
76
  You can access the raw command line results by tapping into the execution attribute of the result or error object.
73
77
 
74
- Note: Not all options are currently supported. See Pdi's official references for [Pan](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/000) and [Kitchen](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/010) to see all options.
78
+ Note: Not all options are currently supported. See PDI's official references for [Pan](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/000) and [Kitchen](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/010) to see all options.
75
79
 
76
80
  ## Contributing
77
81
 
data/lib/pdi.rb CHANGED
@@ -8,8 +8,10 @@
8
8
  #
9
9
 
10
10
  require 'acts_as_hashable'
11
+ require 'English'
11
12
  require 'forwardable'
12
13
  require 'open3'
14
+ require 'timeout'
13
15
 
14
16
  require_relative 'pdi/executor'
15
17
  require_relative 'pdi/spoon'
@@ -13,20 +13,42 @@ module Pdi
13
13
  # This class is the library's "metal" layer, the one which actually makes the system call and
14
14
  # interacts with the operating system (through Ruby's standard library.)
15
15
  class Executor
16
+ attr_reader :timeout_in_seconds
17
+
18
+ def initialize(timeout_in_seconds: nil)
19
+ @timeout_in_seconds = timeout_in_seconds
20
+
21
+ freeze
22
+ end
23
+
16
24
  def run(args)
17
25
  args = Array(args).map(&:to_s)
18
26
 
19
- out, err, status = Open3.capture3(*args)
20
-
21
- Result.new(
22
- args: args,
23
- status: {
24
- code: status.exitstatus,
25
- out: out,
26
- err: err,
27
- pid: status.pid
28
- }
29
- )
27
+ IO.popen(args, err: %i[child out]) do |io|
28
+ begin
29
+ io_read =
30
+ if timeout_in_seconds
31
+ Timeout.timeout(timeout_in_seconds) { io.read }
32
+ else
33
+ io.read
34
+ end
35
+
36
+ io.close
37
+ status = $CHILD_STATUS
38
+
39
+ Result.new(
40
+ args: args,
41
+ status: {
42
+ code: status.exitstatus,
43
+ out: io_read,
44
+ pid: status.pid
45
+ }
46
+ )
47
+ rescue Timeout::Error => e
48
+ Process.kill(9, io.pid)
49
+ raise e
50
+ end
51
+ end
30
52
  end
31
53
  end
32
54
  end
@@ -18,7 +18,7 @@ module Pdi
18
18
 
19
19
  attr_reader :args, :status
20
20
 
21
- def_delegators :status, :code, :out, :err, :pid
21
+ def_delegators :status, :code, :out, :pid
22
22
 
23
23
  def initialize(args:, status: {})
24
24
  @args = args
@@ -13,12 +13,11 @@ module Pdi
13
13
  class Status
14
14
  acts_as_hashable
15
15
 
16
- attr_reader :code, :out, :err, :pid
16
+ attr_reader :code, :out, :pid
17
17
 
18
- def initialize(code:, out: '', err: '', pid:)
18
+ def initialize(code:, out: '', pid:)
19
19
  @code = code
20
20
  @out = out
21
- @err = err
22
21
  @pid = pid
23
22
 
24
23
  freeze
@@ -7,33 +7,41 @@
7
7
  # LICENSE file in the root directory of this source tree.
8
8
  #
9
9
 
10
+ require_relative 'spoon/kitchen_error'
10
11
  require_relative 'spoon/options'
12
+ require_relative 'spoon/pan_error'
11
13
  require_relative 'spoon/parser'
12
14
  require_relative 'spoon/result'
13
15
 
14
16
  module Pdi
15
- # This class is the main wrapper for Pdi's pan and kitchen scripts.
17
+ # This class is the main wrapper for PDI's pan and kitchen scripts.
16
18
  class Spoon
17
19
  DEFAULT_KITCHEN = 'kitchen.sh'
18
20
  DEFAULT_PAN = 'pan.sh'
19
21
 
22
+ TYPES_TO_ERRORS = {
23
+ Options::Type::JOB => KitchenError,
24
+ Options::Type::TRANSFORMATION => PanError
25
+ }.freeze
26
+
20
27
  attr_reader :args, :dir, :kitchen, :pan
21
28
 
22
29
  def initialize(
23
30
  args: [],
24
31
  dir:,
25
32
  kitchen: DEFAULT_KITCHEN,
26
- pan: DEFAULT_PAN
33
+ pan: DEFAULT_PAN,
34
+ timeout_in_seconds: nil
27
35
  )
28
36
  assert_required(:dir, dir)
29
37
  assert_required(:kitchen, kitchen)
30
38
  assert_required(:pan, pan)
31
39
 
32
40
  @args = Array(args)
33
- @dir = dir.to_s
41
+ @dir = File.expand_path(dir.to_s)
34
42
  @kitchen = kitchen.to_s
35
43
  @pan = pan.to_s
36
- @executor = Executor.new
44
+ @executor = Executor.new(timeout_in_seconds: timeout_in_seconds)
37
45
  @parser = Parser.new
38
46
 
39
47
  freeze
@@ -55,11 +63,11 @@ module Pdi
55
63
  # Returns an Executor::Result instance when PDI returns error code 0 or else raises
56
64
  # a PanError (transformation) or KitchenError (job).
57
65
  def run(options)
58
- options = Options.make(options)
59
- final_args = [pan_path] + args + options.to_args
66
+ options = Options.make(options)
67
+ all_args = run_args(options)
60
68
 
61
- executor.run(final_args).tap do |result|
62
- raise(options.error_constant, result) if result.code != 0
69
+ executor.run(all_args).tap do |result|
70
+ raise(error_constant(options), result) if result.code != 0
63
71
  end
64
72
  end
65
73
 
@@ -67,6 +75,22 @@ module Pdi
67
75
 
68
76
  attr_reader :executor, :parser
69
77
 
78
+ def error_constant(options)
79
+ TYPES_TO_ERRORS.fetch(options.type)
80
+ end
81
+
82
+ def run_args(options)
83
+ [script_path(options.type)] + args + options.to_args
84
+ end
85
+
86
+ def script_path(options_type)
87
+ if options_type == Options::Type::JOB
88
+ kitchen_path
89
+ elsif options_type == Options::Type::TRANSFORMATION
90
+ pan_path
91
+ end
92
+ end
93
+
70
94
  def kitchen_path
71
95
  File.join(dir, kitchen)
72
96
  end
@@ -11,6 +11,8 @@ module Pdi
11
11
  class Spoon
12
12
  # This class subclasses Ruby's StandardError and provides a mapping to custom Kitchen
13
13
  # specific error codes to messages.
14
+ # You can find the list of errors in Pentaho's documentation site, for example:
15
+ # https://help.pentaho.com/Documentation/8.0/Products/Data_Integration/Command_Line_Tools
14
16
  class KitchenError < StandardError
15
17
  MESSAGES = {
16
18
  '1' => 'Errors occurred during processing',
@@ -7,10 +7,8 @@
7
7
  # LICENSE file in the root directory of this source tree.
8
8
  #
9
9
 
10
- require_relative 'kitchen_error'
11
10
  require_relative 'options/level'
12
11
  require_relative 'options/param'
13
- require_relative 'pan_error'
14
12
 
15
13
  module Pdi
16
14
  class Spoon
@@ -23,11 +21,6 @@ module Pdi
23
21
  TRANSFORMATION = :transformation
24
22
  end
25
23
 
26
- TYPES_TO_ERRORS = {
27
- Type::JOB => KitchenError,
28
- Type::TRANSFORMATION => PanError
29
- }.freeze
30
-
31
24
  TYPES_TO_KEYS = {
32
25
  Type::JOB => Arg::Key::JOB,
33
26
  Type::TRANSFORMATION => Arg::Key::TRANS
@@ -63,10 +56,6 @@ module Pdi
63
56
  base_args + param_args
64
57
  end
65
58
 
66
- def error_constant
67
- TYPES_TO_ERRORS.fetch(type)
68
- end
69
-
70
59
  private
71
60
 
72
61
  def key
@@ -11,6 +11,8 @@ module Pdi
11
11
  class Spoon
12
12
  # This class subclasses Ruby's StandardError and provides a mapping to custom Pan
13
13
  # specific error codes to messages.
14
+ # You can find the list of errors in Pentaho's documentation site, for example:
15
+ # https://help.pentaho.com/Documentation/8.0/Products/Data_Integration/Command_Line_Tools
14
16
  class PanError < StandardError
15
17
  MESSAGES = {
16
18
  '1' => 'Errors occurred during processing',
@@ -8,5 +8,5 @@
8
8
  #
9
9
 
10
10
  module Pdi
11
- VERSION = '1.0.0-alpha.1'
11
+ VERSION = '2.0.0-alpha'
12
12
  end
@@ -0,0 +1,10 @@
1
+ #!/bin/bash
2
+
3
+ echo 'std_out'
4
+ echo 'err_out' >&2
5
+
6
+ sleep $1
7
+
8
+ echo 'after_sleep'
9
+
10
+ exit $2
@@ -0,0 +1,52 @@
1
+ # frozen_string_literal: true
2
+
3
+ #
4
+ # Copyright (c) 2018-present, Blue Marble Payroll, LLC
5
+ #
6
+ # This source code is licensed under the MIT license found in the
7
+ # LICENSE file in the root directory of this source tree.
8
+ #
9
+
10
+ require 'spec_helper'
11
+
12
+ describe Pdi::Executor do
13
+ let(:script) { File.join('spec', 'mocks', 'spoon', 'sleep.sh') }
14
+ let(:one_second) { 1 }
15
+ let(:return_code) { 33 }
16
+
17
+ describe '#run' do
18
+ context 'with a timeout' do
19
+ # do not make these too high, bugs could cause the entire script to still be executed without
20
+ # killing it so these high values could draw out results.
21
+ let(:thirty_seconds) { 30 }
22
+
23
+ subject { described_class.new(timeout_in_seconds: one_second) }
24
+
25
+ # This will run a script that will take 30 seconds to process, but by limiting the
26
+ # timeout using the #run argument, it should raise an error after one second.
27
+ it 'times out and kills process after 5 seconds' do
28
+ args = [script, thirty_seconds]
29
+
30
+ expect { subject.run(args) }.to raise_error(Timeout::Error)
31
+ end
32
+ end
33
+
34
+ context 'without a timeout' do
35
+ it 'returns right exit status as code' do
36
+ args = [script, one_second, return_code]
37
+
38
+ result = subject.run(args)
39
+
40
+ expect(result.code).to eq(return_code)
41
+ end
42
+
43
+ it 'returns right standard output and error as out' do
44
+ args = [script, one_second, return_code]
45
+
46
+ result = subject.run(args)
47
+
48
+ expect(result.out).to eq("std_out\nerr_out\nafter_sleep\n")
49
+ end
50
+ end
51
+ end
52
+ end
@@ -0,0 +1,42 @@
1
+ # frozen_string_literal: true
2
+
3
+ #
4
+ # Copyright (c) 2018-present, Blue Marble Payroll, LLC
5
+ #
6
+ # This source code is licensed under the MIT license found in the
7
+ # LICENSE file in the root directory of this source tree.
8
+ #
9
+
10
+ require 'spec_helper'
11
+
12
+ describe Pdi::Spoon::KitchenError do
13
+ describe 'initialization' do
14
+ [1, 2, 7, 8, 9].each do |code|
15
+ specify "code #{code} should have message" do
16
+ result = Pdi::Executor::Result.new(
17
+ args: [],
18
+ status: {
19
+ code: code,
20
+ pid: 123
21
+ }
22
+ )
23
+
24
+ expect(described_class.new(result).message).not_to eq('Unknown')
25
+ end
26
+ end
27
+
28
+ [-1, 0, 3, 4, 5, 6, 10, 11].each do |code|
29
+ specify "code #{code} should not have message" do
30
+ result = Pdi::Executor::Result.new(
31
+ args: [],
32
+ status: {
33
+ code: code,
34
+ pid: 123
35
+ }
36
+ )
37
+
38
+ expect(described_class.new(result).message).to eq('Unknown')
39
+ end
40
+ end
41
+ end
42
+ end
@@ -14,6 +14,21 @@ describe Pdi::Spoon do
14
14
  let(:mocks_dir) { %w[spec mocks spoon] }
15
15
  let(:dir) { File.join(*mocks_dir) }
16
16
 
17
+ describe '#initialize' do
18
+ it 'sets executor' do
19
+ timeout_in_seconds = 987
20
+
21
+ subject = described_class.new(dir: dir, timeout_in_seconds: timeout_in_seconds)
22
+
23
+ # Private/internal testing is not recommended, but I really wanted to ensure
24
+ # this class is properly configuring the Executor instance, that way I can rely
25
+ # mainly on the Executor unit tests instead of integration tests at this level.
26
+ executor = subject.send('executor')
27
+
28
+ expect(executor.timeout_in_seconds).to eq(timeout_in_seconds)
29
+ end
30
+ end
31
+
17
32
  describe '#run' do
18
33
  context 'transformations' do
19
34
  let(:options) do
@@ -24,6 +39,14 @@ describe Pdi::Spoon do
24
39
  }
25
40
  end
26
41
 
42
+ it 'will call pan script' do
43
+ subject = described_class.new(args: 0, dir: dir, pan: script)
44
+
45
+ result = subject.run(options)
46
+
47
+ expect(result.args.first).to include(script)
48
+ end
49
+
27
50
  context 'when code is 0' do
28
51
  it 'returns correct stdout, stderr and code' do
29
52
  subject = described_class.new(
@@ -35,8 +58,7 @@ describe Pdi::Spoon do
35
58
 
36
59
  result = subject.run(options)
37
60
 
38
- expect(result.out).to eq("output to stdout\n")
39
- expect(result.err).to eq("output to sterr\n")
61
+ expect(result.out).to eq("output to stdout\noutput to sterr\n")
40
62
  expect(result.code).to eq(0)
41
63
  end
42
64
  end
@@ -70,6 +92,14 @@ describe Pdi::Spoon do
70
92
  }
71
93
  end
72
94
 
95
+ it 'will call kitchen script' do
96
+ subject = described_class.new(args: 0, dir: dir, kitchen: script)
97
+
98
+ result = subject.run(options)
99
+
100
+ expect(result.args.first).to include(script)
101
+ end
102
+
73
103
  context 'when code is 0' do
74
104
  it 'returns correct stdout, stderr and code' do
75
105
  subject = described_class.new(
@@ -81,8 +111,7 @@ describe Pdi::Spoon do
81
111
 
82
112
  result = subject.run(options)
83
113
 
84
- expect(result.out).to eq("output to stdout\n")
85
- expect(result.err).to eq("output to sterr\n")
114
+ expect(result.out).to eq("output to stdout\noutput to sterr\n")
86
115
  expect(result.code).to eq(0)
87
116
  end
88
117
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pdi
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0.pre.alpha.1
4
+ version: 2.0.0.pre.alpha
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matthew Ruggio
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-02-18 00:00:00.000000000 Z
11
+ date: 2020-05-08 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: acts_as_hashable
@@ -160,11 +160,14 @@ files:
160
160
  - lib/pdi/version.rb
161
161
  - pdi.gemspec
162
162
  - spec/mocks/spoon/return_code.sh
163
+ - spec/mocks/spoon/sleep.sh
163
164
  - spec/mocks/spoon/version.sh
164
- - spec/pdi/spoon/error_spec.rb
165
+ - spec/pdi/executor_spec.rb
166
+ - spec/pdi/spoon/kitchen_error_spec.rb
165
167
  - spec/pdi/spoon/options/arg_spec.rb
166
168
  - spec/pdi/spoon/options/param_spec.rb
167
169
  - spec/pdi/spoon/options_spec.rb
170
+ - spec/pdi/spoon/pan_error_spec.rb
168
171
  - spec/pdi/spoon_spec.rb
169
172
  - spec/spec_helper.rb
170
173
  homepage: https://github.com/bluemarblepayroll/pdi
@@ -192,10 +195,13 @@ specification_version: 4
192
195
  summary: Ruby wrapper for invoking Pentaho Data Integration
193
196
  test_files:
194
197
  - spec/mocks/spoon/return_code.sh
198
+ - spec/mocks/spoon/sleep.sh
195
199
  - spec/mocks/spoon/version.sh
196
- - spec/pdi/spoon/error_spec.rb
200
+ - spec/pdi/executor_spec.rb
201
+ - spec/pdi/spoon/kitchen_error_spec.rb
197
202
  - spec/pdi/spoon/options/arg_spec.rb
198
203
  - spec/pdi/spoon/options/param_spec.rb
199
204
  - spec/pdi/spoon/options_spec.rb
205
+ - spec/pdi/spoon/pan_error_spec.rb
200
206
  - spec/pdi/spoon_spec.rb
201
207
  - spec/spec_helper.rb