pdi 1.0.0.pre.alpha.1 → 2.0.0.pre.alpha

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e653f6aed9fd26cce60e4f84cfdf662e0be847f40828142dd28fcc6118150cc7
4
- data.tar.gz: 6ab7d8da5c628dc867d0607bcb3a3db6a5b4c030f07d91adc3f47516a2b3350f
3
+ metadata.gz: 3adae07a9c634b7075a87a47c810c0d4e3c90137ee85a08e01e09038baf990e1
4
+ data.tar.gz: 9f7ecf0e06ace02def29a7e710ea87d5ed5e2af82282a1dba935fa3628116533
5
5
  SHA512:
6
- metadata.gz: 2e2688a153fea7a256e80a9e682b6488e71923d2752e32f37c23a39c7899f33faf2ffd99c91b5770e4e4279dd0af0ea9f3bdd6d52057b9780d6f798f7a44fedc
7
- data.tar.gz: 66f618cc1203d313091cd849403501c127907a719f2b6503792cffe606f02062d0f2f31e179a45395e4460c47245de5461c1b78496208ad5ddf68437f7889558
6
+ metadata.gz: 5104d4d97f12a70e7f2d492dee6f8f870c388fea8091bc26d60fd87efb826331479a5efa851de7cbeb278d48aad9368d12cbad7bbb675b9cd9a20baee50f087e
7
+ data.tar.gz: 3ff6caa7e64806a1f03f918b7a6692e25c7493aa6280c0f9edabc99dd937e8890d76f3e8e0f060584964b993791c990621dc3cd8c908f91b5e81a25ac3e9f5ff
@@ -1,4 +1,4 @@
1
- Metrics/LineLength:
1
+ Layout/LineLength:
2
2
  Max: 100
3
3
  Exclude:
4
4
  - pdi.gemspec
@@ -0,0 +1,19 @@
1
+ # 2.0.0 (May 7th, 2020)
2
+
3
+ Breaking Changes:
4
+
5
+ * Standard error and output have been combined into one stream (out). Early feedback indicated that reading both at the same time was preferable.
6
+
7
+ Enhancements:
8
+
9
+ * Added optional `timeout_in_seconds` argument to Pdi::Spoon#initialize.
10
+
11
+ # 1.0.1 (February 19th, 2020)
12
+
13
+ Fixes:
14
+
15
+ * It now properly calls kitchen for jobs (it was previously calling pan.)
16
+
17
+ # 1.0.0 (February 19th, 2020)
18
+
19
+ Initial release.
data/README.md CHANGED
@@ -1,4 +1,4 @@
1
- # Pdi
1
+ # PDI
2
2
 
3
3
  ---
4
4
 
@@ -36,17 +36,21 @@ Pull Requests are welcome for:
36
36
 
37
37
  ## Examples
38
38
 
39
- All examples assume Pdi has been installed to your home directory: `~/data-integration`.
39
+ All examples assume PDI has been installed to your home directory: `~/data-integration`.
40
40
 
41
41
  ### Creating a Spoon Instance
42
42
 
43
- `Pdi::Spoon` is the common interface you will use when interacting with Pdi. It will use Pan and Kitchen for executing Spoon commands.
43
+ `Pdi::Spoon` is the common interface you will use when interacting with PDI. It will use Pan and Kitchen for executing Spoon commands.
44
44
 
45
45
  ```ruby
46
- spoon = Pdi::Spoon.new(dir: `~/data-integration`)
46
+ spoon = Pdi::Spoon.new(dir: '~/data-integration')
47
47
  ```
48
48
 
49
- Note: You can also override the names of the scripts using the `kitchen` and `pan` constructor keyword arguments. The defaults are `kitchen.sh` and `pan.sh`, respectively.
49
+ Notes:
50
+
51
+ * You can also override the names of the scripts using the `kitchen` and `pan` constructor keyword arguments. The defaults are `kitchen.sh` and `pan.sh`, respectively.
52
+ * For other command line arguments that are not supported first-class in the Options objects below you can utilize the `args` argument when instantiating a `Spoon` instance.
53
+ * Another optional argument is `timeout_in_seconds`. If set it will ensure the sub-process runs within a given window. If it times out the sub-process will be terminated and a Timeout::Error will be raised.
50
54
 
51
55
  ### Executing a Job/Transformation
52
56
 
@@ -71,7 +75,7 @@ result = spoon.run(options)
71
75
 
72
76
  You can access the raw command line results by tapping into the execution attribute of the result or error object.
73
77
 
74
- Note: Not all options are currently supported. See Pdi's official references for [Pan](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/000) and [Kitchen](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/010) to see all options.
78
+ Note: Not all options are currently supported. See PDI's official references for [Pan](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/000) and [Kitchen](https://help.pentaho.com/Documentation/6.1/0L0/0Y0/070/010) to see all options.
75
79
 
76
80
  ## Contributing
77
81
 
data/lib/pdi.rb CHANGED
@@ -8,8 +8,10 @@
8
8
  #
9
9
 
10
10
  require 'acts_as_hashable'
11
+ require 'English'
11
12
  require 'forwardable'
12
13
  require 'open3'
14
+ require 'timeout'
13
15
 
14
16
  require_relative 'pdi/executor'
15
17
  require_relative 'pdi/spoon'
@@ -13,20 +13,42 @@ module Pdi
13
13
  # This class is the library's "metal" layer, the one which actually makes the system call and
14
14
  # interacts with the operating system (through Ruby's standard library.)
15
15
  class Executor
16
+ attr_reader :timeout_in_seconds
17
+
18
+ def initialize(timeout_in_seconds: nil)
19
+ @timeout_in_seconds = timeout_in_seconds
20
+
21
+ freeze
22
+ end
23
+
16
24
  def run(args)
17
25
  args = Array(args).map(&:to_s)
18
26
 
19
- out, err, status = Open3.capture3(*args)
20
-
21
- Result.new(
22
- args: args,
23
- status: {
24
- code: status.exitstatus,
25
- out: out,
26
- err: err,
27
- pid: status.pid
28
- }
29
- )
27
+ IO.popen(args, err: %i[child out]) do |io|
28
+ begin
29
+ io_read =
30
+ if timeout_in_seconds
31
+ Timeout.timeout(timeout_in_seconds) { io.read }
32
+ else
33
+ io.read
34
+ end
35
+
36
+ io.close
37
+ status = $CHILD_STATUS
38
+
39
+ Result.new(
40
+ args: args,
41
+ status: {
42
+ code: status.exitstatus,
43
+ out: io_read,
44
+ pid: status.pid
45
+ }
46
+ )
47
+ rescue Timeout::Error => e
48
+ Process.kill(9, io.pid)
49
+ raise e
50
+ end
51
+ end
30
52
  end
31
53
  end
32
54
  end
@@ -18,7 +18,7 @@ module Pdi
18
18
 
19
19
  attr_reader :args, :status
20
20
 
21
- def_delegators :status, :code, :out, :err, :pid
21
+ def_delegators :status, :code, :out, :pid
22
22
 
23
23
  def initialize(args:, status: {})
24
24
  @args = args
@@ -13,12 +13,11 @@ module Pdi
13
13
  class Status
14
14
  acts_as_hashable
15
15
 
16
- attr_reader :code, :out, :err, :pid
16
+ attr_reader :code, :out, :pid
17
17
 
18
- def initialize(code:, out: '', err: '', pid:)
18
+ def initialize(code:, out: '', pid:)
19
19
  @code = code
20
20
  @out = out
21
- @err = err
22
21
  @pid = pid
23
22
 
24
23
  freeze
@@ -7,33 +7,41 @@
7
7
  # LICENSE file in the root directory of this source tree.
8
8
  #
9
9
 
10
+ require_relative 'spoon/kitchen_error'
10
11
  require_relative 'spoon/options'
12
+ require_relative 'spoon/pan_error'
11
13
  require_relative 'spoon/parser'
12
14
  require_relative 'spoon/result'
13
15
 
14
16
  module Pdi
15
- # This class is the main wrapper for Pdi's pan and kitchen scripts.
17
+ # This class is the main wrapper for PDI's pan and kitchen scripts.
16
18
  class Spoon
17
19
  DEFAULT_KITCHEN = 'kitchen.sh'
18
20
  DEFAULT_PAN = 'pan.sh'
19
21
 
22
+ TYPES_TO_ERRORS = {
23
+ Options::Type::JOB => KitchenError,
24
+ Options::Type::TRANSFORMATION => PanError
25
+ }.freeze
26
+
20
27
  attr_reader :args, :dir, :kitchen, :pan
21
28
 
22
29
  def initialize(
23
30
  args: [],
24
31
  dir:,
25
32
  kitchen: DEFAULT_KITCHEN,
26
- pan: DEFAULT_PAN
33
+ pan: DEFAULT_PAN,
34
+ timeout_in_seconds: nil
27
35
  )
28
36
  assert_required(:dir, dir)
29
37
  assert_required(:kitchen, kitchen)
30
38
  assert_required(:pan, pan)
31
39
 
32
40
  @args = Array(args)
33
- @dir = dir.to_s
41
+ @dir = File.expand_path(dir.to_s)
34
42
  @kitchen = kitchen.to_s
35
43
  @pan = pan.to_s
36
- @executor = Executor.new
44
+ @executor = Executor.new(timeout_in_seconds: timeout_in_seconds)
37
45
  @parser = Parser.new
38
46
 
39
47
  freeze
@@ -55,11 +63,11 @@ module Pdi
55
63
  # Returns an Executor::Result instance when PDI returns error code 0 or else raises
56
64
  # a PanError (transformation) or KitchenError (job).
57
65
  def run(options)
58
- options = Options.make(options)
59
- final_args = [pan_path] + args + options.to_args
66
+ options = Options.make(options)
67
+ all_args = run_args(options)
60
68
 
61
- executor.run(final_args).tap do |result|
62
- raise(options.error_constant, result) if result.code != 0
69
+ executor.run(all_args).tap do |result|
70
+ raise(error_constant(options), result) if result.code != 0
63
71
  end
64
72
  end
65
73
 
@@ -67,6 +75,22 @@ module Pdi
67
75
 
68
76
  attr_reader :executor, :parser
69
77
 
78
+ def error_constant(options)
79
+ TYPES_TO_ERRORS.fetch(options.type)
80
+ end
81
+
82
+ def run_args(options)
83
+ [script_path(options.type)] + args + options.to_args
84
+ end
85
+
86
+ def script_path(options_type)
87
+ if options_type == Options::Type::JOB
88
+ kitchen_path
89
+ elsif options_type == Options::Type::TRANSFORMATION
90
+ pan_path
91
+ end
92
+ end
93
+
70
94
  def kitchen_path
71
95
  File.join(dir, kitchen)
72
96
  end
@@ -11,6 +11,8 @@ module Pdi
11
11
  class Spoon
12
12
  # This class subclasses Ruby's StandardError and provides a mapping to custom Kitchen
13
13
  # specific error codes to messages.
14
+ # You can find the list of errors in Pentaho's documentation site, for example:
15
+ # https://help.pentaho.com/Documentation/8.0/Products/Data_Integration/Command_Line_Tools
14
16
  class KitchenError < StandardError
15
17
  MESSAGES = {
16
18
  '1' => 'Errors occurred during processing',
@@ -7,10 +7,8 @@
7
7
  # LICENSE file in the root directory of this source tree.
8
8
  #
9
9
 
10
- require_relative 'kitchen_error'
11
10
  require_relative 'options/level'
12
11
  require_relative 'options/param'
13
- require_relative 'pan_error'
14
12
 
15
13
  module Pdi
16
14
  class Spoon
@@ -23,11 +21,6 @@ module Pdi
23
21
  TRANSFORMATION = :transformation
24
22
  end
25
23
 
26
- TYPES_TO_ERRORS = {
27
- Type::JOB => KitchenError,
28
- Type::TRANSFORMATION => PanError
29
- }.freeze
30
-
31
24
  TYPES_TO_KEYS = {
32
25
  Type::JOB => Arg::Key::JOB,
33
26
  Type::TRANSFORMATION => Arg::Key::TRANS
@@ -63,10 +56,6 @@ module Pdi
63
56
  base_args + param_args
64
57
  end
65
58
 
66
- def error_constant
67
- TYPES_TO_ERRORS.fetch(type)
68
- end
69
-
70
59
  private
71
60
 
72
61
  def key
@@ -11,6 +11,8 @@ module Pdi
11
11
  class Spoon
12
12
  # This class subclasses Ruby's StandardError and provides a mapping to custom Pan
13
13
  # specific error codes to messages.
14
+ # You can find the list of errors in Pentaho's documentation site, for example:
15
+ # https://help.pentaho.com/Documentation/8.0/Products/Data_Integration/Command_Line_Tools
14
16
  class PanError < StandardError
15
17
  MESSAGES = {
16
18
  '1' => 'Errors occurred during processing',
@@ -8,5 +8,5 @@
8
8
  #
9
9
 
10
10
  module Pdi
11
- VERSION = '1.0.0-alpha.1'
11
+ VERSION = '2.0.0-alpha'
12
12
  end
@@ -0,0 +1,10 @@
1
+ #!/bin/bash
2
+
3
+ echo 'std_out'
4
+ echo 'err_out' >&2
5
+
6
+ sleep $1
7
+
8
+ echo 'after_sleep'
9
+
10
+ exit $2
@@ -0,0 +1,52 @@
1
+ # frozen_string_literal: true
2
+
3
+ #
4
+ # Copyright (c) 2018-present, Blue Marble Payroll, LLC
5
+ #
6
+ # This source code is licensed under the MIT license found in the
7
+ # LICENSE file in the root directory of this source tree.
8
+ #
9
+
10
+ require 'spec_helper'
11
+
12
+ describe Pdi::Executor do
13
+ let(:script) { File.join('spec', 'mocks', 'spoon', 'sleep.sh') }
14
+ let(:one_second) { 1 }
15
+ let(:return_code) { 33 }
16
+
17
+ describe '#run' do
18
+ context 'with a timeout' do
19
+ # do not make these too high, bugs could cause the entire script to still be executed without
20
+ # killing it so these high values could draw out results.
21
+ let(:thirty_seconds) { 30 }
22
+
23
+ subject { described_class.new(timeout_in_seconds: one_second) }
24
+
25
+ # This will run a script that will take 30 seconds to process, but by limiting the
26
+ # timeout using the #run argument, it should raise an error after one second.
27
+ it 'times out and kills process after 5 seconds' do
28
+ args = [script, thirty_seconds]
29
+
30
+ expect { subject.run(args) }.to raise_error(Timeout::Error)
31
+ end
32
+ end
33
+
34
+ context 'without a timeout' do
35
+ it 'returns right exit status as code' do
36
+ args = [script, one_second, return_code]
37
+
38
+ result = subject.run(args)
39
+
40
+ expect(result.code).to eq(return_code)
41
+ end
42
+
43
+ it 'returns right standard output and error as out' do
44
+ args = [script, one_second, return_code]
45
+
46
+ result = subject.run(args)
47
+
48
+ expect(result.out).to eq("std_out\nerr_out\nafter_sleep\n")
49
+ end
50
+ end
51
+ end
52
+ end
@@ -0,0 +1,42 @@
1
+ # frozen_string_literal: true
2
+
3
+ #
4
+ # Copyright (c) 2018-present, Blue Marble Payroll, LLC
5
+ #
6
+ # This source code is licensed under the MIT license found in the
7
+ # LICENSE file in the root directory of this source tree.
8
+ #
9
+
10
+ require 'spec_helper'
11
+
12
+ describe Pdi::Spoon::KitchenError do
13
+ describe 'initialization' do
14
+ [1, 2, 7, 8, 9].each do |code|
15
+ specify "code #{code} should have message" do
16
+ result = Pdi::Executor::Result.new(
17
+ args: [],
18
+ status: {
19
+ code: code,
20
+ pid: 123
21
+ }
22
+ )
23
+
24
+ expect(described_class.new(result).message).not_to eq('Unknown')
25
+ end
26
+ end
27
+
28
+ [-1, 0, 3, 4, 5, 6, 10, 11].each do |code|
29
+ specify "code #{code} should not have message" do
30
+ result = Pdi::Executor::Result.new(
31
+ args: [],
32
+ status: {
33
+ code: code,
34
+ pid: 123
35
+ }
36
+ )
37
+
38
+ expect(described_class.new(result).message).to eq('Unknown')
39
+ end
40
+ end
41
+ end
42
+ end
@@ -14,6 +14,21 @@ describe Pdi::Spoon do
14
14
  let(:mocks_dir) { %w[spec mocks spoon] }
15
15
  let(:dir) { File.join(*mocks_dir) }
16
16
 
17
+ describe '#initialize' do
18
+ it 'sets executor' do
19
+ timeout_in_seconds = 987
20
+
21
+ subject = described_class.new(dir: dir, timeout_in_seconds: timeout_in_seconds)
22
+
23
+ # Private/internal testing is not recommended, but I really wanted to ensure
24
+ # this class is properly configuring the Executor instance, that way I can rely
25
+ # mainly on the Executor unit tests instead of integration tests at this level.
26
+ executor = subject.send('executor')
27
+
28
+ expect(executor.timeout_in_seconds).to eq(timeout_in_seconds)
29
+ end
30
+ end
31
+
17
32
  describe '#run' do
18
33
  context 'transformations' do
19
34
  let(:options) do
@@ -24,6 +39,14 @@ describe Pdi::Spoon do
24
39
  }
25
40
  end
26
41
 
42
+ it 'will call pan script' do
43
+ subject = described_class.new(args: 0, dir: dir, pan: script)
44
+
45
+ result = subject.run(options)
46
+
47
+ expect(result.args.first).to include(script)
48
+ end
49
+
27
50
  context 'when code is 0' do
28
51
  it 'returns correct stdout, stderr and code' do
29
52
  subject = described_class.new(
@@ -35,8 +58,7 @@ describe Pdi::Spoon do
35
58
 
36
59
  result = subject.run(options)
37
60
 
38
- expect(result.out).to eq("output to stdout\n")
39
- expect(result.err).to eq("output to sterr\n")
61
+ expect(result.out).to eq("output to stdout\noutput to sterr\n")
40
62
  expect(result.code).to eq(0)
41
63
  end
42
64
  end
@@ -70,6 +92,14 @@ describe Pdi::Spoon do
70
92
  }
71
93
  end
72
94
 
95
+ it 'will call kitchen script' do
96
+ subject = described_class.new(args: 0, dir: dir, kitchen: script)
97
+
98
+ result = subject.run(options)
99
+
100
+ expect(result.args.first).to include(script)
101
+ end
102
+
73
103
  context 'when code is 0' do
74
104
  it 'returns correct stdout, stderr and code' do
75
105
  subject = described_class.new(
@@ -81,8 +111,7 @@ describe Pdi::Spoon do
81
111
 
82
112
  result = subject.run(options)
83
113
 
84
- expect(result.out).to eq("output to stdout\n")
85
- expect(result.err).to eq("output to sterr\n")
114
+ expect(result.out).to eq("output to stdout\noutput to sterr\n")
86
115
  expect(result.code).to eq(0)
87
116
  end
88
117
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pdi
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0.pre.alpha.1
4
+ version: 2.0.0.pre.alpha
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matthew Ruggio
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-02-18 00:00:00.000000000 Z
11
+ date: 2020-05-08 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: acts_as_hashable
@@ -160,11 +160,14 @@ files:
160
160
  - lib/pdi/version.rb
161
161
  - pdi.gemspec
162
162
  - spec/mocks/spoon/return_code.sh
163
+ - spec/mocks/spoon/sleep.sh
163
164
  - spec/mocks/spoon/version.sh
164
- - spec/pdi/spoon/error_spec.rb
165
+ - spec/pdi/executor_spec.rb
166
+ - spec/pdi/spoon/kitchen_error_spec.rb
165
167
  - spec/pdi/spoon/options/arg_spec.rb
166
168
  - spec/pdi/spoon/options/param_spec.rb
167
169
  - spec/pdi/spoon/options_spec.rb
170
+ - spec/pdi/spoon/pan_error_spec.rb
168
171
  - spec/pdi/spoon_spec.rb
169
172
  - spec/spec_helper.rb
170
173
  homepage: https://github.com/bluemarblepayroll/pdi
@@ -192,10 +195,13 @@ specification_version: 4
192
195
  summary: Ruby wrapper for invoking Pentaho Data Integration
193
196
  test_files:
194
197
  - spec/mocks/spoon/return_code.sh
198
+ - spec/mocks/spoon/sleep.sh
195
199
  - spec/mocks/spoon/version.sh
196
- - spec/pdi/spoon/error_spec.rb
200
+ - spec/pdi/executor_spec.rb
201
+ - spec/pdi/spoon/kitchen_error_spec.rb
197
202
  - spec/pdi/spoon/options/arg_spec.rb
198
203
  - spec/pdi/spoon/options/param_spec.rb
199
204
  - spec/pdi/spoon/options_spec.rb
205
+ - spec/pdi/spoon/pan_error_spec.rb
200
206
  - spec/pdi/spoon_spec.rb
201
207
  - spec/spec_helper.rb