json-mapping-transform 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +12 -0
- data/.rspec +3 -0
- data/.rubocop.yml +31 -0
- data/.travis.yml +7 -0
- data/CODE_OF_CONDUCT.md +74 -0
- data/Gemfile +4 -0
- data/LICENSE.txt +21 -0
- data/README.md +227 -0
- data/Rakefile +6 -0
- data/bin/console +14 -0
- data/bin/setup +8 -0
- data/json-mapping-transform.gemspec +32 -0
- data/lib/conditions.rb +178 -0
- data/lib/json_mapping.rb +184 -0
- metadata +149 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: d98b188ca6cf86ba7a386096fe708a4ca2e7986e18471e1e53c85f84baddb98e
|
4
|
+
data.tar.gz: ecdf870a41b97c13c574fdc1982e69c239d417422bfcbfb2372b02a78150141a
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 45898c72eedfca9afa4c2ba5837fb08b4002ff55da24d475325781a99483d440f5b973b808d5b5c9e9d052a8b058f69ddf876199c93c311f6733594e59cd7ddd
|
7
|
+
data.tar.gz: e980815abd5856ee5f0193b672214d113fd543138d711680f3c0aff56756b6cb8e3c62eb2b0c72bd67b71d86afce3ef1481da8e830e0e8821c7abf42c30297b7
|
data/.gitignore
ADDED
data/.rspec
ADDED
data/.rubocop.yml
ADDED
@@ -0,0 +1,31 @@
|
|
1
|
+
AllCops:
|
2
|
+
Exclude:
|
3
|
+
- 'spec/spec_helper.rb'
|
4
|
+
- 'json-mapping-transform.gemspec'
|
5
|
+
- 'vendor/**/*'
|
6
|
+
- 'coverage/**'
|
7
|
+
- 'bin/**'
|
8
|
+
- '.bundle/**'
|
9
|
+
Layout/LineLength:
|
10
|
+
Max: 140
|
11
|
+
|
12
|
+
Metrics/AbcSize:
|
13
|
+
Enabled: false
|
14
|
+
Metrics/MethodLength:
|
15
|
+
Max: 50
|
16
|
+
Metrics/ClassLength:
|
17
|
+
Max: 200
|
18
|
+
Metrics/PerceivedComplexity:
|
19
|
+
Max: 15
|
20
|
+
Metrics/CyclomaticComplexity:
|
21
|
+
Max: 15
|
22
|
+
Metrics/BlockLength:
|
23
|
+
Exclude:
|
24
|
+
- spec/**/*
|
25
|
+
|
26
|
+
Style/IfUnlessModifier:
|
27
|
+
Enabled: false
|
28
|
+
Style/FrozenStringLiteralComment:
|
29
|
+
Enabled: false
|
30
|
+
Style/SingleLineMethods:
|
31
|
+
Enabled: false
|
data/.travis.yml
ADDED
data/CODE_OF_CONDUCT.md
ADDED
@@ -0,0 +1,74 @@
|
|
1
|
+
# Contributor Covenant Code of Conduct
|
2
|
+
|
3
|
+
## Our Pledge
|
4
|
+
|
5
|
+
In the interest of fostering an open and welcoming environment, we as
|
6
|
+
contributors and maintainers pledge to making participation in our project and
|
7
|
+
our community a harassment-free experience for everyone, regardless of age, body
|
8
|
+
size, disability, ethnicity, gender identity and expression, level of experience,
|
9
|
+
nationality, personal appearance, race, religion, or sexual identity and
|
10
|
+
orientation.
|
11
|
+
|
12
|
+
## Our Standards
|
13
|
+
|
14
|
+
Examples of behavior that contributes to creating a positive environment
|
15
|
+
include:
|
16
|
+
|
17
|
+
* Using welcoming and inclusive language
|
18
|
+
* Being respectful of differing viewpoints and experiences
|
19
|
+
* Gracefully accepting constructive criticism
|
20
|
+
* Focusing on what is best for the community
|
21
|
+
* Showing empathy towards other community members
|
22
|
+
|
23
|
+
Examples of unacceptable behavior by participants include:
|
24
|
+
|
25
|
+
* The use of sexualized language or imagery and unwelcome sexual attention or
|
26
|
+
advances
|
27
|
+
* Trolling, insulting/derogatory comments, and personal or political attacks
|
28
|
+
* Public or private harassment
|
29
|
+
* Publishing others' private information, such as a physical or electronic
|
30
|
+
address, without explicit permission
|
31
|
+
* Other conduct which could reasonably be considered inappropriate in a
|
32
|
+
professional setting
|
33
|
+
|
34
|
+
## Our Responsibilities
|
35
|
+
|
36
|
+
Project maintainers are responsible for clarifying the standards of acceptable
|
37
|
+
behavior and are expected to take appropriate and fair corrective action in
|
38
|
+
response to any instances of unacceptable behavior.
|
39
|
+
|
40
|
+
Project maintainers have the right and responsibility to remove, edit, or
|
41
|
+
reject comments, commits, code, wiki edits, issues, and other contributions
|
42
|
+
that are not aligned to this Code of Conduct, or to ban temporarily or
|
43
|
+
permanently any contributor for other behaviors that they deem inappropriate,
|
44
|
+
threatening, offensive, or harmful.
|
45
|
+
|
46
|
+
## Scope
|
47
|
+
|
48
|
+
This Code of Conduct applies both within project spaces and in public spaces
|
49
|
+
when an individual is representing the project or its community. Examples of
|
50
|
+
representing a project or community include using an official project e-mail
|
51
|
+
address, posting via an official social media account, or acting as an appointed
|
52
|
+
representative at an online or offline event. Representation of a project may be
|
53
|
+
further defined and clarified by project maintainers.
|
54
|
+
|
55
|
+
## Enforcement
|
56
|
+
|
57
|
+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
58
|
+
reported by contacting the project team at parandea1.7@gmail.com. All
|
59
|
+
complaints will be reviewed and investigated and will result in a response that
|
60
|
+
is deemed necessary and appropriate to the circumstances. The project team is
|
61
|
+
obligated to maintain confidentiality with regard to the reporter of an incident.
|
62
|
+
Further details of specific enforcement policies may be posted separately.
|
63
|
+
|
64
|
+
Project maintainers who do not follow or enforce the Code of Conduct in good
|
65
|
+
faith may face temporary or permanent repercussions as determined by other
|
66
|
+
members of the project's leadership.
|
67
|
+
|
68
|
+
## Attribution
|
69
|
+
|
70
|
+
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
71
|
+
available at [http://contributor-covenant.org/version/1/4][version]
|
72
|
+
|
73
|
+
[homepage]: http://contributor-covenant.org
|
74
|
+
[version]: http://contributor-covenant.org/version/1/4/
|
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2020 Anmol Parande
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,227 @@
|
|
1
|
+
# JSON Map Transform
|
2
|
+
|
3
|
+
## Overview
|
4
|
+
When building data pipelines, it is often useful to extract and transfrom data from an input JSON and output it in a different format. The standard process for doing this in Ruby is to write a series of if-else logic coupled with for-loops. This code ends up being largely redundant, confusing, and difficult to maintain or change. This Gem provides an easy and extensible solution to this problem by allowing you to define your mapping in YAML and apply it to any JSON object in a single line of code.
|
5
|
+
|
6
|
+
The general format of the transform mapping looks as follows:
|
7
|
+
``` yaml
|
8
|
+
---
|
9
|
+
conditions:
|
10
|
+
condition_name:
|
11
|
+
class: (required)
|
12
|
+
predicate: (optional)
|
13
|
+
objects:
|
14
|
+
- name: (required)
|
15
|
+
path: (optional)
|
16
|
+
default: (optional)
|
17
|
+
attributes: (optional)
|
18
|
+
transform: (optional)
|
19
|
+
conditions: (optional)
|
20
|
+
- name: (required)
|
21
|
+
output: (optional)
|
22
|
+
field: (optional)
|
23
|
+
```
|
24
|
+
|
25
|
+
## Installation
|
26
|
+
Add the gem to the Gemfile
|
27
|
+
```ruby
|
28
|
+
gem 'json-mapping-transform'
|
29
|
+
```
|
30
|
+
Require the mapping in your code
|
31
|
+
```ruby
|
32
|
+
require 'json_mapping'
|
33
|
+
```
|
34
|
+
|
35
|
+
## Objects
|
36
|
+
Objects are the output keys of the mapping. `JsonMapper#map` will output a single Ruby Hash when applied to an input.
|
37
|
+
The following rules apply to objects:
|
38
|
+
- Each object has a name that translates to its key in the output JSON
|
39
|
+
- The **path** specifies the input key in the source JSON that the object corresponds to
|
40
|
+
- Paths are defined from the top level of the JSON: `/`
|
41
|
+
- When `*` is included in the path, the result will be an array
|
42
|
+
- When a path is not found (or not proviided), the object evaluates to `nil`
|
43
|
+
- Objects can have a **default** value which is returned if the path evaluates to `nil`
|
44
|
+
- Objects can have **attributes** which are a list of more objects (nested JSON objects)
|
45
|
+
- **Note:** Paths in nested objects are relative to the path of the top-level object
|
46
|
+
|
47
|
+
## Conditions
|
48
|
+
Conditions are `if` statements performed on an extracted value. They are defined as a hash in the mapping file.
|
49
|
+
- By default, conditions are evaluated against the object path
|
50
|
+
- If **field** is specified, the condition is evaluated against the path relative to the object path
|
51
|
+
- If the extracted value satisfies the condition, the output will be set to **output** (to the extracted value if output is not specified)
|
52
|
+
- If the extracted value does not satisfy the condition, the output will be set to the object's default
|
53
|
+
- If the extracted value is `nil`, conditions are not evaluated
|
54
|
+
- Conditions are referenced by **name** in the object definition
|
55
|
+
- If multiple conditions are defined and satisfied, the output will be an array
|
56
|
+
|
57
|
+
There are several built-in condition types which can be used for the `class` field of the condition.
|
58
|
+
- `InCondition`: Check if an object/Array is in/intersects with the **predicate**, an array
|
59
|
+
- `RegexCondition`: Check if a string matches the **predicate**, a regular expression
|
60
|
+
- `AnyCondition`: Check if an object/Array is/contains a truthy value
|
61
|
+
- `LessThanCondition`: Check if a `Numeric` is less than the **predicate**, another `Numeric`
|
62
|
+
- `GreaterThanCondition`: Check if a `Numeric` is less than the **predicate**, another `Numeric`
|
63
|
+
- `AndCondition`: Check if an object satisfies all conditions provided in the **predicate**
|
64
|
+
- `OrCondition`: Check if an object satisfies at least one condition provided in the **predicate**
|
65
|
+
- `NotCondition`: Check if an object does not satisfy the condition provided as the **predicate**
|
66
|
+
|
67
|
+
Developers can create their own custom conditions by extending `BaseCondition` inside of the `Conditions` module
|
68
|
+
|
69
|
+
## Transforms
|
70
|
+
- Transforms are arbitrary blocks of code which act on the extracted value for an object
|
71
|
+
- Transforms are applied after conditions (i.e they will only be applied if at least one condition is satisfied)
|
72
|
+
- If the extracted value is `nil` (or all conditions fail), then transforms are not evaluated
|
73
|
+
- Transforms are referenced by name in YAML. You must pass in a hash of them to the `JsonMapper` during initialization
|
74
|
+
|
75
|
+
## Failure Cases
|
76
|
+
### Graceful Failures
|
77
|
+
The mapping will gracefully fail (fall back on default) when
|
78
|
+
1. Encountering a null object in the original object
|
79
|
+
2. Encountering non-existent paths
|
80
|
+
3. Indexing an array out of bounds
|
81
|
+
|
82
|
+
### Exceptions
|
83
|
+
The mapping will raise an exception when
|
84
|
+
1. The YAML map is not formatted properly (`JsonMapper::FormatError`)
|
85
|
+
2. A condition is referenced but not defined (`Conditions::ConditionError`)
|
86
|
+
3. Unknown condition type (`NameError`)
|
87
|
+
4. A condition is defined with an incorrect predicate (`Conditions::ConditionError`)
|
88
|
+
5. A condition is given a value it can't compare to the predicate (`Conditions::ConditionError`)
|
89
|
+
6. A provided transform is not callable (`JsonMapper::TransformError`)
|
90
|
+
7. The `*` operator is used on a non-array (`JsonMapper::PathError`)
|
91
|
+
8. An exception is encountered while applying a transform (`StandardError`)
|
92
|
+
|
93
|
+
## Examples
|
94
|
+
For all the examples provided below, this is the input JSON that is being mapped:
|
95
|
+
```json
|
96
|
+
{
|
97
|
+
"name": "Trader Joe's",
|
98
|
+
"location": "Berkeley, California",
|
99
|
+
"weeklyVisitors": 5000,
|
100
|
+
"storeId": 1234,
|
101
|
+
"employees": [
|
102
|
+
{ "name": "Jim Shoes" },
|
103
|
+
{ "name": "Kay Oss" }
|
104
|
+
],
|
105
|
+
"inventory": [
|
106
|
+
{ "itemName": "Apples", "price": 0.5, "unit": "lb" },
|
107
|
+
{ "itemName": "Oranges", "price": 2, "unit": "lb" },
|
108
|
+
{ "itemName": "Bag of Carrots", "price": 1.5, "unit": "count" }
|
109
|
+
]
|
110
|
+
}
|
111
|
+
```
|
112
|
+
### Basic Example
|
113
|
+
An simple example which just converts between two objects
|
114
|
+
#### Mapping
|
115
|
+
```yaml
|
116
|
+
---
|
117
|
+
objects:
|
118
|
+
- name: name
|
119
|
+
path: "/name"
|
120
|
+
- name: profits
|
121
|
+
default: 0
|
122
|
+
- name: location
|
123
|
+
path: "/location"
|
124
|
+
- name: weekly_visitors
|
125
|
+
path: "/weeklyVisitors"
|
126
|
+
- name: store_id
|
127
|
+
path: "/storeId"
|
128
|
+
- name: employees
|
129
|
+
path: "/employees/*/name"
|
130
|
+
- name: inventory
|
131
|
+
path: "/inventory/*"
|
132
|
+
attributes:
|
133
|
+
- name: item_name
|
134
|
+
path: /itemName
|
135
|
+
- name: price
|
136
|
+
path: /price
|
137
|
+
- name: unit
|
138
|
+
path: /unit
|
139
|
+
```
|
140
|
+
#### Output
|
141
|
+
```json
|
142
|
+
{
|
143
|
+
"name": "Trader Joe\'s",
|
144
|
+
"profits": 0,
|
145
|
+
"location": "Berkeley, California",
|
146
|
+
"weekly_visitors": 5000,
|
147
|
+
"store_id": 1234,
|
148
|
+
"employees": ["Jim Shoes", "Kay Oss"],
|
149
|
+
"inventory": [
|
150
|
+
{ "item_name": "Apples", "price": 0.5, "unit": "lb" },
|
151
|
+
{ "item_name": "Oranges", "price": 2, "unit": "lb" },
|
152
|
+
{ "item_name": "Bag of Carrots", "price": 1.5, "unit": "count" }
|
153
|
+
]
|
154
|
+
}
|
155
|
+
```
|
156
|
+
### Transforms Example
|
157
|
+
An example of a custom transformation
|
158
|
+
#### Mapping
|
159
|
+
```yaml
|
160
|
+
---
|
161
|
+
objects:
|
162
|
+
- name: name
|
163
|
+
path: "/name"
|
164
|
+
- name: inventory
|
165
|
+
path: "/inventory/*/"
|
166
|
+
transform: listing_transform
|
167
|
+
```
|
168
|
+
#### Code
|
169
|
+
```ruby
|
170
|
+
transforms = {
|
171
|
+
'listing_transform' => ->(list) { list.map { |x| "#{x['itemName']} at $#{x['price']}/#{x['unit']}" } }
|
172
|
+
}
|
173
|
+
output = JsonMapping.new(path, transforms).map(store_fixture)
|
174
|
+
```
|
175
|
+
#### Output
|
176
|
+
```json
|
177
|
+
{
|
178
|
+
"name": "Trader Joe\'s",
|
179
|
+
"inventory": ["Apples at $0.5/lb", "Oranges at $2/lb", "Bag of Carrots at $1.5/count"]
|
180
|
+
}
|
181
|
+
```
|
182
|
+
### Conditions Example
|
183
|
+
An example using conditions
|
184
|
+
```yaml
|
185
|
+
---
|
186
|
+
conditions:
|
187
|
+
apple_condition:
|
188
|
+
class: AppleCondition
|
189
|
+
high_performance_condition:
|
190
|
+
class: AndCondition
|
191
|
+
predicate:
|
192
|
+
- class: LessThanCondition
|
193
|
+
predicate: 10000
|
194
|
+
- class: GreaterThanCondition
|
195
|
+
predicate: 1000
|
196
|
+
|
197
|
+
objects:
|
198
|
+
- name: performance
|
199
|
+
path: "/weeklyVisitors"
|
200
|
+
conditions:
|
201
|
+
- name: high_performance_condition
|
202
|
+
output: high
|
203
|
+
- name: apple
|
204
|
+
path: "/inventory"
|
205
|
+
conditions:
|
206
|
+
- name: apple_condition
|
207
|
+
```
|
208
|
+
#### Code
|
209
|
+
```ruby
|
210
|
+
module Conditions
|
211
|
+
class AppleCondition < BaseCondition
|
212
|
+
def apply(value)
|
213
|
+
puts value
|
214
|
+
value.is_a?(Hash) && value['itemName'] == 'Apples'
|
215
|
+
end
|
216
|
+
end
|
217
|
+
end
|
218
|
+
|
219
|
+
output = JsonMapping.new(path).map(store_fixture)
|
220
|
+
```
|
221
|
+
#### Output
|
222
|
+
```json
|
223
|
+
{
|
224
|
+
"performance": "high",
|
225
|
+
"apple": [{ "itemName": "Apples", "price": 0.5, "unit": "lb" }]
|
226
|
+
}
|
227
|
+
```
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "json_mapping"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
10
|
+
# require "pry"
|
11
|
+
# Pry.start
|
12
|
+
|
13
|
+
require "irb"
|
14
|
+
IRB.start(__FILE__)
|
data/bin/setup
ADDED
@@ -0,0 +1,32 @@
|
|
1
|
+
lib = File.expand_path("lib", __dir__)
|
2
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
3
|
+
|
4
|
+
Gem::Specification.new do |spec|
|
5
|
+
spec.name = 'json-mapping-transform'
|
6
|
+
spec.version = '0.1.0'
|
7
|
+
spec.authors = ['Anmol Parande']
|
8
|
+
spec.email = ['parande.anmol@gmail.com']
|
9
|
+
|
10
|
+
spec.summary = 'Map one JSON format into another JSON format'
|
11
|
+
spec.description = 'When building data pipelines, it is often useful to extract and transfrom data from an input JSON and output it in a different format. The standard process for doing this in Ruby is to write a series of if-else logic coupled with for-loops. This code ends up being largely redundant, confusing, and difficult to maintain or change. This Gem provides an easy and extensible solution to this problem by allowing you to define your mapping in YAML and apply it to any JSON object in a single line of code.'
|
12
|
+
spec.homepage = 'https://github.com/aparande/json-mapping-transform'
|
13
|
+
spec.license = 'MIT'
|
14
|
+
|
15
|
+
spec.metadata['homepage_uri'] = spec.homepage
|
16
|
+
spec.metadata['source_code_uri'] = 'https://github.com/aparande/json-mapping-transform'
|
17
|
+
# spec.metadata['changelog_uri'] = "TODO: Put your gem's CHANGELOG.md URL here."
|
18
|
+
|
19
|
+
# Specify which files should be added to the gem when it is released.
|
20
|
+
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
21
|
+
spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
|
22
|
+
`git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
23
|
+
end
|
24
|
+
spec.require_paths = ['lib']
|
25
|
+
|
26
|
+
spec.add_development_dependency 'bundler', '~> 2.0'
|
27
|
+
spec.add_development_dependency 'rake', '~> 10.0'
|
28
|
+
spec.add_development_dependency 'redcarpet'
|
29
|
+
spec.add_development_dependency 'rspec', '~> 3.0'
|
30
|
+
spec.add_development_dependency 'simplecov'
|
31
|
+
spec.add_development_dependency 'yard'
|
32
|
+
end
|
data/lib/conditions.rb
ADDED
@@ -0,0 +1,178 @@
|
|
1
|
+
##
|
2
|
+
# Stores conditions accessible to +JsonMapper+
|
3
|
+
module Conditions
|
4
|
+
##
|
5
|
+
# Thrown when a condition encounters an error
|
6
|
+
class ConditionError < StandardError; end
|
7
|
+
|
8
|
+
##
|
9
|
+
# Abstract class from which all conditions inherit
|
10
|
+
class BaseCondition
|
11
|
+
##
|
12
|
+
# @param [Any] predicate A predicate value which will be compared against input values
|
13
|
+
def initialize(predicate)
|
14
|
+
@predicate = predicate
|
15
|
+
end
|
16
|
+
|
17
|
+
def apply(_) true; end
|
18
|
+
end
|
19
|
+
|
20
|
+
##
|
21
|
+
# Checks if an element (or elements of an array) belong to an array
|
22
|
+
# Can be used as an "equals to" condition
|
23
|
+
class InCondition < BaseCondition
|
24
|
+
##
|
25
|
+
# @param [Array] predicate An array predicate
|
26
|
+
def initialize(predicate)
|
27
|
+
raise ConditionError, "In condition value must be an Array, not #{predicate.class}" unless predicate.is_a? Array
|
28
|
+
|
29
|
+
super(predicate)
|
30
|
+
end
|
31
|
+
|
32
|
+
##
|
33
|
+
# @param [Any, Array] value A value to be checked against the predicate
|
34
|
+
# @return [true] if value and predicate have overlapping values
|
35
|
+
# @return [false] if value and predicate have no overlapping values
|
36
|
+
def apply(value)
|
37
|
+
value = [value] unless value.is_a? Array
|
38
|
+
(value & @predicate).any?
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
##
|
43
|
+
# Compares a value against a regular expression
|
44
|
+
class RegexCondition < BaseCondition
|
45
|
+
##
|
46
|
+
# @param [String] predicate A valid regular expression
|
47
|
+
def initialize(predicate)
|
48
|
+
predicate = Regexp.new(predicate.to_s)
|
49
|
+
super(predicate)
|
50
|
+
rescue RegexpError => e
|
51
|
+
raise ConditionError, e.inspect
|
52
|
+
end
|
53
|
+
|
54
|
+
##
|
55
|
+
# @param [String] value
|
56
|
+
# @return [true] if value matches regular expression
|
57
|
+
# @return [false] if value does not match regular expression
|
58
|
+
def apply(value)
|
59
|
+
@predicate.match?(value)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
##
|
64
|
+
# Checks if any values in the array are true
|
65
|
+
class AnyCondition < BaseCondition
|
66
|
+
##
|
67
|
+
# @param [Any, Array] value
|
68
|
+
# @return [true] if value is truthy (or has at least one truthy value)
|
69
|
+
# @return [false] if the value is not truthy (or has no truthy values)
|
70
|
+
def apply(value)
|
71
|
+
value = [value] unless value.is_a? Array
|
72
|
+
value.any?
|
73
|
+
end
|
74
|
+
end
|
75
|
+
|
76
|
+
##
|
77
|
+
# Checks if two conditions are true
|
78
|
+
class AndCondition < BaseCondition
|
79
|
+
##
|
80
|
+
# @param [Array] predicate An array of Hashes representating conditions (Greater than length 2)
|
81
|
+
def initialize(predicate)
|
82
|
+
raise ConditionError, 'And condition predicate must be an array of conditions' unless predicate.is_a?(Array) && predicate.length > 1
|
83
|
+
|
84
|
+
predicate.map! { |x| Object.const_get("Conditions::#{x['class']}").new(x['predicate']) }
|
85
|
+
super(predicate)
|
86
|
+
end
|
87
|
+
|
88
|
+
##
|
89
|
+
# @param [Any] value
|
90
|
+
# @return [true] if +value+ matches all conditions
|
91
|
+
# @return [false] if +value+ does not match one condition
|
92
|
+
def apply(value)
|
93
|
+
@predicate.map { |x| x.apply(value) }.all?
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
##
|
98
|
+
# Checks if either or both of two conditions are true
|
99
|
+
class OrCondition < BaseCondition
|
100
|
+
##
|
101
|
+
# @param [Array] predicate An array of Hashes representating conditions (Greater than length 2)
|
102
|
+
def initialize(predicate)
|
103
|
+
raise ConditionError, 'Or condition predicate must be an array of conditions' unless predicate.is_a?(Array) && predicate.length > 1
|
104
|
+
|
105
|
+
predicate.map! { |x| Object.const_get("Conditions::#{x['class']}").new(x['predicate']) }
|
106
|
+
super(predicate)
|
107
|
+
end
|
108
|
+
|
109
|
+
##
|
110
|
+
# @param [Any] value
|
111
|
+
# @return [true] if +value+ matches any condition
|
112
|
+
# @return [false] if +value+ matches no conditions
|
113
|
+
def apply(value)
|
114
|
+
@predicate.map { |x| x.apply(value) }.any?
|
115
|
+
end
|
116
|
+
end
|
117
|
+
|
118
|
+
##
|
119
|
+
# Checks if a condition is not true
|
120
|
+
class NotCondition < BaseCondition
|
121
|
+
##
|
122
|
+
# @param [Hash] predicate A hash representing a condition
|
123
|
+
def initialize(predicate)
|
124
|
+
raise ConditionError, 'Not condition predicate a condition' unless predicate.is_a?(Hash) && predicate.key?('class')
|
125
|
+
|
126
|
+
predicate = Object.const_get("Conditions::#{predicate['class']}").new(predicate['predicate'])
|
127
|
+
super(predicate)
|
128
|
+
end
|
129
|
+
|
130
|
+
##
|
131
|
+
# @param [Any] value
|
132
|
+
# @return [true] if +value+ does not satisfy the condition
|
133
|
+
# @return [false] if +value+ satisfies the condition
|
134
|
+
def apply(value)
|
135
|
+
!@predicate.apply(value)
|
136
|
+
end
|
137
|
+
end
|
138
|
+
|
139
|
+
##
|
140
|
+
# Checks if value is less than a predicate
|
141
|
+
class LessThanCondition < BaseCondition
|
142
|
+
##
|
143
|
+
# @param [Numeric] predicate
|
144
|
+
def initialize(predicate)
|
145
|
+
raise ConditionError, 'LessThan condition predicate must a number' unless predicate.is_a?(Numeric)
|
146
|
+
|
147
|
+
super(predicate)
|
148
|
+
end
|
149
|
+
|
150
|
+
##
|
151
|
+
# @param [Numeric] value
|
152
|
+
# @return [true] if +value < predicate+
|
153
|
+
# @return [false] if +value >= predicate+
|
154
|
+
def apply(value)
|
155
|
+
value < @predicate
|
156
|
+
end
|
157
|
+
end
|
158
|
+
|
159
|
+
##
|
160
|
+
# Checks if value is greater than a predicate
|
161
|
+
class GreaterThanCondition < BaseCondition
|
162
|
+
##
|
163
|
+
# @param [Numeric] predicate
|
164
|
+
def initialize(predicate)
|
165
|
+
raise ConditionError, 'GreaterThan condition predicate must a number' unless predicate.is_a?(Numeric)
|
166
|
+
|
167
|
+
super(predicate)
|
168
|
+
end
|
169
|
+
|
170
|
+
##
|
171
|
+
# @param [Numeric] value
|
172
|
+
# @return [true] if +value > predicate+
|
173
|
+
# @return [false] if +value <= predicate+
|
174
|
+
def apply(value)
|
175
|
+
value > @predicate
|
176
|
+
end
|
177
|
+
end
|
178
|
+
end
|
data/lib/json_mapping.rb
ADDED
@@ -0,0 +1,184 @@
|
|
1
|
+
require 'yaml'.freeze
|
2
|
+
require 'logger'.freeze
|
3
|
+
require 'conditions'.freeze
|
4
|
+
|
5
|
+
##
|
6
|
+
# Stores and applies a mapping to an input ruby Hash
|
7
|
+
class JsonMapping
|
8
|
+
##
|
9
|
+
# Thrown when a transform is not found or not callable
|
10
|
+
class TransformError < StandardError; end
|
11
|
+
##
|
12
|
+
# Thrown when parsing an invalid path
|
13
|
+
class PathError < StandardError; end
|
14
|
+
##
|
15
|
+
# Thrown when the YAML transform is not formatted properly
|
16
|
+
class FormatError < StandardError; end
|
17
|
+
|
18
|
+
##
|
19
|
+
# @param [String] schema_path The path to the YAML schema
|
20
|
+
# @param [Hash] transforms A hash of callable objects (Procs/Lambdas). Keys must match transform names specified in YAML
|
21
|
+
def initialize(schema_path, transforms = {})
|
22
|
+
schema = YAML.safe_load(File.read(schema_path))
|
23
|
+
|
24
|
+
@conditions = (schema['conditions'] || {}).map do |key, value|
|
25
|
+
[key, Object.const_get("Conditions::#{value['class']}").new(value['predicate'])]
|
26
|
+
end.to_h
|
27
|
+
|
28
|
+
@object_schemas = schema['objects']
|
29
|
+
@transforms = transforms || {}
|
30
|
+
@logger = Logger.new($stdout)
|
31
|
+
end
|
32
|
+
|
33
|
+
##
|
34
|
+
# @param [Hash] input_hash A ruby hash onto which the schema should be applied
|
35
|
+
# @return [Array] An array of output hashes representing the mapped objects
|
36
|
+
def apply(input_hash)
|
37
|
+
raise FormatError, 'Must define objects under the \'objects\' name' if @object_schemas.nil?
|
38
|
+
|
39
|
+
@object_schemas.map { |schema| parse_object(input_hash, schema) }.reduce(&:merge)
|
40
|
+
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
##
|
45
|
+
# Maps an object schema to an object in the output
|
46
|
+
# @param [Hash] input_hash The hash onto which the schema should be mapped
|
47
|
+
# @param [Hash] schema A hash representing the schema which should be applied to the input
|
48
|
+
# Raises +FormatError+ if +schema+ is not a +Hash+ or has no key +name+
|
49
|
+
# @return [Hash] The output object
|
50
|
+
def parse_object(input_hash, schema)
|
51
|
+
raise FormatError, "Object should be a hash: #{schema}" unless schema.is_a? Hash
|
52
|
+
raise FormatError, "Object needs a name: #{schema}" unless schema.key?('name')
|
53
|
+
|
54
|
+
output = {}
|
55
|
+
# Its an object
|
56
|
+
if schema.key?('attributes')
|
57
|
+
output[schema['name']] = schema['default']
|
58
|
+
|
59
|
+
object_hash = parse_path(input_hash, schema['path'])
|
60
|
+
return output if object_hash.nil?
|
61
|
+
|
62
|
+
unless object_hash.is_a? Array
|
63
|
+
object_hash = [object_hash]
|
64
|
+
end
|
65
|
+
|
66
|
+
attrs = []
|
67
|
+
object_hash.each do |obj|
|
68
|
+
attributes_hash = {}
|
69
|
+
schema['attributes'].each do |attribute|
|
70
|
+
attr_hash = parse_object(obj, attribute)
|
71
|
+
attributes_hash = attributes_hash.merge(attr_hash)
|
72
|
+
end
|
73
|
+
attrs << attributes_hash
|
74
|
+
end
|
75
|
+
|
76
|
+
output[schema['name']] = attrs.length == 1 && schema['path'][-1] != '*' ? attrs[0] : attrs
|
77
|
+
else # Its a value
|
78
|
+
output = map_value(input_hash, schema)
|
79
|
+
end
|
80
|
+
|
81
|
+
output
|
82
|
+
end
|
83
|
+
|
84
|
+
##
|
85
|
+
# Maps a schema to a single field in the output schema
|
86
|
+
# @param [Hash] input_hash The input hash to be mapped
|
87
|
+
# @param [Hash] schema The schema which should be applied
|
88
|
+
# @return [Hash] A Hash which represents the applied schema
|
89
|
+
def map_value(input_hash, schema)
|
90
|
+
raise FormatError, "Schema should be a hash: #{schema}" unless schema.is_a? Hash
|
91
|
+
|
92
|
+
output = {}
|
93
|
+
output[schema['name']] = schema['default']
|
94
|
+
return output if schema['path'].nil?
|
95
|
+
|
96
|
+
value = parse_path(input_hash, schema['path'])
|
97
|
+
return output if value.nil?
|
98
|
+
|
99
|
+
if schema.key?('conditions')
|
100
|
+
value = apply_conditions(value, schema['conditions']) || output[schema['name']]
|
101
|
+
end
|
102
|
+
|
103
|
+
if schema.key?('transform') && value != output[schema['name']]
|
104
|
+
raise TransformError, "Undefined transform named #{schema['transform']}" unless @transforms.key?(schema['transform'])
|
105
|
+
raise TransformError, 'Transforms should respond to the \'call\' method' unless @transforms[schema['transform']].respond_to?(:call)
|
106
|
+
|
107
|
+
value = @transforms[schema['transform']].call(value)
|
108
|
+
end
|
109
|
+
|
110
|
+
output[schema['name']] = value
|
111
|
+
output
|
112
|
+
end
|
113
|
+
|
114
|
+
##
|
115
|
+
# @param [Hash] input_hash The input hash
|
116
|
+
# @param [String] path The path at which to grab the value
|
117
|
+
# @return [Any] The value at the particular path
|
118
|
+
def parse_path(input_hash, path)
|
119
|
+
raise ArgumentError, "path must be string, not #{path.class}" unless path.is_a? String
|
120
|
+
|
121
|
+
parts = path.split('/')
|
122
|
+
value = input_hash
|
123
|
+
|
124
|
+
parts.each_with_index do |part, idx|
|
125
|
+
if value.nil?
|
126
|
+
@logger.warn("Could not find #{path} in #{input_hash}")
|
127
|
+
break
|
128
|
+
end
|
129
|
+
|
130
|
+
if part == '*'
|
131
|
+
raise PathError, "#{parts[0, idx].join('/')} in #{input_hash} is not an array" unless value.is_a? Array
|
132
|
+
|
133
|
+
return value.map { |obj| parse_path(obj, parts[idx + 1..-1].join('/')) }
|
134
|
+
else
|
135
|
+
next if part.empty?
|
136
|
+
|
137
|
+
if value.is_a? Array
|
138
|
+
part = part.to_i
|
139
|
+
|
140
|
+
if part >= value.length
|
141
|
+
@logger.warn("Index went out of bounds while parsing #{path} in #{input_hash}")
|
142
|
+
value = nil
|
143
|
+
break
|
144
|
+
end
|
145
|
+
end
|
146
|
+
|
147
|
+
value = value[part]
|
148
|
+
end
|
149
|
+
end
|
150
|
+
value
|
151
|
+
end
|
152
|
+
|
153
|
+
##
|
154
|
+
# Applies conditions to a value
|
155
|
+
# @param [Any] value A value to compare the condition predicates against
|
156
|
+
# @param [Array] conds An array of conditions
|
157
|
+
# @return [Array] If multiple conditions are satisfied
|
158
|
+
# @return [Any] If one condition is satisfied
|
159
|
+
# @return [nil] If no conditions are satisfied
|
160
|
+
def apply_conditions(value, conds)
|
161
|
+
output = []
|
162
|
+
conds.each do |cond|
|
163
|
+
input_val = value
|
164
|
+
raise FormatError, "Conditions are a hash: #{cond}" unless cond.is_a? Hash
|
165
|
+
raise Conditions::ConditionError, "Unknown condition named #{cond['name']}" unless @conditions.key?(cond['name'])
|
166
|
+
|
167
|
+
condition = @conditions[cond['name']]
|
168
|
+
|
169
|
+
input_val = [input_val] unless input_val.is_a? Array
|
170
|
+
input_val = input_val.select do |x|
|
171
|
+
x = parse_path(x, cond['field']) if cond.key?('field')
|
172
|
+
condition.apply(x)
|
173
|
+
end
|
174
|
+
|
175
|
+
next if input_val.empty?
|
176
|
+
|
177
|
+
# Maintain the original data-type of the value (i.e Array or single element)
|
178
|
+
input_val = input_val[0] if input_val.length == 1 && !value.is_a?(Array)
|
179
|
+
output << (cond['output'] || input_val)
|
180
|
+
end
|
181
|
+
|
182
|
+
return (output.length == 1 ? output[0] : output) unless output.empty?
|
183
|
+
end
|
184
|
+
end
|
metadata
ADDED
@@ -0,0 +1,149 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: json-mapping-transform
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Anmol Parande
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2020-08-21 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: bundler
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '2.0'
|
20
|
+
type: :development
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '2.0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: rake
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '10.0'
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '10.0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: redcarpet
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rspec
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '3.0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '3.0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: simplecov
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: yard
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ">="
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
type: :development
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
description: When building data pipelines, it is often useful to extract and transfrom
|
98
|
+
data from an input JSON and output it in a different format. The standard process
|
99
|
+
for doing this in Ruby is to write a series of if-else logic coupled with for-loops.
|
100
|
+
This code ends up being largely redundant, confusing, and difficult to maintain
|
101
|
+
or change. This Gem provides an easy and extensible solution to this problem by
|
102
|
+
allowing you to define your mapping in YAML and apply it to any JSON object in a
|
103
|
+
single line of code.
|
104
|
+
email:
|
105
|
+
- parande.anmol@gmail.com
|
106
|
+
executables: []
|
107
|
+
extensions: []
|
108
|
+
extra_rdoc_files: []
|
109
|
+
files:
|
110
|
+
- ".gitignore"
|
111
|
+
- ".rspec"
|
112
|
+
- ".rubocop.yml"
|
113
|
+
- ".travis.yml"
|
114
|
+
- CODE_OF_CONDUCT.md
|
115
|
+
- Gemfile
|
116
|
+
- LICENSE.txt
|
117
|
+
- README.md
|
118
|
+
- Rakefile
|
119
|
+
- bin/console
|
120
|
+
- bin/setup
|
121
|
+
- json-mapping-transform.gemspec
|
122
|
+
- lib/conditions.rb
|
123
|
+
- lib/json_mapping.rb
|
124
|
+
homepage: https://github.com/aparande/json-mapping-transform
|
125
|
+
licenses:
|
126
|
+
- MIT
|
127
|
+
metadata:
|
128
|
+
homepage_uri: https://github.com/aparande/json-mapping-transform
|
129
|
+
source_code_uri: https://github.com/aparande/json-mapping-transform
|
130
|
+
post_install_message:
|
131
|
+
rdoc_options: []
|
132
|
+
require_paths:
|
133
|
+
- lib
|
134
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
135
|
+
requirements:
|
136
|
+
- - ">="
|
137
|
+
- !ruby/object:Gem::Version
|
138
|
+
version: '0'
|
139
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
140
|
+
requirements:
|
141
|
+
- - ">="
|
142
|
+
- !ruby/object:Gem::Version
|
143
|
+
version: '0'
|
144
|
+
requirements: []
|
145
|
+
rubygems_version: 3.0.3
|
146
|
+
signing_key:
|
147
|
+
specification_version: 4
|
148
|
+
summary: Map one JSON format into another JSON format
|
149
|
+
test_files: []
|