json_data_extractor 0.0.1 → 0.0.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +62 -12
- data/json_data_extractor.gemspec +22 -19
- data/lib/json_data_extractor.rb +27 -13
- data/lib/src/version.rb +1 -1
- metadata +10 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 101c20f78189fbd300dedf5a03779d19558d65fb4f4d2d4767ec8b1f638aafde
|
4
|
+
data.tar.gz: 924d26135a261281c14051ab7aa305debcec61688fd6deda4aeb322da8028ecd
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: df55bd36b0e00ac0bd4f427451dcf69425538dc06382632b261ec03aaa6cb24b0648873645b80c609b1e7329e3f9aac85fa97612afdcc492efb6efa91d062f83
|
7
|
+
data.tar.gz: df3a0c646a823792fa1bce461cab8eeb037b6be9906fbb1f5608cf69da4e13133c11f2e71aa90033b3924ea4ccc11f6a9900ca17b263c44702583c6eb3b679fa
|
data/README.md
CHANGED
@@ -1,7 +1,13 @@
|
|
1
1
|
# JsonDataExtractor
|
2
2
|
|
3
|
-
|
4
|
-
|
3
|
+
NOTE: This is still a very early beta.
|
4
|
+
|
5
|
+
Transform JSON data structures with the help of a simple schema and JsonPath expressions.
|
6
|
+
Use the JsonDataExtractor gem to extract and modify data from complex JSON structures using a straightforward syntax
|
7
|
+
and a range of built-in or custom modifiers.
|
8
|
+
|
9
|
+
_Another try to make something for JSON that is XSLT for XML.
|
10
|
+
We transform one JSON into another JSON with the help of a third JSON!!!111!!eleventy!!_
|
5
11
|
|
6
12
|
Remap one JSON structure into another with some basic rules and [jsonpath](https://github.com/joshbuddy/jsonpath).
|
7
13
|
|
@@ -25,9 +31,11 @@ Or install it yourself as:
|
|
25
31
|
|
26
32
|
## Usage
|
27
33
|
|
28
|
-
|
34
|
+
JsonDataExtractor allows you to remap one JSON structure into another with some basic rules
|
35
|
+
and [JSONPath](https://goessner.net/articles/JsonPath/) expressions. The process involves defining a schema that maps
|
36
|
+
the input JSON structure to the desired output structure.
|
29
37
|
|
30
|
-
|
38
|
+
We'll base our examples on the following source:
|
31
39
|
|
32
40
|
```json
|
33
41
|
{
|
@@ -68,7 +76,17 @@ With the following source:
|
|
68
76
|
}
|
69
77
|
```
|
70
78
|
|
71
|
-
|
79
|
+
### Defining a Schema
|
80
|
+
|
81
|
+
A schema consists of one or more mappings that specify how to extract data from the input JSON and where to place it in
|
82
|
+
the output JSON.
|
83
|
+
|
84
|
+
Each mapping has a path field that specifies the JsonPath expression to use for data extraction, and an optional
|
85
|
+
modifier field that specifies one or more modifiers to apply to the extracted data. Modifiers are used to transform the
|
86
|
+
data in some way before placing it in the output JSON.
|
87
|
+
|
88
|
+
Here's an example schema that extracts the authors and categories from a JSON structure similar to the one used in the
|
89
|
+
previous example (here it's in YAML just for readability):
|
72
90
|
|
73
91
|
```yaml
|
74
92
|
schemas:
|
@@ -77,7 +95,9 @@ schemas:
|
|
77
95
|
modifier: downcase
|
78
96
|
categories: $..category
|
79
97
|
```
|
98
|
+
|
80
99
|
The resulting json will be:
|
100
|
+
|
81
101
|
```json
|
82
102
|
{
|
83
103
|
"authors": [
|
@@ -96,22 +116,50 @@ The resulting json will be:
|
|
96
116
|
|
97
117
|
```
|
98
118
|
|
99
|
-
Modifiers can be supplied on object creation and/or added later by calling `#add_modifier` method.
|
119
|
+
Modifiers can be supplied on object creation and/or added later by calling `#add_modifier` method. Please see specs for
|
120
|
+
examples.
|
100
121
|
|
101
|
-
|
122
|
+
### Nested schemas
|
123
|
+
|
124
|
+
JDE supports nested schemas. Just provide your element with a type of `array` and add a `schema` key for its data.
|
102
125
|
|
103
|
-
|
126
|
+
E.g. this is a valid real-life schema with nested data:
|
127
|
+
|
128
|
+
```ruby
|
129
|
+
{
|
130
|
+
name: '$.Name',
|
131
|
+
code: '$.Code',
|
132
|
+
services: '$.Services[*].Code',
|
133
|
+
locations: {
|
134
|
+
path: '$.Locations[*]',
|
135
|
+
type: 'array',
|
136
|
+
schema: {
|
137
|
+
name: '$.Name',
|
138
|
+
type: '$.Type',
|
139
|
+
code: '$.Code'
|
140
|
+
}
|
141
|
+
}
|
142
|
+
}
|
143
|
+
```
|
144
|
+
|
145
|
+
## TODO
|
104
146
|
|
147
|
+
Update this readme for better usage cases. Add info on arrays and modifiers.
|
105
148
|
|
106
149
|
## Development
|
107
150
|
|
108
|
-
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can
|
151
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can
|
152
|
+
also run `bin/console` for an interactive prompt that will allow you to experiment.
|
109
153
|
|
110
|
-
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
|
154
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
|
155
|
+
version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version,
|
156
|
+
push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
111
157
|
|
112
158
|
## Contributing
|
113
159
|
|
114
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/austerlitz/json_data_extractor. This project
|
160
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/austerlitz/json_data_extractor. This project
|
161
|
+
is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to
|
162
|
+
the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
115
163
|
|
116
164
|
## License
|
117
165
|
|
@@ -119,4 +167,6 @@ The gem is available as open source under the terms of the [MIT License](https:/
|
|
119
167
|
|
120
168
|
## Code of Conduct
|
121
169
|
|
122
|
-
Everyone interacting in the JsonDataExtractor project’s codebases, issue trackers, chat rooms and mailing lists is
|
170
|
+
Everyone interacting in the JsonDataExtractor project’s codebases, issue trackers, chat rooms and mailing lists is
|
171
|
+
expected to follow
|
172
|
+
the [code of conduct](https://github.com/austerlitz/json_data_extractor/blob/master/CODE_OF_CONDUCT.md).
|
data/json_data_extractor.gemspec
CHANGED
@@ -1,33 +1,36 @@
|
|
1
|
-
|
2
|
-
lib = File.expand_path("../lib", __FILE__)
|
1
|
+
lib = File.expand_path('../lib', __FILE__)
|
3
2
|
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
-
require
|
3
|
+
require 'src/version'
|
5
4
|
|
6
5
|
Gem::Specification.new do |spec|
|
7
|
-
spec.name
|
8
|
-
spec.version
|
9
|
-
spec.authors
|
10
|
-
spec.email
|
6
|
+
spec.name = 'json_data_extractor'
|
7
|
+
spec.version = JsonDataExtractor::VERSION
|
8
|
+
spec.authors = ['Max Buslaev']
|
9
|
+
spec.email = ['max@buslaev.net']
|
11
10
|
|
12
|
-
spec.summary
|
13
|
-
|
14
|
-
|
15
|
-
spec.
|
11
|
+
spec.summary = %q{Transform JSON data structures with the help of a simple schema and JsonPath expressions.
|
12
|
+
Use the JsonDataExtractor gem to extract and modify data from complex JSON structures using a straightforward syntax
|
13
|
+
and a range of built-in or custom modifiers.}
|
14
|
+
spec.description = %q{json_data_extractor makes it easy to extract data from complex JSON structures,
|
15
|
+
such as API responses or configuration files, using a schema that defines the path to the data and any necessary
|
16
|
+
transformations. The schema is defined as a simple Ruby hash that maps keys to paths and optional modifiers.}
|
17
|
+
spec.homepage = 'https://github.com/austerlitz/json_data_extractor'
|
18
|
+
spec.license = 'MIT'
|
16
19
|
|
17
|
-
|
20
|
+
# Specify which files should be added to the gem when it is released.
|
18
21
|
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
19
22
|
spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
|
20
23
|
`git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
21
24
|
end
|
22
|
-
spec.bindir =
|
25
|
+
spec.bindir = 'exe'
|
23
26
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
24
|
-
spec.require_paths = [
|
27
|
+
spec.require_paths = ['lib']
|
25
28
|
|
26
|
-
spec.add_development_dependency
|
27
|
-
spec.add_development_dependency
|
28
|
-
spec.add_development_dependency
|
29
|
-
spec.add_development_dependency
|
30
|
-
spec.add_development_dependency
|
29
|
+
spec.add_development_dependency 'bundler', '~> 1.17'
|
30
|
+
spec.add_development_dependency 'rake', '~> 10.0'
|
31
|
+
spec.add_development_dependency 'rspec', '~> 3.0'
|
32
|
+
spec.add_development_dependency 'pry'
|
33
|
+
spec.add_development_dependency 'amazing_print'
|
31
34
|
|
32
35
|
spec.add_dependency 'jsonpath'
|
33
36
|
end
|
data/lib/json_data_extractor.rb
CHANGED
@@ -1,32 +1,50 @@
|
|
1
1
|
require "src/version"
|
2
2
|
|
3
3
|
class JsonDataExtractor
|
4
|
-
attr_reader :
|
4
|
+
attr_reader :data, :modifiers
|
5
5
|
|
6
6
|
def initialize(json_data, modifiers = {})
|
7
|
-
@
|
8
|
-
@modifiers = modifiers
|
7
|
+
@data = json_data
|
8
|
+
@modifiers = modifiers.transform_keys(&:to_sym) # todo address this later
|
9
9
|
end
|
10
10
|
|
11
|
+
# @param modifier_name [String, Symbol]
|
11
12
|
def add_modifier(modifier_name, &block)
|
13
|
+
modifier_name = modifier_name.to_sym unless modifier_name.is_a?(Symbol)
|
12
14
|
modifiers[modifier_name] = block
|
13
15
|
end
|
14
16
|
|
17
|
+
# @param schema [Hash] schema of the expected data mapping
|
15
18
|
def extract(schema)
|
16
19
|
results = {}
|
17
20
|
schema.each do |key, val|
|
18
|
-
|
19
|
-
|
21
|
+
if val.is_a?(Hash)
|
22
|
+
val.transform_keys!(&:to_sym)
|
23
|
+
path = val[:path]
|
24
|
+
modifiers = Array(val[:modifiers] || val[:modifier]).map(&:to_sym)
|
25
|
+
array_type = 'array' == val[:type]
|
26
|
+
nested = val.delete(:schema)
|
27
|
+
else
|
28
|
+
path = val
|
29
|
+
modifiers = []
|
30
|
+
end
|
20
31
|
|
21
|
-
json_path
|
22
|
-
extracted_data = json_path.on(@
|
32
|
+
json_path = JsonPath.new(path)
|
33
|
+
extracted_data = json_path.on(@data)
|
23
34
|
|
24
35
|
if extracted_data.empty?
|
25
36
|
results[key] = nil
|
26
37
|
else
|
27
38
|
results[key] = apply_modifiers(extracted_data, modifiers)
|
28
|
-
|
39
|
+
|
40
|
+
# TODO yeah, this looks ugly. Address it later.
|
41
|
+
if !array_type
|
29
42
|
results[key] = results[key].first unless 1 < results[key].size
|
43
|
+
elsif nested
|
44
|
+
results[key] = []
|
45
|
+
Array(extracted_data).each do |item|
|
46
|
+
results[key] << self.class.new(item).extract(nested)
|
47
|
+
end
|
30
48
|
end
|
31
49
|
end
|
32
50
|
end
|
@@ -39,11 +57,7 @@ class JsonDataExtractor
|
|
39
57
|
data.map do |value|
|
40
58
|
modified_value = value
|
41
59
|
modifiers.each do |modifier|
|
42
|
-
|
43
|
-
modified_value = @modifiers[modifier.to_sym].call(modified_value)
|
44
|
-
else
|
45
|
-
modified_value = apply_single_modifier(modifier, modified_value)
|
46
|
-
end
|
60
|
+
modified_value = apply_single_modifier(modifier, modified_value)
|
47
61
|
end
|
48
62
|
modified_value
|
49
63
|
end
|
data/lib/src/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: json_data_extractor
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Max Buslaev
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-03-
|
11
|
+
date: 2023-03-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -94,7 +94,10 @@ dependencies:
|
|
94
94
|
- - ">="
|
95
95
|
- !ruby/object:Gem::Version
|
96
96
|
version: '0'
|
97
|
-
description:
|
97
|
+
description: |-
|
98
|
+
json_data_extractor makes it easy to extract data from complex JSON structures,
|
99
|
+
such as API responses or configuration files, using a schema that defines the path to the data and any necessary
|
100
|
+
transformations. The schema is defined as a simple Ruby hash that maps keys to paths and optional modifiers.
|
98
101
|
email:
|
99
102
|
- max@buslaev.net
|
100
103
|
executables: []
|
@@ -136,5 +139,8 @@ requirements: []
|
|
136
139
|
rubygems_version: 3.0.3
|
137
140
|
signing_key:
|
138
141
|
specification_version: 4
|
139
|
-
summary: Transform JSON data structures with the help of simple schema
|
142
|
+
summary: Transform JSON data structures with the help of a simple schema and JsonPath
|
143
|
+
expressions. Use the JsonDataExtractor gem to extract and modify data from complex
|
144
|
+
JSON structures using a straightforward syntax and a range of built-in or custom
|
145
|
+
modifiers.
|
140
146
|
test_files: []
|