database-exporter 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +39 -0
- data/.travis.yml +6 -0
- data/CHANGELOG.md +4 -0
- data/Gemfile +7 -0
- data/Gemfile.lock +74 -0
- data/LICENSE +22 -0
- data/README.md +682 -0
- data/Rakefile +7 -0
- data/bin/database-exporter +48 -0
- data/database_exporter.gemspec +34 -0
- data/example_data/contentful_model.json +316 -0
- data/example_data/contentful_structure.json +89 -0
- data/example_data/example_settings.yml +25 -0
- data/example_data/mapping.json +119 -0
- data/lib/cli.rb +13 -0
- data/lib/configuration.rb +69 -0
- data/lib/converters/content_types_structure_creator.rb +58 -0
- data/lib/converters/contentful_model_to_json.rb +78 -0
- data/lib/database/export.rb +74 -0
- data/lib/database/modules/json_export.rb +79 -0
- data/lib/database/modules/relations_export.rb +270 -0
- data/lib/database/modules/utils.rb +20 -0
- data/lib/migrator.rb +29 -0
- data/lib/version.rb +3 -0
- data/spec/fixtures/database/data/assets/image/image_1.json +9 -0
- data/spec/fixtures/database/data/assets/image/image_2.json +9 -0
- data/spec/fixtures/database/data/assets/image/image_3.json +9 -0
- data/spec/fixtures/database/data/assets/image/image_4.json +9 -0
- data/spec/fixtures/database/data/collections/comment.json +18 -0
- data/spec/fixtures/database/data/collections/job_skills.json +13 -0
- data/spec/fixtures/database/data/collections/jobs.json +44 -0
- data/spec/fixtures/database/data/collections/profile.json +19 -0
- data/spec/fixtures/database/data/collections/user.json +36 -0
- data/spec/fixtures/database/data/entries/comment/comment_1.json +9 -0
- data/spec/fixtures/database/data/entries/comment/comment_2.json +9 -0
- data/spec/fixtures/database/data/entries/comment/comment_3.json +9 -0
- data/spec/fixtures/database/data/entries/comment/comment_4.json +9 -0
- data/spec/fixtures/database/data/entries/comment/comment_5.json +9 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_1.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_10.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_2.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_3.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_4.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_5.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_6.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_7.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_8.json +7 -0
- data/spec/fixtures/database/data/entries/job_skills/job_skills_9.json +7 -0
- data/spec/fixtures/database/data/entries/jobs/jobs_1.json +56 -0
- data/spec/fixtures/database/data/entries/jobs/jobs_2.json +55 -0
- data/spec/fixtures/database/data/entries/jobs/jobs_4.json +49 -0
- data/spec/fixtures/database/data/entries/profile/profile_1.json +12 -0
- data/spec/fixtures/database/data/entries/profile/profile_2.json +12 -0
- data/spec/fixtures/database/data/entries/user/user_1.json +24 -0
- data/spec/fixtures/database/data/entries/user/user_2.json +20 -0
- data/spec/fixtures/database/data/helpers/job_add_id_comments.json +11 -0
- data/spec/fixtures/database/data/helpers/job_add_id_job_add_skills.json +24 -0
- data/spec/fixtures/database/data/helpers/user_id_job_adds.json +9 -0
- data/spec/fixtures/database/data/helpers/user_id_profiles.json +8 -0
- data/spec/fixtures/database/data/table_names.json +10 -0
- data/spec/fixtures/database/table_names.json +4 -0
- data/spec/fixtures/development.sqlite3 +0 -0
- data/spec/fixtures/json_responses/transformed_row.json +7 -0
- data/spec/fixtures/json_row/row.json +6 -0
- data/spec/fixtures/settings/contentful_model.json +316 -0
- data/spec/fixtures/settings/contentful_structure.json +89 -0
- data/spec/fixtures/settings/contentful_structure_test.json +82 -0
- data/spec/fixtures/settings/mapping.json +119 -0
- data/spec/fixtures/settings/settings.yml +27 -0
- data/spec/lib/configuration_spec.rb +17 -0
- data/spec/lib/database/export_spec.rb +49 -0
- data/spec/lib/database/json_export_spec.rb +49 -0
- data/spec/lib/database/relations_export_spec.rb +201 -0
- data/spec/lib/migrator_spec.rb +112 -0
- data/spec/spec_helper.rb +12 -0
- data/spec/support/db_rows_json.rb +9 -0
- data/spec/support/shared_configuration.rb +27 -0
- metadata +358 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 06cd3d41d41504c75c8b22d74cdaee9d77ca79d5
|
4
|
+
data.tar.gz: 5655c082e08d86f9d5e438593102f69a51defed9
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: a9180c514f3224204ea8da91a38dce8ea9c991ccb72d8f9566cae25f8ce4e86c0f86b20046488c14d4938d31ca0c13d53f5144be45e2568eaf22f7a8a1a629fe
|
7
|
+
data.tar.gz: 9e85b77bc5982b7866c82642d6a0744aa5a1c42f31f5872857acd651c75862857dc105c6abdc164b2ab4559efe878a301e706ab80f6753a5d6e4e44bc6134f17
|
data/.gitignore
ADDED
@@ -0,0 +1,39 @@
|
|
1
|
+
*.gem
|
2
|
+
*.rbc
|
3
|
+
/.config
|
4
|
+
/coverage/
|
5
|
+
/InstalledFiles
|
6
|
+
/pkg/
|
7
|
+
/spec/reports/
|
8
|
+
/test/tmp/
|
9
|
+
/test/version_tmp/
|
10
|
+
/tmp/
|
11
|
+
|
12
|
+
## Specific to RubyMotion:
|
13
|
+
.dat*
|
14
|
+
.repl_history
|
15
|
+
build/
|
16
|
+
|
17
|
+
## Documentation cache and generated files:
|
18
|
+
/.yardoc/
|
19
|
+
/_yardoc/
|
20
|
+
/doc/
|
21
|
+
/rdoc/
|
22
|
+
|
23
|
+
## Environment normalisation:
|
24
|
+
/.bundle/
|
25
|
+
/lib/bundler/man/
|
26
|
+
|
27
|
+
# for a library or gem, you might want to ignore these files since the code is
|
28
|
+
# intended to run in multiple environments; otherwise, check them in:
|
29
|
+
# Gemfile.lock
|
30
|
+
# .ruby-version
|
31
|
+
# .ruby-gemset
|
32
|
+
|
33
|
+
# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
|
34
|
+
.rvmrc
|
35
|
+
|
36
|
+
#RubyMine
|
37
|
+
.idea
|
38
|
+
|
39
|
+
.DS_Store
|
data/.travis.yml
ADDED
data/CHANGELOG.md
ADDED
data/Gemfile
ADDED
data/Gemfile.lock
ADDED
@@ -0,0 +1,74 @@
|
|
1
|
+
PATH
|
2
|
+
remote: .
|
3
|
+
specs:
|
4
|
+
database-exporter (0.0.1)
|
5
|
+
activesupport (~> 4.1)
|
6
|
+
escort (~> 0.4.0)
|
7
|
+
http (~> 0.6)
|
8
|
+
i18n (~> 0.6)
|
9
|
+
multi_json (~> 1)
|
10
|
+
mysql2 (~> 0.3)
|
11
|
+
pg (~> 0.17.0)
|
12
|
+
sequel (~> 4.15)
|
13
|
+
sqlite3 (~> 1.3.10)
|
14
|
+
|
15
|
+
GEM
|
16
|
+
remote: https://rubygems.org/
|
17
|
+
specs:
|
18
|
+
activesupport (4.1.8)
|
19
|
+
i18n (~> 0.6, >= 0.6.9)
|
20
|
+
json (~> 1.7, >= 1.7.7)
|
21
|
+
minitest (~> 5.1)
|
22
|
+
thread_safe (~> 0.1)
|
23
|
+
tzinfo (~> 1.1)
|
24
|
+
diff-lcs (1.2.5)
|
25
|
+
docile (1.1.5)
|
26
|
+
escort (0.4.0)
|
27
|
+
nesty
|
28
|
+
http (0.6.3)
|
29
|
+
http_parser.rb (~> 0.6.0)
|
30
|
+
http_parser.rb (0.6.0)
|
31
|
+
i18n (0.6.11)
|
32
|
+
json (1.8.1)
|
33
|
+
minitest (5.5.0)
|
34
|
+
multi_json (1.10.1)
|
35
|
+
mysql2 (0.3.17)
|
36
|
+
nesty (1.0.2)
|
37
|
+
pg (0.17.1)
|
38
|
+
rake (10.4.2)
|
39
|
+
rspec (3.1.0)
|
40
|
+
rspec-core (~> 3.1.0)
|
41
|
+
rspec-expectations (~> 3.1.0)
|
42
|
+
rspec-mocks (~> 3.1.0)
|
43
|
+
rspec-core (3.1.7)
|
44
|
+
rspec-support (~> 3.1.0)
|
45
|
+
rspec-expectations (3.1.2)
|
46
|
+
diff-lcs (>= 1.2.0, < 2.0)
|
47
|
+
rspec-support (~> 3.1.0)
|
48
|
+
rspec-its (1.1.0)
|
49
|
+
rspec-core (>= 3.0.0)
|
50
|
+
rspec-expectations (>= 3.0.0)
|
51
|
+
rspec-mocks (3.1.3)
|
52
|
+
rspec-support (~> 3.1.0)
|
53
|
+
rspec-support (3.1.2)
|
54
|
+
sequel (4.17.0)
|
55
|
+
simplecov (0.9.1)
|
56
|
+
docile (~> 1.1.0)
|
57
|
+
multi_json (~> 1.0)
|
58
|
+
simplecov-html (~> 0.8.0)
|
59
|
+
simplecov-html (0.8.0)
|
60
|
+
sqlite3 (1.3.10)
|
61
|
+
thread_safe (0.3.4)
|
62
|
+
tzinfo (1.2.2)
|
63
|
+
thread_safe (~> 0.1)
|
64
|
+
|
65
|
+
PLATFORMS
|
66
|
+
ruby
|
67
|
+
|
68
|
+
DEPENDENCIES
|
69
|
+
bundler (~> 1.6)
|
70
|
+
database-exporter!
|
71
|
+
rake
|
72
|
+
rspec (~> 3)
|
73
|
+
rspec-its (~> 1.1.0)
|
74
|
+
simplecov
|
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2014 Contentful
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE.
|
22
|
+
|
data/README.md
ADDED
@@ -0,0 +1,682 @@
|
|
1
|
+
Database to Contentful exporter
|
2
|
+
=================
|
3
|
+
|
4
|
+
## Description
|
5
|
+
|
6
|
+
Migrate content from a relational database to [contentful.com](https://www.contentful.com).
|
7
|
+
|
8
|
+
This tool allows you to fetch content from your database system and prepare it for the import.
|
9
|
+
|
10
|
+
|
11
|
+
## Installation
|
12
|
+
|
13
|
+
```bash
|
14
|
+
gem install database-exporter
|
15
|
+
```
|
16
|
+
|
17
|
+
This will install the `database-exporter` executable on your system.
|
18
|
+
|
19
|
+
## Usage
|
20
|
+
|
21
|
+
Once you installed the Gem and created the `settings.yml` file, you can invoke the tool using:
|
22
|
+
|
23
|
+
```bash
|
24
|
+
database-exporter --config-file settings.yml --action
|
25
|
+
```
|
26
|
+
|
27
|
+
## Step by Step
|
28
|
+
|
29
|
+
1. Create a YAML file with the required parameters (e.g. settings.yml):
|
30
|
+
|
31
|
+
```yml
|
32
|
+
#PATH to all data, this will create a folder in your current working directory
|
33
|
+
data_dir: PATH_TO_ALL_DATA
|
34
|
+
|
35
|
+
#Connecting to the database
|
36
|
+
adapter: postgres
|
37
|
+
host: localhost
|
38
|
+
database: database_name
|
39
|
+
user: username
|
40
|
+
password: username
|
41
|
+
|
42
|
+
# Extract data from models:
|
43
|
+
mapped:
|
44
|
+
tables:
|
45
|
+
- :table_name_1
|
46
|
+
- :table_name_2
|
47
|
+
- :table_name_3
|
48
|
+
- :table_name_4
|
49
|
+
|
50
|
+
## MAPPING ##
|
51
|
+
mapping_dir: example_data/mapping.json
|
52
|
+
contentful_structure_dir: example_data/contentful_structure.json
|
53
|
+
|
54
|
+
## CONVERT
|
55
|
+
content_model_json: example_data/contentful_model.json
|
56
|
+
converted_model_dir: example_data/contentful_structure.json
|
57
|
+
```
|
58
|
+
|
59
|
+
2. Create the `contentful_structure.json` file: First you should create a content model using the [Contentful ](www.contentful.com) web application. Then you can download the content model using the content management api and use it as the schema for your imports.
|
60
|
+
|
61
|
+
```bash
|
62
|
+
curl -X GET \
|
63
|
+
-H 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
|
64
|
+
'https://api.contentful.com/spaces/SPACE_ID/content_types' > contentful_model.json
|
65
|
+
```
|
66
|
+
|
67
|
+
It will create a `contentful_model.json` file, which you need to transform into the `contentful_structure.json` file using:
|
68
|
+
|
69
|
+
```bash
|
70
|
+
database-exporter --config-file settings.yml --convert-content-model-to-json
|
71
|
+
```
|
72
|
+
|
73
|
+
The converted content model will be saved as a JSON file in the `converted_model_dir` path.
|
74
|
+
|
75
|
+
3. Create content types files: Based on the ```contentful_structure.json``` file, create content types json files, which represents your contentful structure, use:
|
76
|
+
|
77
|
+
```bash
|
78
|
+
database-exporter --config-file settings.yml --create-content-model-from-json
|
79
|
+
```
|
80
|
+
|
81
|
+
It will extract your content types and store it as a separate JSON file in `data_dir/collections` directory.
|
82
|
+
|
83
|
+
4. After filling in the required parameters to [connect to the database](https://github.com/contentful/database-adapter#database-connection---define-adapter), the tables we want to fetch the content from need to be specified.
|
84
|
+
You can skip `joining table names`, if you do not want to map them to a separate content type.
|
85
|
+
|
86
|
+
```bash
|
87
|
+
database-exporter --config-file settings.yml --list-tables
|
88
|
+
```
|
89
|
+
|
90
|
+
It will create the `table_names.json` file with the names of all tables contained in database.
|
91
|
+
|
92
|
+
Example:
|
93
|
+
|
94
|
+
```javascript
|
95
|
+
[
|
96
|
+
"schema_migrations",
|
97
|
+
"skills",
|
98
|
+
"comments",
|
99
|
+
"images",
|
100
|
+
"job_add_skills",
|
101
|
+
"users",
|
102
|
+
"job_adds",
|
103
|
+
"profiles"
|
104
|
+
]
|
105
|
+
```
|
106
|
+
|
107
|
+
5. Extract data from the database: Create the `mapping.json` file with the mapped the structure of your database.
|
108
|
+
|
109
|
+
Example structure for `user` table.
|
110
|
+
|
111
|
+
```javascript
|
112
|
+
"Users": {
|
113
|
+
"content_type": "User",
|
114
|
+
"type": "entry",
|
115
|
+
"fields": {
|
116
|
+
},
|
117
|
+
"links": {
|
118
|
+
}
|
119
|
+
}
|
120
|
+
```
|
121
|
+
|
122
|
+
After defining the structure for each table you want to extract in the JSON file, use:
|
123
|
+
|
124
|
+
```bash
|
125
|
+
database-exporter --config-file settings.yml --extract-to-json
|
126
|
+
```
|
127
|
+
|
128
|
+
This will extract data from tables and store it as JSON. The `data_dir/entries` directory will be created with subdirectories that represent the data from each table.
|
129
|
+
The sub-directories name depends on the *content_type* parameter used in the mapping.json file.
|
130
|
+
|
131
|
+
6. Mapping data to content types: The `mapping.json` file contains the structure of your database. All the relationships between the models need to be specified there.
|
132
|
+
A description of how to build those relationships [can be found here](https://github.com/contentful/database-adapter#relations-types).
|
133
|
+
|
134
|
+
To begin the mapping procedure, use:
|
135
|
+
|
136
|
+
```bash
|
137
|
+
database-exporter --config-file settings.yml --prepare-json
|
138
|
+
```
|
139
|
+
|
140
|
+
It will change the structure of files in the `entries` directory. If the mapping has been done correctly, you can proceed to import the data into Contentful.
|
141
|
+
|
142
|
+
7. Use the [contentful-importer](https://github.com/contentful/generic-importer.rb) to import the content to [contentful.com](https://www.contentful.com)
|
143
|
+
|
144
|
+
|
145
|
+
## Configuration File
|
146
|
+
|
147
|
+
You need to create a configuration file and fill in the following information:
|
148
|
+
|
149
|
+
```yml
|
150
|
+
#PATH to all data
|
151
|
+
data_dir: PATH_TO_ALL_DATA
|
152
|
+
|
153
|
+
#Connecting to a database
|
154
|
+
adapter: postgres
|
155
|
+
host: localhost
|
156
|
+
database: database_name
|
157
|
+
user: username
|
158
|
+
password: password
|
159
|
+
|
160
|
+
# Extract data from models:
|
161
|
+
mapped:
|
162
|
+
tables:
|
163
|
+
- :table_name_1
|
164
|
+
- :table_name_2
|
165
|
+
- :table_name_3
|
166
|
+
|
167
|
+
# Mapping
|
168
|
+
mapping_dir: PATH_TO_MAPPING_FILE/mapping.json
|
169
|
+
contentful_structure_dir: PATH_TO_CONTENTFUL_STURCTURE_FILE/contentful_structure.json
|
170
|
+
|
171
|
+
# Convert
|
172
|
+
content_model_json: PATH_TO_CONTENT_MODEL/contentful_model.json
|
173
|
+
converted_model_dir: PATH_TO_CONVERTED_CONTENT_MODEL_FILE/contentful_structure.json
|
174
|
+
|
175
|
+
```
|
176
|
+
|
177
|
+
## Actions
|
178
|
+
|
179
|
+
To display all actions use the `-h` option:
|
180
|
+
|
181
|
+
```bash
|
182
|
+
database-exporter -h
|
183
|
+
```
|
184
|
+
|
185
|
+
#### --list-tables
|
186
|
+
|
187
|
+
This action will create a JSON file including all table names from your database and write them to `data_dir/table_names.json`. The table names are needed to extract the content from the database.
|
188
|
+
|
189
|
+
#### --extract-to-json
|
190
|
+
|
191
|
+
In the [settings.yml](https://github.com/contentful/database-adapter#setting-file) file, you need to define the table names that should be exported from the database.
|
192
|
+
|
193
|
+
The recommended way to get the table names, is using [--list-tables](https://github.com/contentful/database-adapter#--list-tables).
|
194
|
+
|
195
|
+
After specifying the table names you want to extract in your settings, run the `--extract-to-json` command.
|
196
|
+
This will save each object from the database into its own JSON file, ready to be transformed and imported.
|
197
|
+
|
198
|
+
Path to JSON data: ***data_dir/entries/content_type_name_defined_in_mapping_json_file***
|
199
|
+
|
200
|
+
#### --prepare-json
|
201
|
+
|
202
|
+
Prepares the generated JSON files so they can be imported to Contentful.
|
203
|
+
|
204
|
+
|
205
|
+
### FIELDS
|
206
|
+
|
207
|
+
To change the name of a field in the database to a new one in the content type, we need to add a new mapping for that field:
|
208
|
+
```javascript
|
209
|
+
"fields": {
|
210
|
+
"model_name": "new_api_contentful_field_name",
|
211
|
+
"model_name": "new_api_contentful_field_name",
|
212
|
+
"model_name": "new_api_contentful_field_name"
|
213
|
+
},
|
214
|
+
```
|
215
|
+
|
216
|
+
|
217
|
+
### Relation Types/Joins
|
218
|
+
|
219
|
+
The following relational associations behave similar to the Active Record associations.
|
220
|
+
|
221
|
+
#### belongs_to
|
222
|
+
|
223
|
+
The `belongs_to` method should only be used if this table contains the foreign key. If the other table contains the foreign key, then you should use `has_one` instead.
|
224
|
+
|
225
|
+
At the beginning and we are looking for `type` and `id` of the linked object in file `contentful_structure.json`.
|
226
|
+
It's very important to maintain consistency for the content type names in `mapping.json` and `contentful_structure.json`.
|
227
|
+
The next step is to check if the object has defined a foreign key itself. After that an object with type and ID is created.
|
228
|
+
|
229
|
+
Example:
|
230
|
+
|
231
|
+
```javascript
|
232
|
+
"Comments": {
|
233
|
+
"content_type": "Comments",
|
234
|
+
"type": "entry",
|
235
|
+
"fields": {
|
236
|
+
},
|
237
|
+
"links": {
|
238
|
+
"belongs_to": [
|
239
|
+
{
|
240
|
+
"relation_to": "ModelName",
|
241
|
+
"foreign_id": "model_foreign_id"
|
242
|
+
}
|
243
|
+
]
|
244
|
+
}
|
245
|
+
}
|
246
|
+
```
|
247
|
+
|
248
|
+
It will assign the associated object and save its ID `(model_name + id)` in the JSON file.
|
249
|
+
|
250
|
+
Result:
|
251
|
+
|
252
|
+
```javascript
|
253
|
+
{
|
254
|
+
"id": "model_name_ID",
|
255
|
+
...
|
256
|
+
"job_add_id": {
|
257
|
+
"type": "Entry",
|
258
|
+
"id": "model_name_3"
|
259
|
+
},
|
260
|
+
}
|
261
|
+
```
|
262
|
+
|
263
|
+
#### has_one
|
264
|
+
|
265
|
+
The `has_one` method should be used if the other table contains the foreign key. If the current table contains the foreign key, then you should use belongs_to instead.
|
266
|
+
|
267
|
+
At the beginning the tool builds a helper file which contains the primary id as key and the foreign id as values. This file lives in `data_dir/helpers`.
|
268
|
+
|
269
|
+
After that we modify only those files whose ID is located in the helper file as a key. Value is written as a Hash value.
|
270
|
+
|
271
|
+
Example:
|
272
|
+
|
273
|
+
```javascript
|
274
|
+
"Users": {
|
275
|
+
"content_type": "Users",
|
276
|
+
"type": "entry",
|
277
|
+
"fields": {
|
278
|
+
...
|
279
|
+
},
|
280
|
+
"links": {
|
281
|
+
"has_one": [
|
282
|
+
{
|
283
|
+
"relation_to": "ModelName",
|
284
|
+
"primary_id": "primary_key_name"
|
285
|
+
}
|
286
|
+
]
|
287
|
+
}
|
288
|
+
}
|
289
|
+
```
|
290
|
+
|
291
|
+
Result:
|
292
|
+
|
293
|
+
It will assign the associated object, save his ID ```(model_name + id)``` in JSON file.
|
294
|
+
|
295
|
+
```javascript
|
296
|
+
...
|
297
|
+
"model_name": {
|
298
|
+
"type": "profiles",
|
299
|
+
"id": "content_type_id_3"
|
300
|
+
}
|
301
|
+
```
|
302
|
+
|
303
|
+
|
304
|
+
#### many
|
305
|
+
|
306
|
+
The resulting file will be generated in a similar way as for the `has_one` relation.
|
307
|
+
At the beginning the tool builds a helper file which contains the primary id as key and the foreign id as values. This file lives in `data_dir/helpers`.
|
308
|
+
|
309
|
+
After that we modify only those files whose ID is located in the helper file as a key. Related objects are written always as an Array.
|
310
|
+
|
311
|
+
Example:
|
312
|
+
|
313
|
+
```javascript
|
314
|
+
"ModelName": {
|
315
|
+
...
|
316
|
+
},
|
317
|
+
"links": {
|
318
|
+
"many": [
|
319
|
+
{
|
320
|
+
"relation_to": "related_model_name",
|
321
|
+
"primary_id": "primary_key_name"
|
322
|
+
}
|
323
|
+
],
|
324
|
+
}
|
325
|
+
}
|
326
|
+
```
|
327
|
+
|
328
|
+
It will assign the associated objects, save its ID ```(model_name + id)``` in JSON file.
|
329
|
+
|
330
|
+
Result:
|
331
|
+
|
332
|
+
```javascript
|
333
|
+
{
|
334
|
+
"id": "content_type_id",
|
335
|
+
"comments": [
|
336
|
+
{
|
337
|
+
"type": "related_content_type_name",
|
338
|
+
"id": "related_model_name_id"
|
339
|
+
},
|
340
|
+
{
|
341
|
+
"type": "related_content_type_name",
|
342
|
+
"id": "related_model_name_id"
|
343
|
+
},
|
344
|
+
{
|
345
|
+
"type": "related_content_type_name",
|
346
|
+
"id": "related_model_name_id"
|
347
|
+
},
|
348
|
+
{
|
349
|
+
"type": "related_content_type_name",
|
350
|
+
"id": "related_model_name_id"
|
351
|
+
}
|
352
|
+
]
|
353
|
+
}
|
354
|
+
```
|
355
|
+
|
356
|
+
#### many_through
|
357
|
+
|
358
|
+
The resulting file will be generated in a similar way as for the `has_one` relation.
|
359
|
+
After that we modify only those files whose ID is located in the helper file as a key. Related objects are written always as an Array.
|
360
|
+
|
361
|
+
Attributes:
|
362
|
+
|
363
|
+
```
|
364
|
+
relation_to: Name of the related model, defined in the mapping.json file as a key.
|
365
|
+
primary_id: Name of the primary key located in the joining table.
|
366
|
+
foreign_id: Name of the foreign key, located in the joining table. The object with this ID will be added to the mapped object.
|
367
|
+
through: Name of the joining model.
|
368
|
+
```
|
369
|
+
|
370
|
+
|
371
|
+
Example:
|
372
|
+
|
373
|
+
```javascript
|
374
|
+
"ModelName": {
|
375
|
+
...
|
376
|
+
"links": {
|
377
|
+
"many_through": [
|
378
|
+
{
|
379
|
+
"relation_to": "related_model_name",
|
380
|
+
"primary_id": "primary_key_name",
|
381
|
+
"foreign_id": "foreign_key_name",
|
382
|
+
"through": "join_table_name"
|
383
|
+
}
|
384
|
+
]
|
385
|
+
}
|
386
|
+
}
|
387
|
+
```
|
388
|
+
|
389
|
+
It will map the join table and save the objects IDs in the current model.
|
390
|
+
|
391
|
+
Result:
|
392
|
+
|
393
|
+
```javascript
|
394
|
+
"content_type_name": [
|
395
|
+
{
|
396
|
+
"type": "content_type_name",
|
397
|
+
"id": "related_model_foreign_id"
|
398
|
+
},
|
399
|
+
{
|
400
|
+
"type": "content_type_name",
|
401
|
+
"id": "related_model_foreign_id"
|
402
|
+
},
|
403
|
+
{
|
404
|
+
"type": "content_type_name",
|
405
|
+
"id": "related_model_foreign_id"
|
406
|
+
}
|
407
|
+
]
|
408
|
+
```
|
409
|
+
|
410
|
+
#### aggregate_belongs
|
411
|
+
|
412
|
+
`aggregate_belongs` allows to fetch a value from a related model.
|
413
|
+
To add the value, the table must have the `foreign_id` to the related table. Through this key the object is found and the related data is extracted.
|
414
|
+
|
415
|
+
Attributes:
|
416
|
+
|
417
|
+
```
|
418
|
+
relation_to: Name of the related model, defined in the mapping.json file as a key.
|
419
|
+
primary_id: Name of the primary key in the model.
|
420
|
+
field: Name of the attribute, which you want to add.
|
421
|
+
save_as: Name of the attribute whose value is assigned.
|
422
|
+
```
|
423
|
+
|
424
|
+
Example:
|
425
|
+
|
426
|
+
```javascript
|
427
|
+
"links": {
|
428
|
+
"aggregate_belongs": [
|
429
|
+
{
|
430
|
+
"relation_to": "related_model_name",
|
431
|
+
"primary_id": "primary_key_name",
|
432
|
+
"field": "aggregated_field_name",
|
433
|
+
"save_as": "name_of_field"
|
434
|
+
}
|
435
|
+
]
|
436
|
+
}
|
437
|
+
```
|
438
|
+
|
439
|
+
Result:
|
440
|
+
|
441
|
+
```javascript
|
442
|
+
{
|
443
|
+
"id": "model_name_id",
|
444
|
+
"name_of_field": "aggregated_value"
|
445
|
+
}
|
446
|
+
```
|
447
|
+
|
448
|
+
#### aggregate_has_one
|
449
|
+
|
450
|
+
It will save the value with the key of the related model. To add `has_one` value, the table must have the `primary_id` of the related table.
|
451
|
+
|
452
|
+
Attributes:
|
453
|
+
|
454
|
+
```
|
455
|
+
relation_to: Name of the related model, defined in the mapping.json file as a key.
|
456
|
+
primary_id: Name of the primary key in the model.
|
457
|
+
field: Name of the attribute, which you want to add.
|
458
|
+
save_as: Name of the attribute whose value is assigned.
|
459
|
+
```
|
460
|
+
|
461
|
+
Example:
|
462
|
+
|
463
|
+
```javascript
|
464
|
+
"links": {
|
465
|
+
"aggregate_has_one": [
|
466
|
+
{
|
467
|
+
"primary_id": "primary_id",
|
468
|
+
"relation_to": "related_model_name",
|
469
|
+
"field": "name_of_field_to_aggregate",
|
470
|
+
"save_as": "save_as_field_name"
|
471
|
+
}
|
472
|
+
]
|
473
|
+
}
|
474
|
+
```
|
475
|
+
|
476
|
+
Result:
|
477
|
+
|
478
|
+
```javascript
|
479
|
+
{
|
480
|
+
"id": "model_name_id",
|
481
|
+
"name_of_field": "aggregated_value"
|
482
|
+
}
|
483
|
+
```
|
484
|
+
|
485
|
+
#### aggregate_many
|
486
|
+
|
487
|
+
It will save the value with the key of the related table. To add the `has_many` value, the related table must have the `primary_id` of the related model. This will create a new attribute in the model with the type `Array`.
|
488
|
+
|
489
|
+
Example:
|
490
|
+
|
491
|
+
```javascript
|
492
|
+
"links": {
|
493
|
+
"aggregate_many": [
|
494
|
+
{
|
495
|
+
"primary_id": "primary_id",
|
496
|
+
"relation_to": "related_model_name",
|
497
|
+
"field": "name_of_field_to_aggregate",
|
498
|
+
"save_as": "save_as_field_name"
|
499
|
+
}
|
500
|
+
]
|
501
|
+
}
|
502
|
+
```
|
503
|
+
|
504
|
+
Result:
|
505
|
+
|
506
|
+
```javascript
|
507
|
+
{
|
508
|
+
"id": "model_name_id",
|
509
|
+
"name_of_field": [
|
510
|
+
"aggregated_value1",
|
511
|
+
"aggregated_value2",
|
512
|
+
"aggregated_value3",
|
513
|
+
"aggregated_value4"
|
514
|
+
]
|
515
|
+
}
|
516
|
+
```
|
517
|
+
|
518
|
+
#### aggregate_through
|
519
|
+
|
520
|
+
It will save the value with the key of the related model.
|
521
|
+
To add the `has_many, through` value, you need to define the `join model` which contains the `primary_id` and `foreign_id`. Through the `foreign_id` the desired object can be found.
|
522
|
+
|
523
|
+
Attributes:
|
524
|
+
|
525
|
+
```
|
526
|
+
relation_to: Name of related model, defined in mapping.json file as a key.
|
527
|
+
primary_id: Name of primary key located in joining table.
|
528
|
+
foreign_id: Name of foreign key, located in joining table. Object with this ID will be added mapped object.
|
529
|
+
through: Name of joining model.
|
530
|
+
```
|
531
|
+
|
532
|
+
Example:
|
533
|
+
|
534
|
+
```javascript
|
535
|
+
"links": {
|
536
|
+
"aggregate_through": [
|
537
|
+
{
|
538
|
+
"relation_to": "related_model_name",
|
539
|
+
"primary_id": "primary_key_name",
|
540
|
+
"foreign_id": "foreign_key_name",
|
541
|
+
"through": "join_table_name",
|
542
|
+
"field": '"name_of_field_to_aggregate",
|
543
|
+
"save_as": "save_as_field_name"
|
544
|
+
}
|
545
|
+
]
|
546
|
+
}
|
547
|
+
```
|
548
|
+
|
549
|
+
Result:
|
550
|
+
|
551
|
+
```javascript
|
552
|
+
{
|
553
|
+
"id": "model_name_id",
|
554
|
+
"name_of_field": ["aggregated_value1",
|
555
|
+
"aggregated_value2",
|
556
|
+
"aggregated_value3",
|
557
|
+
"aggregated_value4"
|
558
|
+
]
|
559
|
+
}
|
560
|
+
```
|
561
|
+
|
562
|
+
## Contentful Structure
|
563
|
+
|
564
|
+
This file represents our Contentful structure, it defines the remote data types and how they are formed.
|
565
|
+
|
566
|
+
Example:
|
567
|
+
|
568
|
+
```javascript
|
569
|
+
{
|
570
|
+
"Comments": {
|
571
|
+
"id": "comment",
|
572
|
+
"description": "",
|
573
|
+
"displayField": "title",
|
574
|
+
"fields": {
|
575
|
+
"title": "Text",
|
576
|
+
"content": "Text"
|
577
|
+
}
|
578
|
+
},
|
579
|
+
"JobAdd": {
|
580
|
+
"id": "job_add",
|
581
|
+
"description": "Add new job form",
|
582
|
+
"displayField": "name",
|
583
|
+
"fields": {
|
584
|
+
"name": "Text",
|
585
|
+
"specification": "Text",
|
586
|
+
"Images": {
|
587
|
+
"id": "image",
|
588
|
+
"link_type": "Asset"
|
589
|
+
},
|
590
|
+
"Comments": {
|
591
|
+
"id": "comments",
|
592
|
+
"link_type": "Array",
|
593
|
+
"type": "Entry"
|
594
|
+
},
|
595
|
+
"Skills": {
|
596
|
+
"id": "skills",
|
597
|
+
"link_type": "Array",
|
598
|
+
"type": "Entry"
|
599
|
+
}
|
600
|
+
}
|
601
|
+
}
|
602
|
+
```
|
603
|
+
They keys "Images", "Comments", "Skills" are the equivalent of the content types IDs specified in the file **mapping.json**.
|
604
|
+
|
605
|
+
Example:
|
606
|
+
```javascript
|
607
|
+
"SkillsTableName": {
|
608
|
+
"content_type": "Skills",
|
609
|
+
"type": "entry",
|
610
|
+
"fields": { ... }
|
611
|
+
```
|
612
|
+
|
613
|
+
**IMPORTANT**
|
614
|
+
|
615
|
+
To create any relationship between tables, we must remember that the content names given in the **mapping.json** file, must be equal with names in the **contentful_structure.json** file.
|
616
|
+
|
617
|
+
## Settings file
|
618
|
+
|
619
|
+
To be able to extract any content you need to create a `settings.yml` file and define all needed parameters.
|
620
|
+
|
621
|
+
#### Database Connection - Define Adapter
|
622
|
+
|
623
|
+
Assuming we are going to work with a MySQL, SQLite or PostgreSQL database we need to setup the credentials:
|
624
|
+
Following is the example of connecting to a MySQL database `test_import`.
|
625
|
+
|
626
|
+
```yml
|
627
|
+
adapter: mysql2
|
628
|
+
user: username
|
629
|
+
host: localhost
|
630
|
+
database: test_import
|
631
|
+
password: secret_password
|
632
|
+
```
|
633
|
+
|
634
|
+
**Available Adapters**
|
635
|
+
|
636
|
+
```
|
637
|
+
PostgreSQL => postgres
|
638
|
+
MySQL => mysql2
|
639
|
+
SQlite => sqlite
|
640
|
+
```
|
641
|
+
|
642
|
+
#### Mapped tables
|
643
|
+
|
644
|
+
Before we can start exporting the data from the database, the to be used tables need to be specified.
|
645
|
+
The fastest way to get the names is using the [--list-tables](https://github.com/contentful/generic-importer.rb#--list-tables) action.
|
646
|
+
|
647
|
+
Add those to the `settings.yml` file in the following manner:
|
648
|
+
|
649
|
+
```yml
|
650
|
+
mapped:
|
651
|
+
tables:
|
652
|
+
```
|
653
|
+
Example:
|
654
|
+
|
655
|
+
```yml
|
656
|
+
mapped:
|
657
|
+
tables:
|
658
|
+
- :example_1
|
659
|
+
- :example_2
|
660
|
+
- :example_3
|
661
|
+
- :example_4
|
662
|
+
```
|
663
|
+
|
664
|
+
There is no need to specify the names of a join table unless you want to save them as a separate content types.
|
665
|
+
|
666
|
+
### Mapping
|
667
|
+
|
668
|
+
* JSON file with mapping structure that defines relations between models.
|
669
|
+
|
670
|
+
```yml
|
671
|
+
mapping_dir: example_path/mapping.json
|
672
|
+
```
|
673
|
+
|
674
|
+
* JSON file with contentful structure
|
675
|
+
```yml
|
676
|
+
contentful_structure_dir: contentful_import_files/contentful_structure.json
|
677
|
+
```
|
678
|
+
* [Dump JSON file](https://github.com/contentful/generic-importer.rb#--convert-content-model-to-json) with content types from content model:
|
679
|
+
|
680
|
+
```yml
|
681
|
+
converted_model_dir: contentful_import_files/contentful_structure.json
|
682
|
+
```
|