contentful-importer 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5b4cb5b28ec76176d942746f1e3d0c1062acbb39
+  data.tar.gz: e214eff208d733e53483a1397d5f12cbadbdb5a9
+SHA512:
+  metadata.gz: c045429597f9113ee8dc58678b8d8fdea072fe9f17447d2d49ef28c1ead138c2f393294d542874dd80e418f764b40a0c419ebeae2729c11f09daea19aecfef79
+  data.tar.gz: 8f889a25905fa8e8903aad4961793e1f16ae17de1f82f028ed70bd8042d0b0f6d811e5fc3f694ad6da95b2d4a232e1d15e8bed7d2d96bfdc7c1a55232f5b6bd7

data/.gitignore

@@ -0,0 +1,38 @@
+*.gem
+*.rbc
+/.config
+/coverage/
+/InstalledFiles
+/pkg/
+/spec/reports/
+/test/tmp/
+/test/version_tmp/
+/tmp/
+
+## Specific to RubyMotion:
+.dat*
+.repl_history
+build/
+
+## Documentation cache and generated files:
+/.yardoc/
+/_yardoc/
+/doc/
+/rdoc/
+
+## Environment normalisation:
+/lib/bundler/man/
+
+# for a library or gem, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
+# Gemfile.lock
+# .ruby-version
+# .ruby-gemset
+
+# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
+.rvmrc
+
+#RubyMine
+.idea
+
+.DS_Store

data/.rspec

@@ -0,0 +1,3 @@
+--color
+--format progress
+--order random

data/.travis.yml

@@ -0,0 +1,8 @@
+language: ruby
+rvm:
+  - 2.1.2
+  - 2.1.1
+  - 2.1
+  - 2.0.0
+  - 1.9.3
+#  - jruby-19mode

data/CHANGELOG.md

@@ -0,0 +1,4 @@
+# Change Log
+## 0.0.1
+### Other
+* Initial release

data/Gemfile

@@ -0,0 +1,7 @@
+source 'https://rubygems.org'
+# Specify your gem's dependencies in contentful_importer.gemspec
+gemspec
+
+group :test do
+  gem 'simplecov', require: false
+end

data/Gemfile.lock

@@ -0,0 +1,87 @@
+PATH
+  remote: .
+  specs:
+    contentful-importer (0.0.1)
+      activesupport (~> 4.1)
+      api_cache (~> 0.3.0)
+      contentful-management (~> 0.5.0)
+      escort (~> 0.4.0)
+      http (~> 0.6)
+      i18n (~> 0.6)
+      json-schema (~> 2.5.0)
+      multi_json (~> 1)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activesupport (4.2.0)
+      i18n (~> 0.7)
+      json (~> 1.7, >= 1.7.7)
+      minitest (~> 5.1)
+      thread_safe (~> 0.3, >= 0.3.4)
+      tzinfo (~> 1.1)
+    addressable (2.3.6)
+    api_cache (0.3.0)
+    contentful-management (0.5.0)
+      http (~> 0.6)
+      multi_json (~> 1)
+    crack (0.4.2)
+      safe_yaml (~> 1.0.0)
+    diff-lcs (1.2.5)
+    docile (1.1.5)
+    escort (0.4.0)
+      nesty
+    form_data (0.1.0)
+    http (0.7.1)
+      form_data (~> 0.1.0)
+      http_parser.rb (~> 0.6.0)
+    http_parser.rb (0.6.0)
+    i18n (0.7.0)
+    json (1.8.2)
+    json-schema (2.5.0)
+      addressable (~> 2.3)
+    minitest (5.5.1)
+    multi_json (1.10.1)
+    nesty (1.0.2)
+    rake (10.4.2)
+    rspec (3.1.0)
+      rspec-core (~> 3.1.0)
+      rspec-expectations (~> 3.1.0)
+      rspec-mocks (~> 3.1.0)
+    rspec-core (3.1.7)
+      rspec-support (~> 3.1.0)
+    rspec-expectations (3.1.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.1.0)
+    rspec-its (1.1.0)
+      rspec-core (>= 3.0.0)
+      rspec-expectations (>= 3.0.0)
+    rspec-mocks (3.1.3)
+      rspec-support (~> 3.1.0)
+    rspec-support (3.1.2)
+    safe_yaml (1.0.4)
+    simplecov (0.9.1)
+      docile (~> 1.1.0)
+      multi_json (~> 1.0)
+      simplecov-html (~> 0.8.0)
+    simplecov-html (0.8.0)
+    thread_safe (0.3.4)
+    tzinfo (1.2.2)
+      thread_safe (~> 0.1)
+    vcr (2.9.3)
+    webmock (1.20.4)
+      addressable (>= 2.3.6)
+      crack (>= 0.3.2)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.6)
+  contentful-importer!
+  rake
+  rspec (~> 3)
+  rspec-its (~> 1.1.0)
+  simplecov
+  vcr (~> 2.9.3)
+  webmock (>= 1.20)

data/LICENSE

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Contentful
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

data/README.md

@@ -0,0 +1,327 @@
+Contentful Generic-importer
+=================
+
+## Description
+
+This allows you to import structured JSON data to [Contentful](https://www.contentful.com).
+
+You can use one of the following tools to extract your content and make it ready for import:
+
+#### Available exporters
+
+ - [Database](https://github.com/contentful/database-adapter)
+ - [Wordpress](https://github.com/contentful/wordpress-adapter)
+ - [Drupal](https://github.com/contentful/drupal-adapter)
+
+
+## Installation
+
+```bash
+gem install contentful-importer
+```
+
+This will install the ```contentful-importer``` executable.
+
+
+## Usage
+
+Before you can import your content to Contentful, you need to extract and prepare it. We currently support the mentioned exporters, but you can also create your own tools.
+
+As a first step you should create a `settings.yml` file and fill in your credentials:
+
+```yaml
+#Contentful
+access_token: access_token
+organization_od: organization_id
+```
+
+**A Contentful OAuth access token can be created using the [Contentful Management API - documentation](https://www.contentful.com/developers/documentation/content-management-api/#getting-started)**
+
+The Contentful organization id can be found in your account settings.
+
+Once you installed the Gem and created the YAML file with the settings you can invoke the tool using:
+
+```bash
+contentful-importer --config-file settings.yml --action
+```
+
+## Step by step
+
+1. Create YAML file with required parameters (eg. ```settings.yml```):
+
+    ```yaml
+    #PATH to all data
+    data_dir: DEFINE_BEFORE_EXPORTING_DATA
+
+    #Contentful credentials
+    access_token: ACCESS_TOKEN
+    organization_id: ORGANIZATION_ID
+    space_id: DEFINE_AFTER_CREATING_SPACE
+    default_locale: DEFINE_LOCALE_CODE
+
+    ## CONTENTFUL STRUCTURE
+    contentful_structure_dir: PATH_TO_CONTENTFUL_STRUCTURE_JSON_FILE
+
+    ## CONVERT CONTENTFUL MODEL TO CONTENTFUL IMPORT STRUCTURE
+    content_model_json:
+    converted_model_dir:
+    ```
+
+2. Create the contentful_structure.json. First you need to create a content model using the [Contentful web application](www.contentful.com). Then you can download the content model using the content management api and use the content model for the import:
+
+    ```bash
+     curl -X GET \
+          -H 'Authorization: Bearer ACCESS_TOKEN' \
+          'https://api.contentful.com/spaces/SPACE_ID/content_types' > contentful_model.json
+    ```
+
+    It will create ```contentful_model.json``` file, which you need to transform into the ```contentful_structure.json``` using:
+
+    ```bash
+    contentful-importer --config-file settings.yml --convert-content-model-to-json
+    ```
+
+    The converted content model will be saved as JSON file in the ```converted_model_dir``` path.
+
+
+3. Once you have prepared the `content types`, `assets` and `entries` (for example using one of the existing extraction adapters or creating your own) they can be imported. It can be chosen to use one (default) or two parallel threads to speedup this process.
+   There are two steps to import entries and assets.
+
+    **Entries**
+
+    ```bash
+    contentful-importer --config-file settings.yml --import
+    ```
+
+    or
+
+    ```bash
+    contentful-importer --config-file settings.yml --import --threads 2
+    ```
+
+    **Assets**
+
+    ```bash
+    contentful-importer --config-file settings.yml --import-assets
+    ```
+
+    After each request the `success_number_of_thread.csv` or `success_assets.csv` file is updated. You can find those in `data_dir/logs`.
+    If an entry or asset fails to be imported, it will end up in the `failure_number_of_thread.csv` or `assets_failure.csv` including the error message.
+
+
+4. Publish entries and assets. After successfully importing the entries and assets to contentful, they need to be published in order to be available through the delivery API.
+
+    To publish all entries use:
+
+    ```bash
+    contentful-importer --config-file settings.yml --publish-entries
+    ```
+
+    To publish all assets use:
+
+    ```bash
+    contentful-importer --config-file settings.yml --publish-assets
+    ```
+
+    or
+
+    ```bash
+    contentful-importer --config-file settings.yml --publish-assets --threads 2
+    ```
+
+    After each request the ```success_published_entries.csv``` or ```success_published_assets.csv``` file is updated. You can find those in ```data_dir/logs```.
+    If an entry or asset fails to be imported, it will end up in the ```failure_published_entries.csv``` or ```failure_published_assets.csv``` including the error message.
+
+
+## Actions
+
+To display all actions use the `-h` option:
+
+```bash
+contentful-importer -h
+```
+#### --convert-content-model-to-json
+
+If you already have an existing content model for a space it can be downloaded and used for the import:
+
+```bash
+ curl -X GET \
+      -H 'Authorization: Bearer ACCESS_TOKEN' \
+      'https://api.contentful.com/spaces/SPACE_ID/content_types' > contentful_model.json
+```
+
+
+In the **settings.yml** specify the PATH to **contentful_model.json**.
+
+```yaml
+#Dump file with content model.
+content_model_json: example_path/contentful_model.json
+```
+
+and define the PATH where you want to save the converted JSON file:
+
+```yaml
+#File with converted structure of contentful model. Almost ready to import.
+converted_model_dir: example_path/contentful_structure.json
+```
+
+#### --create-contentful-model-from-json
+
+Create the content model based on the structure defined in ```contentful_structure.json```.
+This will generate content types files that are ready to be imported.
+Set the `contentful_structure_dir` variable to point to the structure file in your settings file.
+
+Path to collections: **data_dir/collections**
+
+```bash
+contentful-importer --config-file settings.yml --create-contentful-model-from-json
+```
+
+This is optional if you intend to create your own content structure as JSON files and not use the web application for it.
+
+#### --import-content-types ARGS
+
+To import the content types into an existing space use `--space_id SPACE_ID`:
+
+```bash
+contentful-importer --config-file settings.yml --import-content-types --space_id SPACE_ID
+```
+
+To create a new Space and import content types use `--space_name NAME`:
+
+```bash
+contentful-importer --config-file settings.yml --import-content-types --space_name NAME
+```
+
+#### --import
+
+Import the entries with a single thread:
+
+```bash
+contentful-importer --config-file settings.yml --import
+```
+
+or using two threads
+
+```bash
+contentful-importer --config-file settings.yml --import --threads 2
+```
+
+#### --import-assets
+
+Import the assets:
+
+```bash
+contentful-importer --config-file settings.yml --import-assets
+```
+
+#### --publish-entries
+
+To publish all entries:
+
+```bash
+contentful-importer --config-file settings.yml --publish-entries
+```
+Number of threads that are used in the publishing of entries is dependent on `--threads` argument, which you specified when import data.
+
+#### --publish-assets ARGS
+
+You can publish all assets with single Thread:
+
+```bash
+contentful-importer --config-file settings.yml --publish-assets
+```
+or add ```--threads``` argument to use multiple Threads:
+
+```bash
+contentful-importer --config-file settings.yml --publish-assets --threads 2
+```
+
+#### --test-credentials
+
+Before importing any content you can verify that your credentials in the **settings.yml** file are correct:
+
+```bash
+contentful-importer --config-file settings.yml --test-credentials
+```
+
+#### --validate-schema
+
+After preparing the files to import, you can validate the JSON schema, use command:
+
+```contentful-importer --config-file settings.yml --validate-schema```
+
+This comes in handy when you want to create your own extraction adapter.
+
+## Content Model
+
+This represents an example content model:
+
+```javascript
+{
+    "Comments": {
+        "id": "comment",
+        "description": "",
+        "displayField": "title",
+        "fields": {
+            "title": "Text",
+            "content": "Text"
+        }
+    },
+    "JobAdd": {
+        "id": "job_add",
+        "description": "Add new job form",
+        "displayField": "name",
+        "fields": {
+            "name": "Text",
+            "specification": "Text",
+            "Images": {
+                "id": "image",
+                "link_type": "Asset"
+            },
+            "Comments": {
+                "id": "comments",
+                "link_type": "Array",
+                "type": "Entry"
+            },
+            "Skills": {
+                "id": "skills",
+                "link_type": "Array",
+                "type": "Entry"
+            }
+        }
+    }
+```
+Key names "Images", "Comments", "Skills" are the equivalent of the content type names specified in the file **mapping.json**.
+
+Example:
+
+```javascript
+     "SkillsTableName": {
+         "content_type": "Skills",
+         "type": "entry",
+         "fields": { ... }
+```
+
+
+# Import
+
+Before you start importing the content make sure you read [how to use it](https://github.com/contentful/generic-importer.rb#usage) and [tested your credentials](https://github.com/contentful/generic-importer.rb#--test-credentials).
+
+#### Space ID
+
+After [importing the content types](https://github.com/contentful/generic-importer.rb#--import-content-types-args) to the Space, you need to specify the `space_id` parameter in the settings.
+
+
+Example:
+```yml
+space_id: space_id
+```
+
+#### Default locale
+
+To specify in which locale you want to create all Entries and Assets, set ```default_locale``` parameter in settings.yml file:
+
+```yml
+default_locale: de-DE
+```