contentful-database-importer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 98d3aacd7022ac369a2b9b81cd21048d52d1da11
+  data.tar.gz: 063b897d2875b9ac709da9f930035f108b41c135
+SHA512:
+  metadata.gz: 9b3a9767b6a65e6e5fc9ad2efb26e68190411b358f35e9fa25e8075cbfbae1943e6851f759447d76a819dd9f1baa91e610e86af3277c970e274e914c2309f585
+  data.tar.gz: 99db145e30c2adf4bfb8b3e07c49d5626f6576f331aaccd7ed41a142eacbe0a336ecd074671667cea63ffb5cb25ee7407d4908275b9ab26ebad12fc6ebedacca

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.12.5

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at david.litvakb@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/CONTRIBUTING.md

@@ -0,0 +1,43 @@
+# Contributing
+In the spirit of [free software][free-sw], **everyone** is encouraged to help
+improve this project.
+
+[free-sw]: http://www.fsf.org/licensing/essays/free-sw.html
+
+Here are some ways *you* can contribute:
+
+* by using alpha, beta, and prerelease versions
+* by reporting bugs
+* by suggesting new features
+* by writing or editing documentation
+* by writing specifications
+* by writing code ( **no patch is too small** : fix typos, add comments, clean up inconsistent whitespace )
+* by refactoring code
+* by closing [issues][]
+* by reviewing patches
+
+[issues]: https://github.com/contentful/contentful-database-importer.rb/issues
+
+## Submitting an Issue
+We use the [GitHub issue tracker][issues] to track bugs and features. Before
+submitting a bug report or feature request, check to make sure it hasn't
+already been submitted. When submitting a bug report, please include a [Gist][]
+that includes a stack trace and any details that may be necessary to reproduce
+the bug, including your gem version, Ruby version, and operating system.
+Ideally, a bug report should include a pull request with failing specs.
+
+[gist]: https://gist.github.com/
+
+## Submitting a Pull Request
+1. [Fork the repository.][fork]
+2. [Create a topic branch.][branch]
+3. Add specs for your unimplemented feature or bug fix.
+4. Run `bundle exec rake test`. If your specs pass, return to step 3.
+5. Implement your feature or bug fix.
+6. Run `bundle exec rake test`. If your specs fail, return to step 5.
+7. Add, commit, and push your changes.
+8. [Submit a pull request.][pr]
+
+[fork]: http://help.github.com/fork-a-repo/
+[branch]: http://learn.github.com/p/branching.html
+[pr]: http://help.github.com/send-pull-requests/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in bootstrap-database.gemspec
+gemspec

data/Guardfile

@@ -0,0 +1,14 @@
+guard :rspec, cmd: "bundle exec rspec" do
+  require "guard/rspec/dsl"
+  dsl = Guard::RSpec::Dsl.new(self)
+
+  # RSpec files
+  rspec = dsl.rspec
+  watch(rspec.spec_helper) { rspec.spec_dir }
+  watch(rspec.spec_support) { rspec.spec_dir }
+  watch(rspec.spec_files)
+
+  # Ruby files
+  ruby = dsl.ruby
+  dsl.watch_spec_files_for(ruby.lib_files)
+end

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Contentful GmbH - David Litvak Bruno
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,253 @@
+# Contentful Database Importer
+
+![TravisCI](https://travis-ci.org/contentful/contentful-database-importer.rb.svg?branch=master)
+
+A simple DSL to define your database schemas, their relation to Contentful and import them to Contentful.
+
+This gem is intended to be a replacement to [`database_exporter`](https://github.com/contentful/database-exporter.rb). _Warning_: Both gems are incompatible.
+
+## Contentful
+
+[Contentful](http://www.contentful.com) is a content management platform for web applications, mobile apps and connected devices.
+It allows you to create, edit and manage content in the cloud and publish it anywhere via a powerful API.
+Contentful offers tools for managing editorial teams and enabling cooperation between organizations.
+
+## What does `contentful-database-importer` do?
+
+`contentful-database-importer` let's you define mapping classes between your database and Contentful and allows
+you to generate a JSON file that's a valid [`contentful_bootstrap`](https://github.com/contentful/contentful-bootstrap.rb) JSON Template,
+or directly import to Contentful, creating a new space and using your data to populate the content.
+
+
+## Installation
+
+```bash
+gem install contentful-database-importer
+```
+
+## Usage
+
+* Create a new directory for your import configuration:
+
+```bash
+mkdir my_importer_dir && cd my_importer_dir
+```
+
+* Create a _`Gemfile`_ with the gem:
+
+```ruby
+source 'https://rubygems.org'
+
+gem 'contentful-database-importer'
+```
+
+* Create your importer file, for example _`import.rb`_:
+
+```ruby
+require 'contentful/database_importer'
+
+class MyTable
+  include Contentful::DatabaseImporter::Resource
+
+  # ... your schema definition ... (explained in next section)
+end
+
+# ... more table definitions ...
+
+Contentful::DatabaseImporter.setup do |config|
+  config.space_name = 'My Cool New Space'
+  config.database_connection = 'postgres://user:pass@host:port'
+end
+
+Contentful::DatabaseImporter.run!
+```
+
+* Run your file:
+
+```bash
+bundle exec ruby import.rb
+```
+
+### Defining your Schema
+
+```ruby
+class MyTable
+  include Contentful::DatabaseImporter::Resource
+
+  self.table_name = 'overrides_table_name' # Optional - By default it's the class name in snake case. E.g. 'my_table'
+  self.content_type_id = 'overrides_content_type_id' # Optional - By default it's the class name in snake case. E.g 'my_table'
+  self.content_type_name = 'Overrides Name' # Optional - By default it's the class name
+
+  id Contentful::DatabaseImporter::IdGenerator::Base, template: '{{content_type_id}}_{{foo}}_{{index}}' # Optional - By default it's the IdGenerator::Base(template: '{{content_type_id}}_{{index}}')
+
+  field :foo, type: :string
+  field :bar, maps_to: :not_bar, type: :string
+  field :image, type: :asset
+end
+```
+
+#### Overriding Table and Content Type ID
+
+The methods `::table_name=` and `::content_type_id=` allow you to override the IDs for either the table or content type.
+By default, they are a `snake_cased` version of the class name.
+
+#### Defining the ID generator
+
+You can define the ID generation strategy, there are 2 classes currently provided:
+
+- `Contentful::DatabaseImporter::IdGenerator::Base`: Provides a very basic template engine for generating IDs, this is the default strategy.
+- `Contentful::DatabaseImporter::IdGenerator::ContentfulLike`: Provides a Base62 encode that produces IDs similar to the Contentful provided ones,
+  uses the `Base` strategy, then pads it to a minimum length and then Base62 encode it.
+
+##### ID Templates
+
+Theres a minimal template engine provided for the ID Generators.
+A single template looks like `{{foo}}_{{bar}}` and works by replacing the values enclosed between `{{}}` with the corresponding value for each entry.
+
+There are a few variables globally provided for every class (and will be looked up before the object fields):
+
+- `class_name`: The name of the mapping class
+- `table_name`: The defined table name (or the default)
+- `content_type_id`: The defined content type ID (or the default)
+- `index`: The position of the entry (0-based) on the database table
+
+After those globally provided values, you can use the value for any field on the mapping class (using the `:maps_to` value if present).
+
+For example, using the example template above, if the DB record looks like:
+
+```ruby
+{foo: 'something', bar: 'else', image: 'https://example.com/happycat.jpg'}
+```
+
+Then the resulting template will be `something_else`
+
+**Note**: With relationships, it's useful to use a unique identifier value as part of the ID template.
+
+#### Defining Fields
+
+For defining the field you have the `::field(name, options = {})` method. It defines how to retrieve and later serialize the field.
+
+The options are:
+
+- `type: type_name`: **Required** for coercions. Types defined below.
+- `maps_to: name`: **Optional**. Defaults to field name, and defines the field name in Contentful
+- `pre_process: lambda_or_symbol`: **Optional**. Described below.
+- `exclude_from_output: boolean`: **Optional**. Defaults to false. Defines if the field will not be uploaded to Contentful. Useful for ID generation.
+
+##### Regular Field Types
+
+- `:symbol`, `:string`: Short text field (255 characters maximum) in Contentful.
+- `:text`: Long text field.
+- `:number`: Floating point precision number.
+- `:integer`: Integer number.
+- `:boolean`: Boolean.
+- `:location`: Geographical Location (can be coerced from a String, Hash or Array.)
+- `:date`: An ISO8601 Date (can be coerced from a Date/DateTime object or String).
+- `:object`: A JSON Object. _Currently not supported_
+- `:asset`: A File description. A String containing the file URL needs to be provided.
+- `:array`: An Array of elements.
+
+In the case of using `:array`, an extra parameter `item_type: type` must be provided.
+
+##### Relationship Field Types
+
+If your data has a relationship field, the `type:` value will be the related class, and will require additional parameters specifying
+the relationship type and keys for retrieving the appropiate data.
+
+For example:
+
+```ruby
+class Foo
+  field :bars, type: Bar, relationship: :many, id_field: :id, key: :foo_id
+  field :baz, type: Baz, relationship: :one, id_field: :id, key: :baz_id
+  field :quxs, type: Qux, relationship: :many_through, through: :foo_qux, primary_id_field: :id, primary_key: :foo_id, foreign_key: :qux_id, foreign_id_field: :id
+end
+```
+
+In Contentful, relationships are unidirectional, and if you want bidirectional relationships, you need to declare them in both classes.
+
+Relationship fields have the particularity that they don't require the `:maps_to` property, as Contentful will always use
+the field name for the property in Contentful. You define the name of the field in the database with relationship specific parameters.
+
+Relationship Types:
+
+- `:many`: One to Many relationship, looks for all related objects of the associated class that match the value of the `:id_field` via the `:key`.
+  In the example above, it will look for all `Bar` entries which have a `:foo_id` that match the value of `:id` for the current `Foo` entry.
+- `:one`: One to One relationship, looks for the related object of the associated class that matches the value of the `:key` field in the current entry, with the value of `:id_field` in the related entry.
+  In the example above, it will look for the `Baz` entry which has an ID that matches the value of `:baz_id` in the current entry.
+- `:many_through`: Many to Many relationship, looks for the related object through an intermediate lookup table, after this it behaves like `:many`.
+  In the example above, it will look for all `Qux` entries found in the intermediate table that match the current entry `:id` and looks it up via the `Qux`s `:id`.
+
+**Note**: If you're using relationships, use a custom ID Generator template which includes a unique field for each entry,
+that way, creating the links in Contentful will be successful. This requires including the field in the class definition.
+
+For example: `'{{content_type_id}}_{{id}}'`
+
+**Note**: Ruby requires a class to be defined before using it as a parameter, therefore, you should declare all classes that
+are contained within others, before the one in which you use them. If you want to have circular relationships, you need to define a Merge Class pointing to the same table and content type as the desired class (defined below).
+
+##### Pre-processing
+
+If you want to transform your data before uploading to Contentful,
+you can use the `:pre_process` parameter in the `::field` definition.
+
+The pre-process value can be a lambda function (E.g. `-> (value) { value + 1 }`) or a symbol (E.g. `:pre_process_foo`).
+
+If you use a lambda function, it must receive a single parameter and return a single value.
+
+If you use a Symbol, it must match the name of a method defined within the class you're calling it from.
+This method must receive a single parameter and return a single value.
+
+### Merging Tables
+
+You might want to merge the content of multiple tables into a single content type.
+
+This is supported by default, but ensure that the classes have the same `content_type_id` defined and if you need to
+merge the entries as well, that the ID generator template is set in a way that can match the values from the different classes.
+
+In the case you want to create multiple content types from a single table, the same concepts apply.
+
+In the case of circular references, you will have to create 2 or more classes pointing to the same table and content type, the same concepts apply.
+
+**Note**: Merge classes require at least 1 field declared, even if it's excluded from output.
+
+### Configuration
+
+```ruby
+Contentful::DatabaseImporter.setup do |config|
+  config.space_name = 'My Cool New Space' # Required - the destination space name
+  config.database_connection = 'postgres://user:pass@host:port' # Required - the DB Connection string
+end
+```
+
+`database_connection` allows the following Database URI Strings:
+
+- **[Postgres (Section 31.1.1.2)](https://www.postgresql.org/docs/9.3/static/libpq-connect.html#LIBPQ-CONNSTRING)**: E.g. `'postgres://user:password@host:port/database_name'`.
+- **SQlite**: E.g. `'sqlite://file_path.db'`.
+- **MySQL**: E.g. `'mysql://user:password@host:post/database_name'`.
+
+### Running the Import Tool
+
+You can do any of the following operations:
+
+* Generate a JSON Template as a Ruby Hash for reuse within the script:
+
+```ruby
+Contentful::DatabaseImporter.generate_json
+```
+
+* Generate JSON Template as a prettyfied JSON string:
+
+```ruby
+Contentful::DatabaseImporter.generate_json!
+```
+
+* Generate the JSON and Import it to Contentful (creates a Space with all the content):
+
+```ruby
+Contentful::DatabaseImporter.run!
+```
+
+## Contributing
+
+Feel free to improve this tool by submitting a Pull Request. For more information, please read [CONTRIBUTING.md](./CONTRIBUTING.md)

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/contentful-database-importer.gemspec

@@ -0,0 +1,36 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'contentful/database_importer/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "contentful-database-importer"
+  spec.version       = Contentful::DatabaseImporter::VERSION
+  spec.authors       = ["Contentful GmbH (David Litvak Bruno)"]
+  spec.email         = ["david.litvak@contentful.com"]
+
+  spec.summary       = %q{Tool to import content from a Database to Contentful}
+  spec.description   = %q{Tool to import content from a Database to Contentful}
+  spec.homepage      = "https://contentful.com"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'contentful_bootstrap', '~> 3.0'
+  spec.add_dependency 'contentful-management', '~> 0.9'
+  spec.add_dependency 'sequel', '~> 4.39'
+  spec.add_dependency 'base62-rb'
+  spec.add_dependency 'mimemagic'
+
+  spec.add_development_dependency "bundler", "~> 1.12"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency 'rubocop'
+  spec.add_development_dependency 'simplecov'
+  spec.add_development_dependency 'guard'
+  spec.add_development_dependency 'guard-rspec'
+  spec.add_development_dependency 'pry'
+end

data/lib/contentful/database_importer/config.rb

@@ -0,0 +1,12 @@
+module Contentful
+  module DatabaseImporter
+    # Configuration for Importer
+    class Config
+      attr_accessor :space_name, :database_connection
+
+      def complete?
+        !space_name.nil? && !database_connection.nil?
+      end
+    end
+  end
+end

data/lib/contentful/database_importer/id_generator/base.rb

@@ -0,0 +1,37 @@
+module Contentful
+  module DatabaseImporter
+    module IdGenerator
+      # Base Id Generator
+      class Base
+        attr_reader :options
+
+        def initialize(options = {})
+          @options = options
+        end
+
+        def run(entry_data, index)
+          options.fetch(:template, '').gsub(/\{\{([\w|\.]+)\}\}/) do |match|
+            find(entry_data, index, match.gsub('{{', '').gsub('}}', '').to_sym)
+          end
+        end
+
+        def find(entry_data, index, match)
+          return options[match] if options.key?(match)
+          return index.to_s if match == :index
+
+          find_on_entry(entry_data, match)
+        end
+
+        def find_on_entry(entry_data, match)
+          if entry_data.excluded_fields.key?(match)
+            return entry_data.excluded_fields[match]
+          elsif entry_data.bootstrap_fields.key?(match)
+            return entry_data.bootstrap_fields[match]
+          end
+
+          raise "Template could not be resolved, #{match} not found."
+        end
+      end
+    end
+  end
+end

data/lib/contentful/database_importer/id_generator/contentful_like.rb

@@ -0,0 +1,23 @@
+require 'contentful/database_importer/id_generator/base'
+require 'base62-rb'
+
+module Contentful
+  module DatabaseImporter
+    module IdGenerator
+      # Base62 Encoded Id Generator
+      class ContentfulLike < Base
+        def run(entry_data, index)
+          result = ''
+          id = super(entry_data, index)
+          id.each_char do |c|
+            result << c.ord.to_s
+          end
+
+          result << '9' while result.size < 40
+
+          Base62.encode(result.to_i)
+        end
+      end
+    end
+  end
+end

data/lib/contentful/database_importer/id_generator.rb

@@ -0,0 +1,2 @@
+require 'contentful/database_importer/id_generator/base'
+require 'contentful/database_importer/id_generator/contentful_like'

data/lib/contentful/database_importer/json_generator.rb

@@ -0,0 +1,82 @@
+require 'json'
+
+module Contentful
+  module DatabaseImporter
+    # Json File Generator
+    module JsonGenerator
+      def self.generate_json
+        result = { version: 3, contentTypes: [],
+                   assets: [], entries: {} }
+
+        resources.each do |resource|
+          content_type_definition = resource.content_type_definition
+          create_content_type(content_type_definition, result)
+          create_entries(resource, content_type_definition, result)
+        end
+
+        result
+      end
+
+      def self.resources
+        ObjectSpace.each_object(Class).select do |c|
+          c.included_modules.include? Resource
+        end
+      end
+
+      def self.create_content_type(content_type_definition, result)
+        prev_ct_definition = result[:contentTypes].find do |ct|
+          ct[:id] == content_type_definition[:id]
+        end
+
+        if prev_ct_definition
+          result[:contentTypes].delete(prev_ct_definition)
+          content_type_definition.merge!(prev_ct_definition)
+        end
+
+        result[:contentTypes] << content_type_definition
+      end
+
+      def self.create_entries(resource, content_type_definition, result)
+        result[:entries][content_type_definition[:id]] ||= []
+        resource.all.each do |entry|
+          create_entry(entry, content_type_definition, result)
+        end
+      end
+
+      def self.previous_entry(entry_definition, content_type_definition, result)
+        result[:entries][content_type_definition[:id]].find do |e|
+          e[:sys][:id] == entry_definition[:sys][:id]
+        end
+      end
+
+      def self.merge_entries(entry_definition, content_type_definition, result)
+        prev_entry = previous_entry(
+          entry_definition,
+          content_type_definition,
+          result
+        )
+
+        return unless prev_entry
+
+        result[:entries][content_type_definition[:id]].delete(prev_entry)
+        entry_definition.merge!(prev_entry)
+      end
+
+      def self.create_entry(entry, content_type_definition, result)
+        entry_definition = entry.to_bootstrap
+
+        merge_entries(entry_definition, content_type_definition, result)
+
+        result[:assets].concat(
+          entry.associated_assets
+        ) unless entry.associated_assets.empty?
+
+        result[:entries][content_type_definition[:id]] << entry_definition
+      end
+
+      def self.generate_json!
+        JSON.pretty_generate(generate_json)
+      end
+    end
+  end
+end