lacerda 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b9e353a73a35769ada04cc9efa1a32dfbedc1fee
+  data.tar.gz: a18205aa09824893c34a3c861dff1995f92fd28a
+SHA512:
+  metadata.gz: 605e8d8df037122645b4a113180ab043cb7472ca35792696e570c9398510b2c4f2703600cce3c3fdceee6f3fe473bbba44c1d259b7dfe9842cac85b31cb4ab0b
+  data.tar.gz: 4e9ad371ccb84ba3667c973011d216d0bddae5454a85a4aadfc638053d71c83ddb537cd1ae300561384b6e51c94e2929359c583dac46618b55d2a253f1f6b367

data/.gitignore

@@ -0,0 +1,10 @@
+contracts/**/*.json
+**/*.schema.json
+spec/support/**.json
+tags
+gems.tags
+**/*blueprint-ast.json
+coverage
+**/*.gem
+Gemfile.lock
+tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format documentation

data/.ruby-gemset

@@ -0,0 +1 @@
+minimum-term

data/.ruby-version

@@ -0,0 +1 @@
+2

data/CHANGELOG.markdown

@@ -0,0 +1,8 @@
+# 0.2.4 (23-Oct-15)
+- fix deep clone bug for schema scoping
+
+# 0.2.3 (22-Oct-15)
+- ignore consumed objects from other publishers when validating
+
+# 0.2.2
+- allow missing property type in MSON and default to 'object'

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/Guardfile

@@ -0,0 +1,41 @@
+# A sample Guardfile
+# More info at https://github.com/guard/guard#readme
+
+## Uncomment and set this to only include directories you want to watch
+# directories %w(app lib config test spec features) \
+#  .select{|d| Dir.exists?(d) ? d : UI.warning("Directory #{d} does not exist")}
+
+## Note: if you are using the `directories` clause above and you are not
+## watching the project directory ('.'), then you will want to move
+## the Guardfile to a watched dir and symlink it back, e.g.
+#
+#  $ mkdir config
+#  $ mv Guardfile config/
+#  $ ln -s config/Guardfile .
+#
+# and, you'll have to watch "config/Guardfile" instead of "Guardfile"
+
+guard :bundler do
+  watch('Gemfile')
+  # Uncomment next line if your Gemfile contains the `gemspec' command.
+  watch(/^.+\.gemspec/)
+end
+
+guard 'ctags-bundler', :src_path => ["lib"] do
+  watch(/^(app|lib|spec\/support)\/.*\.rb$/)
+  watch('Gemfile.lock')
+  watch(/^.+\.gemspec/)
+end
+
+guard :rspec, cmd: 'IGNORE_LOW_COVERAGE=1 rspec', all_on_start: true  do
+  watch(%r{^spec/support/.*\.mson$}) { "spec" }
+  watch(%r{^spec/support/.*\.rb$}) { "spec" }
+  watch('spec/spec_helper.rb')  { "spec" }
+
+  # We could run individual specs, sure, but for now I dictate the tests
+  # are only green when we have 100% coverage, so partial runs will never
+  # succeed. Therefore, always run all the things.
+  watch(%r{^(spec/.+_spec\.rb)$}) { "spec" }
+  watch(%r{^lib/(.+)\.rb$})     { "spec" }
+end
+

data/README.md

@@ -0,0 +1,65 @@
+# Lacerda [![Circle CI](https://circleci.com/gh/moviepilot/minimum-term/tree/master.svg?style=svg)](https://circleci.com/gh/moviepilot/minimum-term/tree/master) [![Coverage Status](https://coveralls.io/repos/moviepilot/minimum-term/badge.svg?branch=master&service=github)](https://coveralls.io/github/moviepilot/minimum-term?branch=master) [![Code Climate](https://codeclimate.com/github/moviepilot/minimum-term/badges/gpa.svg)](https://codeclimate.com/github/moviepilot/minimum-term) [![Dependency Status](https://gemnasium.com/moviepilot/minimum-term.svg)](https://gemnasium.com/moviepilot/minimum-term)
+
+![](https://dl.dropboxusercontent.com/u/1953503/lacerda.jpg)
+> «We need total coverage»<sup>[1](http://www.dailyscript.com/scripts/fearandloathing.html)</sup>
+
+This gem can:
+
+- convert MSON to JSON Schema files
+- read a directory which contains one directory per service
+- read a publish.mson and a consume.mson from each service
+- build a model of your infrastructure knowing
+  - which services publishes what to which other service
+  - which service consumes what from which other service
+  - if all services consume and publish conforming to their contracts.
+
+You likely don't want to use it on its own but head on over to the [Zeta](https://github.com/moviepilot/zeta) gem which explains things in more detail. If you're just looking for ONE way to transform MSON files into JSON Schema, read on:
+
+## Getting started
+First, check out [this API Blueprint map](https://github.com/apiaryio/api-blueprint/wiki/API-Blueprint-Map) to understand how _API Blueprint_ documents are laid out:
+
+![API Blueprint map](https://raw.githubusercontent.com/apiaryio/api-blueprint/master/assets/map.png)
+
+You can see that their structure covers a full API use case with resource groups, single resources, actions on those resources including requests and responses. All we want, though, is the little red top level branch called `Data structures`.
+
+We're using a ruby gem called [RedSnow](https://github.com/apiaryio/redsnow), which has bindings to [SnowCrash](https://github.com/apiaryio/snowcrash) which parses _API Blueprints_ into an AST.
+
+Luckily, a rake task does all that for you. To convert all `*.mson` files in `contracts/` into `*.schema.json` files,
+
+put this in your `Rakefile`:
+
+```ruby
+require "lacerda/tasks"
+```
+
+and smoke it:
+
+```shell
+/home/dev/lacerda$ DATA_DIR=contracts/ rake lacerda:mson_to_json_schema
+Converting 4 files:
+OK /home/dev/lacerda/contracts/consumer/consume.mson
+OK /home/dev/lacerda/contracts/invalid_property/consume.mson
+OK /home/dev/lacerda/contracts/missing_required/consume.mson
+OK /home/dev/lacerda/contracts/publisher/publish.mson
+/home/dev/lacerda$
+```
+
+## Tests and development
+  - run `bundle` once
+  - run `guard` in a spare terminal which will run the tests,
+    install gems, and so forth
+  - run `rspec spec` to run all the tests
+  - check out  `open coverage/index.html` or `open coverage/rcov/index.html`
+  - run `bundle console` to play around with a console
+
+## Structure
+
+By converting all files in a directory this gem will build up the following relationships:
+
+- Infrastructure
+  - Service
+    - Contracts
+      - Publish contract
+        - PublishedObjects
+      - Consume contract
+        - ConsumedObjects

data/Rakefile

@@ -0,0 +1,9 @@
+#encoding: utf-8
+#!/usr/bin/env ruby
+
+require 'bundler/gem_tasks'
+Bundler.require
+
+$:.unshift File.join(File.dirname(__FILE__), "lib")
+
+require 'lacerda/tasks'

data/circle.yml

@@ -0,0 +1,3 @@
+machine:
+  ruby:
+    version: 2.2.1

data/lacerda.gemspec

@@ -0,0 +1,35 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'lacerda/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "lacerda"
+  spec.version       = Lacerda::VERSION
+  spec.authors       = ["Jannis Hermanns"]
+  spec.email         = ["jannis@gmail.com"]
+
+  spec.summary       = 'Markdown publish/consume contract parser and validator'
+  spec.description   = 'Specify which objects your services publish or consume in MSON (markdown) and let this gem validate these contracts.'
+  spec.homepage      = "https://github.com/moviepilot/lacerda"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "bin"
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+  spec.license       = 'MIT'
+
+  spec.add_runtime_dependency "activesupport"
+  spec.add_runtime_dependency "rake",          ["~> 10.2"]
+  spec.add_runtime_dependency "json-schema",   ["~> 2.5"]
+  spec.add_runtime_dependency "redsnow",       ["~> 0.4"]
+  spec.add_runtime_dependency "colorize"
+
+  spec.add_development_dependency "bundler",             ["~> 1"]
+  spec.add_development_dependency "guard-bundler",       ["~> 2.1"]
+  spec.add_development_dependency "guard-ctags-bundler", ["~> 1.4"]
+  spec.add_development_dependency "guard-rspec",         ["~> 4.6"]
+  spec.add_development_dependency "rspec",               ["~> 3.3"]
+  spec.add_development_dependency "coveralls",           ["~> 0.8"]
+  spec.add_development_dependency "codeclimate-test-reporter"
+end

data/lib/lacerda.rb

@@ -0,0 +1,10 @@
+require "lacerda/conversion"
+require "lacerda/publish_contract"
+require "lacerda/consume_contract"
+require "lacerda/service"
+require "lacerda/infrastructure"
+require "lacerda/compare/json_schema"
+
+module Lacerda
+  SCOPE_SEPARATOR = ':'
+end

data/lib/lacerda/compare/json_schema.rb

@@ -0,0 +1,123 @@
+module Lacerda
+  module Compare
+    class JsonSchema
+      ERRORS = {
+        :ERR_ARRAY_ITEM_MISMATCH  => nil,
+        :ERR_MISSING_DEFINITION   => nil,
+        :ERR_MISSING_POINTER      => nil,
+        :ERR_MISSING_PROPERTY     => nil,
+        :ERR_MISSING_REQUIRED     => nil,
+        :ERR_MISSING_TYPE_AND_REF => nil,
+        :ERR_TYPE_MISMATCH        => nil,
+        :ERR_NOT_SUPPORTED        => nil
+      }
+
+      attr_reader :errors
+
+      def initialize(containing_schema)
+        @containing_schema = containing_schema
+      end
+
+      def contains?(contained_schema, pry = false)
+        @errors = []
+        @contained_schema = contained_schema
+        definitions_contained?
+      end
+
+      private
+
+      def definitions_contained?
+        @contained_schema['definitions'].each do |property, contained_property|
+          containing_property = @containing_schema['definitions'][property]
+          return _e(:ERR_MISSING_DEFINITION, [property]) unless containing_property
+          return false unless schema_contains?(containing_property, contained_property, [property])
+        end
+        true
+      end
+
+      def _e(error, location, extra = nil)
+        message = [ERRORS[error], extra].compact.join(": ")
+        @errors.push(error: error, message: message, location: location.join("/"))
+        false
+      end
+
+      def schema_contains?(publish, consume, location = [])
+
+        # We can only compare types and $refs, so let's make
+        # sure they're there
+        return _e!(:ERR_MISSING_TYPE_AND_REF) unless
+          (consume['type'] or consume['$ref']) and
+          (publish['type'] or publish['$ref'])
+
+        # There's four possibilities here:
+        #
+        # 1) publish and consume have a type defined
+        # 2) publish and consume have a $ref defined
+        # 3) publish has a $ref defined, and consume an inline object
+        # 4) consume has a $ref defined, and publish an inline object
+        #    (we don't support this yet, as otherwise couldn't check for
+        #    missing definitions, because we could never know if something
+        #    specified in the definitions of the consuming schema exists in
+        #    the publishing schema as an inline property somewhere).
+        #    TODO: check if what I just said makes sense. I'm not sure anymore.
+        # Let's go:
+
+        # 1)
+        if (consume['type'] and publish['type'])
+          if consume['type'] != publish['type']
+            return _e(:ERR_TYPE_MISMATCH, location, "#{consume['type']} != #{publish['type']}")
+          end
+
+        # 2)
+        elsif(consume['$ref'] and publish['$ref'])
+         resolved_consume = resolve_pointer(consume['$ref'], @contained_schema)
+         resolved_publish = resolve_pointer(publish['$ref'], @containing_schema)
+         return schema_contains?(resolved_publish, resolved_consume, location)
+
+        # 3)
+        elsif(consume['type'] and publish['$ref'])
+          if resolved_ref = resolve_pointer(publish['$ref'], @containing_schema)
+            return schema_contains?(resolved_ref, consume, location)
+          else
+            return _e(:ERR_MISSING_POINTER, location, publish['$ref'])
+          end
+
+        # 4)
+        elsif(consume['$ref'] and publish['type'])
+          return _e(:ERR_NOT_SUPPORTED, location)
+        end
+
+        # Make sure required properties in consume are required in publish
+        consume_required = consume['required'] || []
+        publish_required = publish['required'] || []
+        missing = (consume_required - publish_required)
+        return _e(:ERR_MISSING_REQUIRED, location, missing.to_json) unless missing.empty?
+
+        # We already know that publish and consume's type are equal
+        # but if they're objects, we need to do some recursion
+        if consume['type'] == 'object'
+          consume['properties'].each do |property, schema|
+            return _e(:ERR_MISSING_PROPERTY, location, property) unless publish['properties'][property]
+            return false unless schema_contains?(publish['properties'][property], schema, location + [property])
+          end
+        end
+
+        if consume['type'] == 'array'
+          sorted_publish = publish['items'].sort
+          consume['items'].sort.each_with_index do |item, i|
+            next if schema_contains?(sorted_publish[i], item)
+            return _e(:ERR_ARRAY_ITEM_MISMATCH, location)
+          end
+        end
+
+        true
+      end
+
+      def resolve_pointer(pointer, schema)
+        type = pointer[/\#\/definitions\/([^\/]+)$/, 1]
+        return false unless type
+        schema['definitions'][type]
+      end
+    end
+  end
+end

data/lib/lacerda/consume_contract.rb

@@ -0,0 +1,18 @@
+require 'lacerda/contract'
+
+module Lacerda
+  class ConsumeContract < Lacerda::Contract
+    def object_description_class
+      Lacerda::ConsumedObject
+    end
+
+    def scoped_schema(service)
+      # Poor man's deep clone: json 🆗 🆒
+      filtered_schema = JSON.parse(schema.to_json)
+      filtered_schema['definitions'].select! do |k|
+        k.underscore.start_with?(service.name.underscore+Lacerda::SCOPE_SEPARATOR)
+      end
+      filtered_schema
+    end
+  end
+end

data/lib/lacerda/consumed_object.rb

@@ -0,0 +1,16 @@
+require 'lacerda/object_description'
+
+module Lacerda
+  class ConsumedObject < Lacerda::ObjectDescription
+
+    def publisher
+      i = @scoped_name.index(Lacerda::SCOPE_SEPARATOR)
+      return @defined_in_service unless i
+      @defined_in_service.infrastructure.services[@scoped_name[0...i].underscore.to_sym]
+    end
+
+    def consumer
+      @defined_in_service
+    end
+  end
+end

data/lib/lacerda/contract.rb

@@ -0,0 +1,34 @@
+require 'active_support/core_ext/hash/indifferent_access'
+require 'lacerda/published_object'
+require 'lacerda/consumed_object'
+
+module Lacerda
+  class Contract
+    attr_reader :service, :schema
+
+    def initialize(service, schema_or_file)
+      @service = service
+      load_schema(schema_or_file)
+    end
+
+    def objects
+      return [] unless @schema[:definitions]
+      @schema[:definitions].map do |scoped_name, schema|
+        object_description_class.new(service, scoped_name, schema)
+      end
+    end
+
+    private
+
+    def load_schema(schema_or_file)
+      if schema_or_file.is_a?(Hash)
+        @schema = schema_or_file
+      elsif File.readable?(schema_or_file)
+        @schema = JSON.parse(open(schema_or_file).read)
+      else
+        @schema = {}
+      end
+      @schema = @schema.with_indifferent_access
+    end
+  end
+end

data/lib/lacerda/conversion.rb

@@ -0,0 +1,132 @@
+require 'fileutils'
+require 'open3'
+require 'lacerda/conversion/apiary_to_json_schema'
+require 'lacerda/conversion/error'
+require 'redsnow'
+
+module Lacerda
+  module Conversion
+    def self.mson_to_json_schema(filename:, keep_intermediary_files: false, verbose: false)
+      begin
+        mson_to_json_schema!(filename: filename, keep_intermediary_files: keep_intermediary_files, verbose: verbose)
+        puts "OK ".green + filename if verbose
+        true
+      rescue
+        puts "ERROR ".red + filename if verbose
+        false
+      end
+    end
+
+    def self.mson_to_json_schema!(filename:, keep_intermediary_files: false, verbose: true)
+
+      # For now, we'll use the containing directory's name as a scope
+      service_scope = File.dirname(filename).split(File::SEPARATOR).last.underscore
+
+      # Parse MSON to an apiary blueprint AST
+      # (see https://github.com/apiaryio/api-blueprint/wiki/API-Blueprint-Map)
+      ast_file = mson_to_ast_json(filename)
+
+      # Pluck out Data structures from it
+      data_structures = data_structures_from_blueprint_ast(ast_file)
+
+      # Generate json schema from each contained data structure
+      schema = {
+        "$schema"     => "http://json-schema.org/draft-04/schema#",
+        "title"       => service_scope,
+        "definitions" => {},
+        "type"        => "object",
+        "properties"  => {},
+      }
+
+      # The scope for the data structure is the name of the service
+      # publishing the object. So if we're parsing a 'consume' schema,
+      # the containing objects are alredy scoped (because a consume
+      # schema says 'i consume object X from service Y'.
+      basename = File.basename(filename)
+      if basename.end_with?("publish.mson")
+        data_structure_autoscope = service_scope
+      elsif basename.end_with?("consume.mson")
+        data_structure_autoscope = nil
+      else
+        raise Error, "Invalid filename #{basename}, can't tell if it's a publish or consume schema"
+      end
+
+      # The json schema we're constructing contains every known
+      # object type in the 'definitions'. So if we have definitions for
+      # the objects User, Post and Tag, the schema will look like this:
+      #
+      # {
+      #   "$schema": "..."
+      #
+      #   "definitions": {
+      #     "user": { "type": "object", "properties": { ... }}
+      #     "post": { "type": "object", "properties": { ... }}
+      #     "tag":  { "type": "object", "properties": { ... }}
+      #   }
+      #
+      #   "properties": {
+      #     "user": "#/definitions/user"
+      #     "post": "#/definitions/post"
+      #     "tag":  "#/definitions/tag"
+      #   }
+      #
+      # }
+      #
+      # So when testing an object of type `user` against this schema,
+      # we need to wrap it as:
+      #
+      # {
+      #   user: {
+      #     "your": "actual",
+      #     "data": "goes here"
+      #     }
+      # }
+      #
+      data_structures.each do |data|
+        id = data['name']['literal']
+        json= DataStructure.new(id, data, data_structure_autoscope).to_json
+        member = json.delete('title')
+        schema['definitions'][member] = json
+        schema['properties'][member] = {"$ref" => "#/definitions/#{member}"}
+      end
+
+      # Write it in a file
+      outfile = filename.gsub(/\.\w+$/, '.schema.json')
+      File.open(outfile, 'w'){ |f| f.puts JSON.pretty_generate(schema) }
+
+      # Clean up
+      FileUtils.rm_f(ast_file) unless keep_intermediary_files
+      true
+    end
+
+    def self.data_structures_from_blueprint_ast(filename)
+      c = JSON.parse(open(filename).read)['ast']['content'].first
+      return [] unless c
+      c['content']
+    end
+
+    def self.mson_to_ast_json(filename)
+      input = filename
+      output = filename.gsub(/\.\w+$/, '.blueprint-ast.json')
+
+
+      parse_result = FFI::MemoryPointer.new :pointer
+      RedSnow::Binding.drafter_c_parse(open(input).read, 0, parse_result)
+      parse_result = parse_result.get_pointer(0)
+
+      status = -1
+      result = ''
+
+      unless parse_result.null?
+        status = 0
+        result = parse_result.read_string
+      end
+
+      File.open(output, 'w'){ |f| f.puts(result)  }
+
+      output
+    ensure
+      RedSnow::Memory.free(parse_result)
+    end
+  end
+end