orch 0.0.2 → 0.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2505f295b3b9d24ec8cf03009803c9efa3afb0ef
-  data.tar.gz: 85462611db90a1799169b89ee601076d2bc0d7b3
+  metadata.gz: 4fc678adcfcacd47d4c3424fffae37cfba36f7f1
+  data.tar.gz: deba3fb34b70a3b486c400b71ff395b869fa0a9a
 SHA512:
-  metadata.gz: 496fb704230bc3d7410e46b15c4d7e67b3a604e0781461f38b505d862f0e5ac6694883deac23ee29c68510560de71049efd58c543fa1d94cb420a5b7b005fc8b
-  data.tar.gz: 6e5d5fcdbfbe50becbd7227dab42ce6b86932ed9b130127b923d7aff7ad9eb54d25092bd376820608e57f952d2515e851c49b5948b89e8e313f9a46e07a7ec3c
+  metadata.gz: bf766eea2fd37950ac0fa9fcbe2528e4b51748570f0ae104cc73796bc667403d994e64e310c964fd882bfb2adca49aaaaefe4e412e39df7bf2c7ddcf08ef6b95
+  data.tar.gz: 5f1e9ce4837fb1a4ca65cefc4fc12005ffb0d6f94d98a3f58c45b059b972725120310374df0cc02cc7bcce63f447e7e1cfe06e5938bc52257c6d55aa8bccca08

data/README.md

@@ -20,7 +20,133 @@ The general usecase is to run "orch deploy _job_spec_" to deploy a configuration
 
 ## Job spec format
 
-TODO: describe the file
+The orch Yaml format is really a wrapper for multiple Marathon or Chronos job descriptions.  The format is based on yaml.  To get a basic understanding of Yaml files check out: http://www.yaml.org/start.html
+
+The basic orch syntax is as follows:
+```
+version: alpha1
+applications:
+  - kind: Chronos
+    chronos_spec:
+      ...
+```
+
+The **version:** field must currently always be alpha1.  Eventually, this tool may need to support a revised format and this can be used to support multiple versions.
+
+The **applications:** field contains an array of Chronos or Marathon configurations.  Each array must have a **kind:** field with a value of _Chronos_ or _Marathon_ which simply tells orch what type of config this is.  Depending on the value of **kind:** a field of **chronos_spec:** or **marathon_spec:** is also required.
+
+The values of **chronos_spec:** should simply be the yaml equivelent of the chronos json messages.  You can find documentation for chronos json format here: https://mesos.github.io/chronos/docs/api.html
+
+
+Likewise, the values of **marathon_spec:** should simply be the yaml equivelent of the marathon json messages.  You can find documentation for marathon json format here: https://mesosphere.github.io/marathon/docs/rest-api.html
+
+So far all we have is a different way to specify a job.  The real power of orch will come from using meta data to act on these configs in more interesting ways...
+
+### Deployment vars
+
+The **deploy_vars:** field allows you to define some values that can be used to drive deployment of the same config for multiple uses.  Often it is useful to have *dev*, *test* & *prod* environments for our application.  Generally we want to use the same specification for all environments but we need a way to differentiate them and allow the app at run time to know what environment it is running in.
+
+Let's start with an example:
+```
+deploy_vars:
+  DEPLOY_ENV:
+    - dev
+    - test
+    - prod
+```
+This example will define an environment variable named **DEPLOY_ENV**.  (You can name it whatever you want.)  We are defining it here to have values of *dev*, *test*, and *prod*.
+
+You then would need to specify that variable in your application sections like this:
+```
+version: alpha1
+deploy_vars:
+  DEPLOY_ENV:
+    - dev
+    - test
+    - prod
+applications:
+  - kind: Chronos
+    DEPLOY_ENV: dev
+    chronos_spec:
+      name: "myapp-{{DEPLOY_ENV}}"
+      ...
+  - kind: Chronos
+    DEPLOY_ENV: prod
+    chronos_spec:
+      name: "myapp-{{DEPLOY_ENV}}"
+      ...
+```
+
+When orch runs it will insert the environment variable into the *env* or *environmentVariables* sections of your spec.  However, you can also use the **--deploy-env** command line option to tell orch to only deploy a paticular environment.  You can also use the substitution feature to use the value of your environment varaible in other parts of the config like name.  *(e.g. `name: "myapp-{{DEPLOY_ENV}}"` would substitute to `name: "myapp-dev"` for the dev environment)*
+
+### Alternative way to set environment variables
+
+Chronos and Marathon use different syntx to specify environment variables.  Also, you may want to specify certain variables consistently across several applications in your orch spec.  The **env:** field provides a way to do that with a nice clean syntax.
+
+
+The env field could be used at the top level of the spec to be used across all applications in the spec:
+```
+version: alpha1
+env:
+  GOOGLE_URL: http://www.google.com
+  YP_URL: http://www.yp.com
+  applications:
+    ...
+```
+
+Or you can also specify it at the application level as an alternative syntax to specifying it in the *marathon_spec:* or *chronos_spec:* sections - like this:
+```
+version: alpha1
+env:
+  GOOGLE_URL: http://www.google.com
+  YP_URL: http://www.yp.com
+  applications:
+    - kind: Marathon
+      env:
+        GOOGLE_URL: http://www.google.com
+        YP_URL: http://www.yp.com        
+      marathon_spec:
+        ...
+```
+
+Lower level definitions of an env var will supeceed higher level version.  That is if you specify an env value in the *marathon_spec* or *chronos_spec* level it will override anything set at the *app* or *orch* level.
+### Yaml anchors and aliases
+
+The nice thing about using Yaml as the spec format is the ability to use Yaml's anchors (&) and alias (*) syntax.  This allows you to specify sections of your configuration that can be reused across multiple jobs.  Typically if you have a job for multiple deployment environments you what them all to have the same spec - except perhaps override one or two things.  (Like an envionment value or the docker image.)
+
+We will refer you to other documentation on the internet on how to use Yaml anchors and aliases.  However, you can take a look at a couple of the [provided examples](examples/Examples.md) to see how you might use Yaml to its fullest.
+
+### Substition
+
+A primary use case of **orch** is to use the tool from within a Makefile.  You may want to subsitite parts of you configuration from vairables in your Makefile.  In the *Deployment vars* section we showed you how you can use deployment vars as substitution values in other parts of the configuration.
+
+**Orch** also provides the *--subst* option to provide a way to pass additional substitution vars when running your configuration.  A common use case might be to pass in the tag for a Docker image that is managed by your Makefile.  Here is an example:
+```
+todo: provide an example
+```
+
+TODO: finish this section
+
+### Bamboo
+
+[Bamboo](https://github.com/QubitProducts/bamboo) is a tool that creates an HAProxy for providing an easier way to get to a farm of web servers hosted by Marathon.  You define the Marathon id of the application you want to load balance and an acl to tell HAProxy what url to direct to your marathon application.
+
+Orch provides a way to also configure your Bamboo server from your orch config.  Here is an example:
+```
+version: alpha1
+applications:
+  - kind: Marathon
+    marathon_spec:
+      ...
+    bamboo_spec:
+      acl: "hdr(host) -i test-web.int.yp.com"
+```
+
+An additional *bamboo_spec:* field is added to the Marathon application at the same level as the *marathon_spec:* field.  The *bamboo_spec:* field contains one field called *acl:*.  The simply has a acl rule for HAProxy.  See Bamboo documentation for more details.
+
+Note: you will also need to set the **bamboo_url:** feild of your configutation.
+
+### TODO: section on vault
 
 ## Configuration Options
 

data/examples/Examples.md

@@ -5,6 +5,8 @@ This directory contains several examples to show various orch features.
 
 | File                  | Description
 |-----------------------|------------------------------------------------------------------------|
-| shimple-chronos.yml   | A very simple chronos spec for a daily recurring job
-| env-chronos.yml       | Shows how to set up a chronos job for multiple deployment environments.
+| shimple-chronos.yml   | A very simple chronos spec for a daily recurring job.  It also has a simple example of a dependant job.
+| env-chronos.yml       | Shows how to set up a chronos job for multiple deployment environments.  It also gives a good example of how to use Yaml anchors, the use of multiple deploy_vars: fields, and how to use deploy_vars fields in substitutions.
 | simple-marathon.yml   | A simple example of a Marathon job using orch
+| env-marathon.yml      | A more complex marathon example showing multiple deployment environments.
+| bamboo-ex.yml         | Example of using Marathon with Bamboo.  

data/examples/bamboo-ex.yml

@@ -0,0 +1,47 @@
+default_job: &DEFAULT_JOB
+  kind: Marathon
+  DEPLOY_ENV: dev
+  bamboo_spec:
+    acl: "hdr(host) -i test-web-{{DEPLOY_ENV}}.ypec.int.yp.com"
+  marathon_spec: &DEFAULT_SPEC
+    id: "test-bamboo-{{DEPLOY_ENV}}"
+    cpus: 0.1
+    mem: 300
+    instances: 3
+    container: &DEFAULT_CONTAINER
+      type: "DOCKER"
+      docker: &DEFAULT_DOCKER
+        image: "nginx"
+        network: "BRIDGE"
+        portMappings: 
+          - 
+            containerPort: 80
+            hostPort: 0
+            protocol: "tcp"
+        forcePullImage: false
+
+version: alpha1
+deploy_vars:
+  DEPLOY_ENV:
+    - dev
+    - test
+    - prod
+env:
+  DEBUG_FLAG: true
+applications:
+  - <<: *DEFAULT_JOB
+    DEPLOY_ENV: dev
+    marathon_spec:
+      <<: *DEFAULT_SPEC
+      container:
+        <<: *DEFAULT_CONTAINER
+        docker:
+          <<: *DEFAULT_DOCKER
+          forcePullImage: true
+  - <<: *DEFAULT_JOB
+    DEPLOY_ENV: test
+  - <<: *DEFAULT_JOB
+    DEPLOY_ENV: prod
+    env:
+      DEBUG_FLAG: false
+

data/examples/env-chronos.yml

@@ -1,9 +1,8 @@
 
 default_job: &DEFAULT_JOB
   kind: Chronos
-  environment: override me
-  cronos_spec: &DEFAULT_SPEC
-    name: "ray-test-{{DEPLOY_ENV}}"
+  chronos_spec: &DEFAULT_SPEC
+    name: "ray-test-{{DEPLOY_ENV}}-{{COLO}}"
     schedule: "R/2015-01-25T21:00/PT24H"
     scheduleTimeZone: "PST"
     owner: "rjohnson@yp.com"
@@ -18,11 +17,10 @@ default_job: &DEFAULT_JOB
       - 
         name: "LOG_LOCATION"
         value: "/var/{{DEPLOY_ENV}}"
-    forcePullImage: false
     command: "echo \"log here: $LOG_LOCATION\"; echo \"ENV: $DEPLOY_ENV\";sleep 10; echo bye"
 
 version: alpha1
-environment_vars:
+deploy_vars:
   DEPLOY_ENV:
     - dev
     - test
@@ -32,31 +30,26 @@ environment_vars:
     - east
 applications:
   - <<: *DEFAULT_JOB
-    environment: dev
     DEPLOY_ENV: dev
     COLO: east
-    cronos_spec:
+    chronos_spec:
       <<: *DEFAULT_SPEC
       schedule: "R/2015-08-01T21:00/PT24H"
-      forcePullImage: true
   - <<: *DEFAULT_JOB
-    environment: test
     DEPLOY_ENV: test
     COLO: east
-    cronos_spec:
+    chronos_spec:
       <<: *DEFAULT_SPEC
       schedule: "R/2015-08-01T21:10/PT24H"
   - <<: *DEFAULT_JOB
-    environment: prod
     DEPLOY_ENV: prod
     COLO: east
-    cronos_spec:
+    chronos_spec:
       <<: *DEFAULT_SPEC
       schedule: "R/2015-08-01T21:30/PT24H"
   - <<: *DEFAULT_JOB
-    environment: prod
     DEPLOY_ENV: prod
     COLO: west
-    cronos_spec:
+    chronos_spec:
       <<: *DEFAULT_SPEC
       schedule: "R/2015-08-01T21:30/PT24H"

data/examples/env-marathon.yml

@@ -0,0 +1,44 @@
+default_job: &DEFAULT_JOB
+  kind: Marathon
+  DEPLOY_ENV: dev
+  marathon_spec: &DEFAULT_SPEC
+    id: "test-web-{{DEPLOY_ENV}}"
+    cpus: 0.1
+    mem: 300
+    instances: 3
+    container: &DEFAULT_CONTAINER
+      type: "DOCKER"
+      docker: &DEFAULT_DOCKER
+        image: "nginx"
+        network: "BRIDGE"
+        portMappings: 
+          - 
+            containerPort: 80
+            hostPort: 0
+            protocol: "tcp"
+        forcePullImage: false
+
+version: alpha1
+deploy_vars:
+  DEPLOY_ENV:
+    - dev
+    - test
+    - prod
+env:
+  DEBUG_FLAG: true
+applications:
+  - <<: *DEFAULT_JOB
+    DEPLOY_ENV: dev
+    marathon_spec:
+      <<: *DEFAULT_SPEC
+      container:
+        <<: *DEFAULT_CONTAINER
+        docker:
+          <<: *DEFAULT_DOCKER
+          forcePullImage: true
+  - <<: *DEFAULT_JOB
+    DEPLOY_ENV: test
+  - <<: *DEFAULT_JOB
+    DEPLOY_ENV: prod
+    env:
+      DEBUG_FLAG: false

data/examples/simple-chronos.yml

@@ -1,9 +1,9 @@
 version: alpha1
 applications:
   - kind: Chronos
-    cronos_spec:
-      name: "ray-test"
-      schedule: "R/2015-01-25T21:00/PT24H"
+    chronos_spec:
+      name: "orch-test-schedule"
+      schedule: "R/2015-01-25T21:00/PT03H"
       scheduleTimeZone: "PST"
       owner: "rjohnson@yp.com"
       container: 
@@ -15,3 +15,19 @@ applications:
       uris: []
       forcePullImage: true
       command: "echo hello; sleep 10; echo bye"
+  - kind: Chronos
+    chronos_spec:
+      name: "orch-test-dependant-schedule"
+      parents:
+        - orch-test-schedule
+      scheduleTimeZone: "PST"
+      owner: "rjohnson@yp.com"
+      container: 
+        type: "DOCKER"
+        image: "busybox"
+        network: "BRIDGE"
+      cpus: "0.5"
+      mem: "512"
+      uris: []
+      forcePullImage: true
+      command: "echo run-dependant-job; sleep 10; echo bye"

data/examples/simple-marathon.yml

@@ -1,17 +1,12 @@
 version: alpha1
-environment_vars:
-  DEPLOY_ENV:
-    - dev
-    - test
-    - prod
 applications:
   - kind: Marathon
     DEPLOY_ENV: dev
     marathon_spec:
-      id: "ray-web-{{DEPLOY_ENV}}"
+      id: "test-web-app"
       cpus: 0.1
       mem: 300
-      instances: 10
+      instances: 2
       env: 
         FOO: "bar"
       container: 
@@ -24,5 +19,4 @@ applications:
               containerPort: 80
               hostPort: 0
               protocol: "tcp"
-          forcePullImage: true
 

data/lib/bamboo.rb

@@ -0,0 +1,112 @@
+# Chronos interface object
+require 'json'
+require 'net/http'
+require 'uri'
+
+require 'config'
+
+class String
+  def numeric?
+    Float(self) != nil rescue false
+  end
+end
+
+module Orch
+  class Bamboo
+    def initialize(options)
+      @config = Orch::Config.new(options)
+    end
+
+    def deploy(app_id, bamboo_spec)
+      uri = URI(@config.bamboo_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      # Create the real json 
+      bamboo_json = {}
+      bamboo_json["id"] = (app_id[0] == '/') ? app_id : ("/" + app_id)
+      bamboo_json["acl"] = bamboo_spec["acl"]
+
+  # curl -i -X PUT -d '{"id":"/ExampleAppGroup/app1", "acl":"path_beg -i /group/app-1"}' http://localhost:8000/api/services//ExampleAppGroup/app1
+# {"/adam-web-dev":{"Id":"/adam-web-dev","Acl":"hdr(host) -i adam-web-dev.ypec.int.yp.com"
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.put("/api/services/#{app_id}", bamboo_json.to_json, json_headers)
+
+      if response.code == 200.to_s
+        puts "successfully created bamboo spec for marathon job: #{app_id}"
+      else
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+      end
+
+      return response
+    end
+
+    def delete(app_id)
+      uri = URI(@config.chronos_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      puts "curl -i -X DELETE #{uri}/api/services/#{app_id}"
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.delete("/api/services/#{app_id}", json_headers)
+
+      if response.code != 200.to_s
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+      end
+
+      # TODO: handle error codes better?
+
+      return response
+    end
+
+    def verify(app_id, spec)
+      if @config.check_for_bamboo_url == false
+        puts "no bamboo_url - can not verify with server"
+        return
+      end
+
+      uri = URI(@config.bamboo_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      # TODO: will this work or do I need to parse through all services like chronos
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.get("/api/services", json_headers)
+
+      if response.code != 200.to_s
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+        foundDiffs = true 
+      end
+
+      allJobs = Hashie::Mash.new(JSON.parse(response.body))
+
+      # Bamboo api only returns all items
+      jobFound = false
+      allJobs.each do |key, job|
+        if key == app_id
+          jobFound = true
+          # Bamboo returns keys with different case then you send them!
+          # Right now the only field to compar is Acl so we only check it by hand
+          if spec["acl"] != job["Acl"]
+            printf "difference for field: acl\n"
+            printf "    spec:   #{spec["acl"]}\n"
+            printf "    server: #{job["Acl"]}\n"
+            foundDiffs = true 
+          end
+    
+
+        end
+      end
+      
+      if !jobFound
+        puts "bamboo spec for marathon job \"#{app_id}\" not currently deployed"
+        foundDiffs = true 
+      end
+
+      # TODO: handle error codes better?
+
+      return foundDiffs
+    end
+
+  end
+end

data/lib/{deploy.rb → chronos.rb}

@@ -1,3 +1,4 @@
+# Chronos interface object
 require 'json'
 require 'net/http'
 require 'uri'
@@ -11,20 +12,27 @@ class String
 end
 
 module Orch
-  class Deploy
+  class Chronos
     def initialize(options)
       # TODO: get chronos and marathon urls from diffferent ways: param, env, .config
       @config = Orch::Config.new(options)
     end
 
-    def deploy_chronos(json_payload)
+    def deploy(json_payload)
       uri = URI(@config.chronos_url)
       json_headers = {"Content-Type" => "application/json",
                 "Accept" => "application/json"}
 
-      # curl -L -H 'Content-Type: application/json' -X POST -d @$schedule_file $CHRONOS_URL/scheduler/iso8601
+      path = nil
+      path = "/scheduler/iso8601" unless json_payload["schedule"].nil?
+      path = "/scheduler/dependency" unless json_payload["parents"].nil?
+      if path.nil?
+        puts "neither schedule nor parents fields defined for Chronos job"
+        exit 1
+      end
+
       http = Net::HTTP.new(uri.host, uri.port)
-      response = http.post("/scheduler/iso8601", json_payload, json_headers)
+      response = http.post(path, json_payload, json_headers)
 
       if response.code != 204.to_s
         puts "Response #{response.code} #{response.message}: #{response.body}"
@@ -35,7 +43,25 @@ module Orch
       return response
     end
 
-    def verify_chronos(json_payload)
+    def delete(name)
+      uri = URI(@config.chronos_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      # curl -L -X DELETE chronos-node:8080/scheduler/job/request_event_counter_hourly
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.delete("/scheduler/job/#{name}", json_headers)
+
+      if response.code != 204.to_s
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+      end
+
+      # TODO: handle error codes better?
+
+      return response
+    end
+
+    def verify(json_payload)
       if @config.check_for_chronos_url == false
         puts "no chronos_url - can not verify with server"
         return
@@ -47,16 +73,17 @@ module Orch
       json_headers = {"Content-Type" => "application/json",
                 "Accept" => "application/json"}
 
-      # curl -L -H 'Content-Type: application/json' -X POST -d @$schedule_file $CHRONOS_URL/scheduler/iso8601
       http = Net::HTTP.new(uri.host, uri.port)
-      response = http.get("/scheduler/jobs", json_headers)
+      response = http.get("/scheduler/jobs/search?name=#{spec.name}", json_headers)
 
       if response.code != 200.to_s
         puts "Response #{response.code} #{response.message}: #{response.body}"
+        foundDiffs = true 
       end
 
       array = JSON.parse(response.body).map { |hash| Hashie::Mash.new(hash) }
 
+      # Chronos search API could return more than one item - make sure we find the exact match
       jobFound = false
       array.each do |job|
         if job.name == spec.name
@@ -67,11 +94,12 @@ module Orch
       
       if !jobFound
         puts "job \"#{spec.name}\" not currently deployed"
+        foundDiffs = true 
       end
 
       # TODO: handle error codes better?
 
-      return response
+      return foundDiffs
     end
 
     def find_diffs(spec, job)
@@ -84,7 +112,16 @@ module Orch
           end
           next
         end
-        # TODO: not sure how to compare arrays
+        if spec[key].is_a?(Array)
+          if spec[key].length != job[key].length
+            printf "difference for field: #{key} - length of array is different\n"
+            printf "    spec:   #{spec[key].to_json}\n"
+            printf "    server: #{job[key].to_json}\n"
+            foundDiff = true
+            next
+          end
+          # TODO: not sure how to compare arrays
+        end
         specVal = spec[key]
         jobVal = job[key]
         if spec[key].to_s.numeric?
@@ -94,66 +131,23 @@ module Orch
           specVal = spec[key]
           jobVal = job[key]
         end
+        # Chronos changes the case of the Docker argument for some reason
+        if key == "type"
+          specVal = specVal.upcase
+          jobVal = jobVal.upcase
+        end
         if specVal != jobVal
-          puts "#{key}= spec:#{specVal}, server:#{jobVal}"
+          if foundDiff == false
+            puts "Differences found in job"
+          end
+          printf "difference for field: #{key}\n"
+          printf "    spec:   #{specVal}\n"
+          printf "    server: #{jobVal}\n"
           foundDiff = true
         end
       end
 
       return foundDiff
     end
-
-    def deploy_marathon(json_payload)
-      uri = URI(@config.marathon_url)
-      json_headers = {"Content-Type" => "application/json",
-                "Accept" => "application/json"}
-      puts uri
-
-      http = Net::HTTP.new(uri.host, uri.port)
-      response = http.put("/v2/apps/#{json_payload["id"]}", json_payload, json_headers)
-
-      puts "Response #{response.code} #{response.message}: #{response.body}"
-
-      if response.code == 204.to_s
-        puts "success"
-      end
-
-      # TODO: handle error codes
-
-      return response
-    end
-
-    def verify_marathon(json_payload)
-      if @config.check_for_marathon_url == false
-        puts "no marathon_url - can not verify with server"
-        return
-      end
-
-      spec = Hashie::Mash.new(JSON.parse(json_payload))
-
-      uri = URI(@config.marathon_url)
-      json_headers = {"Content-Type" => "application/json",
-                "Accept" => "application/json"}
-
-      http = Net::HTTP.new(uri.host, uri.port)
-      response = http.get("/v2/apps/#{spec.id}", json_headers)
-
-      # puts "Response #{response.code} #{response.message}: #{response.body}"
-
-      if response.code == 404.to_s
-        puts "job: #{spec.id} - is not deployed"
-        return
-      end
-
-      if response.code == 200.to_s
-        job = Hashie::Mash.new(JSON.parse(response.body))
-        foundDiffs = find_diffs(spec, job.app)
-      end
-
-      # TODO: handle error codes
-
-      return response
-    end
-
   end
-end
+end

data/lib/config.rb

@@ -32,6 +32,14 @@ module Orch
       end
     end
 
+    def check_for_bamboo_url
+      if (!  @options.has_key?("bamboo_url")) && (@APP_CONFIG.nil? || @APP_CONFIG["bamboo_url"].nil?)
+        return false
+      else
+        return true
+      end
+    end
+
     def chronos_url
       if @options.has_key?("chronos_url")
         # If passed in on command line override what is in config file
@@ -62,13 +70,32 @@ module Orch
       return @APP_CONFIG["marathon_url"]
     end
 
-    def setup_config(marathon_url, chronos_url)
+    def bamboo_url
+      if @options.has_key?("bamboo_url")
+        # If passed in on command line override what is in config file
+        url = @options["bamboo_url"]
+        return url
+      end
+
+      if @APP_CONFIG.nil? || @APP_CONFIG["bamboo_url"].nil?
+        puts "bamboo_url not specified, use --bamboo_url or set in ~/.orch/config.yml"
+        exit 1
+      end
+
+      return @APP_CONFIG["bamboo_url"]
+    end
+
+    def setup_config(settings)
 
       if @APP_CONFIG.nil?
         @APP_CONFIG = {}
       end
-      @APP_CONFIG['marathon_url'] = marathon_url
-      @APP_CONFIG['chronos_url'] = chronos_url
+
+      settings.each do |key, value|
+        if settings[key] != ""
+          @APP_CONFIG[key] = value
+        end
+      end
 
       if ! File.directory?("#{Dir.home}/.orch")
         Dir.mkdir("#{Dir.home}/.orch")

data/lib/marathon.rb

@@ -0,0 +1,150 @@
+# Marathon interface object
+require 'json'
+require 'net/http'
+require 'uri'
+
+require 'config'
+
+class String
+  def numeric?
+    Float(self) != nil rescue false
+  end
+end
+
+module Orch
+  class Marathon
+    def initialize(options)
+      # TODO: get chronos and marathon urls from diffferent ways: param, env, .config
+      @config = Orch::Config.new(options)
+    end
+
+    def deploy(app_id, json_payload)
+      uri = URI(@config.marathon_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.put("/v2/apps/#{app_id}", json_payload, json_headers)
+
+      # TODO: should we do anyting with version or deploymentId that gets returned?
+      if response.code == 201.to_s
+        puts "successfully created marathon job: #{app_id}"
+      elsif response.code == 200.to_s
+        puts "successfully updated marathon job: #{app_id}"
+      else
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+      end
+
+      return response
+    end
+
+    def delete(id)
+      uri = URI(@config.marathon_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.delete("/v2/apps/#{id}", json_headers)
+
+      if response.code == 200.to_s
+        puts "successfully deleted #{id}"
+      elsif response.code == 404.to_s
+        puts "job: #{id} - does not exist to delete"
+      else
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+      end
+
+      return response
+    end
+
+    def verify(json_payload)
+      if @config.check_for_marathon_url == false
+        puts "no marathon_url - can not verify with server"
+        return
+      end
+
+      spec = Hashie::Mash.new(JSON.parse(json_payload))
+
+      uri = URI(@config.marathon_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.get("/v2/apps/#{spec.id}", json_headers)
+
+      if response.code == 200.to_s
+        job = Hashie::Mash.new(JSON.parse(response.body))
+        foundDiffs = find_diffs(spec, job.app)
+      elsif response.code == 404.to_s
+        puts "job: #{spec.id} - not defined in Marathon"
+        foundDiffs = true 
+      else
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+        foundDiffs = true 
+      end
+
+      return foundDiffs
+    end
+
+    def restart(app_id)
+      # POST /v2/apps/{appId}/restart: Rolling restart of all tasks of the given app
+      uri = URI(@config.marathon_url)
+      json_headers = {"Content-Type" => "application/json",
+                "Accept" => "application/json"}
+
+      http = Net::HTTP.new(uri.host, uri.port)
+      response = http.post("/v2/apps/#{app_id}/restart", {}.to_json, json_headers)
+
+      if response.code == 200.to_s
+        puts "success"
+      else
+        puts "Response #{response.code} #{response.message}: #{response.body}"
+      end
+
+      return response
+    end
+
+    def find_diffs(spec, job)
+      foundDiff = false
+
+      spec.each_key do |key|
+        if spec[key].is_a?(Hash)
+          if find_diffs(spec[key], job[key]) == true
+            foundDiff = true
+          end
+          next
+        end
+        if spec[key].is_a?(Array)
+          if spec[key].length != job[key].length
+            printf "difference for field: #{key} - length of array is different\n"
+            printf "    spec:   #{spec[key].to_json}\n"
+            printf "    server: #{job[key].to_json}\n"
+            foundDiff = true
+          end
+          # TODO: this will not work if arrays are not in same order
+          spec[key].zip(job[key]).each do |subSpec, subJob|
+            if find_diffs(subSpec, subJob) == true
+              foundDiff = true
+            end
+            next
+          end
+          next
+        end
+        specVal = spec[key]
+        jobVal = job[key]
+        if spec[key].to_s.numeric?
+          specVal = Float(spec[key])
+          jobVal = Float(job[key])
+        end
+        if specVal != jobVal
+          printf "difference for field: #{key}\n"
+          printf "    spec:   #{specVal}\n"
+          printf "    server: #{jobVal}\n"
+          foundDiff = true
+        end
+      end
+
+      return foundDiff
+    end  
+  end
+end

data/lib/orch/version.rb

@@ -1,3 +1,3 @@
 module Orch
-  VERSION = "0.0.2"
+  VERSION = "0.0.3"
 end

data/lib/orch.rb

@@ -4,7 +4,9 @@ require 'json'
 require 'thor'
 require "yaml"
 require "parse"
-require "deploy"
+require "chronos"
+require "marathon"
+require "bamboo"
 
 module Orch
   class Application < Thor
@@ -21,16 +23,18 @@ module Orch
         end
       end
       say("Enter values to construct a ~/.orch/config.yaml file")
-      marathon_url = ask("Marathon URL: ")
-      chronos_url = ask("Chronos URL: ")
+      settings = {}
+      settings["marathon_url"] = ask("Marathon URL: ")
+      settings["chronos_url"] = ask("Chronos URL: ")
+      settings["bamboo_url"] = ask("Bamboo URL: ")
 
-      config.setup_config(marathon_url, chronos_url)
+      config.setup_config(settings)
     end
 
     option :deploy_kind, :default => 'all',
            :desc => 'deploys only the given application kind: chronos, marathon, all'
-    option :deploy_env, :default => 'all',
-           :desc => 'ENV_VAR=VALUE deploys only if an environment_var matches the given value'
+    option :deploy_var, :default => 'all',
+           :desc => 'DEPLOY_VAR=VALUE deploys only if a deploy_var matches the given value'
     option :subst,
            :desc => 'KEY=VALUE substitute KEY with VALUE globaly in your config'
     option :show_json, :default => false,
@@ -42,7 +46,10 @@ module Orch
       parser = Orch::Parse.new(file_name, options)
       result = parser.parse(true)
       puts "Number of configs found: #{result.length}"
-      deploy = Orch::Deploy.new(options)
+
+      marathon = Orch::Marathon.new(options)
+      chronos = Orch::Chronos.new(options)
+      bamboo = Orch::Bamboo.new(options)
       result.each do |app|
         printf "Name: %s, Type: %s, Deploy?: %s", app[:name], app[:type], app[:deploy]
         app[:env_vars].each do |key, value|
@@ -53,23 +60,33 @@ module Orch
           pretty = JSON.pretty_generate(app[:json])
           puts "JSON: #{pretty}"
         end
+        foundDiffs = false
         if (app[:type] == "Chronos") && (options[:server_verify] == true)
-          deploy.verify_chronos(app[:json].to_json)
+          foundDiffs = chronos.verify(app[:json].to_json)
         end
         if (app[:type] == "Marathon") && (options[:server_verify] == true)
-          deploy.verify_marathon(app[:json].to_json)
+          foundDiffs = marathon.verify(app[:json].to_json)
+          if app[:bamboo_spec]
+            bambooDiffs = bamboo.verify(app[:name], app[:bamboo_spec])
+            foundDiffs = foundDiffs || bambooDiffs
+          end
+        end
+        if (!foundDiffs) && (options[:server_verify] == true)
+          puts "No differences with server found"
         end
       end
     end
 
     option :deploy_kind, :default => 'all',
            :desc => 'deploys only the given application kind: chronos, marathon, all'
-    option :deploy_env, :default => 'all',
-           :desc => 'ENV_VAR=VALUE deploys only if an environment_var matches the given value'
+    option :deploy_var, :default => 'all',
+           :desc => 'DEPLOY_VAR=VALUE deploys only if a deploy_var matches the given value'
     option :chronos_url,
            :desc => 'url to chronos master'
     option :marathon_url,
            :desc => 'url to marathon master'
+    option :bamboo_url,
+           :desc => 'url to bamboo server'
     option :subst,
            :desc => 'KEY=VALUE substitute KEY with VALUE globaly in your config'
     desc 'deploy PATH', 'Deploys config to mesos frameworks.'
@@ -81,7 +98,9 @@ module Orch
         puts "nothing found to deploy"
       end
 
-      deploy = Orch::Deploy.new(options)
+      marathon = Orch::Marathon.new(options)
+      chronos = Orch::Chronos.new(options)
+      bamboo = Orch::Bamboo.new(options)
       result.each do |app|
         if !app[:deploy]
           puts "skipping app: #{app[:name]}"
@@ -90,13 +109,88 @@ module Orch
         puts "deploying #{app[:name]} to #{app[:type]}"
         #puts "#{app[:json]}"  - should I support show_json here as well?
         if app[:type] == "Chronos"
-          deploy.deploy_chronos(app[:json].to_json)
+          chronos.deploy(app[:json].to_json)
         end
         if app[:type] == "Marathon"
-          deploy.deploy_marathon(app[:json].to_json)
+          marathon.deploy(app[:name], app[:json].to_json)
+          if app[:bamboo_spec]
+            bamboo.deploy(app[:name], app[:bamboo_spec])
+          end
         end
       end
     end
 
+    option :deploy_kind, :default => 'all',
+           :desc => 'deletes only the given application kind: chronos, marathon, all'
+    option :deploy_var, :default => 'all',
+           :desc => 'DEPLOY_VAR=VALUE deletes only if a deploy_var matches the given value'
+    option :chronos_url,
+           :desc => 'url to chronos master'
+    option :marathon_url,
+           :desc => 'url to marathon master'
+    option :bamboo_url,
+           :desc => 'url to bamboo server'
+    option :subst,
+           :desc => 'KEY=VALUE substitute KEY with VALUE globaly in your config'
+    desc 'delete PATH', 'Deletes config from mesos frameworks.'
+    def delete(file_name)
+      parser = Orch::Parse.new(file_name, options)
+      result = parser.parse(false)
+
+      if result.length == 0
+        puts "nothing found to delete"
+      end
+
+      marathon = Orch::Marathon.new(options)
+      chronos = Orch::Chronos.new(options)
+      bamboo = Orch::Bamboo.new(options)
+
+      result.each do |app|
+        if !app[:deploy]
+          puts "skipping app: #{app[:name]}"
+          next
+        end
+        puts "deleting #{app[:name]} from #{app[:type]}"
+        if app[:type] == "Chronos"
+          chronos.delete(app[:name])
+        end
+        if app[:type] == "Marathon"
+          marathon.delete(app[:name])
+          if app[:bamboo_spec]
+            bamboo.delete(app[:name])
+          end
+        end
+      end
+    end
+
+    option :deploy_kind, :default => 'all',
+           :desc => 'deletes only the given application kind: chronos, marathon, all'
+    option :deploy_var, :default => 'all',
+           :desc => 'DEPLOY_VAR=VALUE deletes only if a deploy_var matches the given value'
+    option :marathon_url,
+           :desc => 'url to marathon master'
+    option :subst,
+           :desc => 'KEY=VALUE substitute KEY with VALUE globaly in your config'
+    desc 'restart PATH', 'Restarts specified application(s) on server'
+    def restart(file_name)
+      parser = Orch::Parse.new(file_name, options)
+      result = parser.parse(false)
+
+      if result.length == 0
+        puts "nothing found to restart"
+      end
+
+      marathon = Orch::Marathon.new(options)
+      result.each do |app|
+        if !app[:deploy] || (app[:type] == "Chronos")
+          puts "skipping app: #{app[:name]}"
+          next
+        end
+        puts "restarting #{app[:name]} on #{app[:type]}"
+        if app[:type] == "Marathon"
+          marathon.restart(app[:name])
+        end
+      end
+    end
   end
 end

data/lib/parse.rb

@@ -9,14 +9,13 @@ module Orch
         puts "file does not exist: #{path}"
         exit 1
       end
-      # TODO: try resecue for any syntax issues in yaml file
+
       begin
         yaml = ::YAML.load_file(path)
       rescue Psych::SyntaxError => e
         puts "error parsing yaml file #{e}"
         exit 1
       end
-      #puts yaml.to_json
 
       @options = options
 
@@ -42,6 +41,11 @@ module Orch
       # Check for vault vars
       env_var_values = parse_vault_vars(@spec)
 
+      # Check for env values at spec level
+      if ! spec.env.nil?
+        env_var_values = env_var_values.merge(spec.env)
+      end
+
       if spec.applications.nil?
         puts "required section applications: must have at least one application defined"
         exit 1
@@ -60,39 +64,51 @@ module Orch
           exit 1
         end
 
-        # Generate any environment variables that need to be merged in
-        spec_env_vars = parse_env_vars(app)
-        env_var_values = env_var_values.merge(spec_env_vars)
+        # Generate any deploy variables that need to be merged in
+        deploy_vars = parse_deploy_vars(app)
+        app_var_values = env_var_values.merge(deploy_vars)
+
+        # Check for env values at app level
+        if ! app.env.nil?
+          app_var_values = app_var_values.merge(app.env)
+        end
 
         if (app.kind == "Chronos")
-          chronos_spec = parse_chronos(app, env_var_values)
+          chronos_spec = parse_chronos(app, app_var_values)
 
-          result = {:name => chronos_spec["name"], :type => app.kind, :deploy => should_deploy?(app), :env_vars => env_var_values, :json => chronos_spec}
+          result = {:name => chronos_spec["name"], :type => app.kind, :deploy => should_deploy?(app), :env_vars => deploy_vars, :json => chronos_spec}
           results << result
         end
 
         if (app.kind == "Marathon")
-          marathon_spec = parse_marathon(app, env_var_values)
+          marathon_spec = parse_marathon(app, app_var_values)
+
+          result = {:name => marathon_spec["id"], :type => app.kind, :deploy => should_deploy?(app), :env_vars => deploy_vars, :json => marathon_spec}
+
+          if app.bamboo_spec
+            bamboo_spec = parse_bamboo(app, app_var_values)
+            result[:bamboo_spec] = bamboo_spec
+          end
 
-          result = {:name => marathon_spec["id"], :type => app.kind, :deploy => should_deploy?(app), :env_vars => env_var_values, :json => marathon_spec}
           results << result
         end
+
       end
 
       return results
     end
 
-    def parse_env_vars(app)
+    def parse_deploy_vars(app)
       result = {}
-      if (! @spec.environment_vars.nil?)
-        @spec.environment_vars.each do |key, value|
+      if (! @spec.deploy_vars.nil?)
+        @spec.deploy_vars.each do |key, value|
           if app[key].nil?
             puts "environments_var #{key} specified - but not included in app"
             # TODO: would be nice to put the app name...
             exit 1
           end
-          if ! @spec.environment_vars[key].include? app[key]
-            puts "#{key} value \"#{app[key]}\" not in #{@spec.environment_vars[key].to_s}"
+          if ! @spec.deploy_vars[key].include? app[key]
+            puts "#{key} value \"#{app[key]}\" not in #{@spec.deploy_vars[key].to_s}"
             exit 1
           end
           result[key] = app[key]
@@ -115,29 +131,50 @@ module Orch
     end
 
     def parse_chronos(app, env_var_values)
-      # TODO: check if it exists
-      cronos_spec = app.cronos_spec
+      if app.chronos_spec.nil?
+        puts "App of kind: Chronos requires a 'chronos_spec:' field"
+        exit 1
+      end
+      chronos_spec = app.chronos_spec
 
-      # Augment any spec environment variables with meta values
-      env_var_values.each do |key, value|
+      # Override out high-level env vars with any chronos_spec level vars
+      env_vars = env_var_values
+      (chronos_spec.environmentVariables || []).each do |x|
+        env_vars[x["name"]] = x["value"]
+      end
+
+      # Rewrite the environmentVariables from the hash
+      chronos_spec.environmentVariables = []
+      env_vars.each do |key, value|
         pair = {"name" => key, "value" => value}
-        cronos_spec.environmentVariables << pair        
+        chronos_spec.environmentVariables << pair        
       end
 
       # Do subst processing
-      spec_str = do_subst(cronos_spec, app)
-      cronos_spec = JSON.parse(spec_str)
+      spec_str = do_subst(chronos_spec, app)
+      chronos_spec = JSON.parse(spec_str)
 
-      return cronos_spec
+      return chronos_spec
     end
 
     def parse_marathon(app, env_var_values)
-      # TODO: check if it exists
+      if app.marathon_spec.nil?
+        puts "App of kind: Marathon requires a 'marathon_spec:' field"
+        exit 1
+      end
       marathon_spec = app.marathon_spec
 
-      # Augment any spec environment variables with meta values
+      # Augment any spec environment variables with meta values - but don't overwite
+      marathon_spec.env = {} unless marathon_spec.env
       env_var_values.each do |key, value|
-        marathon_spec.env[key] = value     
+        marathon_spec.env[key] = value.to_s unless marathon_spec.env[key] 
+      end
+
+      if marathon_spec.id
+        marathon_spec.id = (marathon_spec.id[0] == '/') ? marathon_spec.id : ("/" + marathon_spec.id)
+      else
+        puts "id: is a required field for a marathon spec"
+        exit 1
       end
 
       spec_str = do_subst(marathon_spec, app)
@@ -146,25 +183,34 @@ module Orch
       return marathon_spec
     end
 
+    def parse_bamboo(app, env_var_values)
+      # Do any substs
+      spec_str = do_subst(app.bamboo_spec, app)
+      bamboo_spec = JSON.parse(spec_str)
+
+      if bamboo_spec['acl'].nil?
+        puts "required field 'acl:' missing from bamboo_spec"
+      end
+
+      return bamboo_spec
+    end
+
     def should_deploy?(app)
       result = true
-      if (app.kind == "Marathon") && @options[:deploy_kind] == 'chronos'
-        result = false
-      end
-      if (app.kind == "Chronos") && @options[:deploy_kind] == 'marathon'
+      if @options[:deploy_kind] != 'all' && app.kind != @options[:deploy_kind]
         result = false
       end
 
-      if @options[:deploy_env] != 'all'
-        @options[:deploy_env].split(",").each do |x|
+      if @options[:deploy_var] != 'all'
+        @options[:deploy_var].split(",").each do |x|
           pair = x.split("=")
-          envVar = pair[0]
-          envVal = pair[1]
-          if app[envVar].nil?
-            puts "environment var of '#{envVar}' not found in app"
+          deployVar = pair[0]
+          deployVal = pair[1]
+          if app[deployVar].nil?
+            puts "environment var of '#{deployVar}' not found in app"
             exit 1
           end
-          if app[envVar] != envVal
+          if app[deployVar] != deployVal
             result = false
           end
         end
@@ -176,9 +222,9 @@ module Orch
     def do_subst(spec, app)
       spec_str = spec.to_json.to_s
 
-      # Subst any of the environment_vars values
-      if (! @spec.environment_vars.nil?)
-        @spec.environment_vars.each do |key, value|
+      # Subst any of the deploy_vars values
+      if (! @spec.deploy_vars.nil?)
+        @spec.deploy_vars.each do |key, value|
           spec_str = spec_str.gsub(/{{#{key}}}/, app[key])
         end
       end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: orch
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
 platform: ruby
 authors:
 - Ray Johnson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-08-25 00:00:00.000000000 Z
+date: 2015-08-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -81,11 +81,15 @@ files:
 - Rakefile
 - bin/orch
 - examples/Examples.md
+- examples/bamboo-ex.yml
 - examples/env-chronos.yml
+- examples/env-marathon.yml
 - examples/simple-chronos.yml
 - examples/simple-marathon.yml
+- lib/bamboo.rb
+- lib/chronos.rb
 - lib/config.rb
-- lib/deploy.rb
+- lib/marathon.rb
 - lib/orch.rb
 - lib/orch/version.rb
 - lib/parse.rb