ciinabox-ecs 0.1.6

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 353e5b07dfebb932b4853d1a614e59a587877dfa
+  data.tar.gz: a167bceccc62cfd3a4ba48d03fc1442bfe76ba03
+SHA512:
+  metadata.gz: 0de55a2bb67c0cb9bc43bdc98f55b3fa97896f491304d0b02d27bd07033189bb979ec2e1a47d7c37f427e3579db5cacff714083375582e5ac7358ad8d70159a6
+  data.tar.gz: 170ee8dde7a66bc109ad5b5d594e52497e0113be1e7b84bb96766b8d696b572b231d43f9b5bb4e95e6d8b377f897b68609a938dd9adda8ae58fa787dc1cb867b

data/Gemfile

@@ -0,0 +1,7 @@
+source 'https://rubygems.org'
+
+gem 'rake'
+gem 'cfndsl','0.15.2'
+gem 'cfn_manage'
+gem 'deep_merge'
+gem 'rubyzip'

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 base2Services
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,458 @@
+# ciinabox ECS
+
+ciinabox pronounced ciin a box is a set of automation for building
+and managing a bunch of CI tools in AWS using the Elastic Container Service (ECS).
+
+Right Now ciinabox supports deploying:
+
+ * [jenkins](https://jenkins.io/)
+ * [drone](http://docs.drone.io)
+ * [bitbucket](https://www.atlassian.com/software/bitbucket)
+ * [hawtio](http://hawt.io/)
+ * [nexus](http://www.sonatype.org/nexus/)
+ * plus custom tasks and stacks
+
+## Setup
+
+requires ruby 2.3+
+
+install [ciinabox-ecs](https://rubygems.org/gems/ciinabox-ecs/) gem
+
+```bash
+$ gem install ciinabox-ecs
+...
+Done installing documentation for ciinabox-ecs after xx seconds
+1 gem installed
+
+$ ciinabox-ecs help
+Usage: ciinabox-ecs action1 action2 action3 ciinabox_name
+Valid actions:
+name                 |description         
+
+active               |Switch active ciinabox
+create               |Creates the ciinabox environment
+create_server_cert   |Create self-signed SSL certs for use with ciinabox
+create_source_bucket |Creates the source bucket for deploying ciinabox
+deploy               |Deploy Cloudformation templates to S3
+down                 |Turn off your ciinabox environment
+full_install         |Initialize configuration, create required assets in AWS account, create Cloud Formation stack
+generate             |Generate CloudFormation templates
+generate_keypair     |Generate ciinabox AWS keypair
+init                 |Initialise a new ciinabox environment
+package_lambdas      |Package Lambda Functions as ZipFiles
+ssh                  |SSH into your ciinabox environment
+status               |Current status of the active ciinabox
+tear_down            |Deletes/tears down the ciinabox environment
+up                   |Turn on your ciinabox environment
+update               |Updates the ciinabox environment
+update_cert_to_acm   |Replace previously auto-generated IAM certificate with auto-validated ACM certificate (if one exists)
+upload_server_cert   |Uploads SSL server certs for ciinabox
+
+```
+
+If setting your own parameters and additional services, they should be configured as such:
+
+#### User-defined parameters:
+ciinaboxes/ciinabox_name/config/params.yml
+
+e.g:
+```ruby
+log_level: ':debug'
+timezone: 'Australia/Melbourne'
+```
+
+#### User-defined services:
+If you wish to add additional containers to your ciinabox environment, you can specify them like so:
+ciinaboxes/ciinabox_name/config/services.yml
+
+e.g:
+```yaml
+---
+services:
+  - jenkins:
+  - bitbucket:
+      LoadBalancerPort: 22
+      InstancePort: 7999
+      Protocol: TCP
+  - hawtio:
+  - nexus:
+```
+
+Please note that if you wish to do this, that you also need to create a CFNDSL template for the service under templates/services, with the name of the service as the filename (e.g. bitbucket.rb)
+
+Note the drone service requires a minimum yaml configuration of below
+```yml
+services:
+  - drone:
+      params:
+        -
+          VPC:
+            Ref: VPC
+        -
+          SubnetPublicA:
+            Ref: SubnetPublicA
+        -
+          SubnetPublicB:
+            Ref: SubnetPublicB
+        -
+          ECSSubnetPrivateA:
+            Ref: ECSSubnetPrivateA
+        -
+          ECSSubnetPrivateB:
+            Ref: ECSSubnetPrivateB
+        -
+          SecurityGroupBackplane:
+            Ref: SecurityGroupBackplane
+        -
+          SecurityGroupOps:
+            Ref: SecurityGroupOps
+        -
+          SecurityGroupDev:
+            Ref: SecurityGroupDev
+        -
+          SecurityGroupNatGateway:
+            Ref: SecurityGroupNatGateway
+        -
+          SecurityGroupWebHooks:
+            Ref: SecurityGroupWebHooks
+        -
+          ECSENIPrivateIpAddress:
+            Ref: ECSENIPrivateIpAddress
+      tasks:
+        drone-server:
+          env:
+            DRONE_OPEN: true
+```
+to further configure drone ci refer to the drone ci's environment variable in the documentation http://docs.drone.io/installation/, you can add/override drone's environment variable to their corresponding yaml section (`drone-server` and `drone-agent`), example
+```yml
+      tasks:
+        drone-server:
+          env:
+            DRONE_OPEN: true
+            DRONE_SECRET: base2services # if this value is not specified, a secure random hex will be used
+        drone-agent:
+          env:
+            DRONE_SECRET: base2services # if this value is not specified, a secure random hex will be used
+```
+
+## Getting Started
+
+To get started install `ciinabox-ecs` ruby gem
+
+```bash
+$ gem install ciinabox-ecs
+```
+
+### Quick setup
+
+You can be guided through full installation of ciinabox by running `full_install` action. Interactive
+command line prompt will offer you defaults for most of required options.
+
+```bash
+$ ciinabox-ecs full_install
+
+```
+
+### Step by step setup
+
+1. Initialize/Create a new ciinabox environment. Please note that any user-defined services and parameters will be merged during this task into the default templates
+  ```bash
+  $ ciinabox-ecs init
+  Enter the name of ypur ciinabox:
+  myciinabox
+  Enter the id of your aws account you wish to use with ciinabox
+  111111111111
+  Enter the AWS region to create your ciinabox (e.g: ap-southeast-2):
+  us-west-2
+  Enter the name of the S3 bucket to deploy ciinabox to:
+  source.myciinabox.com
+  Enter top level domain (e.g tools.example.com), must exist in Route53 in the same AWS account:
+  myciinabox.com
+  # Enable active ciinabox by executing or override ciinaboxes base directory:
+  export CIINABOXES_DIR="ciinaboxes/"
+  export CIINABOX="myciinabox"
+  ```
+  You can override the default ciinaboxes directory by setting the CIINABOXES_DIR environment variable. Also the DNS domain you entered about must already exist in Route53
+
+
+3. Generate self-signed wild-card cert for your ciinabox
+  ```bash
+  $ ciinabox-ecs create_server_cert [ciinabox_name]
+  Generating a 4096 bit RSA private key
+  .......................................................................................................................................++
+  ....................++
+  writing new private key to 'ciinaboxes/myciinabox/ssl/ciinabox.key'
+  -----
+  ```
+
+4. Create IAM server-certificates
+  ```bash
+  $ ciinabox-ecs upload_server_cert [ciinabox_name]
+  Successfully uploaded server-certificates
+  ```
+
+5. Create ciinabox S3 source deployment bucket
+  ```bash
+  $ ciinabox-ecs create_source_bucket [ciinabox_name] 
+  Successfully created S3 source deployment bucket source.myciinabox.com
+  ```
+
+6. Create ssh ec2 keypair
+  ```bash
+  $ ciinabox-ecs generate_keypair [ciinabox_name]
+  Successfully created ciinabox ssh keypair
+  ```
+
+7. Generate ciinabox cloudformation templates
+  ```bash
+  $ ciinabox-ecs generate [ciinabox_name] 
+  Writing to output/ciinabox.json
+  using extras [[:yaml, "ciinaboxes/myciinabox/config/default_params.yml"], [:yaml, "config/services.yml"], [:ruby, "ext/helper.rb"]]
+  Loading YAML file ciinaboxes/myciinabox/config/default_params.yml
+  Setting local variable ciinabox_version to 0.1
+  Setting local variable ciinabox_name to myciinabox
+  ......
+  ......
+  $ ls -al output/
+  total 72
+  drwxr-xr-x   9 ciinabox  staff    306  9 Sep 21:52 .
+  drwxr-xr-x  14 ciinabox  staff    476 19 Oct 10:26 ..
+  -rw-r--r--   1 ciinabox  staff      0  7 Sep 14:30 .gitkeep
+  -rw-r--r--   1 ciinabox  staff   1856 19 Oct 13:27 ciinabox.json
+  -rw-r--r--   1 ciinabox  staff   6096 19 Oct 13:27 ecs-cluster.json
+  -rw-r--r--   1 ciinabox  staff   1358  9 Sep 17:39 ecs-service-elbs.json
+  -rw-r--r--   1 ciinabox  staff   3250 19 Oct 13:27 ecs-services.json
+  drwxr-xr-x   4 ciinabox  staff    136  9 Sep 21:53 services
+  -rw-r--r--   1 ciinabox  staff  13218 19 Oct 13:27 vpc.json
+  ```
+  This will render the cloudformation templates locally in the output directory
+
+8. Deploy/upload cloudformation templates to source deployment bucket
+  ```bash
+  $ ciinabox-ecs deploy [ciinabox_name]
+  upload: output/vpc.json to s3://source.myciinabox.com/ciinabox/0.1/vpc.json
+  upload: output/ecs-services.json to s3://source.myciinabox.com/ciinabox/0.1/ecs-services.json
+  upload: output/ciinabox.json to s3://source.myciinabox.com/ciinabox/0.1/ciinabox.json
+  upload: output/services/jenkins.json to s3://source.myciinabox.com/ciinabox/0.1/services/jenkins.json
+  upload: output/ecs-service-elbs.json to s3://source.myciinabox.com/ciinabox/0.1/ecs-service-elbs.json
+  upload: output/ecs-cluster.json to s3://source.myciinabox.com/ciinabox/0.1/ecs-cluster.json
+  Successfully uploaded rendered templates to S3 bucket source.myciinabox.com
+  ```
+
+9. Create/Lanuch ciinabox environment
+  ```bash
+  $ ciinabox-ecs create base2
+  Starting updating of ciinabox environment
+  # checking status using
+  $ ciinabox-ecs status base2
+  base2 ciinabox is in state: CREATE_IN_PROGRESS
+  # When your ciinabox environment is ready the status will be
+  base2 ciinabox is alive!!!!
+  ECS cluster private ip:10.xx.xx.xx
+  ```
+  You can access jenkins using http://jenkins.myciinabox.com
+
+## Additional Tasks
+
+### ciinabox-ecs update
+
+Runs a cloudformation update on the current ciinabox environment. You can use this task if you've modified the default_params.yml config file for your ciinabox and you want to apply these changes to your ciinabox.
+
+A common update would be to lock down ip access to your ciinabox environment
+
+1. edit ciinaboxes/myciinabox/config/default_params.yml
+
+  ```yaml
+  ....
+  #Environment Access
+  #add list of public IP addresses you want to access the environment from
+  #default to public access probably best to change this
+  opsAccess:
+    - my-public-ip
+    - my-my-other-ip
+  #add list of public IP addresses for your developers to access the environment
+  #default to public access probably best to change this
+  devAccess:
+    - my-dev-teams-ip
+  ....
+  ```
+
+2. update your ciinabox
+  ```bash
+  $ ciinabox-ecs generate deploy update [ciinabox_name]
+  $ ciinabox-ecs status [ciinabox_name]
+  ```
+
+### ciinabox-ecs tear_down [ciinabox_name]
+
+Tears down your ciinabox environment. But why would you want to :)
+
+
+### ciinabox-ecs up [ciinabox_name]
+
+Relies on [cfn_manage](https://rubygems.org/gems/cfn_manage) gem to bring stack up. Stack needs to be stopped using `ciinabox:down` task
+
+### ciinabox-ecs down [ciinabox_name]
+
+Relies on [cfn_manage](https://rubygems.org/gems/cfn_manage) gem to stop the stack. Will set ASG size to 0 (and optionally set bastion ASG size to 0).
+
+## Adding Custom Templates per ciinabox
+
+Custom templates should be defined under <CIINABOXES_DIR>/<CIINABOX>/templates.
+
+For each stack that needs to be included add a stack under extra_stacks in the config.yml.  
+
+By default the name of the nested stack will be assumed to be the file name when the template is getting called.  This can be overriden.  
+
+Parameters get passed in as a hash and all get passed in from the top level.
+
+\#extra_stacks:
+\#  elk:
+\#    #define template name? - optional
+\#    file_name: elk
+\#    parameters:
+\#      RoleName: search
+\#      CertName: x
+
+# Extra configs
+
+## To restore the volume from a snapshot in an existing ciinabox update the following 2 values
+
+ecs_data_volume_snapshot: (Note: if ciinabox exists this is two step approach you will need to change volume name and change back volume name)
+
+ecs_data_volume_name: override this if you need to re-generate the volume, e.g. from snapshot
+
+\#add if you want ecs docker volume != 22GB - must be > 22
+
+\#ecs_docker_volume_size: 100
+
+\#use this to change volume snapshot for running ciinabox
+
+\#ecs_data_volume_name: "ECSDataVolume2s"
+
+\#set the snapshot to restore from
+
+\#ecs_data_volume_snapshot: snap-49e2b3b5
+
+\#set the size of the ecs data volume -- NOTE: would take a new volume - i.e. change volume name
+
+\#ecs_data_volume_size: 250
+
+\#optional ciinabox name if you need more than one or you want a different name
+
+\#stack_name: ciinabox-tools
+
+## For internal elb for jenkins
+
+```
+internal_elb: false
+
+ - jenkins:
+    LoadBalancerPort: 50000
+    InstancePort: 50000
+    Protocol: TCP
+# needs internal_elb: true
+```
+
+# Ciinabox configuration
+
+## Bastion (Jumpbox) instance
+
+If you have need to access ECS Cluster instance running Jenkins server via secure shell, you may do so by logging
+into bastion host first. By default, bastion is disabled for ciinabox Cloud Formation stack, however you can enable
+it by using `bastion_stack` configuration key. Bastion will be launched as part of AutoScaling Group of size 1,
+allowing it to self heal in case of system or instance check failure.
+
+```yaml
+include_bastion_stack: true
+```
+
+It is also possible to override other bastion host parameters, such as Amazon Machine Image and instance type
+used for Launch Configuration. Defaults are below
+
+```yaml
+bastionInstanceType: t2.micro
+# Amazon Linux 2017.09
+bastionAMI:
+  us-east-1:
+   ami: ami-c5062ba0
+  us-east-2:
+   ami: ami-c5062ba0
+  us-west-2:
+   ami: ami-e689729e
+  us-west-1:
+   ami: ami-02eada62
+  ap-southeast-1:
+   ami: ami-0797ea64
+  ap-southeast-2:
+   ami: ami-8536d6e7
+  eu-west-1:
+   ami: ami-acd005d5
+  eu-west-2:
+   ami: ami-1a7f6d7e
+  eu-central-1:
+   ami: ami-c7ee5ca8
+
+```
+
+
+## IAM Roles
+
+Default IAM permission for ciinabox stack running Jenkins server are set in `config/default_params.yml`, under
+`ecs_iam_role_permissions_default` configuration key. You can extend this permissions on a ciinabox level
+using `ecs_iam_role_permissions_extras` key. E.g.
+
+(within `$CIINABOXES_DIR/$CIINABOX/config/params.yml`)
+```yaml
+
+ecs_iam_role_permissions_extras:
+  -
+    name: allow-bucket-policy
+    actions:
+      - s3:PutBucketPolicy
+
+```
+
+## Allowing connections from NAT gateway
+
+If ECS Cluster and running Jenkins will try to access itself via public route and url, you will need
+to allow such traffic using Security Group rules. As NAT Gateway is used for sending all requests to internet,
+it is NAT Gateways IP address that should be added to Group rules. Use `allow_nat_connections` configuration
+key for this.
+
+```yaml
+allow_nat_connections: false
+```
+
+## Automatic issuance and validation of ACM SSL certificate
+
+This setting is enabled by default in default parameters. During the ciinabox init stage, you will be 
+asked if you want to utilise this functionality. Essentially, custom cloudformation resource based on 
+python [aws-acm-validator](https://pypi.python.org/pypi/aws-acm-cert-validator) python package will 
+request and validate ACM certificate through appropriate Route 53 DNS validation record. 
+
+### To disable during ciinabox setup
+
+Answer question below with 'y' during ciinabox init stage 
+
+```text
+Use selfsigned rather than ACM issued and validated certificate (y/n)? [n]
+```
+
+### To disable for existing ciinaboxes
+
+Within `$CIINABOXES_DIR/$CIINABOX/params.yml`
+
+```yaml
+acm_auto_issue_validate: false
+```
+
+### To migrate previous versions of ciinabox to this functionality
+
+After updating to latest ciinabox version including this functionality, you may want to update value of `default_ssl_cert_id`
+configuration key to ARN of the freshly issued ACM certificate. You can do that using `update_cert_to_acm` action
+
+```yaml
+$ ciinabox-ecs update_cert_to_acm [ciinabox_name]
+Set arn:aws:acm:ap-southeast-2:123456789012:certificate/2f2f3f9f-aaaa-bbbb-cccc-11dac04e7fb9 as default_cert_arn
+```