subspace 0.1.3 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d1a0444e2a77ab17b951b2ea70367a514808945d
-  data.tar.gz: 80f8ca02fbe7ed8c1931e2e542b3643a477ead8a
+  metadata.gz: 6a866534c27d65465d53cc06c495c719197ad402
+  data.tar.gz: f88c9c97b3a26e0876ad83fa99973e586eea8441
 SHA512:
-  metadata.gz: 4f9d38fd9fec7f5e03a6492a75aa287e35f63380758b7d88ea2387d774df734e43f2e28d654c2b895347a83973cd790cc61339037c3455c4761e1e15c59be2a0
-  data.tar.gz: 934f5a090aa313d247cb4f39518dd69cea82a0736443256e69a6fc247e82bdd24e13b1144f62ffd405aec4cd8dc799a9868e4a24858db1e48b9d9d28d7e793a7
+  metadata.gz: 20c7da85e53b22d8e92e02c685b0e03d60ec59cab75370198303083c67d28257e5d51c2a4de47e89961b5d1ac9ba692e0ecf39ad792484de8d30c538f68fa02d
+  data.tar.gz: 6252943b439a6c5f43e9296b1e4035972707f20f54c53ee35818fbf07b9b28053d0495e86cc8b2907e5761b09c86ca16d3f59f0dff86658a9e616b866385b6c6

data/README.md

@@ -6,6 +6,15 @@ http://tvtropes.org/pmwiki/pmwiki.php/Main/SubspaceAnsible
 
 ## Installation
 
+First, install ansible (>2.0)
+
+OSX:
+  brew install ansible
+
+linux:
+  apt-get install ansible
+
+
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -31,10 +40,152 @@ Then run
 Initialize the project for subspace. Creates `config/provision` with all
 necessary files.
 
-* `rake provision:<environment>`
+* `subspace provision <environment>`
 
 Runs the playbook at `config/provision/<environment.yml>`.
 
+* `subspace vars <environment> [--edit] [--create]`
+
+Manage environment variables on different platforms.  The default action is simply to show the vars defined for an environemnt.  Pass --edit to edit them in the system editor.
+
+The new system uses a file in `config/provision/templates/application.yml.template` that contains environment variables for all environments.  The configuration that is not secret is visible and version controlled, while the secrets are stored in the vault files for their environments. The default file created by `subspace init`  looks like this:
+
+```
+# These environment variables are applied to all environments, and can be secret or not
+
+# This is secret and can be changed on all three environment easily by using subspace vars <env> --edit
+SECRET_KEY_BASE: {{secret_key_base}}
+
+#This is not secret, and is the same value for all environments
+ENABLE_SOME_FEATURE: false
+
+development:
+  INSECURE_VARIABLE: "this isn't secret"
+
+dev:
+  INSECURE_VARIABLE: "but it changes"
+
+production:
+  INSECURE_VARIABLE: "on different servers"
+
+```
+
+Further, you can use the extremely command to create a local copy of `config/application.yml`
+
+    # Create a local copy of config/application.yml with the secrets encrypted in vars/development.yml
+    $ subspace vars development --create
+
+This can get you up and running in development securely, the only thing you need to distribute to new team members is the vault password.
+
+NOTE: application.yml should be in the `.gitignore`, since subspace creates a new version on the server and symlinks it on top of whatever is checked in.
+
+# Host configuration
+
+We need to know some info about hosts, but not much.  See the files for details, it's mostly the hostname and the user that can administer the system, eg `ubuntu` on AWS/ubuntu, `ec2-user`, or even `root` (not recommended)
+
+# Role Configuration
+
+This is a description of all the roles that are included by installing subspace, along with their configuration.
+
+## common
+
+This role should almost always be there.  It ties a bunch of stuff together, runs apt-get update or yum upgrade, sets hostnames, and generally makes the server sane.
+
+    project_name: my_project
+    swap_space: 536870912
+    deploy_user: deploy
+
+Note: we grant the deploy user limited sudo access to run `service xyz restart` and also add it to the `adm` group so it can view logs in `/var/log`.
+
+## apache
+
+## collectd
+
+
+
+## delayed_job
+
+## letsencrypt
+
+## logrotate
+
+Installs logrotate and lets you configure logs for automatic rotation.  Example config for rails:
+
+    logrotate_scripts:
+      - name: rails
+        path: "/u/apps/{{project_name}}/shared/log/{{rails_env}}.log"
+        options:
+          - weekly
+          - size 100M
+          - missingok
+          - compress
+          - delaycompress
+          - copytruncate
+
+## memcache
+
+## monit
+
+## mysql
+
+## mysql2_gem
+
+## newrelic
+
+## nginx
+
+## papertrail
+
+## passenger
+
+## postgresql
+
+## puma
+
+## rails
+
+Provisions for a rails app.  This one is probably pretty important.
+
+Default values (these are usually fine)
+
+    database_pool: 5
+    database_name: "{{project_name}}_{{rails_env}}"
+    database_user: "{{project_name}}"
+    job_queues:
+      - default
+      - mailers
+
+Customize:
+
+    rails_env: [whatever]
+
+## redis
+
+## ruby-common
+
+Installs ruby on the machine.  YOu can set a version by picking off the download url and sha hash from ruby-lang.org
+
+    ruby_version: ruby-2.4.1
+    ruby_checksum: a330e10d5cb5e53b3a0078326c5731888bb55e32c4abfeb27d9e7f8e5d000250
+    ruby_download_location: 'https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.1.tar.gz'
+
+
+## sidekiq
+
+
+## Other Internal Roles
+
+Since ansible doesn't support versioning of roles, we cloned the role here so that it doesn't change unexpectedly.  We expect to update from upstream occasionally, please let us know if we're missing something we should have.
+
+You should not include these roles directly in your subspace config files.  For example, instead of including `zenoamaro.postgresql`, simply include our `postgresql` role which depens on zenoamaro's role.
+
+Thanks to the following repositories for making their roles available:
+* https://github.com/zenoamaro/ansible-postgresql
+* https://github.com/mtpereira/ansible-passenger
+
+
+# Development
+
 ## Directory Structure
 
 `ansible/roles`

data/TODO

@@ -0,0 +1,34 @@
+TODO:
+
+subspace init
+  - create ansible.cfg
+    - Add stuff from zach
+  - generate a password and create a file (add to gitignore)
+  - create config/provision
+    + entire directory tree
+  - generate example production.yml, dev.yml
+
+
+  - create the ansible-vault vars files somehow
+
+
+-- 8/21
+
+1. remove 10fw.net as the default hostname
+2. autocreate authorized_keys with ~/.ssh/id_rsa.pub
+3. add readme or something better than this
+4. combine dev and prod.yml into the same shared tasks and update the provision task to just pass in the hosts on the command line, i think that's a better more canonical "ansible" thing to do than have different playbooks for dev and prod.
+5. whats up with the "mtpereira.passenger" role?  if our role wants to depend on that one (see postgres) than file, but its confusing in the role list in dev.yml
+6. create a menu where we ask people what setup they want and they choose each step (apache+passenger) or (nginx+puma) or whatever (or have these in a giant provision.yml and just commented out). similarly they should pick their DB.  Heroku calls these things "buildpacks" and we should do a similar thing.
+7. dont use rake use subspace, eg `subspace provision dev`
+8. not sure why the rake task generates QA and Staging even when the configs dont exist.
+9. AWS stuff shouldn't be in there by default, its way too specific to our use case i think
+10. I can't edit the vault files after doing subspace init.  OK I see I have to be in the provision dir.  We should do that via subspace commands
+11. Needs to prompt for ruby version too or pull it from .ruby-version (does it do this or did festbuddy get lucky?)
+12. Need something like `subspace override <rolename>` that will just make a copy of the role in the provision dir so you can fux0r with it -- also need to think about what happens if we udpate subspace, does it automatically update roles for everyone?  Maybe it should always copy them locally and make you run `subspace update roles` to avoid `bundle update` wrecking people's production servers
+13. `One or more undefined variables: 'letsencrypt_email' is undefined`
+14. We need to figure out app.yml and the secrets file beacause it sucks, also not everyone uses figaro
+15. more options for SSL, like should it auto redirect, what hostnames are allowed, etc
+16. needs an option to set the project name instead of just using the directory
+  - ok this is the there, but the readme should force you to edit this file, and there needs to be a ton more options there (unless we go with provision.rb for configuration, which we could do)
+

data/ansible/roles/apache/tasks/main.yml

@@ -3,28 +3,31 @@
     apt:
       pkg: apache2
       state: present
+    become: true
 
   - name: a2enmod headers
     apache2_module:
       name: headers
       state: present
+    become: true
 
   - name: a2enmod expires
     apache2_module:
       name: expires
       state: present
+    become: true
 
   - name: Create Apache config
     template:
       src: "{{template_src_path}}"
       dest: /etc/apache2/sites-available/{{project_name}}.conf
-    sudo: true
     notify: apache restart
+    become: true
 
   - name: Symlink {{project_name}}.conf to sites-enabled
     file:
       src: /etc/apache2/sites-available/{{project_name}}.conf
       dest: /etc/apache2/sites-enabled/{{project_name}}.conf
       state: link
-    sudo: true
     notify: apache restart
+    become: true

data/ansible/roles/collectd/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+  graphite_host: graphite.example.com
+  graphite_port: "2003"

data/ansible/roles/collectd/handlers/main.yml

@@ -0,0 +1,3 @@
+  - name: restart collectd
+    command: service collectd restart
+    sudo: true

data/ansible/roles/collectd/tasks/main.yml

@@ -0,0 +1,43 @@
+---
+  - name: Create collectd configuration dir
+    become: true
+    file:
+      path: /etc/collectd/collectd.conf.d
+      state: directory
+      mode: 0755
+
+  #required first so that collectd will start even if the FQDN of the host doesnt' resovle
+  - name: Create hostname config
+    template:
+      src: hostname.conf
+      dest: /etc/collectd/collectd.conf.d/hostname.conf
+    become: true
+
+  - name: Install collectd
+    apt:
+      pkg: collectd
+      state: present
+    become: true
+
+  - name: Create graphite config
+    template:
+      src: graphite.conf
+      dest: /etc/collectd/collectd.conf.d/graphite.conf
+    become: true
+    notify: restart collectd
+
+  - name: Create df config
+    template:
+      src: df.conf
+      dest: /etc/collectd/collectd.conf.d/df.conf
+    become: true
+    notify: restart collectd
+
+  - name: create delayed_job_postgres config
+    template:
+      src: delayed_job_postgres.conf
+      dest: /etc/collectd/collectd.conf.d/delayed_job_postgres.conf
+    become: true
+    notify: restart collectd
+    when: collectd_enable_djpg is defined
+

data/ansible/roles/collectd/templates/delayed_job_postgres.conf

@@ -0,0 +1,20 @@
+LoadPlugin postgresql
+<Plugin postgresql>
+  <Query dj_count>
+    Statement "SELECT count(*) as count FROM delayed_jobs"
+    <Result>
+      Type gauge
+      InstancePrefix "dj_count"
+      ValuesFrom count
+    </Result>
+  </Query>
+
+  <Database {{database_name}}>
+    Host "{{database_host}}"
+    Port "5432"
+    User "{{database_user}}"
+    Password "{{database_password}}"
+    Query dj_count
+    Instance "djpg"
+  </Database>
+</Plugin>

data/ansible/roles/collectd/templates/df.conf

@@ -0,0 +1,16 @@
+<Plugin df>
+        FSType rootfs
+        # ignore the usual virtual / temporary file-systems
+        FSType sysfs
+        FSType proc
+        FSType devtmpfs
+        FSType devpts
+        FSType tmpfs
+        FSType fusectl
+        FSType cgroup
+        IgnoreSelected true
+#       ReportByDevice false
+#       ReportInodes false
+#       ValuesAbsolute true
+        ValuesPercentage true
+</Plugin>

data/ansible/roles/collectd/templates/graphite.conf

@@ -0,0 +1,14 @@
+LoadPlugin write_graphite
+<Plugin write_graphite>
+        <Node "example">
+                Host "{{graphite_host}}"
+                Port "{{graphite_port}}"
+                Protocol "tcp"
+                #LogSendErrors true
+                Prefix "collectd."
+                #Postfix "collectd"
+                StoreRates true
+                AlwaysAppendDS false
+                EscapeCharacter "_"
+        </Node>
+</Plugin>

data/ansible/roles/collectd/templates/hostname.conf

@@ -0,0 +1,2 @@
+Hostname "{{hostname}}"
+FQDNLookup false

data/ansible/roles/{delayed_job/files/deploy-service → common/files/sudoers-service}

@@ -1,2 +1 @@
 deploy ALL=(root) NOPASSWD: /usr/sbin/service
-deploy ALL=(root) NOPASSWD: /usr/bin/monit

data/ansible/roles/common/tasks/main.yml

@@ -6,28 +6,44 @@
     template:
       src: motd
       dest: /etc/motd
-    sudo: true
+    become: true
 
   - name: Set hostname
     command: hostname {{hostname}}
-    sudo: true
+    become: true
+
+  - name: Set hostname in /etc/hosts
+    lineinfile:
+      dest: "/etc/hosts"
+      line: "127.0.0.1 {{hostname}}"
+      state: present
+      insertafter: "127.0.0.1 localhost"
+    become: true
 
   - name: update /etc/hostname
     copy:
       content: "{{hostname}}"
       dest: /etc/hostname
+    become: true
 
   - name: Set hostname for systemd
     hostname:
       name: "{{hostname}}"
+    become: true
+
+  - name: install aptitude
+    apt:
+      pkg: aptitude
+      state: present
+    become: true
 
   - name: apt-get update
     apt: update_cache=yes cache_valid_time=86400
-    sudo: true
+    become: true
 
   - name: apt-get upgrade
     apt: upgrade=full
-    sudo: true
+    become: true
 
   - name: Set timezone variables
     copy: content='America/Chicago'
@@ -36,6 +52,7 @@
           group=root
           mode=0644
           backup=yes
+    become: true
     notify:
       - update timezone
 
@@ -45,20 +62,33 @@
       state: present
       generate_ssh_key: yes
       shell: /bin/bash
-    sudo: true
+    become: true
+
+  - name: Add deploy user to adm group so it can view logs in /var/log
+    user:
+      name: "{{deploy_user}}"
+      append: yes
+      groups: "adm"
+    become: true
+
+  - name: Add sudoers.d file so that deploy can restart services without entering password.
+    copy:
+      src: sudoers-service
+      dest: /etc/sudoers.d/service
+    become: true
 
   - name: Update authorized_keys for deploy user
     copy:
       src: authorized_keys
       dest: "/home/{{deploy_user}}/.ssh/authorized_keys"
       owner: "{{deploy_user}}"
-    sudo: true
+    become: true
 
   - name: Create directory to which to deploy
     file:
       path: /u/apps/{{project_name}}
       owner: "{{deploy_user}}"
       state: directory
-    sudo: true
+    become: true
 
   - include: swap.yml

data/ansible/roles/common/tasks/swap.yml

@@ -1,41 +1,43 @@
 - name: set swap_file variable
+  become: true
   set_fact:
     swap_file: /swapfile
 
 - name: check if swap file exists
+  become: true
   stat:
     path: /swapfile
   register: swap_file_check
 
 - name: create swap file
-  sudo: yes
+  become: true
   command: fallocate -l {{ swap_space }} /swapfile
   when: not swap_file_check.stat.exists
 
 - name: set permissions on swap file
-  sudo: yes
+  become: true
   file:
     path: /swapfile
     mode: 0600
 
 - name: format swap file
-  sudo: yes
+  become: true
   command: mkswap /swapfile
   when: not swap_file_check.stat.exists
 
 - name: add to fstab
-  sudo: yes
+  become: true
   lineinfile:
     dest: /etc/fstab
     regexp: /swapfile
     line: "/swapfile none swap sw 0 0"
 
 - name: turn on swap
-  sudo: yes
+  become: true
   command: swapon -a
 
 - name: set swapiness
-  sudo: yes
+  become: true
   sysctl:
     name: vm.swappiness
     value: "1"

data/ansible/roles/common/templates/motd

@@ -1,9 +1,12 @@
-###############################################################################
-This server is provisioned by subspace.
+This server brought to you by:
+ ____        _    ____
+/ ___| _   _| |__/ ___| _ __   __ _  ___ ___
+\___ \| | | | '_ \___ \| '_ \ / _` |/ __/ _ \
+ ___) | |_| | |_) |__) | |_) | (_| | (_|  __/
+|____/ \__,_|_.__/____/| .__/ \__,_|\___\___|
+                       |_|
 
-If you need to make configuration changes to the server, including package
-updates, /etc/* configuration, user access, etc, please consult the
-config/provision directory in the app.
+~~~ https://github.com/tenforwardconsulting/subspace ~~~
 
-For more information, refer to https://github.com/tenforwardconsulting/subspace
-###############################################################################
+If you need to make configuration changes to the server, please modify the
+config/provision directory in the app or risk the changes dissapearing.

data/ansible/roles/common/templates/motd2

@@ -0,0 +1,24 @@
+
+                   .------,      YOU ARE
+      .\/.          |______|    A SPECIAL
+    _\_}{_/_       _|_Ll___|_    SNOWFLAKE
+     / }{ \       [__________]          .\/.
+      '/\'        /          \        _\_\/_/_
+                 ()  o  o    ()        / /\ \
+                  \ ~~~   .  /          '/\'
+             _\/   \ '...'  /    \/_
+              \\   {`------'}    //
+               \\  /`---/',`\\  //
+                \/'  o  | |\ \`//
+                /'      | | \/ /\
+   __,. -- ~~ ~|    o   `\|      |~ ~~ -- . __
+               |                 |
+          jgs  \    o            /  THIS SERVER
+                `._           _.'     IS NOT
+                   ^~- . -  ~^
+
+       This server is managed by SubSpace:
+ https://github.com/tenforwardconsulting/subspace
+
+If you need to make configuration changes to the server, please modify the
+config/provision directory in the app or risk the changes dissapearing.

data/ansible/roles/delayed_job/README.md

@@ -2,8 +2,6 @@
 
 ## Variables
 
-### Optional
+### Required
 
-* `delayed_job_queues`
-  The delayed job queues so the upstart script can start each one.
-  If this is not set, then the upstart script will start delayed\_job without specifying any queue and it will run all of your jobs.
+`delayed_job_command`

data/ansible/roles/delayed_job/defaults/main.yml

@@ -1,4 +1,2 @@
 ---
   delayed_job_command: bin/delayed_job
-  delayed_job_queues:
-    - default

data/ansible/roles/delayed_job/meta/main.yml

@@ -0,0 +1,5 @@
+---
+  dependencies:
+    - {
+      role: monit
+    }

data/ansible/roles/delayed_job/tasks/main.yml

@@ -1,34 +1,19 @@
 ---
-  - name: Install monit
-    apt:
-      name: monit
-      state: present
-
   - name: Install delayed_job monit script
     template:
       src: delayed-job-monit-rc
       dest: /etc/monit/conf.d/delayed_job_{{project_name}}_{{rails_env}}
-    notify: monit reload
-
-  - name: Copy sudoers file so that deploy can restart services without entering password.
-    copy:
-      src: deploy-service
-      dest: /etc/sudoers.d/deploy-service
-    sudo: true
-
-  - name: Copy monit config to enable http from localhost
-    copy:
-      src: monit-http.conf
-      dest: /etc/monit/conf.d/monit-http.conf
     sudo: true
-    notify: monit reload
+    notify: monit
 
   - name: Remove old upstart files
     file:
       path: /etc/init/delayed-job.conf
       state: absent
+    sudo: true
 
   - name: Remove old monit files
     file:
       path: /etc/monit/conf.d/delayed_job
       state: absent
+    sudo: true

data/ansible/roles/delayed_job/templates/delayed-job-monit-rc

@@ -8,7 +8,7 @@
 #
 #   include /var/www/apps/{app_name}/shared/delayed_job.monitrc
 
-{% for queue in delayed_job_queues %}
+{% for queue in job_queues %}
 check process delayed_job_{{queue}}
   with pidfile /u/apps/{{project_name}}/shared/tmp/pids/delayed_job.{{queue}}.pid
   start program = "/bin/su - deploy -c 'cd /u/apps/{{project_name}}/current; RAILS_ENV={{rails_env}} bundle exec {{delayed_job_command}} --identifier={{queue}} --queue={{queue}} start'"

data/ansible/roles/letsencrypt/defaults/main.yml

@@ -1,2 +1,2 @@
 ---
-  certbot_dir: "/home/ubuntu"
+  certbot_dir: "/opt/certbot"