pkgr 0.3.4 → 1.0.1.pre

data/README.md

@@ -1,498 +1,103 @@
 # pkgr
 
-Plug this [Railtie](http://api.rubyonrails.org/classes/Rails/Railtie.html)
-into your Rails 3 app (ruby1.9 only), and you'll be ready to package your
-Rails app as a DEB package. RPM support could/will be added in the short
-future.
+## Goal
 
-This gem originates from hours spent trying to figure out how to package a
-Ruby app for CentOS, and then Debian. It's definitely not trivial, because
-there are so many pieces to put together before getting something working.
-Hopefully this will save time for everyone :)
+Make debian packages out of any app that can run on Heroku.
 
-Fair warning: there are probably a few things that could be done better, or in
-a more standard way. Feel free to open an issue and/or a pull-request.
+## Examples
 
-## Why?
-
-[Capistrano](http://capify.org/) is great for deploying Rails/Ruby
-applications, but the deployment recipe can quickly become a mess, and scaling
-the deployment to more than a few servers can prove to be difficult. Plus, if
-you're already using automation tools such as
-[Puppet](http://www.puppetlabs.com/) to configure your servers, you have to
-run two different processes to configure your infrastructure.
-
-Another issue with Capistrano is that the hook system is not that powerful.
-Compare that with the pre/post-install/upgrade/uninstall steps that you can
-define in a RPM or DEB package, and you'll quickly see the advantage of
-letting a robust package manager such as `apt` or `yum` handle all those
-things for you in a reliable manner.
-
-Last thing, once you built your RPM or DEB package and you tested that it
-works once, you can deploy it on any number of servers at any time and you're
-sure that it will install the required package dependencies, run the hooks,
-and put the files in the directories you specified, creating them as needed.
-Then, you can downgrade or uninstall the whole application in one command.
-
-## How?
-
-The issue with Ruby applications is that most of the gems are not (yet)
-packaged in the various Linux distributions. And even if they were, installing
-multiple Ruby applications that need two different versions of the same Ruby
-library would be impossible, since you can't install two different (minor)
-versions of a library with the package managers.
-
-So, how are we going to easily package Ruby applications and avoid dependency
-issues? Well, I know package maintainers will scream at me, but we'll just
-vendor the required gems in the package we'll build, and use bundler to manage
-those dependencies. Thus, the only dependency we'll put in our package will be
-Ruby1.9 (+rubygems).
-
-## What?
-
-This gem will allow you to package your Rails3 application as a `.deb` package, and sets up a few things for you:
-
-* an `init.d` script to easily start/stop/restart your app, and make it load when the server boots;
-
-* an executable to manually start the server, your rake tasks, or access the console;
-
-* your configuration files will be available in `/etc/app-name/*.yml`;
-
-* defaults for your app (host, port, etc.) can be setup in `/etc/default/app-name`;
-
-* a proper `logrotate` file will be created for you, so that your log files
-  don't eat all the disk space of your server;
-
-* and a few other things.
-
-The default target installation directory for all the other app files will be
-`/opt/local/app-name`. This can be configured.
-
-## Requirements
-
-* You must use Rails3+ and ruby1.9+ in your application. This may work with
-  other rubies but then you'll need to add adapt the dependencies.
-
-* Your Rails application must be able to run with the
-  [`thin`](http://code.macournoyer.com/thin/) web server. Don't forget to add
-  `thin` to your Gemfile!
-
-* Your Rails application must have a `config.ru` file.
-
-* Your application must be checked into a **Git** repository. Your name and
-  email is taken from the git configuration, and the changelog is populated
-  based on the git log between two versions.
-
-## Getting started (quickly)
-
-Aka, how to build a debian package out of your Rails app in 5 minutes. This
-assumes that you have a VM or server running Debian Squeeze readily available
-for building the package.
-
-Install `pkgr`:
-
-    $ gem install pkgr
-
-Create a new Rails app:
-
-    $ rails new my-app
-          create  
-          create  README.rdoc
-          create  Rakefile
-          ...
-          create  vendor/plugins
-          create  vendor/plugins/.gitkeep
-
-Initialize the git repository (if not already done):
-
-    $ cd my-app
-    $ git init
-    $ git add .
-    $ git commit -m "First commit"
-
-Package (this will create a new branch prefixed with `pkgr-`. You can delete
-it afterwards):
-
-    $ pkgr --uri . --bump 0.1.0 --host debian-build-machine
-    ...
-    my-app_0.1.0-1_amd64.deb                                                                     100% 6080KB   5.9MB/s   00:01
-
-Your .deb package is in `pkg/my-app_0.1.0-1_amd64.deb`. Install it on your production server:
-
-    $ scp pkg/my-app_0.1.0-1_amd64.deb production-machine:/tmp/
-    $ ssh production-machine 'sudo dpkg -i /tmp/my-app_0.1.0-1_amd64.deb || sudo apt-get -f -y install'
-
-The app should now be up and running on `0.0.0.0:8000`. You might want to change this in the `/etc/default/my-app` file on the server.
-
-    $ ssh production-machine 'curl -is localhost:8000'
-
-You should get a 404. This is expected since this app does nothing. You can
-now build your app and update the `config/pkgr.yml` file when you have system
-dependencies to add.
-
-Enjoy, and you may want to read the step-by-step guide to learn more about
-what's going on.
-
-## Getting started (step by step)
-
-`pkgr` can be used as a command-line tool (fully automatic), or through the
-rake tasks it installs when you require it in your Gemfile. Below is how you would use the rake tasks to accomplish the same thing as before.
-
-### Setup
-
-Create a new Rails app:
-
-    $ rails new my-app --skip-bundle
-          create  
-          create  README.rdoc
-          create  Rakefile
-          ...
-          create  vendor/plugins
-          create  vendor/plugins/.gitkeep
-
-Go into your app directory, and add `pkgr` to your Gemfile:
-
-    $ cd my-app
-    $ echo "gem 'pkgr'" >> Gemfile
-
-For now, this packaging tool only supports `thin` (would be easy to add others, though), so add it to your Gemfile:
-
-    $ echo "gem 'thin'" >> Gemfile
-
-Install the gems:
-
-    $ bundle install
-
-If it's not already done, initialize a git repository and create a first commit:
-
-    $ git init
-    $ git add .
-    $ git commit -m "First commit"
-
-Setup `pkgr`:
-
-    $ rake pkgr:setup
-    Setting up configuration file...
-    ...
-    Edit '/Users/crohr/tmp/my-app/config/pkgr.yml' and fill in the required information, then enter 'rake pkgr:generate' to generate the debian files.
-
-As outlined, edit `config/pkgr.yml` and fill in your app name. In our example I'll fill in `my-app` as the app name. Also, you should edit the runtime and build dependencies (though the default ones should be fine with a base Rails app).
-
-An example `pkgr.yml` file is given below:
-
-    ---
-    version: 0.4.0
-    name: my-app
-    description: This is a description
-    git_ref: HEAD
-    config_files:
-    - pkgr.yml
-    - database.yml
-    architecture: amd64
-    debian_runtime_dependencies:
-    - ${shlibs:Depends}
-    - ${misc:Depends}
-    - ruby1.9.1-full
-    - git-core
-    - libxml2
-    - libxslt1.1
-    debian_build_dependencies:
-    - debhelper (>= 7)
-    - dpkg-dev
-    - libmysqlclient15-dev
-    - libxml2-dev
-    - libxslt-dev
-    - libsqlite3-dev
-
-### Generate the packaging files
-
-Now generate the required files for packaging:
-
-    $ rake pkgr:generate
-    mkdir -p /Users/crohr/tmp/my-app/debian
-    cp /Users/crohr/.rvm/gems/ruby-1.9.3-p125/gems/pkgr-0.1.0/lib/pkgr/data/debian/changelog /Users/crohr/tmp/my-app/debian/changelog
-    cp /Users/crohr/.rvm/gems/ruby-1.9.3-p125/gems/pkgr-0.1.0/lib/pkgr/data/debian/cron.d /Users/crohr/tmp/my-app/debian/cron.d
-    Correctly set up debian files.
-    mkdir -p /Users/crohr/tmp/my-app/bin
-    cp /Users/crohr/.rvm/gems/ruby-1.9.3-p125/gems/pkgr-0.1.0/lib/pkgr/data/bin/executable /Users/crohr/tmp/my-app/bin/my-app
-    chmod 755 /Users/crohr/tmp/my-app/bin/my-app
-    Correctly set up executable file. Try running './bin/my-app console'.
-
-This will have created the required `debian/` files, plus an executable for your app, so that you're able to do the following:
+* Gitlab package for debian wheezy: <http://deb.pkgr.io/crohr/gitlabhq/>.
 
-    $ ./bin/my-app console development
-    $ ./bin/my-app server start -e development
-    $ ./bin/my-app rake some_task
+## Installation
 
-This is especially useful when the app is deployed on a server, since the executable will be added to the path!
+Note: the new version of pkgr using buildpacks is only available as a pre-release.
 
-By default, you should not have to change anything in the `debian/` folder, so let's package our app. 
+Install `pkgr` on a Debian/Ubuntu machine (only `wheezy` flavour for now):
 
-### Package the app
+    sudo apt-get install ruby1.9.1-full
+    sudo gem install pkgr --version "1.0.1.pre"
 
-First, make sure you committed all your changes:
+## Usage
 
-    $ git add .
-    $ git commit -m "..."
+To package your app, you can either execute `pkgr` locally if your app repository is on the same machine:
 
-Then increase the version number:
+    pkgr path/to/app/repo
 
-    $ rake pkgr:bump:minor
-    Committing changelog and version file...
-    git add debian/changelog /Users/crohr/tmp/my-app/config/pkgr.yml && git commit -m 'v0.1.0' debian/changelog /Users/crohr/tmp/my-app/config/pkgr.yml
-    [master c05dd73] v0.1.0
-     2 files changed, 29 insertions(+), 31 deletions(-)
-     rewrite config/pkgr.yml (82%)
-     create mode 100755 debian/changelog
+Or, assuming your build machine is accessible via SSH by doing `ssh pkgr-build-machine` (set this in your `~/.ssh/config` file), you can do as follows:
 
-Make sure you do not have any staged change (otherwise, commit them):
+    pkgr path/to/app/repo --host pkgr-build-machine
 
-    $ git status
-    # On branch master
-    nothing to commit (working directory clean)
+The resulting .deb package will be in your current working directory.
 
-Finally, ask to build the package on a machine running Debian Squeeze (I generally use my SSH config file to handle the SSH connection details for the specified host):
+Full command line options are given below:
 
-    $ HOST=debian-build-machine rake pkgr:build:deb
+    $ pkgr help package
+    Usage:
+      pkgr package TARBALL
 
-After some time, you should get a final line with the name of your debian package:
+    Options:
+      [--target=TARGET]                        # Target package to build (only 'deb' supported for now)
+                                               # Default: deb
+      [--changelog=CHANGELOG]                  # Changelog
+      [--architecture=ARCHITECTURE]            # Target architecture for the package
+                                               # Default: x86_64
+      [--homepage=HOMEPAGE]                    # Project homepage
+      [--description=DESCRIPTION]              # Project description
+      [--version=VERSION]                      # Package version (if git directory given, it will use the latest git tag available)
+      [--iteration=ITERATION]                  # Package iteration (you should keep the default here)
+                                               # Default: 20131007132226
+      [--user=USER]                            # User to run the app under (defaults to your app name)
+      [--group=GROUP]                          # Group to run the app under (defaults to your app name)
+      [--compile-cache-dir=COMPILE_CACHE_DIR]  # Where to store the files cached between packaging runs
+      [--dependencies=one two three]           # Specific system dependencies that you want to install with the package
+      [--build-dependencies=one two three]     # Specific system dependencies that must be present before building
+      [--before-precompile=BEFORE_PRECOMPILE]  # Provide a script to run just before the buildpack compilation
+      [--host=HOST]                            # Remote host to build on (default: local machine)
+      [--auto]                                 # Automatically attempt to install missing dependencies
+      [--verbose]                              # Run verbosely
+      [--debug]                                # Run very verbosely
+      [--name=NAME]                            # Application name (if directory given, it will default to the directory name)
 
-    [... lots of lines ...]
-    my-app_0.1.0-1_amd64.deb
-
-Make sure it is really here:
-
-    $ ls -l pkg/
-    total 12128
-    -rw-r--r--  1 crohr  staff  6207392 May  4 10:57 my-app_0.1.0-1_amd64.deb
-
-### Use it
-
-Now you can either upload it to an apt repository (if you have one, I'll make a tutorial on how to set up a simple one), or just test that the package works by installing it on your build machine (or another one, for that matter, but you'll have to manually re-install the dependencies):
-
-    $ scp pkg/my-app_0.1.0-1_amd64.deb debian-build-machine:/tmp/
-    $ ssh debian-build-machine
-    debian-build-machine $ sudo dpkg -i /tmp/my-app_0.1.0-1_amd64.deb
-    Selecting previously deselected package my-app.
-    (Reading database ... 53073 files and directories currently installed.)
-    Unpacking my-app (from /tmp/my-app_0.1.0-1_amd64.deb) ...
-    Setting up my-app (0.1.0-1) ...
-    Installing new version of config file /etc/my-app/pkgr.yml ...
-    Adding system user `my-app' (UID 105) ...
-    Adding new group `my-app' (GID 108) ...
-    Adding new user `my-app' (UID 105) with group `my-app' ...
-    Not creating home directory `/home/my-app'.
-    Starting my-app: OK.
-
-Make sure your app is running:
-
-    debian-build-machine $ ps aux | grep my-app | grep -v grep
-    my-app   13928  3.5 10.5 143436 40004 ?        Sl   11:06   0:02 thin server (0.0.0.0:8000) [my-app-0.1.0]
-
-Notice how the process name shows the version number? From experience, this is really useful.
-
-Now you can send a first request:
-
-    $ curl localhost:8000/
-    <!DOCTYPE html>
-    <html>
-    <head>
-      <title>The page you were looking for doesn't exist (404)</title>
-      <style type="text/css">
-        body { background-color: #fff; color: #666; text-align: center; font-family: arial, sans-serif; }
-        div.dialog {
-          width: 25em;
-          padding: 0 4em;
-          margin: 4em auto 0 auto;
-          border: 1px solid #ccc;
-          border-right-color: #999;
-          border-bottom-color: #999;
-        }
-        h1 { font-size: 100%; color: #f00; line-height: 1.5em; }
-      </style>
-    </head>
-
-    <body>
-      <!-- This file lives in public/404.html -->
-      <div class="dialog">
-        <h1>The page you were looking for doesn't exist.</h1>
-        <p>You may have mistyped the address or the page may have moved.</p>
-      </div>
-    </body>
-    </html>
-
-Obviously this app does nothing, so you'll get a 404. So go back to building your app, and then just type `rake pkgr:bump:patch` and `HOST=debian-build-machine rake pkgr:build:deb` to generate a new package !
-
-### Release it (debian)
-
-As of 0.2.0, you can now release the latest package on a server, and add it to
-your list of APT sources for easy installation. In the following we'll assume
-that you want to serve your packages from a host called `apt-server`.
-
-Once you've built the package, run the following to upload it to the
-apt-server and generate the manifest:
-
-    $ HOST=apt-server rake pkgr:release:deb
-
-Note that you need **sudo** privileges for this. If all goes well, you should
-now have a directory named `/var/www/my-app` on `apt-server`. The next step is
-to serve this directory over HTTP. Simply enough, install `apache2` and you're
-good to go:
-
-    $ ssh apt-server 'sudo apt-get install apache2 -y'
-
-Now, on the server on which you want to install the package (let's say
-`production-server`), just add an additional APT source file referencing this
-new APT server:
-
-    
-    production-server # cat /etc/apt/sources.list.d/my-app.list
-    deb http://apt-server.ltd/my-app /
-
-And then:
-
-    production-server # apt-get update && apt-get install my-app
-
-Note that you may need to add the following to a
-`/etc/apt/apt.conf.d/allow-unauthenticated` file if apt complains about
-unauthenticated packages:
-
-    production-server # echo 'APT::Get::AllowUnauthenticated "true";' >> /etc/apt/apt.conf.d/allow-unauthenticated
-
-## General usage
-
-Declare `pkgr` as one of your dependencies in your `Gemfile`:
-
-    gem 'pkgr'
-
-Also add `thin`:
-
-    gem 'thin'
-
-Now make sure you have all the gems installed:
-
-    bundle install
-
-`pkgr` will install a number of new rake tasks to handle the packaging
-workflow. But first, you'll have to create a configuration file to get it
-working:
-
-    rake pkgr:setup
-
-This will create a configuration file at `config/pkgr.yml`. Edit it, and fill
-in details about the `name` of your application, description, and the list of
-runtime dependencies it depends on. Same for dependencies required at build
-time only (most of the time, development headers).
-
-Now you can generate all the files required for building a debian package:
-
-    rake pkgr:generate
-
-A new directory `debian/` should have been created. You can have a look at it,
-but you should not have to edit anything manually.
-
-Once you're ready to package your app, just run the following commands:
-
-* Commit your changes (the `pkgr` app will `git archive HEAD`, which means all
-  your changes must be committed first -- we may want to change this):
-
-        commit -am "..."
-
-* Increment the version number:
-
-        rake pkgr:bump:patch # or rake pkgr:bump:minor or rake pkgr:bump:major
-
-* Build the package on your machine (default, but you better be running a
-  Debian Squeeze, and have an SSH server running), or on a remote machine
-  (recommended, for instance you can get a Vagrant VM in no time):
-
-        HOST=debian-build-machine rake pkgr:build:deb
-        # or HOST=localhost rake pkgr:build:deb, or just rake pkgr:build:deb
-
-  Note that the user with which you're connecting to the build machine **must
-  have `sudo` privileges** (required to install build and runtime
-  dependencies).
-  
-  Also, it's most likely that you'll have to do this a few times at first, as
-  well as adding missing runtime and build dependencies, before your app can
-  be successfully packaged.
-
-* Your .deb package should be made available in the `pkg` directory of your
-  app. Next step is probably to upload it to a local apt repository, and then
-  a simple `apt-get install my-app` will install everything. Enjoy!
-
-## CLI usage
-
-Starting from version 0.3.0, pkgr now comes with an executable, which allows
-to package any app stored in a git repository with one command.
-
-For instance, here is how you would package the [Redmine] [redmine] app:
-
-    $ cd /tmp
-    $ pkgr --uri https://github.com/edavis10/redmine --ref master --bump 1.4.1 \
-      -c https://raw.github.com/crohr/pkgr/master/examples/redmine/configuration.yml \
-      -c https://raw.github.com/crohr/pkgr/master/examples/redmine/database.yml \
-      -c https://raw.github.com/crohr/pkgr/master/examples/redmine/pkgr.yml \
-      --host debian-build
-
-You .deb package will be available in `/tmp/redmine/pkg/`. In this example,
-the given `pkgr.yml` configuration file automatically adds a dependency on
-`mysql-server`, which means that when you install the generated redmine
-package, it will be ready to be accessed on `0.0.0.0:8000`.
-
-Note that for simple projects, you may not need to specify all those
-configuration files on the command line. Redmine is a complex app.
-
-See `pkgr -h` for the list of options available:
-
-    $ pkgr -h
-    * Description
-      pkgr 0.2.0 - Package Rails apps effortlessly.
-    * Usage
-      pkgr --uri GIT_REPOSITORY --config database.yml:http://path/to/database.yml --config ...
-
-    * Common options
-            --uri=                       Sets the Git repository URI (FILE, HTTP, SSH, GIT, etc.) [required]
-        -c, --config=                    Download a configuration file into the config/ folder of the app (HTTP or FILE URIs)
-        -b, --bump=                      Sets the app version [required]
-        -n, --name=                      Sets the app name [optional]
-            --ref=                       Sets the git reference to checkout [default=master]
-            --host=                      Sets the build machine hostname. If none, the process will stop just before building the package.
-
-    * Other
-        -h, --help                       Show this message
-            --version                    Show version
-
-[redmine]: http://www.redmine.org/
-
-## Notes of interest
+## Why?
 
-* your configuration files will be stored in `/etc/my-app/*.yml`, making it easy to manage with Puppet or manually (don't forget to `/etc/init.d/my-app restart` after making changes).
+[Capistrano](http://capify.org/) is great for deploying Rails/Ruby
+applications, but the deployment recipe can quickly become a mess, and scaling
+the deployment to more than a few servers can prove to be difficult. Plus, if
+you're already using automation tools such as
+[Puppet](http://www.puppetlabs.com/) to configure your servers, you have to
+run two different processes to configure your infrastructure.
 
-* you can change how the Thin server is launched by adding options to the `/etc/default/my-app` file.
+Also, once you built your DEB package and you tested that it
+works once, you can deploy it on any number of servers at any time and you're
+sure that it will install the required package dependencies, run the hooks,
+and put the files in the directories you specified, creating them as needed.
+Then, you can downgrade or uninstall the whole application in one command.
 
-* your log files will be stored in `/var/log/my-app/`.
+## What this does
 
-* your db files will be stored in `var/db/my-app/`.
+* Uses Heroku buildpacks to embed all your dependencies within the debian package. That way, no need to mess with stale system dependencies, and no need to install anything by hand. Also, we get for free all the hard work done by Heroku developers to make sure your app runs fine in an isolated way.
 
-* if you've got migrations to run, just do a `my-app rake db:migrate` (we might want to run them automatically as part of the postinstall process).
+* Gives you a nice executable, which closely replicates the Heroku toolbelt utility. For instance, assuming you're packaging an app called `my-app`, you can do the following:
 
-* you can launch a console using `my-app console`.
+        my-app config:set VAR=value
+        my-app config:get VAR
+        my-app run [procfile process]
+        my-app scale web=1 worker=1
+        ...
 
-* use the initd script to start and stop the app: `/etc/init.d/my-app [start|stop|restart|status]`.
+* You app will reside in `/opt/app-name`.
 
-## Todo
+* You'll also get a upstart based initialization script that you can use directly.
 
-* Speed up the packaging process (currently, bundler re-downloads all the gems
-  each time you package an app).
+* Logs will be stored in `/var/log/app-name/`, with a proper logrotate config automatically added.
 
-* Include tasks for building RPMs.
+* Config files can be added in `/etc/app-name/`
 
-* The included initd script sucks. Improve it.
+## Requirements
 
-* Populate system dependencies based on gems declared in the Gemfile.
+* You must have a Procfile.
 
-* Some tests.
+* Your application must be Heroku compatible, meaning you should be able to set your main app's configuration via environment variables.
 
 ## Authors