wilbur 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 417d87a72995963bd253992473a7187df70a3a9d
+  data.tar.gz: ea7e93c56c213dac1aa3516ab3945e6717c05345
+SHA512:
+  metadata.gz: 74b54c9428e99513ef221cf1a59b2895a681c9fb18011163cc5547da8c65f4bd2db8b44c9ab1ae9ef61766585fda837d01c592a5a0c56f677b58fd66defe6eac
+  data.tar.gz: 2505db8a85a9dbb82820fb9326a2570a28f3ee074702f0a3f836e080709967c6269ee0f9ba8b353a5b78406a21665ea9d66065d648ef1bd6521edff2354f2147

data/.gitignore

@@ -0,0 +1,8 @@
+.DS_Store
+*.swp
+*.swo
+.bundle
+.vagrant
+pkg
+vendor
+Gemfile.lock

data/.ruby-version

@@ -0,0 +1 @@
+2.0.0-p195

data/Gemfile

@@ -0,0 +1,5 @@
+source "https://rubygems.org"
+
+gemspec
+
+gem "travis"

data/README.md

@@ -0,0 +1,93 @@
+# Wilbur: Your OpenWRT Build Assistant
+
+This project provides a comprehensive infrastructure for building and
+provisioning OpenWRT images easily and in an automated fashion.
+
+For more information about OpenWRT, please check http://openwrt.org
+
+## Project structure
+* `openwrt_release`: here you specify the name of the release targeted for
+  build and deployment
+* `flavors/`: contains the different kernel and rootfs configurations used
+  to build working OpenWRT images
+  * `flavors.yml`: serves as a db of flavor configurations. Used to provide
+    more understanding over each configuration and for referencing them more
+    easily by the user.
+  * `<flavor_name>/`: contains all files related to the specific flavor
+      * `dot_config`: Buildroot configuration file used to specify kernel
+        configuration and packages to compile/install in the final image
+      * `patches/`: contains all diff patches to be applied to the OpenWRT's
+        source code prior to building it
+* `provisioning/`: contains the resources used to provision project's virtual
+  machines
+  * `manifests/`: contains the Puppet manifests used to bring up the building and
+    provisioning VMs
+  * `scripts/`: contains the scripts used with Vagrant to aid Puppet
+    provisioning
+
+## Usage
+
+At the moment, all facilites are provided as **Thor**
+commands. Also, **Bundler** is used to contain the project in its own folder.
+
+To start using the builder, just let Bundler do its work:
+
+`bundle install --path vendor/bundle`
+
+Then, to see a list of the available commands, run:
+
+`bundle exec thor -T`
+
+To see a description of the use cases supported by the project, read on this
+doc file.
+
+### Declaring OpenWRT release to build
+In the project's root folder there's a file named `openwrt_release`. This file
+must contain a single line with the name of the specific OpenWRT release
+you want to build (e.g. `attitude_adjustment`). If you want to build bleeding
+edge, just set it to `openwrt`.
+
+#### Internals
+The name specified in `openwrt_release` is used to build the URL of the source
+repository to clone locally (so format matters). Basically, source will be fetched
+from
+
+`git://nbd.name/<openwrt_release>.git`
+
+To provide this same name to Puppet during provisioning, some
+"pre-provisioning" is done prior to running Puppet; during this step, a custom
+executable fact will be added to the **building machine**, named
+`openwrt_release`, that will read the content of the file in the shared project
+folder. This way, you can specify the release you want to use once and not
+worry about synchronizing names between building app and provisioning manifest.
+
+### Building a specific flavor
+You can easily start a build by invoking the `build` task:
+
+`bundle exec thor build <flavor>`
+
+To obtain a list of available flavors, you can ask thor:
+
+`bundle exec thor flavor_list`
+
+This will return the list of available flavors. Each row will be in the form:
+
+`<flavor name> <flavor description>`
+
+The description is there to clarify what the flavor's purpose is; you can just
+pick the name and pass it over to the `build` task.
+
+Also, additional options are available for the `build` command:
+
+* `--verbose`: let the compilation process be verbose about what's doing
+  (useful for debugging)
+* `--nuke`: prior to start compilation, delete **everything** that is not bare
+  source code. _Note that in standard operation mode, build command already
+  wipes the output of previous run(s), so in normal circumstances this flag is
+  not needed. You may need it if you mess with the configuration and want to
+  restart with a clean environment. For more information, have a look at
+  [`make distclean` Buildroot command](http://wiki.openwrt.org/doc/howto/build)._
+
+**NOTE**: At the moment, when the build process ends, nothing is done to gather
+the produced files. For now, you have to ssh into the **builder** box and find
+the kernel and rootfs under `attitude_adjustment/bin/<arch>/`.

data/Rakefile

@@ -0,0 +1,16 @@
+require 'rake/testtask'
+require 'wilbur/version'
+
+namespace :gem do
+  require 'bundler/gem_tasks'
+end
+
+namespace :test do
+  Rake::TestTask.new(:end_to_end) do |t|
+    t.verbose = true
+    t.libs << "lib"
+    t.libs << "test"
+    t.test_files = FileList["test/end_to_end/**/*_test.rb"]
+    #t.ruby_opts = ["-w"]
+  end
+end

data/Thorfile

@@ -0,0 +1,145 @@
+# -*- mode: ruby -*-
+# vim: ft=ruby
+
+require 'thor'
+require 'net/ssh'
+
+class Default < Thor
+  desc "build FLAVOR", "builds the specified OpenWRT flavor on the building machine"
+  long_desc <<-EOS
+    Builds the specified OpenWRT flavor on the building node.
+
+    This means compiling the kernel and generating one or more OpenWRT image
+    files.
+    \x5The kernel will be configured with the options specified in the flavor
+    configuration file; in the same file (which is a standard kernel dot file
+    suitable for OpenWRT Buildroot) will be described which packages are to be compiled
+    and which ones are to be installed in the preconfigured image.
+
+    You can get a list of available flavors by running:
+    \x5
+    \x5\t thor flavor_list
+
+    The name specified in the first column can be used as an argument for the
+    build command.
+  EOS
+  option :verbose, :type => :boolean, :aliases => :v
+  option :nuke,    :type => :boolean, :aliases => :n
+  def build(flavor)
+    if flavors.select { |f| f["name"] == flavor}.empty?
+      puts "No flavor found matching `#{flavor}`"
+      exit 1
+    end
+
+    if not Dir.exists?(File.join('flavors', flavor))
+      puts <<-EOS
+        Sorry, no directory flavors/#{flavor} exist. Please read how flavors 
+        management work and fix your configuration before proceeding.
+      EOS
+      exit 1
+    end
+
+    dot_config = File.join(flavor, 'dot_config')
+    if not File.exists?(File.join('flavors', dot_config))
+      puts <<-EOS
+        Sorry, flavors/#{flavor}/dot_config file doesn't exist.  Please read how flavors 
+        management work and fix your configuration before proceeding.
+      EOS
+      exit 1
+    end
+
+    verbose = options[:verbose] ? "V=s" : ""
+
+    invoke :prepare_builder, [], []
+
+    Net::SSH.start('', nil, ssh_options_for(BUILDER_VM)) do |builder|
+      builder.exec! "cd #{openwrt_release}; make distclean" if options[:nuke]
+
+      puts "Preparing building environment"
+      builder.exec! "
+        cd #{openwrt_release}
+        git pull
+        ./scripts/feeds update -a
+        ./scripts/feeds install -a" do |channel, stream, data|
+        $stdout << data
+      end
+
+      puts "Syncing OpenWRT dot config file"
+      builder.exec! "cp #{File.join(VAGRANT_SHARE_ROOT, 'flavors', dot_config)} #{File.join(openwrt_release, '.config')}"
+
+      puts "Starting image build process"
+      builder.exec! "
+        cd #{openwrt_release}
+        make clean
+        make #{verbose}" do |channel, stream, data|
+        $stdout << data
+      end
+
+      puts "Build finished."
+    end
+  end
+
+  desc "prepare_builder", "prepares builder VM for subsequent use"
+  def prepare_builder
+    case `vagrant status #{BUILDER_VM}`
+    when /vm is running/i     then  ['provision']
+    when /to resume this vm/i then  ['up', 'provision']
+    else                            ['up']
+    end.each do |cmd| system 'vagrant', cmd, BUILDER_VM  end
+  end
+
+  desc "flavor_list", "list available configurations for kernel compilation and rootfs configuration"
+  def flavor_list
+    flavors.each do |config|
+      puts "#{config["name"]} : #{config["desc"]}"
+    end
+  rescue Exception => e
+    puts e.message
+    exit 1
+  end
+
+  no_commands do
+    BUILDER_VM = 'builder'.freeze
+    VAGRANT_SHARE_ROOT = '/vagrant'.freeze
+
+    def flavors
+      YAML::load_file("flavors/flavors.yml")
+    end
+
+    def ssh_options_for(host)
+      ssh_config = `vagrant ssh-config #{host}`
+
+      return if ssh_config.empty?
+
+      net_ssh_options = {}
+      hashed_ssh_config(ssh_config).each do |option|
+        param = options_map[option[:key]]
+        net_ssh_options[param] = option[:value] if param
+      end
+      
+      net_ssh_options
+    end
+
+    def hashed_ssh_config(config_string)
+      config_string.split("\n").map do |option|
+        fields = option.strip.split(' ', 2)
+        { :key => fields.first, :value => fields.last.gsub('"','') }
+      end
+    end
+
+    def options_map
+      { "HostName" => :host_name,
+        "User" => :user,
+        "Port" => :port,
+        "IdentityFile" => :keys,
+        "UserKnownHostsFile" => :user_known_hosts_file,
+        "IdentitiesOnly" => :keys_only,
+      }
+    end
+
+    def openwrt_release
+      @openwrt_release = File.read('openwrt_release') unless @openwrt_release
+      @openwrt_release
+    end
+  end
+end

data/Vagrantfile

@@ -0,0 +1,46 @@
+# -*- mode: ruby -*-
+# vi: ft=ruby
+
+# Adds a `openwrt_relase` custom fact, fetched from projects root
+$add_facter = <<SCRIPT
+  sudo mkdir -p /etc/facter/facts.d
+  sudo cp /vagrant/provisioning/scripts/openwrt_release.sh /etc/facter/facts.d
+SCRIPT
+
+Vagrant.configure("2") do |config|
+  config.vm.define :builder do |builder|
+    builder.vm.box = "ubuntu-base"
+
+    builder.vm.hostname = "openwrt.builders.derecom.it"
+
+    builder.vm.provider :virtualbox do |vbox|
+      vbox.customize ["modifyvm", :id, "--memory", "1024"]
+    end
+
+    builder.ssh.username = "ops"
+
+    builder.vm.provision :shell, :inline => $add_facter
+
+    builder.vm.provision :puppet do |puppet|
+      puppet.manifests_path = "provisioning/manifests"
+      puppet.manifest_file  = "builder.pp"
+    end
+  end
+
+  config.vm.define :provisioner do |provisioner|
+    provisioner.vm.box = "centos-base"
+
+    provisioner.vm.hostname = "openwrt.provisioning.derecom.it"
+
+    provisioner.vm.provider :virtualbox do |vbox|
+      vbox.customize ["modifyvm", :id, "--memory", "1024"]
+    end
+
+    provisioner.ssh.username = "ops"
+
+    provisioner.vm.provision :puppet do |puppet|
+      puppet.manifests_path = "provisioning/manifests"
+      puppet.manifest_file  = "provisioner.pp"
+    end
+  end
+end

data/flavors/README.md

@@ -0,0 +1,33 @@
+## Flavors Database
+This folder contains the flavors' database, which is a list of specific
+OpenWRT configurations that along with OpenWRT source code suffice to build
+complete working images. Each flavor might be used as a template for a specific
+model, as a template for a specific model **and** a specific feature (like 3G
+over USB support), or even as a complete deployment specification (including
+particular network configuration, keys, and so on).
+
+The database is defined in `flavors.yml`, which is basically a YAML file
+containing an array on its root.
+
+Each array entry contains the following fields:
+
+* `name`: the (short) name of the flavor.
+* `desc`: a description of the flavor, useful to clarify what it contains and
+          when to use it
+* `type`: can be `rootfs` or `ramdisk`, depending on if the final image is
+          meant to be installed on router's flash or to be loaded entirely in
+          ramdisk (as in the case of a firmware loaded from PXE - useful for
+          provisioning of OpenWRT itself)
+
+Then, there must exist a folder named after the name of the flavor,
+containing at least the following file:
+
+* `dot_config`: typically this will be the `.config` file created in the
+          source root folder after running `make menuconfig` (or similar).
+          Defines the kernel configuration and which packages are to be
+          installed in the final image.
+
+Also, the following optional sub-entries might be present:
+
+* `patches/`: contains the diff files to be applied as patches to OpenWRT's
+          source code prior to start the build process.