wilbur 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 417d87a72995963bd253992473a7187df70a3a9d
4
+ data.tar.gz: ea7e93c56c213dac1aa3516ab3945e6717c05345
5
+ SHA512:
6
+ metadata.gz: 74b54c9428e99513ef221cf1a59b2895a681c9fb18011163cc5547da8c65f4bd2db8b44c9ab1ae9ef61766585fda837d01c592a5a0c56f677b58fd66defe6eac
7
+ data.tar.gz: 2505db8a85a9dbb82820fb9326a2570a28f3ee074702f0a3f836e080709967c6269ee0f9ba8b353a5b78406a21665ea9d66065d648ef1bd6521edff2354f2147
data/.gitignore ADDED
@@ -0,0 +1,8 @@
1
+ .DS_Store
2
+ *.swp
3
+ *.swo
4
+ .bundle
5
+ .vagrant
6
+ pkg
7
+ vendor
8
+ Gemfile.lock
data/.ruby-version ADDED
@@ -0,0 +1 @@
1
+ 2.0.0-p195
data/Gemfile ADDED
@@ -0,0 +1,5 @@
1
+ source "https://rubygems.org"
2
+
3
+ gemspec
4
+
5
+ gem "travis"
data/README.md ADDED
@@ -0,0 +1,93 @@
1
+ # Wilbur: Your OpenWRT Build Assistant
2
+
3
+ This project provides a comprehensive infrastructure for building and
4
+ provisioning OpenWRT images easily and in an automated fashion.
5
+
6
+ For more information about OpenWRT, please check http://openwrt.org
7
+
8
+ ## Project structure
9
+ * `openwrt_release`: here you specify the name of the release targeted for
10
+ build and deployment
11
+ * `flavors/`: contains the different kernel and rootfs configurations used
12
+ to build working OpenWRT images
13
+ * `flavors.yml`: serves as a db of flavor configurations. Used to provide
14
+ more understanding over each configuration and for referencing them more
15
+ easily by the user.
16
+ * `<flavor_name>/`: contains all files related to the specific flavor
17
+ * `dot_config`: Buildroot configuration file used to specify kernel
18
+ configuration and packages to compile/install in the final image
19
+ * `patches/`: contains all diff patches to be applied to the OpenWRT's
20
+ source code prior to building it
21
+ * `provisioning/`: contains the resources used to provision project's virtual
22
+ machines
23
+ * `manifests/`: contains the Puppet manifests used to bring up the building and
24
+ provisioning VMs
25
+ * `scripts/`: contains the scripts used with Vagrant to aid Puppet
26
+ provisioning
27
+
28
+ ## Usage
29
+
30
+ At the moment, all facilites are provided as **Thor**
31
+ commands. Also, **Bundler** is used to contain the project in its own folder.
32
+
33
+ To start using the builder, just let Bundler do its work:
34
+
35
+ `bundle install --path vendor/bundle`
36
+
37
+ Then, to see a list of the available commands, run:
38
+
39
+ `bundle exec thor -T`
40
+
41
+ To see a description of the use cases supported by the project, read on this
42
+ doc file.
43
+
44
+ ### Declaring OpenWRT release to build
45
+ In the project's root folder there's a file named `openwrt_release`. This file
46
+ must contain a single line with the name of the specific OpenWRT release
47
+ you want to build (e.g. `attitude_adjustment`). If you want to build bleeding
48
+ edge, just set it to `openwrt`.
49
+
50
+ #### Internals
51
+ The name specified in `openwrt_release` is used to build the URL of the source
52
+ repository to clone locally (so format matters). Basically, source will be fetched
53
+ from
54
+
55
+ `git://nbd.name/<openwrt_release>.git`
56
+
57
+ To provide this same name to Puppet during provisioning, some
58
+ "pre-provisioning" is done prior to running Puppet; during this step, a custom
59
+ executable fact will be added to the **building machine**, named
60
+ `openwrt_release`, that will read the content of the file in the shared project
61
+ folder. This way, you can specify the release you want to use once and not
62
+ worry about synchronizing names between building app and provisioning manifest.
63
+
64
+ ### Building a specific flavor
65
+ You can easily start a build by invoking the `build` task:
66
+
67
+ `bundle exec thor build <flavor>`
68
+
69
+ To obtain a list of available flavors, you can ask thor:
70
+
71
+ `bundle exec thor flavor_list`
72
+
73
+ This will return the list of available flavors. Each row will be in the form:
74
+
75
+ `<flavor name> <flavor description>`
76
+
77
+ The description is there to clarify what the flavor's purpose is; you can just
78
+ pick the name and pass it over to the `build` task.
79
+
80
+ Also, additional options are available for the `build` command:
81
+
82
+ * `--verbose`: let the compilation process be verbose about what's doing
83
+ (useful for debugging)
84
+ * `--nuke`: prior to start compilation, delete **everything** that is not bare
85
+ source code. _Note that in standard operation mode, build command already
86
+ wipes the output of previous run(s), so in normal circumstances this flag is
87
+ not needed. You may need it if you mess with the configuration and want to
88
+ restart with a clean environment. For more information, have a look at
89
+ [`make distclean` Buildroot command](http://wiki.openwrt.org/doc/howto/build)._
90
+
91
+ **NOTE**: At the moment, when the build process ends, nothing is done to gather
92
+ the produced files. For now, you have to ssh into the **builder** box and find
93
+ the kernel and rootfs under `attitude_adjustment/bin/<arch>/`.
data/Rakefile ADDED
@@ -0,0 +1,16 @@
1
+ require 'rake/testtask'
2
+ require 'wilbur/version'
3
+
4
+ namespace :gem do
5
+ require 'bundler/gem_tasks'
6
+ end
7
+
8
+ namespace :test do
9
+ Rake::TestTask.new(:end_to_end) do |t|
10
+ t.verbose = true
11
+ t.libs << "lib"
12
+ t.libs << "test"
13
+ t.test_files = FileList["test/end_to_end/**/*_test.rb"]
14
+ #t.ruby_opts = ["-w"]
15
+ end
16
+ end
data/Thorfile ADDED
@@ -0,0 +1,145 @@
1
+ # -*- mode: ruby -*-
2
+ # vim: ft=ruby
3
+
4
+ require 'thor'
5
+ require 'net/ssh'
6
+
7
+ class Default < Thor
8
+ desc "build FLAVOR", "builds the specified OpenWRT flavor on the building machine"
9
+ long_desc <<-EOS
10
+ Builds the specified OpenWRT flavor on the building node.
11
+
12
+ This means compiling the kernel and generating one or more OpenWRT image
13
+ files.
14
+ \x5The kernel will be configured with the options specified in the flavor
15
+ configuration file; in the same file (which is a standard kernel dot file
16
+ suitable for OpenWRT Buildroot) will be described which packages are to be compiled
17
+ and which ones are to be installed in the preconfigured image.
18
+
19
+ You can get a list of available flavors by running:
20
+ \x5
21
+ \x5\t thor flavor_list
22
+
23
+ The name specified in the first column can be used as an argument for the
24
+ build command.
25
+ EOS
26
+ option :verbose, :type => :boolean, :aliases => :v
27
+ option :nuke, :type => :boolean, :aliases => :n
28
+ def build(flavor)
29
+ if flavors.select { |f| f["name"] == flavor}.empty?
30
+ puts "No flavor found matching `#{flavor}`"
31
+ exit 1
32
+ end
33
+
34
+ if not Dir.exists?(File.join('flavors', flavor))
35
+ puts <<-EOS
36
+ Sorry, no directory flavors/#{flavor} exist. Please read how flavors
37
+ management work and fix your configuration before proceeding.
38
+ EOS
39
+ exit 1
40
+ end
41
+
42
+ dot_config = File.join(flavor, 'dot_config')
43
+ if not File.exists?(File.join('flavors', dot_config))
44
+ puts <<-EOS
45
+ Sorry, flavors/#{flavor}/dot_config file doesn't exist. Please read how flavors
46
+ management work and fix your configuration before proceeding.
47
+ EOS
48
+ exit 1
49
+ end
50
+
51
+ verbose = options[:verbose] ? "V=s" : ""
52
+
53
+ invoke :prepare_builder, [], []
54
+
55
+ Net::SSH.start('', nil, ssh_options_for(BUILDER_VM)) do |builder|
56
+ builder.exec! "cd #{openwrt_release}; make distclean" if options[:nuke]
57
+
58
+ puts "Preparing building environment"
59
+ builder.exec! "
60
+ cd #{openwrt_release}
61
+ git pull
62
+ ./scripts/feeds update -a
63
+ ./scripts/feeds install -a" do |channel, stream, data|
64
+ $stdout << data
65
+ end
66
+
67
+ puts "Syncing OpenWRT dot config file"
68
+ builder.exec! "cp #{File.join(VAGRANT_SHARE_ROOT, 'flavors', dot_config)} #{File.join(openwrt_release, '.config')}"
69
+
70
+ puts "Starting image build process"
71
+ builder.exec! "
72
+ cd #{openwrt_release}
73
+ make clean
74
+ make #{verbose}" do |channel, stream, data|
75
+ $stdout << data
76
+ end
77
+
78
+ puts "Build finished."
79
+ end
80
+ end
81
+
82
+ desc "prepare_builder", "prepares builder VM for subsequent use"
83
+ def prepare_builder
84
+ case `vagrant status #{BUILDER_VM}`
85
+ when /vm is running/i then ['provision']
86
+ when /to resume this vm/i then ['up', 'provision']
87
+ else ['up']
88
+ end.each do |cmd| system 'vagrant', cmd, BUILDER_VM end
89
+ end
90
+
91
+ desc "flavor_list", "list available configurations for kernel compilation and rootfs configuration"
92
+ def flavor_list
93
+ flavors.each do |config|
94
+ puts "#{config["name"]} : #{config["desc"]}"
95
+ end
96
+ rescue Exception => e
97
+ puts e.message
98
+ exit 1
99
+ end
100
+
101
+ no_commands do
102
+ BUILDER_VM = 'builder'.freeze
103
+ VAGRANT_SHARE_ROOT = '/vagrant'.freeze
104
+
105
+ def flavors
106
+ YAML::load_file("flavors/flavors.yml")
107
+ end
108
+
109
+ def ssh_options_for(host)
110
+ ssh_config = `vagrant ssh-config #{host}`
111
+
112
+ return if ssh_config.empty?
113
+
114
+ net_ssh_options = {}
115
+ hashed_ssh_config(ssh_config).each do |option|
116
+ param = options_map[option[:key]]
117
+ net_ssh_options[param] = option[:value] if param
118
+ end
119
+
120
+ net_ssh_options
121
+ end
122
+
123
+ def hashed_ssh_config(config_string)
124
+ config_string.split("\n").map do |option|
125
+ fields = option.strip.split(' ', 2)
126
+ { :key => fields.first, :value => fields.last.gsub('"','') }
127
+ end
128
+ end
129
+
130
+ def options_map
131
+ { "HostName" => :host_name,
132
+ "User" => :user,
133
+ "Port" => :port,
134
+ "IdentityFile" => :keys,
135
+ "UserKnownHostsFile" => :user_known_hosts_file,
136
+ "IdentitiesOnly" => :keys_only,
137
+ }
138
+ end
139
+
140
+ def openwrt_release
141
+ @openwrt_release = File.read('openwrt_release') unless @openwrt_release
142
+ @openwrt_release
143
+ end
144
+ end
145
+ end
data/Vagrantfile ADDED
@@ -0,0 +1,46 @@
1
+ # -*- mode: ruby -*-
2
+ # vi: ft=ruby
3
+
4
+ # Adds a `openwrt_relase` custom fact, fetched from projects root
5
+ $add_facter = <<SCRIPT
6
+ sudo mkdir -p /etc/facter/facts.d
7
+ sudo cp /vagrant/provisioning/scripts/openwrt_release.sh /etc/facter/facts.d
8
+ SCRIPT
9
+
10
+ Vagrant.configure("2") do |config|
11
+ config.vm.define :builder do |builder|
12
+ builder.vm.box = "ubuntu-base"
13
+
14
+ builder.vm.hostname = "openwrt.builders.derecom.it"
15
+
16
+ builder.vm.provider :virtualbox do |vbox|
17
+ vbox.customize ["modifyvm", :id, "--memory", "1024"]
18
+ end
19
+
20
+ builder.ssh.username = "ops"
21
+
22
+ builder.vm.provision :shell, :inline => $add_facter
23
+
24
+ builder.vm.provision :puppet do |puppet|
25
+ puppet.manifests_path = "provisioning/manifests"
26
+ puppet.manifest_file = "builder.pp"
27
+ end
28
+ end
29
+
30
+ config.vm.define :provisioner do |provisioner|
31
+ provisioner.vm.box = "centos-base"
32
+
33
+ provisioner.vm.hostname = "openwrt.provisioning.derecom.it"
34
+
35
+ provisioner.vm.provider :virtualbox do |vbox|
36
+ vbox.customize ["modifyvm", :id, "--memory", "1024"]
37
+ end
38
+
39
+ provisioner.ssh.username = "ops"
40
+
41
+ provisioner.vm.provision :puppet do |puppet|
42
+ puppet.manifests_path = "provisioning/manifests"
43
+ puppet.manifest_file = "provisioner.pp"
44
+ end
45
+ end
46
+ end
data/flavors/README.md ADDED
@@ -0,0 +1,33 @@
1
+ ## Flavors Database
2
+ This folder contains the flavors' database, which is a list of specific
3
+ OpenWRT configurations that along with OpenWRT source code suffice to build
4
+ complete working images. Each flavor might be used as a template for a specific
5
+ model, as a template for a specific model **and** a specific feature (like 3G
6
+ over USB support), or even as a complete deployment specification (including
7
+ particular network configuration, keys, and so on).
8
+
9
+ The database is defined in `flavors.yml`, which is basically a YAML file
10
+ containing an array on its root.
11
+
12
+ Each array entry contains the following fields:
13
+
14
+ * `name`: the (short) name of the flavor.
15
+ * `desc`: a description of the flavor, useful to clarify what it contains and
16
+ when to use it
17
+ * `type`: can be `rootfs` or `ramdisk`, depending on if the final image is
18
+ meant to be installed on router's flash or to be loaded entirely in
19
+ ramdisk (as in the case of a firmware loaded from PXE - useful for
20
+ provisioning of OpenWRT itself)
21
+
22
+ Then, there must exist a folder named after the name of the flavor,
23
+ containing at least the following file:
24
+
25
+ * `dot_config`: typically this will be the `.config` file created in the
26
+ source root folder after running `make menuconfig` (or similar).
27
+ Defines the kernel configuration and which packages are to be
28
+ installed in the final image.
29
+
30
+ Also, the following optional sub-entries might be present:
31
+
32
+ * `patches/`: contains the diff files to be applied as patches to OpenWRT's
33
+ source code prior to start the build process.