autoproj 1.2.6 → 1.3.0

data/doc/guide/src/autobuild.page

@@ -54,7 +54,7 @@ cmake_package "package_name" do |pkg|
 end
 {coderay}
 
-The doc\_dir assignment above is needed if the package installs its documentation
+The <tt>doc_dir</tt> assignment above is needed if the package installs its documentation
 elsewhere than "doc".
 
 Defining dependencies

data/doc/guide/src/customization.page

@@ -65,7 +65,7 @@ Removing packages from the build
 --------------------------------
 Instead of having to list all packages that you do want to build, it is possible
 to list the packages that you don't want to build. Simply list them in the
-<tt>exclude\_packages</tt> section of the manifest file. In the following example, all
+<tt>exclude_packages</tt> section of the manifest file. In the following example, all
 of the rubim.orocos package set will be built *but* the pocosim-log package.
 
 {coderay:: yaml}
@@ -81,7 +81,7 @@ Using packages that are already installed
 
 If some packages are already installed elsewhere, and you want to use that
 version instead of the one listed in the package sets, list them in the
-<tt>ignore\_packages</tt> section of the manifest. In the following example, the
+<tt>ignore_packages</tt> section of the manifest. In the following example, the
 <tt>orocos/rtt</tt> package will *not* be built by autoproj.
 
 {coderay:: yaml}
@@ -95,9 +95,9 @@ ignore_packages:
   - orocos/rtt
 {coderay}
 
-This differs from the <tt>exclude\_packages</tt> mechanism: packages listed in
-<tt>ignore\_packages</tt> are assumed to be present even though autoproj does not manage
-them, while packages in <tt>exclude\_packages</tt> are assumed to be absent and therefore
+This differs from the <tt>exclude_packages</tt> mechanism: packages listed in
+<tt>ignore_packages</tt> are assumed to be present even though autoproj does not manage
+them, while packages in <tt>exclude_packages</tt> are assumed to be absent and therefore
 issue an error if another package in the installation depend on them.
 
 Local overrides of version control information

data/doc/guide/src/error_messages.page

@@ -0,0 +1,16 @@
+---
+title: Understanding autoproj error messages
+sort_info: 800
+---
+
+I know nothing about a prepackaged package called 'XXXX', ...
+------------------------------------------
+_Short explanation_: an OS dependency is listed by one of the packages, but no
+definition exists in one of the osdeps files.
+
+_Long explanation_: XXXX is listed as an operating system dependency by one of
+the packages that you requested to build. Two solutions: (i) the package should
+*not* depend on it, and you should modify the package's manifest.xml file. (ii)
+the package should depend on it and you should list the OS package in [the
+relevant osdeps file](osdeps.html)
+

data/doc/guide/src/index.page

@@ -32,14 +32,16 @@ Components of an Autoproj installation
 -------------------------------------
 A autoproj installation is seeded by _package sets_. A package set is a local or remote
 directory in which there is:
- * autobuild scripts that describe what can be built and how it should be built.
-   These scripts an also list a set of configuration options that allow to
-   parametrize the build. In general, there should be only a limited number of
-   such options.
- * a source.yml file which describes the package set itself, and where the software
-   packages are located (what version control system and what URL).
- * optionally, a file that describe prepackaged dependencies that can be
-   installed by using the operating system package management system.
+
+ * a list of available packages, and information on how to build these packages.
+   This list, and the build information, are Ruby scripts that use the autobuild
+   API. [More ...](autobuild.html)
+ * version control information: where to get the source code for each of the
+   packages defined in the set.
+
+Then, an autoproj configuration is simply a list of package sets (i.e. a list of
+packages that are available to build), and a file that selects which packages
+should be built. This file is the <tt>manifest</tt>.
 
 Bootstrapping
 -------------
@@ -59,7 +61,28 @@ The canonical way is the following:
    ruby autoproj\_bootstrap
    {.cmdline}
 
- * follow the instructions printed by the script above :)
+ * follow the instructions printed by the script<tt>manifest</tt>.
+
+Additionally, if you are given a reference to a source code repository in which
+an autoproj configuration is stored (i.e. a directory in which a manifest is
+present), you can bootstrap this configuration directly:
+
+   wget http://doudou.github.com/autoproj/autoproj\_bootstrap <br />
+   ruby autoproj\_bootstrap VCS 
+   {.cmdline}
+
+For instance, to build all packages made available by the RubyInMotion project,
+do
+
+   wget http://doudou.github.com/autoproj/autoproj\_bootstrap <br />
+   ruby autoproj\_bootstrap git git://github.com/doudou/rubim.all.git
+   {.cmdline}
+
+Additional options can be given for the version control system. For instance,
+
+   wget http://doudou.github.com/autoproj/autoproj\_bootstrap <br />
+   ruby autoproj\_bootstrap git git://github.com/doudou/rubim.all.git branch=stable
+   {.cmdline}
 
 Software packages in Autoproj
 -----------------------------

data/doc/guide/src/source_yml.page

@@ -15,7 +15,7 @@ Starting a new package set
 Create a subdirectory in autoproj/ and add a source.yml file that looks like:
 
 {coderay:: yaml}
-name my_package_set_name
+name: my_package_set_name
 {coderay}
 
 Et voila ! You have a new empty package set
@@ -40,7 +40,13 @@ orogen_package "my/package" # for orogen packages
 The package name will be used to refer to this particular package later on --
 especially for version control definition. If subdirectories are used, like "my"
 in the above example, the package source will be checked out and built in the
-corresponding subdirectory.
+corresponding subdirectory. For instance, with
+
+{coderay:: ruby}
+cmake_package "drivers/hokuyo"
+{coderay}
+
+the hokuyo driver will _always_ be built in a <tt>drivers/</tt> subdirectory.
 
 Now that the package is declared, we need to add version control information to
 the source.yml file. This needs to be done in the version\_control section of
@@ -62,6 +68,14 @@ version_control:
       url: svn+ssh://svnhosting.com/blabla/trunk/my/package
 {coderay}
 
+For testing purposes, it is possible to tell autoproj to *not* take into account
+any VCS information:
+
+{coderay:: yaml}
+version_control:
+  - my/package: none
+{coderay}
+
 The remaining of this page is a more in-depth description of this process.
 
 Autobuild scripts
@@ -107,9 +121,14 @@ The name field gives a name for the set. It is arbitrary, but the guideline
 is to give a name that is java-like for namespaces, i.e. origin.name.
 
 The <tt>constants:</tt> section lists values that can be reused for different
-packages. Autoproj defines the HOME constant which is the user's home directory,
-and the PACKAGE constant that is the actual package name. We will see later that
-these values can also be configuration variables that are asked to the user.
+packages. Autoproj defines two constants:
+ 
+ * HOME is the user's home directory,
+ * PACKAGE is the actual package name, useful when using wildcards in package
+   names (see below)
+
+It is also possible to use configuration variables, that get asked to the user
+during the build (see below).
 
 Finally, the <tt>version_control:</tt> section describes how to import each
 software package. Its general format is:
@@ -119,11 +138,10 @@ package_name:
   url: repository_url
 {coderay}
 
-Where package\_name is a regular expression that matches the package name, i.e.
-use ".\*" to match all packages for instance. The package name is the one given
-to the <tt>blabla_package</tt> stanza in the autobuild file. In the above
-autobuild example, the package names are for instance "orocos/rtt" and
-"drivers/imu".
+Where package\_name is a regular expression that matches the package name (for
+instance, ".\*" will match all packages and "drivers/.\*" will match packages
+whose name starts with 'drivers'). The package name is the one given to the
+<tt>blabla_package</tt> stanza in the autobuild file.
 
 For the git importer, one of 'branch' or 'tag' options can be provided as well:
 {coderay:: yaml}
@@ -132,18 +150,18 @@ package_name:
   tag: tag_to_stick_to # it is branch OR tag
 {coderay}
 
-The options are applied in order, meaning that the toplevel options override the
-lower level ones. In general, one will have a ".\*" entry to give options for all
-packages, and then override for specific packages:
+The options are applied in order, meaning that the top entries will be overriden
+by the lower ones. In general, one will have a ".\*" entry to give options for
+all packages, and then override for specific packages:
 
 {coderay:: yaml}
 version_control:
-  - "modules/logger": # we don't follow master on this module
-      branch: imoby
-
   - .*: # common options for all packages
       type: git
       url: $ROOT_DIR/$PACKAGE.git
+
+  - "modules/logger": # we don't follow master on this module
+      branch: imoby
 {coderay}
 
 Interaction between package sets definition files
@@ -179,13 +197,16 @@ where the option\_type field can either be "string" or "boolean".
 
 Then, you can use the option\_name as an expansion in the source.yml file.
 
-For instance, with an init.rb looking like:
+For instance, at my lab we are using an share filesystem to store the git
+repositories. Our project's init.rb file has the following option definition:
 {coderay:: ruby}
 configuration_option "MOUNT_POINT", "string",
     :default => "$HOME/nfs",
     :doc => "mount point of the NFS server"
 {coderay}
 
+And the source.yml uses it with:
+
 {coderay:: yaml}
 version_control:
   ".*":

data/doc/guide/src/structure.page

@@ -34,7 +34,7 @@ generated to set up your shell for the use of the installed software.
 
 Listing and adding package sets
 -------------------------------
-Package sets are listed in the <tt>package\_sets</tt> section of
+Package sets are listed in the <tt>package_sets</tt> section of
 <tt>autoproj/manifest</tt> file. This section looks like this:
 
 {coderay:: yaml}

data/lib/autoproj/autobuild.rb

@@ -1,6 +1,12 @@
 require 'autobuild'
 require 'set'
 
+class Autobuild::Package
+    def autoproj_name
+        srcdir.gsub /^#{Regexp.quote(Autoproj.root_dir)}\//, ''
+    end
+end
+
 module Autoproj
     class Reporter < Autobuild::Reporter
         def error(error)

data/lib/autoproj/default.osdeps

@@ -1,4 +1,6 @@
 # The following definitions are needed to bootstrap autoproj
+none: ignore
+
 ruby:
     debian,ubuntu:
         - ruby1.8-dev
@@ -42,5 +44,9 @@ lsb_release:
     debian,ubuntu: lsb-release
     gentoo: sys-apps/lsb-release
     
+archive:
+    debian,ubuntu: [tar, unzip]
+    gentoo: [app-arch/tar, app-arch/unzip]
+
 # vim: expandtab
 

data/lib/autoproj/manifest.rb

@@ -95,7 +95,17 @@ module Autoproj
             Autobuild.send(type, url, options)
         end
 
-        def to_s; "#{type}:#{url}" end
+        def to_s 
+            if type == "none"
+                "none"
+            else
+                desc = "#{type}:#{url}"
+                if !options.empty?
+                    desc = "#{desc} #{options.map { |key, value| "#{key}=#{value}" }.join(" ")}"
+                end
+                desc
+            end
+        end
     end
 
     def self.vcs_definition_to_hash(spec)
@@ -126,7 +136,7 @@ module Autoproj
     # This method normalizes the three forms into a VCSDefinition object
     def self.normalize_vcs_definition(spec)
         spec = vcs_definition_to_hash(spec)
-        if !(spec[:type] && spec[:url])
+        if !(spec[:type] && (spec[:type] == 'none' || spec[:url]))
             raise ConfigError, "the source specification #{spec.inspect} misses either the VCS type or an URL"
         end
 
@@ -167,7 +177,9 @@ module Autoproj
         # True if this source defines nothing
         def empty?
             !source_definition['version_control'] &&
-                !each_package.find { true }
+                !each_package.find { true } &&
+                !File.exists?(File.join(local_dir, "overrides.rb")) &&
+                !File.exists?(File.join(local_dir, "init.rb"))
         end
 
         # The directory in which data for this source will be checked out
@@ -342,16 +354,24 @@ module Autoproj
                             spec = { $1 => spec }
                             name = name.gsub(/ \w+$/, '')
                         end
+
+                        if spec.respond_to?(:to_str)
+                            if spec == "none"
+                                spec = { :type => "none" }
+                            else
+                                raise ConfigError, "invalid VCS specification '#{name}: #{spec}'"
+                            end
+                        end
                     end
 
-                    if Regexp.new(name) =~ package_name
+                    if Regexp.new("^" + name) =~ package_name
                         vcs_spec = vcs_spec.merge(spec)
                     end
                 end
             end
 
             if !vcs_spec.empty?
-                expansions = Hash["PACKAGE" => package_name]
+                expansions = Hash["PACKAGE" => package_name, "PACKAGE_BASENAME" => File.basename(package_name)]
 
                 vcs_spec = expand(vcs_spec, expansions)
                 vcs_spec = Autoproj.vcs_definition_to_hash(vcs_spec)
@@ -407,7 +427,28 @@ module Autoproj
     end
 
     class Manifest
-        FakePackage = Struct.new :name, :srcdir
+        FakePackage = Struct.new :text_name, :name, :srcdir, :importer
+        class FakePackage
+            def autoproj_name; name end
+            def import
+                importer.import(self)
+            end
+
+            def progress(msg)
+                Autobuild.progress(msg % [text_name])
+            end
+
+            # Display a progress message, and later on update it with a progress
+            # value. %s in the string is replaced by the package name
+            def progress_with_value(msg)
+                Autobuild.progress_with_value(msg % [text_name])
+            end
+
+            def progress_value(value)
+                Autobuild.progress_value(value)
+            end
+        end
+
 	def self.load(file)
             begin
                 data = YAML.load(File.read(file))
@@ -478,7 +519,7 @@ module Autoproj
             #
             # And honestly I don't think someone will have 20 000 package sets
             done_something = false
-            each_source do |source| 
+            each_source(false) do |source| 
                 next if source_name && source.name != source_name
                 done_something = true
 
@@ -561,8 +602,6 @@ module Autoproj
                 return @sources.each(&block)
             end
 
-            return if !data['package_sets']
-
             # Load the local source first ...
             local = LocalSource.new
             if load_description
@@ -572,22 +611,27 @@ module Autoproj
             end
             if load_description
                 if !local.empty?
-                    yield(local)
-                    if load_description
-                        @sources = [local]
-                    end
+                    @sources = [local]
+                else
+                    @sources = []
                 end
-                @sources ||= []
             else
                 yield(local)
             end
 
+            return if !data['package_sets']
+
 	    data['package_sets'].each do |spec|
                 source = source_from_spec(spec, load_description)
                 if load_description
                     @sources << source
+                else
+                    yield(source)
                 end
-                yield(source)
+            end
+
+            if load_description
+                @sources.each(&block)
             end
         end
 
@@ -611,36 +655,70 @@ module Autoproj
             packages.each_value { |package, _| yield(package) }
         end
 
-        def self.update_remote_source(source)
-            importer     = source.vcs.create_autobuild_importer
-            return if !importer # updates  have been disabled 
-            fake_package = FakePackage.new(source.automatic_name, source.local_dir)
+        # The VCS object for the main configuration itself
+        attr_reader :vcs
+
+        def each_configuration_source
+            if !block_given?
+                return enum_for(:each_configuration_source)
+            end
 
-            importer.import(fake_package)
-        rescue Autobuild::ConfigException => e
-            raise ConfigError, e.message, e.backtrace
+            if vcs
+                yield(vcs, "autoproj main configuration", "autoproj_config", Autoproj.config_dir)
+            end
+
+            each_remote_source(false) do |source|
+                yield(source.vcs, source.name || source.vcs.url, source.automatic_name, source.local_dir)
+            end
+            self
         end
 
-        attr_reader :vcs
-        def self.import_whole_installation(vcs, into)
+        def self.create_autobuild_package(vcs, text_name, pkg_name, into)
             importer     = vcs.create_autobuild_importer
-            return if !importer # updates  have been disabled 
-            fake_package = FakePackage.new('autoproj main configuration', into)
-            importer.import(fake_package)
+            return if !importer # updates have been disabled by using the 'none' type
+
+            fake_package = FakePackage.new(text_name, pkg_name, into)
+            fake_package.importer = importer
+            fake_package
+
+        rescue Autobuild::ConfigException => e
+            raise ConfigError, "cannot import #{name}: #{e.message}", e.backtrace
+        end
+
+        def self.update_source(vcs, text_name, pkg_name, into)
+            fake_package = create_autobuild_package(vcs, text_name, pkg_name, into)
+            fake_package.import
+
         rescue Autobuild::ConfigException => e
-            raise ConfigError, "cannot import autoproj configuration: #{e.message}", e.backtrace
+            raise ConfigError, "cannot import #{name}: #{e.message}", e.backtrace
         end
 
         def update_yourself
-            Manifest.import_whole_installation(vcs, Autoproj.config_dir)
+            Manifest.update_source(vcs, "autoproj main configuration", "autoproj_conf", Autoproj.config_dir)
         end
 
         def update_remote_sources
             # Iterate on the remote sources, without loading the source.yml
             # file (we're not ready for that yet)
             each_remote_source(false) do |source|
-                Manifest.update_remote_source(source)
+                Manifest.update_source(source.vcs, source.name || source.vcs.url, source.automatic_name, source.local_dir)
+            end
+        end
+
+        def importer_definition_for(package_name, package_source = nil)
+            if !package_source
+                _, package_source, _ = packages.values.find { |pkg, pkg_source, _| pkg.name == package_name }
+            end
+
+            each_source.each do |source|
+                vcs = source.importer_definition_for(package_name)
+                if vcs
+                    return vcs
+                elsif package_source.name == source.name
+                    return
+                end
             end
+            nil
         end
 
         # Sets up the package importers based on the information listed in
@@ -661,14 +739,7 @@ module Autoproj
         #    by S1
         def load_importers
             packages.each_value do |package, package_source, package_source_file|
-                vcs = each_source.find do |source|
-                    vcs = source.importer_definition_for(package.name)
-                    if vcs
-                        break(vcs)
-                    elsif package_source.name == source.name
-                        break
-                    end
-                end
+                vcs = importer_definition_for(package.name, package_source)
 
                 if vcs
                     Autoproj.add_build_system_dependency vcs.type
@@ -728,7 +799,7 @@ module Autoproj
 
         # Looks into the layout setup in the manifest, and yields each layout
         # and sublayout in order
-        def each_package_set(selection, layout_name = '/', layout_def = data['layout'], &block)
+        def each_package_set(selection = nil, layout_name = '/', layout_def = data['layout'], &block)
             if !layout_def
                 yield(layout_name, default_packages, default_packages)
                 return nil
@@ -756,6 +827,36 @@ module Autoproj
             end
         end
 
+        def in_sublayout(name, packages)
+            srcdir  = File.join(Autoproj.root_dir, name)
+            prefix  = File.join(Autoproj.build_dir, name)
+            logdir  = File.join(prefix, "log")
+            Autobuild.logdir = logdir
+            packages.each do |pkg_name|
+                pkg = Autobuild::Package[pkg_name]
+                pkg.srcdir = File.join(srcdir, pkg_name)
+                pkg.prefix = prefix
+                pkg.logdir = logdir
+            end
+
+            yield
+        end
+
+        def handle_enabled_packages(selected_packages)
+            handled_packages = Set.new
+            each_package_set(selected_packages) do |name, packages, enabled_packages|
+                packages         -= handled_packages
+                enabled_packages -= handled_packages
+
+                in_sublayout(name, packages) do
+                    if !packages.empty?
+                        yield(name, packages, enabled_packages)
+                    end
+                end
+                handled_packages |= packages
+            end
+        end
+
         def default_packages
             names = if layout = data['layout']
                         layout_packages(layout, true)
@@ -821,16 +922,20 @@ module Autoproj
         end
 
         def install_os_dependencies(packages)
-            osdeps = known_os_packages
 
             required_os_packages = Set.new
+            package_os_deps = Hash.new { |h, k| h[k] = Array.new }
             packages.each do |pkg_name|
                 if manifest = package_manifests[pkg_name]
-                    required_os_packages |= manifest.each_os_dependency.to_set
+                    pkg_osdeps = manifest.each_os_dependency.to_set
+                    pkg_osdeps.each do |osdep_name|
+                        package_os_deps[osdep_name] << pkg_name
+                    end
+                    required_os_packages |= pkg_osdeps
                 end
             end
 
-            osdeps.install(required_os_packages)
+            known_os_packages.install(required_os_packages, package_os_deps)
         end
 
         # Package selection can be done in three ways: