autoproj 1.0.0

data/doc/guide/src/source_yml.page

@@ -0,0 +1,139 @@
+---
+title: Creating a package set
+sort_info: 100
+---
+A package set is made of three things:
+
+ * the description file (source.yml)
+ * an optional initialization script (init.rb)
+ * autobuild scripts (\*.autobuild)
+
+Autobuild scripts
+-----------------
+The autobuild scripts lists all the packages defined by this set. It is a
+Ruby script (but you don't need to know Ruby to write one). In its most simple
+form, it looks like:
+
+{coderay:: ruby}
+cmake_package "orocos/rtt"
+autotools_package "drivers/imu"
+{coderay}
+
+The above snippet defines two packages. The first one uses CMake as a build
+system and will be installed in the <tt>orocos/rtt</tt> subdirectory of the
+autoproj installation. The second one is an autotools package. There is, for
+now, only support for these two build systems, and for Ruby packages. Package
+definitions can be tweaked quite a lot, including the ability to generate
+documentation. See [the next page](autobuild.html) for more information on how to write autobuild
+scripts.
+
+source.yml
+----------
+The source.yml file gives generic information on the package set itself (most
+importantly its name), and version control information (i.e. where to get the
+packages). It is a YAML file, and looks like:
+
+{coderay:: yaml}
+name: dfki.orocos
+constants:
+  ROOT_DIR: $HOME/share
+  
+version_control:
+  - "modules/.*":
+      type: git
+      url: $ROOT_DIR/$PACKAGE.git
+  - "drivers/.*":
+      type: svn
+      url: svn+ssh://rlbsvn.informatik.uni-bremen.de/trunk/$PACKAGE.git
+{coderay}
+
+The name field gives a name for the set. It is arbitrary, but the guideline
+is to give a name that is java-like for namespaces, i.e. origin.name.
+
+The <tt>constants:</tt> section lists values that can be reused for different
+packages. Autoproj defines the HOME constant which is the user's home directory,
+and the PACKAGE constant that is the actual package name. We will see later that
+these values can also be configuration variables that are asked to the user.
+
+Finally, the <tt>version_control:</tt> section describes how to import each
+software package. Its general format is:
+{coderay:: yaml}
+package_name:
+  type: version_control_type # git, svn, cvs, darcs
+  url: repository_url
+{coderay}
+
+Where package\_name is a regular expression that matches the package name, i.e.
+use ".\*" to match all packages for instance. The package name is the one given
+to the <tt>blabla_package</tt> stanza in the autobuild file. In the above
+autobuild example, the package names are for instance "orocos/rtt" and
+"drivers/imu".
+
+For the git importer, one of 'branch' or 'tag' options can be provided as well:
+{coderay:: yaml}
+package_name:
+  branch: branch_to_track
+  tag: tag_to_stick_to # it is branch OR tag
+{coderay}
+
+The options are applied in order, meaning that the toplevel options override the
+lower level ones. In general, one will have a ".\*" entry to give options for all
+packages, and then override for specific packages:
+
+{coderay:: yaml}
+version_control:
+  - "modules/logger": # we don't follow master on this module
+      branch: imoby
+
+  - .*: # common options for all packages
+      type: git
+      url: $ROOT_DIR/$PACKAGE.git
+{coderay}
+
+Interaction between package sets definition files
+-------------------------------------------
+When multiple package sets are used, it is possible to override the version
+control definitions in low-priority sets with a higher priority one. Autoproj
+has the following rules when searching for version control information:
+
+ * autoproj looks at the package sets *in the order they appear in the
+   installation manifest*
+ * autoproj searches a relevant version\_control field, and stops at the first
+   package set that has one.
+ * autoproj *stops searching* at the package set that defines the package.
+   Consequence: this set *must* have a version\_control field for it, and an
+   error will be issued otherwise.
+
+Using configuration options
+---------------------------
+autoproj offers a configuration system that allows the user to tweak the build
+to its needs. If the version control definitions depend on such configuration
+options (as, for instance, because you want to parametrize the importer URLs),
+then create an init.rb file next to the source.yml file, and add lines looking
+like:
+
+{coderay:: ruby}
+configuration_option "option_name", "option_type",
+    :default => "default_value",
+    :values => ["set", "of", "possible", "values"],
+    :doc => "description of the option"
+{coderay}
+
+where the option\_type field can either be "string" or "boolean".
+
+Then, you can use the option\_name as an expansion in the source.yml file.
+
+For instance, with an init.rb looking like:
+{coderay:: ruby}
+configuration_option "MOUNT_POINT", "string",
+    :default => "$HOME/nfs",
+    :doc => "mount point of the NFS server"
+{coderay}
+
+{coderay:: yaml}
+version_control:
+  ".*":
+    url: $MOUNT_POINT/git/$PACKAGE.git
+    type: git
+{coderay}
+

data/doc/guide/src/structure.page

@@ -0,0 +1,153 @@
+---
+title: Managing autoproj installations
+sort_info: 50
+---
+
+This page describes the structure of a autoproj installation, and describes how
+to manage one. See [the introduction](index.page) for the bootstrapping process.
+
+Structure
+---------
+
+A autoproj installation is characterized by a autoproj/ directory in which all
+autoproj-related configuration is saved. The contained files are as follows:
+
+ * autoproj/manifest: list of available package sets, layout of the
+   installation. See "Management" below.
+ * autoproj/remotes/\*: package sets that are imported from a remote version
+   control system
+ * autoproj/\*/: local sets, i.e. sets that have not been imported from a remote
+   version control system.
+
+The build is done in two steps:
+
+ * each package is being built in a <tt>build</tt> subdirectory of the package's
+   source (<tt>package_directory/build/</tt>)
+ * it is then installed in the build/ directory at the toplevel of the autoproj
+   installation
+
+Moreover, the <tt>build/log</tt> directory contains the output of all commands
+that have been run during the build. Finally, a <tt>env.sh</tt> script is
+generated to set up your shell for the use of the installed software.
+
+Toplevel manifest
+-----------------
+The <tt>autoproj/manifest</tt> file is a yaml file which describes the package
+sets that are available, and how packages are organized in the installation. It
+looks like this:
+
+{coderay:: yaml}
+layout:
+  - autoproj.orocos
+  - asguard:
+    - dfki.imoby
+    
+package_sets:
+  - imoby
+  - type: git
+    url:  git://github.com/doudou/autoproj-orocos.git
+{coderay}
+
+The first section, the <tt>layout</tt> section, lists the packages or package
+sets we want to have in this installation.  It is a layout, meaning that some packages
+can be built and installed in a subdirectory of the main installation
+(see "Layouts and dependencies" below). Packages are referred to by the name they
+have in the autobuild file (see [this page for more details](source_yml.hml)).
+Package sets are referred to by the name given in the [set's <tt>source.yml</tt>
+file](source_yml.html), and are interpreted as "build all packages of the given
+set".
+
+The second section, the <tt>package_sets</tt> section, lists both local and remote
+sets that are available to this installation. Local sets are
+subdirectories of the <tt>autoproj/</tt> directory: for instance, in the above
+example, autoproj will look at the <tt>autoproj/imoby/</tt> directory. Remote
+sets are taken from remote version control systems. Its general format is:
+
+{coderay:: yaml}
+  - type: version_control_type # git, svn, cvs, darcs
+    url: repository_url
+{coderay}
+
+For the git importer, one of 'branch' or 'tag' options can be provided as well:
+{coderay:: yaml}
+  - type: version_control_type # git, svn, cvs, darcs
+    url: repository_url
+    branch: branch_to_track
+    tag: tag_to_stick_to # it is branch OR tag
+{coderay}
+
+Management
+----------
+
+To update and build a autoproj installation, simply do:
+
+autoproj build
+{.commandline}
+
+It will ask the value of newly defined configuration options, import (or update)
+code hosted remotely, and build it. autoproj will *not* ask you again about the
+configuration questions you already answered, so if you want to change them, do:
+
+autoproj build --reconfigure
+{.commandline}
+
+Alternatively, you can edit the autoproj/config.yml file directly.
+
+If you are in a disconnected environment (i.e. no access to remote
+repositories), use the <tt>--no-update</tt> option to skip the update phase.
+autoproj will still have to checkout new packages, though:
+
+autoproj build --no-update
+{.commandline}
+
+If, on the other hand, you only want to update the sources, do
+
+autoproj update
+{.commandline}
+
+To add a new set, one edits the <tt>autoproj/manifest</tt> file and adds it
+there. Then, simply starting the build will update everything and rebuild what
+is needed.
+
+Documentation is generated only when asked explicitely:
+
+autoproj doc
+{.commandline}
+
+Building packages selectively on the command line
+-------------------------------------------------
+
+All the build-related commands given above (i.e. build, doc, and update) can be
+given a package name or a name used as a subdirectory in the layout section of
+the manifest.
+
+In the first case, only the package and the dependencies _that are on the same
+level on the installation layout_ are built. It means that with the following
+layout:
+
+{coderay:: yaml}
+layout:
+  - typelib
+  - asguard:
+    - modules/base
+    - modules/logger
+{coderay}
+
+If the command line is
+
+autoproj build modules/logger
+{.cmdline}
+
+then only modules/logger and modules/base will be built -- assuming
+modules/logger depends on modules/base -- but typelib will be left alone
+_regardless of its state_. It may speed up the build process tremendously, but
+also may generate errors if other packages needed to be updated.
+
+Idem, if the command line is:
+
+autoproj build asguard
+{.cmdline}
+
+then all packages or asguard/ are built _but none of the dependencies that are
+defined in other places in the layout_.
+

data/lib/autoproj.rb

@@ -0,0 +1,23 @@
+module Autoproj
+    class ConfigError < RuntimeError; end
+    class InternalError < RuntimeError; end
+end
+
+require "enumerator"
+require 'autoproj/manifest'
+require 'autoproj/osdeps'
+require 'autoproj/system'
+require 'autoproj/options'
+require 'logger'
+require 'utilrb/logger'
+
+module Autoproj
+    class << self
+        attr_reader :logger
+    end
+    @logger = Logger.new(STDOUT)
+    logger.level = Logger::WARN
+    logger.formatter = lambda { |severity, time, progname, msg| "#{severity}: #{msg}\n" }
+    extend Logger::Forward
+end
+

data/lib/autoproj/autobuild.rb

@@ -0,0 +1,173 @@
+require 'autobuild'
+require 'set'
+
+module Autoproj
+    class Reporter < Autobuild::Reporter
+        def error(error)
+            error_lines = error.to_s.split("\n")
+            STDERR.puts color("Build failed: #{error_lines.shift}", :bold, :red)
+            STDERR.puts error_lines.join("\n")
+        end
+        def success
+            STDERR.puts color("Build finished successfully at #{Time.now}", :bold, :green)
+            if Autobuild.post_success_message
+                puts Autobuild.post_success_message
+            end
+        end
+    end
+
+    def self.warn(message)
+        STDERR.puts Autoproj.console.color("  WARN: #{message}", :magenta)
+    end
+
+    @definition_files = Hash.new
+    @file_stack       = Array.new
+    class << self
+        attr_reader :definition_files
+    end
+
+    def self.package_name_from_options(spec)
+        if spec.kind_of?(Hash)
+            spec.to_a.first.first.to_str
+        else
+            spec.to_str
+        end
+    end
+
+    def self.current_file
+        @file_stack.last
+    end
+
+    def self.define(package_type, spec, &block)
+        package = Autobuild.send(package_type, spec, &block)
+        Autoproj.manifest.register_package package, *current_file
+    end
+
+    @loaded_autobuild_files = Set.new
+    def self.import_autobuild_file(source, path)
+        return if @loaded_autobuild_files.include?(path)
+
+        @file_stack.push([source, File.basename(path)])
+        load path
+        @loaded_autobuild_files << path
+
+    ensure
+        @file_stack.pop
+    end
+end
+
+def ruby_doc(pkg, target = 'doc')
+    pkg.doc_task do
+        pkg.doc_disabled unless File.file?('Rakefile')
+        Autobuild::Subprocess.run pkg.name, 'doc', 'rake', target
+    end
+
+end
+
+# Common setup for packages hosted on groupfiles/Autonomy
+def package_common(package_type, spec)
+    package_name = Autoproj.package_name_from_options(spec)
+
+    begin
+        Rake::Task[package_name]
+        Autoproj.warn "#{package_name} in #{Autoproj.current_file[0]} has been overriden in #{Autoproj.definition_source(package_name)}"
+    rescue
+    end
+
+    Autoproj.define(package_type, spec) do |pkg|
+        pkg.srcdir   = pkg.name
+        yield(pkg) if block_given?
+    end
+end
+
+def cmake_package(options, &block)
+    package_common(:cmake, options) do |pkg|
+        Autoproj.add_build_system_dependency 'cmake'
+        yield(pkg) if block_given?
+        unless pkg.has_doc?
+            pkg.with_doc do
+                doc_html = File.join('doc', 'html')
+                if File.directory? doc_html
+                    pkg.doc_dir = doc_html
+                end
+            end
+        end
+    end
+end
+
+# Use this method to import and build CMake packages that are hosted on the
+# groupfiles server, on the autonomy project folder
+def autotools_package(options, &block)
+    package_common(:autotools, options) do |pkg|
+        Autoproj.add_build_system_dependency 'autotools'
+        yield(pkg) if block_given?
+        unless pkg.has_doc?
+            pkg.with_doc do
+                doc_html = File.join('doc', 'html')
+                if File.directory? doc_html
+                    pkg.doc_dir = doc_html
+                end
+            end
+        end
+    end
+end
+
+def ruby_common(pkg)
+    def pkg.prepare
+        super
+        Autobuild.update_environment srcdir
+    end
+
+    pkg.post_install do
+        Autobuild.progress "  setting up Ruby package #{pkg.name}"
+        Autobuild.update_environment pkg.srcdir
+        if File.file?('Rakefile')
+            if File.directory?('ext')
+                Autobuild::Subprocess.run pkg.name, 'post-install', 'rake', 'setup'
+            end
+        end
+    end
+end
+
+def env_set(name, value)
+    Autoproj.env_set(name, value)
+end
+def env_add(name, value)
+    Autoproj.env_add(name, value)
+end
+
+def ruby_package(options)
+    package_common(:import, options) do |pkg|
+        class << pkg
+            attr_accessor :doc_target
+        end
+
+        ruby_common(pkg)
+        yield(pkg) if block_given?
+        unless pkg.has_doc?
+            ruby_doc(pkg, pkg.doc_target || 'redocs')
+        end
+    end
+end
+
+def orogen_package(options, &block)
+    package_common(:orogen, options) do |pkg|
+        yield(pkg) if block_given?
+    end
+end
+
+def source_package(options)
+    package_common(options) do |pkg|
+        pkg.srcdir   = pkg.name
+        yield(pkg) if block_given?
+    end
+end
+
+def configuration_option(*opts, &block)
+    Autoproj.configuration_option(*opts, &block)
+end
+
+def user_config(*opts)
+    Autoproj.user_config(*opts)
+end
+