autoproj 1.0.0

data/History.txt

@@ -0,0 +1,2 @@
+= Version 1.0
+ * initial release

data/Manifest.txt

@@ -0,0 +1,37 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+bin/autoproj
+doc/guide/config.yaml
+doc/guide/ext/init.rb
+doc/guide/ext/previous_next.rb
+doc/guide/ext/rdoc_links.rb
+doc/guide/src/autobuild.page
+doc/guide/src/autoproj_bootstrap
+doc/guide/src/default.css
+doc/guide/src/default.template
+doc/guide/src/htmldoc.metainfo
+doc/guide/src/images/bodybg.png
+doc/guide/src/images/contbg.png
+doc/guide/src/images/footerbg.png
+doc/guide/src/images/gradient1.png
+doc/guide/src/images/gradient2.png
+doc/guide/src/index.page
+doc/guide/src/source_yml.page
+doc/guide/src/structure.page
+lib/autoproj.rb
+lib/autoproj/autobuild.rb
+lib/autoproj/default.osdeps
+lib/autoproj/manifest.rb
+lib/autoproj/options.rb
+lib/autoproj/osdeps.rb
+lib/autoproj/system.rb
+lib/autoproj/version.rb
+samples/manifest
+samples/manifest.xml
+samples/osdeps.yml
+test/data/test_manifest/autoproj/local/local.autobuild
+test/data/test_manifest/autoproj/manifest
+test/test_debian.rb
+test/test_manifest.rb

data/README.txt

@@ -0,0 +1,79 @@
+What is Autoproj
+----------------
+Autoproj allows to easily install and maintain software that is under source
+code form (usually from a version control system). It has been designed to support a
+package-oriented development process, where each package can have its own
+version control repository (think "distributed version control"). It also
+provides an easy integration of the local operating system (Debian, Ubuntu,
+Fedora, maybe MacOSX at some point).
+
+This tool has been developped in the frame of the RubyInMotion project
+(http://sites.google.com/site/rubyinmotion), to install robotics-related
+software -- that is often bleeding edge. Unlike [the ROS build
+system](http://ros.org), it is not bound to one build system, one VCS and one
+integration framework. The philosophy behind autoproj
+is:
+ * supports both CMake and autotools, and can be adapted to other tools
+ * supports different VCS: cvs, svn, git, plain tarballs.
+ * software packages are plain packages, meaning that they can be built and
+   installed /outside/ an autoproj tree, and are not tied *at all* to the
+   autoproj build system.
+ * leverage the actual OS package management system. Right now, only Debian-like
+   systems (like Ubuntu) are supported, simply because it is the only one I have
+   access to.
+ * handle code generation properly
+
+It tries as much as possible to follow the lead of Willow Garage on the package
+specification. More specifically, the package manifest files are common between
+ROS package management and autoproj (more details in the following of this
+document).
+
+Components of an Autoproj installation
+-------------------------------------
+A autoproj installation is seeded by _package sets_. A package set is a local or remote
+directory in which there is:
+ * autobuild scripts that describe what can be built and how it should be built.
+   These scripts an also list a set of configuration options that allow to
+   parametrize the build. In general, there should be only a limited number of
+   such options.
+ * a source.yml file which describes the package set itself, and where the software
+   packages are located (what version control system and what URL).
+ * optionally, a file that describe prepackaged dependencies that can be
+   installed by using the operating system package management system.
+
+Bootstrapping
+-------------
+"Bootstrapping" means getting autoproj itself before it can work its magic ...
+The canonical way is the following:
+
+ * install Ruby by yourself. On Debian or Ubuntu, this is done with
+   done with
+
+   sudo apt-get install wget ruby
+   {.cmdline}
+
+ * then, [download this script](autoproj_bootstrap) *in the directory where
+   you want to create an autoproj installation*, and run it. This can be done with
+
+   wget http://doudou.github.com/autoproj/autoproj\_bootstrap <br />
+   ruby autoproj\_bootstrap
+   {.cmdline}
+
+ * follow the instructions printed by the script above :)
+
+Software packages in Autoproj
+-----------------------------
+In the realm of autoproj, a software package should be a self-contained build
+system, that could be built outside of an autoproj tree. In practice, it means
+that the package writer should leverage its build system (for instance, cmake)
+to discover if the package dependencies are installed, and what are the
+appropriate build options that should be given (for instance, include
+directories or library names).
+
+As a guideline, we recommend that inter-package dependencies are managed by
+using pkg-config.
+
+To describe the package, and more importantly to setup cross-package
+dependencies, an optional manifest file can be added. This manifest file is
+named manifest.xml. Its format is described later in this user's guide.
+

data/Rakefile

@@ -0,0 +1,84 @@
+$LOAD_PATH.unshift File.join(Dir.pwd, 'lib')
+
+begin
+    require 'hoe'
+    namespace 'dist' do
+        config = Hoe.spec 'autoproj' do
+            self.developer "Sylvain Joyeux", "sylvain.joyeux@dfki.de"
+
+            self.rubyforge_name = 'autobuild'
+            self.summary = 'Easy installation and management of robotics software'
+            self.description = paragraphs_of('README.txt', 0..1).join("\n\n")
+            self.changes     = paragraphs_of('History.txt', 0..1).join("\n\n")
+
+            extra_deps << 
+                ['autobuild',   '>= 1.3.0'] <<
+                ['rmail',   '>= 1.0.0'] <<
+                ['utilrb', '>= 1.3.3'] <<
+                ['nokogiri', '>= 1.3.3'] <<
+                ['highline', '>= 1.5.0']
+
+            extra_dev_deps <<
+                ['webgen', '>= 0.5.9'] <<
+                ['rdoc', '>= 2.4.0']
+        end
+
+        desc "generate the bootstrap script"
+        task 'bootstrap' do
+            osdeps_code = File.read(File.join(Dir.pwd, 'lib', 'autoproj', 'osdeps.rb'))
+            bootstrap_code = File.read(File.join(Dir.pwd, 'bin', 'autoproj_bootstrap.in')).
+                gsub('OSDEPS_CODE', osdeps_code)
+            File.open(File.join(Dir.pwd, 'doc', 'guide', 'src', 'autoproj_bootstrap'), 'w') do |io|
+                io.write bootstrap_code
+            end
+        end
+    end
+
+    # Define our own documentation handling. Rake.clear_tasks is defined by Hoe
+    Rake.clear_tasks(/dist:(re|clobber_|)docs/)
+
+rescue LoadError
+    STDERR.puts "cannot load the Hoe gem. Distribution is disabled"
+rescue Exception => e
+    STDERR.puts "cannot load the Hoe gem, or Hoe fails. Distribution is disabled"
+    STDERR.puts "error message is: #{e.message}"
+end
+
+do_doc = begin
+             require 'webgen/webgentask'
+             require 'rdoc/task'
+             true
+         rescue LoadError => e
+             STDERR.puts "ERROR: cannot load webgen and/or RDoc, documentation generation disabled"
+             STDERR.puts "ERROR:   #{e.message}"
+         end
+
+if do_doc
+    task 'doc' => 'doc:all'
+    task 'clobber_docs' => 'doc:clobber'
+    task 'redocs' do
+        Rake::Task['doc:clobber'].invoke
+        Rake::Task['doc'].invoke
+    end
+
+    namespace 'doc' do
+        task 'all' => %w{guide api}
+        task 'clobber' => 'clobber_guide'
+        Webgen::WebgenTask.new('guide') do |website|
+            website.clobber_outdir = true
+            website.directory = File.join(Dir.pwd, 'doc', 'guide')
+            website.config_block = lambda do |config|
+                config['output'] = ['Webgen::Output::FileSystem', File.join(Dir.pwd, 'doc', 'html')]
+            end
+        end
+        task 'guide' => 'dist:bootstrap'
+        RDoc::Task.new("api") do |rdoc|
+            rdoc.rdoc_dir = 'doc/html/api'
+            rdoc.title    = "oroGen"
+            rdoc.options << '--show-hash'
+            rdoc.rdoc_files.include('lib/**/*.rb')
+        end
+    end
+end
+
+

data/bin/autoproj

@@ -0,0 +1,465 @@
+#! /usr/bin/env ruby
+
+require 'autoproj'
+require 'autoproj/autobuild'
+require 'open-uri'
+
+require 'highline'
+include Autoproj
+
+InputError = Autoproj::InputError
+module Autoproj
+    @verbose = false
+    @console = HighLine.new
+
+    class << self
+        attr_accessor :verbose
+        attr_reader :console
+    end
+end
+
+# Some configuration variables
+mail_config = Hash.new
+only_update_sources = false
+only_do_status      = false
+no_os_deps = false
+debug = false
+
+# Parse the configuration options
+parser = OptionParser.new do |opts|
+    opts.banner = <<-EOBANNER
+autoproj mode [options]
+where 'mode' is one of:
+  build:  import, build and install all selected packages
+  doc:    generate and install documentation for packages that have some
+  update: only import/update packages, do not build them
+  status: displays the state of the packages w.r.t. their source VCS
+  bootstrap: starts a new autoproj installation. Usage:
+    autoproj bootstrap [manifest_url|source_vcs source_url opt1=value1 opt2=value2 ...]
+  
+  list-sources:   list all available sources
+  update-sources: update all remote sources, but do not build
+
+Additional options:
+    EOBANNER
+    opts.on("--reconfigure", "re-ask all configuration options (build mode only)") do
+        Autoproj.reconfigure = true
+    end
+    opts.on("--no-update", "do not update already checked-out sources (build mode only)") do
+        Autobuild.do_update = false
+    end
+
+    opts.on("--no-osdeps", "do not install prepackaged dependencies") do
+        no_os_deps = true
+    end
+
+    opts.on("--verbose", "verbose output") do
+        Autoproj.verbose  = true
+        Autobuild.verbose = true
+        Rake.application.options.trace = false
+    end
+    opts.on("--debug", "debugging output") do
+        Autoproj.verbose  = true
+        Autobuild.verbose = true
+        Rake.application.options.trace = true
+        debug = true
+    end
+    opts.on('--nice NICE', Integer, 'nice the subprocesses to the given value') do |value|
+        Autobuild.nice = value
+    end
+    opts.on("-h", "--help", "Show this message") do
+        puts opts
+        exit
+    end
+    opts.on("--mail-from EMAIL", String, "From: field of the sent mails") do |from_email|
+        mail[:from] = from_email
+    end
+    opts.on("--mail-to EMAILS", String, "comma-separated list of emails to which the reports should be sent") do |emails| 
+        mail[:to] ||= []
+        mail[:to] += emails.split(',')
+    end
+    opts.on("--mail-subject SUBJECT", String, "Subject: field of the sent mails") do |subject_email|
+        mail[:subject] = subject_email
+    end
+    opts.on("--mail-smtp HOSTNAME", String, " address of the mail server written as hostname[:port]") do |smtp|
+        raise "invalid SMTP specification #{smtp}" unless smtp =~ /^([^:]+)(?::(\d+))?$/
+            mail[:smtp] = $1
+        mail[:port] = Integer($2) if $2 && !$2.empty?
+    end
+    opts.on("--mail-only-errors", "send mail only on errors") do
+        mail[:only_errors] = true
+    end
+end
+
+args = ARGV.dup
+parser.parse!(args)
+mode     = args.shift
+selected_packages = args.dup
+
+def color(*args)
+    Autoproj.console.color(*args)
+end
+
+def do_status(packages)
+    console = Autoproj.console
+
+    Autobuild::Reporting.report do
+        last_was_in_sync = false
+        packages.sort.each do |pkg|
+            pkg = Autobuild::Package[pkg]
+
+            if !pkg.importer.respond_to?(:status)
+                # This importer does not support status display
+                STDERR.puts color("  this package's importer does not support status display", :bold, :red)
+                next
+            end
+
+            lines = []
+            status = pkg.importer.status(pkg)
+            if status.uncommitted_code
+                lines << color("  contains uncommitted modifications", :red)
+            end
+
+            case status.status
+            when Autobuild::Importer::Status::UP_TO_DATE
+                if !status.uncommitted_code
+                    if last_was_in_sync
+                        STDERR.print ", #{pkg.name}"
+                    else
+                        STDERR.print pkg.name
+                    end
+                    last_was_in_sync = true
+                    next
+                else
+                    lines << color("  local and remote are in sync", :green)
+                end
+            when Autobuild::Importer::Status::ADVANCED
+                lines << color("  local contains #{status.local_commits.size} commit that remote does not have:", :magenta)
+                status.local_commits.each do |line|
+                    lines << color("    #{line}", :magenta)
+                end
+            when Autobuild::Importer::Status::SIMPLE_UPDATE
+                lines << color("  remote contains #{status.remote_commits.size} commit that local does not have:", :magenta)
+                status.remote_commits.each do |line|
+                    lines << color("    #{line}", :magenta)
+                end
+            when Autobuild::Importer::Status::NEEDS_MERGE
+                lines << color("  local and remote have diverged with respectively #{status.local_commits.size} and #{status.remote_commits.size} commits each", :magenta)
+                lines << "  -- local commits --"
+                status.local_commits.each do |line|
+                    lines << color("   #{line}", :magenta)
+                end
+                lines << "  -- remote commits --"
+                status.remote_commits.each do |line|
+                    lines << color("   #{line}", :magenta)
+                end
+            end
+
+            if last_was_in_sync
+                STDERR.puts color(": local and remote are in sync", :green)
+            end
+
+            last_was_in_sync = false
+            STDERR.print "#{pkg.name}:"
+
+            if lines.size == 1
+                STDERR.puts lines.first
+            else
+                STDERR.puts
+                STDERR.puts lines.join("\n")
+            end
+        end
+        if last_was_in_sync
+            STDERR.puts color(": local and remote are in sync", :green)
+        end
+    end
+end
+
+
+def do_bootstrap(*args)
+    if args.empty? # no argument, simply add a manifest template
+        sample_dir = File.expand_path(File.join("..", "samples"), File.dirname(__FILE__))
+        FileUtils.mkdir_p "autoproj"
+        manifest_data = File.read(File.join(sample_dir, "manifest"))
+
+    elsif args.size == 1 # must be a manifest file
+        manifest_url = args.first
+        STDERR.puts color("autoproj: downloading manifest file #{manifest_url}", :bold)
+        manifest_data = open(manifest_url) { |file| file.read }
+
+    else # must be a source definition
+        manifest_data = <<-EOF
+sources:
+    - type: #{args.shift}
+      url: #{args.shift}
+        EOF
+        while !args.empty?
+            name, value = args.shift.split("=")
+            manifest_data << "      #{name}: #{value}\n"
+        end
+
+    end
+
+    File.open(File.join(Autoproj.config_dir, "manifest"), "w") do |io|
+        io.write(manifest_data)
+    end
+end
+
+case mode
+when "bootstrap"
+    # If there is no argument, We need one more argument. It is either a VCS type or a path to a
+    # manifest.
+    #
+    # In the first case, we create a new manifest and add the source given on
+    # the command line to it. In the second case, we simply use it as a
+    # manifest.
+    do_bootstrap(*args)
+    only_update_sources = true
+
+when "build"
+when "update"
+    Autobuild.do_build  = false
+when "status"
+    only_do_status      = true
+    Autobuild.do_update = false
+    no_os_deps = true
+when "update-sources"
+    only_update_sources = true
+when "list-sources"
+    only_update_sources = true
+    Autobuild.do_update = false
+    
+when "doc"
+    Autobuild.do_update = false
+    Autobuild.do_doc    = true
+    Autobuild.only_doc  = true
+else
+    puts parser
+    exit(1)
+end
+
+# Find the autoproj root dir
+root_dir = Autoproj.root_dir
+# Load user configuration
+Autoproj.load_config
+# Set the initial environment
+Autoproj.set_initial_env
+# Set up some important autobuild parameters
+Autobuild.prefix  = Autoproj.build_dir
+Autobuild.srcdir  = root_dir
+Autobuild.doc_errors = false
+Autobuild.do_doc = false
+Autobuild::Reporting << Autoproj::Reporter.new
+if mail_config[:to]
+    Autobuild::Reporting << MailReporter.new(mail_config)
+end
+
+
+Dir.chdir(root_dir)
+begin
+    # Load the installation's manifest
+    manifest = Autoproj.manifest = Manifest.load(File.join('autoproj', 'manifest'))
+    source_os_dependencies = manifest.each_remote_source(false).
+        inject(Set.new) do |set, source|
+            set << source.vcs.type if !source.local?
+        end
+
+    # If we need to install some packages to import our remote sources, do it
+    if !no_os_deps && !source_os_dependencies.empty?
+        STDERR.puts color("autoproj: installing prepackaged dependencies to access the source definitions", :bold)
+        osdeps = manifest.known_os_packages
+        osdeps.install(source_os_dependencies)
+    end
+
+    # Update the remote sources if there are any
+    if manifest.has_remote_sources?
+        STDERR.puts color("autoproj: updating remote sources", :bold)
+        Autobuild::Reporting.report do
+            manifest.update_remote_sources
+        end
+        STDERR.puts
+    end
+
+    # If in verbose mode, or if we only update sources, list the sources
+    #
+    # Note that we can't have the Manifest class load the source.yml file, as it
+    # cannot resolve all constants. So we need to do it ourselves to get the
+    # name ...
+    if Autoproj.verbose || only_update_sources
+        sources = manifest.each_source(false).to_a
+
+        if sources.empty?
+            STDERR.puts color("autoproj: no sources defined in autoproj/manifest", :bold, :red)
+        else
+            STDERR.puts color("autoproj: available sources", :bold)
+            manifest.each_source(false) do |source|
+                source_yml = source.raw_description_file
+                STDERR.puts "  #{source_yml['name']}"
+                if source.local?
+                    STDERR.puts "    local source in #{source.local_dir}"
+                else
+                    STDERR.puts "    vcs:   #{source.vcs}, #{source.vcs.options.inspect}"
+                    STDERR.puts "    local: #{source.local_dir}"
+                end
+            end
+        end
+    end
+    if only_update_sources
+        exit(0)
+    end
+
+    # Load init.rb files. each_source must not load the source.yml file, as
+    # init.rb may define configuration options that are used there
+    manifest.each_source(false) do |source|
+        init_rb = File.join(source.local_dir, "init.rb")
+        if File.exists?(init_rb)
+            load init_rb
+        end
+    end
+
+    # Load the required autobuild definitions
+    STDERR.puts color("autoproj: loading build files and configuring build", :bold)
+    if !Autoproj.reconfigure?
+        STDERR.puts color("run 'autoproj --reconfigure' to change configuration values", :bold)
+    end
+    manifest.each_autobuild_file do |source, name|
+        Autoproj.import_autobuild_file source, name
+    end
+
+    # The user is asked for configuration values both during the manifest
+    # loading and the loading of autobuild files. Save it now.
+    Autoproj.save_config
+
+    # Now, load the package's importer configurations (from the various
+    # source.yml files)
+    manifest.load_importers
+    if Autoproj.verbose
+        # List defined packages, and in which autobuild files they are defined
+        STDERR.puts "Available packages:"
+        Autoproj.manifest.packages.each_value do |package, source, file|
+            STDERR.puts "  #{package.name}: #{file} from #{source.name}"
+        end
+    end
+
+    # Create the build target from the manifest if the user did not provide an
+    # explicit one
+    if selected_packages.empty?
+        selected_packages = Autoproj.manifest.default_packages
+    end
+    if Autoproj.verbose
+        STDERR.puts "will install #{selected_packages.to_a.join(", ")}"
+    end
+    if selected_packages.empty? # no packages, terminate
+        STDERR.puts
+        STDERR.puts color("autoproj: no packages defined", :red)
+        exit 0
+    end
+    selected_packages = selected_packages.to_set
+
+    # Install prepackaged dependencies needed to import and build the packages
+    if !no_os_deps
+        STDERR.puts
+        STDERR.puts color("autoproj: installing prepackaged dependencies for build system & version control", :bold)
+        osdeps = manifest.known_os_packages
+        osdeps.install(Autoproj.build_system_dependencies - source_os_dependencies)
+    end
+
+    # Now starts a different stage of the whole build. Until now, we were
+    # working on the whole package set. Starting from now, we need to build the
+    # package sets based on the layout file
+    Autoproj.manifest.each_package_set(selected_packages) do |name, packages, enabled_packages|
+        srcdir  = File.join(Autoproj.root_dir, name)
+        prefix  = File.join(Autoproj.build_dir, name)
+        logdir  = File.join(prefix, "log")
+        Autobuild.logdir = logdir
+        packages.each do |pkg_name|
+            pkg = Autobuild::Package[pkg_name]
+            pkg.srcdir = File.join(srcdir, pkg_name)
+            pkg.prefix = prefix
+            pkg.logdir = logdir
+        end
+
+        # We are doing a status, now is the right time
+        STDERR.puts
+        if only_do_status
+            do_status(packages)
+            next
+        end
+
+
+        # ... but build only the selected packages (and avoid display of
+        # progress messages if there is nothing to build)
+        if !enabled_packages.empty?
+            STDERR.puts
+            STDERR.puts color("autoproj: now building #{name}", :bold)
+
+            STDERR.puts color("  updating packages", :bold)
+            Autobuild::Reporting.report do
+                import_targets = enabled_packages.map { |pkg| "#{pkg}-import" }
+                task "autoproj-#{name}-import" => import_targets
+                Rake::Task["autoproj-#{name}-import"].invoke
+            end
+
+            # Load package manifests, apply dependencies to the autobuild definitions
+            # and install OS packages
+            manifest.load_package_manifests(packages)
+            if !no_os_deps
+                STDERR.puts color("  installing prepackaged dependencies", :bold)
+                manifest.install_os_dependencies
+            end
+
+            # Call the prepare target now, after we did the import *and* loaded
+            # the manifests
+            Autobuild::Reporting.report do
+                prepare_targets = enabled_packages.map { |pkg| "#{pkg}-prepare" }
+                task "autoproj-#{name}-prepare" => prepare_targets
+                Rake::Task["autoproj-#{name}-prepare"].invoke
+            end
+
+            # And now build
+            if Autobuild.only_doc
+                STDERR.puts color("  building and installing documentation", :bold)
+            else
+                STDERR.puts color("  building and installing packages", :bold)
+            end
+
+            Autobuild::Reporting.report do
+                Autobuild.apply(enabled_packages, "autoproj-#{name}")
+                Autobuild::Reporting.success
+            end
+        end
+
+        # Now call the prepare target for all packages as it may be useful for
+        # the rest of the builds and for the generation of the env.sh file
+        #
+        # Note that we don't really have to care about dependencies, but we do
+        # it anyway
+        old_update_flag = Autobuild.do_update
+        begin
+            Autobuild.do_update = false
+            Autobuild::Reporting.report do
+                prepare_targets = (packages - enabled_packages).map { |pkg_name| "#{pkg_name}-prepare" }
+                task "autoproj-#{name}-prepare" => prepare_targets
+                Rake::Task["autoproj-#{name}-prepare"].invoke
+            end
+        ensure
+            Autobuild.do_update = old_update_flag
+        end
+
+        Autoproj.export_env_sh(name)
+    end
+
+rescue ConfigError => e
+    STDERR.puts
+    STDERR.puts color(e.message, :red, :bold)
+    if debug then raise
+    else exit 1
+    end
+rescue Interrupt
+    STDERR.puts
+    STDERR.puts color("Interrupted by user", :red, :bold)
+    if debug then raise
+    else exit 1
+    end
+end
+