AptDownloader 1.0.0

data/History.txt

@@ -0,0 +1,9 @@
+=== 1.0.0 / 2008-12-21
+
+* 1 major enhancement
+
+  * Birthday!
+  * Can download to a given directory for any architecture
+  * no reuse of temp files
+  * no uninstall
+  

data/Manifest.txt

@@ -0,0 +1,7 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+bin/apt_downloader
+lib/apt_downloader.rb
+test/test_apt_downloader.rb

data/README.txt

@@ -0,0 +1,88 @@
+= AptDownloader
+
+http://rubyforge.org/projects/uwruby/
+http://uwruby/rubyforge.org/apt_downloader
+
+== DESCRIPTION:
+
+Apt-Downloader is intended as a  useful tool for
+developers working on embedded linux. 
+
+When working on embedded linux it can be difficult to get programs working
+on your target due to the shear number of dependencies needed to run
+them.  Debian has done an excellent job of building and testing their
+software packages on a large number of architectures.  The debian apt system
+makes it easy for debian users to get access to this software on their own
+platforms, but it's a bit obsfuscated how to use thier software to say, 
+download a package and all of it's dependencies to the root filesystem of
+your target and target architecture.  
+
+Apt-Downloader is front end to apt that is intended to make this easy.
+
+== FEATURES/PROBLEMS:
+
+Features:
+ *Automatically builds up package lists from debian
+ *Automatically determines all dependency packages and dowloads them.
+ *Allows users to pick an output directory for either deb packages, the
+extraction of such packages, or both.
+ *Cleans up after itself - all temp files are deleted at program exit.
+ *Clean and simple to use.
+
+Problems:
+ *Takes a little while: building package lists is a slow process, especially
+over a slow internet connection
+ *Does not (yet) support reusing the temp files that take a while to generate/
+download before packages can be installed.
+ *Does not (yet) support reusing any downloaded files.  Files are re-downloaded
+from scratch each time they are needed.
+ *Checks your package names against the host system.  If host and target servers
+don't match, this could create a problem.
+ *Not capable of uninstall.  If you don't like what you get, you will need to
+regenerate your rootFS (however you could just install apt in your root fs
+with this tool and then use it nativiely if your target has an internet
+connection
+
+== SYNOPSIS:
+
+   ad = AptDownloader.new
+   ad.architecture = 'arm'
+   ad.addPackages 'perl', 'ruby', 'ffmpeg'
+   ad.output_directory = '~/arm_deb_packages'
+   ad.go
+
+== REQUIREMENTS:
+
+You must have the apt (Advanced Package Tool) toolset  
+installed on your distrobution of linux.  Apt is was designed for
+Debian and is also in use on Ubuntu and FIXME-- ADD OTHER DISTROS HERE
+
+== INSTALL:
+
+sudo gem install
+thats it.
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2008 FIX
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,12 @@
+# -*- ruby -*-
+
+require 'rubygems'
+require 'hoe'
+require './lib/apt_downloader.rb'
+
+Hoe.new('AptDownloader', AptDownloader::VERSION) do |p|
+   p.rubyforge_name = 'uwruby'
+   p.developer('Mathew Chasan', 'aftermathew@gmail.com')
+end
+
+# vim: syntax=Ruby

data/bin/apt_downloader

@@ -0,0 +1,126 @@
+#!/usr/bin/ruby
+
+require 'rubygems'
+require 'optparse'
+require 'pp'
+require 'bin/apt_downloader'
+
+class ThisApplication
+  def parse_options ops = ARGV
+    @option_parser = OptionParser.new do |opts|
+      opts.banner = "Usage apt_downloader [ops] debian-package [more packages]"
+
+      opts.on("-a", "--architecture ARCH", "Debian architecure to download " +
+              "packages for.  Options are:" +
+              "#{AptDownloader::VALID_DEBIAN_ARCHITECTURES.map{|x| "\n" +"\t"*5+"#{x}"} }") do |a|
+        @architecure = a
+      end
+      
+      opts.on("-o", "--output-dir DIR", "Copy all deb files to directory " +
+              "(directory created if needed)") do |dir|
+        puts dir
+        @output_dir = dir
+      end
+
+      opts.on("-x", "--extract-dir DIR", "Extract all downloaded deb files to" +
+              " the supplied directory.  Debs will be extracted without" +
+              " root privelages.") do |x|
+        @extract_dir = x
+        @sudo_extract = false
+      end
+
+      opts.on("-X", "--sudo-extract-dir DIR", "Extract all downloaded deb files" +
+              " to the supplied directory.  Debs will be extracted with" +
+              " root privelages (for extracting to your target root" +
+              " filesystem).  Users will be asked for sudo password.") do |x|
+        @extract_dir = x
+        @sudo_extract = true
+      end
+
+      # these features not yet implemented
+      if nil
+        opts.on("-s", "--save-dir-structure", "NOT YET IMPLEMENTED " +
+                "Save the temp filesystem so that the" +
+                " next call to apt_downloader can build on what you have" +
+                " already downloaded and not from scratch") do |s|
+          @save_fs = true
+        end
+
+        opts.on("-r", "--reuse-dir-structure", "NOT YET IMPLEMENTED " +
+                "Reuse a previously saved  directory structure" +
+                " (and save new downloads to it)." +
+                " Structure should have been created with a previous run" +
+                " of apt_downloader with the -s option") do |dir|
+          @reuse_fs = dir
+        end
+      end
+
+      opts.on("--", "--end",
+              "Don't parse any more options") do
+        OptionParser.terminate
+      end
+    end #end option parser block
+
+    @packages = begin @option_parser.parse ops
+                rescue OptionParser::InvalidOption => e
+                  puts e
+                  puts option_parser
+                  exit 1
+                end
+  end
+
+  def end_badly
+    puts @option_parser
+    exit 1
+  end
+
+  def initialize
+    @output_dir = nil
+    @extract_dir = nil
+    @sudo_extract =false
+    @architecture = nil
+
+    parse_options
+    
+    unless @packages.size > 0
+      end_badly
+    end
+   
+    unless @output_dir || @extract_dir
+      puts "This program is being run without saving or extracting *any* deb" +
+        " files!  Cowardly refusing to run!"
+      end_badly
+    end
+
+    ad = AptDownloader.new
+    ad.download_dir = @output_dir if @output_dir
+    ad.extract_dir = @extract_dir if @extract_dir
+    ad.sudo_extract = @sudo_extract
+   
+    begin
+      unless @architecture then
+        puts "Architecture left blank, defaulting to armel"
+        @architecture = 'armel'
+      end
+      ad.architecture = @architecture
+    rescue
+      puts "#{@architecture} is not a valid architecture"
+      puts 'Giving up...'
+      end_badly
+    end
+
+    @packages.each{  |package|
+      begin
+        ad.addPackage package
+      rescue
+        puts "#{package} was not found as a valid package!"
+        puts 'Giving up...'
+        end_badly
+      end
+    }
+
+    ad.go
+  end
+end
+
+ThisApplication.new

data/lib/apt_downloader.rb

@@ -0,0 +1,271 @@
+#!/usr/bin/ruby -w
+
+
+require 'pp'
+require 'tempfile'
+require 'fileutils'
+
+class AptDownloader
+  VERSION = '1.0.0'
+  VALID_DEBIAN_ARCHITECTURES = %w[alpha amd64 arm armel hppa i386 ia64 mips
+                                  mipsel powerpc s390 sparc]
+  APT_FS_PATH =  "#{Dir::tmpdir}/apt_fs_#{$$}_"
+
+  DEFAULT_DEBIAN_SERVER_LIST= ['deb     http://security.debian.org/ testing/updates main contrib non-free',
+                               'deb-src http://security.debian.org/ testing/updates main contrib non-free',
+                               'deb     http://security.debian.org/ stable/updates main contrib non-free',
+                               'deb-src http://security.debian.org/ stable/updates main contrib non-free',
+                               # Main archives
+                               'deb     http://ftp.us.debian.org/debian/ stable main non-free contrib',
+                               'deb-src http://ftp.us.debian.org/debian/ stable main non-free contrib',
+                               'deb     http://ftp.us.debian.org/debian/ testing main non-free contrib',
+                               'deb-src http://ftp.us.debian.org/debian/ testing main non-free contrib',
+                               'deb     http://ftp.us.debian.org/debian/ unstable main non-free contrib',
+                               'deb-src http://ftp.us.debian.org/debian/ unstable main non-free contrib',
+
+                               'deb     http://ftp.us.debian.org/debian/ ' +
+                               'experimental main non-free contrib',
+
+                               'deb-src http://ftp.us.debian.org/debian/ ' +
+                               'experimental main non-free contrib',
+
+                               # --- debian-multimedia
+                               # deb http://www.debian-multimedia.org etch main'
+                               'deb http://www.debian-multimedia.org unstable main',
+                               'deb http://www.debian-multimedia.org testing main',
+                               'deb http://www.debian-multimedia.org experimental main',
+                               'deb-src http://www.debian-multimedia.org sid main',
+                              ]
+
+  def config_file_contents
+    return <<-EOS
+APT "";
+APT::Architecture "#{@architecture}";
+APT::Build-Essential "";
+APT::Build-Essential:: "build-essential";
+APT::Install-Recommends "1";
+APT::Install-Suggests "0";
+APT::Acquire "";
+APT::Acquire::Translation "environment";
+APT::Authentication "";
+APT::Authentication::TrustCDROM "true";
+APT::NeverAutoRemove "";
+APT::NeverAutoRemove:: "^linux-image.*";
+APT::NeverAutoRemove:: "^linux-restricted-modules.*";
+APT::Default-Release "testing";
+APT::Cache-Limit "141943904";
+Dir "#{APT_FS_PATH}/";
+Dir::State "#{APT_FS_PATH}/var/lib/apt/";
+Dir::State::lists "lists/";
+Dir::State::cdroms "cdroms.list";
+Dir::State::userstatus "status.user";
+Dir::State::status "#{APT_FS_PATH}/var/lib/dpkg/status";
+Dir::Cache "#{APT_FS_PATH}/var/cache/apt/";
+Dir::Cache::archives "archives/";
+Dir::Cache::srcpkgcache "srcpkgcache.bin";
+Dir::Cache::pkgcache "pkgcache.bin";
+Dir::Etc "etc/apt/";
+Dir::Etc::sourcelist "sources.list";
+Dir::Etc::sourceparts "sources.list.d";
+Dir::Etc::vendorlist "vendors.list";
+Dir::Etc::vendorparts "vendors.list.d";
+Dir::Etc::main "apt.conf";
+Dir::Etc::parts "apt.conf.d";
+Dir::Etc::preferences "preferences";
+Dir::Bin "";
+Dir::Bin::methods "/usr/lib/apt/methods";
+Dir::Bin::dpkg "/usr/bin/dpkg";
+Dir::Log "#{APT_FS_PATH}/var/log/apt";
+Dir::Log::Terminal "term.log";
+DPkg "";
+DPkg::Pre-Install-Pkgs "";
+DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
+DPkg::Post-Invoke "";
+DPkg::Post-Invoke:: "if [ -x /usr/bin/debsums ]; then /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives; fi";
+
+  EOS
+
+  end
+
+  
+  attr_accessor :architecture, :download_dir, :apt_tmp_fs, :extract_dir
+  attr_accessor :sudo_extract
+
+  # at this point packages once added can't be deleted,
+  # until i implement apt-remove to get rid of them from
+  # apt-tmp-fs
+  attr_reader :packages
+
+  def create_config_file
+    @apt_config = Tempfile.new 'config_file'
+    @apt_config.sync = true
+    @apt_config.write config_file_contents
+  end
+
+  # create a temprorary directory structure that imititates the parts of the
+  # linux filesystem that apt cares about
+  def create_apt_fs
+    # create a temporary directory to do all downloads to
+    @apt_tmp_fs = FileUtils.mkpath APT_FS_PATH
+
+    %w[/etc/apt/
+       /var/cache/apt/archives/partial
+       /var/lib/apt/lists/partial
+       /var/lib/dpkg/
+      ].each{ |directory| FileUtils.mkpath(APT_FS_PATH + directory) }
+
+    %w[/var/lib/dpkg/lock
+       /var/lib/dpkg/status
+       /var/lib/apt/lists/lock
+       /etc/apt/sources.list
+      ].each{ |file| FileUtils.touch(APT_FS_PATH + file) }
+
+    File.open(APT_FS_PATH + "/etc/apt/sources.list",
+              File::WRONLY |
+              File::TRUNC |
+              File::CREAT ) { |sources_file|
+      DEFAULT_DEBIAN_SERVER_LIST.each{ |source| sources_file.puts source }
+    }
+
+    # set up directory for removal on program delete
+    at_exit{ FileUtils.rm_rf APT_FS_PATH  }
+  end
+
+
+  def download packages
+    call = "apt-get -c #{@apt_config.path} update"
+    puts "** apt-downloader: #{call}"
+    system(call) unless @already_updated
+
+    @already_updated = true
+
+    call = "apt-get -c #{@apt_config.path} --download-only " +
+           "--assume-yes --force-yes install #{packages}"
+    puts "\n** apt-downloader: #{call}"
+    system(call)
+  end
+
+  def initialize architecture='armel', *packages
+    @packages         = []
+    packages.each{ |package| addPackage package }
+
+    @already_updated  = false
+    @apt_config       = nil
+    @apt_tmp_fs       = nil
+    
+    @architecture     = architecture
+    @download_dir     = nil
+    @extract_dir      = nil
+    @sudo_extract     = false
+    @dependencies     = []
+    @suggestions      = []
+    @all_dependencies = []
+    @all_suggestions  = []
+  end
+
+  protected
+  def _handle_debs path
+    mypath = File.expand_path path
+    unless File.exist?(mypath)
+      FileUtils.mkpath mypath
+    end
+
+    unless File.directory?(mypath)
+      raise(ArgumentError,
+            "Cannot save to #{mypath} since it is not a directory")
+    end
+
+    yield mypath
+  end
+
+  # internal use only
+  def _copy_debs_to path
+    _handle_debs(@download_dir){ |path|
+      FileUtils.cp(Dir.glob(@apt_tmp_fs + "/var/cache/apt/archives/*.deb"),path)
+    }
+  end
+
+  def _extract_debs_to path
+    _handle_debs(@extract_dir) { |path|
+      Dir.glob(@apt_tmp_fs + "/var/cache/apt/archives/*.deb").each { |archive|
+        puts "** Extracting: #{archive}"
+        sudo = @sudo_extract ? 'sudo' : ''
+        system "#{sudo} dpkg -x #{archive} #{path}" 
+      }
+    }
+  end
+
+  public
+  def go
+    create_config_file unless @apt_config
+    create_apt_fs unless @apt_tmp_fs
+
+    packages_string = ''
+    packages.each{ |package| packages_string += package + ' ' }
+    download packages_string
+
+    _copy_debs_to @download_dir if (@download_dir)
+    _extract_debs_to @extract_dir if(@extract_dir)
+  end
+
+  # Returns true if packageName is a real debian package, and false otherwise
+  def AptDownloader.validPackage? packageName
+    result = %x[apt-cache pkgnames]
+    result.split("\n").include?(packageName)
+  end
+
+  def addPackages(*packageNames)
+    packageNames.each { |name|
+      unless AptDownloader::validPackage? name
+        raise ArgumentError, "Package #{name} does not exist."
+      end
+      @packages << name
+    }
+  end
+
+  def addPackage(package)
+    addPackages package
+  end
+
+  def architecture= ( architecture )
+    arch = architecture.downcase
+    unless VALID_DEBIAN_ARCHITECTURES.include? arch
+      raise ArgumentError, "Invalid architecture"
+    end
+    @architecture = arch
+  end
+
+  #internal use only
+  def _get_dependencies_backend callword, searchword, recurse=false
+    unless @packages
+      raise StandardError, "No packages set!"
+    end
+    dependencies = packages.map { |package|
+      deps = %x[apt-cache #{recurse ? '--recurse' : ''} #{callword} #{package}]
+      deps = deps.map{ |x|
+        # get rid of coflicts/and other messages we aren't lookign for
+        x = x.sub!(searchword, '') ? x.strip! : ''
+        x = x.include?('|') ? '' : x
+      }
+    }
+    dependencies.flatten!.uniq!
+    dependencies.delete ''
+    dependencies
+  end
+
+  def depends
+    @dependencies = _get_dependencies_backend 'depends', 'Depends:'
+  end
+
+  def suggests
+    @suggestions = _get_dependencies_backend 'depends', 'Suggests:'
+  end
+
+  def recursiveDepends
+    @all_dependencies = _get_dependencies_backend 'depends', 'Depends:', true
+  end
+
+  def recursiveSuggests
+    @all_suggestions = _get_dependencies_backend 'depends', 'Suggests:', true
+  end
+end

data/test/test_apt_downloader.rb

@@ -0,0 +1,71 @@
+#!/usr/bin/env ruby -w
+
+require 'test/unit'
+require 'apt_downloader'
+
+
+
+class TestAptDownloader < Test::Unit::TestCase
+  def setup
+    @downloader = AptDownloader.new
+  end
+
+  def test_invalid_package
+    assert_raise(ArgumentError){
+      @downloader.addPackages  "FakePackageName"
+    }
+  end
+
+  def test_valid_package
+    assert(AptDownloader.validPackage?('ruby'))
+  end
+
+  def test_add_many_packages
+    assert_nothing_raised{
+      @downloader.addPackages 'ruby', 'ruby1.9', 'ruby1.8'
+    }
+    assert_equal(@downloader.packages, %w[ruby ruby1.9 ruby1.8])
+    assert_nothing_raised{
+      @downloader.addPackages 'rake', 'ruby-dev'
+    }
+    assert_equal(@downloader.packages, %w[ruby ruby1.9 ruby1.8 rake ruby-dev])
+  end
+  
+  def test_invalid_architecture
+    assert_raise(ArgumentError){
+      @downloader.architecture =  "vax"
+    }
+  end
+
+  def test_valid_architecture
+    assert_nothing_raised{
+      @downloader.architecture = "arm"
+    }
+  end
+
+  def test_dependencies
+    @downloader.addPackage "ruby1.8"
+    assert_equal( %w[libc6 libruby1.8], @downloader.depends )
+    assert_equal( %w[ruby1.8-examples rdoc1.8 ri1.8], @downloader.suggests )
+  end
+
+  def test_recursive_dependencies
+    @downloader.addPackage 'ffmpeg'
+    assert_equal(["libavcodec51", "libavdevice52", "libavformat52", \
+                  "libavutil49", "libc6", "libfreetype6", "libimlib2",\
+                  "libsdl1.2debian", "libswscale0"], @downloader.depends)
+    # i thought a lot about how to handle this.  It actually returns 1967
+    # different dependencies but that is liable to change as debian changes.
+    # Mostly i wanted make sure this is big
+    d = @downloader.recursiveDepends
+    assert(d.size > 1900)
+    assert_equal(d.compact, d)
+    assert_equal(d.uniq, d)
+  end
+
+  def test_download
+  end
+
+  def teardown
+  end
+end

metadata

@@ -0,0 +1,72 @@
+--- !ruby/object:Gem::Specification 
+name: AptDownloader
+version: !ruby/object:Gem::Version 
+  version: 1.0.0
+platform: ruby
+authors: 
+- Mathew Chasan
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2008-12-23 00:00:00 -08:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: hoe
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.8.2
+    version: 
+description: Apt-Downloader is intended as a  useful tool for developers working on embedded linux.   When working on embedded linux it can be difficult to get programs working on your target due to the shear number of dependencies needed to run them.  Debian has done an excellent job of building and testing their software packages on a large number of architectures.  The debian apt system makes it easy for debian users to get access to this software on their own platforms, but it's a bit obsfuscated how to use thier software to say,  download a package and all of it's dependencies to the root filesystem of your target and target architecture.    Apt-Downloader is front end to apt that is intended to make this easy.
+email: 
+- aftermathew@gmail.com
+executables: 
+- apt_downloader
+extensions: []
+
+extra_rdoc_files: 
+- History.txt
+- Manifest.txt
+- README.txt
+files: 
+- History.txt
+- Manifest.txt
+- README.txt
+- Rakefile
+- bin/apt_downloader
+- lib/apt_downloader.rb
+- test/test_apt_downloader.rb
+has_rdoc: true
+homepage: http://rubyforge.org/projects/uwruby/
+post_install_message: 
+rdoc_options: 
+- --main
+- README.txt
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: uwruby
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Apt-Downloader is intended as a  useful tool for developers working on embedded linux
+test_files: 
+- test/test_apt_downloader.rb