gjp 0.29.0 → 0.30.0

data/README.md

@@ -1,27 +1,21 @@
 # gjp – Green Java Packager's tools
 
-`gjp` (pronounced _/ˈdʒiː ˈaɪ ˈd͡ʒəʊ/_) is a set of tools to ease and partially automate Linux packaging of Java projects.
+`gjp` is a set of tools to ease creating Linux packages for Java projects.
 
-The project objective is to strongly reduce manual packaging efforts by enabling a new and much simpler workflow.
+## Why?
 
-## Status
-
-`gjp` is a research project currently in beta state. All basic features have been coded, non-essential ones are being planned, packages are currently being built. If you are a packager you can try to use it, any feedback would be **very** welcome!
-
-At the moment `gjp` is tested on openSUSE and can only output RPMs for the openSUSE and SLES distributions. Fedora, RHEL and other distros could be supported in the future.
+Packaging of Java projects is sometimes hard because build tools like Maven partially overlap with distro tools like RPM in functionality.
 
-## Contact
+We are trying to come up with **a tool that drastically simplifies packaging**.
 
-Are you using `gjp` or even just reading this? Let's get in touch!
+## How to install?
 
-    smoioli at suse dot de
-
-## Requirements and installation
+First you need:
 
 * [Ruby 1.9](https://www.ruby-lang.org/en/);
 * [git](http://git-scm.com/);
 * a JDK that can compile whatever software you need to package;
-* only for the optional `set-up-nonet-user` subcommand, `sudo` and `iptables`;
+* only for the optional `set-up-nonet-user` subcommand, `sudo` and `iptables`.
 
 You can install gjp via RubyGems:
 
@@ -29,38 +23,27 @@ You can install gjp via RubyGems:
 
 ## Workflow
 
-Building a package with `gjp` is quite unusual — this is a [deliberate choice](#motivation) to minimize packaging efforts.
-
-### Overview
-
-The basic process is:
-
-* a `gjp` project is created;
-* package sources are added in `src/<package name>`;
-* any other file that is needed for the build, except the JDK, is added in binary form (jars, the Maven executable, plugins,  etc.) in `kit/`. In `gjp` a **kit** is a set of binary files that satisfies all build dependencies in a `gjp` project;
-* a build is attempted and during the build, `gjp` keeps track of file changes. When it finishes, `gjp` restores `src/` in its original state, making it a "repeatable dry-run build". `gjp` will retain any files that were automatically downloaded by Maven or other similar tools in `kit/`, along with other binary build dependencies, and note any produced file for later spec generation;
-* `gjp` produces spec files for two packages: one for the project itself and one for the kit needed to build it;
-* kit and project packages can be submitted to [OBS](http://en.opensuse.org/openSUSE:Build_Service). Project package will rebuild cleanly because it needs no Internet access - all files were already downloaded during the dry-run phase above and are included in the kit.
-
-Note that:
+Building a package with `gjp` is quite unusual — this is a [deliberate choice](#motivation), so don't worry. Basic steps are:
 
-* the project's build time dependency graph is very simple: just its kit and the JDK;
-* the kit is basically a binary blob. If its sources are needed for proper packaging, for example to comply with the GPL, [a separate step](#kit-sources) is needed to add them;
-* the kit is needed at build time only (by OBS), no end user should ever install it;
-* a `gjp` project can be used to build a number of packages that share one binary kit. This can help if the kit becomes big in size;
-* `gjp` will take advantage of Maven's pom files to generate its specs if they are available. This allows to precompile most spec fields automatically.
+* `gjp init` a new project;
+* add sources to `src/<package name>` and everything else you need in binary form in `kit/` (eg. a Maven binary distribution);
+* execute `gjp dry-run`, any command needed to compile your software (possibly `gjp mvn package`), then `gjp finish`;
+* execute `gjp generate-all`;
+* done! Spec files and tarballs are baked in `output` automatically by `gjp`.
 
-### A sample Maven project (commons-collections)
+## How can that possibly work? (commons-collections walkthrough)
 
-#### Initialization and project setup
+Let's review all steps in detail with a concrete Maven-based example, commons-collections.
 
-Ceate a new `gjp` project, in this example named "galaxy":
+First, ceate a new `gjp` project, in this example named "myproject":
 
-    mkdir galaxy
-    cd galaxy
+    mkdir myproject
+    cd myproject
     gjp init
 
-`gjp init` generates a folder structure and assumes you respect it, in particular, you should place all your projects' source files in `src/`. Every `src/` subfolder will become a separate package named after the folder itself, so use the following commands to create a `commons-collections` folders and populate it:
+(`gjp init` generates a folder structure for you).
+
+Second, place all your projects' source files in `src/`. Every `src/` subfolder will become a separate package named after the folder itself, so use the following commands to create a `commons-collections` folders and populate it:
 
     cd src
     mkdir commons-collections
@@ -69,116 +52,91 @@ Ceate a new `gjp` project, in this example named "galaxy":
     unzip commons-collections-3.2.1-src.zip
     rm commons-collections-3.2.1-src.zip
 
-Now let's move to the kit (which, unsurprisingly, should be placed in the `kit/` directory). commons-collections needs Maven 3 to build, so we should simply unzip a copy in `kit/`:
+Third, all non-source files needed for the build should go into `kit`. This includes all build dependencies and tools excluding the JDK, so in this case we need Maven:
 
     cd ../../kit
-    wget http://apache.fastbull.org/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.zip
-    unzip apache-maven-3.1.0-bin.zip
-    rm apache-maven-3.1.0-bin.zip
-    cd ..
-
-This is actually everything needed to do a first dry-run build.
-
-#### First dry-run build
+    wget http://www.eu.apache.org/dist/maven/binaries/apache-maven-3.1.1-bin.zip
+    unzip apache-maven-3.1.1-bin.zip
+    rm apache-maven-3.1.1-bin.zip
 
-Let's call `gjp dry-run` to let `gjp` know we are building and then call Maven. Note that `gjp mvn` is used instead of plain `mvn`: `gjp` will take care of locating the Maven installation in `kit/` and ensure it will store all downloaded files there.
+Fourth, you need to show `gjp` how to build your package by executing `gjp dry-run` and `gjp finish`. Bash history will be recorded to generate a build script (possibly only a starting point, maybe sufficient in simple cases):
 
     gjp dry-run
     cd src/commons-collections/commons-collections-3.2.1-src/
     gjp mvn package
-
-Success! Now we have to tell gjp to return in normal mode:
-
     gjp finish
 
-At this point `gjp` restored `src/` as it was before the build while taking note of any produced file, so that later it will be able to output `%install` and `%files` sections of the project spec file automatically.
-
-Note that, if the build was unsusccesful, the following command can be used to cancel it and return to pre-dry running state:
+Note that we used `gjp mvn package` instead of `mvn package` so that `gjp` will use Maven from `kit/` instead of the system-wide installation you might have. `gjp` will also add some options on the mvn commandline so that any downloaded files will be retained in `kit/m2` so that this build can be replayed (even without network access) later. Also note that this being a dry-run build, sources will be brought back to their original state after `gjp finish`. This ensures build repeatability.
 
-    gjp finish --abort
+Finally, use this command to let `gjp` to generate build scripts, spec files and tarballs:
 
-#### Generating a build script
-
-`gjp` expects that all commands needed to build a package are in a `build.sh` script in `src/<package name>`. If you are a Bash user you are lucky - `gjp` can create one for you by looking at your command history! Just type:
-
-    gjp generate-package-script
+    gjp generate-all
 
-Note that `gjp` will substitute the `gjp mvn` calls with equivalent lines that are actually runnable on a build host without `gjp` itself.
+Note that gjp will also generate a special binary-only package called a **kit**, which contains basically the `kit/` folder. This is the only build-time requirement of any `gjp` package in your project.
 
-Of course this script can also be manually modified, and it must be in more difficult cases. You don't even have to be afraid of regenerating it later. `gjp` will run a three-way merge and warn if conflicts arise!
+## Special cases
 
-#### Generating archives and spec files
+### Failing builds
 
-The following command will generate the kit archive in `output/galaxy-kit/`:
+If youyr build fails for whatever reason, abort it with `gjp finish --abort`. `gjp` will restore all project files as they were before build.
 
-    gjp generate-kit-archive
+### Manual changes
 
-Note that, in later runs, only an additional "diff" tar.xz file will be created to ease uploads. You can use the `--full` option to regenerate a single complete archive.
+You can do any manual changes to spec and build.sh files and regenerate them later, `gjp` will reconcile changes with a [three-way merge](http://en.wikipedia.org/wiki/Three-way_merge#Three-way_merge) and alert about any conflicts. You can generate single files with the following commands:
 
-The following command will generate the kit spec:
+* `gjp generate-kit-archive`: (re)generates the kit tarball;
+* `gjp generate-kit-spec`: (re)generates the kit spec;
+* `gjp generate-package-script`: (re)generates the `build.sh` file from the latest bash history (assumes `gjp dry-run` and `gjp finish`have been used). Assumes your current working directory is in a package folder (that is, a subdirectory of `src/<package name>/`);
+* `gjp generate-package-archive`: (re)generates a package tarball;
+* `gjp generate-package-spec`: (re)generates a package spec;
 
-    gjp generate-kit-spec
+Note that, by default, `generate-kit-archive` will generate additional "diff" tar.xz files instead of rewriting the whole archive - this will result in faster uploads if you use OBS (see below). You can use the `--full` option to regenerate a single complete archive.
 
-You can inspect the generated "galaxy-kit.spec" file, but in general you should not need to edit it.
+### [OBS](build.opensuse.org)
 
-You can then generate the project spec and archive files provided you have a pom file (more formats will be supported in future). In this case:
+If you want to submit packages to OBS, you can do so by replacing the `output/` directory in your `gjp` project with a symlink to your OBS project directory.
 
-    gjp generate-package-archive
-    gjp generate-package-spec
-    less ../commons-collections.spec
+Packages will rebuild cleanly in OBS because no Internet access is needed - all files were already downloaded during dry-run and are included in the kit.
 
-commons-collection BuldRequires galaxy-kit, its archive contains only source files and it will install any produced .jar file in `/usr/lib/java`.
-You can also edit the specs file manually if you want. When you later regenerate it, `gjp` will automatically try to reconcile changes with a [three-way merge](http://en.wikipedia.org/wiki/Three-way_merge#Three-way_merge).
+Note that the kit package is needed at build time only by OBS, no end user should ever install it.
 
-OBS users: note that the output/ directory created by gjp can be submitted or used as OBS project. Feel free to replace it with a symlink pointing at your home OBS project, or use symlinks from your OBS project to its contents.
+### Kit sources
 
-#### Quicker workflow
+Your kit is basically a binary blob. If its sources are needed for proper packaging, for example to comply with the GPL, some extra steps are needed.
 
-`gjp` also has a `generate-all` subcommand that will generate everything in one step. Thus, for Maven-based projects, the minimal workflow is:
+If you use Maven, most (~90%) sources can be automatically downloaded:
 
-    cd src
-    mkdir <package_name>
-    cd <package_name>
-    wget <sources>
-    ...
-    gjp dry-run
-    gjp mvn package
-    gjp finish
-    gjp generate-all
-
-#### Kit sources
+    gjp get-maven-source-jars
 
-If you want to redistribute your RPMs, chances are that you need source files for kit contents.
+The remaining (mostly very outdated) jars will be listed by `gjp` when the download ends. You need to manually find corresponding sources for them, you can use:
 
-If you use Maven, most sources for binary jars in your kit can be automatically downloaded:
+    gjp get-source /path/to/<jar name>.pom
 
-    gjp get-maven-source-jars
+to get pointers to relevant sites if available in the pom itself.
 
-For non-Maven jars, or Maven jars that have no available sources, some extra manual work is needed. First of all, you should get a `pom.xml` file for your jar, if you don't have it already:
+A list of commonly used jars can be found [below](#frequently-used-jar-sources).
 
-    gjp get-pom jarname.jar
+### Ant builds
 
-This will create a `jarname.pom` file (if it can be found in Maven Central, otherwise you will need to search the Internet manually).
+`gjp` is currently optimized for Maven as it is the most common build tool, but it can work with any other. In particular, support for Ant has already been implemented and `gjp ant` works like `gjp mvn`.
 
-If you are lucky, the pom file will contain an URL to a site where sources can be downloaded, or even an SCM address. At the moment automatic source retrieval is not automated, but `gjp` can help you with the following utility subcommands:
+Sometimes you will have jar files distributed along with the source archive that will end up in `src/`: you don't want that! Run
 
-* `gjp get-parent-pom POM` will attempt to download a pom's parent from search.maven.org, where `POM` is a filename or URI;
-* `gjp get-source-address POM` will attempt to find the SCM Internet address of a pom.xml from the file itself or through api.github.com. `POM` can either be a filename or a URI;
-* `gjp get-source POM ADDRESS` downloads the source of a pom.xml's project from its SCM at ADDRESS;
+    gjp purge-jars
 
-More comprehensive support is planned in future releases.
+to have them moved to `kit/jars`. The command will generate a symlink back to the original, so builds will work as expected.
 
-#### Optional: Ant packages
+When generating spec files, be sure to have a `pom.xml` in your package directory even if you are not using Maven: `gjp` will automatically take advantage of information from it to compile many fields.
 
-Building Ant packages is not really different from Maven ones, as `gjp ant` will operate exactly like `gjp mvn`.
+You can also ask `gjp` to find one via `gjp get-pom <filename>.jar` (be sure to have Maven in your kit).
 
-Sometimes you will have jar files distributed along with the source archive that will end up in `src/`: you don't want that, just run `gjp purge-jars` to have them moved to the kit. The command will generate a symlink back to the original, so builds will not fail.
+### Other builds
 
-Once built, you should get a pom.xml file (via `gjp get-pom`, see above) in order to generate its spec.
+Other build tools are currently unsupported but will be added in the future. You can nevertheless use them - the only rule is that you have to keep all of their files in `kit`.
 
-#### Optional: running networkless dry-run builds
+### Optional: networkless dry-run builds
 
-`gjp` has a subcommand to setup a `nonet` user without Internet access, courtesy of `iptables`. You can simply retry the build using that user to see if it works. Note that the following commands will alter group permissions to allow both your current user and `nonet` to work on the same files.
+If you want to be 100% sure your package builds without network access, you can use a special `gjp` subcommand to setup a `nonet` user that cannot use the Internet. Then you can simply retry the build using that user to see if it works. Note that the following commands will alter group permissions to allow both your current user and `nonet` to work on the same files.
 
     gjp set-up-nonet-user
     chmod -R g+rw ../../..
@@ -190,9 +148,6 @@ Once built, you should get a pom.xml file (via `gjp get-pom`, see above) in orde
     exit
     gjp finish
 
-The above is not mandatory, but it can be useful for debugging purposes.
-
-
 ### Gotchas
 
 * `gjp` internally uses `git` to keep track of files, any gjp project is actually also a `git` repo. Feel free to navigate it, you can commit, push and pull as long as the `gjp` tags are preserved. You can also delete commits and tags, effectively rewinding gjp history (just make sure to delete all tags pointing to a certain commit when you discard it);
@@ -201,6 +156,13 @@ The above is not mandatory, but it can be useful for debugging purposes.
 
     export NO_BRP_CHECK_BYTECODE_VERSION=true
 
+* if packages build at first but then fail after a few days because Maven tries to connect to the Internet, add the `--option` flag to the `mvn` line in `build.sh`;
+
+### Frequently used jar sources
+
+* ant-launcher-1.6.5: is included in ant-1.6.5 sources, you most prbably have it already in `kit/m2/ant/ant/1.6.5`;
+* doxia-sink-api-1.0-alpha-4: use `svn checkout http://svn.apache.org/repos/asf/maven/doxia/doxia/tags/doxia-sink-api-1.0-alpha-4/ doxia-sink-api-1.0-alpha-4`;
+* kxml2-2.2.2: use http://downloads.sourceforge.net/project/kxml/kxml2/2.2.2/kxml2-src-2.2.2.zip;
 
 ## Motivation
 
@@ -219,17 +181,24 @@ The current solution in openSUSE is having the packager handle those differences
 
 The Fedora community is experimenting with another set of tools, [XMvn](http://mizdebsk.fedorapeople.org/xmvn/site/), which goals are similar to `gjp`'s. 
 
-### Kit rationale
+## Kit rationale
 
 `gjp` simplifies the packaging process mostly because of its use of a binary blob package that contains all build time dependencies for a set of packages called a **kit**.
 
 Building software from a binary blob is unusual for Linux distros, and it surely has some drawbacks. It is anyway believed that benefits outweigh them, in fact using prebuilt software:
 
-* drastically reduces packaging efforts. A very basic and relatively simple package like [commons-collections](http://commons.apache.org/proper/commons-collections/) needs about [150 jars](https://build.opensuse.org/package/show/home:SilvioMoioli/galaxy-kit) just to be compiled and tested. Those should be packaged, roughly, one-by-one!
+* drastically reduces packaging efforts. A very basic and relatively simple package like [commons-collections](http://commons.apache.org/proper/commons-collections/) needs about [150 jars](https://build.opensuse.org/package/show/home:SilvioMoioli/myproject-kit) just to be compiled and tested. Those should be packaged, roughly, one-by-one!
 * is just the way all Java developers out there build, test and use their software — this is how they expect it to work. Any different approach is necessarily error-prone and could result in unexpected bugs;
 * does not affect the ability of providing patches to Java projects, as only build time requirements are in the kit. In virtually all cases patching a piece of software does not require to patch its build toolchain;
 * does not affect the ability of complying to software licenses like the GPL. In fact those licenses only require to redistribute a project's source code - not the whole toolchain needed to build it. [Sources can be added](#kit-sources) for GPL'd parts of the kit, if any.
 
+## Status
+
+`gjp` is a research project currently in beta state. If you are a packager you can try to use it, any feedback would be **very** welcome!
+
+At the moment `gjp` is tested on openSUSE.
+
+
 ## Sources
 
 `gjp`'s Git repo is available on GitHub, which can be browsed at:
@@ -240,6 +209,12 @@ and cloned with:
 
     git clone git@github.com:SilvioMoioli/gjp.git
 
+## Contact
+
+Are you using `gjp` or even just reading this? Let's get in touch!
+
+    smoioli at suse dot de
+
 ## License
 
 MIT license.

data/lib/gjp.rb

@@ -9,10 +9,10 @@ require "gjp/version_matcher"
 require "gjp/maven_website"
 require "gjp/limited_network_user"
 require "gjp/pom_getter"
-require "gjp/source_address_getter"
 require "gjp/source_getter"
-require "gjp/parent_pom_getter"
 require "gjp/kit_runner"
+require "gjp/ant_runner"
+require "gjp/maven_runner"
 require "gjp/kit_spec_adapter"
 require "gjp/package_spec_adapter"
 require "gjp/spec_generator"

data/lib/gjp/ant_runner.rb

@@ -0,0 +1,27 @@
+# encoding: UTF-8
+
+module Gjp
+  # runs Ant with gjp-specific options
+  class AntRunner < KitRunner
+    include Logger
+
+    # runs ant in a subprocess
+    def ant(options)
+      run_executable("#{get_ant_commandline(@project.full_path)} #{options.join(' ')}")
+    end
+
+    # returns a command line for running Ant from the specified
+    # prefix
+    def get_ant_commandline(prefix)
+      executable = find_executable("ant")
+
+      if executable != nil
+        ant_path = File.join(prefix, executable)
+
+        "#{ant_path}"
+      else
+        raise ExecutableNotFoundError.new("ant")
+      end
+    end
+  end
+end

data/lib/gjp/cli.rb

@@ -73,7 +73,7 @@ module Gjp
         checking_exceptions do
           project = Gjp::Project.new(".")
           ensure_dry_running(true, project) do
-            Gjp::KitRunner.new(project).mvn(@options)
+            Gjp::MavenRunner.new(project).mvn(@options)
           end
         end
       end
@@ -91,7 +91,7 @@ module Gjp
         checking_exceptions do
           project = Gjp::Project.new(".")
           ensure_dry_running(true, project) do
-            Gjp::KitRunner.new(project).ant(@options)
+            Gjp::AntRunner.new(project).ant(@options)
           end
         end
       end
@@ -301,42 +301,43 @@ module Gjp
 
             puts "#{format_path(path, project)} written, #{text_status}"
           else
-            puts "#{name} not found. Try http://google.com/#q=#{URI::encode(pom_getter.cleanup_name(name))}"
+            puts "#{name}'s pom not found. Try:"
+            puts "http://google.com/#q=#{URI::encode(pom_getter.cleanup_name(name) + " pom")}"
           end
         end
       end
     end
 
-    subcommand "get-parent-pom", "Retrieves a pom that is the parent of an existing pom" do
-      parameter "POM", "a pom file path or URI"
-      def execute
-        checking_exceptions do
-          puts Gjp::ParentPomGetter.new.get_parent_pom(pom)
-        end
-      end
-    end
-      
-    subcommand "get-source-address", "Retrieves a project's SCM Internet address" do
+    subcommand "get-source", "Attempts to retrieve a project's sources" do
       parameter "POM", "a pom file path or URI"
 
       def execute
         checking_exceptions do
-          puts Gjp::SourceAddressGetter.new.get_source_address(pom)
+          project = Gjp::Project.new(".")
+          source_getter = Gjp::SourceGetter.new
+
+          puts "Attempting to find source through Maven..."
+          if source_getter.get_maven_source_jar(project, pom)
+            puts "Source jar found and added to Maven repository."
+          else
+            effective_pom_path = Gjp::MavenRunner.new(project).get_effective_pom(pom)
+            puts "Source jar not found in Maven. Try looking here:"
+            pom = Gjp::Pom.new(effective_pom_path)
+            if pom.url && !pom.url.empty?
+              puts "Website: #{pom.url}"
+            end
+            if pom.scm_connection && !pom.scm_connection.empty?
+              puts "SCM connection: #{pom.scm_connection}"
+            end
+            if pom.scm_url && !pom.scm_url.empty?
+              puts "SCM connection: #{pom.scm_url}"
+            end
+            puts "The effective POM: #{effective_pom_path}"
+            puts "Google: http://google.com/#q=#{URI::encode(pom.name + ' sources')}"
+          end
         end
       end
     end
-    
-    subcommand "get-source", "Retrieves a project's source code directory" do
-      parameter "ADDRESS", "project's SCM Internet address"
-      parameter "POM", "project's pom file path or URI"
-      parameter "[DIRECTORY]", "directory in which to save the source code", :default => "."
-
-      def execute
-        checking_exceptions do
-          puts Gjp::SourceGetter.new.get_source(address, pom, directory)
-        end    
-      end    
-    end
 
     private
 

data/lib/gjp/kit_runner.rb

@@ -27,51 +27,11 @@ module Gjp
       nil
     end
 
-    # runs an external executable
+    # runs an external executable, returns true on success
     def run_executable(full_commandline)
       log.debug "running #{full_commandline}"
       Process.wait(Process.spawn(full_commandline))
-      $?
-    end
-
-    # runs mvn in a subprocess
-    def mvn(options)
-      run_executable "#{get_maven_commandline(@project.full_path)} #{options.join(' ')}"
-    end
-
-    # returns a command line for running Maven from the specified
-    # prefix
-    def get_maven_commandline(prefix)
-      executable = find_executable("mvn")
-
-      if executable != nil
-        mvn_path = File.join(prefix, executable)
-        repo_path = File.join(prefix, "kit", "m2")
-        config_path = File.join(prefix, "kit", "m2", "settings.xml")
-
-        "#{mvn_path} -Dmaven.repo.local=#{repo_path} -s#{config_path}"
-      else
-        raise ExecutableNotFoundError.new("mvn")
-      end
-    end
-
-    # runs mvn in a subprocess
-    def ant(options)
-      run_executable "#{get_ant_commandline(@project.full_path)} #{options.join(' ')}"
-    end
-
-    # returns a command line for running Ant from the specified
-    # prefix
-    def get_ant_commandline(prefix)
-      executable = find_executable("ant")
-
-      if executable != nil
-        ant_path = File.join(prefix, executable)
-
-        "#{ant_path}"
-      else
-        raise ExecutableNotFoundError.new("ant")
-      end
+      $?.exitstatus == 0
     end
   end