gjp 0.15.7 → 0.16.0

data/README.md

@@ -1,44 +1,204 @@
-gjp – Green Java Packager's tools
-===
+# gjp – Green Java Packager's tools
 
 `gjp` (pronounced _/ˈdʒiː ˈaɪ ˈd͡ʒəʊ/_) is a set of tools to ease and partially automate Linux packaging of Java projects.
 
-The project focus is on producing rpm packages for SUSE distributions, but it is general enough to be useful even for other distributions.
+The project objective is to strongly reduce manual packaging efforts by enabling a new and much simpler workflow.
+
+## Status
+
+`gjp` is a research project currently in alpha state. Basic concepts seem to be viable, packages are currently being built with the new approach to identify problem areas but not all basic features have been coded yet. If you are a packager you can try to use it (any feedback would be **very** welcome!), but be warned that anything can still change at this point. 
+
+## Contact
+
+Are you using `gjp` or even just reading this? Let's get in touch!
+
+    smoioli at suse dot de
 
 
 ## Install
 
-Easiest install is via RubyGems:
+Via RubyGems:
 
     $ gem install gjp
 
-## Usage
-
-Main workflow subcommands:
-* `gjp init` inits a new gjp project in the current directory, generating minimal directories and files;
-* `gjp gather` starts a gathering phase, to add source and kit files. You should place source files in src/<package name> and binary dependency files in kit/;
-* `gjp dry-run` starts a dry-run phase, to attempt a build. Any change to src/ will be reverted after you call `gjp finish`;
-* `gjp mvn` during a dry run, locates and runs Maven from any directory in kit/, using options to force repository in kit/m2 and settings in kit/m2/settings.xml;
-* `gjp status` prints the current phase;
-* `gjp finish` ends the current phase;
-* `gjp generate-kit-spec` creates or refreshes a spec file for the kit. Use after `gjp finish`;
-* `gjp generate-kit-archive` creates or refreshes an archive file for the kit. Use after `gjp finish`;
-* `gjp generate-package-spec NAME POM` creates or refreshes a spec file for the package in src/<NAME>. Use after `gjp finish`;
-* `gjp generate-package-archive NAME` creates or refreshes an archive file for package in src/<NAME>. Use after `gjp finish`;
-
-Optional workflow subcommands:
-* `gjp set-up-nonet-user` sets up a user named `nonet` without Internet access you can use for networkless dry runs. Requires `iptables` and 
-superuser privileges;
-* `gjp tear-down-nonet-user` removes a user previously created by gjp;
-
-Other available tools:
+Note: `gjp` requires `git` in order to work. Some non-essential subcommands also need `iptables`, `sudo` and a JDK.
+
+## Workflow
+
+Building a package with `gjp` is quite unusual — this is a [deliberate choice](#motivation) to minimize packaging efforts.
+
+### Overview
+
+The basic process is:
+
+* a `gjp` project is created;
+* sources are added to the project, `gjp` keeps track of them;
+* any other file that is needed for the build, except the JDK, is added in binary form (jars, the Maven executable, its plugins,  etc.). Again, `gjp` keeps track of what you add, noting that those were binary build dependencies and not sources;
+* a build is attempted. After the build is successful, `gjp` notes what files were produced and restores sources in their original state, making it a "repeatable dry-run build". `gjp` also retains any files that were automatically downloaded by Maven or other similar tools during the dry-run as binary build dependencies;
+* `gjp` produces spec files for two packages: one for the project itself and one for all of its binary build dependencies, called a **kit**;
+* kit and project packages can be submitted to [OBS](http://en.opensuse.org/openSUSE:Build_Service). Project package will rebuild cleanly because it needs no Internet access - all files were already downloaded during the dry-run above and are included in the kit.
+
+Note that:
+
+* the project's build time dependency graph is very simple: just its kit and the JDK;
+* the kit is basically a binary blob. If its sources are needed for proper packaging, for example to comply with the GPL, [a separate step](#kit-sources) is needed to add them;
+* the kit is needed at build time only (by OBS), no end user should ever install it;
+* a `gjp` project can be used to build a number of packages that share one binary kit. This can help if the kit becomes big in size;
+* `gjp` will take advantage of Maven's pom files to generate its specs if they are available. This allows to precompile most spec fields automatically.
+
+In `gjp`, the build process can be in one of the following phases at any given moment:
+
+* gathering: in this phase you add sources and kit files. New projects start in this phase, you can always enter it later with `gjp gather`;
+* dry-running: in this phase you attempt a build. Any change in the sources will be reverted after it ends, while files added to the kit will be retained. You can enter this phase with `gjp dry-run`;
+* finishing: in this phase `gjp` generates specs and archive files. You can enter it running `gjp finish`;
+
+### Sample project (commons-io)
+
+Ceate a new `gjp` project, in this example named "galaxy":
+
+    mkdir galaxy
+    cd galaxy
+    gjp init
+
+As you can see from the output, `gjp init` starts a new gathering phase in which you can add sources and kit files. It also generated a folder structure and assumes you respect it, in particular, you should place all your projects' source files in `src`. Every `src` subfolder will become a separate package named after the folder itself, so use the following commands to create a `commons-collections` folders and populate it:
+
+    cd src
+    mkdir commons-collections
+    cd commons-collections
+    wget http://archive.apache.org/dist/commons/collections/source/commons-collections-3.2.1-src.zip
+    unzip commons-collections-3.2.1-src.zip
+    rm commons-collections-3.2.1-src.zip
+
+Now let's move to the kit (which, unsurprisingly, should be placed in the `kit` folder). commons-lang needs Maven 3 to build, so we should simply unzip a copy in `kit`:
+
+    cd ../../kit
+    wget http://apache.fastbull.org/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.zip
+    unzip apache-maven-3.1.0-bin.zip
+    rm apache-maven-3.1.0-bin.zip
+    cd ..
+
+Nothing else is needed for a first build. Let's call `gjp dry-run` to let `gjp` know we are building and then call Maven. Note that `gjp mvn` is used instead of plain `mvn`: `gjp` will take care of locating the Maven installation we have in the `kit` and ensure it will store all downloaded files there.
+
+    gjp dry-run
+    cd src/commons-collections/commons-collections-3.2.1-src/
+    gjp mvn package
+    gjp finish
+
+Success! At this point `gjp` took note of all needed files, and restored `src` as it was before the build. This should be sufficient to be able to repeat the build on a machine with no Internet access, but what if we wanted to be 100% sure of that?
+
+`gjp` has a subcommand to setup a `nonet` user without Internet access, courtesy of `iptables`. You can simply retry the build using that user to see if it works. Note that the following commands will alter group permissions to allow both your current user and `nonet` to work on the same files. 
+
+    gjp set-up-nonet-user
+    chmod -R g+rw ../../..
+    gjp dry-run
+    su nonet
+    ping www.google.com #this should fail!
+    gjp mvn package
+    chmod -R g+rw .
+    exit
+
+The above is obviously not mandatory, but it can be useful for debugging purposes.
+
+One last thing before generating packages is to setup a build script. By default `gjp` will generate a spec file which assumes a `build.sh` script in the source folder of your project that contains all commands needed to build the package itself. At the moment, this needs to be done manually, but it will hopefully be automated in a future release.
+
+Let's start a new gathering phase and add that:
+
+    gjp gather
+    vi ../build.sh
+
+Add the following lines:
+
+    #!/bin/sh
+    cd src/commons-collections/commons-collections-3.2.1-src/
+    ../../../kit/apache-maven-3.1.0/bin/mvn -Dmaven.repo.local=`readlink -e ../../../kit/m2` -s`readlink -e ../../../kit/m2/settings.xml` package
+
+Now complete the gathering:
+
+    gjp finish
+    cd ../../..
+
+Note that `build.sh` gets called from the `gjp` project root, hence the `cd` line, and the Maven line was taken directly from `gjp mvn` output above and pasted verbatim.
+
+The following command will generate the kit spec:
+
+    gjp generate-kit-spec
+    less specs/galaxy-kit.spec
+
+Nothing fancy here, just a bunch of binaries installed in a proper location. You can also edit the spec file manually if you want. When you later regenerate it, `gjp` will automatically try to reconcile changes with a [three-way merge](http://en.wikipedia.org/wiki/Three-way_merge#Three-way_merge).
+
+You can also generate the corresponding .tar.xz file with:
+
+    gjp generate-kit-archive
+
+The contents of this file were tracked by `gjp` during gathering and dry-run phases, and are listed in `file_lists/kit`. You can also edit it if you want.
+
+You can then generate the project spec and archive files provided you have a pom file (more formats will be supported in future). In this case:
+
+    gjp generate-package-spec commons-collections src/commons-collections/commons-collections-3.2.1-src/pom.xml
+    gjp generate-package-archive commons-collections
+    less specs/commons-collections.spec
+
+As you can see, this package BuildRequires its kit, contains only the source files and installs any .jar files that were produced during dry runs. Archive is generated from `file_lists/commons-collections_input`, which lists source files. Output files are in `file_lists/commons-collections_output` and are used to compile the `%install` and `%files` sections of the project spec (by default jar files are included, see `gjp generate-package-spec --help`).
+
+    less file_lists/commons-collections_input
+    less file_lists/commons-collections_output
+
+Packages are ready to be submitted to an OBS project. As OBS integration is not yet implemented, refer to OBS documentation to do that.
+
+### Kit sources
+
+If kit sources are needed for license compliance, some extra work is needed. Fortunately, finding jar source files and adding them to the kit is much easier than packaging its contents in proper RPMs!
+
+If the project you are packaging uses Maven, you can ask Maven itself to find source jars for dependencies. Running the following command will add them to the kit:
+
+    gjp mvn dependency:sources
+
+Unfortunately this will not take care of Maven itself, Maven's plugins and their dependencies so some extra work might be needed.
+
+At the moment `gjp`'s supprort to kit source retrieval is limited to the following subcommands:
+
 * `gjp get-pom NAME` will attempt to find a pom. `NAME` can be a jar file on your disk, a project directory, or simply a `name-version` string. `gjp` will get the pom either from the package itself or through search.maven.org using heuristic searching;
 * `gjp get-parent-pom POM` will attempt to download a pom's parent from search.maven.org, where `POM` is a filename or URI;
 * `gjp get-source-address POM` will attempt to find the SCM Internet address of a pom.xml from the file itself or through api.github.com. `POM` can either be a filename or a URI;
 * `gjp get-source POM ADDRESS` downloads the source of a pom.xml's project from its SCM at ADDRESS;
-* `gjp scaffold-jar-table DIRECTORY` looks for jars in the project's DIRECTORY and classifies them as build-time dependencies (b), run-time dependencies (r) or products (p);
 
-## Source
+More comprehensive support is planned in future releases.
+
+You are advised to use [Maven Central](http://search.maven.org/) to search for sources and other information about projects.
+
+### Implementation note
+
+`gjp` internally uses `git` to keep track of files, any gjp project is actually also a `git` repo. Feel free to navigate it, you can commit, push and pull freely as long as the `gjp` tags are preserved.
+
+## Motivation
+
+The Java developer world has packages (jars, wars, ears...), tools ([ant](http://ant.apache.org/), [Maven](http://maven.apache.org/), [Ivy](http://ant.apache.org/ivy/), [Gradle](http://www.gradle.org/)...) and established workflows to handle software distribution while Linux distros have their own ([zypper](http://en.opensuse.org/Portal:Zypper), [yum](http://en.wikipedia.org/wiki/Yellowdog_Updater,_Modified), [apt](http://en.wikipedia.org/wiki/Advanced_Packaging_Tool)...). Since the two communities have different goals and requirements, Linux distros typically want to repackage Java software with their own format, tools and workflows. Reasons range from ease of installation, predictable rebuildability, support to (security) patching, management tools, legal issues, etc.
+
+Unfortunately those two "schemes" became very different over time, so automatic translation/repackaging is not possible except in very simple cases. This leads to a lot of tedious, error-prone manual work that Linux packagers have to do in order to fit the "alien" Java model into distro packaging rules that were thought and optimized with a different ecosystem in mind.
+
+A typical example is packaging any software built by Maven on SUSE distros. A number of pain points arise:
+
+* Maven requires Internet access and downloads precompiled code as part of the build. RPMs have to be built on a standalone machine in [OBS](http://en.opensuse.org/openSUSE:Build_Service), and they should build code from sources they are provided beforehands exclusively;
+* [Maven is basically a plugin container](http://maven.apache.org/plugins/), so hundreds of different plugins have to be installed to build real-life projects (the exact plugin set and their dependencies is determined at build time). While this is no big deal for Java developers, since they get the corresponding jars prebuilt from Maven itself, it is a nightmare for distros, because all shipping code is supposed to be built from scratch and packaged!
+* Maven often uses multiple versions of a same library or plugin during the same build. Usually distros do not maintain more than one version of any given library to reduce maintenance;
+* Maven requires itself in order to build. To be more exact, Maven needs Nexus, which in turn needs Maven and Nexus. To be more exact, its build dependency graph is a very complicated mess with lots of cycles that have to be broken manually.
+
+The current solution in openSUSE is having the packager handle those differences, but this limits the amount of software the community is able to package due to the high effort required to overcome them.
+
+The Fedora community is experimenting with another set of tools, [XMvn](http://mizdebsk.fedorapeople.org/xmvn/site/), which goals are similar to `gjp`'s. 
+
+### Kit rationale
+
+`gjp` simplifies the packaging process mostly because of its use of a binary blob package that contains all build time dependencies for a set of packages called a **kit**.
+
+Building software from a binary blob is unusual for Linux distros, and it surely has some drawbacks. It is anyway believed that benefits outweigh them, in fact using prebuilt software:
+
+* drastically reduces packaging efforts. A very basic and relatively simple package like [commons-lang](http://commons.apache.org/proper/commons-lang/) needs about [150 jars](https://build.opensuse.org/package/show/home:SilvioMoioli/galaxy-kit) just to be compiled and tested. Those should be packaged, roughly, one-by-one!
+* is just the way all Java developers out there build, test and use their software — this is how they expect it to work. Any different approach is necessarily error-prone and could result in unexpected bugs;
+* does not affect the ability of providing patches to Java projects, as only build time requirements are in the kit. In virtually all cases patching a piece of software does not require to patch its build toolchain;
+* does not affect the ability of complying to software licenses like the GPL. In fact those licenses only require to redistribute a project's source code - not the whole toolchain needed to build it. [Sources can be added](#kit-sources) for GPL'd parts of the kit, if any.
+
+## Sources
 
 `gjp`'s Git repo is available on GitHub, which can be browsed at:
 

data/lib/gjp/package_spec_adapter.rb

@@ -20,16 +20,19 @@ module Gjp
     def initialize(project, package_name, pom, filter)
       @name = package_name
       @version = pom.version
-      @license = pom.license_name
-      clean_description = pom.description.gsub(/[\s]+/, ' ').strip
-      @summary = clean_description[0..60].gsub(/\s\w+$/, '...')
+      @license = if pom.license_name != ""
+        pom.license_name
+      else
+        "Apache-2.0"
+      end
+      @summary = cleanup_description(pom.description, 60)
       @url = pom.url
       @project_name = project.name
       @group_id = pom.group_id
       @artifact_id = pom.artifact_id
       @version = pom.version
       @runtime_dependency_ids = pom.runtime_dependency_ids
-      @description = clean_description
+      @description = cleanup_description(pom.description, 1500)
 
       output_list = File.join(project.full_path, "file_lists", "#{@name}_output")
       @outputs = File.open(output_list).readlines.map do |line|
@@ -42,6 +45,15 @@ module Gjp
     def get_binding
       binding
     end
+
+    def cleanup_description(raw, max_length)
+      raw
+        .gsub(/[\s]+/, " ")
+        .strip
+        .slice(0..max_length -1)
+        .sub(/\s\w+$/, "")
+        .sub(/\.+$/, "")
+    end
   end
 end
 

data/lib/gjp/version.rb

@@ -1,5 +1,5 @@
 # encoding: UTF-8
 
 module Gjp
-  VERSION = "0.15.7"
+  VERSION = "0.16.0"
 end

data/lib/template/kit.spec

@@ -24,8 +24,8 @@ Url:            https://github.com/SilvioMoioli/gjp
 Group:          Development/Libraries/Java
 Source0:        %{name}.tar.xz
 BuildRoot:      %{_tmppath}/%{name}-%{version}-build
-BuildRequires:  java-devel
 BuildArch:      noarch
+BuildRequires:  xz
 Provides:       gjp(kit)
 # no two kits should ever be installed at any given time
 Conflicts:      otherproviders(gjp(kit))
@@ -50,7 +50,6 @@ cp -a * %{buildroot}%{_datadir}/gjp/%{name}/
 %files
 %defattr(-,root,root)
 %doc ../README.SUSE
-%{_datadir}/gjp
-%{_datadir}/gjp/%{name}/
+%{_datadir}/gjp/
 
 %changelog

data/lib/template/package.spec

@@ -24,6 +24,8 @@ Url:            <%= url %>
 Group:          Development/Libraries/Java
 Source0:        %{name}.tar.xz
 BuildRoot:      %{_tmppath}/%{name}-%{version}-build
+BuildRequires:  xz
+BuildRequires:  java-devel
 BuildRequires:  <%= project_name %>-kit
 BuildArch:      noarch
 Provides:       mvn(<%= group_id %>:<%= artifact_id %>) == <%= version %>
@@ -32,15 +34,20 @@ Requires:       mvn(<%= dependency_id[0] %>:<%= dependency_id[1] %>) <% if depen
 <% end %>
 
 %description
-<%= description %>
+<%=
+  description
+%>
 
 %prep
-%setup -q -c
+%setup -q -c -n src/<%= name %>
+ln -sf %{_datadir}/gjp/<%= project_name %>-kit ../../kit
 
 %build
-./build.sh
+cd ../../
+sh src/<%= name %>/build.sh
 
 %install
+mkdir -p %{buildroot}%{_javadir}
 <% outputs.each do |output| %>
 cp -a <%= output %> %{buildroot}%{_javadir}/<%= File.basename(output) %>
 <% end %>
@@ -48,7 +55,7 @@ cp -a <%= output %> %{buildroot}%{_javadir}/<%= File.basename(output) %>
 %files
 %defattr(-,root,root)
 <% outputs.each do |output| %>
-cp -a <%= output %> %{buildroot}%{_javadir}/<%= File.basename(output) %>
+%{_javadir}/<%= File.basename(output) %>
 <% end %>
 
 %changelog

data/spec/lib/scaffolder_spec.rb

@@ -115,7 +115,7 @@ describe Gjp::Scaffolder do
         spec_lines = File.readlines(File.join("specs", "test.spec"))
         spec_lines.should include("Name:           test\n")
         spec_lines.should include("License:        The Apache Software License, Version 2.0\n")
-        spec_lines.should include("Summary:        Nailgun is a client, protocol, and server for running Java...\n")
+        spec_lines.should include("Summary:        Nailgun is a client, protocol, and server for running Java\n")
         spec_lines.should include("Url:            http://martiansoftware.com/nailgun\n")
         spec_lines.should include("BuildRequires:  #{@project.name}-kit\n")
         spec_lines.should include("Provides:       mvn(com.martiansoftware:nailgun-all) == 0.9.1\n")

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: gjp
 version: !ruby/object:Gem::Version
-  version: 0.15.7
+  version: 0.16.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-03 00:00:00.000000000 Z
+date: 2013-10-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake