gjp 0.30.0 → 0.31.0

data/README.md

@@ -114,7 +114,13 @@ The remaining (mostly very outdated) jars will be listed by `gjp` when the downl
 
 to get pointers to relevant sites if available in the pom itself.
 
-A list of commonly used jars can be found [below](#frequently-used-jar-sources).
+A list of commonly used jars can be found [below](#frequently-used-sources).
+
+You can also use:
+
+    gjp list-kit-missing-sources
+
+To get a list of jars that have one or more `.class` file which does not have a corresponding `.java` file in `kit/` (or zip files in `kit/`).
 
 ### Ant builds
 
@@ -158,11 +164,27 @@ If you want to be 100% sure your package builds without network access, you can
 
 * if packages build at first but then fail after a few days because Maven tries to connect to the Internet, add the `--option` flag to the `mvn` line in `build.sh`;
 
-### Frequently used jar sources
-
-* ant-launcher-1.6.5: is included in ant-1.6.5 sources, you most prbably have it already in `kit/m2/ant/ant/1.6.5`;
-* doxia-sink-api-1.0-alpha-4: use `svn checkout http://svn.apache.org/repos/asf/maven/doxia/doxia/tags/doxia-sink-api-1.0-alpha-4/ doxia-sink-api-1.0-alpha-4`;
-* kxml2-2.2.2: use http://downloads.sourceforge.net/project/kxml/kxml2/2.2.2/kxml2-src-2.2.2.zip;
+### Frequently used sources
+
+* antlr-2.7.7: `wget http://www.antlr2.org/download/antlr-2.7.7.tar.gz -O kit/m2/antlr/antlr/2.7.7/antlr-2.7.7-sources.tar.gz`;
+* ant-launcher-1.6.5: sources included in ant-1.6.5;
+* asm-3.3.1: `wget http://download.forge.ow2.org/asm/asm-3.3.1.tar.gz -O kit/m2/asm/asm/3.3.1/asm-3.3.1-sources.tar.gz`;
+* jsr305-1.3.9: sources included in jar;
+* jsr305-2.0.1: sources included in jar;
+* commons-lang-1.0: `wget http://archive.apache.org/dist/commons/lang/source/lang-1.0-src.tar.gz -O kit/m2/commons-lang/commons-lang/1.0/commons-lang-1.0-sources.tar.gz`;
+* commons-logging-api-1.1: `wget http://archive.apache.org/dist/commons/logging/source/commons-logging-1.1-src.tar.gz -O kit/m2/commons-logging/commons-logging/1.1/commons-logging-1.1-sources.tar.gz` (included in commons-logging-1.1);
+* commons-logging-1.0: `wget http://archive.apache.org/dist/commons/logging/source/commons-logging-1.0-src.tar.gz -O kit/m2/commons-logging/commons-logging/1.0/commons-logging-1.0-sources.tar.gz`;
+* dom4j-1.1: `wget http://dom4j.cvs.sourceforge.net/viewvc/dom4j/dom4j/?view=tar&pathrev=dom4j-1-1 -O kit/m2/dom4j/dom4j/1.1/dom4j-1.1-sources.tar.gz`;
+* doxia-sink-api-1.0-alpha-4: use `svn export http://svn.apache.org/repos/asf/maven/doxia/doxia/tags/doxia-sink-api-1.0-alpha-4/ kit/m2/doxia/doxia-sink-api/1.0-alpha-4/doxia-sink-api-1.0-alpha-4-sources`;
+* kxml2-2.2.2: use `wget http://sourceforge.net/projects/kxml/files/kxml2/2.2.2/kxml2-src-2.2.2.zip/download -O kit/m2/net/sf/kxml/kxml2/2.2.2/kxml2-2.2.2-sources.zip`;
+* log4j-1.2.12: `wget http://archive.apache.org/dist/logging/log4j/1.2.12/logging-log4j-1.2.12.tar.gz -O kit/m2/log4j/log4j/1.2.12/log4j-1.2.12-sources.tar.gz`;
+* stringtemplate-3.2: `wget http://www.stringtemplate.org/download/stringtemplate-3.2.tar.gz -O kit/m2/org/antlr/stringtemplate/3.2/stringtemplate-3.2-sources.tar.gz`;
+* velocity-tools-2.0: `wget http://archive.apache.org/dist/velocity/tools/2.0/velocity-tools-2.0-src.tar.gz -O kit/m2/org/apache/velocity/velocity-tools/2.0/velocity-tools-2.0-sources.tar.gz`;
+* velocity-1.5: `wget http://archive.apache.org/dist/velocity/engine/1.5/velocity-1.5.tar.gz -O kit/m2/org/apache/velocity/velocity/1.5/velocity-1.5-sources.tar.gz`;
+* trilead-ssh2-build213-svnkit-1.3: `svn export http://svn.svnkit.com/repos/svnkit/tags/1.3.5/ kit/m2/org/tmatesoft/svnkit/svnkit/1.3.5/svnkit-1.3.5-sources` (included in svnkit 1.3.5 full sources because of custom patch);
+* xercesImpl-2.9.1: `wget http://archive.apache.org/dist/xerces/j/source/Xerces-J-src.2.9.1.tar.gz -O kit/m2/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1-sources.tar.gz`;
+* xmlpull-1.1.3.1: wget `http://www.extreme.indiana.edu/xmlpull-website/v1/download/xmlpull_1_1_3_4c_src.tgz -O kit/m2/xmlpull/xmlpull/1.1.3.1/xmlpull-1.1.3.1-sources.tar.gz`
+* xmlunit-1.3: `wget http://sourceforge.net/projects/xmlunit/files/xmlunit%20for%20Java/XMLUnit%20for%20Java%201.3/xmlunit-1.3-src.zip/download -O kit/m2/xmlunit/xmlunit/1.3/xmlunit-1.3-sources.zip`;
 
 ## Motivation
 

data/lib/gjp.rb

@@ -17,5 +17,6 @@ require "gjp/kit_spec_adapter"
 require "gjp/package_spec_adapter"
 require "gjp/spec_generator"
 require "gjp/archiver"
+require "gjp/kit_checker"
 
 require "gjp/cli"

data/lib/gjp/cli.rb

@@ -339,6 +339,23 @@ module Gjp
       end
     end
 
+    subcommand "list-kit-missing-sources", "Locates jars in kit/ that have no source files" do
+      def execute
+        checking_exceptions do
+          project = Gjp::Project.new(".")
+          kit_checker = Gjp::KitChecker.new(project)
+
+          ensure_dry_running(false, project) do
+            puts "Some source files were not found in these archives:"
+            kit_checker.get_unsourced.each do |archive|
+              percentage = "%.2f" % (100.0 * archive[:unsourced_class_names].length()/archive[:class_names].length())
+              puts "#{format_path(archive[:archive], project)} (~#{percentage}% missing)"
+            end
+          end
+        end
+      end
+    end
+
     private
 
     def ensure_dry_running(state, project)

data/lib/gjp/kit_checker.rb

@@ -0,0 +1,107 @@
+# encoding: UTF-8
+
+require "find"
+
+module Gjp
+  # checks kits for errors
+  class KitChecker
+    include Logger
+
+    def initialize(project)      
+      @project = project
+    end
+
+    # returns an array of [path, archive] couples found in kit/
+    # archive is not nil if path is inside a zip file
+    def kit_file_paths
+      @project.from_directory("kit") do
+        plain_file_paths = Dir[File.join("**", "*")].select do |path|
+          File.file?(path)
+        end.map do |path|
+          [path, nil]
+        end
+
+        archived_file_paths = plain_file_paths.select do |path, archive|
+          path. =~ /\.(zip)|([jwe]ar)$/
+        end.map do |path, archive|
+          result = []
+          Zip::ZipFile.foreach(path) do |entry|
+            if entry.file?
+              result << [entry.to_s, path]
+            end
+          end
+          result
+        end.flatten(1)
+
+        plain_file_paths + archived_file_paths
+      end
+    end
+
+    # returns a list of class names for which
+    # we have source files in kit/
+    def get_source_class_names(paths)
+      source_paths = paths.select do |path, archive|
+        path =~ /\.java$/
+      end
+
+      # heuristically add all possible package names, walking 
+      # back the directory tree all the way back to root.
+      # This could add non-existent names, but allows not looking
+      # in the file at all
+      class_names = source_paths.map do |path, archive|
+        class_name = path_to_class(path)
+        parts = class_name.split(".")
+        last_index = parts.length() -1
+        (0..last_index).map do |i|
+          parts[i..last_index].join(".")
+        end
+      end.flatten
+
+      Set.new(class_names)
+    end
+
+    # returns a list of class names for which
+    # we have binary files in kit/
+    def get_compiled_classes(paths)
+      result = {}
+      compiled_paths = paths.select do |path, archive|
+        path =~ /\.class$/
+      end.each do |path, archive|
+        class_name = path_to_class(path)
+        if result[archive] == nil
+          result[archive] = [class_name]
+        else
+          result[archive] << class_name
+        end
+      end
+
+      result
+    end
+
+    # returns a hash that associates archive names and
+    # the unsourced classes within them
+    def get_unsourced
+      paths = kit_file_paths
+      source_class_names = get_source_class_names(paths)
+      archive_paths_to_class_names = get_compiled_classes(paths)
+
+      result = archive_paths_to_class_names.map do |archive, class_names|
+        unsourced_class_names = class_names.select do |class_name|
+          source_class_names.include?(class_name) == false
+        end
+        {:archive => archive, :class_names => class_names, :unsourced_class_names => unsourced_class_names}
+      end.select do |archive|
+        archive[:unsourced_class_names].any?
+      end
+    end
+
+    private
+    # attempts to turn a .java/.class file name into a class name
+    def path_to_class(path)
+      path
+        .gsub(/\$.+?\./, ".") # remove $InnerClasses
+        .gsub(/\..+?$/, "") # remove .extension
+        .gsub("/", ".") # convert paths in package names
+    end
+  end
+end

data/lib/gjp/version.rb

@@ -1,5 +1,5 @@
 # encoding: UTF-8
 
 module Gjp
-  VERSION = "0.30.0"
+  VERSION = "0.31.0"
 end

data/spec/lib/kit_checker_spec.rb

@@ -0,0 +1,119 @@
+# encoding: UTF-8
+
+require "spec_helper"
+require "rubygems"
+require "zip/zip"
+
+describe Gjp::KitChecker do
+  include Gjp::Mockers
+
+  before(:each) do
+    create_mock_project
+    @kit_checker = Gjp::KitChecker.new(@project)
+  end
+
+  after(:each) do
+    delete_mock_project
+  end
+
+  describe "#kit_file_paths"  do
+    it "returns an array of paths found in kit" do
+      @project.from_directory("kit") do
+        FileUtils.touch("top_level")
+        FileUtils.mkdir_p("directory")
+        FileUtils.touch(File.join("directory","in_directory"))
+        Zip::ZipFile.open("zipped.zip", Zip::ZipFile::CREATE) do |zipfile|
+          Dir[File.join("directory", "**", "**")].each do |file|
+            zipfile.add(file.sub("/directory", ""), file)
+          end
+        end
+      end
+
+      all_files = @kit_checker.kit_file_paths 
+
+      all_files.should include ["top_level", nil]
+      all_files.should_not include ["directory", nil]
+      all_files.should include ["directory/in_directory", nil]
+      all_files.should include ["zipped.zip", nil]
+      all_files.should include ["directory/in_directory", "zipped.zip"]
+    end
+  end
+
+  describe "#get_source_classes"  do
+    it "distills source class names in kit" do
+      all_files = [
+        ["path/to/ClassOne.java", nil],
+        ["path/to/ClassTwo.java", "path/to/archive.jar"],
+        ["ClassThree.java", "another_archive.jar"],
+        ["path/to/CompiledClass.class", "yet_another.jar"],
+      ]
+
+      class_names = @kit_checker.get_source_class_names(all_files)
+
+      class_names.should include "path.to.ClassOne"
+      class_names.should include "path.to.ClassTwo"
+      class_names.should include "ClassThree"
+      class_names.should_not include "path.to.CompiledClass"
+    end
+  end
+
+  describe "#get_compiled_classes"  do
+    it "distills source class names in kit" do
+      all_files = [
+        ["path/to/ClassOne.class", nil],
+        ["path/to/ClassTwo.class", "path/to/archive.jar"],
+        ["ClassThree.class", "another_archive.jar"],
+        ["path/to/SourceClass.java", "yet_another.jar"],
+      ]
+
+      classes = @kit_checker.get_compiled_classes(all_files)
+
+      classes[nil].should include "path.to.ClassOne"
+      classes["path/to/archive.jar"].should include "path.to.ClassTwo"
+      classes["another_archive.jar"].should include "ClassThree"
+      classes["yet_another.jar"].should be_nil
+    end
+  end
+
+  describe "#get_unsourced" do
+    it "returns a list of jars wich source files are missing" do
+      @project.from_directory("kit") do
+        FileUtils.mkdir_p("package1")
+        FileUtils.touch(File.join("package1","UnsourcedClass.class"))
+
+        FileUtils.mkdir_p("package2")
+        FileUtils.touch(File.join("package2","SourcedClass.java"))
+        Zip::ZipFile.open("zipped-source-2.jar", Zip::ZipFile::CREATE) do |zipfile|
+          Dir[File.join("package2", "**", "**")].each do |file|
+            zipfile.add(file.sub("/package2", ""), file)
+          end
+        end
+        FileUtils.rm(File.join("package2","SourcedClass.java"))
+        FileUtils.touch(File.join("package2","SourcedClass.class"))
+        Zip::ZipFile.open("zipped-2.jar", Zip::ZipFile::CREATE) do |zipfile|
+          Dir[File.join("package2", "**", "**")].each do |file|
+            zipfile.add(file.sub("/package2", ""), file)
+          end
+        end
+
+        FileUtils.mkdir_p("package3")
+        FileUtils.touch(File.join("package3","SourcedSameArchive.java"))
+        FileUtils.touch(File.join("package3","SourcedSameArchive.class"))
+        Zip::ZipFile.open("zipped-3.zip", Zip::ZipFile::CREATE) do |zipfile|
+          Dir[File.join("package3", "**", "**")].each do |file|
+            zipfile.add(file.sub("/package3", ""), file)
+          end
+        end
+      end
+
+      unsourced = @kit_checker.get_unsourced
+      unsourced.length().should eq 1
+
+      unsourced.first[:archive].should be_nil
+      unsourced.first[:class_names].should include "package1.UnsourcedClass"
+      unsourced.first[:class_names].should include "package2.SourcedClass"
+      unsourced.first[:class_names].should include "package3.SourcedSameArchive"
+      unsourced.first[:unsourced_class_names].should include "package1.UnsourcedClass"
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: gjp
 version: !ruby/object:Gem::Version
-  version: 0.30.0
+  version: 0.31.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-12-27 00:00:00.000000000 Z
+date: 2013-12-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -161,6 +161,7 @@ files:
 - lib/gjp/archiver.rb
 - lib/gjp/cli.rb
 - lib/gjp/git.rb
+- lib/gjp/kit_checker.rb
 - lib/gjp/kit_runner.rb
 - lib/gjp/kit_spec_adapter.rb
 - lib/gjp/limited_network_user.rb
@@ -204,6 +205,7 @@ files:
 - spec/lib/ant_runner_spec.rb
 - spec/lib/archiver_spec.rb
 - spec/lib/git_spec.rb
+- spec/lib/kit_checker_spec.rb
 - spec/lib/kit_runner_spec.rb
 - spec/lib/limited_network_user_spec_interactive.rb
 - spec/lib/maven_runner_spec.rb
@@ -262,6 +264,7 @@ test_files:
 - spec/lib/ant_runner_spec.rb
 - spec/lib/archiver_spec.rb
 - spec/lib/git_spec.rb
+- spec/lib/kit_checker_spec.rb
 - spec/lib/kit_runner_spec.rb
 - spec/lib/limited_network_user_spec_interactive.rb
 - spec/lib/maven_runner_spec.rb