rhack 0.4.0 → 0.4.1

data/LICENSE

@@ -1,52 +1,20 @@
 Copyright (c) 2010-2013
     Sergey Baev <tinbka@gmail.com>
-Curb is free software licensed under the following terms:
 
-  1. You may make and give away verbatim copies of the source form of the
-     software without restriction, provided that you duplicate all of the
-     original copyright notices and associated disclaimers.
+This work is licensed under the same license as Ruby language.
+    
+This licence also implies a compliance with details of CURB-LICENSE.
 
-  2. You may modify your copy of the software in any way, provided that
-     you do at least ONE of the following:
+THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
 
-       a) place your modifications in the Public Domain or otherwise
-          make them Freely Available, such as by posting said
-	  modifications to Usenet or an equivalent medium, or by allowing
-	  the author to include your modifications in the software.
 
-       b) use the modified software only within your corporation or
-          organization.
-
-       c) give non-standard binaries non-standard names, with
-          instructions on where to get the original software distribution.
-
-       d) make other distribution arrangements with the author.
-
-  3. You may distribute the software in object code or binary form,
-     provided that you do at least ONE of the following:
-
-       a) distribute the binaries and library files of the software,
-	  together with instructions (in the manual page or equivalent)
-	  on where to get the original distribution.
-
-       b) accompany the distribution with the machine-readable source of
-	  the software.
-
-       c) give non-standard binaries non-standard names, with
-          instructions on where to get the original software distribution.
-
-       d) make other distribution arrangements with the author.
-
-  4. You may modify and include the part of the software into any other
-     software (possibly commercial).
-
-  5. The scripts and library files supplied as input to or produced as 
-     output from the software do not automatically fall under the
-     copyright of the software, but belong to whomever generated them, 
-     and may be sold commercially, and may be aggregated with this
-     software.
-
-  6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
-     IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
-     WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-     PURPOSE.

data/Manifest.txt

@@ -1,27 +1,19 @@
 ext/curb/curb_errors.c
 ext/curb/curb_errors.h
-ext/curb/curb_errors.o
 ext/curb/Makefile
 ext/curb/curb_macros.h
 ext/curb/curb_multi.c
 ext/curb/curb_multi.h
-ext/curb/curb_multi.o
 ext/curb/curb_upload.c
 ext/curb/curb_upload.h
-ext/curb/curb_upload.o
 ext/curb/curb_config.h
 ext/curb/extconf.rb
 ext/curb/curb.c
 ext/curb/curb.h
-ext/curb/curb.o
-ext/curb/curb_core.so
 ext/curb/curb_easy.c
 ext/curb/curb_easy.h
-ext/curb/curb_easy.o
-ext/curb/mkmf.log
 ext/curb/curb_postfield.c
 ext/curb/curb_postfield.h
-ext/curb/curb_postfield.o
 ext/curb-original/curb_errors.c
 ext/curb-original/curb_errors.h
 ext/curb-original/curb_macros.h
@@ -61,9 +53,8 @@ test/test_scout.rb
 test/test_frame.rb
 ./Gemfile.lock
 ./LICENSE
+./README.md
 ./Rakefile
 ./Manifest.txt
 ./CURB-LICENSE
-./README.txt
-./Gemfile
-./History.txt
+./Gemfile

data/README.md

@@ -0,0 +1,76 @@
+### RHACK
+[github](https://github.com/tinbka/rhack)
+
+RHACK is Ruby Http ACcess Kit -- curl-based web-client for developing web-scrapers/bots.
+
+##### Features
+* Asynchronous, still EventMachine independent. Synchronization can be turned on with 1sec per waiting penalty
+* Fast as on simple queries as on high load. Can process up to thousands (limited by a net interface, of course) different parallel requests of any HTTP method with no penalty
+* Flexibly configurable on 3 levels:
+* * Curl::Easy (simplest request configuration, inherited from [curb gem](http://github.com/taf2/curb))
+* * ::Scout (Curl::Easy wrapper with transparent cookies and extendable anonimization processing, detailed request/response info, callbacks and retry configuration)
+* * ::Frame (Scout array wrapper with smart request interpretor, load balancing and extendable response processor)
+* Included support of javascript processing on loaded pages (johnson gem)
+* Web-service abstraction for creating scrapers, that implement some examples of how to use this library
+
+---
+
+It's still randomly documented since it's just my working tool.
+
+#### Expected to complete:
+
+* Redis-based configurable cache for ::Service and for downloads
+* Full javascript processing, including linked scripts; maybe support of other javascript engines gems
+
+### CHANGES
+
+##### Version 0.4.1
+
+* Сhanged ::Frame @static behaviour, :static option now accept hash with :procotol key (::Frame#validate comment)
+* Changed log level in curl-global.rb
+* Described the library and *marked down* this readme
+
+##### Version 0.4
+
+* Fixed bugs
+* * idle execution in Rails application thread
+* * Curl::Easy default callback
+* * some misspelling-bugs
+* Minified ::ScoutSquad#next waiting time
+* ::Service
+* * added meta-methods #login (sync only) and #scrape!(<::Page>)
+* ::Frame
+* * made new cache prototype. Call #use_cache!(false?) for (in)activate and #drop_cache! for clearance
+* * added :xhr exec option
+* ::Page
+* * #title returns full title by default
+* * #html is auto-encoded to UTF-8 during #process
+
+##### Version 0.3
+
+* Adjusted cookie processor in accordance with web-servers and entrust redirection process to ::Scout
+* Added some shortcuts to ::Frame and Curl modules
+* Сonfig defaults are now taken from rails
+* Removed crappy database usage from lib/words.rb
+* curb_multi.c: Moved callbacks out of rb_rescue so that I could know wtf was happen there
+
+##### Version 0.2
+
+* Nastily pulled down curb-0.8.1 extension sources and harshly patched by changes made long before, so that the core will be as modern as possible and with necessary features
+* Fixed syntax for Ruby 1.9
+
+##### Version 0.1
+
+* A long time ago in a galaxy far, far away...
+* A library had been created based on Net::HTTP
+* In a few months its base had been changed by curb-0.4.4 because of poorness and incovinience of Net::HTTP
+* Had been made background-mode for Curl::Multi and multipart body setting for Curl::Easy so that Curl could be both sync and async
+* Had been added a couple of wrappers for Curl::Easy and its results, proxy lists processor, scrapers for a few web-services, and plugin for libxml-ruby that lives at rmtools gem now
+
+### License
+
+RHACK is copyright (c) 2010-2013 Sergey Baev <tinbka@gmail.com>, and released under the terms of the Ruby license. 
+See the LICENSE file for the details. 
+Rhack is also include slightly modified Curb gem extension source code. For original 
+Curb gem code you may want to check ext/curb-original or visit <http://github.com/taf2/curb/tree/master>.
+See the CURB-LICENSE file for the details. 

data/Rakefile

@@ -8,14 +8,14 @@ rescue LoadError
 end
 
 compile_manifest
-RHACK_VERSION = '0.4.0'
+RHACK_VERSION = '0.4.1'
 begin
     require 'hoe'
     config = Hoe.spec "rhack" do |h|
         h.developer("Sergey Baev", "tinbka@gmail.com")
 
-        h.description = 'Webscraping library based on curb gem extension and libxml-ruby (and optionally Johnson and ActiveRecord)'
-       # h.summary = h.description
+        h.description = 'Ruby Http ACcess Kit -- curl-based web-client for developing web-scrapers/bots'
+        h.summary = h.description
         h.urls = ['https://github.com/tinbka/rhack']
        
         h.extra_deps << ['rmtools','>= 1.2.12']

data/ext/curb/Makefile

@@ -12,11 +12,11 @@ ECHO = $(ECHO1:0=@echo)
 #### Start of system configuration section. ####
 
 srcdir = .
-topdir = /home/shinku/.rvm/rubies/ruby-1.9.3-p194/include/ruby-1.9.1
-hdrdir = /home/shinku/.rvm/rubies/ruby-1.9.3-p194/include/ruby-1.9.1
-arch_hdrdir = /home/shinku/.rvm/rubies/ruby-1.9.3-p194/include/ruby-1.9.1/$(arch)
+topdir = /home/shinku/.rvm/rubies/ruby-1.9.3-p327/include/ruby-1.9.1
+hdrdir = /home/shinku/.rvm/rubies/ruby-1.9.3-p327/include/ruby-1.9.1
+arch_hdrdir = /home/shinku/.rvm/rubies/ruby-1.9.3-p327/include/ruby-1.9.1/$(arch)
 VPATH = $(srcdir):$(arch_hdrdir)/ruby:$(hdrdir)/ruby
-prefix = $(DESTDIR)/home/shinku/.rvm/rubies/ruby-1.9.3-p194
+prefix = $(DESTDIR)/home/shinku/.rvm/rubies/ruby-1.9.3-p327
 rubylibprefix = $(libdir)/$(RUBY_BASE_NAME)
 exec_prefix = $(prefix)
 vendorhdrdir = $(rubyhdrdir)/vendor_ruby
@@ -59,21 +59,22 @@ LIBRUBY = $(LIBRUBY_SO)
 LIBRUBY_A = lib$(RUBY_SO_NAME)-static.a
 LIBRUBYARG_SHARED = -Wl,-R -Wl,$(libdir) -L$(libdir) -l$(RUBY_SO_NAME)
 LIBRUBYARG_STATIC = -Wl,-R -Wl,$(libdir) -L$(libdir) -l$(RUBY_SO_NAME)-static
-OUTFLAG = -o 
-COUTFLAG = -o 
+empty =
+OUTFLAG = -o $(empty)
+COUTFLAG = -o $(empty)
 
 RUBY_EXTCONF_H = curb_config.h
 cflags   =  $(optflags) $(debugflags) $(warnflags)
 optflags = -O3
 debugflags = -ggdb
-warnflags = -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration
-CFLAGS   = -fPIC $(cflags)  -fPIC  -g -Wall $(ARCH_FLAG)
+warnflags = -Wall -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration
+CFLAGS   = -fPIC -I/home/shinku/.rvm/usr/include -fPIC  -g -Wall $(ARCH_FLAG)
 INCFLAGS = -I. -I$(arch_hdrdir) -I$(hdrdir)/ruby/backward -I$(hdrdir) -I$(srcdir)
 DEFS     = -D_FILE_OFFSET_BITS=64
-CPPFLAGS = -DRUBY_EXTCONF_H=\"$(RUBY_EXTCONF_H)\" -I/home/shinku/.rvm/usr/include  $(DEFS) $(cppflags)
+CPPFLAGS = -DRUBY_EXTCONF_H=\"$(RUBY_EXTCONF_H)\"  $(DEFS) $(cppflags) -I/home/shinku/.rvm/usr/include
 CXXFLAGS = $(CFLAGS) $(cxxflags)
-ldflags  = -L.  -rdynamic -Wl,-export-dynamic
-dldflags = 
+ldflags  = -L. -L/home/shinku/.rvm/usr/lib -rdynamic -Wl,-export-dynamic -L/home/shinku/.rvm/usr/lib  -Wl,-R/home/shinku/.rvm/usr/lib 
+dldflags = -L/home/shinku/.rvm/usr/lib  -Wl,-R/home/shinku/.rvm/usr/lib 
 ARCH_FLAG = 
 DLDFLAGS = $(ldflags) $(dldflags) $(ARCH_FLAG)
 LDSHARED = $(CC) -shared
@@ -87,7 +88,7 @@ RUBY_SO_NAME = ruby
 arch = i686-linux
 sitearch = $(arch)
 ruby_version = 1.9.1
-ruby = /home/shinku/.rvm/rubies/ruby-1.9.3-p194/bin/ruby
+ruby = /home/shinku/.rvm/rubies/ruby-1.9.3-p327/bin/ruby
 RUBY = $(ruby)
 RM = rm -f
 RM_RF = $(RUBY) -run -e rm -- -rf
@@ -97,6 +98,7 @@ INSTALL = /usr/bin/install -c
 INSTALL_PROG = $(INSTALL) -m 0755
 INSTALL_DATA = $(INSTALL) -m 644
 COPY = cp
+TOUCH = exit >
 
 #### End of system configuration section. ####
 
@@ -115,8 +117,8 @@ extout_prefix =
 target_prefix = 
 LOCAL_LIBS = 
 LIBS = $(LIBRUBYARG_SHARED)  -lpthread -lrt -ldl -lcrypt -lm   -lc -lcurl
-SRCS = curb_easy.c curb_upload.c curb.c curb_multi.c curb_errors.c curb_postfield.c
-OBJS = curb_easy.o curb_upload.o curb.o curb_multi.o curb_errors.o curb_postfield.o
+SRCS = curb.c curb_errors.c curb_multi.c curb_upload.c curb_easy.c curb_postfield.c
+OBJS = curb.o curb_errors.o curb_multi.o curb_upload.o curb_easy.o curb_postfield.o
 TARGET = curb_core
 DLLIB = $(TARGET).so
 EXTSTATIC = 
@@ -124,8 +126,8 @@ STATIC_LIB =
 
 BINDIR        = $(bindir)
 RUBYCOMMONDIR = $(sitedir)$(target_prefix)
-RUBYLIBDIR = /home/shinku/.rvm/gems/ruby-1.9.3-p194@global/gems/rhack-0.3.3/lib$(target_prefix)
-RUBYARCHDIR = /home/shinku/.rvm/gems/ruby-1.9.3-p194@global/gems/rhack-0.3.3/lib$(target_prefix)
+RUBYLIBDIR = /home/shinku/.rvm/gems/ruby-1.9.3-p327/gems/rhack-0.4.0/lib$(target_prefix)
+RUBYARCHDIR = /home/shinku/.rvm/gems/ruby-1.9.3-p327/gems/rhack-0.4.0/lib$(target_prefix)
 HDRDIR        = $(rubyhdrdir)/ruby$(target_prefix)
 ARCHHDRDIR    = $(rubyhdrdir)/$(arch)/ruby$(target_prefix)
 
@@ -155,19 +157,21 @@ distclean: clean distclean-so distclean-rb-default distclean-rb
 realclean: distclean
 install: install-so install-rb
 
-install-so: $(RUBYARCHDIR)
 install-so: $(RUBYARCHDIR)/$(DLLIB)
 $(RUBYARCHDIR)/$(DLLIB): $(DLLIB)
-	@-$(MAKEDIRS) $(@D)
+	-$(Q)$(MAKEDIRS) $(@D)
 	$(INSTALL_PROG) $(DLLIB) $(@D)
+clean-static::
+	-$(Q)$(RM) $(STATIC_LIB)
 install-rb: pre-install-rb install-rb-default
 install-rb-default: pre-install-rb-default
 pre-install-rb: Makefile
 pre-install-rb-default: Makefile
 pre-install-rb-default:
 	$(ECHO) installing default curb_core libraries
-$(RUBYARCHDIR):
-	$(Q) $(MAKEDIRS) $@
+./.RUBYARCHDIR.time:
+	$(Q) $(MAKEDIRS) $(RUBYARCHDIR)
+	$(Q) $(TOUCH) $@
 
 site-install: site-install-so site-install-rb
 site-install-so: install-so
@@ -205,7 +209,7 @@ site-install-rb: install-rb
 
 $(DLLIB): $(OBJS) Makefile
 	$(ECHO) linking shared-object $(DLLIB)
-	@-$(RM) $(@)
+	-$(Q)$(RM) $(@)
 	$(Q) $(LDSHARED) -o $@ $(OBJS) $(LIBPATH) $(DLDFLAGS) $(LOCAL_LIBS) $(LIBS)
 
 

data/lib/curl-global.rb

@@ -3,13 +3,13 @@ module Curl
   
   def execute(unless_allready=false)
     if unless_allready and Curl.status
-      return L.debug "Non-nil status! Avoid executing"
+      return L.log "Non-nil status! Avoid executing"
     end
     if $CarierThread and s = $CarierThread.status
-      L.debug "Carier thread allready started and has status #{s}"
+      L.log "Carier thread allready started and has status #{s}"
     else
       if s = Curl.status(false) then L.warn s end
-      L.debug($CarierThread ? "Resetting Carier thread" : "Setting Carier thread up")
+      L.log($CarierThread ? "Resetting Carier thread" : "Setting Carier thread up")
       $CarierThread = Thread.new {
         error = nil
         begin
@@ -22,7 +22,7 @@ module Curl
           begin
               # with true argument (idle) it would break only if no requests to perform
             break unless $Carier.perform true
-            L.debug "Nothing to perform; idling..."
+            L.log "Nothing to perform; idling..."
           rescue => error
             break
               # but ruby mystically crashes if next sequence occur:
@@ -44,7 +44,7 @@ module Curl
   
   def wait
     if $CarierThread and $CarierThread.status
-      if !(within = Thread.current == $CarierThread)
+      unless within = Thread.current == $CarierThread
         # We can't set `perform' timeout lesser than 1 second in the curl binding
         # because in that case thread status would always be "run"
         # so here we wait for exactly 1 sec
@@ -52,13 +52,14 @@ module Curl
       end
       # Also, if thread do Kernel.sleep, it would skip Curl.wait here
       if !$Carier.sheduled and ($CarierThread.status == 'sleep' or within && $Carier.reqs.empty?)
-        L.debug "No shedule to wait"
+        L.log "No shedule to wait"
       else
-        L.log "Waiting for Carier to complete in #{within ? 'it\'s thread' : Thread.main == Thread.current ? 'main thread' : 'thread '+Thread.current.object_id}"
+        this_thread = within ? 'it\'s thread' : Thread.main == Thread.current ? 'main thread' : 'thread '+Thread.current.object_id
+        L.log "Waiting for Carier to complete in #{this_thread}"
         begin
-          L.log { "Trying to change $CarierThreadIsJoined #{$CarierThreadIsJoined} -> true from #{within ? 'it\'s thread' : Thread.main == Thread.current ? 'main thread' : 'thread '+Thread.current.object_id}" }
+          L.log { "Trying to change $CarierThreadIsJoined #{$CarierThreadIsJoined} -> true from #{this_thread}" }
           if within 
-            L.debug "calling this from one of callbacks to wait for the rest to complete"
+            L.log "calling this from one of callbacks to wait for the rest to complete"
             begin
               $Carier.perform
             rescue RuntimeError => e
@@ -74,12 +75,16 @@ module Curl
           recall!
           L.info "Carier thread recalled by keyboard"
         ensure
-          L.log "trying to change $CarierThreadIsJoined #{$CarierThreadIsJoined} -> false from #{within ? 'it\'s' : 'main'} thread"
+          L.log "trying to change $CarierThreadIsJoined #{$CarierThreadIsJoined} -> false from #{this_thread}"
           if !within
             $CarierThreadIsJoined = false
+            # using Curl#execute from different threads may cause problems here when you don't control input,
+            # for example, in a daemonized ruby process
+            # just do not get $CarierThread joined from non-main thread
             if $CarierThread and e = $CarierThread.value
               # this will raise thread-safely in main thread
               # in case of unrescued error in CarierThread
+              L.log(([e.message]+RMTools.format_trace(e.backtrace))*"\n")
               recall!
               raise e 
             end
@@ -98,23 +103,23 @@ module Curl
   def recall
     L.debug caller
     if $CarierThread
-      L.debug "Recalling Carier thread"
+      L.log "Recalling Carier thread"
       $CarierThread.kill
       sleep 1
     else
-      L.debug "No thread to recall"
+      L.log "No thread to recall"
     end
   end
   alias :stop :recall
   
   def recall!
     if $CarierThread
-      L.info "Recalling thread and resetting Carier!!!"
+      L.warn "Recalling thread and resetting Carier!!!"
       $CarierThread.kill
       $CarierThread = nil
       $Carier.reset
     else
-      L.debug "No thread to recall!"
+      L.log "No thread to recall!"
     end
   end
   alias :stop! :recall!
@@ -147,10 +152,10 @@ module Curl
           e
         end
       else
-        L.debug "Carier Thread is exited without error"
+        L.log "Carier Thread is exited without error"
       end
     else
-      L.debug "There is no Carier Thread atm"
+      L.log "There is no Carier Thread atm"
     end
   end
   alias :st :status

data/lib/frame.rb

@@ -34,7 +34,7 @@ module HTTPAccessKit
         'http://' >> uri if uri !~ /^\w+:\/\//
         @loc = uri.parse:uri
         # be careful, if you set :static => false, frame will be unable to use implicit url
-        @static = @opts.fetch(:static, true).b
+        @static = @opts.fetch(:static, true)
       else
         @loc = {}
         @static = false
@@ -76,7 +76,7 @@ module HTTPAccessKit
     end
     
     def inspect
-      "<#Frame @ #{@ss.untargeted ? 'no target' : @loc.root}: #{'scout'.x @ss.size}#{', static' if @static}, cookies #{@ss[0].cookieProc ? 'on' : 'off'}>"
+      "<#Frame @ #{@ss.untargeted ? 'no target' : @loc.root}: #{'scout'.x @ss.size}#{', static'+(' => '+@static.protocol if @static.is(Hash)) if @static}, cookies #{@ss[0].cookieProc ? 'on' : 'off'}>"
     end
     
     # opts are :eval, :json, :hash, :wait, :proc_result, :save_result, :load_scripts, 
@@ -426,15 +426,28 @@ module HTTPAccessKit
       end
     end
     
+    # :static option now can accept hash with :procotol key, in that case Frame can be relocated to the same domain on another protocol and default protocol would be the value of @static.protocol
     def validate(uri)
       if uri
         loc = uri.parse:uri
         if loc.root and loc.root != @loc.root
-          raise TargetError, "failed to get #{uri} by static frame #{@loc.host}, you should first update it with new target" if @static 
-          @loc.root = loc.root
+          if @static
+            if @static.is Hash
+              if loc.host != @loc.host
+                raise TargetError, "unable to get #{uri} by static frame [#{@static.protocol}://]#{@loc.host}, you should first update it with new target"
+              end
+            else
+              raise TargetError, "unable to get #{uri} by static frame #{@loc.root}, you should first update it with new target"
+            end
+          end
+          @loc.root, @loc.host, @loc.protocol = loc.root, loc.host, loc.protocol
           uri
         elsif !loc.root
           raise TargetError if !@static
+          if @static.is Hash
+            @loc.protocol = @static.protocol
+            @loc.root = @loc.protocol+'://'+@loc.host
+          end
           File.join @loc.root, uri
         else uri
         end