sinew 2.0.4 → 2.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7e7426a91f427a3c97969eb501ce0bd55a658ece54af0dfea994f8faffd479f7
-  data.tar.gz: baeb12b6af0fa2c5c11390de16ed1837458f52b4b00393dce8f9e66d4eb898a3
+  metadata.gz: b383fb9d0a1d57acfafd78d8e3ff0185b81acb5bcd368d4c0cca9a8999aa0a52
+  data.tar.gz: 0dede0f01c7a53056a38705c6fc134cd33a2f15868a9b4fee1b6f9fa85361d31
 SHA512:
-  metadata.gz: 1698acfc26dbab92c390cde0956a72011f22f8f9bb4c5ebb194d131ae0a8dfe6c58e2224d36e840167210c2bf472efd18fd8e0cc92b1c9640df590f1faf71473
-  data.tar.gz: e9a2688616dd866792cd286a1ebff094fe32ad9d8552ce77e7465648022fd65db83149ee558f88140f907d7a2da422c41312e8a32f73f0d1930f865862eab90f
+  metadata.gz: e867516dd43bed9f6dd524475c70b933b30029788521224eed08d269ea1264f5d52a3740e967a2cc61ec96861e1875ecaf2f43af403bd917cac05ce2fd394119
+  data.tar.gz: 719ab64ac523e9cf553171bf318a59938655e4a40fd4c6544ec5c5188c7a81227a30b9ba7c70a6ace90a41b476a62081f2ac4cc88a490d0be2e05ade7e8b3dce

data/README.md

@@ -20,24 +20,24 @@ gem 'sinew'
 
 <!--- markdown-toc --no-firsth1 --maxdepth 1 readme.md -->
 
-* [Sinew 2](#sinew-2-may-2018)
-* [Quick Example](#quick-example)
-* [How it Works](#how-it-works)
-* [DSL Reference](#dsl-reference)
-* [Hints](#hints)
-* [Limitations](#limitations)
-* [Changelog](#changelog)
-* [License](#license)
+- [Sinew 2](#sinew-2-may-2018)
+- [Quick Example](#quick-example)
+- [How it Works](#how-it-works)
+- [DSL Reference](#dsl-reference)
+- [Hints](#hints)
+- [Limitations](#limitations)
+- [Changelog](#changelog)
+- [License](#license)
 
 ## Sinew 2 (May 2018)
 
 I am pleased to announce the release of Sinew 2.0, a complete rewrite of Sinew for the modern era. Enhancements include:
 
-* Remove dependencies on active_support, curl and tidy. We use HTTParty now.
-* Much easier to customize requests in `.sinew` files. For example, setting User-Agent or Bearer tokens.
-* More operations like `post_json` or the generic `http`. These methods are thin wrappers around HTTParty.
-* New end-of-run report.
-* Tests, rubocop, vscode settings, travis, etc.
+- Remove dependencies on active_support, curl and tidy. We use HTTParty now.
+- Much easier to customize requests in `.sinew` files. For example, setting User-Agent or Bearer tokens.
+- More operations like `post_json` or the generic `http`. These methods are thin wrappers around HTTParty.
+- New end-of-run report.
+- Tests, rubocop, vscode settings, travis, etc.
 
 **Breaking change**
 
@@ -124,72 +124,82 @@ Because all requests are cached, you can run Sinew repeatedly with confidence. R
 
 #### Making requests
 
-* `get(url, query = {})` - fetch a url with HTTP GET. URL parameters can be added using `query.
-* `post(url, form = {})` - fetch a url with HTTP POST, using `form` as the URL encoded POST body.
-* `post_json(url, json = {})` - fetch a url with HTTP POST, using `json` as the POST body.
-* `http(method, url, options = {})` - use this for more complex requests
+- `get(url, query = {})` - fetch a url with HTTP GET. URL parameters can be added using `query.
+- `post(url, form = {})` - fetch a url with HTTP POST, using `form` as the URL encoded POST body.
+- `post_json(url, json = {})` - fetch a url with HTTP POST, using `json` as the POST body.
+- `http(method, url, options = {})` - use this for more complex requests
 
 #### Parsing the response
 
 These variables are set after each HTTP request.
 
-* `raw` - the raw response from the last request
-* `html` - like `raw`, but with a handful of HTML-specific whitespace cleanups
-* `noko` - parse the response as HTML and return a [Nokogiri](http://nokogiri.org) document
-* `xml` - parse the response as XML and return a [Nokogiri](http://nokogiri.org) document
-* `json` - parse the response as JSON, with symbolized keys
-* `url` - the url of the last request. If the request goes through a redirect, `url` will reflect the final url.
-* `uri` - the URI of the last request. This is useful for resolving relative URLs.
+- `raw` - the raw response from the last request
+- `html` - like `raw`, but with a handful of HTML-specific whitespace cleanups
+- `noko` - parse the response as HTML and return a [Nokogiri](http://nokogiri.org) document
+- `xml` - parse the response as XML and return a [Nokogiri](http://nokogiri.org) document
+- `json` - parse the response as JSON, with symbolized keys
+- `url` - the url of the last request. If the request goes through a redirect, `url` will reflect the final url.
+- `uri` - the URI of the last request. This is useful for resolving relative URLs.
 
 #### Writing CSV
 
-* `csv_header(keys)` - specify the columns for CSV output. If you don't call this, Sinew will use the keys from the first call to `csv_emit`.
-* `csv_emit(hash)` - append a row to the CSV file
+- `csv_header(keys)` - specify the columns for CSV output. If you don't call this, Sinew will use the keys from the first call to `csv_emit`.
+- `csv_emit(hash)` - append a row to the CSV file
 
 ## Hints
 
 Writing Sinew recipes is fun and easy. The builtin caching means you can iterate quickly, since you won't have to re-fetch the data. Here are some hints for writing idiomatic recipes:
 
-* Sinew doesn't (yet) check robots.txt - please check it manually.
-* Prefer Nokogiri over regular expressions wherever possible. Learn [CSS selectors](http://www.w3schools.com/cssref/css_selectors.asp).
-* In Chrome, `$` in the console is your friend.
-* Fallback to regular expressions if you're desperate. Depending on the site, use either `raw` or `html`. `html` is probably your best bet. `raw` is good for crawling Javascript, but it's fragile if the site changes.
-* Learn to love `String#[regexp]`, which is an obscure operator but incredibly handy for Sinew.
-* Laziness is useful. Keep your CSS selectors and regular expressions simple, so maybe they'll work again the next time you need to crawl a site.
-* Don't be afraid to mix CSS selectors, regular expressions, and Ruby:
+- Sinew doesn't (yet) check robots.txt - please check it manually.
+- Prefer Nokogiri over regular expressions wherever possible. Learn [CSS selectors](http://www.w3schools.com/cssref/css_selectors.asp).
+- In Chrome, `$` in the console is your friend.
+- Fallback to regular expressions if you're desperate. Depending on the site, use either `raw` or `html`. `html` is probably your best bet. `raw` is good for crawling Javascript, but it's fragile if the site changes.
+- Learn to love `String#[regexp]`, which is an obscure operator but incredibly handy for Sinew.
+- Laziness is useful. Keep your CSS selectors and regular expressions simple, so maybe they'll work again the next time you need to crawl a site.
+- Don't be afraid to mix CSS selectors, regular expressions, and Ruby:
 
 ```ruby
 noko.css("table")[4].css("td").select { |i| i[:width].to_i > 80 }.map(&:text)
 ```
 
-* Debug your recipes using plain old `puts`, or better yet use `ap` from [awesome_print](https://github.com/michaeldv/awesome_print).
-* Run `sinew -v` to get a report on every `csv_emit`. Very handy.
-* Add the CSV files to your git repo. That way you can version them and get diffs!
+- Debug your recipes using plain old `puts`, or better yet use `ap` from [awesome_print](https://github.com/michaeldv/awesome_print).
+- Run `sinew -v` to get a report on every `csv_emit`. Very handy.
+- Add the CSV files to your git repo. That way you can version them and get diffs!
 
 ## Limitations
 
-* Caching is based on URL, so use caution with cookies and other forms of authentication
-* Almost no support for international (non-english) characters
+- Caching is based on URL, so use caution with cookies and other forms of authentication
+- Almost no support for international (non-english) characters
 
 ## Changelog
 
+#### 2.0.5 (unreleased)
+
+- Supports multiple proxies (`--proxy host1,host2,...`)
+
+#### 2.0.4 (May 2018)
+
+- Handle and cache more errors (too many redirects, connection failures, etc.)
+- Support for adding uri.scheme in generate_cache_key
+- Added status `code`, a peer to `uri`, `raw`, etc.
+
 #### 2.0.3 (May 2018)
 
-* &amp; now normalizes to & (not and)
+- &amp; now normalizes to & (not and)
 
 #### 2.0.2 (May 2018)
 
-* Support for `--limit`, `--proxy` and the `xml` variable
-* Dedup - warn and ignore if row[:url] has already been emitted
-* Auto gunzip if contents are compressed
+- Support for `--limit`, `--proxy` and the `xml` variable
+- Dedup - warn and ignore if row[:url] has already been emitted
+- Auto gunzip if contents are compressed
 
 #### 2.0.1 (May 2018)
 
-* Support for legacy cached `head` files from Sinew 1
+- Support for legacy cached `head` files from Sinew 1
 
 #### 2.0.0 (May 2018)
 
-* Complete rewrite. See above.
+- Complete rewrite. See above.
 
 #### 1.0.3 (June 2012)
 

data/lib/sinew/main.rb

@@ -15,12 +15,6 @@ module Sinew
       @runtime_options = RuntimeOptions.new
       @request_tm = Time.at(0)
       @request_count = 0
-
-      if options[:proxy]
-        addr, port = options[:proxy].split(':')
-        runtime_options.httparty_options[:http_proxyaddr] = addr
-        runtime_options.httparty_options[:http_proxyport] = port || 80
-      end
     end
 
     def run
@@ -105,6 +99,7 @@ module Sinew
         else
           "req #{request.uri}"
         end
+        msg = "#{msg} => #{request.proxy}" if request.proxy
         $stderr.puts msg
       end
 

data/lib/sinew/request.rb

@@ -24,19 +24,36 @@ module Sinew
       @cache_key = calculate_cache_key
     end
 
+    def proxy
+      @proxy ||= begin
+        if proxies = sinew.options[:proxy]
+          proxies.split(',').sample
+        end
+      end
+    end
+
     # run the request, return the result
     def perform
       validate!
 
-      # merge optons
-      options = self.options.merge(sinew.runtime_options.httparty_options)
+      party_options = options.dup
+
+      # merge proxy
+      if proxy = self.proxy
+        addr, port = proxy.split(':')
+        party_options[:http_proxyaddr] = addr
+        party_options[:http_proxyport] = port || 80
+      end
+
+      # now merge runtime_options
+      party_options = party_options.merge(sinew.runtime_options.httparty_options)
 
       # merge headers
       headers = sinew.runtime_options.headers
-      headers = headers.merge(options[:headers]) if options[:headers]
-      options[:headers] = headers
+      headers = headers.merge(party_options[:headers]) if party_options[:headers]
+      party_options[:headers] = headers
 
-      party_response = HTTParty.send(method, uri, options)
+      party_response = HTTParty.send(method, uri, party_options)
       Response.from_network(self, party_response)
     end
 

data/lib/sinew/version.rb

@@ -1,4 +1,4 @@
 module Sinew
   # Gem version
-  VERSION = '2.0.4'.freeze
+  VERSION = '2.0.5'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sinew
 version: !ruby/object:Gem::Version
-  version: 2.0.4
+  version: 2.0.5
 platform: ruby
 authors:
 - Adam Doppelt
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-05-24 00:00:00.000000000 Z
+date: 2019-03-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: awesome_print