web_fetch 0.1.3 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0db44428d9307f942a78b30e7be62b1134e18771e6633adacfff5f95ea3a2f78
-  data.tar.gz: 27ce789c7134dc90f1ae47b9815a4d7fbe72e53edfa5dae53ea6820dffc23cf6
+  metadata.gz: efc9744475219db51912a9786698c869c139ad85334380566124c0f42096beb9
+  data.tar.gz: 358340a89ee0a65519dd0b0cfb7adf6e15cb0e9f1d01b6fc25351394df62ffe4
 SHA512:
-  metadata.gz: 7788c4d849fd56d27e0b167d8d2a00cb5b64d019556dfcd3196d755742179e2e7059a4e3745feff7faef6843418a43cb9bc628428b128add145a834cfc9d198e
-  data.tar.gz: 9c17c80925224f08099a0717a4afd8fd489effee2012be7c63b59d42f14451b066fb990f2497ae1d8781a1e4ac8eea9eac1239346b30f09bf10b7a556a2caf85
+  metadata.gz: 9c341ee83a5877d1537f6e4afe992fd19b45651b694c4cb18ffa9ce7131d6fd7669eecad2b368f235f586dd0083e556ab203a7e17cf1f458d0eca444b839e619
+  data.tar.gz: a34dc8a42298aa11ae2a4cb24ce90dfc799bed47330db04e919a595a094584ca0183840f039c5aebf1bcd81742f1e2a5cffd514502a36824c026d4d277c539f1

data/README.md

@@ -2,20 +2,18 @@
 
 ## Overview
 
-WebFetch is an asynchronous HTTP proxy server that accepts multiple requests for HTTP retrieval, immediately returning a token for each request, and then allowing that token to be redeemed later when the entity has fully responded.
+WebFetch executes concurrent, asynchronous HTTP requests. It is itself an HTTP server implementing a RESTful API, wrapped by a Ruby client interface. Instead of returning a response, WebFetch immediately returns a *promise* which can be redeemed later when the response has been processed.
 
-This permits issuing multiple HTTP requests in parallel, in a fully encapsulated and external process, without having to resort to multi-threading, multi-processing, or complex non-blocking IO implementations.  [EventMachine][1] is used to handle the heavy lifting.
+This permits issuing multiple HTTP requests in parallel, in a fully encapsulated and external process, without having to resort to multi-threading, multi-processing, or complex non-blocking IO implementations. [EventMachine][1] is used to handle the heavy lifting.
 
 ![WebFetch architecture][2]
 
 ## Getting Started
 
-Although WebFetch runs as a web server and provides all functionality over a RESTful API (see below), the simplest way to use it is with its Ruby client implementation, which wraps the HTTP API for you, using [Faraday][3]. This also serves as a [reference][4] for writing WebFetch clients in other languages.
-
 In your `Gemfile`, add:
 
 ``` ruby
-gem 'web_fetch', git: 'https://github.com/bobf/web_fetch.git'
+gem 'web_fetch'
 ```
 
 and update your bundle:
@@ -24,100 +22,181 @@ and update your bundle:
 bundle install
 ```
 
-Create, connect to, and wrap a Ruby client object around a new WebFetch server instance, listening as `localhost` on port `8077`:
+Require WebFetch in your application:
 
 ``` ruby
 require 'web_fetch'
+```
+
+### Launch or connect to a server
+
+Launch the server from your application (recommended for familiarising yourself
+with WebFetch):
+
+``` ruby
 client = WebFetch::Client.create('localhost', 8077)
 ```
 
-Issue some requests [asynchronously]:
+Or connect to an existing WebFetch server (recommended for production systems - see [below](#process-management) for more details):
 
 ``` ruby
-requests = [{ url: 'http://foobar.baz/' },
-            { url: 'http://barfoo.baz/foobar',
-              headers: { 'User-Agent' => 'Foo Browser' } },
-              query: { foo: 'what is foo', bar: 'what is baz' } ]
-jobs = client.gather(requests)
+client = WebFetch::Client.new('localhost', 8077)
 ```
 
-Retrieve the responses [synchronously - *any result that has not yet arrived will block until it has arrived while other requests continue to run in parallel*]:
+### Create a request
+
+Create a WebFetch request. Note that the request will not begin until the next step:
 
 ``` ruby
-responses = []
-jobs.each do |job|
-  response = client.retrieve_by_uid(job[:uid])
-  responses.push(response)
+request = WebFetch::Request.new do |req|
+  req.url = 'http://foobar.baz'
+  req.headers = { 'User-Agent' => 'Foo Browser' }
+  req.query = { foobar: 'baz' }
+  req.method = :get
+  req.body = 'foo bar baz'
+  req.custom = { my_id: '123' }
 end
 ```
 
-See [a working example][5]
+Only `url` is required. The default HTTP method is `GET`.
 
-## HTTP API
+Anything assigned to `custom` will be returned with the final result (available by calling `#custom` on the result). This may be useful if you need to tag each request with your own custom identifier, for example. Anything you assign here will have no bearing whatsoever on the HTTP request.
 
-If you need to use the WebFetch server's HTTP API directly refer to the [Swagger API Reference][6]
+If you prefer to build a request from a hash, you can call `WebFetch::Request.from_hash`
 
-## Managing the WebFetch process yourself
+``` ruby
+request = WebFetch::Request.from_hash(
+  url: 'http://foobar.baz',
+  headers: { 'User-Agent' => 'Foo Browser' },
+  query: { foobar: 'baz' },
+  method: :get,
+  body: 'foo bar baz',
+  custom: { my_id: '123' }
+)
+```
 
-You may want to run the WebFetch server yourself rather than instantiate it via the client. For this case, the executable `bin/web_fetch_control` is provided.
+### Gather responses
 
-WebFetch can be started in the terminal with output going to STDOUT or as a daemon.
+Ask WebFetch to begin gathering your HTTP requests in the background:
 
-Run the server as a daemon:
+``` ruby
+promises = client.gather([request])
+```
+
+`WebFetch::Client#gather` accepts an array of `WebFetch::Request` objects and immediately returns an array of `WebFetch::Promise` objects. WebFetch will process all requests in the background concurrently.
 
+To retrieve the result of a request, call `WebFetch::Promise#fetch`
+
+``` ruby
+result = promises.first.fetch
+
+# Available methods:
+result.body
+result.headers
+result.status # HTTP status code
+result.success? # False if a network error (not HTTP error) occurred
+result.error # Underlying network error if applicable
 ```
-$ bundle exec bin/web_fetch_control start -- --log /tmp/web_fetch.log
+
+Note that `WebFech::Promise#fetch` will block until the result is complete by default. If you want to continue executing other code if the result is not ready (e.g. to see if any other results are ready), you can pass `wait: false`
+
+``` ruby
+result = promises.first.fetch(wait: false)
 ```
 
-**Note that you should always pass `--log` when running as a daemon otherwise all output will go to the null device.**
+A special value `:pending` will be returned if the result is still processing.
 
-Run the server in the terminal:
+Alternatively, you can call `WebFetch::Promise#complete?` to check if a request has finished before waiting for the response:
 
+``` ruby
+result = promises.first.fetch if promises.first.complete?
 ```
-$ bundle exec bin/web_fetch_control run -- --port 8080
+
+### Fetching results later
+
+In some cases you may need to fetch the result of a request in a different context to which you initiated it. A unique ID is available for each *Promise* which can be used to fetch the result from a separate *Client* instance:
+
+``` ruby
+client = WebFetch::Client.new('localhost', 8077)
+promises = client.gather([
+  WebFetch::Request.new { |req| req.url = 'http://foobar.com' }
+])
+uid = promises.first.uid
+
+# Later ...
+client = WebFetch::Client.new('localhost', 8077)
+result = client.fetch(uid)
 ```
 
-It is further recommended to use a process management tool to monitor the pidfile (pass `--pidfile /path/to/file.pid` to specify an explicit location).
+This can be useful if your web application initiates requests in one controller action and fetches them in another; the `uid` can be stored in a database and used to fetch the request later on.
 
-To connect to an existing process, use `WebFetch::Client.new` rather than `WebFetch::Client.create`. For example:
+### Stopping the server
+
+When you have finished using the web server, call `WebFetch::Client#stop`
 
 ``` ruby
-WebFetch::Client.new('localhost', 8087)
+client.stop
 ```
 
-## WebFetch Client request options
+The server will not automatically stop when your program exits.
 
-`WebFetch::Client#gather` accepts an array of hashes which may contain the following parameters:
+## Examples
 
-  * `url`: The target URL [string]
-  * `headers`: HTTP headers [hash]
-  * `query`: Query parameters [hash]
-  * `method`: HTTP method (default: `"GET"`) [string]
-  * `body`: HTTP body [string]
+[Runnable examples][5] are provided for more detailed usage.
 
-These parameters will all be used (where provided) when initiating the HTTP request on the target.
+## HTTP API
 
-Arbitrary parameters can also be passed and will be returned by `#gather` (though they will not be used to construct the HTTP request). This allows tagging requests with arbitrary information if you need to identify them in a particular way. For example, you may want to generate your own unique identifier for a request, in which case you could do:
+If you need to use the WebFetch server's HTTP API directly refer to the [Swagger API Reference][6]
 
-``` ruby
-client.gather([{ url: 'http://foobar.baz', my_unique_id: '123-456-789' }])
-# [{:request=>{:url=>"http://foobar.baz", :my_unique_id=>"123-456-789"}, :hash=>"7c511911d16e1072363fa1653bdd93df65208901", :uid=>"1fb4ee7a-9fc0-4896-9af2-7cbdf234a468"}]
+## Managing the WebFetch process yourself <a name='process-management'></a>
+
+For production systems it is advised that you run the WebFetch server separately rather than instantiate it via the client. For this case, the executable `bin/web_fetch_control` is provided. Daemonisation is handled by the [daemons][7] gem.
+
+WebFetch can be started in the terminal with output going to STDOUT or as a daemon.
+
+Run the server as a daemon:
+
+```
+$ web_fetch_control start
 ```
 
-## Logging
+Run the server in the terminal:
 
-WebFetch logs to STDOUT by default. An alternative log file can be set either
-by passing `--log /path/to/logfile` to the command line server, or by passing
-`log: '/path/to/logfile'` to `WebFetch::Client.create`:
+```
+$ web_fetch_control run
+```
+
+Stop the server:
 
 ```
-$ bundle exec bin/web_fetch_server --log /tmp/web_fetch.log
+$ web_fetch_control stop
 ```
 
+To pass options to WebFetch, pass `--` to `web_fetch_control` and add all WebFetch options afterwards.
+
+Available options:
+
 ```
-client = WebFetch::Client.create('localhost', 8077, log: '/tmp/web_fetch.log')
+--port 60087
+--host localhost
+--pidfile /tmp/web_fetch.pid
+--log /var/log/web_fetch.log
 ```
 
+e.g.:
+
+```
+web_fetch_control run -- --port 8000 --host 0.0.0.0
+```
+
+No pid file will be created unless the `--pidfile` parameter is passed. It is recommended to use a process monitoring tool (e.g. `monit` or `systemd`) to monitor the WebFetch process.
+
+When running as a daemon, WebFetch will log to the null device so it is advised to always pass `--log` in this case.
+
+## Docker
+
+To use WebFetch in Docker you can either use the provided [`Dockerfile`](docker/Dockerfile) or the public image [`web_fetch/web_fetch`](https://hub.docker.com/r/webfetch/webfetch/)
+
 ## Contributing
 
 WebFetch uses `rspec` for testing:
@@ -138,12 +217,13 @@ Feel free to fork and create a pull request if you would like to make any change
 
 ## License
 
-WebFetch is licensed under the [MIT License][7]. You are encouraged to re-use the code in any way you see fit as long as you give credit to the original author. If you do use the code for any other projects then feel free to let me know but, of course, this is not required.
+WebFetch is licensed under the [MIT License][8]. You are encouraged to re-use the code in any way you see fit as long as you give credit to the original author. If you do use the code for any other projects then feel free to let me know but, of course, this is not required.
 
 [1]: https://github.com/eventmachine/eventmachine
 [2]: doc/web_fetch_architecture.png
 [3]: https://github.com/lostisland/faraday
 [4]: lib/web_fetch/client.rb
-[5]: doc/client_example.rb
+[5]: doc/examples/
 [6]: swagger.yaml
-[7]: LICENSE
+[7]: https://github.com/thuehlinger/daemons
+[8]: LICENSE

data/config/locales/en.yml

@@ -10,3 +10,7 @@ en:
 
   uid_not_found: "Provided `uid` has not yet been requested"
   hash_not_found: "Provided `hash` has not yet been requested"
+
+  pending: "Your request is still being processed"
+
+  no_request: "No active request found for UID: %{uid}"

data/doc/examples/blocking_requests.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+# Run me with:
+# bundle exec ruby doc/examples/blocking_requests.rb
+#
+# rubocop:disable all
+require 'web_fetch'
+
+def blocking_requests
+  urls = ['https://rubygems.org/',
+          'http://lycos.com/',
+          'http://google.com/']
+
+  requests = urls.map do |url|
+    WebFetch::Request.new do |request|
+      request.url = url
+    end
+  end
+
+  client = WebFetch::Client.create('localhost', 8077)
+  promises = client.gather(requests)
+
+  promises.each do |promise|
+    result = promise.fetch(wait: true) # Will block (default behaviour)
+    puts result.body[0..100]
+    puts "Success: #{result.success?}"
+  end
+
+  client.stop
+end
+
+blocking_requests
+# rubocop:enable all

data/doc/examples/non_blocking_requests.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+# Run me with:
+# bundle exec ruby doc/examples/non_blocking_requests.rb
+#
+# rubocop:disable all
+require 'web_fetch'
+
+def non_blocking_requests
+  urls = ['https://rubygems.org/',
+          'http://lycos.com/',
+          'http://google.com/']
+
+  requests = urls.map do |url|
+    WebFetch::Request.new do |request|
+      request.url = url
+    end
+  end
+
+  client = WebFetch::Client.create('localhost', 8077)
+  promises = client.gather(requests)
+
+  while promises.any? { |promise| !promise.complete? }
+    promises.each do |promise|
+      result = promise.fetch(wait: false) # Will not block
+      puts result.body[0..100] unless result == :pending
+    end
+  end
+
+  client.stop
+end
+
+non_blocking_requests
+# rubocop:enable all

data/doc/examples/use_uid_for_request.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+# Run me with:
+# bundle exec ruby doc/examples/use_uid_for_request.rb
+#
+# rubocop:disable all
+require 'web_fetch'
+
+def use_uid_for_request
+  request = WebFetch::Request.new do |request|
+    request.url = 'https://rubygems.org/'
+  end
+
+  client = WebFetch::Client.create('localhost', 8077)
+  promises = client.gather([request])
+  uid = promises.first.uid
+  while true
+    result = client.fetch(uid, wait: false)
+    break unless result == :pending
+  end
+
+  puts result.body[0..100]
+
+  client.stop
+end
+
+use_uid_for_request
+# rubocop:enable all

data/docker/Dockerfile

@@ -0,0 +1,3 @@
+FROM library/ruby
+RUN gem install web_fetch
+CMD ["web_fetch_control", "run", "--", "--port", "8077", "--host", "0.0.0.0"]

data/lib/web_fetch/client.rb

@@ -4,6 +4,10 @@ module WebFetch
   # Client to be used in application code. Capable of spawning a server and
   # interacting with it to gather requests and retrieve them
   class Client
+    include ClientHttp
+
+    attr_reader :host, :port
+
     def initialize(host, port, options = {})
       @host = host
       @port = port
@@ -38,15 +42,35 @@ module WebFetch
     end
 
     def gather(requests)
-      json = JSON.dump(requests: requests)
+      json = JSON.dump(requests: requests.map(&:to_h))
       response = post('gather', json)
-      return nil unless response.success?
 
-      JSON.parse(response.body, symbolize_names: true)[:requests]
+      handle_error(JSON.parse(response.body)['error']) unless response.success?
+
+      requests = JSON.parse(response.body, symbolize_names: true)[:requests]
+      promises(requests)
+    end
+
+    def fetch(uid, options = {})
+      block = options.fetch(:wait, true)
+
+      outcome = block ? retrieve_by_uid(uid) : find_by_uid(uid)
+      no_request_error(uid) if outcome.nil?
+
+      return :pending if outcome[:pending]
+
+      new_result(outcome)
     end
 
     def retrieve_by_uid(uid)
-      response = get('retrieve', uid: uid)
+      response = get("retrieve/#{uid}")
+      return nil unless response.success?
+
+      JSON.parse(response.body, symbolize_names: true)
+    end
+
+    def find_by_uid(uid)
+      response = get("find/#{uid}")
       return nil unless response.success?
 
       JSON.parse(response.body, symbolize_names: true)
@@ -78,23 +102,29 @@ module WebFetch
 
     private
 
-    def base_uri
-      "http://#{@host}:#{@port}"
+    def handle_error(errors)
+      raise WebFetch::ClientError, errors
     end
 
-    def get(endpoint, params = {})
-      conn = Faraday.new(url: base_uri)
-      conn.get do |request|
-        request.url "/#{endpoint}"
-        request.params.merge!(params)
-      end
+    def no_request_error(uid)
+      raise RequestNotFoundError, [I18n.t('no_request', uid: uid)]
+    end
+
+    def new_result(outcome)
+      response = outcome[:response]
+      Result.new(
+        body: response[:body],
+        headers: response[:headers],
+        status: response[:status],
+        success: response[:success],
+        error: response[:error],
+        uid: outcome[:uid]
+      )
     end
 
-    def post(endpoint, body)
-      conn = Faraday.new(url: base_uri)
-      conn.post do |request|
-        request.url "/#{endpoint}"
-        request.body = body
+    def promises(requests)
+      requests.map do |request|
+        Promise.new(self, uid: request[:uid], request: request[:request])
       end
     end
   end

data/lib/web_fetch/concerns/client_http.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module WebFetch
+  module ClientHttp
+    def base_uri
+      "http://#{@host}:#{@port}"
+    end
+
+    def get(endpoint, params = {})
+      conn = Faraday.new(url: base_uri)
+      conn.get do |request|
+        request.url "/#{endpoint}"
+        request.params.merge!(params)
+      end
+    end
+
+    def post(endpoint, body)
+      conn = Faraday.new(url: base_uri)
+      conn.post do |request|
+        request.url "/#{endpoint}"
+        request.body = body
+      end
+    end
+  end
+end

data/lib/web_fetch/errors.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module WebFetch
+  class Error < StandardError
+    attr_reader :errors
+
+    def initialize(errors = nil)
+      @errors = errors
+    end
+  end
+
+  class ClientError < Error; end
+
+  class RequestNotFoundError < Error; end
+end

data/lib/web_fetch/event_machine_helpers.rb

@@ -3,31 +3,44 @@
 module WebFetch
   # EventMachine layer-specific helpers
   module EventMachineHelpers
-    def wait_for_response(deferred, response)
-      deferred[:http].callback do
-        Logger.debug("HTTP fetch complete for uid: #{deferred[:uid]}")
-        deferred[:succeeded] = true
+    def request_async(request)
+      async_request = EM::HttpRequest.new(request[:url])
+      method = request.fetch(:method, 'GET').downcase.to_sym
+      async_request.public_send(
+        method,
+        head: request[:headers],
+        query: request.fetch(:query, {}),
+        body: request.fetch(:body, nil)
+      )
+    end
+
+    def apply_callbacks(request)
+      request[:deferred].callback do
+        Logger.debug("HTTP fetch complete for uid: #{request[:uid]}")
+        request[:succeeded] = true
       end
 
-      deferred[:http].errback do
-        Logger.debug("HTTP fetch failed for uid: #{deferred[:uid]}")
-        deferred[:failed] = true
+      request[:deferred].errback do
+        Logger.debug("HTTP fetch failed for uid: #{request[:uid]}")
+        request[:failed] = true
       end
+    end
 
-      tick_loop(deferred, response)
+    def wait_for_response(request, response)
+      tick_loop(request, response)
     end
 
-    def tick_loop(deferred, response)
+    def tick_loop(request, response)
       # XXX There may be a much nicer way to wait for an async task to complete
       # before returning a response but I couldn't figure it out, so I used
       # EM.tick_loop which effectively does the same as a Twisted deferred
       # callback chain, just much more explicitly.
       EM.tick_loop do
-        if deferred[:succeeded]
-          succeed(deferred, response)
+        if request[:succeeded]
+          succeed(request, response)
           :stop
-        elsif deferred[:failed]
-          fail_(deferred, response)
+        elsif request[:failed]
+          fail_(request, response)
           :stop
         end
       end

data/lib/web_fetch/http_helpers.rb

@@ -9,6 +9,16 @@ module WebFetch
       response.send_response
     end
 
+    def pending(result, response)
+      respond_immediately({
+                            payload: {
+                              uid: result[:request][:uid],
+                              pending: true,
+                              message: I18n.t(:pending)
+                            }
+                          }, response)
+    end
+
     def compress(string)
       return string unless accept_gzip?
 
@@ -35,31 +45,33 @@ module WebFetch
       JSON.parse(@http_post_content, symbolize_names: true)
     end
 
-    def succeed(deferred, response)
+    def succeed(request, response)
       response.status = 200
-      response.content = compress(JSON.dump(success(deferred)))
+      response.content = compress(JSON.dump(success(request)))
       response.send_response
+      storage.delete(request[:uid])
     end
 
-    def success(deferred)
-      result = deferred[:http]
+    def success(request)
+      result = request[:deferred]
       { response: {
         success: true,
         body: result.response,
         headers: result.headers,
         status: result.response_header.status
       },
-        uid: deferred[:uid] }
+        uid: request[:uid] }
     end
 
-    def fail_(deferred, response)
+    def fail_(request, response)
       response.status = 200
-      response.content = compress(JSON.dump(failure(deferred)))
+      response.content = compress(JSON.dump(failure(request)))
       response.send_response
+      storage.delete(request[:uid])
     end
 
-    def failure(deferred)
-      result = deferred[:http]
+    def failure(request)
+      result = request[:deferred]
       { response: {
         success: false,
         body: result.response,
@@ -67,7 +79,7 @@ module WebFetch
         status: result.response_header.status,
         error: (result.error&.inspect)
       },
-        uid: deferred[:uid] }
+        uid: request[:uid] }
     end
 
     def accept_gzip?