batch_api2 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/MIT-LICENSE +20 -0
- data/Rakefile +30 -0
- data/changelog.md +74 -0
- data/lib/batch_api.rb +28 -0
- data/lib/batch_api/batch_error.rb +41 -0
- data/lib/batch_api/configuration.rb +36 -0
- data/lib/batch_api/error_wrapper.rb +44 -0
- data/lib/batch_api/internal_middleware.rb +87 -0
- data/lib/batch_api/internal_middleware/decode_json_body.rb +28 -0
- data/lib/batch_api/internal_middleware/response_filter.rb +27 -0
- data/lib/batch_api/operation.rb +2 -0
- data/lib/batch_api/operation/rack.rb +76 -0
- data/lib/batch_api/operation/rails.rb +42 -0
- data/lib/batch_api/processor.rb +113 -0
- data/lib/batch_api/processor/executor.rb +18 -0
- data/lib/batch_api/processor/sequential.rb +29 -0
- data/lib/batch_api/rack_middleware.rb +37 -0
- data/lib/batch_api/response.rb +38 -0
- data/lib/batch_api/utils.rb +17 -0
- data/lib/batch_api/version.rb +3 -0
- data/lib/tasks/batch_api_tasks.rake +4 -0
- data/readme.md +243 -0
- data/spec/dummy/Gemfile +1 -0
- data/spec/dummy/Gemfile.lock +8 -0
- data/spec/dummy/README.rdoc +261 -0
- data/spec/dummy/Rakefile +15 -0
- data/spec/dummy/app/assets/javascripts/application.js +15 -0
- data/spec/dummy/app/assets/javascripts/endpoints.js +2 -0
- data/spec/dummy/app/assets/stylesheets/application.css +13 -0
- data/spec/dummy/app/assets/stylesheets/endpoints.css +4 -0
- data/spec/dummy/app/controllers/application_controller.rb +3 -0
- data/spec/dummy/app/controllers/endpoints_controller.rb +36 -0
- data/spec/dummy/app/helpers/application_helper.rb +2 -0
- data/spec/dummy/app/helpers/endpoints_helper.rb +2 -0
- data/spec/dummy/app/views/endpoints/get.html.erb +2 -0
- data/spec/dummy/app/views/endpoints/post.html.erb +2 -0
- data/spec/dummy/app/views/layouts/application.html.erb +14 -0
- data/spec/dummy/bin/bundle +3 -0
- data/spec/dummy/bin/rails +4 -0
- data/spec/dummy/bin/rake +4 -0
- data/spec/dummy/bin/setup +29 -0
- data/spec/dummy/config.ru +4 -0
- data/spec/dummy/config/application.rb +32 -0
- data/spec/dummy/config/boot.rb +3 -0
- data/spec/dummy/config/database.yml +25 -0
- data/spec/dummy/config/environment.rb +5 -0
- data/spec/dummy/config/environments/development.rb +41 -0
- data/spec/dummy/config/environments/production.rb +79 -0
- data/spec/dummy/config/environments/test.rb +42 -0
- data/spec/dummy/config/initializers/assets.rb +11 -0
- data/spec/dummy/config/initializers/backtrace_silencers.rb +7 -0
- data/spec/dummy/config/initializers/cookies_serializer.rb +3 -0
- data/spec/dummy/config/initializers/filter_parameter_logging.rb +4 -0
- data/spec/dummy/config/initializers/inflections.rb +16 -0
- data/spec/dummy/config/initializers/mime_types.rb +4 -0
- data/spec/dummy/config/initializers/secret_token.rb +7 -0
- data/spec/dummy/config/initializers/session_store.rb +3 -0
- data/spec/dummy/config/initializers/wrap_parameters.rb +14 -0
- data/spec/dummy/config/locales/en.yml +23 -0
- data/spec/dummy/config/routes.rb +64 -0
- data/spec/dummy/config/secrets.yml +22 -0
- data/spec/dummy/db/test.sqlite3 +0 -0
- data/spec/dummy/public/404.html +26 -0
- data/spec/dummy/public/422.html +26 -0
- data/spec/dummy/public/500.html +25 -0
- data/spec/dummy/public/favicon.ico +0 -0
- data/spec/dummy/script/rails +6 -0
- data/spec/dummy/test/functional/endpoints_controller_test.rb +14 -0
- data/spec/dummy/test/unit/helpers/endpoints_helper_test.rb +4 -0
- data/spec/lib/batch_api_spec.rb +20 -0
- data/spec/lib/batch_error_spec.rb +23 -0
- data/spec/lib/configuration_spec.rb +30 -0
- data/spec/lib/error_wrapper_spec.rb +68 -0
- data/spec/lib/internal_middleware/decode_json_body_spec.rb +44 -0
- data/spec/lib/internal_middleware/response_filter_spec.rb +61 -0
- data/spec/lib/internal_middleware_spec.rb +93 -0
- data/spec/lib/operation/rack_spec.rb +246 -0
- data/spec/lib/operation/rails_spec.rb +100 -0
- data/spec/lib/processor/executor_spec.rb +22 -0
- data/spec/lib/processor/sequential_spec.rb +39 -0
- data/spec/lib/processor_spec.rb +136 -0
- data/spec/lib/rack_middleware_spec.rb +103 -0
- data/spec/lib/response_spec.rb +53 -0
- data/spec/rack-integration/rails_spec.rb +10 -0
- data/spec/rack-integration/shared_examples.rb +273 -0
- data/spec/rack-integration/sinatra_integration_spec.rb +19 -0
- data/spec/spec_helper.rb +42 -0
- data/spec/support/sinatra_app.rb +54 -0
- data/spec/support/sinatra_xhr.rb +13 -0
- metadata +214 -0
@@ -0,0 +1,42 @@
|
|
1
|
+
require 'batch_api/operation/rack'
|
2
|
+
|
3
|
+
module BatchApi
|
4
|
+
# Public: an individual batch operation.
|
5
|
+
module Operation
|
6
|
+
class Rails < Operation::Rack
|
7
|
+
# Public: create a new Rails Operation. It does all that Rack does
|
8
|
+
# and also some additional Rails-specific processing.
|
9
|
+
def initialize(op, base_env, app)
|
10
|
+
super
|
11
|
+
@params = params_with_path_components
|
12
|
+
end
|
13
|
+
|
14
|
+
# Internal: customize the request environment. This is currently done
|
15
|
+
# manually and feels clunky and brittle, but is mostly likely fine, though
|
16
|
+
# there are one or two environment parameters not yet adjusted.
|
17
|
+
def process_env
|
18
|
+
# parameters
|
19
|
+
super
|
20
|
+
@env["action_dispatch.request.parameters"] = @params
|
21
|
+
@env["action_dispatch.request.request_parameters"] = @params
|
22
|
+
end
|
23
|
+
|
24
|
+
private
|
25
|
+
|
26
|
+
# Internal: process the params the Rails way, merging in the
|
27
|
+
# path_parameters. If the route can't be recognized, it will
|
28
|
+
# leave the params unchanged.
|
29
|
+
#
|
30
|
+
# Returns the updated params.
|
31
|
+
def params_with_path_components
|
32
|
+
begin
|
33
|
+
path_params = ::Rails.application.routes.recognize_path(@url, @op)
|
34
|
+
@params.merge(path_params)
|
35
|
+
rescue
|
36
|
+
@params
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
@@ -0,0 +1,113 @@
|
|
1
|
+
require 'batch_api/processor/sequential'
|
2
|
+
require 'batch_api/operation'
|
3
|
+
|
4
|
+
module BatchApi
|
5
|
+
class Processor
|
6
|
+
attr_reader :ops, :options, :app
|
7
|
+
|
8
|
+
# Public: create a new Processor.
|
9
|
+
#
|
10
|
+
# env - a Rack environment hash
|
11
|
+
# app - a Rack application
|
12
|
+
#
|
13
|
+
# Raises OperationLimitExceeded if more operations are requested than
|
14
|
+
# allowed by the BatchApi configuration.
|
15
|
+
# Raises Errors::BadOptionError if other provided options are invalid.
|
16
|
+
# Raises ArgumentError if no operations are provided (nil or []).
|
17
|
+
#
|
18
|
+
# Returns the new Processor instance.
|
19
|
+
def initialize(request, app)
|
20
|
+
@app = app
|
21
|
+
@request = request
|
22
|
+
@env = request.env
|
23
|
+
@ops = self.process_ops
|
24
|
+
@options = self.process_options
|
25
|
+
end
|
26
|
+
|
27
|
+
# Public: the processing strategy to use, based on the options
|
28
|
+
# provided in BatchApi setup and the request.
|
29
|
+
# Currently only Sequential is supported.
|
30
|
+
def strategy
|
31
|
+
BatchApi::Processor::Sequential
|
32
|
+
end
|
33
|
+
|
34
|
+
# Public: run the batch operations according to the appropriate strategy.
|
35
|
+
#
|
36
|
+
# Returns a set of BatchResponses
|
37
|
+
def execute!
|
38
|
+
stack = InternalMiddleware.batch_stack(self)
|
39
|
+
format_response(stack.call(middleware_env))
|
40
|
+
end
|
41
|
+
|
42
|
+
protected
|
43
|
+
|
44
|
+
def middleware_env
|
45
|
+
{
|
46
|
+
ops: @ops,
|
47
|
+
rack_env: @env,
|
48
|
+
rack_app: @app,
|
49
|
+
options: @options
|
50
|
+
}
|
51
|
+
end
|
52
|
+
|
53
|
+
# Internal: format the result of the operations, and include
|
54
|
+
# any other appropriate information (such as timestamp).
|
55
|
+
#
|
56
|
+
# result - the array of batch operations
|
57
|
+
#
|
58
|
+
# Returns a hash ready to go to the user
|
59
|
+
def format_response(operation_results)
|
60
|
+
{
|
61
|
+
"results" => operation_results
|
62
|
+
}
|
63
|
+
end
|
64
|
+
|
65
|
+
# Internal: Validate that an allowable number of operations have been
|
66
|
+
# provided, and turn them into BatchApi::Operation objects.
|
67
|
+
#
|
68
|
+
# ops - a series of operations
|
69
|
+
#
|
70
|
+
# Raises Errors::OperationLimitExceeded if more operations are requested than
|
71
|
+
# allowed by the BatchApi configuration.
|
72
|
+
# Raises Errors::NoOperationsError if no operations are provided.
|
73
|
+
#
|
74
|
+
# Returns an array of BatchApi::Operation objects
|
75
|
+
def process_ops
|
76
|
+
ops = @request.params.delete("ops")
|
77
|
+
if !ops || ops.empty?
|
78
|
+
raise Errors::NoOperationsError, "No operations provided"
|
79
|
+
elsif ops.length > BatchApi.config.limit
|
80
|
+
raise Errors::OperationLimitExceeded,
|
81
|
+
"Only #{BatchApi.config.limit} operations can be submitted at once, " +
|
82
|
+
"#{ops.length} were provided"
|
83
|
+
else
|
84
|
+
ops.map do |op|
|
85
|
+
self.class.operation_klass.new(op, @env, @app)
|
86
|
+
end
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
90
|
+
# Internal: which operation class to used.
|
91
|
+
#
|
92
|
+
# Returns Batch::Operation::(Rack|Rails) depending on the environment
|
93
|
+
def self.operation_klass
|
94
|
+
BatchApi.rails? ? Operation::Rails : Operation::Rack
|
95
|
+
end
|
96
|
+
|
97
|
+
# Internal: Processes any other provided options for validity.
|
98
|
+
# Currently, the :sequential option is REQUIRED (until parallel
|
99
|
+
# implementation is created).
|
100
|
+
#
|
101
|
+
# options - an options hash
|
102
|
+
#
|
103
|
+
# Raises Errors::BadOptionError if sequential is not provided.
|
104
|
+
#
|
105
|
+
# Returns the valid options hash.
|
106
|
+
def process_options
|
107
|
+
unless @request.params["sequential"]
|
108
|
+
raise Errors::BadOptionError, "Sequential flag is currently required"
|
109
|
+
end
|
110
|
+
@request.params
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
module BatchApi
|
2
|
+
class Processor
|
3
|
+
# Public: a simple middleware that lives at the end of the internal chain
|
4
|
+
# and simply executes each batch operation.
|
5
|
+
class Executor
|
6
|
+
|
7
|
+
# Public: initialize the middleware.
|
8
|
+
def initialize(app)
|
9
|
+
@app = app
|
10
|
+
end
|
11
|
+
|
12
|
+
# Public: execute the batch operation.
|
13
|
+
def call(env)
|
14
|
+
env[:op].execute
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
@@ -0,0 +1,29 @@
|
|
1
|
+
module BatchApi
|
2
|
+
class Processor
|
3
|
+
class Sequential
|
4
|
+
# Public: initialize with the app.
|
5
|
+
def initialize(app)
|
6
|
+
@app = app
|
7
|
+
end
|
8
|
+
|
9
|
+
# Public: execute all operations sequentially.
|
10
|
+
#
|
11
|
+
# ops - a set of BatchApi::Operations
|
12
|
+
# options - a set of options
|
13
|
+
#
|
14
|
+
# Returns an array of BatchApi::Response objects.
|
15
|
+
def call(env)
|
16
|
+
env[:ops].collect do |op|
|
17
|
+
# set the current op
|
18
|
+
env[:op] = op
|
19
|
+
|
20
|
+
# execute the individual request inside the operation-specific
|
21
|
+
# middeware, then clear out the current op afterward
|
22
|
+
middleware = InternalMiddleware.operation_stack
|
23
|
+
middleware.call(env).tap {|r| env.delete(:op) }
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
@@ -0,0 +1,37 @@
|
|
1
|
+
module BatchApi
|
2
|
+
class RackMiddleware
|
3
|
+
def initialize(app, &block)
|
4
|
+
@app = app
|
5
|
+
yield BatchApi.config if block
|
6
|
+
end
|
7
|
+
|
8
|
+
def call(env)
|
9
|
+
if batch_request?(env)
|
10
|
+
begin
|
11
|
+
request = request_klass.new(env)
|
12
|
+
result = BatchApi::Processor.new(request, @app).execute!
|
13
|
+
[200, self.class.content_type, [result.to_json]]
|
14
|
+
rescue => err
|
15
|
+
ErrorWrapper.new(err).render
|
16
|
+
end
|
17
|
+
else
|
18
|
+
@app.call(env)
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
def self.content_type
|
23
|
+
{"Content-Type" => "application/json"}
|
24
|
+
end
|
25
|
+
|
26
|
+
private
|
27
|
+
|
28
|
+
def batch_request?(env)
|
29
|
+
env["PATH_INFO"] == BatchApi.config.endpoint &&
|
30
|
+
env["REQUEST_METHOD"] == BatchApi.config.verb.to_s.upcase
|
31
|
+
end
|
32
|
+
|
33
|
+
def request_klass
|
34
|
+
defined?(ActionDispatch) ? ActionDispatch::Request : Rack::Request
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
module BatchApi
|
2
|
+
# Public: a response from an internal operation in the Batch API.
|
3
|
+
# It contains all the details that are needed to describe the call's
|
4
|
+
# outcome.
|
5
|
+
class Response
|
6
|
+
# Public: the attributes of the HTTP response.
|
7
|
+
attr_accessor :status, :body, :headers
|
8
|
+
|
9
|
+
# Public: create a new response representation from a Rack-compatible
|
10
|
+
# response (e.g. [status, headers, response_object]).
|
11
|
+
def initialize(response)
|
12
|
+
@status, @headers = *response
|
13
|
+
@body = process_body(response[2])
|
14
|
+
end
|
15
|
+
|
16
|
+
# Public: convert the response to JSON. nil values are ignored.
|
17
|
+
def as_json(options = {})
|
18
|
+
{}.tap do |result|
|
19
|
+
result[:body] = @body unless @body.nil?
|
20
|
+
result[:headers] = @headers unless @headers.nil?
|
21
|
+
result[:status] = @status unless @status.nil?
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
def process_body(body_pieces)
|
28
|
+
# bodies have to respond to .each, but may otherwise
|
29
|
+
# not be suitable for JSON serialization
|
30
|
+
# (I'm looking at you, ActionDispatch::Response)
|
31
|
+
# so turn it into a string
|
32
|
+
base_body = ""
|
33
|
+
body_pieces.each {|str| base_body << str}
|
34
|
+
base_body
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
@@ -0,0 +1,17 @@
|
|
1
|
+
module BatchApi
|
2
|
+
module Utils
|
3
|
+
|
4
|
+
def self.deep_dup(object)
|
5
|
+
if object.is_a?(Hash)
|
6
|
+
duplicate = object.dup
|
7
|
+
duplicate.each_pair do |k,v|
|
8
|
+
tv = duplicate[k]
|
9
|
+
duplicate[k] = tv.is_a?(Hash) && v.is_a?(Hash) ? deep_dup(tv) : v
|
10
|
+
end
|
11
|
+
duplicate
|
12
|
+
else
|
13
|
+
object
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
data/readme.md
ADDED
@@ -0,0 +1,243 @@
|
|
1
|
+
[](http://travis-ci.org/arsduo/batch_api)
|
2
|
+
|
3
|
+
## What's this?
|
4
|
+
|
5
|
+
A gem that provides a RESTful Batch API for Rails and other Rack applications.
|
6
|
+
In this system, batch requests are simply collections of regular REST calls,
|
7
|
+
whose results are returned as an equivalent collection of regular REST results.
|
8
|
+
|
9
|
+
This is heavily inspired by [Facebook's Batch API](https://developers.facebook.com/docs/graph-api/making-multiple-requests).
|
10
|
+
|
11
|
+
## A Quick Example
|
12
|
+
|
13
|
+
Making a batch request:
|
14
|
+
|
15
|
+
```
|
16
|
+
# POST /batch
|
17
|
+
# Content-Type: application/json
|
18
|
+
|
19
|
+
{
|
20
|
+
ops: [
|
21
|
+
{method: "get", url: "/patrons"},
|
22
|
+
{method: "post", url: "/orders/new", params: {dish_id: 123}},
|
23
|
+
{method: "get", url: "/oh/no/error", headers: {break: "fast"}},
|
24
|
+
{method: "delete", url: "/patrons/456"}
|
25
|
+
],
|
26
|
+
sequential: true
|
27
|
+
}
|
28
|
+
```
|
29
|
+
|
30
|
+
Reading the response:
|
31
|
+
|
32
|
+
```
|
33
|
+
{
|
34
|
+
results: [
|
35
|
+
{status: 200, body: [{id: 1, name: "Jim-Bob"}, ...], headers: {}},
|
36
|
+
{status: 201, body: {id: 4, dish_name: "Spicy Crab Legs"}, headers: {}},
|
37
|
+
{status: 500, body: {error: {oh: "noes!"}}, headers: {Problem: "woops"}},
|
38
|
+
{status: 200, body: null, headers: {}}}
|
39
|
+
]
|
40
|
+
}
|
41
|
+
```
|
42
|
+
|
43
|
+
### How It Works
|
44
|
+
|
45
|
+
#### Requests
|
46
|
+
|
47
|
+
As you can see from the example above, each request in the batch (an
|
48
|
+
"operation", in batch parlance) describes the same features any HTTP request
|
49
|
+
would include:
|
50
|
+
|
51
|
+
* _url_ - the API endpoint to hit, formatted exactly as you would for a regular
|
52
|
+
REST API request, leading / and all. (required)
|
53
|
+
* _method_ - what type of request to make -- GET, POST, PUT, etc. If no method
|
54
|
+
is supplied, GET is assumed. (optional)
|
55
|
+
* _args_ - a hash of arguments to the API. This can be used for both GET and
|
56
|
+
PUT/POST/PATCH requests. (optional)
|
57
|
+
* _headers_ - a hash of request-specific headers. The headers sent in the
|
58
|
+
request will be included as well, with operation-specific headers taking
|
59
|
+
precendence. (optional)
|
60
|
+
* _silent_ - whether to return a response for this request. You can save on
|
61
|
+
transfer if, for instance, you're making several PUT/POST requests, then
|
62
|
+
executing a GET at the end.
|
63
|
+
|
64
|
+
These individual operations are supplied as the "ops" parameter in the
|
65
|
+
overall request. Other options include:
|
66
|
+
|
67
|
+
* _sequential_ - execute all operations sequentially, rather than in parallel.
|
68
|
+
*This parameter is currently REQUIRED and must be set to true.* (In the future
|
69
|
+
the Batch API will offer parallel processing for thread-safe apps, and hence
|
70
|
+
this parameter must be supplied in order to explicitly preserve expected
|
71
|
+
behavior.)
|
72
|
+
|
73
|
+
Other options may be provided in the future for both the global request
|
74
|
+
and individual operations.
|
75
|
+
|
76
|
+
### Responses
|
77
|
+
|
78
|
+
The Batch API will always return a 200, with a JSON body containing the
|
79
|
+
individual responses under the "results" key. Those responses, in turn,
|
80
|
+
contain the same main components of any HTTP response:
|
81
|
+
|
82
|
+
* _status_ - the HTTP status (200, 201, 400, etc.)
|
83
|
+
* _body_ - the rendered body
|
84
|
+
* _headers_ - any response headers
|
85
|
+
|
86
|
+
### Errors
|
87
|
+
|
88
|
+
Errors in individual Batch API requests will be returned inline, with the
|
89
|
+
same status code and body they would return as individual requests.
|
90
|
+
|
91
|
+
If the Batch API itself returns a non-200 status code, that indicates a global
|
92
|
+
problem.
|
93
|
+
|
94
|
+
## Installation
|
95
|
+
|
96
|
+
Setting up the Batch API is simple. Just add the gem to your middlewares:
|
97
|
+
|
98
|
+
```ruby
|
99
|
+
# in application.rb
|
100
|
+
config.middleware.use BatchApi::RackMiddleware do |batch_config|
|
101
|
+
# you can set various configuration options:
|
102
|
+
batch_config.verb = :put # default :post
|
103
|
+
batch_config.endpoint = "/batchapi" # default /batch
|
104
|
+
batch_config.limit = 100 # how many operations max per request, default 50
|
105
|
+
|
106
|
+
# default middleware stack run for each batch request
|
107
|
+
batch_config.batch_middleware = Proc.new { }
|
108
|
+
# default middleware stack run for each individual operation
|
109
|
+
batch_config.operation_middleware = Proc.new { }
|
110
|
+
end
|
111
|
+
```
|
112
|
+
|
113
|
+
That's it! Just fire up your curl, hit your endpoint with the right verb and a properly formatted request, and enjoy some batch API action.
|
114
|
+
|
115
|
+
## Why a Batch API?
|
116
|
+
|
117
|
+
Batch APIs, though unRESTful, are useful for reducing HTTP overhead
|
118
|
+
by combining requests; this is particularly valuable for mobile clients,
|
119
|
+
which may generate groups of offline actions and which desire to
|
120
|
+
reduce battery consumption while connected by making fewer, better-compressed
|
121
|
+
requests.
|
122
|
+
|
123
|
+
### Why not HTTP Pipelining?
|
124
|
+
|
125
|
+
HTTP pipelining is an awesome and promising technology, and would provide a
|
126
|
+
simple and effortless way to parallel process many requests; however, using
|
127
|
+
pipelining raised several issues for us, one of which was a blocker:
|
128
|
+
|
129
|
+
* [Lack of browser
|
130
|
+
support](http://en.wikipedia.org/wiki/HTTP_pipelining#Implementation_in_web_browsers):
|
131
|
+
a number of key browsers do not yet support HTTP pipelining (or have it
|
132
|
+
disabled by default). This will of course change in time,
|
133
|
+
but for now this takes pipelining out of consideration. (There a similar but
|
134
|
+
more minor issue
|
135
|
+
with [many web
|
136
|
+
proxies](http://en.wikipedia.org/wiki/HTTP_pipelining#Implementation_in_web_proxies).)
|
137
|
+
* The HTTP pipelining specification states that non-idempotent requests (e.g.
|
138
|
+
[POST](http://en.wikipedia.org/wiki/HTTP_pipelining) and
|
139
|
+
[in some
|
140
|
+
descriptions](http://www-archive.mozilla.org/projects/netlib/http/pipelining-faq.html) PUT)
|
141
|
+
shouldn't be made via pipelining. Though I have heard that some server
|
142
|
+
implementations do support POST requests (putting all subsequent requests on
|
143
|
+
hold until it's done), for applications that submit a lot of POSTs this raised
|
144
|
+
concerns as well.
|
145
|
+
|
146
|
+
Given this state of affairs -- and my desire to hack up a Batch API gem :P --,
|
147
|
+
we decided to implement an API-based solution.
|
148
|
+
|
149
|
+
### Why this Approach?
|
150
|
+
|
151
|
+
There are two main approaches to writing batch APIs:
|
152
|
+
|
153
|
+
* A limited, specialized batch endpoint (or endpoints), which usually handles
|
154
|
+
updates and creates. DHH sketched out such a bulk update/create endpoint
|
155
|
+
for Rails 3.2 [in a gist](https://gist.github.com/981520) last year.
|
156
|
+
* A general-purpose RESTful API that can handle anything in your application,
|
157
|
+
a la the Facebook Batch API.
|
158
|
+
|
159
|
+
The second approach, IMO, minimizes code duplication and complexity. Rather
|
160
|
+
than have two systems that manage resources (or a more complicated one that
|
161
|
+
can handle both batch and individual requests), we simply route requests as we
|
162
|
+
always would.
|
163
|
+
|
164
|
+
This solution has several specific benefits:
|
165
|
+
|
166
|
+
* Less complexity - non-batch endpoints don't need any extra code, which means
|
167
|
+
less to maintain on your end.
|
168
|
+
* Complete flexibility - as you add new features to your application,
|
169
|
+
they become immediately and automatically available via the Batch API.
|
170
|
+
* More RESTful - as individual operations are simply actions on RESTful
|
171
|
+
resources, you preserve an important characteristic of your API.
|
172
|
+
|
173
|
+
As well as the general benefits of all batch operations:
|
174
|
+
|
175
|
+
* Reuse of state - user authentication, request stack processing, and
|
176
|
+
similar processing only needs to be done once.
|
177
|
+
* Better for clients - clients need to make fewer requests, as described above.
|
178
|
+
* Parallelizable - in the future, we could run requests in parallel (if
|
179
|
+
our app is thread-safe). Clients would be able to explicitly specify
|
180
|
+
dependencies between operations (or simply run all sequentially). This
|
181
|
+
should make for some fun experimentation :)
|
182
|
+
|
183
|
+
There's only one downside I can think of to this approach as opposed to a
|
184
|
+
specialized endpoint:
|
185
|
+
|
186
|
+
* Reduced ability to optimize - unlike a specialized API endpoint, each request
|
187
|
+
will be treated in isolation, which makes it harder to optimize the
|
188
|
+
underlying database queries via more efficient (read: complicated) SQL logic.
|
189
|
+
(Better identity maps would help with this, and since the main pain point
|
190
|
+
this approach addresses is at the HTTP connection layer, I submit we can
|
191
|
+
accept this.)
|
192
|
+
|
193
|
+
## Implementation
|
194
|
+
|
195
|
+
The Batch API is implemented as a Rack middleware. Here's how it works:
|
196
|
+
|
197
|
+
First, if the request isn't a batch request (as defined by the endpoint and
|
198
|
+
method in BatchApi.config), it gets processed normally by your app.
|
199
|
+
|
200
|
+
If it is a batch request, we:
|
201
|
+
* Read and validate the parameters for the request, constructing a
|
202
|
+
representation of the operation.
|
203
|
+
* Compile a customized Rack environment hash with the appropriate parameters,
|
204
|
+
so that your app interprets the request as being for the appropriate action.
|
205
|
+
(This is requires a bit of extra processing for Rails.)
|
206
|
+
* Send each request up the middleware stack as normal, collecting the results.
|
207
|
+
Errors are caught and recorded appropriately.
|
208
|
+
* Send you back the results.
|
209
|
+
|
210
|
+
At both the batch level (processing all requests) and the individual operation
|
211
|
+
request, there is an internal, customizable midleware stack that you can
|
212
|
+
customize to insert additional custom behavior, such as handling authentication
|
213
|
+
or decoding JSON bodies for individual requests (this latter comes
|
214
|
+
pre-included). Check out the lib/batch_api/internal_middleware.rb for more
|
215
|
+
information.
|
216
|
+
|
217
|
+
## To Do
|
218
|
+
|
219
|
+
The core of the Batch API is complete and solid, and so ready to go that it's
|
220
|
+
in use at 6Wunderkinder already :P
|
221
|
+
|
222
|
+
Here are some immediate tasks:
|
223
|
+
|
224
|
+
* Test against additional frameworks (beyond Rails and Sinatra)
|
225
|
+
* Write more usage docs / create a wiki.
|
226
|
+
* Add additional features inspired by the Facebook API, such as the ability to
|
227
|
+
surpress output for individual requests, etc.
|
228
|
+
* Add RDoc to the spec task and ensure all methods are documented.
|
229
|
+
* Research and implement parallelization and dependency management.
|
230
|
+
|
231
|
+
## Thanks
|
232
|
+
|
233
|
+
To 6Wunderkinder, for all their support for this open-source project, and their
|
234
|
+
general awesomeness.
|
235
|
+
|
236
|
+
To Facebook, for providing inspiration and a great implementation in this and
|
237
|
+
many other things.
|
238
|
+
|
239
|
+
To [JT Archie](http://github.com/jtarchie) for his help and feedback.
|
240
|
+
|
241
|
+
## Issues? Questions? Ideas?
|
242
|
+
|
243
|
+
Open a ticket or send a pull request!
|