prefetcher 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: bda803d355067711c1c81142eaaf80928a4bd2ac
4
+ data.tar.gz: 55f9e91d9f0f4f7d2d50d47d6454d799a031bbad
5
+ SHA512:
6
+ metadata.gz: 11f7a3809331c5895df81bbc9882712f573bca07c47da1af20e96b06ad511dbde9155a9cfc70d3609aaa698be3ed87135f50ed61ea27438b76f18b8ac0406e59
7
+ data.tar.gz: 0d4081babac5f0ed943fed7944ff4c02ba436030ce6d49b3b3cd53b5f112ba733b6a5297d0bd3e074d2182a450c7f08291c8a79944834252f35471c55eab0ba8
@@ -0,0 +1,22 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
18
+ *.bundle
19
+ *.so
20
+ *.o
21
+ *.a
22
+ mkmf.log
@@ -0,0 +1,18 @@
1
+ language: ruby
2
+
3
+ addons:
4
+ code_climate:
5
+ repo_token: c17041d1834d6f6ff02afdcc14e2537fb8255218240f72de22db6951c4efdd82
6
+
7
+ rvm:
8
+ - 1.9.3
9
+ - 2.0.0
10
+ - 2.1.1
11
+ - 2.1.2
12
+ - rbx-2.2.6
13
+ - jruby
14
+
15
+ matrix:
16
+ allow_failures:
17
+ - rvm: rbx-2.2.6
18
+ - rvm: jruby
data/Gemfile ADDED
@@ -0,0 +1,15 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in fetcher.gemspec
4
+ gemspec
5
+
6
+ gem 'mock_redis'
7
+
8
+ gem 'rspec'
9
+
10
+ gem 'ffaker'
11
+ gem 'fakeweb'
12
+ gem 'pry'
13
+ gem 'pry-nav'
14
+
15
+ gem "codeclimate-test-reporter", group: :test, require: nil
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2014 Alex Rozumey
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,62 @@
1
+ # Prefetcher
2
+ [![Build Status](https://travis-ci.org/brain-geek/prefetcher.svg?branch=master)](https://travis-ci.org/brain-geek/prefetcher)
3
+ [![Code Climate](https://codeclimate.com/github/brain-geek/prefetcher/badges/gpa.svg)](https://codeclimate.com/github/brain-geek/prefetcher)
4
+ [![Test Coverage](https://codeclimate.com/github/brain-geek/prefetcher/badges/coverage.svg)](https://codeclimate.com/github/brain-geek/prefetcher)
5
+
6
+ This gem provides a simple-to-use interface to work with frequently requested http requests from your api. It gets request response from memory, if possible. But also this means you have to update this cache from time to time (using [whenever](https://github.com/javan/whenever), for example). Any kind of non-200 responses will not be memoized, so you can be always sure that you don't use broken data. Redis is used to store data. [RDoc](http://rdoc.info/github/brain-geek/prefetcher/master/frames)
7
+
8
+ ## Installation
9
+
10
+ Add this line to your application's Gemfile:
11
+
12
+ gem 'prefetcher'
13
+
14
+ And then execute:
15
+
16
+ $ bundle
17
+
18
+ Or install it yourself as:
19
+
20
+ $ gem install prefetcher
21
+
22
+ You can also override redis connection details (if not using default localhost:6379 ):
23
+
24
+ Prefetcher.redis_connection = Redis.new(:host => "10.0.1.1", :port => 6380, :db => 15)
25
+
26
+ See [redis gem documentation](https://github.com/redis/redis-rb#getting-started) for more options when creating redis connection.
27
+
28
+ ## Usage
29
+
30
+ ### Using cached requests
31
+
32
+ After installing project you can request any URL:
33
+
34
+ Prefetcher::HttpPrefetcher.new('http://www.reddit.com/r/ruby').get
35
+
36
+ Calling #get any number of times will return data from cache.
37
+
38
+ ### Force get
39
+
40
+ If you want to force request (and save the response), you can call #fetch:
41
+
42
+ Prefetcher::HttpPrefetcher.new('http://www.reddit.com/r/ruby').fetch
43
+
44
+ This will cause actual http request.
45
+
46
+ ### Updating cache
47
+
48
+ Calling manualy. You can call *Prefetcher.update_all* to fetch all URLs right now.
49
+
50
+ You can also automate this call using [whenever](https://github.com/javan/whenever). Just add this code to your schedule.rb .
51
+
52
+ every 30.minutes do
53
+ runner "Prefetcher.update_all"
54
+ end%
55
+
56
+ ## Contributing
57
+
58
+ 1. Fork it ( https://github.com/brain-geek/prefetcher/fork )
59
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
60
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
61
+ 4. Push to the branch (`git push origin my-new-feature`)
62
+ 5. Create a new Pull Request
@@ -0,0 +1,6 @@
1
+ require "bundler/gem_tasks"
2
+ require "rspec/core/rake_task"
3
+
4
+ RSpec::Core::RakeTask.new
5
+
6
+ task :default => :spec
@@ -0,0 +1,26 @@
1
+ require "redis"
2
+ require "active_support/core_ext/string/output_safety"
3
+ require "active_support/core_ext/hash/except"
4
+
5
+ require "prefetcher/http_fetcher"
6
+ require "prefetcher/http_memoizer"
7
+
8
+ require "prefetcher/version"
9
+
10
+ module Prefetcher
11
+
12
+ # Updates all memoized requests
13
+ def self.update_all(options = {})
14
+ HttpMemoizer.new(options).get_list.each do |url|
15
+ HttpFetcher.new(options.merge(url: url)).fetch
16
+ end
17
+ end
18
+
19
+ def self.redis_connection
20
+ @redis_connection ||= Redis.new
21
+ end
22
+
23
+ def self.redis_connection=(conn)
24
+ @redis_connection = conn
25
+ end
26
+ end
@@ -0,0 +1,47 @@
1
+ module Prefetcher
2
+ class HttpFetcher
3
+ attr_reader :url, :redis_connection, :memoizer
4
+
5
+ def initialize(params = {})
6
+ @url = params.fetch(:url)
7
+ @redis_connection = params.fetch(:redis_connection, Prefetcher.redis_connection)
8
+ @memoizer = params.fetch(:memoizer, HttpMemoizer.new(redis_connection: @redis_connection))
9
+ end
10
+
11
+ # Makes request to given URL
12
+ def fetch
13
+ uri = URI(URI.encode(self.url))
14
+
15
+ http = Net::HTTP.new(uri.host, uri.port)
16
+ request = Net::HTTP::Get.new(uri.request_uri)
17
+
18
+ response = http.request(request)
19
+
20
+ if response.code == "200"
21
+ memoize(response.body)
22
+ response.body
23
+ else
24
+ ''
25
+ end
26
+ end
27
+
28
+ # Returns cached version if availible. If not cached - makes request using #fetch .
29
+ def get
30
+ (get_from_memory || fetch).html_safe.force_encoding('utf-8')
31
+ end
32
+
33
+ protected
34
+ def cache_key
35
+ "cached-url-#{url}"
36
+ end
37
+
38
+ def get_from_memory
39
+ @redis_connection.get(cache_key)
40
+ end
41
+
42
+ def memoize(response)
43
+ memoizer.push(url)
44
+ @redis_connection.set(cache_key, response)
45
+ end
46
+ end
47
+ end
@@ -0,0 +1,28 @@
1
+ module Prefetcher
2
+ class HttpMemoizer
3
+ attr_reader :redis_connection
4
+
5
+ def initialize(params = {})
6
+ @redis_connection = params.fetch(:redis_connection, Prefetcher.redis_connection)
7
+ end
8
+
9
+ # Add URL to memoized list
10
+ def push(url)
11
+ redis.sadd(cache_key, url)
12
+ end
13
+
14
+ # Get all memoized URLs
15
+ def get_list
16
+ redis.smembers cache_key
17
+ end
18
+
19
+ protected
20
+ def cache_key
21
+ "urls-list"
22
+ end
23
+
24
+ def redis
25
+ Prefetcher.redis_connection
26
+ end
27
+ end
28
+ end
@@ -0,0 +1,3 @@
1
+ module Prefetcher
2
+ VERSION = "0.0.2"
3
+ end
@@ -0,0 +1,26 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'prefetcher/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "prefetcher"
8
+ spec.version = Prefetcher::VERSION
9
+ spec.authors = ["Alex Rozumiy"]
10
+ spec.email = ["brain-geek@yandex.ua"]
11
+ spec.summary = %q{Prefetching/caching tool for external requests}
12
+ spec.description = %q{This gem provides possibility to have 'fresh' prefetched result of external http request all the time}
13
+ spec.homepage = ""
14
+ spec.license = "MIT"
15
+
16
+ spec.files = `git ls-files -z`.split("\x0")
17
+ spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
18
+ spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
19
+ spec.require_paths = ["lib"]
20
+
21
+ spec.add_dependency "redis"
22
+ spec.add_dependency "activesupport"
23
+
24
+ spec.add_development_dependency "bundler", "~> 1.6"
25
+ spec.add_development_dependency "rake"
26
+ end
@@ -0,0 +1,135 @@
1
+ require 'spec_helper'
2
+
3
+ describe Prefetcher::HttpFetcher do
4
+ let(:redis_connection) { MockRedis.new }
5
+
6
+ let(:default_params) { Hash[url: url, redis_connection: redis_connection] }
7
+ let(:params) { default_params }
8
+ let(:object) { described_class.new(params) }
9
+
10
+ let(:url) { Faker::Internet.http_url }
11
+ let(:request_body) { Faker::HTMLIpsum.ul_short }
12
+
13
+ describe "#initialize" do
14
+ it "fails if no url given" do
15
+ expect { described_class.new(default_params.except(:url)).to raise_error }
16
+ end
17
+
18
+ it "uses default connection if not set explicitly" do
19
+ Prefetcher.redis_connection = (connection = double('Redis'))
20
+ object = described_class.new(default_params.except(:redis_connection))
21
+
22
+ expect(object.redis_connection).to be connection
23
+ expect(object.memoizer.redis_connection).to be connection
24
+ end
25
+ end
26
+
27
+ describe "#fetch" do
28
+ subject { object.fetch }
29
+
30
+ describe "200 response" do
31
+ before { FakeWeb.register_uri(:get, url, :body => request_body) }
32
+
33
+ it "gets data from real world http query" do
34
+ expect(subject).to eq request_body
35
+ end
36
+
37
+ it "makes http requests the same number of times as called" do
38
+ FakeWeb.register_uri(:get, url,
39
+ [{:body => "1", :status => ["200", "OK"]},
40
+ {:body => "2", :status => ["200", "OK"]},
41
+ {:body => "3", :status => ["200", "OK"]}])
42
+
43
+ expect(object.fetch).to eq "1"
44
+ expect(object.fetch).to eq "2"
45
+ expect(object.fetch).to eq "3"
46
+ expect(object.fetch).to eq "3" # fakeweb feature - when no more responces, it uses last
47
+ end
48
+
49
+ describe "saves output to redis" do
50
+ it "to corresponding key" do
51
+ expect(redis_connection.get("cached-url-#{url}")).to be_nil
52
+
53
+ subject
54
+
55
+ expect(redis_connection.get("cached-url-#{url}")).to eq request_body
56
+ end
57
+ end
58
+
59
+ it "also pushes given url to url_memoizer" do
60
+ expect(object.memoizer).to receive(:push).with(url)
61
+ subject
62
+ end
63
+ end
64
+
65
+ describe "500 response" do
66
+ before do
67
+ FakeWeb.register_uri(:get, url, body: request_body, status: ["500", "Internal Server Error"])
68
+ end
69
+
70
+ it "returns empty string" do
71
+ expect(subject).to be_empty
72
+ end
73
+
74
+ it "does not write this to cache" do
75
+ subject
76
+ expect(redis_connection.get("cached-url-#{url}")).to be_nil
77
+ end
78
+ end
79
+
80
+ describe "404 response" do
81
+ before do
82
+ FakeWeb.register_uri(:get, url, body: request_body, status: ["404", "Not Found"])
83
+ end
84
+
85
+ it "returns empty string" do
86
+ expect(subject).to be_empty
87
+ end
88
+
89
+ it "does not write this to cache" do
90
+ subject
91
+ expect(redis_connection.get("cached-url-#{url}")).to be_nil
92
+ end
93
+ end
94
+ end
95
+
96
+ describe "#get" do
97
+ let(:request_body) { Faker::HTMLIpsum.ul_short.force_encoding('US-ASCII') } # real-world case with encoding
98
+ subject { object.get }
99
+
100
+ describe "when no data already present in cache" do
101
+ before { expect(object).to receive(:fetch).and_return(request_body) }
102
+
103
+ it "calls #fetch" do
104
+ expect(subject).to eq request_body
105
+ end
106
+
107
+ it "returns html safe value" do
108
+ expect(subject).to be_html_safe
109
+ end
110
+
111
+ it "returns utf-8 encoded string" do
112
+ expect(subject.encoding.to_s).to eq "UTF-8"
113
+ end
114
+ end
115
+
116
+ describe "when there is data in cache" do
117
+ before do
118
+ expect(object).to_not receive(:fetch)
119
+ redis_connection.set("cached-url-#{url}", request_body)
120
+ end
121
+
122
+ it "returns data only from cache" do
123
+ expect(subject).to eq request_body
124
+ end
125
+
126
+ it "returns html safe value" do
127
+ expect(subject).to be_html_safe
128
+ end
129
+
130
+ it "returns utf-8 encoded string" do
131
+ expect(subject.encoding.to_s).to eq "UTF-8"
132
+ end
133
+ end
134
+ end
135
+ end
@@ -0,0 +1,31 @@
1
+ require 'spec_helper'
2
+
3
+ describe Prefetcher::HttpMemoizer do
4
+ describe "#get_list" do
5
+ let(:memoizer) { described_class.new }
6
+ subject { memoizer.get_list }
7
+
8
+ it "returns empty array by default" do
9
+ expect(subject).to be_empty
10
+ end
11
+
12
+ it "returns the url once, even if it was memorized multiple times" do
13
+ url = Faker::Internet.http_url
14
+
15
+ memoizer.push url
16
+ memoizer.push url
17
+
18
+ expect(subject).to eq [url]
19
+ end
20
+
21
+ it "returns all given urls" do
22
+ urls = 3.times.map { Faker::Internet.http_url }
23
+
24
+ urls.each do |url|
25
+ memoizer.push url
26
+ end
27
+
28
+ expect(subject.sort).to eq urls.sort
29
+ end
30
+ end
31
+ end
@@ -0,0 +1,40 @@
1
+ require 'spec_helper'
2
+
3
+ describe Prefetcher do
4
+ describe "redis connection accessors" do
5
+ it "returns previously set connection" do
6
+ described_class.redis_connection = (conn = double("Redis"))
7
+
8
+ expect(described_class.redis_connection).to eq conn
9
+ end
10
+
11
+ it "uses Redis.new without arguments by default" do
12
+ expect(Redis).to receive(:new).with(no_args).and_return(conn = double("Redis"))
13
+
14
+ expect(described_class.redis_connection).to_not eq conn
15
+
16
+ described_class.redis_connection = nil
17
+
18
+ expect(described_class.redis_connection).to eq conn
19
+ end
20
+ end
21
+
22
+ describe "::update_all" do
23
+ let(:url) { Faker::Internet.http_url }
24
+
25
+ it "should update data in HttpFetcher fetched URLs" do
26
+ FakeWeb.register_uri(:get, url,
27
+ [{:body => "1", :status => ["200", "OK"]},
28
+ {:body => "2", :status => ["200", "OK"]}])
29
+
30
+ obj = Prefetcher::HttpFetcher.new(url: url)
31
+
32
+ expect(obj.get).to eq "1"
33
+ expect(obj.get).to eq "1"
34
+
35
+ described_class.update_all
36
+
37
+ expect(obj.get).to eq "2"
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,18 @@
1
+ require 'rubygems'
2
+ require 'bundler/setup'
3
+
4
+ require "codeclimate-test-reporter"
5
+ CodeClimate::TestReporter.start
6
+
7
+ require 'prefetcher'
8
+
9
+ Bundler.require
10
+
11
+ # Disabling old rspec should syntax
12
+ RSpec.configure do |config|
13
+ config.raise_errors_for_deprecations!
14
+
15
+ config.before(:each) do
16
+ Prefetcher.redis_connection = MockRedis.new
17
+ end
18
+ end
metadata ADDED
@@ -0,0 +1,120 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: prefetcher
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.2
5
+ platform: ruby
6
+ authors:
7
+ - Alex Rozumiy
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2014-08-26 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: redis
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - '>='
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - '>='
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: activesupport
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - '>='
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - '>='
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: bundler
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ~>
46
+ - !ruby/object:Gem::Version
47
+ version: '1.6'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ~>
53
+ - !ruby/object:Gem::Version
54
+ version: '1.6'
55
+ - !ruby/object:Gem::Dependency
56
+ name: rake
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - '>='
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - '>='
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ description: This gem provides possibility to have 'fresh' prefetched result of external
70
+ http request all the time
71
+ email:
72
+ - brain-geek@yandex.ua
73
+ executables: []
74
+ extensions: []
75
+ extra_rdoc_files: []
76
+ files:
77
+ - .gitignore
78
+ - .travis.yml
79
+ - Gemfile
80
+ - LICENSE.txt
81
+ - README.md
82
+ - Rakefile
83
+ - lib/prefetcher.rb
84
+ - lib/prefetcher/http_fetcher.rb
85
+ - lib/prefetcher/http_memoizer.rb
86
+ - lib/prefetcher/version.rb
87
+ - prefetcher.gemspec
88
+ - spec/prefetcher/http_fetcher_spec.rb
89
+ - spec/prefetcher/http_memoizer_spec.rb
90
+ - spec/prefetcher_spec.rb
91
+ - spec/spec_helper.rb
92
+ homepage: ''
93
+ licenses:
94
+ - MIT
95
+ metadata: {}
96
+ post_install_message:
97
+ rdoc_options: []
98
+ require_paths:
99
+ - lib
100
+ required_ruby_version: !ruby/object:Gem::Requirement
101
+ requirements:
102
+ - - '>='
103
+ - !ruby/object:Gem::Version
104
+ version: '0'
105
+ required_rubygems_version: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - '>='
108
+ - !ruby/object:Gem::Version
109
+ version: '0'
110
+ requirements: []
111
+ rubyforge_project:
112
+ rubygems_version: 2.0.6
113
+ signing_key:
114
+ specification_version: 4
115
+ summary: Prefetching/caching tool for external requests
116
+ test_files:
117
+ - spec/prefetcher/http_fetcher_spec.rb
118
+ - spec/prefetcher/http_memoizer_spec.rb
119
+ - spec/prefetcher_spec.rb
120
+ - spec/spec_helper.rb