tsumamigui 0.1.1 → 0.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +19 -0
- data/CODE_OF_CONDUCT.md +74 -0
- data/README.md +1 -2
- data/lib/tsumamigui/parser.rb +9 -5
- data/lib/tsumamigui/request.rb +4 -2
- data/lib/tsumamigui/response.rb +5 -1
- data/lib/tsumamigui/version.rb +1 -1
- metadata +4 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 56fa99aed4b2fe61611547e801cbd2ad15896abf
|
4
|
+
data.tar.gz: 5e573542ae77da0b3b39a4113af032ab0a4a1996
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3abaced2852c68fc95646b3eb006d5076d391626c173d2760fec1475d6f100d273795693fa9d2dbe7ede76827c8ced1bcb9f901ce6534de976a7f036a0234587
|
7
|
+
data.tar.gz: ae8e0cab6e93a2740092fa94acaa25c8b156cc499f468cbbc54a54f6f7094ef0558204a6d0e58c72c823c2510b8756b6378e75f91f98dac008a0c4d9eb4acf02
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
# 0.1.2 / 2017-10-24
|
2
|
+
|
3
|
+
## Notes
|
4
|
+
|
5
|
+
* Revise code comments :memo:
|
6
|
+
|
7
|
+
# 0.1.1 / 2017-07-12
|
8
|
+
|
9
|
+
## Bug fixes
|
10
|
+
|
11
|
+
* Fix lint error :bug:
|
12
|
+
|
13
|
+
# 0.1.0 / 2017-07-12
|
14
|
+
|
15
|
+
`Tsumamigui` has been released! :tada:
|
16
|
+
|
17
|
+
## Features
|
18
|
+
|
19
|
+
* Scrape urls with xpath.
|
data/CODE_OF_CONDUCT.md
ADDED
@@ -0,0 +1,74 @@
|
|
1
|
+
# Contributor Covenant Code of Conduct
|
2
|
+
|
3
|
+
## Our Pledge
|
4
|
+
|
5
|
+
In the interest of fostering an open and welcoming environment, we as
|
6
|
+
contributors and maintainers pledge to making participation in our project and
|
7
|
+
our community a harassment-free experience for everyone, regardless of age, body
|
8
|
+
size, disability, ethnicity, gender identity and expression, level of experience,
|
9
|
+
nationality, personal appearance, race, religion, or sexual identity and
|
10
|
+
orientation.
|
11
|
+
|
12
|
+
## Our Standards
|
13
|
+
|
14
|
+
Examples of behavior that contributes to creating a positive environment
|
15
|
+
include:
|
16
|
+
|
17
|
+
* Using welcoming and inclusive language
|
18
|
+
* Being respectful of differing viewpoints and experiences
|
19
|
+
* Gracefully accepting constructive criticism
|
20
|
+
* Focusing on what is best for the community
|
21
|
+
* Showing empathy towards other community members
|
22
|
+
|
23
|
+
Examples of unacceptable behavior by participants include:
|
24
|
+
|
25
|
+
* The use of sexualized language or imagery and unwelcome sexual attention or
|
26
|
+
advances
|
27
|
+
* Trolling, insulting/derogatory comments, and personal or political attacks
|
28
|
+
* Public or private harassment
|
29
|
+
* Publishing others' private information, such as a physical or electronic
|
30
|
+
address, without explicit permission
|
31
|
+
* Other conduct which could reasonably be considered inappropriate in a
|
32
|
+
professional setting
|
33
|
+
|
34
|
+
## Our Responsibilities
|
35
|
+
|
36
|
+
Project maintainers are responsible for clarifying the standards of acceptable
|
37
|
+
behavior and are expected to take appropriate and fair corrective action in
|
38
|
+
response to any instances of unacceptable behavior.
|
39
|
+
|
40
|
+
Project maintainers have the right and responsibility to remove, edit, or
|
41
|
+
reject comments, commits, code, wiki edits, issues, and other contributions
|
42
|
+
that are not aligned to this Code of Conduct, or to ban temporarily or
|
43
|
+
permanently any contributor for other behaviors that they deem inappropriate,
|
44
|
+
threatening, offensive, or harmful.
|
45
|
+
|
46
|
+
## Scope
|
47
|
+
|
48
|
+
This Code of Conduct applies both within project spaces and in public spaces
|
49
|
+
when an individual is representing the project or its community. Examples of
|
50
|
+
representing a project or community include using an official project e-mail
|
51
|
+
address, posting via an official social media account, or acting as an appointed
|
52
|
+
representative at an online or offline event. Representation of a project may be
|
53
|
+
further defined and clarified by project maintainers.
|
54
|
+
|
55
|
+
## Enforcement
|
56
|
+
|
57
|
+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
58
|
+
reported by contacting the project team at obi.yuta@gmail.com. All
|
59
|
+
complaints will be reviewed and investigated and will result in a response that
|
60
|
+
is deemed necessary and appropriate to the circumstances. The project team is
|
61
|
+
obligated to maintain confidentiality with regard to the reporter of an incident.
|
62
|
+
Further details of specific enforcement policies may be posted separately.
|
63
|
+
|
64
|
+
Project maintainers who do not follow or enforce the Code of Conduct in good
|
65
|
+
faith may face temporary or permanent repercussions as determined by other
|
66
|
+
members of the project's leadership.
|
67
|
+
|
68
|
+
## Attribution
|
69
|
+
|
70
|
+
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
71
|
+
available at [http://contributor-covenant.org/version/1/4][version]
|
72
|
+
|
73
|
+
[homepage]: http://contributor-covenant.org
|
74
|
+
[version]: http://contributor-covenant.org/version/1/4/
|
data/README.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
# Tsumamigui
|
2
|
-
[![Gem Version](https://badge.fury.io/rb/tsumamigui.svg)](https://badge.fury.io/rb/tsumamigui) ![circleci](https://circleci.com/gh/obiyuta/tsumamigui.svg?&style=shield&circle-token=8e2bda0f04504c2fe43d3fe70ea8ab1b6184806f) [![Code Climate](https://codeclimate.com/github/obiyuta/tsumamigui/badges/gpa.svg)](https://codeclimate.com/github/obiyuta/tsumamigui) [![Test Coverage](https://codeclimate.com/github/obiyuta/tsumamigui/badges/coverage.svg)](https://codeclimate.com/github/obiyuta/tsumamigui/coverage) [![Dependency Status](https://gemnasium.com/badges/github.com/obiyuta/tsumamigui.svg)](https://gemnasium.com/github.com/obiyuta/tsumamigui) [![Inline docs](http://inch-ci.org/github/obiyuta/tsumamigui.svg?branch=master)](http://inch-ci.org/github/obiyuta/tsumamigui)
|
2
|
+
[![Gem Version](https://badge.fury.io/rb/tsumamigui.svg)](https://badge.fury.io/rb/tsumamigui) [![circleci](https://circleci.com/gh/obiyuta/tsumamigui.svg?&style=shield&circle-token=8e2bda0f04504c2fe43d3fe70ea8ab1b6184806f)](https://circleci.com/gh/obiyuta/tsumamigui/tree/master) [![Code Climate](https://codeclimate.com/github/obiyuta/tsumamigui/badges/gpa.svg)](https://codeclimate.com/github/obiyuta/tsumamigui) [![Test Coverage](https://codeclimate.com/github/obiyuta/tsumamigui/badges/coverage.svg)](https://codeclimate.com/github/obiyuta/tsumamigui/coverage) [![Dependency Status](https://gemnasium.com/badges/github.com/obiyuta/tsumamigui.svg)](https://gemnasium.com/github.com/obiyuta/tsumamigui) [![Inline docs](http://inch-ci.org/github/obiyuta/tsumamigui.svg?branch=master)](http://inch-ci.org/github/obiyuta/tsumamigui)
|
3
3
|
|
4
4
|
Tsumamigui(つまみぐい) is a simple and hussle-free Ruby web scraping library.
|
5
5
|
|
@@ -52,7 +52,6 @@ ___Important:___ Tsumamigui requests each urls at intervals of 1.0~3.0sec automa
|
|
52
52
|
## TODO
|
53
53
|
|
54
54
|
- [ ] Custom request headers.
|
55
|
-
- [ ] SSL support. (unchecked)
|
56
55
|
|
57
56
|
etc...
|
58
57
|
|
data/lib/tsumamigui/parser.rb
CHANGED
@@ -3,12 +3,13 @@ require 'tsumamigui/error'
|
|
3
3
|
|
4
4
|
module Tsumamigui
|
5
5
|
class Parser
|
6
|
+
# xpath to getting the data from
|
6
7
|
attr_reader :xpath
|
7
8
|
|
8
9
|
class << self
|
9
10
|
# @param [Array<Tsumamigui::Response>] responses
|
10
11
|
# @param [Hash] xpath
|
11
|
-
# @return [Array<Hash>]
|
12
|
+
# @return [Array<Hash>] parsed responses
|
12
13
|
def parse(responses, xpath)
|
13
14
|
new(xpath).send(:parse, responses)
|
14
15
|
end
|
@@ -21,7 +22,9 @@ module Tsumamigui
|
|
21
22
|
|
22
23
|
private
|
23
24
|
|
24
|
-
#
|
25
|
+
# Parse response data into hash object
|
26
|
+
# @param [Array<Tsumamigui::Response>] responses
|
27
|
+
# @return [Array<Hash>] parsed responses
|
25
28
|
# @raise [Tsumamigui::ParserError]
|
26
29
|
def parse(responses)
|
27
30
|
results = []
|
@@ -36,13 +39,14 @@ module Tsumamigui
|
|
36
39
|
raise ParserError, e.message
|
37
40
|
end
|
38
41
|
|
42
|
+
# Extract data from parsed html with xpath
|
39
43
|
# @param [Object] document Nokogiri::HTML::Document
|
40
|
-
# @return [Hash]
|
44
|
+
# @return [Hash] xpath and its key
|
41
45
|
# @raise [Tsumamigui::ParserError]
|
42
|
-
def extract(
|
46
|
+
def extract(document)
|
43
47
|
result = {}
|
44
48
|
@xpath.each do |k, v|
|
45
|
-
result[k] =
|
49
|
+
result[k] = document.xpath(v).to_s
|
46
50
|
end
|
47
51
|
result
|
48
52
|
rescue => e
|
data/lib/tsumamigui/request.rb
CHANGED
@@ -7,7 +7,7 @@ module Tsumamigui
|
|
7
7
|
class << self
|
8
8
|
# @param [String] urls
|
9
9
|
# @param [Array<String>] urls
|
10
|
-
# @return [Array<Tsumamigui::Response>]
|
10
|
+
# @return [Array<Tsumamigui::Response>] response instances
|
11
11
|
def run(*urls)
|
12
12
|
new(urls).send(:run)
|
13
13
|
end
|
@@ -27,7 +27,8 @@ module Tsumamigui
|
|
27
27
|
|
28
28
|
private
|
29
29
|
|
30
|
-
#
|
30
|
+
# Execute request and return responses
|
31
|
+
# @return [Array<Tsumamigui::Response>] response instances
|
31
32
|
def run
|
32
33
|
results = []
|
33
34
|
@urls.each do |url|
|
@@ -36,6 +37,7 @@ module Tsumamigui
|
|
36
37
|
results
|
37
38
|
end
|
38
39
|
|
40
|
+
# Fetch HTML from url and convert to response object
|
39
41
|
# @param [String] url
|
40
42
|
# @return [Tsumamigui::Response] response
|
41
43
|
# @raise [Tsumamigui::RequestError]
|
data/lib/tsumamigui/response.rb
CHANGED
@@ -1,7 +1,10 @@
|
|
1
1
|
module Tsumamigui
|
2
2
|
class Response
|
3
|
+
# request url
|
3
4
|
attr_reader :url
|
5
|
+
# fetched html
|
4
6
|
attr_reader :html
|
7
|
+
# charset of fetched html
|
5
8
|
attr_reader :charset
|
6
9
|
|
7
10
|
# @param [String] url
|
@@ -13,7 +16,8 @@ module Tsumamigui
|
|
13
16
|
@charset = charset
|
14
17
|
end
|
15
18
|
|
16
|
-
#
|
19
|
+
# Convert request data to array
|
20
|
+
# @return [Array] [url, html, charset]
|
17
21
|
def to_array
|
18
22
|
[@url, @html, @charset]
|
19
23
|
end
|
data/lib/tsumamigui/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: tsumamigui
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- obiyuta
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-
|
11
|
+
date: 2017-10-24 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: nokogiri
|
@@ -161,6 +161,8 @@ files:
|
|
161
161
|
- ".pryrc"
|
162
162
|
- ".rspec"
|
163
163
|
- ".rubocop.yml"
|
164
|
+
- CHANGELOG.md
|
165
|
+
- CODE_OF_CONDUCT.md
|
164
166
|
- Gemfile
|
165
167
|
- LICENSE.txt
|
166
168
|
- README.md
|