tomosia_wallhere_crawl 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: b52398c94e10a3e7d0b5efaacf9f51e5c77565d0c053017221abca624d025932
4
+ data.tar.gz: e8568e2f02a76cea021e43512019b66a554d952819aea064229b02bf3ab2d93f
5
+ SHA512:
6
+ metadata.gz: 336d1c213416c431882c12cdae1296860720466d1e77ca5febe0cc86f182910c37f4a0aa282fdd125ab051b68ffb688e659f62ae42e0988ba36501953d283234
7
+ data.tar.gz: ea2ffbc8c6760cc615184b7978e1066bb15f2b276bb0935af1db7dfaf68cfaac977818afbb3e09f24f832fdaabb38aa46bb23daf3c4836c1e17209b75d2820ae
@@ -0,0 +1,74 @@
1
+ # Contributor Covenant Code of Conduct
2
+
3
+ ## Our Pledge
4
+
5
+ In the interest of fostering an open and welcoming environment, we as
6
+ contributors and maintainers pledge to making participation in our project and
7
+ our community a harassment-free experience for everyone, regardless of age, body
8
+ size, disability, ethnicity, gender identity and expression, level of experience,
9
+ nationality, personal appearance, race, religion, or sexual identity and
10
+ orientation.
11
+
12
+ ## Our Standards
13
+
14
+ Examples of behavior that contributes to creating a positive environment
15
+ include:
16
+
17
+ * Using welcoming and inclusive language
18
+ * Being respectful of differing viewpoints and experiences
19
+ * Gracefully accepting constructive criticism
20
+ * Focusing on what is best for the community
21
+ * Showing empathy towards other community members
22
+
23
+ Examples of unacceptable behavior by participants include:
24
+
25
+ * The use of sexualized language or imagery and unwelcome sexual attention or
26
+ advances
27
+ * Trolling, insulting/derogatory comments, and personal or political attacks
28
+ * Public or private harassment
29
+ * Publishing others' private information, such as a physical or electronic
30
+ address, without explicit permission
31
+ * Other conduct which could reasonably be considered inappropriate in a
32
+ professional setting
33
+
34
+ ## Our Responsibilities
35
+
36
+ Project maintainers are responsible for clarifying the standards of acceptable
37
+ behavior and are expected to take appropriate and fair corrective action in
38
+ response to any instances of unacceptable behavior.
39
+
40
+ Project maintainers have the right and responsibility to remove, edit, or
41
+ reject comments, commits, code, wiki edits, issues, and other contributions
42
+ that are not aligned to this Code of Conduct, or to ban temporarily or
43
+ permanently any contributor for other behaviors that they deem inappropriate,
44
+ threatening, offensive, or harmful.
45
+
46
+ ## Scope
47
+
48
+ This Code of Conduct applies both within project spaces and in public spaces
49
+ when an individual is representing the project or its community. Examples of
50
+ representing a project or community include using an official project e-mail
51
+ address, posting via an official social media account, or acting as an appointed
52
+ representative at an online or offline event. Representation of a project may be
53
+ further defined and clarified by project maintainers.
54
+
55
+ ## Enforcement
56
+
57
+ Instances of abusive, harassing, or otherwise unacceptable behavior may be
58
+ reported by contacting the project team at nguyensontung18183. All
59
+ complaints will be reviewed and investigated and will result in a response that
60
+ is deemed necessary and appropriate to the circumstances. The project team is
61
+ obligated to maintain confidentiality with regard to the reporter of an incident.
62
+ Further details of specific enforcement policies may be posted separately.
63
+
64
+ Project maintainers who do not follow or enforce the Code of Conduct in good
65
+ faith may face temporary or permanent repercussions as determined by other
66
+ members of the project's leadership.
67
+
68
+ ## Attribution
69
+
70
+ This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
71
+ available at [https://contributor-covenant.org/version/1/4][version]
72
+
73
+ [homepage]: https://contributor-covenant.org
74
+ [version]: https://contributor-covenant.org/version/1/4/
data/Gemfile ADDED
@@ -0,0 +1,6 @@
1
+ source "https://rubygems.org"
2
+
3
+ # Specify your gem's dependencies in tomosia_wallhere_crawl.gemspec
4
+ gemspec
5
+
6
+ gem "rake", "~> 12.0"
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2020 TODO: Write your name
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
@@ -0,0 +1,44 @@
1
+ # TomosiaWallhereCrawl
2
+
3
+ Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/tomosia_wallhere_crawl`. To experiment with that code, run `bin/console` for an interactive prompt.
4
+
5
+ TODO: Delete this and the text above, and describe your gem
6
+
7
+ ## Installation
8
+
9
+ Add this line to your application's Gemfile:
10
+
11
+ ```ruby
12
+ gem 'tomosia_wallhere_crawl'
13
+ ```
14
+
15
+ And then execute:
16
+
17
+ $ bundle install
18
+
19
+ Or install it yourself as:
20
+
21
+ $ gem install tomosia_wallhere_crawl
22
+
23
+ ## Usage
24
+
25
+ TODO: Write usage instructions here
26
+
27
+ ## Development
28
+
29
+ After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
30
+
31
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
32
+
33
+ ## Contributing
34
+
35
+ Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/tomosia_wallhere_crawl. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/tomosia_wallhere_crawl/blob/master/CODE_OF_CONDUCT.md).
36
+
37
+
38
+ ## License
39
+
40
+ The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
41
+
42
+ ## Code of Conduct
43
+
44
+ Everyone interacting in the TomosiaWallhereCrawl project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/tomosia_wallhere_crawl/blob/master/CODE_OF_CONDUCT.md).
@@ -0,0 +1,2 @@
1
+ require "bundler/gem_tasks"
2
+ task :default => :spec
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require "bundler/setup"
4
+ require "tomosia_wallhere_crawl"
5
+
6
+ # You can add fixtures and/or initialization code here to make experimenting
7
+ # with your gem easier. You can also use a different console, if you like.
8
+
9
+ # (If you use this, don't forget to add pry to your Gemfile!)
10
+ # require "pry"
11
+ # Pry.start
12
+
13
+ require "irb"
14
+ IRB.start(__FILE__)
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+ IFS=$'\n\t'
4
+ set -vx
5
+
6
+ bundle install
7
+
8
+ # Do any other automated setup that you need to do here
@@ -0,0 +1,72 @@
1
+ require 'open-uri'
2
+ require 'nokogiri'
3
+ require 'writeexcel'
4
+ module TomosiaWallhereCrawl
5
+ class CrawlImage
6
+ def savedata (data = {}, description)
7
+ workbook = WriteExcel.new("#{description}/InfoImage.xls")
8
+ worksheet = workbook.add_worksheet
9
+ data.each_with_index do |row, stt|
10
+ row.each do |key, value|
11
+ worksheet.write_string(stt, 0, row['name'])
12
+ worksheet.write_url(stt, 1, row['url'])
13
+ worksheet.write_string(stt, 2, row['extension'])
14
+ worksheet.write_string(stt, 3, row['size'])
15
+ end
16
+ end
17
+ workbook.close
18
+ end
19
+ def crawldata(key,description,max)
20
+ data=[]
21
+
22
+ sum = 0
23
+ index = 1
24
+ while sum != max do
25
+ # Open url
26
+ url = "https://wallhere.com/en/wallpapers?q=#{key}&page=#{index}"
27
+ document = open(url)
28
+ content = document.read
29
+ parsed_content = Nokogiri::HTML(content)
30
+ length = parsed_content.css('.item').to_a.length - 1
31
+ if length == -1
32
+ break
33
+ else
34
+ i = 0
35
+ for i in i..length
36
+ nameimg = File.basename(parsed_content.css('.item').to_a[i].children.children.first.to_h['src']).delete('.jpg!s')
37
+ # Get url img
38
+ urlimg = parsed_content.css('.item').to_a[i].children.children.first.to_h['src']
39
+ open(urlimg) do |image|
40
+ File.open("#{description}#{nameimg}", "w+") do |file|
41
+ file.write(image.read)
42
+ #Get info img
43
+ n = nameimg
44
+ ui = parsed_content.css('.item').to_a[i].children.children.first.to_h['src']
45
+ ex = File.extname(parsed_content.css('.item').to_a[i].children.children.first.to_h['src']).delete('.!s')
46
+ size = File.size("#{description}#{nameimg}")
47
+ s = "#{size} kb"
48
+ row = {'stt'=>i, 'name'=>n, 'url'=>ui, 'extension'=>ex, 'size'=>s}
49
+ data.push(row)
50
+ end
51
+ end
52
+ sum += 1
53
+ if max == sum
54
+ break
55
+ end
56
+ end
57
+ end
58
+ index += 1
59
+ end
60
+ savedata(data,description)
61
+ end
62
+ def crawl(key,description,max = nil)
63
+ mutithread = (0..2).map do
64
+ Thread.new do
65
+ crawldata(key,description,max)
66
+ end
67
+ end
68
+ mutithread.map(&:join)
69
+ end
70
+ end
71
+ end
72
+
@@ -0,0 +1,3 @@
1
+ module TomosiaWallhereCrawl
2
+ VERSION = "0.1.0"
3
+ end
@@ -0,0 +1,22 @@
1
+ require_relative 'lib/tomosia_wallhere_crawl/version'
2
+
3
+ Gem::Specification.new do |spec|
4
+ spec.name = "tomosia_wallhere_crawl"
5
+ spec.version = TomosiaWallhereCrawl::VERSION
6
+ spec.authors = ["nguyen son tung"]
7
+ spec.email = ["nguyensontung18183"]
8
+
9
+ spec.summary = %q{tomosia_wallhere_crawl using download image from wallhere}
10
+ spec.description = %q{Write a longer description or delete this line.}
11
+ spec.homepage = "https://github.com/nguyensontung183183/tomosia_wallhere_crawl.git"
12
+ spec.license = "MIT"
13
+ spec.required_ruby_version = Gem::Requirement.new(">= 2.3.0")
14
+ # Specify which files should be added to the gem when it is released.
15
+ # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
16
+ spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
17
+ `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
18
+ end
19
+ spec.bindir = "exe"
20
+ spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
21
+ spec.require_paths = ["lib"]
22
+ end
metadata ADDED
@@ -0,0 +1,53 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: tomosia_wallhere_crawl
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - nguyen son tung
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2020-08-05 00:00:00.000000000 Z
12
+ dependencies: []
13
+ description: Write a longer description or delete this line.
14
+ email:
15
+ - nguyensontung18183
16
+ executables: []
17
+ extensions: []
18
+ extra_rdoc_files: []
19
+ files:
20
+ - CODE_OF_CONDUCT.md
21
+ - Gemfile
22
+ - LICENSE.txt
23
+ - README.md
24
+ - Rakefile
25
+ - bin/console
26
+ - bin/setup
27
+ - lib/tomosia_wallhere_crawl.rb
28
+ - lib/tomosia_wallhere_crawl/version.rb
29
+ - tomosia_wallhere_crawl.gemspec
30
+ homepage: https://github.com/nguyensontung183183/tomosia_wallhere_crawl.git
31
+ licenses:
32
+ - MIT
33
+ metadata: {}
34
+ post_install_message:
35
+ rdoc_options: []
36
+ require_paths:
37
+ - lib
38
+ required_ruby_version: !ruby/object:Gem::Requirement
39
+ requirements:
40
+ - - ">="
41
+ - !ruby/object:Gem::Version
42
+ version: 2.3.0
43
+ required_rubygems_version: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ requirements: []
49
+ rubygems_version: 3.0.1
50
+ signing_key:
51
+ specification_version: 4
52
+ summary: tomosia_wallhere_crawl using download image from wallhere
53
+ test_files: []