docapurl 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 99de94810d60e3d3c5e934c7dff632f7b6d23c841d02d1359590191f091e6c16
-  data.tar.gz: 46c12c213e11dce773bc5982cb7327ec985830893007f8b6e4277cf681455611
+  metadata.gz: c175e73c80948788100dfec7dab654d13cd0ab9c2f1e1f195cb27be55c858b23
+  data.tar.gz: e8c774fd4072aa0cd047afbc7ec7345e9f9a145717ba3c058342502cae6ff6ba
 SHA512:
-  metadata.gz: 3058e5f01ee00295ad7d5c9442799e2a41d65507cb8cc189e674f21447d524b5ad14b99f215fab12d9924baf09f2c035c81d7cc9c5ea9318d40425399448bc93
-  data.tar.gz: e031c7ab870d9c5cfb53ef1c90231c4a1b7250b32fc1507a4a4fbfc0ca07d6d4c6f69debc94379f921ba68e4c67b749eddfb09e1caed3770f38a3411a5bfc040
+  metadata.gz: 1aa1857f6b4309ca714ddf6eb53c36bfdde2191db93efc07d2e23df4209a507ade2d48a8e0959710dd229c560642261d2c204dad99262b57d5878cbaa9321c65
+  data.tar.gz: 8478697b36cc173f4efb4e02f01ea42e34d3c31300df6e1300599c19d2bf1b36dc159ed3389238780ffa46847ef9dc67c3d3b0a28802a2d15378a9f96b916835

data/CH_README.md

@@ -0,0 +1,77 @@
+# Docapurl
+一个ruby的命令行截图网页的工具
+
+## 抓取网页线上服务
+
+[https://www.urlprint.com](https://www.urlprint.com) 提供在线抓取网页生成图片的服务, 支持 REST API 调用, 欢迎各位大佬试用
+
+## 安装
+
+gem install docapurl
+
+
+## 前置条件
+
+- ruby 环境
+- chrome
+
+需要安装chrome 浏览器, 并推荐把chrome的path 加入到环境变量PATH 或者 BROWSER_PATH 中
+
+## 使用方式
+
+on terminal
+
+`docapurl cap [url] [image_path]`
+
+使用 `docapurl help cap` 获得更多帮助
+
+## Example
+
+```
+ docapurl cap https:/www.bilibili.com 1.jpg --pagedown-to-bottom
+```
+
+## FAQ
+
+-  为啥docapurl 需要 chrome 以及在哪下载chrome?
+
+因为 docapurl 封装了 ferrum gem 的一些截图功能, 而 ferrum 依赖于 headless Chrome .
+linux 的chrome 一般都不太靠谱, 所以在这里下载官方的 https://www.chromium.org/getting-involved/download-chromium
+
+windows 和mac 用户, 则可以在这里下载 https://www.google.com/chrome/.
+
+下载安装完毕 chrome 后, 强烈建议将浏览器path 加入到 path 变量中
+
+- 我能否跳过加入浏览器path 到 env PATH 中呢?
+
+可以, 只不过在使用截图时, 需要指定浏览器path `--browser-path=you-path/to/chrome`
+
+
+- 能否在 EC2 或者 ECS 上用?
+
+可以, 只不过你要在机器中先装好 Chrome
+
+特别提示, AWS ec2 上没中文字体, 需要安装中文字体如果你截图到是中文网页的话.
+
+
+- 为啥截图网页的时候 网页底部有些图片没有出来?
+
+因为这个网页可能使用了懒加载图片的技术, 需要图片在浏览器视区内才进行加载, cap 默认 5次 pageDown 如果你想确保都加载, 请使用 --pagedown-to-bottom 参数
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/docapurl. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Docapurl project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/docapurl/blob/master/CODE_OF_CONDUCT.md).

data/Gemfile.lock

@@ -1,9 +1,9 @@
 PATH
   remote: .
   specs:
-    docapurl (0.1.0)
+    docapurl (0.2.1)
       ferrum (~> 0.11)
-      thor (~> 0.20)
+      thor
 
 GEM
   remote: https://rubygems.org/
@@ -19,7 +19,7 @@ GEM
       concurrent-ruby (~> 1.1)
       websocket-driver (>= 0.6, < 0.8)
     public_suffix (4.0.6)
-    rake (10.5.0)
+    rake (13.0.3)
     rspec (3.10.0)
       rspec-core (~> 3.10.0)
       rspec-expectations (~> 3.10.0)
@@ -33,7 +33,7 @@ GEM
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.10.0)
     rspec-support (3.10.2)
-    thor (0.20.3)
+    thor (1.1.0)
     websocket-driver (0.7.3)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.5)
@@ -44,7 +44,7 @@ PLATFORMS
 DEPENDENCIES
   bundler (~> 2.0)
   docapurl!
-  rake (~> 10.0)
+  rake (>= 12.3.3)
   rspec (~> 3.0)
 
 BUNDLED WITH

data/README.md

@@ -1,8 +1,14 @@
 # Docapurl
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/docapurl`. To experiment with that code, run `bin/console` for an interactive prompt.
+A tool to screenshot the webpage on terminal.
+
+[chinese_readme 中文说明](https://github.com/jicheng1014/docapurl/blob/master/CH_README.md)
+
+## Capture url as a Service
+
+[https://www.urlprint.com](https://www.urlprint.com) provides the capture url service, REST API supported.
+
 
-TODO: Delete this and the text above, and describe your gem
 
 ## Installation
 
@@ -19,10 +25,59 @@ And then execute:
 Or install it yourself as:
 
     $ gem install docapurl
+## Prerequisites
+
+Chrome browser is required.
+By the default, docapurl will invoke chrome in `PATH` ENV or ENV `BROWSER_PATH`
 
 ## Usage
 
-TODO: Write usage instructions here
+on terminal
+
+`docapurl cap [url] [image_path]`
+
+use `docapurl help cap` to know more details
+
+## Example
+
+```
+ docapurl cap https:/www.bilibili.com 1.jpg --pagedown-to-bottom
+```
+
+or if u wanna know more details on screenshot
+
+```
+ docapurl cap https:/www.bilibili.com 1.jpg --pagedown-to-bottom --no-headless
+```
+
+## FAQ
+
+-  Why docapurl needs chrome and where to download Chrome?
+
+Because docapurl just encapsulates functions from ferrum gem, and ferrum depends on headless Chrome.
+There's no official Chrome or Chromium package for Linux don't install it this way because it's either outdated or unofficial, both are bad. Download it from official https://www.chromium.org/getting-involved/download-chromium
+
+For mac and windows, u can download Chrome from https://www.google.com/chrome/.
+
+
+
+- Could i skip  adding browser path to ENV path?
+
+Yes, use param `--browser-path=you-path/to/chrome`
+
+
+- Could use it on aws EC2 or aliyun ECS servers?
+
+Yes, but u should installed chrome on serverw first.
+
+
+- Why some images in the page didn't show in the screenshot?
+
+Because the website may use lazy loading technology to the images, the images loaded when the images in browser viewscreen.
+docapurl invokes keyborard PageDown 5 times default. use param `--pagedown-to-bottom` could ensure all images load happen.
+
+
+
 
 ## Development
 

data/docapurl.gemspec

@@ -29,9 +29,9 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_development_dependency "bundler", "~> 2.0"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rake", ">= 12.3.3"
   spec.add_development_dependency "rspec", "~> 3.0"
   spec.add_dependency 'ferrum', '~>0.11'
-  spec.add_dependency "thor", "~> 0.20"
+  spec.add_dependency "thor"
 
 end

data/lib/docapurl.rb

@@ -1,9 +1,5 @@
 require_relative './docapurl/version'
 require_relative './docapurl/browser'
-begin
-  require 'byebug'
-rescue
-end
 require 'thor'
 
 module Docapurl

data/lib/docapurl/browser.rb

@@ -24,17 +24,25 @@ module Docapurl
       options[:path] ||= "screenshot-#{host.to_s == '' ? '' : "#{host}-"}#{Time.now.strftime('%F-%T')}.jpg"
       logger.info "browser begin to visit url #{url}"
 
+      set_callback("before_visit_func", options)
       browser.go_to(url)
+      set_callback("after_visit_func", options)
+
+
       logger.info 'visited'
       max_pagedown = options[:max_pagedown] || 5
       pagedown_to_bottom = options.delete :pagedown_to_bottom
       visit_whole_page(browser, max_pagedown: max_pagedown, pagedown_to_bottom: pagedown_to_bottom)
 
       sleep_before_screen = options.delete :sleep_before_screen
-      logger.info "sleep #{sleep_before_screen.to_i} second before screen"
+      logger.info "sleep #{sleep_before_screen.to_i} second before screenshot"
       sleep(sleep_before_screen.to_i)
 
+
+      set_callback("before_screenshot_func", options)
       browser.screenshot(**options)
+      set_callback("after_screenshot_func", options)
+
       logger.info "screenshot ended, path = #{options[:path]}"
     end
 
@@ -67,6 +75,18 @@ module Docapurl
       browser.keyboard.type(:Home)
     end
 
+    private
+
+    def set_callback(name, options)
+      the_function = options.delete name
+      if the_function.nil?
+        the_function = options.delete name.to_sym
+      end
+
+      the_function.call(self) unless the_function.nil?
+    end
+
+
     class << self
       def cap(url, path = nil, browser_options = {}, cap_options = {})
         browser = new(browser_options)

data/lib/docapurl/version.rb

@@ -1,3 +1,3 @@
 module Docapurl
-  VERSION = "0.1.0"
+  VERSION = "0.2.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: docapurl
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.1
 platform: ruby
 authors:
 - atpking
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-04-24 00:00:00.000000000 Z
+date: 2021-05-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -28,16 +28,16 @@ dependencies:
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: 12.3.3
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: 12.3.3
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -70,16 +70,16 @@ dependencies:
   name: thor
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '0.20'
+        version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '0.20'
+        version: '0'
 description:
 email:
 - atpking@gmail.com
@@ -91,6 +91,7 @@ files:
 - ".gitignore"
 - ".rspec"
 - ".travis.yml"
+- CH_README.md
 - CODE_OF_CONDUCT.md
 - Gemfile
 - Gemfile.lock