html-get 2.15.0-0 → 2.15.0

package/README.md

@@ -9,26 +9,16 @@
 [![Coverage Status](https://img.shields.io/coveralls/microlinkhq/html-get.svg?style=flat-square)](https://coveralls.io/github/microlinkhq/html-get)
 [![NPM Status](https://img.shields.io/npm/dm/html-get.svg?style=flat-square)](https://www.npmjs.org/package/html-get)
 
-> Get the HTML from any website, using prerendering when is necessary.
+> Get the HTML from any website, fine-tuned for correction & speed.
 
 ## Features
 
-- Get HTML markup from any website (client side apps as well).
-- Prerendering detection based on domains list.
-- Speed up process blocking ads trackers.
-- Encoding body response properly.
+- Get HTML markup for any URL, including images, video, audio, or pdf.
+- Block ads tracker or any non-necessary network subrequest.
+- Handle unreachable or timeout URLs gracefully.
+- Ensure HTML markup is appropriately encoded.
 
-<br>
-
-Headless technology like [puppeteer](https://github.com/GoogleChrome/puppeteer) brings us to get the HTML markup from any website, even when the target URL is client side app and we need to wait until dom events fire for getting the real markup.
-
-Generally this approach better than a simple GET request from the target URL, but because you need to wait for dom events, prerendering could be slow and in some scenario unnecessary (sites that use server side rendering could be resolved with a simple GET).
-
-**html-get** bring the best of both worlds, doing the following algorithm:
-
-- Determinate if the target URL actually needs prerendering (internally it has a [list of popular site domains](https://github.com/microlinkhq/html-get/blob/master/src/auto-domains.js) that don't need it).
-- If it needs prerendering, perform the action using Headless technology, blocking ads trackers requests for speed up the process, trying to resolve the main request in the minimum amount of time.
-- If it does not need prerendering or prerendering fails for any reason (for example, timeout), the request will be resolved doing a GET request.
+**html-get** takes advantage of [puppeteer](https://github.com/GoogleChrome/puppeteer) headless technology when is needed, such as client-side apps that needs to be prerender.
 
 ## Install
 
@@ -89,68 +79,70 @@ Type: `string`
 
 The target URL for getting the HTML markup.
 
+#### options
+
+##### encoding
+
+Type: `string`
+Default: `'utf-8'`
+
+It ensures the HTML markup is encoded to the encoded value provided.
+
+The value will be passes to [`html-encode`](https://github.com/kikobeats/html-encode) 
+
 ##### getBrowserless
 
 *Required*<br>
-Type: `function`<br>
+Type: `function`
 
 A function that should return a [browserless](https://browserless.js.org/) instance to be used for interact with puppeteer:
 
-#### options
-
-##### prerender
+##### getMode
 
-Type: `boolean`|`string`<br>
-Default: `'auto'`
+Type: `function`
 
-Enable or disable prerendering as mechanism for getting the HTML markup explicitly.
+It determines the strategy to use based on the `url`, being the possibles values `'fetch'` or `'prerender'` .
 
-The value `auto` means that that internally use a list of websites that don't need to use prerendering by default. This list is used for speedup the process, using `fetch` mode for these websites.
+##### getTemporalFile
 
-See [getMode parameter](#getMode) for know more.
+Type: `function`
 
-##### encoding
+It creates a temporal file.
 
-Type: `string`<br>
-Default: `'utf-8'`
+##### gotOpts
 
-Encoding the HTML markup properly from the body response.
+Type: `object`
 
-It determines the encode to use A Node.js library for converting HTML documents of arbitrary encoding into a target encoding (utf8, utf16, etc).
+It passes configuration object to [got](https://www.npmjs.com/package/got) under `'fetch'` strategy.
 
 ##### headers
 
-Type: `object`<br>
+Type: `object`
 
 Request headers that will be passed to fetch/prerender process.
 
-##### getMode
+##### mutoolPath
 
-Type: `function`<br>
+Type: `function`
 
-A function evaluation that will be invoked to determinate the resolutive `mode` for getting the HTML markup from the target URL.
+It returns the path for [mutool](https://mupdf.com/) binary, used for turning PDF files into HTML markup.
 
-The default `getMode` is:
+##### prerender
 
-```js
-const getMode = (url, { prerender }) => {
-  if (prerender === false) return 'fetch'
-  if (prerender !== 'auto') return 'prerender'
-  return autoDomains.includes(getDomain(url)) ? 'fetch' : 'prerender'
-}
-```
+Type: `boolean`|`string`<br>
+Default: `'auto'`
 
-##### gotOptions
+Enable or disable prerendering as mechanism for getting the HTML markup explicitly.
 
-Type: `object`<br>
+The value `auto` means that that internally use a list of websites that don't need to use prerendering by default. This list is used for speedup the process, using `fetch` mode for these websites.
 
-Under `mode=fetch`, pass configuration object to [got](https://www.npmjs.com/package/got).
+See [getMode parameter](#getMode) for know more.
 
 ##### puppeteerOpts
 
 Type: `object`
 
-Under non `mode=fetch`, pass configuration object to [puppeteer](https://www.npmjs.com/package/puppeteer).
+It passes coniguration object to [puppeteer](https://www.npmjs.com/package/puppeteer) under `'prerender'` strategy.
 
 ##### rewriteUrls
 

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "html-get",
-  "description": "Get the HTML from any website, using prerendering when is necessary.",
+  "description": "Get the HTML from any website, fine-tuned for correction & speed",
   "homepage": "https://nicedoc.com/microlinkhq/html-get",
-  "version": "2.15.0-0",
+  "version": "2.15.0",
   "main": "src/index.js",
   "bin": {
     "html-get": "bin/index.js"
@@ -20,20 +20,25 @@
     "url": "https://github.com/microlinkhq/html-get/issues"
   },
   "keywords": [
+    "audio",
     "fetch",
     "get",
     "got",
     "headless",
     "html",
+    "image",
+    "markup",
+    "pdf",
     "prerender",
-    "request"
+    "request",
+    "video"
   ],
   "dependencies": {
     "@kikobeats/time-span": "~1.0.3",
-    "@metascraper/helpers": "~5.43.0",
+    "@metascraper/helpers": "~5.43.4",
     "cheerio": "~1.0.0-rc.12",
     "css-url-regex": "~4.0.0",
-    "debug-logfmt": "~1.2.0",
+    "debug-logfmt": "~1.2.2",
     "execall": "~2.0.0",
     "got": "~11.8.6",
     "html-encode": "~2.1.6",
@@ -45,7 +50,7 @@
     "p-retry": "~4.6.0",
     "replace-string": "~3.1.0",
     "tinyspawn": "~1.2.6",
-    "top-sites": "~1.1.202"
+    "top-sites": "~1.1.205"
   },
   "devDependencies": {
     "@commitlint/cli": "latest",
@@ -77,6 +82,21 @@
     "scripts",
     "src"
   ],
+  "scripts": {
+    "clean": "rm -rf node_modules",
+    "contributors": "(npx git-authors-cli && npx finepack && git add package.json && git commit -m 'build: contributors' --no-verify) || true",
+    "lint": "standard-markdown README.md && standard",
+    "postinstall": "node scripts/postinstall",
+    "postrelease": "npm run release:tags && npm run release:github && (ci-publish || npm publish --access=public)",
+    "prerelease": "npm run update:check",
+    "pretest": "npm run lint",
+    "release": "standard-version -a",
+    "release:github": "github-generate-release",
+    "release:tags": "git push --follow-tags origin HEAD:master",
+    "test": "c8 ava",
+    "update": "ncu -u",
+    "update:check": "ncu -- --error-level 2"
+  },
   "license": "MIT",
   "ava": {
     "files": [
@@ -106,20 +126,5 @@
   "simple-git-hooks": {
     "commit-msg": "npx commitlint --edit",
     "pre-commit": "npx nano-staged"
-  },
-  "scripts": {
-    "clean": "rm -rf node_modules",
-    "contributors": "(npx git-authors-cli && npx finepack && git add package.json && git commit -m 'build: contributors' --no-verify) || true",
-    "lint": "standard-markdown README.md && standard",
-    "postinstall": "node scripts/postinstall",
-    "postrelease": "npm run release:tags && npm run release:github && (ci-publish || npm publish --access=public)",
-    "prerelease": "npm run update:check && npm run contributors",
-    "pretest": "npm run lint",
-    "release": "standard-version -a",
-    "release:github": "github-generate-release",
-    "release:tags": "git push --follow-tags origin HEAD:master",
-    "test": "c8 ava",
-    "update": "ncu -u",
-    "update:check": "ncu -- --error-level 2"
   }
-}
+}

package/src/auto-domains.json

@@ -1 +1 @@
-[[["domainWithoutSuffix","google"]],[["domainWithoutSuffix","youtube"]],[["domainWithoutSuffix","microsoft"]],[["domainWithoutSuffix","apple"]],[["domainWithoutSuffix","wikipedia"]],[["domainWithoutSuffix","wordpress"]],[["domainWithoutSuffix","blogspot"]],[["domainWithoutSuffix","vimeo"]],[["domainWithoutSuffix","github"]],[["domainWithoutSuffix","bbc"]],[["domainWithoutSuffix","slideshare"]],[["domainWithoutSuffix","theguardian"]],[["domainWithoutSuffix","imdb"]],[["domainWithoutSuffix","nytimes"]],[["domainWithoutSuffix","spotify"]],[["domainWithoutSuffix","huffingtonpost"]],[["domainWithoutSuffix","soundcloud"]],[["domainWithoutSuffix","twitter"]],[["domainWithoutSuffix","telegraph"]],[["domainWithoutSuffix","pinterest"]],[["domainWithoutSuffix","yelp"]],[["domainWithoutSuffix","eventbrite"]],[["domainWithoutSuffix","techcrunch"]],[["domainWithoutSuffix","zoom"]],[["domainWithoutSuffix","engadget"]],[["domainWithoutSuffix","stackoverflow"]],[["domain","abc.net.au"]],[["domainWithoutSuffix","theverge"]],[["domainWithoutSuffix","flickr"]],[["domainWithoutSuffix","giphy"]],[["domainWithoutSuffix","digg"]],[["domainWithoutSuffix","substack"]],[["domainWithoutSuffix","csdn"]],[["domainWithoutSuffix","deviantart"]],[["domainWithoutSuffix","dribbble"]],[["domainWithoutSuffix","etsy"]],[["domainWithoutSuffix","ghost"]],[["domainWithoutSuffix","gitlab"]],[["domainWithoutSuffix","imgur"]],[["domainWithoutSuffix","meetup"]],[["domainWithoutSuffix","producthunt"]],[["domainWithoutSuffix","sourceforge"]],[["domainWithoutSuffix","tumblr"]],[["domainWithoutSuffix","ycombinator"]]]
+[[["domainWithoutSuffix","google"]],[["domainWithoutSuffix","youtube"]],[["domainWithoutSuffix","microsoft"]],[["domainWithoutSuffix","apple"]],[["domainWithoutSuffix","wikipedia"]],[["domainWithoutSuffix","wordpress"]],[["domainWithoutSuffix","blogspot"]],[["domainWithoutSuffix","github"]],[["domainWithoutSuffix","vimeo"]],[["domainWithoutSuffix","theguardian"]],[["domainWithoutSuffix","imdb"]],[["domainWithoutSuffix","bbc"]],[["domainWithoutSuffix","slideshare"]],[["domainWithoutSuffix","nytimes"]],[["domainWithoutSuffix","spotify"]],[["domainWithoutSuffix","twitter"]],[["domainWithoutSuffix","soundcloud"]],[["domainWithoutSuffix","telegraph"]],[["domainWithoutSuffix","pinterest"]],[["domainWithoutSuffix","huffingtonpost"]],[["domainWithoutSuffix","yelp"]],[["domainWithoutSuffix","techcrunch"]],[["domainWithoutSuffix","zoom"]],[["domainWithoutSuffix","stackoverflow"]],[["domain","abc.net.au"]],[["domainWithoutSuffix","eventbrite"]],[["domainWithoutSuffix","engadget"]],[["domainWithoutSuffix","theverge"]],[["domainWithoutSuffix","substack"]],[["domainWithoutSuffix","giphy"]],[["domainWithoutSuffix","imgur"]],[["domainWithoutSuffix","csdn"]],[["domainWithoutSuffix","deviantart"]],[["domainWithoutSuffix","digg"]],[["domainWithoutSuffix","dribbble"]],[["domainWithoutSuffix","etsy"]],[["domainWithoutSuffix","flickr"]],[["domainWithoutSuffix","ghost"]],[["domainWithoutSuffix","gitlab"]],[["domainWithoutSuffix","meetup"]],[["domainWithoutSuffix","producthunt"]],[["domainWithoutSuffix","sourceforge"]],[["domainWithoutSuffix","tumblr"]],[["domainWithoutSuffix","ycombinator"]]]

package/src/index.js

@@ -24,7 +24,14 @@ const ABORT_TYPES = ['image', 'stylesheet', 'font']
 const fetch = PCancelable.fn(
   async (
     url,
-    { reflect = false, toEncode, timeout = REQ_TIMEOUT, mutoolPath, getTemporalFile, ...opts },
+    {
+      reflect = false,
+      toEncode,
+      timeout = REQ_TIMEOUT,
+      mutoolPath,
+      getTemporalFile,
+      ...opts
+    },
     onCancel
   ) => {
     const reqTimeout = reflect ? timeout / 2 : timeout
@@ -53,8 +60,7 @@ const fetch = PCancelable.fn(
           return (await $(`mutool draw -q -F html ${file.path}`)).stdout
         }
 
-        return contentType.startsWith('text/html') ||
-          !isMediaUrl(url)
+        return contentType.startsWith('text/html') || !isMediaUrl(url)
           ? await toEncode(res.body, res.headers['content-type'])
           : res.body
       })()
@@ -175,17 +181,34 @@ const defaultGetMode = (url, { prerender }) => {
 
 const defaultGetTemporalFile = (url, ext) => {
   const hash = crypto.createHash('sha256').update(url).digest('hex')
-  const filepath = path.join(os.tmpdir(), ext === undefined ? hash : `${hash}.${ext}`)
+  const filepath = path.join(
+    os.tmpdir(),
+    ext === undefined ? hash : `${hash}.${ext}`
+  )
   return { path: filepath }
 }
 
-const defaultMutoolPath = () => (() => { try { return execSync('which mutool').toString().trim() } catch (_) {} })()
+const defaultMutoolPath = () =>
+  (() => {
+    try {
+      return execSync('which mutool').toString().trim()
+    } catch (_) {}
+  })()
 
 const getContent = PCancelable.fn(
   (
     url,
     mode,
-    { getBrowserless, gotOpts, headers, puppeteerOpts, rewriteUrls, toEncode, mutoolPath, getTemporalFile },
+    {
+      getBrowserless,
+      getTemporalFile,
+      gotOpts,
+      headers,
+      mutoolPath,
+      puppeteerOpts,
+      rewriteUrls,
+      toEncode
+    },
     onCancel
   ) => {
     const isFetchMode = mode === 'fetch'