lighthouse 8.0.0-dev.20210709 → 8.0.0

package/docs/lantern.md

@@ -0,0 +1,55 @@
+# Lantern
+
+## Overview
+
+Project Lantern is an ongoing effort to reduce the run time of Lighthouse and improve audit quality by modeling page activity and simulating browser execution. This document details the accuracy of these models and captures the expected natural variability.
+
+## Deep Dive
+
+[![Lantern Deep Dive](https://img.youtube.com/vi/0dkry1r49xw/0.jpg)](https://www.youtube.com/watch?v=0dkry1r49xw)
+
+## Accuracy
+
+All of the following accuracy stats are reported on a set of 300 URLs sampled from the Alexa top 1000, HTTPArchive dataset, and miscellaneous ad landing pages. Median was collected for *9 runs* in one environment and compared to the median of *9 runs* in a second environment.
+
+Stats were collected using the [trace-evaluation](https://github.com/patrickhulce/lighthouse-trace-evaluations) scripts. Table cells contain [Spearman's rho](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) and [MAPE](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error) for the respective metric.
+
+### Lantern Accuracy Stats
+| Comparison | FCP | FMP | TTI |
+| -- | -- | -- | -- |
+| Lantern predicting Default LH | .811 : 23.1% | .811 : 23.6% | .869 : 42.5% |
+| Lantern predicting LH on WPT | .785 : 28.3% | .761 : 33.7% | .854 : 45.4% |
+
+### Reference Stats
+| Comparison | FCP | FMP | TTI |
+| -- | -- | -- | -- |
+| Unthrottled LH predicting Default LH | .738 : 27.1% | .694 : 33.8% | .743 : 62.0% |
+| Unthrottled LH predicting WPT | .691 : 33.8% | .635 : 33.7% | .712 : 66.4% |
+| Default LH predicting WPT | .855 : 22.3% | .813 : 27.0% | .889 : 32.3% |
+
+## Conclusions
+
+### Lantern Accuracy Conclusions
+We conclude that Lantern is ~6-13% more inaccurate than DevTools throttling. When evaluating rank performance, Lantern achieves correlations within ~.04-.07 of DevTools throttling.
+
+* For the single view use case, our original conclusion that Lantern's inaccuracy is roughly equal to the inaccuracy introduced by expected variance seems to hold. The standard deviation of single observations from DevTools throttling is ~9-13%, and given Lantern's much lower variance, single observations from Lantern are not significantly more inaccurate on average than single observations from DevTools throttling.
+* For the repeat view use case, we can conclude that Lantern is systematically off by ~6-13% more than DevTools throttling.
+
+### Metric Variability Conclusions
+The reference stats demonstrate that there is high degree of variability with the user-centric metrics and strengthens the position that every load is just an observation of a point drawn from a distribution and to understand the entire experience, multiple draws must be taken, i.e. multiple runs are needed to have sufficiently small error bounds on the median load experience.
+
+The current size of confidence intervals for DevTools throttled performance scores are as follows.
+
+* 95% confidence interval for **1-run** of site at median: 50 **+/- 15** = 65-35
+* 95% confidence interval for **3-runs** of site at median: 50 **+/- 11** = 61-39
+* 95% confidence interval for **5-runs** of site at median: 50 **+/- 8** = 58-42
+
+## Links
+
+* [Lighthouse Variability and Accuracy Analysis](https://docs.google.com/document/d/1BqtL-nG53rxWOI5RO0pItSRPowZVnYJ_gBEQCJ5EeUE/edit?usp=sharing)
+* [Lantern Deck](https://docs.google.com/presentation/d/1EsuNICCm6uhrR2PLNaI5hNkJ-q-8Mv592kwHmnf4c6U/edit?usp=sharing)
+* [Lantern Design Doc](https://docs.google.com/a/chromium.org/document/d/1pHEjtQjeycMoFOtheLfFjqzggY8VvNaIRfjC7IgNLq0/edit?usp=sharing)
+* [WPT Trace Data Set Half 1](https://drive.google.com/open?id=1Y_duiiJVljzIEaYWEmiTqKQFUBFWbKVZ) (access on request)
+* [WPT Trace Data Set Half 2](https://drive.google.com/open?id=1EoHk8nQaBv9aoaVv81TvR7UfXTUu2fiu) (access on request)
+* [Unthrottled Trace Data Set Half 1](https://drive.google.com/open?id=1axJf9R3FPpzxhR7FKOvXPLFLxxApfwD0) (access on request)
+* [Unthrottled Trace Data Set Half 2](https://drive.google.com/open?id=1krcWq5DF0oB1hq90G29bEwIP7zDcJrYY) (access on request)

package/docs/new-audits.md

@@ -0,0 +1,99 @@
+So, you want to create a new audit? Great! We're excited that you want to add to the Lighthouse project :) The goal of this 
+document is to help you understand what constitutes as a "good" audit for Lighthouse, and steps you can follow if you want
+to propose a new audit. 
+
+## New audit principles
+
+Lighthouse audits that surface in the report should:
+- be applicable to a significant portion of web developers (based on scale and severity of impact) 
+- contribute significantly towards making the mobile web experience better for end users. 
+- not have a significant impact on our runtime performance or bundle size. 
+- be new, not something that is already measured by existing audits. 
+- be measurable (especially for performance audits) or have clear pass/fail states.
+- be actionable - when failing, specific advice should be given. If the failure can be tied to a specific resource (a DOM element, script, line of code), use the appropriate detail type (see below). If multiple failures can occur for a page, return a table.
+- not use 3rd party APIs for completing the audit check. 
+
+## Actionability
+
+1. Specific advice should be given if the audit fails. If an audit can fail in multiple ways, each way should have  specific guidance that the user should take to resolve the problem.
+1. If the failure can be applied to a specific resource, use the appropriate detail type (see subsection).
+1. If multiple failures can occur on a single page, show each (use a table - don't just return a binary score).
+
+### Detail Types
+
+An audit can return a number of different [detail types](https://github.com/GoogleChrome/lighthouse/blob/master/types/audit-details.d.ts).
+
+| detail type               | resource              | notes                                  |
+|---------------------------|-----------------------|----------------------------------------|
+| `'node'`                  | DOM element           | set path to a devtoolsNodePath         |
+| `'source-location'`       | Code Network Resource | use to point to specific line, column  |
+| `'code'`                  | N/A; freeform         | render as monospace font `like this`   |
+| `'url'`                   | Network Resource      | we will make it a pretty link          |
+| `'thumbnail'`             | Image Resource        | same as above, but we show a thumbnail |
+| `'link'`                  | -                     | arbitrary link / url combination       |
+| `'text'\|'ms'\|'numeric'` | -                     |                                        |
+
+
+<!--- https://docs.google.com/document/d/1KS6PGPYDfE_TWrRdw55Rd67P-g_MU4KdMetT3cTPHjI/edit#heading=h.32w9jjm4c70w -->
+![Detail type examples](../assets/detail-type-examples.png)
+
+## Naming
+
+### Audit ID
+
+The audit ID should be based on the noun of the subject matter that it surfaces to the user.
+
+The filename should match the audit ID.
+
+**Policy**
+
+- No verbs.
+- No `no-` prefixes.
+- Use the noun of the items it surfaces or concept it centers around.
+- Adjective modifiers are acceptable and encouraged if the noun would be too broad without specificty.
+- If an adjective modifier will result in describing either the passing or failing state, prefer the failing state.
+
+**Examples**
+
+- ~~no-vulnerable-dependencies~~ vulnerable-dependencies (no `no-`)
+- ~~redirects-http~~ http-redirect (no verbs)
+- ~~uses-long-cache-ttl~~ cache-headers (no verbs)
+- ~~is-crawlable~~ crawlability (no verbs)
+- ~~images~~ oversized-images (too broad)
+- ~~used-css~~ unused-css (prefer failing state adjective)
+
+### Audit Title / Failure Title
+
+Audit titles vary based on report context and audit type.
+
+- Opportunities should have an *imperative* `title` describing the action the developer should take to fix the issue.
+- Standard audits should have both a *descriptive* `title` and a `failureTitle` that describe what the page is currently doing that resulted in a passing/failing state.
+
+Opportunity `title`: "Compress large images"
+Standard Audit `title`: "Page works offline"
+Standard Audit `failureTitle`: "Page does not work offline"
+
+
+## Process for creating a new audit
+
+1. Scan the criteria we’ve laid out above. If you think the principles match with your proposed new audit, then proceed! 
+1. Next step is to create an issue on GitHub with answers to the following questions: 
+```
+#### Provide a basic description of the audit
+#### How would the audit appear in the report? 
+<!-- How would the test look when passing? Would there be additional details available?
+     How would the test look when failing? What additional details are available? 
+     If the details are tabular, what are the columns?
+     If not obvious, how would passing/failing be defined? -->
+#### How is this audit different from existing ones?
+#### What % of developers/pages will this impact? 
+<!-- (Estimates OK, data points preferred) -->
+#### How is the new audit making a better web for end users?
+<!-- (Data points preferred) -->
+#### What is the resourcing situation? 
+<!-- Who will create the audits, write the documentation, and maintain both? -->
+#### Any other links or documentation that we should check out?
+```
+3. Once the proposal is submitted, then Lighthouse team will take a look and followup. We will discuss possible implementation approaches, and associated runtime overhead.
+With this new information we can better understand the impl cost and effort required and prioritize the audit into our sprint/roadmap. 
+1. Depending on the prioritization, we'll then work with you to figure out the necessary engineering/UX/product details. 

package/docs/performance-budgets.md

@@ -0,0 +1,187 @@
+# Performance Budgets (budget.json)
+
+Use a [performance budget](https://web.dev/performance-budgets-101/) to assert thresholds for performance metrics. `budget.json` supports three types of budgets:
+- Timing budgets: Assert thresholds for time-based performance metrics like First Contentful Paint, Maximum First Input Delay, and Speed Index.
+- Resource counts: Assert thresholds for the quantity of resources on a page. These thresholds can be defined per resource type or for the page overall.
+- Resource sizes: Assert thresholds for the transfer size of resources on a page. These thresholds can be defined per resource type or for the page overall.
+
+If performance budgets have been configured, the Lighthouse report will include a "Performance Budgets" section.
+
+To make performance budgets a part of your CI process, refer to [Lighthouse CI](https://github.com/GoogleChrome/lighthouse-ci).
+
+## Usage
+
+Performance budgets are supported in the CLI version of lighthouse. To use:
+1. Create a `budget.json` file.
+2. When running Lighthouse from the command line, pass the `--budget-path` flag followed by the path to budget file in order to calculate whenever a category is over budget.
+```
+lighthouse https://youtube.com --budget-path=budget.json
+```
+
+## budget.json
+The `budget.json` file is an array containing one or more `Budget` objects.
+
+```json
+[
+  {
+    "path": "/*",
+    "options": {
+       "firstPartyHostnames": ["*.my-site.com", "my-site.cdn.com"]
+    },
+    "timings": [
+      {
+        "metric": "interactive",
+        "budget": 5000
+      },
+      {
+        "metric": "first-meaningful-paint",
+        "budget": 2000
+      }
+    ],
+    "resourceSizes": [
+      {
+        "resourceType": "total",
+        "budget": 500
+      },
+      {
+        "resourceType": "script",
+        "budget": 150
+      }
+    ],
+    "resourceCounts": [
+      {
+        "resourceType": "third-party",
+        "budget": 100
+      }
+    ]
+  },
+  {
+    "options": {
+       "firstPartyHostnames": ["*.my-site.com", "my-site.cdn.com"]
+    },
+    "path": "/checkout",
+    "resourceSizes": [
+      {
+        "resourceType": "script",
+        "budget": 200
+      }
+    ]
+  }
+]
+``` 
+
+## Further Explanation
+
+### Timing Budgets
+
+_Lighthouse 6 & up_
+
+Use the optional `timings` property to define budgets for time-based performance metrics. In this context, budgets are defined in  milliseconds.
+
+```json
+"timings": [
+   {
+         "metric": "interactive",
+         "budget": 5000
+   }
+]
+```
+
+Supported timing metrics:
+
+- `first-contentful-paint`
+- `interactive`
+- `first-meaningful-paint`
+- `max-potential-fid`
+- `total-blocking-time`
+- `speed-index`
+- `largest-contentful-paint`
+- `cumulative-layout-shift`
+
+### Resource Budgets
+
+Use the optional `resourceSizes` property to define budgets for the *size* of page resources. In this context, budgets are defined in kibibytes (1 KiB = 1024 bytes).
+
+```json
+"resourceSizes": [
+   {
+      "resourceType": "script",
+      "budget": 300
+   }
+]
+```
+
+Use the optional `resourceCounts` property to define budgets for the *quantity* of page resources. In this context, budgets are defined in # of requests.
+
+```json
+"resourceCounts": [
+   {
+      "resourceType": "script",
+      "budget": 10
+   }
+]
+```
+
+Budgets can be set for the follow resource types.
+
+*   `document`
+*   `font`
+*   `image`
+*   `media`
+*   `other`
+*   `script`
+*   `stylesheet`
+*   `third-party`
+*   `total`
+
+### Using the `path` property
+
+_Lighthouse 5.3 & up_
+
+The `path` property indciates the pages that a budget applies to. This string should follow the [robots.txt](https://developers.google.com/search/reference/robots_txt#examples-of-valid-robotstxt-urls) format.
+
+If `path` is not supplied, a budget will apply to all pages.
+
+If a page's URL path matches the `path` property of more than one budget in `budget.json`, then the last matching budget will be applied. As a result, global budgets (e.g. `"path": "/*"`) should be listed first in `budget.json`, followed by the budgets that override the global budget (e.g. `"path": "/blog"`). 
+
+Examples:
+
+```
+"path": "/"
+```
+Result: Matches all URL paths. This is equivalent to writing `"path": "/*"`
+
+```
+"path": "/articles"
+```
+Result: Matches all URL paths starting with `/articles`. For example, `/articles` and `/articles/jun_02_2019.html`
+
+```
+"path": "/store/*/details$"
+```
+Result: Matches `/store/clothes/item123/details` but does not match `/store/details`.
+
+### Identification of third-party resources
+
+_Lighthouse 6 & up_
+
+`options.firstPartyHostnames` can be used to indicate which resources should be considered first-party. Wildcards can optionally be used to match a hostname and all of its subdomains.
+
+If this property is not set, the root domain and all its subdomains are considered first party.
+
+```json
+"options": {
+   "firstPartyHostnames": ["*.my-site.com", "my-site.cdn.com"]
+}
+```
+
+Examples:
+```
+"firstPartyHostnames": ["pets.com"]
+```
+Result: pets.com is considerated first-party, but fishes.pets.com is not.
+
+```
+"firstPartyHostnames": ["*.pets.com"]
+```
+Result: Both pets.com and fishes.pets.com are considered first party.