universal-parcel-scraper 0.2.0 → 0.3.0-main.298

package/README.md

@@ -1,124 +1,262 @@
 <div align="center">
 
+<img src="docs/assets/logo.svg" width="88" alt="">
+
 # Universal Parcel Scraper
 
-**A parcel. One timeline. Your infrastructure.**
+**Parcel tracking that asks the carrier directly, from your own machine.**
 
+[![npm](https://img.shields.io/npm/v/universal-parcel-scraper)](https://www.npmjs.com/package/universal-parcel-scraper)
 [![CI](https://github.com/plhery/universal-parcel-scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/plhery/universal-parcel-scraper/actions/workflows/ci.yml)
 [![License: Apache-2.0](https://img.shields.io/badge/core-Apache--2.0-blue.svg)](LICENSE)
 
-Detect a carrier, fetch its tracking history, and get consistent stages and scan times.
-A TypeScript library, a CLI, and a small HTTP server. No tracking-service account required.
-
 <!-- GENERATED:summary -->
 **105 carriers · 85 active dedicated adapters · 58 countries represented**
 <!-- /GENERATED:summary -->
 
+[Try it](#try-it) · [Ways to run it](#ways-to-run-it) · [Coverage](#how-much-can-it-track) · [Carriers](carriers/) · [Add a carrier](CONTRIBUTING.md)
+
+<img src="docs/assets/demo.svg" width="840" alt="The track command printing a parcel's history as JSON, next to a timeline drawn from it">
+
 </div>
 
-DHL, UPS, FedEx, USPS, Swiss Post, DPD, PostNL, and many more.
-[Browse the carriers](carriers/) · [How it works](ARCHITECTURE.md) · [Add a carrier](CONTRIBUTING.md)
+Give it a tracking number. It works out which carrier the number belongs to and fetches the
+history from that carrier's own website. What comes back has one shape, so a UPS parcel and
+a Poczta Polska parcel look the same to your code.
+
+There is no account to open and no API key. Every carrier publishes tracking in its own way,
+and hosted tracking APIs smooth that over for a fee while seeing every number you look up.
+Here the same work is open code: one adapter per carrier, each in [its own folder](carriers/)
+with a README on how that site is read.
+
+## Try it
+
+You need Node.js 24 or newer.
+
+```sh
+npx universal-parcel-scraper detect 1Z999AA10123456784
+```
+
+Detection is offline. It checks the number against each carrier's formats and checksums and
+answers `ups` without sending anything anywhere.
+
+```sh
+npx universal-parcel-scraper track YOUR_TRACKING_NUMBER
+```
+
+Tracking does go out, to the carrier that owns the number. If a number fits several carriers,
+name one with `--carrier dhl`. A few carriers also want the delivery postcode (`--postcode`)
+or the link from the shipping email (`--tracking-url`).
 
-## Get started
+## Ways to run it
 
-Node.js 24 or newer:
+### Command line
 
 ```sh
-npm install universal-parcel-scraper
-npx parcel-scraper detect 1Z999AA10123456784
-npx parcel-scraper track YOUR_TRACKING_NUMBER --carrier ups
+npm install -g universal-parcel-scraper
 ```
 
+The command is `parcel-scraper`:
+
+```sh
+parcel-scraper detect <number, link or text>   # which carrier is this? no network
+parcel-scraper track <number>                  # the parcel's history
+parcel-scraper recognize <number>              # ask the possible carriers which one knows it
+parcel-scraper carriers                        # the catalog, with the ids --carrier accepts
+parcel-scraper serve                           # the HTTP API described below
+```
+
+Everything prints JSON, so it pipes:
+
+```sh
+parcel-scraper track "$NUMBER" | jq -r '.result.current_stage'
+```
+
+`parcel-scraper --help` lists the environment variables.
+
+### Node
+
 ```js
 import { createTracker } from 'universal-parcel-scraper/node';
 
 const tracker = createTracker();
-const { carrier, source, result } = await tracker.track({
-  number: process.env.PARCEL_NUMBER,
-  carrier: 'ups',
-});
-console.log(carrier, source, result.current_stage, result.events);
+const { carrier, source, result } = await tracker.track({ number: process.env.PARCEL_NUMBER });
+
+console.log(carrier, result.current_stage);
+for (const scan of result.events) {
+  console.log(scan.time, scan.location, scan.description, scan.stage);
+}
 ```
 
-Detection also works in the browser, with no network calls:
+Each scan keeps the carrier's wording and gains a `stage` from [one shared list](data/stages.json).
+`source` names who answered, and `attempts` lists everything that was tried. When nothing
+returns a history, `track()` throws a `TrackingError` carrying those attempts and a hint about
+the cause. [examples/node.mjs](examples/node.mjs) is a runnable version.
+
+### Browser
 
 ```js
 import { parseTrackingInput } from 'universal-parcel-scraper';
-const match = parseTrackingInput('1Z999AA10123456784'); // synthetic example
+
+parseTrackingInput('1Z999AA10123456784');
+// { carrier: 'ups', confidence: 'high', candidates: ['ups'], … }
 ```
 
-Direct HTTP works out of the box. Some carriers need Chromium, image processing, or the
-optional [TRAWL browser service](trawl/README.md). Install the transports you need:
+The root import has no Node imports and makes no requests, so it can sit behind a form field
+and name the carrier as someone types.
+
+### HTTP
+
+For anything that is not JavaScript:
 
 ```sh
-npm install playwright-core sharp onnxruntime-web
+parcel-scraper serve --port 8080
 ```
 
-Pass `chromiumPath` or `trawlUrl` to `createTracker()`. Commercial aggregators are opt-in:
-`createTracker({ providers: ['ParcelsApp', 'Ship24', '17TRACK', 'Postal Ninja', 'UPU'] })`.
-The default is direct adapters plus UPU for eligible postal numbers.
+```sh
+curl http://127.0.0.1:8080/v1/track \
+  -H 'Content-Type: application/json' \
+  -d '{"number":"1Z999AA10123456784","carrier":"ups"}'
+```
 
-## How much can it track?
+`/v1/detect`, `/v1/recognize` and `/v1/carriers` sit next to it, and the
+[OpenAPI file](server/openapi.json) describes them all. The server caches recent answers in
+memory and rate-limits callers. It has no database and never polls on its own.
 
-Carrier count tells you what the catalog knows. Tracking history tells you what a source
-actually returned. Here is the recorded comparison on our **100-carrier reference set**:
+Set `SCRAPER_TOKEN` to require a bearer token. `SCRAPER_DEMO_PAGE=true` serves a small page
+at `/` for trying numbers by hand. Behind a reverse proxy, set `SCRAPER_TRUSTED_PROXIES` so
+the rate limit counts each caller and not the proxy. The other settings are in
+[.env.example](.env.example).
 
-<!-- GENERATED:coverage -->
-| Source | Carriers with history |
-| --- | ---: |
-| **Universal Parcel Scraper, all fallbacks enabled** | **70 / 100** |
-| Dedicated adapters alone | 51 / 100 |
-| ParcelsApp | 50 / 100 |
-| Postal Ninja | 41 / 100 |
-| 17TRACK | 40 / 100 |
-| Ship24 | 38 / 100 |
-| UPU | 11 / 100 |
-<!-- /GENERATED:coverage -->
+### Docker
 
-Generated from [coverage.json](providers/coverage.json). These are recorded outcomes for
-the comparison reference of each carrier, including partial histories; alternate samples
-are excluded. This curated set is not a global market-share ranking or a live availability
-promise. Old parcels expire, formats differ, and browser challenges change.
-[Full results and limitations](providers/COVERAGE.md).
+```sh
+docker run --rm -p 127.0.0.1:8080:8080 ghcr.io/plhery/universal-parcel-scraper:0.3.0
+```
+
+The image runs `serve` and ships Chromium, for the carriers that only answer a real browser.
+
+## What you can build with it
+
+- **A tracking screen in your own app.** [Delivery Tracker](https://github.com/plhery/delivery-tracker),
+  an open-source iPhone app and PWA, runs on this package.
+- **A Home Assistant sensor** for the parcel you are waiting on.
+  [The config is in examples](examples/home-assistant.yaml).
+- **Order status inside a shop or help desk**, so customers are not sent off to the carrier's
+  site. Any backend that speaks HTTP can call the server.
+- **Tracking numbers pulled out of shipping emails.** `detect` takes pasted text and carrier
+  links as well as bare numbers.
+- **A carrier field that fills itself in**, with the browser import.
+- **A cron job** that pings you when `current_stage` turns `delivered`.
+
+The library answers one lookup at a time. Storing parcels and deciding when to check again
+are yours to do.
 
-Hosted APIs advertise much larger carrier catalogs and handle hosting for you. This project
-gives you the retrieval code, control over which sources receive a number, and no per-lookup
-subscription. Catalog totals from vendors are not comparable to this reference test.
+## How it works
 
-## HTTP, Docker, and Home Assistant
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/how-it-works-dark.svg">
+  <img src="docs/assets/how-it-works-light.svg" width="840" alt="An input is detected offline and tracked by the carrier's dedicated adapter. With no history, the lookup moves to the fallbacks you enabled. Either way the result is one timeline.">
+</picture>
+
+Each adapter uses plain HTTP wherever the carrier's site allows it. Some sites only answer a
+real browser and others need image processing, so those adapters can drive a local Chromium
+or the optional [TRAWL browser service](trawl/README.md). Install what your carriers need:
 
 ```sh
-npx parcel-scraper serve --port 8080
-curl http://127.0.0.1:8080/v1/track \
-  -H 'Content-Type: application/json' \
-  -d '{"number":"1Z999AA10123456784","carrier":"ups"}'
+npm install playwright-core sharp onnxruntime-web
 ```
 
-The server shares cached answers, spaces universal-provider calls, and limits requests.
-It stores no parcel list and runs no background polling. Set `SCRAPER_TOKEN` for bearer
-authentication, `SCRAPER_PROVIDERS` to select fallbacks, and `SCRAPER_DEMO_PAGE=true` for a
-small one-off tracking page. [HTTP contract](server/openapi.json) · [Configuration](.env.example).
+Then pass `chromiumPath` or `trawlUrl` to `createTracker()`. The CLI and the server read
+`TRACKING_CHROMIUM_PATH` and `FLARESOLVERR_URL`.
 
-The versioned container includes Chromium and the optional transports:
+Where a carrier accepts a plain client, the adapter names itself with the package's default
+User-Agent. Pass `userAgent`, or set `SCRAPER_USER_AGENT`, to send your own.
 
-```sh
-docker run --rm -p 127.0.0.1:8080:8080 ghcr.io/plhery/universal-parcel-scraper:0.2.0
+When the adapter finds no history, the lookup can move on to an aggregator. Only UPU is on
+by default, for the postal numbers it can serve. The commercial ones are opt-in, and each
+one you enable receives the tracking number:
+
+```js
+createTracker({ providers: ['ParcelsApp', 'Ship24', '17TRACK', 'Postal Ninja', 'UPU'] });
 ```
 
-A [Home Assistant REST sensor](examples/home-assistant.yaml) can call the same API.
-[Node example](examples/node.mjs) · [Development](CONTRIBUTING.md).
+[ARCHITECTURE.md](ARCHITECTURE.md) has the rest.
+
+## How much can it track?
+
+A carrier count says what the catalog knows about. Whether a real parcel comes back with its
+history is a different question, so each source was run on its own against the same
+**100-carrier reference set**. The top bar counts a carrier when any source below it returned
+a history.
 
-## Data and limits
+<!-- GENERATED:coverage -->
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/coverage-dark.svg">
+  <img src="docs/assets/coverage-light.svg" alt="Carriers with tracking history: This project, all fallbacks enabled 70, This project, dedicated adapters alone 51, ParcelsApp 50, Postal Ninja 41, 17TRACK 40, Ship24 38, UPU 11." width="760">
+</picture>
 
-Lookups send the tracking number and any required postcode or capability URL to the selected
-carrier. Enabled commercial fallbacks receive the number; ParcelsApp also receives a supplied
-postcode. A configured TRAWL instance handles the requested pages. Browser pages can load the
-carrier's or provider's challenge resources. The library has no telemetry destination;
-HTTP logs contain route names and outcomes, never parcel inputs.
+<details>
+<summary>The same numbers as a table</summary>
 
-Consumers own polling and persistence. Respect the catalog's refresh limits and the upstream
-retry advice. Carrier sites can change or refuse automated requests; some need a postcode,
-a full tracking link, or an account and remain limited. Unresolved local clocks stay unresolved.
+| Source | Carriers with history |
+| --- | ---: |
+| **Universal Parcel Scraper, all fallbacks enabled** | **70 / 100** |
+| Dedicated adapters alone | 51 / 100 |
+| ParcelsApp | 50 / 100 |
+| Postal Ninja | 41 / 100 |
+| 17TRACK | 40 / 100 |
+| Ship24 | 38 / 100 |
+| UPU | 11 / 100 |
+
+</details>
+<!-- /GENERATED:coverage -->
 
-The core is [Apache-2.0](LICENSE). The optional TRAWL derivative is [AGPL-3.0](trawl/LICENSE).
-Data and model credits are in [NOTICE](NOTICE).
+The numbers come from [coverage.json](providers/coverage.json): the recorded outcome for each
+carrier's comparison reference, partial histories included and alternate samples left out.
+The set is curated. Read it as a comparison between sources, not as a market-share ranking
+or a promise about today, because parcels expire and carrier sites change.
+[Carrier-by-carrier results and limitations](providers/COVERAGE.md).
+
+## Next to a hosted tracking API
+
+| | Universal Parcel Scraper | Hosted tracking API |
+| --- | --- | --- |
+| Account | None | Sign-up and an API key |
+| Cost | Your own compute | A plan or a per-shipment price |
+| Who sees the tracking number | The carrier, plus any fallback you enable | The vendor, then the carrier |
+| Carriers | The [catalog](carriers/) in this repository | A much larger catalog |
+| Updates | You poll | Webhooks |
+| When a carrier changes its site | The adapter breaks until it is fixed here | The vendor deals with it |
+| Hosting | Yours | Theirs |
+
+If you want the widest catalog and nothing to run, a hosted API is the better choice. This
+project is for when the numbers should stay on your side, or when a price per parcel makes
+no sense for what you are building. Vendor catalog totals are not comparable with the
+reference test above.
+
+## Privacy and limits
+
+A lookup sends the tracking number to the carrier, along with the postcode or tracking link
+when that carrier requires one. Fallbacks you enabled receive the number too, and ParcelsApp
+also gets a postcode if you supplied it. Pages opened in a browser, locally or through TRAWL,
+can load the carrier's or provider's challenge scripts. The library has no telemetry, and
+the server's logs hold route names and outcomes, never parcel inputs.
+
+This is scraping, so things break. Carriers redesign their sites and turn away automated
+requests. A few need more than the number, such as a postcode or the link from the shipping
+email, and the ones that require an account stay limited. Keep to the catalog's refresh
+limits and to the retry advice that comes with an error. A scan time with no known UTC offset
+is returned as the carrier wrote it, with no zone guessed.
+
+## Contributing
+
+Adapters break when carriers change their sites. Each carrier lives in its own folder with
+synthetic fixtures and offline tests, so a fix stays local to that folder. For a new carrier,
+`npm run carrier:new` scaffolds the folder and [CONTRIBUTING.md](CONTRIBUTING.md) covers the
+rest. Report security issues through [SECURITY.md](SECURITY.md).
+
+## License
+
+The core is [Apache-2.0](LICENSE). The optional TRAWL image is a derivative under
+[AGPL-3.0](trawl/LICENSE). Data and model credits are in [NOTICE](NOTICE).

package/data/carrier.schema.json

@@ -205,7 +205,7 @@
           "minLength": 1
         },
         "canaryUrl": {
-          "description": "Public, credential-free HTTPS URL the daily canary probes. Required for automatic carriers.",
+          "description": "Public, credential-free HTTPS URL the weekly canary probes. Required for automatic carriers.",
           "type": "string"
         },
         "shows": {
@@ -273,9 +273,11 @@
       ],
       "properties": {
         "field": {
-          "description": "Input name; the generator checks it against the validators it accepts.",
-          "type": "string",
-          "minLength": 1
+          "description": "Input name. The generator checks that the validator is one this input accepts.",
+          "enum": [
+            "trackingUrl",
+            "postcode"
+          ]
         },
         "validator": {
           "type": "string",

package/data/catalog.json

@@ -713,7 +713,7 @@
       "adapter": "gls-ch",
       "requirements": [
         {
-          "field": "dpdPostcode",
+          "field": "postcode",
           "validator": "swissPostcode",
           "label": "Delivery postcode",
           "type": "text",
@@ -780,7 +780,7 @@
       "adapter": "dpd",
       "requirements": [
         {
-          "field": "dpdPostcode",
+          "field": "postcode",
           "validator": "swissPostcode",
           "optional": true,
           "label": "Delivery postcode",
@@ -879,7 +879,7 @@
       "adapter": "mondial-relay",
       "requirements": [
         {
-          "field": "dpdPostcode",
+          "field": "postcode",
           "whenTrackingNumber": "^(?![0-9]{26}$).*$",
           "validator": "francePostcode",
           "label": "Delivery postcode",
@@ -1268,7 +1268,7 @@
       "adapter": "heppner",
       "requirements": [
         {
-          "field": "dpdPostcode",
+          "field": "postcode",
           "validator": "swissOrFrancePostcode",
           "label": "Delivery postcode",
           "type": "text",
@@ -1350,7 +1350,7 @@
       "adapter": "paack",
       "requirements": [
         {
-          "field": "dpdPostcode",
+          "field": "postcode",
           "validator": "paackPostcode",
           "label": "Delivery postcode",
           "type": "text",
@@ -1545,7 +1545,7 @@
       "adapter": "gls-de",
       "requirements": [
         {
-          "field": "dpdPostcode",
+          "field": "postcode",
           "validator": "swissOrFrancePostcode",
           "label": "Delivery postcode",
           "type": "text",

package/dist/app.d.ts

@@ -0,0 +1,11 @@
+/**
+ * Helpers shaped for the parcel app this package was extracted from: its
+ * parcel view, carrier-name hints for provider results, and the clock helpers
+ * its sync uses. They change with that app and carry no stability promise;
+ * the tracking contract is the other entry points.
+ */
+export * from './core/catalog/parcel.js';
+export * from './core/catalog/hints.js';
+export * from './core/time/result.js';
+export { universalCarrierHints } from './providers/shared/hints.js';
+export { recognitionAskedCarriers } from './core/catalog/recognition.js';

package/dist/app.js

@@ -0,0 +1,11 @@
+/**
+ * Helpers shaped for the parcel app this package was extracted from: its
+ * parcel view, carrier-name hints for provider results, and the clock helpers
+ * its sync uses. They change with that app and carry no stability promise;
+ * the tracking contract is the other entry points.
+ */
+export * from './core/catalog/parcel.js';
+export * from './core/catalog/hints.js';
+export * from './core/time/result.js';
+export { universalCarrierHints } from './providers/shared/hints.js';
+export { recognitionAskedCarriers } from './core/catalog/recognition.js';