@hackerhouse/xpub-scan 1.0.0

package/.claude/settings.local.json

@@ -0,0 +1,8 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(npm run build:*)",
+      "Bash(npm test:*)"
+    ]
+  }
+}

package/CONTRIBUTING.md

@@ -0,0 +1,130 @@
+# Welcome!
+
+Thank you for investing your time in contributing to Xpub Scan!
+
+In this guide, you will get an overview of its core principles as well as the contribution workflow.
+
+# Core Principles
+
+Xpub Scan is based on four core principles that are expected to be reflected in each incremental development.
+
+- It is product agnostic
+- It works out of the box
+- It is stateless
+- It is privacy-focused
+
+## 1. Product Agnostic
+
+One consequence of this principle is that Xpub Scan is required to work with Live and LES (Ledger Enterprise Solutions) products when using the comparison feature.
+
+To help enforce this principle, the regression tests are targeting both the Live (Live-common, Live Desktop) and the LES components.
+
+Rationale: Xpub Scan is used by both business-to-consumer and business-to-business teams.
+
+## 2. Works Out the Box
+
+Anyone should be able to just clone, build, and run Xpub Scan. It should just work without additional configuration.
+
+Advanced users should also be able to customize the tool to use robust external providers.
+
+As a consequence, Xpub Scan uses two categories of providers: free providers (referred to by the `DEFAULT_API_URLS` variable in `src/configuration/settings.ts`) and a paid providers (CryptoAPIs: `CRYPTOAPIS_URL`).
+
+In the code, the free providers are labeled as _default_ providers while CryptoAPIs is labeled as the _custom_ provider.
+
+The free providers are not as reliable as the paid provider. Consequently, Xpub Scan has to display a warning message when the free provider is being used to inform the user about such limitations. The current warning message is: “_(only the last ~50 operations by address are displayed)_". This is a hard requirement.
+
+Rationale: Xpub Scan is used outside Ledger and should not depend on one provider.
+
+## 3. Stateless
+
+Xpub Scan is not a wallet: there is no need to maintain an internal state.
+
+Using a database would complexify the tool, notably by implementing a mechanism ensuring the correct synchronization of such a database.
+
+## 4. Privacy-Focused
+
+Xpub Scan should _never_ send xpubs to external providers and only fetch the minimum information (transactions per address that it derives itself).
+
+# Setting up the development environment
+
+Use Node **LTS**. The most recent versions of Node may trigger [OpenSSL-related errors](https://github.com/webpack/webpack/issues/14532)).
+
+At the root of the project, a `.env` file, containing a valid [CryptoAPIs API v2 key](https://developers.cryptoapis.io/technical-documentation/general-information/overview), is required to enable the custom provider.
+
+```
+XPUB_SCAN_CUSTOM_API_KEY_V2=<your key>
+```
+
+# Implementing
+
+## Properly Format the Code
+
+Your code has to be correctly formatted. To this effect, a set of rules are defined in `.prettierrc.json` and automatically enforced with this command:
+
+```
+$ npm run prettier
+```
+
+## Add Unit Tests
+
+When implementing a new feature, please add corresponding unit tests.
+
+You can also contribute to the project by enhancing the current code coverage.
+
+The general idea is to progressively reach near 100% coverage.
+
+# Testing
+
+Once your fix or feature has been implemented, you can locally ensure that the project will pass the tests in the CI.
+
+(Note: you can run all tests at once: `$ npm run dev:test:all`).
+
+## Up-to-date Requirements
+
+For security reasons, the dependencies have to be up-to-date and strictly pinpointed.
+
+To this effect, a dedicated command verifies the dependencies:
+
+```
+$ npm run check:dep
+```
+
+If this command reveals that dependencies are outdated, update them in `package.json` **using strict versioning**:
+
+- `1.2.3` ✅
+- `^1.2.3` ❌
+- `~1.2.3` ❌
+
+Run `check:dep` again and, if the check passes, run the subsequent tests to ensure that the newest dependencies do not break anything.
+
+## Unit Tests
+
+To run the unit tests, do:
+
+```
+$ npm run ci
+```
+
+To generate the coverage (prerequisites: `jest` and `ts-node`, run:
+
+```
+$ jest --collect-coverage
+```
+
+An HTML report will be generated in `./coverage/lcov-report/`.
+
+## Regression Tests
+
+Regression tests are essential to ensure that the core logic of Xpub Scan is not negatively affected by new features and/or fixes. They consist of scanning ad hoc xpubs and performing comparisons using products exports (see: `.github/workflows/regression_tests/datasets.json`). Both positive and negative tests are running: positive tests ensure that Xpub Scan does not detect erroneous discrepancies; negative tests ensure that Xpub Scan can detect fake discrepancies, injected for testing purposes into the exports of the products.
+
+To run the regression tests locally, use the following command:
+
+```
+$ npm run regression
+```
+
+(Important: the tests will fail if no CryptoAPIs API v2 key is provided).
+
+# Any contributions you make will be under the MIT Software License
+
+In short, when you submit code changes, your submissions are understood to be under the same [MIT License](https://github.com/LedgerHQ/xpub-scan/blob/main/LICENSE) that covers the project.

package/LICENSE

@@ -0,0 +1,23 @@
+MIT License
+
+Copyright (c) 2021—2022 Ledger
+
+(Creator and main contributor: Guillaume Lethuillier)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,215 @@
+# Xpub Scan
+
+![XPUB](./doc/logo.png)
+
+Blockchain test oracle and forensics tool.
+
+Given an extended public key (xpub, Ltub, _etc._), get the balances of its derived active addresses and the associated list of operations, or check whether an address has been derived from it.
+
+![Xpub Scan demo](./doc/demo.gif)
+
+## Features
+
+- Privacy Friendly: master public keys are never sent over the Internet: only their derived addresses are
+- Derives specific addresses (by account+index) or all active ones
+- Searches if a given address has been derived from a given master public key (perfect and partial match)
+- Supports legacy, SegWit, and Native SegWit
+- Automatically checks supported CSV or JSON files containing operations history
+
+## Prerequisites
+
+- Node.js **LTS** (note: using Node v17 may trigger [this OpenSSL-related error](https://github.com/webpack/webpack/issues/14532))
+- Docker (if you want to use the docker image)
+
+## Install
+
+```
+$ npm i -g @ledgerhq/xpub-scan
+```
+
+## Usage
+
+### Bitcoin, Litecoin
+
+```
+$ xpub-scan <xpub> [options]
+```
+
+### Ethereum
+
+```
+$ xpub-scan <address> --currency eth [options]
+```
+
+Running programmatically example:
+
+```
+import { Scanner } from "../src/actions/scanner";
+
+const scanResult = async (xpub: string) => {
+  await new Scanner({ itemToScan: xpub }).scan();
+};
+
+scanResult("your_xpub");
+```
+
+## Development
+
+Install dependencies:
+
+```
+$ npm run ci
+```
+
+Build the project
+
+```
+$ npm run build
+```
+
+Run the built version:
+
+```
+$ node ./lib/scan.js [options] <xpub>
+```
+
+_Alternatively, if you want to use the locally-built version of xpub-scan with just `xpub-scan` command, you can:_
+
+```
+$ npm link
+```
+
+### Currencies
+
+By default, Bitcoin xpubs and Litecoin ltubs are automatically detected.
+
+To scan Bitcoin Cash xpubs, use the `--currency bch` argument:
+
+`$ xpub-scan --currency bch <xpub> …`
+
+To scan Litecoin xpubs, use the `--currency ltc` argument:
+
+`$ xpub-scan --currency ltc <xpub> …`
+
+## Usage 1. Check Balances
+
+_In the following instructions, the generic `xpub` term is used to designate a master public key. It can be substituted with another type of supported public key, such as `Ltub` (Litecoin)._
+
+### Scan for a Specific Account and an Index
+
+`$ xpub-scan <xpub> --account <account> --index <index>`
+
+Example:
+`$ xpub-scan xpub6C...44dXs7p -a 0 -i 10` [addresses at account `0`, index `10`]
+
+### ScanLimits Scan | Scan Addresses Derived From Index A to Index B
+
+`$ xpub-scan <xpub> --account <account> --from-index <index A> [--to-index <index B>]`
+
+Example 1:
+`$ xpub-scan xpub6C...44dXs7p -a 0 --from-index 0 --to-index 100` [addresses at account `0`, from index `0` to index `100`, included]
+
+Example 2:
+`$ xpub-scan xpub6C...44dXs7p -a 1 --from-index 29` [addresses at account `0`, from index `29` until no active address]
+
+Note: in order to perform the analysis in this context, Xpub Scan needs to pre-derive addresses. By default, it derives 2,000 addresses per account number. If needed, this number of addresses can be adjusted using the `--pre-derivation-size` option (useful with xpub associated with a large number of addresses).
+
+### Full Scan | Scan All Active Addresses
+
+`$ xpub-scan <xpub>`
+
+Example:
+`$ xpub-scan xpub6C...44dXs7p`
+
+### Compare Imported Data With Actual Data
+
+**Balance `--balance <balance>`**
+
+`$ xpub-scan <xpub> --balance <balance (in base unit such as satoshis)>`
+
+**Addresses `--addresses <filepath>`**
+
+_(Not implemented yet)_
+
+`$ xpub-scan <xpub> --addresses <file path>`
+
+**UTXOs `--utxos <filepath>`**
+
+_(Not implemented yet)_
+
+`$ xpub-scan <xpub> --utxos <file path>`
+
+**Operations `--operations <filepath>`**
+
+`$ xpub-scan <xpub> --operations <file path>`
+
+Example:
+`$ xpub-scan xpub6C...44dXs7p --operations /Users/Test/Downloads/export.csv`
+
+**General Example**
+
+`$ xpub-scan xpub6C...44dXs7p --operations /Users/Test/Downloads/export.csv --balance 12345 --diff` displays at the end of the analysis the results of the comparison between the `12345` satoshis balance and the actual one, as well as the potential mismatches between the imported operations and the actual ones.
+
+### Generate JSON and HTML Reports (Scan Only)
+
+`$ xpub-scan <xpub> [args...] --save <directory>`
+
+The files are saved as `<xpub>.json` and `<xpub>.html`.
+
+Note: `--save stdout` can be used to display the JSON instead of saving the files. Furthermore, the `--quiet` option does not display the analysis progress while the `--silent` option does not display the progress _nor the results_.
+
+### Comparisons-Related Option
+
+- `--diff` displays the mismatches (if any) between the imported and actual operations.
+
+## Usage 2. Check Address Against Xpub
+
+_Check if an address has been derived from a master public key._
+
+### Perfect Match
+
+`$ xpub-scan <xpub> --address <address>`
+
+### Partial Match
+
+Add `?` where there is uncertainty about a character in the address. For instance: `1MYaYeZhDp?m3YtvqBMXQQN??YCz?7NqgF`
+
+## Docker
+
+Build: `$ docker build -t xpubscan .`
+
+Run: `$ docker run xpubscan <xpub> [optional: <args>]`
+
+## Interface
+
+### Check Balance
+
+When an analysis is performed, 3 elements are displayed in the following order:
+
+- The analysis of each derived active address (type, path, address, current balance, total in `←`, total out `→`)
+- The transactions ordered by date (date, block number, address, in `←` | out `→` | sent to self `⮂` | sent to sibling `↺` | received as change from non-sibling `c`)
+- A summary: total number of transactions and total balance by address type
+
+### Xpub and Address Comparison
+
+The derived addresses are displayed during the analysis. Perfect matches are displayed in green (with the corresponding derivation path). Partial matches are displayed in blue (also with the derivation path). No matches are rendered in red.
+
+## Configure
+
+### Change Settings
+
+1. Modify `./src/settings.ts`
+2. rebuild the tool: `$ npm run build`
+3. Re-run it: `$ node ./lib/scan.js <xpub> …`
+
+### Change External Provider
+
+**It is strongly encouraged to use the custom provider for reliable results.**
+
+[Crypto APIs](https://cryptoapis.io/) can be used as a custom provider. In this context, a valid [v2 API key](https://dashboard.cryptoapis.io/account/api-keys) is required.
+
+1. At the root of the project, rename `.env.template` to `.env`
+2. In `.env`, set the `XPUB_SCAN_CUSTOM_API_KEY_V2` (corresponding to your Crypto APIs v2 API key—e.g.: `XPUB_SCAN_CUSTOM_API_KEY_V2=abcd6eacca264f7530eb2f7025a84f8`)
+3. rebuild the tool: `$ npm run build`
+4. Re-run it: `$ node ./lib/scan.js <xpub> …`
+5. Ensure that, when running the tool, it shows that the _custom_ provider is being used

package/__tests__/checkAddresses/checkBitcoinAddressesNegative.test.ts

@@ -0,0 +1,75 @@
+import { _private } from "../../src/actions/checkAddress";
+import { DERIVATION_SCOPE } from "../../src/configuration/settings";
+
+const testSearchFunction = (xpub: string, providedAddress: string) => {
+  return _private.search(
+    xpub,
+    providedAddress,
+    DERIVATION_SCOPE.quick_search,
+    "quick search",
+  );
+};
+
+describe("check whether addresses belong or not to xpubs", () => {
+  /* eslint-disable */
+  console.log = function () {};
+
+  const xpub =
+    "xpub6C9vKwUFiBLbQKS6mhEAtEYhS24sVz8MkvMjxQSECTZVCnFmy675zojLthvXVuQf15RT6ggmt7PTgLBV2tLHHdJenoEkNWe5VPBETncxf2q";
+
+  describe("addresses NOT belonging to the xpub", () => {
+    it("complete legacy address", () => {
+      const providedAddress = "1C3DK2BwhPRZ7e14V7pHj1jDWbTrP3qFnA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial legacy address", () => {
+      const providedAddress = "1C3DK2BwhPRZ7e?4V7pHj1jDWbTrP3qF?A";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("complete SegWit address", () => {
+      const providedAddress = "3Bb1aYVQXiwowamm5kZDweK5KxVnzNKABA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial SegWit address", () => {
+      const providedAddress = "3Bb1aY??Xiwowamm5kZDweK5KxVnzNKA?A";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("complete native SegWit address", () => {
+      const providedAddress = "bc1qxrkzds606tun4qghc8e6c5z4vuk3c0692cy0fA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial native SegWit address", () => {
+      const providedAddress = "bc1qxrkzds???tun4qghc8e6c5z4vuk3c0692cy0fA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+  });
+});

package/__tests__/checkAddresses/checkBitcoinAddressesPositive.test.ts

@@ -0,0 +1,87 @@
+import { _private } from "../../src/actions/checkAddress";
+import { DERIVATION_SCOPE } from "../../src/configuration/settings";
+
+const testSearchFunction = (xpub: string, providedAddress: string) => {
+  return _private.search(
+    xpub,
+    providedAddress,
+    DERIVATION_SCOPE.quick_search,
+    "quick search",
+  );
+};
+
+describe("check whether addresses belong or not to xpubs", () => {
+  /* eslint-disable */
+  console.log = function () {};
+
+  const xpub =
+    "xpub6C9vKwUFiBLbQKS6mhEAtEYhS24sVz8MkvMjxQSECTZVCnFmy675zojLthvXVuQf15RT6ggmt7PTgLBV2tLHHdJenoEkNWe5VPBETncxf2q";
+
+  describe("addresses belonging to the xpub", () => {
+    it("complete legacy address — m/2/588", () => {
+      const providedAddress = "1C3DK2BwhPRZ7e14V7pHj1jDWbTrP3qFnt";
+      const expectedResult = { account: 2, index: 588 };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial legacy address — m/2/588", () => {
+      const providedAddress = "1C3DK?BwhPRZ7e14V7?Hj1jDWbTrP3?Fnt";
+      const expectedResult = {
+        account: 2,
+        index: 588,
+        partial: "1C3DK2BwhPRZ7e14V7pHj1jDWbTrP3qFnt",
+      };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("complete SegWit address — m/3/999", () => {
+      const providedAddress = "3Bb1aYVQXiwowamm5kZDweK5KxVnzNKABp";
+      const expectedResult = { account: 3, index: 999 };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial SegWit address — m/3/999", () => {
+      const providedAddress = "3Bb1aYVQ??wowamm5kZDweK5KxVnzNKA??";
+      const expectedResult = {
+        account: 3,
+        index: 999,
+        partial: "3Bb1aYVQXiwowamm5kZDweK5KxVnzNKABp",
+      };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("complete native SegWit address — m/2/278", () => {
+      const providedAddress = "bc1qxrkzds606tun4qghc8e6c5z4vuk3c0692cy0fl";
+      const expectedResult = { account: 2, index: 278 };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial native SegWit address — m/2/278", () => {
+      const providedAddress = "bc1qxrkzds?06tun4qghc8e6c5z4vuk3c0692cy0??";
+      const expectedResult = {
+        account: 2,
+        index: 278,
+        partial: "bc1qxrkzds606tun4qghc8e6c5z4vuk3c0692cy0fl",
+      };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+  });
+});

package/__tests__/checkAddresses/checkDogeAddressesPositive.test.ts

@@ -0,0 +1,43 @@
+import { _private } from "../../src/actions/checkAddress";
+import { DERIVATION_SCOPE } from "../../src/configuration/settings";
+
+const testSearchFunction = (xpub: string, providedAddress: string) => {
+  return _private.search(
+    xpub,
+    providedAddress,
+    DERIVATION_SCOPE.quick_search,
+    "quick search",
+  );
+};
+
+describe("check whether addresses belong or not to xpubs", () => {
+  /* eslint-disable */
+  console.log = function () {};
+
+  const xpub =
+    "dgub8rDzyFqzw35B8zSqrW9sHDivXL2YmcGEVKyRMhPh8SoeUmkGozD5YzNyEkSH1T9zhvy9iQzns7igxrhQkg4jETWna4X1AUC4MT3YgHenTMB";
+
+  describe("addresses belonging to the xpub", () => {
+    it("complete address — m/1/640", () => {
+      const providedAddress = "DSZsASgFRbGVhRnPE9bmeyd8s9izatDRXL";
+      const expectedResult = { account: 1, index: 640 };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial address — m/1/640", () => {
+      const providedAddress = "DSZsASgFR??VhRnPE9bmeyd8s9izatDR?L";
+      const expectedResult = {
+        account: 1,
+        index: 640,
+        partial: "DSZsASgFRbGVhRnPE9bmeyd8s9izatDRXL",
+      };
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+  });
+});

package/__tests__/checkAddresses/checkLitecoinAddressesNegative.test.ts

@@ -0,0 +1,75 @@
+import { _private } from "../../src/actions/checkAddress";
+import { DERIVATION_SCOPE } from "../../src/configuration/settings";
+
+const testSearchFunction = (xpub: string, providedAddress: string) => {
+  return _private.search(
+    xpub,
+    providedAddress,
+    DERIVATION_SCOPE.quick_search,
+    "quick search",
+  );
+};
+
+describe("check whether addresses belong or not to xpubs", () => {
+  /* eslint-disable */
+  console.log = function () {};
+
+  const xpub =
+    "Ltub2ZoLFXBt7mLeb4out9duUxMmqCSX9mqi73NcQ8nyjQsJEm76JP1poExdj9rRCFfHyuUVKgj5t2B2EBsmFvKivKzVpmKKZ4XiAVx65s3WZ8j";
+
+  describe("addresses NOT belonging to the xpub", () => {
+    it("complete legacy address", () => {
+      const providedAddress = "LXSUUS8Phpm4zGsdfA7pFsDN17yyK8N2KA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial legacy address", () => {
+      const providedAddress = "LXSUUS8Ph?m4zGsdfA7pFsDN17yyK8N2?A";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("complete SegWit address", () => {
+      const providedAddress = "MNY33b2fP4zLQ7W9oBGgw9TfFAm2TxWPAA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial SegWit address", () => {
+      const providedAddress = "MNY33b2fP4z?Q7W9oBGgw9TfFAm2T??PAA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("complete native SegWit address", () => {
+      const providedAddress = "bc1qxrkzds606tun4qghc8e6c5z4vuk3c0692cy0AA";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+
+    it("partial native SegWit address", () => {
+      const providedAddress = "bc1qxrkzds606tun?qghc8e6c5z4vuk3c0692cy0A?";
+      const expectedResult = {};
+
+      const match = testSearchFunction(xpub, providedAddress);
+
+      expect(match).toEqual(expectedResult);
+    });
+  });
+});