@alexbosworth/fiat 2.0.0 → 3.0.0

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Versions
 
+## 3.0.0
+
+- Add `chainRates` to make a `getChainRate` method that gets the USD price of
+    BTC in cents on the day of a date from chain data via LND, for dates from
+    October 20, 2021, pricing each day as the time weighted median of windows
+    of about two hours of its blocks and refusing a day whose price the data do
+    not support rather than guessing, answering the days up to the release from
+    prices shipped with the package and keeping derived prices in a versioned
+    prices file that records how every price was derived, along with the
+    provisional price of the current day, which is reused for about an hour,
+    removing a stored price for good once a reorganization replaces its blocks,
+    and keeping a stored price from blocks that LND has not reached yet, as
+    when it is catching up after being down
+- Add `date` to `getPrices` to get the rates at the hour of a date from
+    Coinbase Exchange, in USD, EUR or GBP, back to 2015 without an API key,
+    fetching the hourly prices of a day once for all of its dates when the same
+    request function is used
+
+### Breaking Changes
+
+- Require node.js 22 or higher
+
 ## 2.0.0
 
 - Require node.js 16 or higher

package/README.md

@@ -2,11 +2,46 @@
 
 Methods for working with fiat prices.
 
+`chainRates` derives prices from chain data alone; how and why is described in
+[docs/METHODOLOGY.md](docs/METHODOLOGY.md). The prices of the days up to a
+release ship with the package in `chain/known/known_prices.json`.
+
+## `chainRates`
+
+Make a method that gets historic USD prices of BTC from chain data
+
+    {
+      fs: {
+        getFile: <Read File Contents Function> (path, cbk) => {}
+        makeDirectory: <Make Directory Function> (path, cbk) => {}
+        renameFile: <Rename File Function> (from, to, cbk) => {}
+        writeFile: <Write File Contents Function> (path, contents, cbk) => {}
+      }
+      lnd: <Authenticated LND API Object>
+      [logger]: <Winston Logger Object>
+      path: <Prices File Path String>
+    }
+
+    @throws
+    <Error>
+
+    @returns
+    {
+      getChainRate: <Get Chain Rate Method> ({
+        currency: <Currency To Price String> // BTC
+        date: <Date To Price ISO 8601 Date String> // priced on its day in UTC
+        fiat: <Fiat Currency To Price In String> // USD
+      }, cbk) => (err, {
+        cents: <US Cents Per BTC on the Day of the Date Number>
+      })
+    }
+
 ## `getPrices`
 
 Get exchange rates from a rate provider
 
     {
+      [date]: <Rates At the Hour of ISO 8601 Date String>
       from: <From Rate Provider String>
       request: <Request Function> {url}, (err, {statusCode}, body) => {}
       symbols: [<Fiat Symbol String>] // empty defaults to USD
@@ -15,8 +50,65 @@ Get exchange rates from a rate provider
     @returns via cbk or Promise
     {
       tickers: [{
-        date: <Rate Updated At ISO 8601 Date String>
+        date: <Rate Updated At or Hour Priced At ISO 8601 Date String>
         rate: <Exchange Rate in Cents Number>
         ticker: <Ticker Symbol String>
       }]
     }
+
+## Scripts
+
+- `npm test` runs the quick tests and reports their coverage
+- `npm run test-all` also runs the slower tests in `test/slow`, which derive
+    prices from simulated chains, and reports the coverage of all the tests
+- `npm run check-prices` compares chain prices with the hourly prices of days
+    on Coinbase Exchange, with the options described under Validation in the
+    methodology. Without days it checks the latest three complete days, or as
+    many as `--days`
+- `npm run update-known-prices` brings the shipped prices up to date, as
+    described under Releasing
+- `npm publish` first runs `scripts/check_known_prices.js`, which stops the
+    publish when the shipped prices are not ready for a release
+
+For example, to compare two days priced without a neighboring price as a
+guide:
+
+    npm run check-prices -- 2024-08-05 2024-08-08 --unguided
+
+The scripts that read the chain reach Bitcoin Core at `http://127.0.0.1:8332`
+with the cookie in the default data directory, or as set by `BITCOIN_RPC_URL`,
+`BITCOIN_RPC_COOKIE`, or `BITCOIN_RPC_USER` and `BITCOIN_RPC_PASSWORD`.
+
+## Releasing
+
+A release normally ships the prices of the days up to it, so that lookups of
+those days do not have to derive them from the chain:
+
+1. With Bitcoin Core synced, run `npm run update-known-prices`. It derives
+    the prices of the last week of shipped days again, which have to come out
+    the same, and then adds a price for every complete day after them. Each
+    new price has to be within the range its day traded in on Coinbase
+    Exchange and within 5% of the median of its hourly prices there, and a day
+    that fails a check stops the run without changing the table
+2. Look over the summary at the end of the run, which shows how far the new
+    prices are from the median of the hourly prices of their days. A higher
+    median or 95th percentile, or a bias, means the prices agree with the
+    exchange less well than before, even though every day passed its checks
+3. Run `npm run test-all`, which includes the slow tests that `npm test`
+    leaves out
+4. Commit `chain/known/known_prices.json` with the release
+5. Run `npm publish`, which checks that the shipped prices are of the current
+    version of the method, start on the earliest supported day, have a price
+    for every day and reach to within a week of the present, and stops when
+    they don't
+
+When the method changes version, every day is derived again, which takes
+several hours, and the summary also shows how far the prices moved from the
+last table.
+
+To release without new prices, publish with `npm publish --ignore-scripts`,
+which skips the check. The table still has to be of the current version of
+the method, since prices of another version are not used, so first run
+`node scripts/check_known_prices.js` and see that the age of the prices is the
+only problem it reports. Lookups of the days after the last shipped price then
+derive them from the chain.

package/arrays/index.js

@@ -0,0 +1,3 @@
+const sortBy = require('./sort_by');
+
+module.exports = {sortBy};

package/arrays/sort_by.js

@@ -0,0 +1,44 @@
+const {isArray} = Array;
+
+const equalTo = 0;
+const greaterThan = 1;
+const lessThan = -1;
+
+/** Sort array by attribute, lowest to highest
+
+  {
+    array: [<Array Element Object>]
+    attribute: <Attribute String>
+  }
+
+  @throws
+  <Error>
+
+  @returns
+  {
+    sorted: [<Sorted Element Object>]
+  }
+*/
+module.exports = ({array, attribute}) => {
+  if (!isArray(array)) {
+    throw new Error('ExpectedArrayToSortByAttribute');
+  }
+
+  if (!attribute) {
+    throw new Error('ExpectedAttributeToSortArrayBy');
+  }
+
+  const sorted = array.slice().sort((a, b) => {
+    if (a[attribute] > b[attribute]) {
+      return greaterThan;
+    }
+
+    if (b[attribute] > a[attribute]) {
+      return lessThan;
+    }
+
+    return equalTo;
+  });
+
+  return {sorted};
+};

package/chain/add_days.js

@@ -0,0 +1,22 @@
+const {dayLength} = require('./../dates/constants');
+const {msPerDay} = require('./constants');
+
+const dayStart = day => Date.parse(`${day}T00:00:00.000Z`);
+
+/** Add days to a day
+
+  {
+    day: <Day YYYY-MM-DD String>
+    days: <Days to Add Number>
+  }
+
+  @returns
+  {
+    day: <Day YYYY-MM-DD String>
+  }
+*/
+module.exports = ({day, days}) => {
+  const date = new Date(dayStart(day) + days * msPerDay);
+
+  return {day: date.toISOString().substring(Number(), dayLength)};
+};

package/chain/blocks/block_header_hash.js

@@ -0,0 +1,29 @@
+const {createHash} = require('crypto');
+
+const bufferAsHex = buffer => buffer.toString('hex');
+const hexAsBuffer = hex => Buffer.from(hex, 'hex');
+const isHeaderHex = n => /^[0-9A-F]{160}$/i.test(n);
+const sha256 = buffer => createHash('sha256').update(buffer).digest();
+
+/** Get the hash of a block from its header
+
+  {
+    header: <Raw Block Header Hex String>
+  }
+
+  @throws
+  <Error>
+
+  @returns
+  {
+    hash: <Block Hash Hex String>
+  }
+*/
+module.exports = ({header}) => {
+  if (!isHeaderHex(header)) {
+    throw new Error('ExpectedRawBlockHeaderHexStringToGetBlockHeaderHash');
+  }
+
+  // Block ids are the double SHA256 of the header, shown byte-reversed
+  return {hash: bufferAsHex(sha256(sha256(hexAsBuffer(header))).reverse())};
+};

package/chain/blocks/block_header_time.js

@@ -0,0 +1,25 @@
+const hexAsBuffer = hex => Buffer.from(hex, 'hex');
+const isHeaderHex = n => /^([0-9A-F]{2}){80,}$/i.test(n);
+const timeOffset = 68;
+
+/** Get the time of a block from its header
+
+  {
+    header: <Raw Block Header or Block Hex String>
+  }
+
+  @throws
+  <Error>
+
+  @returns
+  {
+    time: <Block Header Timestamp Epoch Seconds Number>
+  }
+*/
+module.exports = ({header}) => {
+  if (!isHeaderHex(header)) {
+    throw new Error('ExpectedRawBlockHeaderHexStringToGetBlockHeaderTime');
+  }
+
+  return {time: hexAsBuffer(header).readUInt32LE(timeOffset)};
+};

package/chain/blocks/find_complete_day.js

@@ -0,0 +1,90 @@
+const asyncAuto = require('async/auto');
+const asyncDetectSeries = require('async/detectSeries');
+const {returnResult} = require('asyncjs-util');
+
+const addDays = require('./../add_days');
+const {dayLength} = require('./../../dates/constants');
+const getBlocksForDate = require('./get_blocks_for_date');
+const getMedianTime = require('./get_median_time');
+const {msPerSecond} = require('./../constants');
+
+const isHeight = n => Number.isInteger(n) && n >= 0;
+const maxDaysBack = 7;
+const yesterday = -1;
+
+/** Find the most recent complete day of blocks before the chain tip
+
+  {
+    height: <Current Best Chain Height Number>
+    lnd: <Authenticated LND API Object>
+    [times]: <Block Times By Height Map Object>
+  }
+
+  @returns via cbk or Promise
+  {
+    day: <Day YYYY-MM-DD String>
+  }
+*/
+module.exports = (args, cbk) => {
+  return new Promise((resolve, reject) => {
+    return asyncAuto({
+      // Check arguments
+      validate: cbk => {
+        if (!isHeight(args.height)) {
+          return cbk([400, 'ExpectedCurrentChainHeightToFindCompleteDay']);
+        }
+
+        if (!args.lnd) {
+          return cbk([400, 'ExpectedAuthenticatedLndToFindCompleteDay']);
+        }
+
+        return cbk();
+      },
+
+      // Block times are kept to avoid retrieving them twice
+      times: ['validate', ({}, cbk) => cbk(null, args.times || new Map())],
+
+      // Get the median time of the chain tip
+      getTipTime: ['times', ({times}, cbk) => {
+        return getMedianTime({
+          times,
+          height: args.height,
+          lnd: args.lnd,
+        },
+        cbk);
+      }],
+
+      // Walk back from the day before the tip until a complete day is found
+      findDay: ['getTipTime', 'times', ({getTipTime, times}, cbk) => {
+        const tipDate = new Date(getTipTime.time * msPerSecond).toISOString();
+
+        const tipDay = tipDate.substring(Number(), dayLength);
+
+        const days = [...Array(maxDaysBack).keys()].map(daysBack => {
+          return addDays({day: tipDay, days: yesterday - daysBack}).day;
+        });
+
+        return asyncDetectSeries(days, (date, cbk) => {
+          return getBlocksForDate({
+            date,
+            times,
+            height: args.height,
+            lnd: args.lnd,
+          },
+          (err, res) => cbk(err, !err && !!res.is_complete));
+        },
+        cbk);
+      }],
+
+      // The most recent complete day
+      day: ['findDay', ({findDay}, cbk) => {
+        if (!findDay) {
+          return cbk([503, 'FailedToFindCompleteDayBeforeChainTip']);
+        }
+
+        return cbk(null, {day: findDay});
+      }],
+    },
+    returnResult({reject, resolve, of: 'day'}, cbk));
+  });
+};

package/chain/blocks/find_height_for_time.js

@@ -0,0 +1,93 @@
+const asyncAuto = require('async/auto');
+const asyncTimesSeries = require('async/timesSeries');
+const {returnResult} = require('asyncjs-util');
+
+const getMedianTime = require('./get_median_time');
+
+const {ceil} = Math;
+const {floor} = Math;
+const isHeight = n => Number.isInteger(n) && n >= 0;
+const {log2} = Math;
+
+/** Find the heights where the median time past of blocks crosses a time
+
+  {
+    height: <Current Best Chain Height Number>
+    lnd: <Authenticated LND API Object>
+    low: <Lowest Height to Consider Number>
+    time: <Time Epoch Seconds Number>
+    [times]: <Block Times By Height Map Object>
+  }
+
+  @returns via cbk or Promise
+  {
+    at_or_after: <First Height With Median Time At Or After Time Number>
+    before: <Last Height With Median Time Before Time Number>
+  }
+*/
+module.exports = (args, cbk) => {
+  return new Promise((resolve, reject) => {
+    return asyncAuto({
+      // Check arguments
+      validate: cbk => {
+        if (!isHeight(args.height)) {
+          return cbk([400, 'ExpectedCurrentChainHeightToFindHeightForTime']);
+        }
+
+        if (!args.lnd) {
+          return cbk([400, 'ExpectedAuthenticatedLndToFindHeightForTime']);
+        }
+
+        if (!isHeight(args.low)) {
+          return cbk([400, 'ExpectedLowestHeightToFindHeightForTime']);
+        }
+
+        if (!isHeight(args.time)) {
+          return cbk([400, 'ExpectedTimeToFindHeightForTime']);
+        }
+
+        return cbk();
+      },
+
+      // Binary search for the first height with a median time at or after
+      search: ['validate', ({}, cbk) => {
+        const bounds = {high: args.height + 1, low: args.low};
+        const {lnd} = args;
+        const times = args.times || new Map();
+
+        // A binary search takes a bounded number of steps
+        const steps = ceil(log2(bounds.high - bounds.low + 1)) + 1;
+
+        return asyncTimesSeries(steps, (i, cbk) => {
+          // Exit early when the search has narrowed to a single height
+          if (bounds.low >= bounds.high) {
+            return cbk();
+          }
+
+          const mid = floor((bounds.low + bounds.high) / 2);
+
+          return getMedianTime({lnd, times, height: mid}, (err, res) => {
+            if (!!err) {
+              return cbk(err);
+            }
+
+            if (res.time >= args.time) {
+              bounds.high = mid;
+            } else {
+              bounds.low = mid + 1;
+            }
+
+            return cbk();
+          });
+        },
+        err => cbk(err, bounds.low));
+      }],
+
+      // Determine the crossing heights
+      crossing: ['search', ({search}, cbk) => {
+        return cbk(null, {at_or_after: search, before: search - 1});
+      }],
+    },
+    returnResult({reject, resolve, of: 'crossing'}, cbk));
+  });
+};

package/chain/blocks/get_block_hash.js

@@ -0,0 +1,60 @@
+const asyncAuto = require('async/auto');
+const {returnResult} = require('asyncjs-util');
+
+const blockHeaderHash = require('./block_header_hash');
+const getBlockHeader = require('./get_block_header');
+
+const isHeight = n => Number.isInteger(n) && n >= 0;
+
+/** Get the hash of the block at a height in the chain
+
+  {
+    height: <Block Height Number>
+    lnd: <Authenticated LND API Object>
+  }
+
+  @returns via cbk or Promise
+  {
+    hash: <Block Hash Hex String>
+  }
+*/
+module.exports = ({height, lnd}, cbk) => {
+  return new Promise((resolve, reject) => {
+    return asyncAuto({
+      // Check arguments
+      validate: cbk => {
+        if (!isHeight(height)) {
+          return cbk([400, 'ExpectedBlockHeightToGetBlockHash']);
+        }
+
+        if (!lnd) {
+          return cbk([400, 'ExpectedAuthenticatedLndToGetBlockHash']);
+        }
+
+        return cbk();
+      },
+
+      // Get the block header
+      getHeader: ['validate', ({}, cbk) => getBlockHeader({height, lnd}, cbk)],
+
+      // Hash the block header
+      getHash: ['getHeader', ({getHeader}, cbk) => {
+        const hashed = (() => {
+          try {
+            return blockHeaderHash({header: getHeader.header});
+          } catch (err) {
+            return {};
+          }
+        })();
+
+        // Exit early when the header cannot be hashed
+        if (!hashed.hash) {
+          return cbk([503, 'ExpectedBlockHeaderToGetBlockHash']);
+        }
+
+        return cbk(null, {hash: hashed.hash});
+      }],
+    },
+    returnResult({reject, resolve, of: 'getHash'}, cbk));
+  });
+};

package/chain/blocks/get_block_header.js

@@ -0,0 +1,45 @@
+const asyncAuto = require('async/auto');
+const {getBlockHeader} = require('ln-service');
+const {returnResult} = require('asyncjs-util');
+
+const isHeight = n => Number.isInteger(n) && n >= 0;
+
+/** Get the header of a block in the chain using the LND chain backend
+
+  {
+    height: <Block Height Number>
+    lnd: <Authenticated LND API Object>
+  }
+
+  @returns via cbk or Promise
+  {
+    header: <Raw Block Header Hex String>
+  }
+*/
+module.exports = ({height, lnd}, cbk) => {
+  return new Promise((resolve, reject) => {
+    return asyncAuto({
+      // Check arguments
+      validate: cbk => {
+        if (!isHeight(height)) {
+          return cbk([400, 'ExpectedBlockHeightToGetBlockHeader']);
+        }
+
+        if (!lnd) {
+          return cbk([400, 'ExpectedAuthenticatedLndToGetBlockHeader']);
+        }
+
+        return cbk();
+      },
+
+      // Get the block header
+      getHeader: ['validate', ({}, cbk) => getBlockHeader({height, lnd}, cbk)],
+
+      // The block header
+      header: ['getHeader', ({getHeader}, cbk) => {
+        return cbk(null, {header: getHeader.header});
+      }],
+    },
+    returnResult({reject, resolve, of: 'header'}, cbk));
+  });
+};

package/chain/blocks/get_block_payments.js

@@ -0,0 +1,100 @@
+const asyncAuto = require('async/auto');
+const {getBlock} = require('ln-service');
+const {returnResult} = require('asyncjs-util');
+
+const {blockHeaderHexLength} = require('./../constants');
+const blockHeaderTime = require('./block_header_time');
+const paymentOutputsFromBlock = require('./payment_outputs_from_block');
+
+const flatten = arr => [].concat(...arr);
+const headerFromBlock = block => block.slice(Number(), blockHeaderHexLength);
+const isHeight = n => Number.isInteger(n) && n >= 0;
+
+/** Get the payment outputs and the time of a block
+
+  {
+    [blocks]: <Retrieved Blocks Map Object>
+    height: <Block Height Number>
+    lnd: <Authenticated LND API Object>
+  }
+
+  @returns via cbk or Promise
+  {
+    outputs: [<Payment Output Tokens Number>]
+    time: <Block Header Time Number>
+    transactions: <Payment Transactions Count Number>
+  }
+*/
+module.exports = ({blocks, height, lnd}, cbk) => {
+  return new Promise((resolve, reject) => {
+    return asyncAuto({
+      // Check arguments
+      validate: cbk => {
+        if (!!blocks && !(blocks instanceof Map)) {
+          return cbk([400, 'ExpectedBlocksMapToGetBlockPayments']);
+        }
+
+        if (!isHeight(height)) {
+          return cbk([400, 'ExpectedBlockHeightToGetBlockPayments']);
+        }
+
+        if (!lnd) {
+          return cbk([400, 'ExpectedAuthenticatedLndToGetBlockPayments']);
+        }
+
+        return cbk();
+      },
+
+      // Get the block unless it was already retrieved
+      getBlock: ['validate', ({}, cbk) => {
+        // Exit early when the block was already retrieved
+        if (!!blocks && blocks.has(height)) {
+          return cbk();
+        }
+
+        return getBlock({height, lnd}, cbk);
+      }],
+
+      // Pull out the payment outputs and the time of the block
+      payments: ['getBlock', ({getBlock}, cbk) => {
+        // Exit early when the block was already retrieved
+        if (!getBlock) {
+          return cbk(null, blocks.get(height));
+        }
+
+        const parsed = (() => {
+          try {
+            const {transactions} = paymentOutputsFromBlock({
+              block: getBlock.block,
+            });
+
+            const header = headerFromBlock(getBlock.block);
+
+            return {
+              payments: {
+                outputs: flatten(transactions.map(tx => tx.outputs)),
+                time: blockHeaderTime({header}).time,
+                transactions: transactions.length,
+              },
+            };
+          } catch (err) {
+            return {err};
+          }
+        })();
+
+        // Exit early when the block cannot be parsed
+        if (!!parsed.err) {
+          return cbk([503, 'FailedToParseBlockToGetPrice', {err: parsed.err}]);
+        }
+
+        // Keep the block for other estimates from the same blocks
+        if (!!blocks) {
+          blocks.set(height, parsed.payments);
+        }
+
+        return cbk(null, parsed.payments);
+      }],
+    },
+    returnResult({reject, resolve, of: 'payments'}, cbk));
+  });
+};