@alexbosworth/fiat 2.0.0 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +93 -1
- package/arrays/index.js +3 -0
- package/arrays/sort_by.js +44 -0
- package/chain/add_days.js +22 -0
- package/chain/blocks/block_header_hash.js +29 -0
- package/chain/blocks/block_header_time.js +25 -0
- package/chain/blocks/find_complete_day.js +90 -0
- package/chain/blocks/find_height_for_time.js +93 -0
- package/chain/blocks/get_block_hash.js +60 -0
- package/chain/blocks/get_block_header.js +45 -0
- package/chain/blocks/get_block_payments.js +100 -0
- package/chain/blocks/get_blocks_for_date.js +110 -0
- package/chain/blocks/get_median_time.js +92 -0
- package/chain/blocks/index.js +19 -0
- package/chain/blocks/payment_outputs_from_block.js +178 -0
- package/chain/blocks/sample_heights.js +33 -0
- package/chain/blocks/window_heights.js +34 -0
- package/chain/chain_rates.js +129 -0
- package/chain/constants.json +26 -0
- package/chain/estimates/check_known_estimates.js +119 -0
- package/chain/estimates/current_estimates.js +66 -0
- package/chain/estimates/estimate_and_remember.js +199 -0
- package/chain/estimates/estimate_day_price.js +361 -0
- package/chain/estimates/estimate_record.js +107 -0
- package/chain/estimates/estimate_windows.js +246 -0
- package/chain/estimates/find_starting_day.js +86 -0
- package/chain/estimates/forget_estimates.js +113 -0
- package/chain/estimates/get_chain_price.js +167 -0
- package/chain/estimates/get_price_for_date.js +607 -0
- package/chain/estimates/get_price_from_blocks.js +281 -0
- package/chain/estimates/get_provisional_price.js +163 -0
- package/chain/estimates/index.js +25 -0
- package/chain/estimates/is_alias_ratio.js +38 -0
- package/chain/estimates/is_supported_estimate.js +81 -0
- package/chain/estimates/nearest_estimate.js +49 -0
- package/chain/estimates/price_window.js +25 -0
- package/chain/estimates/remember_estimate.js +83 -0
- package/chain/estimates/store_estimate.js +198 -0
- package/chain/histogram/best_position.js +63 -0
- package/chain/histogram/histogram_excess.js +50 -0
- package/chain/histogram/histogram_peak.js +87 -0
- package/chain/histogram/implied_prices_histogram.js +53 -0
- package/chain/histogram/index.js +33 -0
- package/chain/histogram/is_round_tokens.js +26 -0
- package/chain/histogram/kernel_width.js +25 -0
- package/chain/histogram/lower_bound.js +28 -0
- package/chain/histogram/noise_of_neighbors.js +28 -0
- package/chain/histogram/noise_strength.js +35 -0
- package/chain/histogram/noise_strengths.js +29 -0
- package/chain/histogram/outputs_histogram.js +80 -0
- package/chain/histogram/parabola_peak.js +28 -0
- package/chain/histogram/payments_near_position.js +31 -0
- package/chain/histogram/peak_width.js +37 -0
- package/chain/histogram/position_score.js +38 -0
- package/chain/histogram/score_position.js +94 -0
- package/chain/histogram/smooth_bins.js +29 -0
- package/chain/histogram/sorted_neighbors.js +89 -0
- package/chain/histogram/spike_strengths.js +33 -0
- package/chain/histogram/template_agreement.js +27 -0
- package/chain/index.js +3 -0
- package/chain/known/earliest_supported_day.js +12 -0
- package/chain/known/index.js +11 -0
- package/chain/known/known_prices.js +43 -0
- package/chain/known/known_prices.json +1836 -0
- package/chain/known/method_version.js +26 -0
- package/chain/known/parse_known_prices.js +67 -0
- package/chain/outputs/alias_shifts.js +31 -0
- package/chain/outputs/cap_near_round_values.js +83 -0
- package/chain/outputs/cap_repeated_values.js +33 -0
- package/chain/outputs/dollar_bin_for_price.js +19 -0
- package/chain/outputs/estimate_from_outputs.js +321 -0
- package/chain/outputs/index.js +11 -0
- package/chain/outputs/peak_around_price.js +86 -0
- package/chain/outputs/price_for_dollar_bin.js +21 -0
- package/chain/outputs/price_from_outputs.js +98 -0
- package/chain/outputs/refine_price.js +73 -0
- package/chain/outputs/round_amounts.js +49 -0
- package/chain/outputs/template.json +22 -0
- package/coinbase/constants.json +5 -0
- package/coinbase/get_coinbase_day_candles.js +155 -0
- package/coinbase/get_coinbase_historic_price.js +135 -0
- package/coinbase/index.js +7 -1
- package/dates/constants.json +3 -0
- package/dates/day_of_date.js +26 -0
- package/dates/index.js +5 -0
- package/dates/is_calendar_day.js +16 -0
- package/dates/time_of_date.js +61 -0
- package/docs/METHODOLOGY.md +984 -0
- package/index.js +2 -1
- package/package.json +29 -8
- package/rates/get_prices.js +51 -8
- package/storage/constants.json +4 -0
- package/storage/get_prices_file.js +169 -0
- package/storage/index.js +4 -0
- package/storage/is_prices.js +31 -0
- package/storage/put_prices_file.js +252 -0
- package/storage/temp_path.js +22 -0
- package/.gitattributes +0 -1
- package/.tm_properties +0 -1
- package/test/coinbase/test_get_coinbase_current_price.js +0 -77
- package/test/coindesk/test_get_coindesk_current_price.js +0 -95
- package/test/coingecko/test_get_coingecko_rates.js +0 -93
- package/test/rates/test_get_current_price.js +0 -74
- package/test/rates/test_get_prices.js +0 -78
|
@@ -0,0 +1,984 @@
|
|
|
1
|
+
# Deriving Prices From Chain Data
|
|
2
|
+
|
|
3
|
+
This document describes how the `chain` methods derive a daily USD price of
|
|
4
|
+
BTC using nothing but the blocks of the Bitcoin chain, why each step is there,
|
|
5
|
+
how the method tells whether it is working, and what its limits are.
|
|
6
|
+
|
|
7
|
+
## Purpose
|
|
8
|
+
|
|
9
|
+
Historic exchange rates are normally looked up from a third party. That means
|
|
10
|
+
a network request to a service that sees which dates are being asked about,
|
|
11
|
+
that may rate limit or remove access to older data, and whose answer cannot be
|
|
12
|
+
reproduced later by anyone else. The chain is the one data source every node
|
|
13
|
+
already has, so a price that is derived from it can be computed offline, gives
|
|
14
|
+
the same answer to everyone with the same blocks, and can be audited by
|
|
15
|
+
re-running the derivation over the recorded block range.
|
|
16
|
+
|
|
17
|
+
The derived price is not an exchange price. It is an estimate of the rate at
|
|
18
|
+
which people were paying for things in bitcoin on that day, read from the
|
|
19
|
+
amounts they paid. What exactly that number means is defined below.
|
|
20
|
+
|
|
21
|
+
## The Signal
|
|
22
|
+
|
|
23
|
+
People pay in round amounts of their own currency. Someone sending $100 of
|
|
24
|
+
bitcoin produces an output whose value in satoshis is `100 / price * 1e8`. On
|
|
25
|
+
any given day there are thousands of such payments, at $10, $20, $50, $100,
|
|
26
|
+
$500, $1,000 and so on, and all of them are divided by the same price.
|
|
27
|
+
|
|
28
|
+
Take the logarithm of every output value. A payment of a round amount `U`
|
|
29
|
+
lands at `log10(U) + c`, where `c = log10(1e8 / price)` is the same unknown
|
|
30
|
+
for every payment that day. So the round payments form a fixed pattern of
|
|
31
|
+
spikes, at the log positions of the popular round amounts, that is translated
|
|
32
|
+
as a whole by `c`. Finding the translation finds the price.
|
|
33
|
+
|
|
34
|
+
Each spike is blurred because the rates behind the payments are not all the
|
|
35
|
+
same: the price moves within the day, and wallets quote slightly different
|
|
36
|
+
rates at the same moment. So a spike spans roughly one to a few percent of its
|
|
37
|
+
value. The spikes sit on top of a background of everything else: non-round
|
|
38
|
+
payments, change outputs, consolidations, and so on. The background is dense
|
|
39
|
+
and its shape is unknown, but it has no reason to form spikes at round dollar
|
|
40
|
+
positions.
|
|
41
|
+
|
|
42
|
+
## Selecting Payments
|
|
43
|
+
|
|
44
|
+
Not every output is a payment, and the cleaner the input the sharper the
|
|
45
|
+
spikes. A transaction is treated as a payment when it is not a coinbase, has
|
|
46
|
+
at most 5 inputs, has exactly 2 outputs, has no OP_RETURN output, and has no
|
|
47
|
+
input carrying more than 500 bytes of witness data. Two outputs is the shape of
|
|
48
|
+
a regular payment with change. Many inputs, many outputs, and OP_RETURN mark
|
|
49
|
+
batching, consolidation and protocol traffic rather than purchases. Large
|
|
50
|
+
witness data marks inscriptions and other data embedding whose output values
|
|
51
|
+
are arbitrary.
|
|
52
|
+
|
|
53
|
+
Every transaction is judged on its own. An earlier version also dropped
|
|
54
|
+
transactions that spent outputs created earlier in the same set of blocks, to
|
|
55
|
+
remove chains of exchange plumbing, but whether a transaction is dropped by
|
|
56
|
+
that rule depends on which other blocks are being looked at, so a sample of a
|
|
57
|
+
day's blocks and the whole day would not agree on which outputs count. The
|
|
58
|
+
rule's benefit was never measured and what it removes is background, which
|
|
59
|
+
the matching below is indifferent to, so it is gone.
|
|
60
|
+
|
|
61
|
+
These filters are plausible rather than proven. Their effect on real blocks
|
|
62
|
+
should be measured by running the estimator with and without each one on days
|
|
63
|
+
with known prices, and the filters kept only where they sharpen the result.
|
|
64
|
+
Change outputs cannot be told from payments by their values, so about half
|
|
65
|
+
of the outputs that pass the filters are change. They have no reason to be
|
|
66
|
+
round dollar amounts, so they add to the background rather than forming false
|
|
67
|
+
spikes, which is what the background removal below is for.
|
|
68
|
+
|
|
69
|
+
Output values of 1,000 sats or less, or of 100,000 BTC or more, are ignored.
|
|
70
|
+
|
|
71
|
+
### Round Bitcoin Amounts
|
|
72
|
+
|
|
73
|
+
Values that are round amounts of bitcoin, any multiple of 1,000 sats, form
|
|
74
|
+
spikes of their own that do not move with the price. Each round value is
|
|
75
|
+
compared with the values immediately around it, within a tenth of a percent
|
|
76
|
+
on either side, and the excess of its count over what those neighbors lead one
|
|
77
|
+
to expect is removed as a round bitcoin spike. A round value that sits inside
|
|
78
|
+
a spike of round dollar payments has elevated neighbors, so it keeps the
|
|
79
|
+
density of that spike and loses only what rises above it.
|
|
80
|
+
|
|
81
|
+
What cannot be told apart is payers of a round dollar amount who all used
|
|
82
|
+
exactly one rate that put them on a round value: their payments look like a
|
|
83
|
+
round bitcoin spike and are removed with it. On a day where every payer of an
|
|
84
|
+
amount used a rate like that, the amount is lost. Payers whose rates differ
|
|
85
|
+
even slightly spread over the neighboring values and are kept. A wallet that
|
|
86
|
+
rounds its payments to the nearest 1,000 sats loses most of its contribution
|
|
87
|
+
to the signal but does not bias it, since its payments spread over the round
|
|
88
|
+
values on both sides of the true position.
|
|
89
|
+
|
|
90
|
+
The price is also derived with round values counted in full, and the round
|
|
91
|
+
shift is how far that moves the price. On synthetic days it is zero to a
|
|
92
|
+
hundredth of a percent; a large round shift means the round values carry a
|
|
93
|
+
pattern of their own, which is either round bitcoin payments lining up with
|
|
94
|
+
the template at some price or round dollar payers all at one rate.
|
|
95
|
+
|
|
96
|
+
### Repeated Amounts
|
|
97
|
+
|
|
98
|
+
Payments of round dollar amounts are made at rates that move through the day
|
|
99
|
+
and differ from one service to the next, so their values in sats spread over
|
|
100
|
+
many neighboring values and an exact value rarely repeats more than a few
|
|
101
|
+
times in a day. A flood of transactions from one sender is different: on
|
|
102
|
+
real days, chains of one input transactions repeated a handful of exact
|
|
103
|
+
amounts tens of thousands of times, forming spikes that the matching can take
|
|
104
|
+
for round dollar amounts. Each exact value that is not a round amount of
|
|
105
|
+
bitcoin is therefore counted at most ten times in an estimate, which keeps a
|
|
106
|
+
flood to a few outputs while leaving the spread of real payments as it was.
|
|
107
|
+
Round bitcoin values are left to the comparison with their neighbors above.
|
|
108
|
+
|
|
109
|
+
What this costs is payers who all paid one amount at exactly one rate, such as
|
|
110
|
+
a service that prices in dollars with a rate it holds for hours: the payments
|
|
111
|
+
of each amount beyond the tenth are not counted. They are a small part of
|
|
112
|
+
the signal on real days, where the cap changed the typical estimate by a
|
|
113
|
+
thousandth of a percent.
|
|
114
|
+
|
|
115
|
+
The cap is a count of exact values, not a share of a spike, so what it does
|
|
116
|
+
depends on how many values in sats an amount spreads over. At today's prices a
|
|
117
|
+
round amount spreads over hundreds of values and the cap leaves real spikes
|
|
118
|
+
alone, taking about 1% of the $10 spike at $100,000 in synthetic days of 20,000
|
|
119
|
+
round payments with rates spread by 1%. At a million dollars $10 is 1,000 sats
|
|
120
|
+
and the same spread covers a few dozen values, and the cap takes about two
|
|
121
|
+
thirds of the $10 spike, a third of the $20 spike and a seventh of the $50
|
|
122
|
+
spike. That bends the measured strengths towards the larger amounts, the
|
|
123
|
+
direction that favors the tenfold alias described under Aliases and Confidence.
|
|
124
|
+
A cap relative to the spread of each amount, as the comparison of round values
|
|
125
|
+
with their neighbors is relative, would not, and is left for the next version
|
|
126
|
+
of the method.
|
|
127
|
+
|
|
128
|
+
## Estimating the Price
|
|
129
|
+
|
|
130
|
+
### The Histogram
|
|
131
|
+
|
|
132
|
+
The output values are counted into a histogram on a log10 scale with 200 bins
|
|
133
|
+
per decade, so each bin is about 1.16% wide. That is a little narrower than
|
|
134
|
+
the blur of a spike on a calm day, which lets the spike be located precisely
|
|
135
|
+
without splitting it over too many bins. The histogram is normalized to sum to
|
|
136
|
+
one so that a day with many payments and a day with few are comparable.
|
|
137
|
+
|
|
138
|
+
### Excess Over the Local Background
|
|
139
|
+
|
|
140
|
+
A spike is only a spike relative to what surrounds it. For each bin, the local
|
|
141
|
+
background is the median of the bins within 12 bins on either side, leaving
|
|
142
|
+
out the two bins immediately next to it, and the excess is how far the bin
|
|
143
|
+
rises above that background, floored at zero. The template is matched against
|
|
144
|
+
the excess, not the raw histogram.
|
|
145
|
+
|
|
146
|
+
This matters more than it might seem. Matching against the raw histogram
|
|
147
|
+
rewards any position where the background happens to be dense, spikes or not,
|
|
148
|
+
and compensating for that requires assumptions about where the background
|
|
149
|
+
sits in dollar terms, which is exactly the thing that drifts between years and
|
|
150
|
+
price regimes. Matching against the excess makes the estimator indifferent to
|
|
151
|
+
the shape and position of the background, so the same procedure works at $300
|
|
152
|
+
and at $300,000 without tuning.
|
|
153
|
+
|
|
154
|
+
The fixed neighborhood is a compromise. On a day where the price signal is
|
|
155
|
+
widely dispersed, a spike is wider than the two excluded bins and its own
|
|
156
|
+
shoulders enter the background estimate, which understates the spike. That
|
|
157
|
+
affects every spike the same way so it does not move the position, only the
|
|
158
|
+
measured strength and the margins, and the width of the peak is measured and
|
|
159
|
+
reported so that such days can be recognized.
|
|
160
|
+
|
|
161
|
+
### The Template
|
|
162
|
+
|
|
163
|
+
The template is the set of round USD amounts people pay. It is not a list of
|
|
164
|
+
observations but is generated from the way people round numbers: the leading
|
|
165
|
+
digits 1, 1.5, 2, 2.5, 3, 4, 5, 6 and 7.5 at every power of ten from $1 to
|
|
166
|
+
$100,000, which gives 46 amounts. These positions are as stable as the habit
|
|
167
|
+
of rounding itself.
|
|
168
|
+
|
|
169
|
+
Each amount also has a prior weight, and this is the one place where the method
|
|
170
|
+
encodes an assumption about behavior rather than deriving it. The weight has
|
|
171
|
+
two parts. The digit part says that amounts led by 1, 2 and 5 are common and
|
|
172
|
+
amounts led by 1.5, 2.5, 3, 4, 6 and 7.5 are rare, which follows the
|
|
173
|
+
denominations money comes in. The envelope part says that round payments are
|
|
174
|
+
mostly between ten and a thousand dollars: it is a bell curve over powers of
|
|
175
|
+
ten, centered on a hundred dollars with a width of one decade, so an amount a
|
|
176
|
+
decade away from a hundred dollars keeps 61% of its weight, two decades away
|
|
177
|
+
14%, and three decades away about 1%. The prior is a dozen numbers and both
|
|
178
|
+
parts can be checked against the measured spike strengths that every estimate
|
|
179
|
+
reports.
|
|
180
|
+
|
|
181
|
+
For a candidate price, the template says which bin each amount should land
|
|
182
|
+
in. An amount whose spike would fall below 1,000 sats at that price cannot be
|
|
183
|
+
seen there and takes no part, for or against; the share of the prior weight
|
|
184
|
+
that is observable at a position is its evidence. The strength of a spike at
|
|
185
|
+
an observable amount is the excess at its bin plus its neighbors weighted by a
|
|
186
|
+
small kernel spanning two bins either side, which absorbs the blur.
|
|
187
|
+
|
|
188
|
+
Excess is floored at zero, so even where there is no spike the bins around an
|
|
189
|
+
amount add up to something positive, and a position that can see more
|
|
190
|
+
amounts would collect more of that noise and win for that reason alone. So
|
|
191
|
+
each spike strength is taken above the strength that noise alone produces
|
|
192
|
+
there, the trimmed mean of the excess in the surrounding bins with the spike
|
|
193
|
+
itself and the highest fifth of the neighbors left out, which brings the
|
|
194
|
+
score close to zero wherever there is no pattern, however many amounts are
|
|
195
|
+
observable. It is not exactly zero: leaving out the highest neighbors keeps
|
|
196
|
+
other spikes out of the noise but also puts the noise a little low, so on
|
|
197
|
+
simulated payments with no round amounts at all a position scored about 1% to
|
|
198
|
+
2% of what a real pattern of round payments scores at its price, slightly less
|
|
199
|
+
where fewer amounts were observable, a difference far smaller than the margin
|
|
200
|
+
a price needs over its aliases. The score of a position is the sum of these
|
|
201
|
+
strengths above
|
|
202
|
+
noise, weighted by the template weights, over the observable amounts, and
|
|
203
|
+
it is not scaled up for the amounts that cannot be seen: a position with less
|
|
204
|
+
to show for itself is not helped. The search scores every position in the
|
|
205
|
+
window, one bin at a time, and a parabola through the best bin and its two
|
|
206
|
+
neighbors places the peak between bins.
|
|
207
|
+
|
|
208
|
+
The search under the prior weights decides the alignment, that is, which
|
|
209
|
+
spike is which amount. The strength of every spike is then measured at that
|
|
210
|
+
alignment, and the measured strengths are used as the weights to place the
|
|
211
|
+
peak within the alignment, within two bins of the position the prior chose,
|
|
212
|
+
and to weight the refinement below. The measured strengths cannot be allowed
|
|
213
|
+
to choose between alignments: the alignment they were measured at always fits
|
|
214
|
+
them best, so using them for that would only confirm whatever the first pass
|
|
215
|
+
found. The alignment decision stays with the prior, and everything that
|
|
216
|
+
judges whether that decision was right is computed under the prior too.
|
|
217
|
+
|
|
218
|
+
### Aliases and Confidence
|
|
219
|
+
|
|
220
|
+
The round dollar pattern looks a lot like itself when shifted. Shift it by a
|
|
221
|
+
factor of two and $50 lands where $100 was, $20 where $10 was, $500 where
|
|
222
|
+
$1,000 was. The same is true, to a lesser degree, for a factor of five, of
|
|
223
|
+
ten, of two and a half, and for every other ratio between two amounts in the
|
|
224
|
+
pattern. No estimator can escape this: a pattern that is nearly invariant
|
|
225
|
+
under scaling cannot by itself pin down the scale. The data alone cannot say
|
|
226
|
+
whether people paid $100 at one price or $200 at half that price.
|
|
227
|
+
|
|
228
|
+
What breaks the symmetry is that the amounts are not equally popular. A
|
|
229
|
+
doubling puts the $20 and $200 spikes onto $40 and $400, amounts hardly anyone
|
|
230
|
+
pays, and puts $50 onto $100, which should be the strongest spike but is then
|
|
231
|
+
weaker than $200. A tenfold shift keeps every digit in place and is only told
|
|
232
|
+
apart by the envelope, that payments of $1,000 are rarer than payments of
|
|
233
|
+
$100. That is what the prior weights are for, and it is the only thing they
|
|
234
|
+
are for.
|
|
235
|
+
|
|
236
|
+
Not every ratio is an equally serious alias. Under the prior, and under
|
|
237
|
+
realistic spike strengths, the pattern shifted by two scores about 0.8 of the
|
|
238
|
+
pattern in place, by five or ten about 0.75, by two and a half or four about
|
|
239
|
+
0.4. The ratios below two, 1.2, 1.25, 1.33, 1.5, 1.6, 1.67 and 1.875, score
|
|
240
|
+
between 0.2 and 0.01 of the pattern in place, because few of the amounts land
|
|
241
|
+
on other amounts under those shifts. So the aliases at ratios of two and
|
|
242
|
+
above are the ones that can be mistaken for the price, and the ones below two
|
|
243
|
+
are weak but real.
|
|
244
|
+
|
|
245
|
+
Every estimate reports a confidence: the score of the best position divided by
|
|
246
|
+
the best score among its alias positions, which are the best position shifted
|
|
247
|
+
by the ratio between any two amounts in the template, in either direction, at
|
|
248
|
+
possible prices. An alias position only competes when it shows a
|
|
249
|
+
multi-denomination pattern of its own, several observable amounts with spikes
|
|
250
|
+
of at least a fifth of its strongest, so that a lone stray spike that happens
|
|
251
|
+
to land on one template amount is not treated as a rival price. Each position
|
|
252
|
+
is scored against the whole histogram through the amounts it can see there; two
|
|
253
|
+
candidate prices explain the same bins through different amounts, so this is
|
|
254
|
+
the like for like comparison, and the noise correction keeps a position from
|
|
255
|
+
winning merely by seeing more amounts. Both scores use the prior weights. The
|
|
256
|
+
confidence is reported as at most 100, which is what it is when no alias
|
|
257
|
+
position competes, because none shows a pattern that scores above noise, and
|
|
258
|
+
also when the best position scores more than a hundred times better than the
|
|
259
|
+
best alias. Real days are nowhere near that: every confidence logged on real
|
|
260
|
+
days while developing the method was below 1.3. A clear day gives a confidence
|
|
261
|
+
around 1.2 in synthetic data. A confidence near 1 means the data cannot
|
|
262
|
+
distinguish the price from a multiple of itself, and a confidence below 1 means
|
|
263
|
+
an alias outside the search window fits better than anything inside it, which
|
|
264
|
+
is a sign that the window was in the wrong place.
|
|
265
|
+
|
|
266
|
+
At high prices the small amounts fall under 1,000 sats and cannot be seen: $10
|
|
267
|
+
at a million dollars, $50 at five million. The floor is in sats while the
|
|
268
|
+
envelope is in dollars, so as the lower side of the envelope disappears, the
|
|
269
|
+
position a decade below the price sees amounts that the envelope favors about
|
|
270
|
+
as much as the price itself does. In synthetic days of 20,000 round payments
|
|
271
|
+
drawn from the prior, an estimate without a guide at a million dollars found
|
|
272
|
+
the price but was refused for a confidence of 1.09 to 1.10 against the 1.12 it
|
|
273
|
+
needed. At three and five million dollars it found a tenth of the price, with a
|
|
274
|
+
confidence of 1.09 to 1.20 against the 1.11 it needed and every other measure
|
|
275
|
+
healthy, and was supported in three of four runs. So far above today's prices a
|
|
276
|
+
starting price can be off by a factor of ten. A guided estimate cannot land on
|
|
277
|
+
a tenth of its guide, since neither its window nor the widened window reaches
|
|
278
|
+
that far, but the alias is still scored against it, since alias positions are
|
|
279
|
+
only limited to prices that are possible at all. In the same synthetic days,
|
|
280
|
+
guided by the true price, the estimate found the price every time, but its
|
|
281
|
+
confidence was 1.07 to 1.10 at a million dollars, about 1 at 1.8 million, 0.95
|
|
282
|
+
to 0.98 at two million and 0.88 to 0.92 at three million, so from about two
|
|
283
|
+
million dollars a guided day is refused. That is a regime the prior has not
|
|
284
|
+
been built for, and it is discussed under Limits.
|
|
285
|
+
|
|
286
|
+
The confidence says how well the prior separates the scale. It does not say
|
|
287
|
+
whether the price is right in any other sense, which is why the other health
|
|
288
|
+
measures below exist.
|
|
289
|
+
|
|
290
|
+
### Refinement
|
|
291
|
+
|
|
292
|
+
The rough price is accurate to a fraction of a bin. To do better, every output
|
|
293
|
+
is asked which round amount it is close to at the rough price. An output
|
|
294
|
+
within nine percent of a template amount implies the exact price at which it
|
|
295
|
+
would have been exactly that amount, and those implied prices are counted into
|
|
296
|
+
a fine histogram with 1,000 bins per decade, each weighted by the measured
|
|
297
|
+
strength of its amount. Nine percent is less than the spacing between
|
|
298
|
+
neighboring template amounts, so an output can imply at most one price.
|
|
299
|
+
|
|
300
|
+
Round payments all imply the true price and pile into a peak. The other
|
|
301
|
+
outputs imply prices spread smoothly across the histogram, because their
|
|
302
|
+
values are spread smoothly. The background of each fine bin is the median of
|
|
303
|
+
the bins that are not near it and is subtracted, the excess is smoothed with a
|
|
304
|
+
small triangular kernel to gather the peak, and a parabola through the top of
|
|
305
|
+
the peak places its center. When the peak is wider than the kernel, the kernel
|
|
306
|
+
is widened to match, up to a limit. The histogram is then rebuilt centered on
|
|
307
|
+
the center that was found and the center taken again, until it moves by less
|
|
308
|
+
than half a fine bin, so that the result does not depend on where in the
|
|
309
|
+
histogram the peak first landed. The center of the settled peak is the price.
|
|
310
|
+
Recentering stops after five passes, and a peak that is still moving then is
|
|
311
|
+
reported as unsettled. That alone does not refuse the estimate: an unsettled
|
|
312
|
+
estimate that passes every rule under Supported, Weak and Undetermined is
|
|
313
|
+
accepted, which mostly means its refinement moved by less than one percent
|
|
314
|
+
when started three percent higher or lower, as when a broad peak wanders the
|
|
315
|
+
same way from every start. On synthetic days that trended 8% to 25%, about
|
|
316
|
+
half of the refinements of whole days did not settle, and a few of those
|
|
317
|
+
passed every rule, one of them 7.2% from the middle of a day that moved 25%.
|
|
318
|
+
A full day is normally priced from its windows of about two hours, which
|
|
319
|
+
rarely trend that much, so an unsettled estimate of a whole day only sets the
|
|
320
|
+
price of a day whose supported windows cover less than half of it, while an
|
|
321
|
+
unsettled bridging estimate can set the window of the next day. Refusing an
|
|
322
|
+
unsettled estimate, at least when its search was not confined to a window, is
|
|
323
|
+
left for the next version of the method, since it changes which days are
|
|
324
|
+
supported and so needs the shipped prices to be derived again.
|
|
325
|
+
|
|
326
|
+
The refinement is then repeated from starting prices three percent above and
|
|
327
|
+
below the rough price, and with a range of six percent instead of nine, and
|
|
328
|
+
the stability is the most the price changed. On synthetic days it is zero to
|
|
329
|
+
a hundredth of a percent on calm days and around half a percent on days with
|
|
330
|
+
a wide dispersion of rates.
|
|
331
|
+
|
|
332
|
+
This replaces an earlier refinement that rounded every output to the nearest
|
|
333
|
+
five dollars and took the median of the implied prices. That approach was
|
|
334
|
+
anchored to its own starting point: outputs that are not round payments imply
|
|
335
|
+
prices that are spread symmetrically around whatever rough price was used, so
|
|
336
|
+
the median was a compromise between the rough price and the truth, weighted by
|
|
337
|
+
how much of the data was signal, and it could return the rough price almost
|
|
338
|
+
unchanged while appearing to have refined it. Removing the background before
|
|
339
|
+
taking the center, and recentering until the center settles, means the rough
|
|
340
|
+
price only decides where to look and cannot pull the answer towards itself.
|
|
341
|
+
|
|
342
|
+
### What the Daily Price Means
|
|
343
|
+
|
|
344
|
+
The price at the center of the peak of round payments is the rate at which
|
|
345
|
+
the typical round payment was made, on a log scale, after the background is
|
|
346
|
+
removed. For a calm day that is close to a payment weighted median of the
|
|
347
|
+
day's rates. On a day where the price trended it sits near the rate at which
|
|
348
|
+
the most payments happened, which need not be the middle of the range, since
|
|
349
|
+
payments crowd into the hours people are awake. Measured against the median
|
|
350
|
+
of the day's hourly prices, the peak of a whole day that moved 5% to 6% was up
|
|
351
|
+
to 3.8% away.
|
|
352
|
+
|
|
353
|
+
So a full day is priced from consecutive windows of about two hours, each with
|
|
354
|
+
a sharp peak of its own, and the price of the day is the time weighted median
|
|
355
|
+
of the windows: the price the day spent as long above as below, which is what
|
|
356
|
+
the reference used for validation measures too. Across the five years of prices
|
|
357
|
+
shipped with the method, the ten days that moved most, 10.4% to 15.4% from open
|
|
358
|
+
to close, were all within 1% of the reference, six of them within 0.1%. The
|
|
359
|
+
width of the peak in each window is reported as its dispersion, and the windows
|
|
360
|
+
themselves show how the price moved through the day.
|
|
361
|
+
|
|
362
|
+
## Health Measures
|
|
363
|
+
|
|
364
|
+
Every estimate reports several measures of its own quality, each catching a
|
|
365
|
+
different way the method could fail.
|
|
366
|
+
|
|
367
|
+
The payments count is the number of outputs in the refinement peak above the
|
|
368
|
+
background. It estimates how many genuine round payments the price rests on,
|
|
369
|
+
unlike a count of outputs that merely fall in a window, which would include
|
|
370
|
+
the background. It is a count at the peak, not for the day: on a day whose
|
|
371
|
+
price moved several percent the round payments are spread along the path the
|
|
372
|
+
price took and only those near the mode are counted, so the count falls as
|
|
373
|
+
the move grows, which on real days is a strong relationship. The same count
|
|
374
|
+
is taken at every other position in the
|
|
375
|
+
refinement range away from the peak, and the largest of them is the noise:
|
|
376
|
+
the best that a peak made of nothing achieved across all the positions that
|
|
377
|
+
were searched. The significance is the payments beyond the noise, in standard
|
|
378
|
+
deviations of the counts in the peak region. It is calibrated against the
|
|
379
|
+
search rather than against the winning position alone, so that a peak that
|
|
380
|
+
only slightly beats the best noise peak counts for little however many
|
|
381
|
+
outputs there were.
|
|
382
|
+
|
|
383
|
+
The confidence, described above, is the margin over the patterned aliases
|
|
384
|
+
under the prior, and the agreement is the cosine similarity between the
|
|
385
|
+
measured spike strengths and the prior weights. In synthetic data the
|
|
386
|
+
agreement is about 0.9 at the true price and about 0.6 at an alias, so a
|
|
387
|
+
falling agreement over time is the signal that the prior is drifting away from
|
|
388
|
+
how people pay, and the reported spike strengths are the means to refit it.
|
|
389
|
+
The evidence is the share of the prior weight that was observable at the
|
|
390
|
+
price; it is 1 at today's prices and falls as the price rises and the small
|
|
391
|
+
amounts drop below 1,000 sats.
|
|
392
|
+
|
|
393
|
+
The coverage is the share of all spike-like excess, across the span of the
|
|
394
|
+
template, that sits at round dollar positions. Spikes that are not at round
|
|
395
|
+
dollar positions, whether from another currency or from a service paying a
|
|
396
|
+
fixed odd amount, lower it.
|
|
397
|
+
|
|
398
|
+
The rivalry is the best score the pattern achieves at any other position
|
|
399
|
+
within a factor of two of the price that shows a pattern of its own, beyond
|
|
400
|
+
the dispersion of the peak, divided by the score at the best position. That
|
|
401
|
+
includes the weak aliases at ratios below two, which no window wide enough for
|
|
402
|
+
a daily move can exclude, and it includes round payments in a second currency
|
|
403
|
+
at its exchange rate: in synthetic data a euro pattern with half the volume of
|
|
404
|
+
the dollar pattern gives a rivalry of 0.5 at exactly the exchange rate, and
|
|
405
|
+
one with equal volume gives a rivalry near 1, at which point the method cannot
|
|
406
|
+
say which currency it is pricing. On a clean day the rivalry is about 0.2,
|
|
407
|
+
from the 1.5 ratio alias.
|
|
408
|
+
|
|
409
|
+
The dispersion is the width of the refinement peak at half its height, as a
|
|
410
|
+
fraction of the price, beyond the width that the smoothing of the histogram
|
|
411
|
+
gives every peak. The smoothing turns even a perfectly sharp peak into one
|
|
412
|
+
about 2% wide, so the smoothed width is a floor, not a measurement, on a calm
|
|
413
|
+
day; the floor is removed in quadrature, so a peak as sharp as the histogram
|
|
414
|
+
can resolve has a dispersion of zero, and a calm real day reads about 1%. The
|
|
415
|
+
smoothed width itself is still what decides how far from the peak a rival
|
|
416
|
+
position must be. The dispersion is that of the observed price signal, not a
|
|
417
|
+
measure of market movement: it reflects both the price moving during the
|
|
418
|
+
blocks that were used and wallets quoting different rates at the same time.
|
|
419
|
+
A day whose price stepped from one level to another does not read as wide at
|
|
420
|
+
all, since each level forms its own sharp peak; it is the rivalry that reports
|
|
421
|
+
the second level, and the drift that reports the step.
|
|
422
|
+
To tell the two apart, when a day's worth of blocks is used the price is also
|
|
423
|
+
estimated from the first half of the blocks and from the second half, and the
|
|
424
|
+
drift is how much the second differs from the first. A price that moved shows
|
|
425
|
+
a drift of the same order as the dispersion; dispersion with little drift is
|
|
426
|
+
rate scatter. The spikes themselves are reported as the measured strength of
|
|
427
|
+
each template amount relative to the strongest, and the round shift and the
|
|
428
|
+
stability are described above.
|
|
429
|
+
|
|
430
|
+
## Supported, Weak and Undetermined
|
|
431
|
+
|
|
432
|
+
An estimate is supported when at least three template amounts form spikes of at
|
|
433
|
+
least a fifth of the strongest, at least 100 round payments are in the peak,
|
|
434
|
+
the peak is at least four standard deviations beyond the best noise peak, the
|
|
435
|
+
refinement moves the price by less than one percent when its start or its range
|
|
436
|
+
is changed, every patterned rival position scores less than nine tenths of the
|
|
437
|
+
best, at least half of the prior weight was observable, and the price beats
|
|
438
|
+
every patterned alias. When the search was confined to a window around a known
|
|
439
|
+
price, beating the aliases means a confidence of at least 1. When it was not,
|
|
440
|
+
because there was no known price or because the window was widened, it means a
|
|
441
|
+
confidence of at least 1.1 with the whole pattern observable, and a margin that
|
|
442
|
+
grows as the observable evidence shrinks, one plus a tenth divided by the
|
|
443
|
+
evidence, since then the prior is the only thing excluding the aliases and less
|
|
444
|
+
of the pattern is there to support it.
|
|
445
|
+
|
|
446
|
+
A sample estimate that is not supported, or that rests on fewer than 500
|
|
447
|
+
payments, is weak, and is retried with twice as many blocks until it is
|
|
448
|
+
supported or all the blocks of the day have been used.
|
|
449
|
+
|
|
450
|
+
A full day is estimated both from all of its blocks at once and in consecutive
|
|
451
|
+
windows of twelve blocks, about two hours each, with a last window of fewer
|
|
452
|
+
than six blocks joined to the one before it. Each window is judged by the same
|
|
453
|
+
rules in the same price window, and a window whose search had to be widened is
|
|
454
|
+
not counted, even when the price it found is inside the price window after all,
|
|
455
|
+
as happens when its best position is within three bins of an edge. A window
|
|
456
|
+
covers the time from the block before it to its own last block, by their median
|
|
457
|
+
times past, described below. Timestamps can run backwards from one window to
|
|
458
|
+
the next, and the time they went back would then be counted again by the next
|
|
459
|
+
window, while the median time past never goes backwards, so every second of the
|
|
460
|
+
day is counted once. When the counted windows cover at least half of the day,
|
|
461
|
+
the day is supported, and its price is their time weighted median; otherwise
|
|
462
|
+
the estimate from all of the blocks at once is used when it is supported. On a
|
|
463
|
+
day that trends to an edge of the price window, the windows that were widened
|
|
464
|
+
can leave less than half of the day counted, and the day is then priced from
|
|
465
|
+
all of its blocks at once; counting such windows is left for the next version
|
|
466
|
+
of the method. The measures reported for a day priced from windows are those of
|
|
467
|
+
the median window, except for the outputs and payments, which are the totals of
|
|
468
|
+
the counted windows. A sample that is still not supported once all of the
|
|
469
|
+
blocks of its day are used is estimated in windows the same way.
|
|
470
|
+
|
|
471
|
+
A day that its windows do not support either is undetermined. It is not
|
|
472
|
+
remembered, not used to guide any other day, and the price lookup fails with
|
|
473
|
+
an error that names the day, so that a report built on it fails loudly rather
|
|
474
|
+
than carrying a guess.
|
|
475
|
+
|
|
476
|
+
The thresholds are set from synthetic data and are deliberately conservative.
|
|
477
|
+
They need to be calibrated on real blocks, as described under Validation.
|
|
478
|
+
|
|
479
|
+
## Finding the Blocks of a Day
|
|
480
|
+
|
|
481
|
+
A day is a UTC calendar day, and a block belongs to the day its median time
|
|
482
|
+
past falls on. The median time past of a block is the median of the
|
|
483
|
+
timestamps of the block and the ten blocks before it. Consensus requires every
|
|
484
|
+
block's timestamp to be later than the median time past of the blocks before
|
|
485
|
+
it, so unlike timestamps, which miners can set a couple of hours off and which
|
|
486
|
+
can run backwards from one block to the next, the median time past never
|
|
487
|
+
decreases with height. That makes every day a contiguous range of heights that
|
|
488
|
+
a binary search finds exactly, and that can never change because a later block
|
|
489
|
+
carries an earlier time. The median time past lags the clock by about an hour,
|
|
490
|
+
which is immaterial for a daily price.
|
|
491
|
+
|
|
492
|
+
A day is complete when at least six blocks have been mined after its last
|
|
493
|
+
block. With the day defined by median time past, completion is purely about
|
|
494
|
+
the chance of a chain reorganization replacing the blocks of the day, which six
|
|
495
|
+
confirmations make negligible. Only complete days get a final price.
|
|
496
|
+
|
|
497
|
+
## Estimating Day After Day
|
|
498
|
+
|
|
499
|
+
Because of the aliases, the estimator should never search blind when it does
|
|
500
|
+
not have to. Once one day's price is known, the next day's search is confined
|
|
501
|
+
to a window from 0.62 to 1.2 times that price. The window spans less than a
|
|
502
|
+
factor of two, so no price in the window has its double or its half in the
|
|
503
|
+
window too, which keeps out every alias at a ratio of two or more: the ones
|
|
504
|
+
that score close to the price itself. The weak aliases at ratios between 1.2
|
|
505
|
+
and 1.875 can be inside the window with the price; they are not excluded by
|
|
506
|
+
the window but are checked by the rivalry, which must stay below nine tenths
|
|
507
|
+
for the estimate to be supported. The window reaches further down than up
|
|
508
|
+
because falls have been sharper than rises.
|
|
509
|
+
|
|
510
|
+
The window is a guide, not a wall. When the best position is within three bins
|
|
511
|
+
of the edge of the window, or when an alias outside the window scores better
|
|
512
|
+
than anything inside it, the search is repeated in a window twice as wide on
|
|
513
|
+
each side, and the result is accepted only if it beats its aliases by the clear
|
|
514
|
+
margin, and is reported as widened, even when the price it finds turns out to
|
|
515
|
+
be inside the window. A real move of forty percent in a day is found that way.
|
|
516
|
+
|
|
517
|
+
A window placed on the wrong scale is not stable either. Suppose a day were
|
|
518
|
+
stored at double its true price. The next day's window would be centered on
|
|
519
|
+
that double, the true price would be outside it, and the best position inside
|
|
520
|
+
it would be the alias at double the truth. But the true price is an alias of
|
|
521
|
+
that position, so it is scored too, under the prior, and when the prior
|
|
522
|
+
prefers the truth, the confidence comes out below 1, the search widens, and
|
|
523
|
+
the true price is found with its clear margin. The chain of estimates heals
|
|
524
|
+
at the next day. What it cannot do is decide which of the two days was wrong:
|
|
525
|
+
the widened result then sits at an alias ratio of the day that guided it, and
|
|
526
|
+
one of the two is misplaced. That is treated as a scale conflict, the estimate
|
|
527
|
+
is not stored, and the lookup fails with an error naming both days and both
|
|
528
|
+
prices, so that the question is settled by checking one of them against the
|
|
529
|
+
world rather than by storing a file that disagrees with itself.
|
|
530
|
+
|
|
531
|
+
The conflict test takes a ratio within six percent of a ratio between two
|
|
532
|
+
amounts of the pattern, of two or more, as an alias ratio, and nearly every
|
|
533
|
+
ratio from 1.89 up is within six percent of one, apart from narrow bands such
|
|
534
|
+
as 2.12 to 2.26. So a real move from one day to the next of 47% or more down,
|
|
535
|
+
or 89% or more up, almost always ends as a scale conflict rather than a price,
|
|
536
|
+
and such a day is settled against the world like any other conflict.
|
|
537
|
+
|
|
538
|
+
The case that nothing inside the method can detect is a prior that prefers the
|
|
539
|
+
wrong scale on every day, which is discussed under Limits. It is the reason
|
|
540
|
+
that a price estimated without a guide is logged as a starting point and should
|
|
541
|
+
be checked against any exchange, and the reason the validation below exists.
|
|
542
|
+
|
|
543
|
+
When no price is known at all, the most recent complete day is estimated first
|
|
544
|
+
with an unconstrained search, which is where the clear margin applies. The
|
|
545
|
+
present is the regime the prior describes best, and the result can be checked
|
|
546
|
+
against any exchange immediately. A day that does not clear the margin on its
|
|
547
|
+
own, which on real days happens about once a month when the day's mix of
|
|
548
|
+
amounts happens to read well under a relabeling, is not a reason to fail: the
|
|
549
|
+
day before it is tried, and the one before that, for up to a week, and the
|
|
550
|
+
first day that identifies its price on its own is the starting point. The
|
|
551
|
+
same start is made when every known price is more than a week farther from
|
|
552
|
+
the requested day than the most recent complete day, since bridging back from
|
|
553
|
+
the present is then the shorter way.
|
|
554
|
+
|
|
555
|
+
Days between known estimates and the requested day are bridged one day at a
|
|
556
|
+
time. A bridging estimate uses an evenly spaced sample of 12 of the day's
|
|
557
|
+
blocks, one from the middle of each twelfth of the day's blocks, and doubles
|
|
558
|
+
the sample while the estimate is weak, keeping the blocks it has and adding the
|
|
559
|
+
blocks halfway between them, so that each doubling only fetches new blocks. A
|
|
560
|
+
day that is still not supported once the sample is the whole day is estimated
|
|
561
|
+
in windows, so that a day the price moved a lot in does not stop the bridge.
|
|
562
|
+
Each block is retrieved once however many estimates of the day use it. Bridging
|
|
563
|
+
estimates are kept as priors, used only to guide neighboring searches, and are
|
|
564
|
+
replaced when a day gets a full price.
|
|
565
|
+
|
|
566
|
+
A day that cannot be priced stops a lookup that has to bridge through it, with
|
|
567
|
+
an error that names the day, so that it can be checked against an exchange.
|
|
568
|
+
|
|
569
|
+
The day of the requested date itself is priced from all of its blocks, in
|
|
570
|
+
windows as described under What the Daily Price Means. A date on a day that is
|
|
571
|
+
not yet complete gets a provisional price from the confirmed blocks of the day
|
|
572
|
+
so far, padded with the blocks just before the day when the day has fewer than
|
|
573
|
+
36 confirmed blocks, so that a provisional price measures the same thing as the
|
|
574
|
+
final price as far as the data allows. The latest complete day is bridged to
|
|
575
|
+
first when it has no estimate, so that the guide of a provisional price is
|
|
576
|
+
never further back than that day, which is the day before except in the hour or
|
|
577
|
+
so after midnight, when the day before is not complete yet. When the latest
|
|
578
|
+
complete day is undetermined, the provisional price is guided by the nearest
|
|
579
|
+
other estimate instead of failing. A provisional price
|
|
580
|
+
is kept apart from the full day prices with the blocks it was derived from and
|
|
581
|
+
reused until about six more blocks have been mined, about an hour, after which
|
|
582
|
+
it is derived again, so that looking up the current day again does not fetch
|
|
583
|
+
its blocks each time. Once the day is complete, its full price replaces it. A
|
|
584
|
+
day that starts more than six hours after the median time of the chain tip is
|
|
585
|
+
refused, since the chain has not reached it and a provisional price would only
|
|
586
|
+
come from older blocks, as when the node is still syncing; six hours is the
|
|
587
|
+
span of the padding, and well beyond how far the median time lags the clock,
|
|
588
|
+
even when blocks are slow.
|
|
589
|
+
|
|
590
|
+
## Remembering Prices
|
|
591
|
+
|
|
592
|
+
Full day prices are kept as rates, bridging estimates as priors and the
|
|
593
|
+
provisional prices of the latest days apart from both in a JSON file. The file
|
|
594
|
+
carries a version of its own format, so that a reader can tell a file written
|
|
595
|
+
in another layout from one it understands. Each record carries the price, the
|
|
596
|
+
block range it was derived from, the hash of the last block in that range, and
|
|
597
|
+
a version made of the method version and a hash of the template, which is
|
|
598
|
+
separate from the file format version. Rates also carry the output and payment
|
|
599
|
+
counts and the main health measures, which show how well a price was supported
|
|
600
|
+
without its blocks, though not every input of the support rule, and they mark a
|
|
601
|
+
price that was estimated without a guide. Checking a price fully means deriving
|
|
602
|
+
it again from its block range, which the hash and the version make
|
|
603
|
+
reproducible.
|
|
604
|
+
|
|
605
|
+
Records whose method version does not match the running method are ignored, so
|
|
606
|
+
a change to the method or the template causes prices to be derived again rather
|
|
607
|
+
than mixing prices from different methods, without discarding the file. Records
|
|
608
|
+
whose last block is within about a week of the chain tip are checked against
|
|
609
|
+
the chain before use. A record whose block hash no longer
|
|
610
|
+
matches, because the chain was reorganized or the file came from another chain,
|
|
611
|
+
is removed from the file for good, as long as the file still holds that same
|
|
612
|
+
record rather than a newer price for the day, and the day is derived again when
|
|
613
|
+
it is needed. A record whose last block is beyond the chain tip, as when the
|
|
614
|
+
node is catching up after being down or another node shares the file, cannot be
|
|
615
|
+
checked yet, so it is left out of that lookup and kept in the file, to be
|
|
616
|
+
checked once the node reaches its block.
|
|
617
|
+
|
|
618
|
+
Every write merges with what is already in the file, so two processes deriving
|
|
619
|
+
prices at the same time add to each other's work instead of replacing it, and
|
|
620
|
+
each write goes to a temporary file of its own that is swapped into place so
|
|
621
|
+
that a reader never sees a partial file. Two writes that overlap by moments can
|
|
622
|
+
still lose one new entry, which is only derived again later. A full day rate
|
|
623
|
+
replaces the prior and the provisional price of its day only when they are of
|
|
624
|
+
its own version, so that a rate left by another version does not take the place
|
|
625
|
+
of current prices. A file that cannot be read, or whose format is not
|
|
626
|
+
understood, is an error rather than an empty file, and nothing is written over
|
|
627
|
+
it, so prices that could not be read are never replaced. An empty file is an
|
|
628
|
+
empty set of prices, and an entry that is not a price for a day of the calendar
|
|
629
|
+
is dropped rather than read. A lookup of a shipped day reads neither the node
|
|
630
|
+
nor the file, so a damaged file only fails the days it is needed for.
|
|
631
|
+
|
|
632
|
+
## Prices Shipped With the Method
|
|
633
|
+
|
|
634
|
+
Deriving a price takes the blocks of its day, and reaching a day takes a bridge
|
|
635
|
+
from a known price, so a first lookup far from the present waits for a long
|
|
636
|
+
bridge. The prices of every day up to a release are therefore derived before
|
|
637
|
+
the release and shipped with the package, and a lookup on one of those days is
|
|
638
|
+
answered from them without asking the node for anything. A day after them is
|
|
639
|
+
reached from the last shipped day, so a lookup a month after a release bridges
|
|
640
|
+
a month rather than from the present, unless the present is more than a week
|
|
641
|
+
closer, as it is for a recent day looked up long after a release.
|
|
642
|
+
|
|
643
|
+
The shipped prices are derived exactly as a lookup derives a requested day,
|
|
644
|
+
from all of the blocks of each day, guided by the day before it, using a local
|
|
645
|
+
Bitcoin Core. Before they are written, every new day has to be priced, and its
|
|
646
|
+
price has to fall within the range the day traded in on Coinbase Exchange, from
|
|
647
|
+
its lowest to its highest trade, which catches a day placed on the wrong scale
|
|
648
|
+
or pulled away by a flood while leaving room for where the payments of a day
|
|
649
|
+
that moved a lot were made. A price a percent or two off would still pass that
|
|
650
|
+
check, so every new price is also compared with the median of the hourly prices
|
|
651
|
+
of its day, and a price more than 5% from it stops the run to be looked into,
|
|
652
|
+
since none of the 1,824 shipped days measured against that median was more than
|
|
653
|
+
3.06% from it. On a version change each new price is also compared with the
|
|
654
|
+
price the day had before, and the run ends with the median, 95th percentile,
|
|
655
|
+
bias and largest of each comparison, to be looked over before the prices are
|
|
656
|
+
committed. The most recent shipped days are also derived again and have to come
|
|
657
|
+
out exactly as they were, which catches a change to the method that was not
|
|
658
|
+
given a new version. When the method version changes, every day is derived
|
|
659
|
+
again. When earlier days become supported, only they are derived, bridging back
|
|
660
|
+
from the first shipped day and then forward a day at a time, and the first
|
|
661
|
+
shipped day is derived again from its new neighbor and has to come out exactly
|
|
662
|
+
as it was.
|
|
663
|
+
|
|
664
|
+
The file is JSON holding the method version and the price in cents of each
|
|
665
|
+
day under its date, about 27 bytes a day, and it is only read the first time
|
|
666
|
+
a price is needed. Shipped prices are never written to a user's prices file,
|
|
667
|
+
and anyone with a node can derive them again with the same script.
|
|
668
|
+
|
|
669
|
+
## Accuracy
|
|
670
|
+
|
|
671
|
+
On synthetic days built from a realistic mix of round payments over a dense
|
|
672
|
+
background, the estimate is within 0.05% of the true price across prices from
|
|
673
|
+
$250 to $1,000,000, including days where the background is centered far from
|
|
674
|
+
where payments usually are, days where only 1% of outputs are round payments,
|
|
675
|
+
and days where the rates behind the payments were dispersed by 1%. Bridging
|
|
676
|
+
through 30 synthetic days that include falls of 35% and rises of 27% tracks
|
|
677
|
+
every day within 0.12%, with the large moves found by widening the window.
|
|
678
|
+
|
|
679
|
+
Adversarial synthetic days behave as follows. A day where only 0.3% of outputs
|
|
680
|
+
are round payments is misread by a factor of two and is correctly refused for
|
|
681
|
+
a significance below 1. A day whose rates trend 15% across the day is placed
|
|
682
|
+
within 1.6% of its midpoint with a dispersion of 10% and a stability of 0.6%;
|
|
683
|
+
a day with a 30% trend is placed 9% from its midpoint and is refused for a
|
|
684
|
+
stability of 1.5%, which is the right outcome, since such a day has no single
|
|
685
|
+
price. Payers who round their payments to the nearest 1,000 sats cost signal
|
|
686
|
+
but no accuracy, even at 80% of payers, and move the round shift by two
|
|
687
|
+
hundredths of a percent. Round payments in euros at half the dollar volume
|
|
688
|
+
have no effect on the price and give a rivalry of 0.5; at equal volume the
|
|
689
|
+
method prices the euro pattern and is refused for a rivalry of 0.99 and a
|
|
690
|
+
significance of 3. A service paying one fixed odd amount in 10% of all payment
|
|
691
|
+
outputs would move the price by 20% but is refused because it leaves fewer
|
|
692
|
+
than three spikes in the pattern. A window placed on the wrong scale by a
|
|
693
|
+
factor of two or five is detected with a confidence well below 1. A day on
|
|
694
|
+
which every payer of an amount used one rate that put it on a round value, at
|
|
695
|
+
a price of exactly $50,000, loses the pattern and is refused, with a round
|
|
696
|
+
shift of over 200% showing that the round values carried it.
|
|
697
|
+
|
|
698
|
+
## Validation
|
|
699
|
+
|
|
700
|
+
Synthetic data can only show that the mechanism works and that its failure
|
|
701
|
+
modes are the ones expected. The thresholds above, how well the prior
|
|
702
|
+
describes real payments, how large the margins really are, and what the health
|
|
703
|
+
measures look like on real blocks, have to be measured against known prices on
|
|
704
|
+
real blocks. `npm run check-prices` does that for a list of dates without
|
|
705
|
+
storing anything: it derives each price from a cold start, takes a reference
|
|
706
|
+
price for the day from an exchange, and reports every estimate's record
|
|
707
|
+
alongside its difference from the reference. It runs with a local Bitcoin
|
|
708
|
+
Core, like the script that updates the shipped prices.
|
|
709
|
+
|
|
710
|
+
The reference is the median of the typical prices of the day's hourly candles
|
|
711
|
+
on Coinbase Exchange, which go back to early 2015 and need no key: the price
|
|
712
|
+
the day traded above for half of its hours and below for the other half. A
|
|
713
|
+
time weighted measure is chosen on purpose. The chain price is where the most
|
|
714
|
+
round payments were made, and payments are spread through the day with the
|
|
715
|
+
hours people are awake, while exchange volume piles into the hours when the
|
|
716
|
+
price moves most violently, so a volume weighted average is dragged toward a
|
|
717
|
+
crash in a way the payments are not. The volume weighted average is reported
|
|
718
|
+
too, along with the day's first and last trades, and on a calm day they all
|
|
719
|
+
agree. When a day has no candles, CoinGecko's prices for the start of the day
|
|
720
|
+
and of the next day stand in, with their geometric mean as the reference, but
|
|
721
|
+
CoinGecko's public API only serves the past year.
|
|
722
|
+
|
|
723
|
+
The first measurements below were made against that pair of midnight prices
|
|
724
|
+
for every day, before the hourly reference existed, and it is a crude
|
|
725
|
+
instrument: two midnight prices say little about where a day that trended
|
|
726
|
+
spent its payments, so on a day that moved a few percent a difference of
|
|
727
|
+
around a percent was expected, with its sign following the trend.
|
|
728
|
+
|
|
729
|
+
The same comparison can be run unguided, with `--unguided`: each date
|
|
730
|
+
is estimated from the blocks of its own day with no price window from a
|
|
731
|
+
neighbor and no bridging, exactly as a cold start is, and the estimate is
|
|
732
|
+
reported whether or not the data support it, together with whether they do.
|
|
733
|
+
This is the direct test of the method's central claim, that one day's
|
|
734
|
+
payments identify the price on their own, and because it costs one day of
|
|
735
|
+
blocks per date rather than bridging from the present it is the practical
|
|
736
|
+
way to cover many dates across price regimes: the few hundred dollar days
|
|
737
|
+
with sparse payments, fee spikes and inscription waves that fill blocks with
|
|
738
|
+
dust, crash days, days when the price sat near a round number so that round
|
|
739
|
+
BTC and round USD values coincide, and days above $100,000 where the smallest
|
|
740
|
+
denominations fall below the minimum observable output. The estimates to study
|
|
741
|
+
are the ones that are wrong while supported and right while unsupported,
|
|
742
|
+
since those are what would change the support rule.
|
|
743
|
+
|
|
744
|
+
A cold start happens on the latest complete day. Every other day is reached by
|
|
745
|
+
bridging, with a neighboring day's price as its guide, which is the day after
|
|
746
|
+
it when bridging back from the present, so a day that cannot be priced on its
|
|
747
|
+
own blocks is not by itself a problem. What would be is a guided
|
|
748
|
+
estimate that is wrong and still supported, since bridging accepts it and uses
|
|
749
|
+
it to guide the day before. The comparison can be run reference-guided to look
|
|
750
|
+
for that, with `--reference-guided`: each date is estimated from the
|
|
751
|
+
blocks of its own day in the window that a correctly bridged neighbor would
|
|
752
|
+
give it, the window around the reference price of the next day, or around its
|
|
753
|
+
own reference price with `--own-reference`, and the search is widened and
|
|
754
|
+
checked for a scale conflict with its guide exactly as in a derivation. It also
|
|
755
|
+
costs one day of blocks per date. The guide stands in for a neighbor's chain
|
|
756
|
+
price, which on recent days is within a fraction of a percent of the reference.
|
|
757
|
+
The estimate is of the whole day, as for the requested day of a lookup, or with
|
|
758
|
+
`--sampled` of a sample of its blocks, the way bridging estimates every other
|
|
759
|
+
day: a dozen evenly spaced blocks, doubled until the estimate is supported and
|
|
760
|
+
rests on at least 500 round payments or the whole day is used.
|
|
761
|
+
|
|
762
|
+
The first real measurements, guided, were on the 30 consecutive days ending
|
|
763
|
+
October 16, 2026, with 18,000 to 36,000 round payments a day. All 30 were
|
|
764
|
+
derived, the median absolute difference from the reference was 0.25%, 22 were
|
|
765
|
+
within 0.5% and 27 within 1%, and the signed mean was +0.06%, so there was no
|
|
766
|
+
bias. The difference correlated 0.85 with the day's move from open to next
|
|
767
|
+
open, the chain price lay between the two opens on 21 of the 30 days and was
|
|
768
|
+
never more than 0.5% outside that range, and on the two days that moved 6% to
|
|
769
|
+
7% the chain price sat near the next day's open, the moves having happened
|
|
770
|
+
early in the day, with differences of 2.4% and 3.0% that are the reference's
|
|
771
|
+
failing rather than the chain's. The payment count correlated -0.73 with the
|
|
772
|
+
size of the move, as expected of a count at the peak. The health measures were
|
|
773
|
+
where the synthetic data put them: confidence 1.05 to 1.21 with a median of
|
|
774
|
+
1.17, rivalry 0.21 to 0.34 except for 0.55 on the day with the step change,
|
|
775
|
+
where the earlier level formed the rival, and stability at most 0.001. The two
|
|
776
|
+
lowest confidences, 1.05 and 1.11, were on the two days with the most
|
|
777
|
+
payments, both priced to within 0.2%. On the lower of the two the strongest
|
|
778
|
+
spike was at $2.50 instead of the usual $100 or $50, a burst of payments of
|
|
779
|
+
that size, and under a relabeling at twice the price $2.50 reads as $5, one of
|
|
780
|
+
the heaviest amounts in the prior, which narrowed the margin over that alias.
|
|
781
|
+
Derived on its own blocks with no guide, the day came to the same price to the
|
|
782
|
+
cent but was not supported, as it should not be when its payments alone
|
|
783
|
+
distinguish the price from its double less clearly than usual, and that is why
|
|
784
|
+
a cold start steps back from a day it cannot support rather than failing.
|
|
785
|
+
Those are the easy regime, recent days with abundant payments, and leave the
|
|
786
|
+
thresholds with wide margins there.
|
|
787
|
+
|
|
788
|
+
The first unguided measurements covered seven days from December 2023 to
|
|
789
|
+
October 2026, again against the pair of midnight prices. All seven were
|
|
790
|
+
supported on their own blocks. The day of the 2024 halving, when the fees paid
|
|
791
|
+
for a new token protocol left 5,341 round payments, about a quarter of the
|
|
792
|
+
count of a day a month before, was within 0.52%, and a day in the inscription
|
|
793
|
+
wave of December 2023 with 11,326 was within 0.42%. The three days above
|
|
794
|
+
$100,000, where the $1 amount falls below the smallest observable output, were
|
|
795
|
+
all supported; they were picked for large moves inside the day, and differed
|
|
796
|
+
from the midnight prices by 1.6% to 4.6%, which is what the hourly reference
|
|
797
|
+
is meant to settle, so they are to be measured again against it. October 13,
|
|
798
|
+
2026 came to 83,269.28 on its own blocks, the same price to the cent as in two
|
|
799
|
+
runs that reached it by bridging from a neighbor, which is the window doing
|
|
800
|
+
what it is meant to do: excluding aliases without moving the price.
|
|
801
|
+
|
|
802
|
+
Five days of the earlier regime have been measured unguided, against the hourly
|
|
803
|
+
reference, and none was supported. Three were wrong: December 17, 2017 came to
|
|
804
|
+
$102.01 against $19,489.49, June 26, 2019 to $5,008.25 against $12,603.35, and
|
|
805
|
+
March 12, 2020 to $10,023.34 against $6,170.90. Each locked onto a price just
|
|
806
|
+
above $100, $5,000 or $10,000, where round amounts of bitcoin are also round
|
|
807
|
+
dollar amounts: at $10,000, 0.001 BTC is $10. Round amounts of bitcoin sent
|
|
808
|
+
with the fee taken out of the amount would do that, since they sit just below
|
|
809
|
+
the round values, where the comparison with neighboring values, which only
|
|
810
|
+
removes exact round values, leaves them in place, and they read as round
|
|
811
|
+
dollars at a price just above the round number. The first two fell short of the
|
|
812
|
+
margin over the aliases, with confidences of 1.03 and 1.01, but March 12, 2020
|
|
813
|
+
cleared it at 1.43 and was refused by another of the checks, so the margin
|
|
814
|
+
alone would not have caught it. The other two days, May 19, 2021 and November
|
|
815
|
+
9, 2022, both of which fell 14%, were right, within 0.93% and 0.25%, and fell
|
|
816
|
+
short of the margin, with confidences of 0.94 and 1.05 and rivalries of 0.42
|
|
817
|
+
and 0.44, above the recent days' range. The coincidence prices, one, two or
|
|
818
|
+
five times a power of ten, do not move with the price, and no two are more than
|
|
819
|
+
two and a half times apart while a guided window spans almost two, so most
|
|
820
|
+
windows contain one; the window for June 26, 2019 contains $10,000. A guided
|
|
821
|
+
estimate only needs a confidence of 1 to be supported, so the next measurements
|
|
822
|
+
are of the same days, reference-guided.
|
|
823
|
+
|
|
824
|
+
A year of days, October 20, 2025 to October 18, 2026, was then checked
|
|
825
|
+
reference-guided from samples, the way bridging estimates them, against the
|
|
826
|
+
hourly reference. 358 of the 364 days were supported, with a median absolute
|
|
827
|
+
difference of 0.12%, 95% of them within 1.33% and none more than 3.88% off, and
|
|
828
|
+
no bias. All 161 days that moved less than 1% from open to close were within
|
|
829
|
+
0.95%, and every difference over 2% was on a day that moved at least 2%, where
|
|
830
|
+
a single price for the day says little. Three of the six days that were not
|
|
831
|
+
supported were floods of repeated amounts. On May 9, 2026 eight exact amounts
|
|
832
|
+
from 1,046 to 2,146 sats, nearly all in one input transactions and most of
|
|
833
|
+
those spending a transaction of the same amount, made 22% of the day's payment
|
|
834
|
+
outputs and pulled the estimate onto a pattern of $1 and $1.50, and on March 22
|
|
835
|
+
and 23 a family of amounts from 3,451 to 3,553 sats, each repeated hundreds to
|
|
836
|
+
thousands of times, put March 23 at twice its price. Counting each exact amount
|
|
837
|
+
at most ten times priced all three within 0.8% from the first sample, and that
|
|
838
|
+
is now part of the method. Across the whole year it supported 361 of the 364
|
|
839
|
+
days, moved the typical price by a thousandth of a percent and 95% of days by
|
|
840
|
+
less than 0.04%, brought 95% of days within 1.2%, and improved nine of the ten
|
|
841
|
+
days it moved by more than 0.1%, among them the three floods and days next to
|
|
842
|
+
them that were partly flooded. The other three, December 2 and 15, 2025 and
|
|
843
|
+
February 25, 2026, moved 2% to 6% during the day and were right as a whole but
|
|
844
|
+
not supported, because the payments spread over the day's range read as noise.
|
|
845
|
+
Estimated in windows of twelve blocks, every window of those three days was
|
|
846
|
+
supported, and so was every window of two days that moved 5% to 6% and of a
|
|
847
|
+
calm day, 71 windows in all. The windows followed Coinbase over the same hours
|
|
848
|
+
with a median difference of 0.04% to 0.13% a day and at most 1.08%, in the
|
|
849
|
+
window a crash happened in, and the time weighted medians of the windows were
|
|
850
|
+
within 0.2% of the reference on all six days. On seven days that moved 4.8% to
|
|
851
|
+
12.3%, the time weighted median of the windows was within 0.95% of the
|
|
852
|
+
reference where the peak of the whole day was up to 3.8% away, and since then
|
|
853
|
+
every full day is priced from its windows. With both changes in the method, a
|
|
854
|
+
lookup that bridged from October 19, 2026 back to October 20, 2025 reached
|
|
855
|
+
every day, and the six days it priced in full, among them two of the floods
|
|
856
|
+
and the three days that had needed windows, were within 0.78% of the
|
|
857
|
+
reference.
|
|
858
|
+
|
|
859
|
+
The four years before, October 20, 2021 to October 19, 2025, were checked the
|
|
860
|
+
same way with the method as it now stands. Two days, January 31 and February 1,
|
|
861
|
+
2023, were left unchecked when Coinbase answered the request for their prices
|
|
862
|
+
with a server error, and every one of the other 1,459 days was supported, with
|
|
863
|
+
a median absolute difference of 0.12%, 95% of them within 0.93% and none more
|
|
864
|
+
than 3.79% off, and no bias. All 613 days that moved less than 1% from open to
|
|
865
|
+
close were within 1.93%, the eleven differences over 2% were on days that moved
|
|
866
|
+
more than 1% and were estimated from samples of about two hours, and the 121
|
|
867
|
+
days within 3% of $20,000, $50,000 or $100,000, where round amounts of bitcoin
|
|
868
|
+
are also round dollar amounts, were all supported and within 2.33%, among them
|
|
869
|
+
the summer of 2022 that the price spent near $20,000. November 9, 2022, which
|
|
870
|
+
fell 14% and was short of the margin on its own blocks, was within 0.68%
|
|
871
|
+
guided. Prices are therefore derived from October 20, 2021 on.
|
|
872
|
+
|
|
873
|
+
The shipped prices, every day from October 20, 2021 to October 22, 2026, were
|
|
874
|
+
then derived, each day from all of its blocks and guided by the day before, and
|
|
875
|
+
every day fell within the range it traded in on Coinbase. Against the hourly
|
|
876
|
+
reference, on the 1,824 days it covers, the median difference was 0.06%, 95% of
|
|
877
|
+
days were within 0.38%, 1,813 within 1% and all but two within 2%, with no
|
|
878
|
+
bias, and each year's median was between 0.04% and 0.07%. The largest
|
|
879
|
+
differences were 3.06% on September 13, 2022, a day that fell 9.9%, where the
|
|
880
|
+
price was within 0.08% of the day's volume weighted average price, and 2.12% on
|
|
881
|
+
March 4, 2024, a day that rose 8.2%. The 51 days within 3% of $20,000 had a
|
|
882
|
+
median difference of 0.07% and were all within 1.7%. Counting the time of each
|
|
883
|
+
window by the median time past of its blocks instead of their timestamps moved
|
|
884
|
+
526 of the days from the previous table, by a median of 0.07% and at most
|
|
885
|
+
2.15%, and brought August 8, 2024, a day that rose 11.9%, from 2.63% to 0.42%
|
|
886
|
+
away from the reference.
|
|
887
|
+
|
|
888
|
+
The calibration should be done on one set of historical periods and tested on
|
|
889
|
+
another, across price regimes, from the thousand dollar prices of early 2017,
|
|
890
|
+
the earliest days the method is meant to support, to the present, and it should
|
|
891
|
+
look hardest at the estimates that are wrong while passing every check, since
|
|
892
|
+
those are the ones a user would never notice. The expected pattern is that the
|
|
893
|
+
measures separate the wrong estimates from the right ones; where they do not,
|
|
894
|
+
the measures are incomplete, not the data.
|
|
895
|
+
|
|
896
|
+
## Manipulation
|
|
897
|
+
|
|
898
|
+
Someone who wanted to skew the derived price would have to create payments
|
|
899
|
+
that look like round dollar payments at a different price, in enough volume to
|
|
900
|
+
outweigh the real ones. The estimator weighs outputs, not value, so the cost
|
|
901
|
+
is in transaction fees: a two output transaction is about 140 virtual bytes,
|
|
902
|
+
so at a fee rate of 1 to 5 sats per virtual byte each forged payment costs
|
|
903
|
+
about $0.10 to $0.50 at a $70,000 price. Real round payments are on the order
|
|
904
|
+
of tens of thousands a day, so matching them costs thousands of dollars a day
|
|
905
|
+
and takes a few percent of the day's block space, and doing so for a whole
|
|
906
|
+
month costs on the order of a hundred thousand dollars and is visible as a
|
|
907
|
+
sustained change in the payment count and the spike pattern. Forging a single
|
|
908
|
+
spike is cheaper but is refused as not being a pattern; forging the whole
|
|
909
|
+
pattern at a wrong scale is detected as a rival when the volumes are
|
|
910
|
+
comparable, and only succeeds silently when the forged volume clearly exceeds
|
|
911
|
+
the real one.
|
|
912
|
+
|
|
913
|
+
Weighting payments by the fees they paid would raise the cost of such an
|
|
914
|
+
attack, but fees are not visible in the block data available here, since they
|
|
915
|
+
require the values of the inputs being spent, and the proxies that are
|
|
916
|
+
visible, such as position in the block, are weak. The practical defense is
|
|
917
|
+
the one above: the measures that change when the data changes, and the record
|
|
918
|
+
of what every price was derived from.
|
|
919
|
+
|
|
920
|
+
## Limits
|
|
921
|
+
|
|
922
|
+
The price is for the currency whose round payments dominate, which is the
|
|
923
|
+
dollar. Round payments in other currencies form their own pattern at their own
|
|
924
|
+
rate; while the dollar pattern is the larger, the others only lower the
|
|
925
|
+
coverage and raise the rivalry, and if another currency's round payments ever
|
|
926
|
+
matched the dollar's, the method would price that currency instead and the
|
|
927
|
+
rivalry near 1 would say so. The method is mainnet only, and dates before
|
|
928
|
+
October 20, 2021 are refused for now. Every day since then has been checked
|
|
929
|
+
against exchange prices, while earlier years have only been sampled, and of the
|
|
930
|
+
four days measured before then none was supported on its own and three locked
|
|
931
|
+
onto prices where round amounts of bitcoin are also round dollar amounts, which
|
|
932
|
+
needs work before those years are supported.
|
|
933
|
+
|
|
934
|
+
The prior weights are the one assumption about behavior, and the scale of
|
|
935
|
+
every price rests on them. If people's payment habits shifted so that every
|
|
936
|
+
amount was paid as often as twice that amount is paid today, the data would be
|
|
937
|
+
indistinguishable from the same habits at half the price and no method could
|
|
938
|
+
tell the two apart. If payments of a thousand dollars became as common as
|
|
939
|
+
payments of a hundred, the envelope would be wrong and the scale could be off
|
|
940
|
+
by ten. A prior that is wrong in one of these ways is wrong on every day, so
|
|
941
|
+
the chain of estimates would be coherent and no check inside the method would
|
|
942
|
+
fire; the agreement would drift and the measured strengths would stop
|
|
943
|
+
resembling the prior, but only a comparison with the world settles it. Such an
|
|
944
|
+
error can only enter at a cold start, since a day guided by a correct neighbor
|
|
945
|
+
searches a window where no strong alias can be, and a day guided by a wrong
|
|
946
|
+
neighbor is caught as a scale conflict when the prior prefers the truth.
|
|
947
|
+
|
|
948
|
+
A day whose round payments come in only one or two amounts cannot be placed
|
|
949
|
+
by the data, since one spike could be any amount, and the method refuses it
|
|
950
|
+
when the pattern has fewer than three spikes but can be misled when it has
|
|
951
|
+
exactly the wrong two. A day on which every payer of an amount used the same
|
|
952
|
+
rate, and that rate put the amount on a multiple of 1,000 sats, loses that
|
|
953
|
+
amount. A day on which the price signal is dispersed by more than ten percent
|
|
954
|
+
has no single price; the estimate is somewhere within the range, the
|
|
955
|
+
dispersion says how wide the range was, and the stability check refuses the
|
|
956
|
+
widest.
|
|
957
|
+
|
|
958
|
+
The estimate depends on there being many round payments in a day. Sampled
|
|
959
|
+
estimates of a few thousand outputs can be weak and are escalated; full days
|
|
960
|
+
of a hundred thousand or more outputs are not weak in testing. A large share
|
|
961
|
+
of non-round junk concentrated in a narrow band of values can create a spike
|
|
962
|
+
of its own and mislead the matcher if it is accompanied by enough other spikes
|
|
963
|
+
to pass as a pattern.
|
|
964
|
+
|
|
965
|
+
Far above today's prices two parts of the method stop working as intended,
|
|
966
|
+
because they count in sats rather than in dollars. The floor of 1,000 sats
|
|
967
|
+
hides the smallest amounts, and the cap of ten on each exact value trims real
|
|
968
|
+
spikes of small amounts once a round amount spreads over few values in sats,
|
|
969
|
+
about two thirds of the $10 spike at a million dollars with rates spread by 1%.
|
|
970
|
+
Both push an estimate without a guide towards a tenth of the price, which from
|
|
971
|
+
about three million dollars it can accept, as described under Aliases and
|
|
972
|
+
Confidence. A guided estimate still finds the price, but from about two
|
|
973
|
+
million dollars a tenth of the price outscores it, and since widening the
|
|
974
|
+
window cannot reach that alias, the day is refused and bridging stops there.
|
|
975
|
+
Neither matters at today's prices, but prices at such levels should be checked
|
|
976
|
+
against an exchange, and a floor and a cap that scale with the price are left
|
|
977
|
+
for the next version of the method.
|
|
978
|
+
|
|
979
|
+
Finally, this is slow compared to a lookup. A day is about 144 blocks, which
|
|
980
|
+
is a few hundred megabytes through the LND chain backend: tens of seconds per
|
|
981
|
+
day with a local full node, minutes per day when blocks have to be fetched
|
|
982
|
+
from peers, plus about a second of computation per estimate for the
|
|
983
|
+
refinement, its stability variants and the round value comparison. Prices are
|
|
984
|
+
remembered precisely so that this cost is only paid once per day.
|