commonmeta-ruby 3.2.14 → 3.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile.lock +2 -2
- data/bin/commonmeta +1 -1
- data/lib/commonmeta/author_utils.rb +1 -1
- data/lib/commonmeta/cli.rb +14 -0
- data/lib/commonmeta/crossref_utils.rb +56 -14
- data/lib/commonmeta/readers/json_feed_reader.rb +30 -2
- data/lib/commonmeta/utils.rb +34 -0
- data/lib/commonmeta/version.rb +1 -1
- data/spec/cli_spec.rb +12 -3
- data/spec/fixtures/vcr_cassettes/Commonmeta_CLI/doi_prefix/doi_prefix_by_blog.yml +997 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_CLI/doi_prefix/doi_prefix_by_uuid.yml +256 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_CLI/encode/by_blog.yml +997 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_CLI/encode/by_uuid.yml +256 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_doi_prefix_for_blog/by_blog_id.yml +997 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_doi_prefix_for_blog/by_blog_post_uuid.yml +389 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_doi_prefix_for_blog/by_blog_post_uuid_specific_prefix.yml +389 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_json_feed_item/by_uuid.yml +136 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_json_feed_item_metadata/blog_post_with_non-url_id.yml +136 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_json_feed_item_metadata/ghost_post_with_organizational_author.yml +91 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/get_json_feed_item_metadata/substack_post_with_broken_reference.yml +1316 -0
- data/spec/fixtures/vcr_cassettes/Commonmeta_Metadata/write_metadata_as_crossref/json_feed_item_from_rogue_scholar_with_organizational_author.yml +91 -0
- data/spec/readers/json_feed_reader_spec.rb +280 -186
- data/spec/utils_spec.rb +8 -0
- data/spec/writers/crossref_xml_writer_spec.rb +28 -0
- metadata +14 -2
@@ -0,0 +1,1316 @@
|
|
1
|
+
---
|
2
|
+
http_interactions:
|
3
|
+
- request:
|
4
|
+
method: get
|
5
|
+
uri: https://rogue-scholar.org/api/posts/2b105b29-acbc-4eae-9ff1-368803f36a4d
|
6
|
+
body:
|
7
|
+
encoding: UTF-8
|
8
|
+
string: ''
|
9
|
+
headers:
|
10
|
+
Connection:
|
11
|
+
- close
|
12
|
+
Host:
|
13
|
+
- rogue-scholar.org
|
14
|
+
User-Agent:
|
15
|
+
- http.rb/5.1.1
|
16
|
+
response:
|
17
|
+
status:
|
18
|
+
code: 200
|
19
|
+
message: OK
|
20
|
+
headers:
|
21
|
+
Age:
|
22
|
+
- '0'
|
23
|
+
Cache-Control:
|
24
|
+
- public, max-age=0, must-revalidate
|
25
|
+
Content-Length:
|
26
|
+
- '34607'
|
27
|
+
Content-Type:
|
28
|
+
- application/json; charset=utf-8
|
29
|
+
Date:
|
30
|
+
- Sat, 17 Jun 2023 18:41:34 GMT
|
31
|
+
Etag:
|
32
|
+
- '"xkb42zldhmqpb"'
|
33
|
+
Server:
|
34
|
+
- Vercel
|
35
|
+
Strict-Transport-Security:
|
36
|
+
- max-age=63072000
|
37
|
+
X-Matched-Path:
|
38
|
+
- "/api/posts/[slug]"
|
39
|
+
X-Vercel-Cache:
|
40
|
+
- MISS
|
41
|
+
X-Vercel-Id:
|
42
|
+
- fra1::iad1::27p8s-1687027290531-713fbf76c0ba
|
43
|
+
Connection:
|
44
|
+
- close
|
45
|
+
body:
|
46
|
+
encoding: UTF-8
|
47
|
+
string: '{"id":"https://doi.org/10.59350/bbcsr-r4b59","uuid":"2b105b29-acbc-4eae-9ff1-368803f36a4d","url":"https://markrubin.substack.com/p/the-preregistration-prescriptiveness","title":"The
|
48
|
+
Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science","summary":"Comments
|
49
|
+
on Van Drimmelen (2023)","date_published":"2023-06-06T21:22:00Z","date_modified":null,"date_indexed":"2023-06-17T17:51:53.479476+00:00","authors":[{"url":null,"name":"Mark
|
50
|
+
Rubin"}],"image":"https://substack-post-media.s3.amazonaws.com/public/images/195477de-91ce-4a76-bae8-b6334fb5bcc7_500x379.jpeg","content_html":"<h4>Abstract</h4><p>I
|
51
|
+
discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’
|
52
|
+
decision making during the research process. In particular, I consider his
|
53
|
+
evidence that researchers’ discretion over research decisions is unavoidable
|
54
|
+
when they follow research plans that are either overdetermined (i.e., too
|
55
|
+
prescriptive) or underdetermined (i.e., too vague). I argue that this evidence
|
56
|
+
points to a <em>prescriptiveness trade-off</em> when writing preregistered
|
57
|
+
plans: All other things being equal, plans that are more prescriptive are
|
58
|
+
more likely to result in deviations that turn their confirmatory tests into
|
59
|
+
exploratory tests, and plans that are less prescriptive are more likely to
|
60
|
+
result in confirmatory tests that are susceptible to questionable research
|
61
|
+
practices. I also consider Van Drimmelen’s idea that researchers may
|
62
|
+
make unconscious, implicit decisions during the research process. I relate
|
63
|
+
these implicit decisions to Rumsfeld’s (2002) concept of <em>unknown
|
64
|
+
unknowns</em>: “the things we don’t know we don’t know”!
|
65
|
+
I argue that scientists can report their <em>known knowns</em> (what they
|
66
|
+
know they did and found), and they can be transparent and speculative about
|
67
|
+
their <em>known unknowns</em> (what they know they didn’t do and may
|
68
|
+
find), but that they can’t say much about their <em>unknown unknowns
|
69
|
+
</em>(including their unconscious, implicit decisions) because, by definition,
|
70
|
+
they don’t know what they are! Nonetheless, I think that it’s
|
71
|
+
important to acknowledge unknown unknowns in science because doing so helps
|
72
|
+
to contextualise research efforts as being highly tentative and fallible.</p><h4>Introduction</h4><p>The
|
73
|
+
videos of the presentations at the Metascience2023 conference were recently
|
74
|
+
made available <a href=\"https://www.youtube.com/playlist?list=PLChfyH8TVDGlPTcKrVboBaf6QAGSJt1hQ\">here</a>.
|
75
|
+
One that caught my eye is by Tom van Drimmelen, titled “Researchers’
|
76
|
+
decision making: Navigating ambiguity in research practice”: </p><div
|
77
|
+
id=\"youtube2-5omWDTPcsNo\" class=\"youtube-wrap\" data-attrs=\"{\"videoId\":\"5omWDTPcsNo\",\"startTime\":null,\"endTime\":null}\"
|
78
|
+
data-component-name=\"Youtube2ToDOM\"><div class=\"youtube-inner\"><iframe
|
79
|
+
src=\"https://www.youtube-nocookie.com/embed/5omWDTPcsNo?rel=0&autoplay=0&showinfo=0&enablejsapi=0\"
|
80
|
+
frameborder=\"0\" loading=\"lazy\" gesture=\"media\" allow=\"autoplay; fullscreen\"
|
81
|
+
allowautoplay=\"true\" allowfullscreen=\"true\" width=\"728\" height=\"409\"></iframe></div></div><p>Here,
|
82
|
+
I summarise Van Drimmelen’s excellent work and share my thoughts on
|
83
|
+
his ideas.</p><h4>What is Researcher Discretion?</h4><p>Van Drimmelen (2023)
|
84
|
+
explained that, during the research process, researchers have discretion over
|
85
|
+
numerous decisions that they make about their methodology and analyses. During
|
86
|
+
the data analysis stage, this <em>researcher discretion</em> may be perceived
|
87
|
+
negatively as <em>researcher degrees of freedom</em> that is susceptible to
|
88
|
+
<em>researcher bias</em> and <em>questionable research practices</em> (John
|
89
|
+
et al., 2012; Simmons et al., 2011).</p><div class=\"captioned-image-container\"><figure><a
|
90
|
+
class=\"image-link is-viewable-img image2\" target=\"_blank\" href=\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png\"
|
91
|
+
data-component-name=\"Image2ToDOM\"><div class=\"image2-inset\"><picture><source
|
92
|
+
type=\"image/webp\" srcset=\"https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
93
|
+
424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
94
|
+
848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
95
|
+
1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
96
|
+
1456w\" sizes=\"100vw\"><img src=\"https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png\"
|
97
|
+
width=\"1430\" height=\"759\" data-attrs=\"{\"src\":\"https://substack-post-media.s3.amazonaws.com/public/images/d6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png\",\"fullscreen\":null,\"imageSize\":null,\"height\":759,\"width\":1430,\"resizeWidth\":null,\"bytes\":511459,\"alt\":null,\"title\":null,\"type\":\"image/png\",\"href\":null,\"belowTheFold\":false,\"topImage\":true,\"internalRedirect\":null}\"
|
98
|
+
class=\"sizing-normal\" alt=\"\" srcset=\"https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
99
|
+
424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
100
|
+
848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
101
|
+
1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6f03976-0133-406f-8572-cdbcbbdc3aef_1430x759.png
|
102
|
+
1456w\" sizes=\"100vw\" fetchpriority=\"high\"></picture><div class=\"image-link-expand\"><svg
|
103
|
+
xmlns=\"http://www.w3.org/2000/svg\" width=\"16\" height=\"16\" viewBox=\"0
|
104
|
+
0 24 24\" fill=\"none\" stroke=\"#FFFFFF\" stroke-width=\"2\" stroke-linecap=\"round\"
|
105
|
+
stroke-linejoin=\"round\" class=\"lucide lucide-maximize2\"><polyline points=\"15
|
106
|
+
3 21 3 21 9\"></polyline><polyline points=\"9 21 3 21 3 15\"></polyline><line
|
107
|
+
x1=\"21\" x2=\"14\" y1=\"3\" y2=\"10\"></line><line x1=\"3\" x2=\"10\" y1=\"21\"
|
108
|
+
y2=\"14\"></line></svg></div></div></a><figcaption class=\"image-caption\"><em>Figure
|
109
|
+
1. </em>Tom van Drimmelen illustrates the numerous decisions that researchers
|
110
|
+
make during the research process.</figcaption></figure></div><h4>Researcher
|
111
|
+
Discretion Emerges in Both Over- and Under-Determined Research Plans</h4><p>In
|
112
|
+
theory, it is possible to reduce and/or reveal researcher discretion during
|
113
|
+
data analyses by preregistering one or more methodological and analytical
|
114
|
+
research paths. However, Van Drimmelen (2023) and colleagues were interested
|
115
|
+
in how researcher discretion operates <em>in practice</em>. To investigate,
|
116
|
+
they conducted an ethnographic study in which they observed researcher discretion
|
117
|
+
in two different research groups. They found that researcher discretion emerged
|
118
|
+
when researchers used two categories of research plan.</p><p>1.
|
119
|
+
<em>Underdetermined plans</em>: Researcher discretion emerged when research
|
120
|
+
plans were not precise enough to determine specific actions for the researchers.
|
121
|
+
In this case, researchers needed to use their own discretion to fill in the
|
122
|
+
gaps of the research plan and figure out how to proceed. Researchers often
|
123
|
+
find themselves in this situation because preregistered research plans are
|
124
|
+
often vague (e.g., Bakker et al., 2020; Heirene et al., 2021; Van den Akker
|
125
|
+
et al., 2023).</p><p>2. <em>Overdetermined plans</em>:
|
126
|
+
Researcher discretion also emerged when there were unforeseen changes in the
|
127
|
+
research situation that made the original research plan undesirable or impossible,
|
128
|
+
necessitating an alternative research approach. Again, this situation is common,
|
129
|
+
because most researchers do not possess “godlike planning abilities”
|
130
|
+
(Navarro, 2020). To attempt to address this issue, researchers can preregister
|
131
|
+
“if…then” contingencies (i.e., decision trees) that accommodate
|
132
|
+
<em>anticipated</em> changes in the research situation (Nosek et al., 2018,
|
133
|
+
p. 2602). In addition, it may be possible to update a preregistered analysis
|
134
|
+
plan prior to the data analysis in order to take account of <em>unanticipated</em>
|
135
|
+
events (Nosek et al., 2019, p. 817). However, it is often the case that <em>unanticipated</em>
|
136
|
+
events arise <em>during data analyses</em>. As Reinhart (2015, p. 95) explained
|
137
|
+
(paraphrasing Helmuth von Moltke), “no analysis plan survives contact
|
138
|
+
with the data.” In these cases, researchers either need to stick to
|
139
|
+
an inadequate preregistered analysis plan or deviate and adapt their approach
|
140
|
+
to address the current situation. The evidence shows that researchers often
|
141
|
+
deviate from their preregistered plans (e.g., Abrams et al., 2020; Claesen
|
142
|
+
et al., 2021; Heirene et al., 2021).</p><h4>A Prescriptiveness Trade-Off in
|
143
|
+
Preregistered Plans</h4><p>Van Drimmelen’s (2023) work shows us that
|
144
|
+
researchers are forced to either deviate from overdetermined plans to overcome
|
145
|
+
unforeseen events or fill in the gaps of vague, underdetermined plans. Hence,
|
146
|
+
researcher discretion emerges in the case of both over- and under-determined
|
147
|
+
plans. As Van Drimmelen (2023) put it, “researcher discretion is an
|
148
|
+
integral and unavoidable part of the research practice.”</p><p>I think
|
149
|
+
Van Drimmelen’s (2023) evidence also points to a trade-off between the
|
150
|
+
prescriptiveness of preregistered research plans and the feasibility of their
|
151
|
+
implementation in an unpredictable world. As preregistered plans become more
|
152
|
+
precise and prescriptive, there is a greater chance that researchers will
|
153
|
+
need to deviate from them in order to accommodate unforeseen and/or uncontrolled
|
154
|
+
events that occur during the implementation of the plan. For example, researchers
|
155
|
+
are more likely to deviate from a plan to test “300 participants”
|
156
|
+
than they are to deviate from a less prescriptive plan to test “<em>around</em>
|
157
|
+
300 participants,” because uncontrolled events (e.g., the number of
|
158
|
+
participant exclusions during data analysis) may result in a final sample
|
159
|
+
size that is only close to 300 participants and not exactly 300 participants
|
160
|
+
(e.g., Claesen et al., 2021, pp. 6-7). Consequently, as preregistered plans
|
161
|
+
become more prescriptive, they are more likely to result in deviations that
|
162
|
+
turn their planned confirmatory tests into unplanned exploratory tests.<sup>1</sup>
|
163
|
+
Of course, preregistration continues to distinguish between confirmatory and
|
164
|
+
exploratory tests in this situation, and it prevents exploratory tests from
|
165
|
+
being falsely portrayed as confirmatory tests. Nonetheless, it remains the
|
166
|
+
case that, as plans for confirmatory tests become more prescriptive, the actual
|
167
|
+
implementation of these confirmatory tests in the real world becomes less
|
168
|
+
likely.</p><p>On the other hand, as preregistered plans become vaguer and
|
169
|
+
less prescriptive, confirmatory hypothesis tests will become not only more
|
170
|
+
feasible to achieve without deviation, but also more susceptible to researcher
|
171
|
+
degrees of freedom and questionable research practices. For example, it is
|
172
|
+
more feasible to test “<em>around</em> 300 participants” than
|
173
|
+
to test exactly “300 participants.” However, the first sampling
|
174
|
+
plan does not prevent the questionable research practice of optional stopping
|
175
|
+
(John et al., 2021; Simmons et al., 2011), whereas the second, more prescriptive
|
176
|
+
plan does.</p><p>In summary, researchers need to consider a <em>prescriptiveness
|
177
|
+
trade-off </em>when writing their preregistered plans: All other things being
|
178
|
+
equal, plans that are more prescriptive are more likely to result in deviations
|
179
|
+
that turn their confirmatory tests into exploratory tests, and plans that
|
180
|
+
are less prescriptive are more likely to result in confirmatory tests that
|
181
|
+
are susceptible to questionable research practices. Hence, researchers face
|
182
|
+
a difficult choice between prescriptive confirmatory tests that they are less
|
183
|
+
likely to carry out in practice and vague confirmatory tests that are more
|
184
|
+
prone to questionable research practices.</p><div class=\"captioned-image-container\"><figure><a
|
185
|
+
class=\"image-link is-viewable-img image2\" target=\"_blank\" href=\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg\"
|
186
|
+
data-component-name=\"Image2ToDOM\"><div class=\"image2-inset\"><picture><source
|
187
|
+
type=\"image/webp\" srcset=\"https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
188
|
+
424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
189
|
+
848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
190
|
+
1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
191
|
+
1456w\" sizes=\"100vw\"><img src=\"https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg\"
|
192
|
+
width=\"284\" height=\"429.408\" data-attrs=\"{\"src\":\"https://substack-post-media.s3.amazonaws.com/public/images/26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg\",\"fullscreen\":null,\"imageSize\":null,\"height\":756,\"width\":500,\"resizeWidth\":284,\"bytes\":93786,\"alt\":null,\"title\":null,\"type\":\"image/jpeg\",\"href\":null,\"belowTheFold\":true,\"topImage\":false,\"internalRedirect\":null}\"
|
193
|
+
class=\"sizing-normal\" alt=\"\" srcset=\"https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
194
|
+
424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
195
|
+
848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
196
|
+
1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26ad757c-146e-4d8f-88a5-f34cf5c71c58_500x756.jpeg
|
197
|
+
1456w\" sizes=\"100vw\" loading=\"lazy\"></picture><div class=\"image-link-expand\"><svg
|
198
|
+
xmlns=\"http://www.w3.org/2000/svg\" width=\"16\" height=\"16\" viewBox=\"0
|
199
|
+
0 24 24\" fill=\"none\" stroke=\"#FFFFFF\" stroke-width=\"2\" stroke-linecap=\"round\"
|
200
|
+
stroke-linejoin=\"round\" class=\"lucide lucide-maximize2\"><polyline points=\"15
|
201
|
+
3 21 3 21 9\"></polyline><polyline points=\"9 21 3 21 3 15\"></polyline><line
|
202
|
+
x1=\"21\" x2=\"14\" y1=\"3\" y2=\"10\"></line><line x1=\"3\" x2=\"10\" y1=\"21\"
|
203
|
+
y2=\"14\"></line></svg></div></div></a></figure></div><h4>Researchers Sometimes
|
204
|
+
Make Implicit Decisions</h4><p>Van Drimmelen and colleagues (2023) also identified
|
205
|
+
another interesting issue in their work. At times, they found it difficult
|
206
|
+
to identify researcher discretion because there were multiple potential methodological
|
207
|
+
or analytical approaches available that researchers did not consider. In other
|
208
|
+
words, sometimes <em>researchers</em> <em>were not aware that they were making
|
209
|
+
decisions</em>!</p><p>This idea of unconscious and implicit research decisions
|
210
|
+
reminds me of something that Donald Rumsfeld, the United States Secretary
|
211
|
+
of Defence, said at a press briefing in 2002:</p><blockquote><p>There are
|
212
|
+
known knowns; there are things we know we know. We also know there are known
|
213
|
+
unknowns; that is to say we know there are some things we do not know. But
|
214
|
+
there are also unknown unknowns—the ones we don’t know we don’t
|
215
|
+
know. (Wikipedia, 2023)</p></blockquote><p>In the context of Figure 1, our
|
216
|
+
<em>known knowns</em> are the path or paths that we reported following in
|
217
|
+
our research (e.g., the green paths). In these cases, we know what we did,
|
218
|
+
and we know what we found as a result. In contrast, our <em>known unknowns</em>
|
219
|
+
are the paths that we actively decided not to follow (e.g., all the other
|
220
|
+
paths in Figure 1). We know these paths exist, but we don’t yet know
|
221
|
+
what results lie at the end of them.</p><p>But what about our <em>unknown
|
222
|
+
unknowns</em>? These paths don’t appear in Figure 1! They relate to
|
223
|
+
decisions that we didn’t even consider, at least not at a conscious
|
224
|
+
level. For example, we may not have considered the time of day of testing,
|
225
|
+
the ambient temperature, or a host of other unspecified variables that may
|
226
|
+
be influential on our results. Instead, we unconsciously relegated these unknown
|
227
|
+
unknowns to a <em>ceteris paribus</em> clause (i.e., “all other things
|
228
|
+
being equal”), which assumes that no other influential factors are at
|
229
|
+
play (Meehl, 1990). The problem is that our unconscious and implicit decision
|
230
|
+
to assume the irrelevance of these factors may be wrong! Indeed, the history
|
231
|
+
of science is full of cases in which factors that were not initially considered
|
232
|
+
to be important subsequently turned out to be crucial moderators or boundary
|
233
|
+
conditions of effects (for some examples, see Firestein, 2016).</p><div class=\"captioned-image-container\"><figure><a
|
234
|
+
class=\"image-link is-viewable-img image2\" target=\"_blank\" href=\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg\"
|
235
|
+
data-component-name=\"Image2ToDOM\"><div class=\"image2-inset\"><picture><source
|
236
|
+
type=\"image/webp\" srcset=\"https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
237
|
+
424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
238
|
+
848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
239
|
+
1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
240
|
+
1456w\" sizes=\"100vw\"><img src=\"https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg\"
|
241
|
+
width=\"296\" height=\"444\" data-attrs=\"{\"src\":\"https://substack-post-media.s3.amazonaws.com/public/images/96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg\",\"fullscreen\":null,\"imageSize\":null,\"height\":750,\"width\":500,\"resizeWidth\":296,\"bytes\":89822,\"alt\":null,\"title\":null,\"type\":\"image/jpeg\",\"href\":null,\"belowTheFold\":true,\"topImage\":false,\"internalRedirect\":null}\"
|
242
|
+
class=\"sizing-normal\" alt=\"\" srcset=\"https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
243
|
+
424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
244
|
+
848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
245
|
+
1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96e8d69a-f2c1-4c22-b375-8c5c4fd72689_500x750.jpeg
|
246
|
+
1456w\" sizes=\"100vw\" loading=\"lazy\"></picture><div class=\"image-link-expand\"><svg
|
247
|
+
xmlns=\"http://www.w3.org/2000/svg\" width=\"16\" height=\"16\" viewBox=\"0
|
248
|
+
0 24 24\" fill=\"none\" stroke=\"#FFFFFF\" stroke-width=\"2\" stroke-linecap=\"round\"
|
249
|
+
stroke-linejoin=\"round\" class=\"lucide lucide-maximize2\"><polyline points=\"15
|
250
|
+
3 21 3 21 9\"></polyline><polyline points=\"9 21 3 21 3 15\"></polyline><line
|
251
|
+
x1=\"21\" x2=\"14\" y1=\"3\" y2=\"10\"></line><line x1=\"3\" x2=\"10\" y1=\"21\"
|
252
|
+
y2=\"14\"></line></svg></div></div></a></figure></div><h4>We Can’t Be
|
253
|
+
Transparent About Unknown Unknowns, But We Should Acknowledge They Exist!</h4><p>Van
|
254
|
+
Drimmelen (2023) concluded that “we might not be able to be effectively
|
255
|
+
transparent because <em>we don’t know which decisions we’ve actually
|
256
|
+
made</em>” (my emphasis). In other words, our lack of awareness about
|
257
|
+
our potentially important implicit decisions means that we can’t be
|
258
|
+
transparent about them: We can report our known knowns, and we can be transparent
|
259
|
+
and speculative about our known unknowns, but we can’t say much about
|
260
|
+
our unknown unknowns because we don’t know what they are! By definition,
|
261
|
+
as soon as we start to speculate on a potentially influential factor in our
|
262
|
+
research, it becomes a <em>known</em> unknown about which we need to make
|
263
|
+
conscious decisions!</p><p>Despite our inability to specify our unknown unknowns,
|
264
|
+
Van Drimmelen’s (2023) work reminds us that it’s important to
|
265
|
+
acknowledge this type of ignorance in science because it helps to contextualise
|
266
|
+
our research efforts as being highly tentative and fallible steps in a collective
|
267
|
+
process of discovery and theory development. Acknowledging our unknown unknowns
|
268
|
+
also helps to address recent calls for greater intellectual humility and modesty
|
269
|
+
in science (Hoekstra & Vazire, 2021; Ramsey, 2021).</p><p>I’ll end with
|
270
|
+
a nice example of scientists acknowledging unknown unknowns. Here’s
|
271
|
+
Errington et al. (2021, p. 12) discussing the challenges of assessing replicability
|
272
|
+
in preclinical cancer biology:</p><blockquote><p>It might also be that, in
|
273
|
+
some cases, a failure to replicate was caused by the replication team deviating
|
274
|
+
from the protocol in some way that was not recognized, or that a key part
|
275
|
+
of the procedure was left out of the protocol inadvertently. It is also possible
|
276
|
+
that the effect reported in the original paper depended on methodological
|
277
|
+
factors that were not identified by original authors, the replication team,
|
278
|
+
or any other experts involved in the peer review of the original paper or
|
279
|
+
the Registered Report.</p></blockquote><h4>References</h4><p>Abrams, E., Libgober,
|
280
|
+
J., & List, J. A. (2020). Research registries: Facts, myths, and possible
|
281
|
+
improvements (No. w27250). <em>National Bureau of Economic Research. </em><a
|
282
|
+
href=\"https://doi.org/10.3386/w27250\">https://doi.org/10.3386/w27250</a></p><p>Bakker,
|
283
|
+
M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek,
|
284
|
+
B. A.,…& Wicherts, J. M. (2020). Ensuring the quality and specificity
|
285
|
+
of preregistrations. <em>PLoS Biology, 18</em>(12), e3000937. <a href=\"https://doi.org/10.1371/journal.pbio.3000937\">https://doi.org/10.1371/journal.pbio.3000937</a></p><p>Claesen,
|
286
|
+
A., Gomes, S., Tuerlinckx, F., Vanpaemel, W., & Leuven, K. U. (2021). Comparing
|
287
|
+
dream to reality: An assessment of adherence of the first generation of preregistered
|
288
|
+
studies. <em>Royal Society Open Science, 8</em>(10), 1–11. <a href=\"https://doi.org/10.1098/rsos.211037\">https://doi.org/10.1098/rsos.211037</a></p><p>Errington,
|
289
|
+
T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Reproducibility
|
290
|
+
in cancer biology: Challenges for assessing replicability in preclinical cancer
|
291
|
+
biology. <em>Elife, 10,</em> Article e67995. <a href=\"https://doi.org/10.7554/eLife.67995\">https://doi.org/10.7554/eLife.67995</a></p><p>Fife,
|
292
|
+
D., & Rodgers, J. L. (2019). Moving beyond the \"replication crisis\": Understanding
|
293
|
+
the exploratory/confirmatory data analysis continuum. <em>PsyArXiv</em>. <a
|
294
|
+
href=\"https://doi.org/10.31234/osf.io/5vfq6\">https://doi.org/10.31234/osf.io/5vfq6</a></p><p>Firestein,
|
295
|
+
S. (2016, February 14). Why failure to replicate findings can actually be
|
296
|
+
good for science. <em>LA Times.</em> <a href=\"https://www.latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html\">https://www.latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html</a></p><p>Heirene,
|
297
|
+
R., LaPlante, D., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A.,
|
298
|
+
& Gainsbury, S. M. (2021). Preregistration specificity & adherence: A review
|
299
|
+
of preregistered gambling studies & cross-disciplinary comparison. <em>PsyArXiv</em>.
|
300
|
+
<a href=\"https://psyarxiv.com/nj4es/\">https://psyarxiv.com/nj4es/</a></p><p>Hoekstra,
|
301
|
+
R., & Vazire, S. (2021). Aspiring to greater intellectual humility in science.
|
302
|
+
<em>Nature Human Behaviour, 5</em>(12), 1602-1607. <a href=\"https://doi.org/10.1038/s41562-021-01203-8\">https://doi.org/10.1038/s41562-021-01203-8</a></p><p>John,
|
303
|
+
L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable
|
304
|
+
research practices with incentives for truth telling. <em>Psychological Science,
|
305
|
+
23</em>(5), 524–532. <a href=\"https://doi.org/10.1177/0956797611430953\">https://doi.org/10.1177/0956797611430953</a></p><p>Meehl,
|
306
|
+
P. E. (1990). Appraising and amending theories: The strategy of Lakatosian
|
307
|
+
defense and two principles that warrant it. <em>Psychological Inquiry, 1</em>(2),
|
308
|
+
108-141. <a href=\"https://doi.org/10.1207/s15327965pli0102_1\">https://doi.org/10.1207/s15327965pli0102_1</a></p><p>Navarro,
|
309
|
+
D. (2020, September 23). Paths in strange spaces: A comment on preregistration.<em>
|
310
|
+
PsycArXiv. </em><a href=\"https://doi.org/10.31234/osf.io/wxn58\">https://doi.org/10.31234/osf.io/wxn58</a></p><p>Nosek,
|
311
|
+
B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor,
|
312
|
+
D. T., van ‘t Veer, A. E., & Vazire, S. (2019). Preregistration is hard,
|
313
|
+
and worthwhile. <em>Trends in Cognitive Sciences, 23,</em> 815–818.
|
314
|
+
<a href=\"https://doi.org/10.1016/j.tics.2019.07.009\">https://doi.org/10.1016/j.tics.2019.07.009</a></p><p>Ramsey,
|
315
|
+
R. (2021). A call for greater modesty in psychology and cognitive neuroscience.
|
316
|
+
<em>Collabra: Psychology, 7</em>(1), 24091. <a href=\"https://doi.org/10.1525/collabra.24091\">https://doi.org/10.1525/collabra.24091</a></p><p>Reinhart,
|
317
|
+
A. (2015). <em>Statistics done wrong: The woefully complete guide.</em> No
|
318
|
+
Starch Press.</p><p>Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011).
|
319
|
+
False-positive psychology: Undisclosed flexibility in data collection and
|
320
|
+
analysis allows presenting anything as significant. <em>Psychological Science,
|
321
|
+
22(</em>11), 1359–1366. <a href=\"https://doi.org/10.1177%20/0956797611417632\">https://doi.org/10.1177
|
322
|
+
/0956797611417632</a></p><p>Van den Akker, O., Bakker, M., van Assen, M. A.
|
323
|
+
L. M., Pennington, C. R., Verweij, L., Elsherif, M. M.,…Wicherts, J.
|
324
|
+
M. (2023, May 10). The effectiveness of preregistration in psychology: Assessing
|
325
|
+
preregistration strictness and preregistration-study consistency. <em>PsyArXiv</em>.
|
326
|
+
<a href=\"https://doi.org/10.31222/osf.io/h8xjw\">https://doi.org/10.31222/osf.io/h8xjw</a></p><p>Van
|
327
|
+
Drimmelen, T., Slagboom, N., Reis, R., Bouter, L., Van der Steen, J. (2023,
|
328
|
+
May 9). <em>Researchers’ decision making: Navigating ambiguity in research
|
329
|
+
practice.</em> Metascience2023, Washington, DC. Abstract: <a href=\"https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/\">https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/</a>
|
330
|
+
Preregistration: <a href=\"https://osf.io/tqwgp/\">https://osf.io/tqwgp/</a></p><p>Wikipedia.
|
331
|
+
(2023, March 16). There are unknown unknowns. <em>Wikipedia</em>. https://en.wikipedia.org/wiki/There_are_unknown_unknowns</p><h4>Endnotes</h4><p>1. Here,
|
332
|
+
I define confirmatory tests as planned tests, and exploratory tests as unplanned
|
333
|
+
tests. This conceptualization is consistent with the frequentist rationale
|
334
|
+
for preregistration, according to which unplanned tests undermine assumptions
|
335
|
+
regarding Type I error rates (Nosek et al., 2019, p. 816). I acknowledge that
|
336
|
+
there are other definitions for “confirmatory” and “exploratory”,
|
337
|
+
including definitions that view them as existing at opposite ends of a continuum
|
338
|
+
rather than as discrete categories (e.g., Fife & Rodgers, 2019). According
|
339
|
+
to this continuum perspective, larger deviations from preregistered plans
|
340
|
+
result in “more exploratory” tests than smaller deviations. I
|
341
|
+
don’t subscribe to this continuum perspective, because it is unclear
|
342
|
+
how to formally assess some deviations as being “larger” or “smaller”
|
343
|
+
than others. I agree that research studies can be described as being more
|
344
|
+
confirmatory or exploratory, because they can contain a greater proportion
|
345
|
+
of planned or unplanned tests. However, in my view, the tests themselves can
|
346
|
+
only be either confirmatory or exploratory, because they can only be either
|
347
|
+
planned or unplanned.</p><h4>Acknowledgements</h4><p>I’m grateful to
|
348
|
+
Tom van Drimmelen for his comments on an earlier version of this article.</p><h4>PDF
|
349
|
+
Download</h4><p>You can download a PDF version of this article <a href=\"https://doi.org/10.31222/osf.io/3t7pc\">here</a>.</p><h4>Reference</h4><p>The
|
350
|
+
reference for this article is:</p><p>Rubin, M. (2023, June 7). The preregistration
|
351
|
+
prescriptiveness trade-off and unknown unknowns in science: Comments on Van
|
352
|
+
Drimmelen (2023). <em>Critical Metascience: MetaArXiv</em>. <a href=\"https://doi.org/10.31222/osf.io/3t7pc\">https://doi.org/10.31222/osf.io/3t7pc</a></p><h4>Please
|
353
|
+
Subscribe and/or Share!</h4><p class=\"button-wrapper\" data-attrs=\"{\"url\":\"https://markrubin.substack.com/subscribe?\",\"text\":\"Subscribe
|
354
|
+
now\",\"action\":null,\"class\":null}\" data-component-name=\"ButtonCreateButton\"><a
|
355
|
+
class=\"button primary\" href=\"https://markrubin.substack.com/subscribe?\"><span>Subscribe
|
356
|
+
now</span></a></p><p class=\"button-wrapper\" data-attrs=\"{\"url\":\"https://markrubin.substack.com/p/the-preregistration-prescriptiveness?utm_source=substack&utm_medium=email&utm_content=share&action=share\",\"text\":\"Share\",\"action\":null,\"class\":null}\"
|
357
|
+
data-component-name=\"ButtonCreateButton\"><a class=\"button primary\" href=\"https://markrubin.substack.com/p/the-preregistration-prescriptiveness?utm_source=substack&utm_medium=email&utm_content=share&action=share\"><span>Share</span></a></p>","tags":[],"language":"en","references":[{"doi":"https://doi.org/10.3386/w27250","key":"ref1"},{"doi":"https://doi.org/10.1371/journal.pbio.3000937","key":"ref2"},{"doi":"https://doi.org/10.1098/rsos.211037","key":"ref3"},{"doi":"https://doi.org/10.7554/elife.67995","key":"ref4"},{"doi":"https://doi.org/10.31234/osf.io/5vfq6","key":"ref5"},{"key":"ref6","url":"https://latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html"},{"key":"ref7","url":"https://psyarxiv.com/nj4es"},{"doi":"https://doi.org/10.1038/s41562-021-01203-8","key":"ref8"},{"doi":"https://doi.org/10.1177/0956797611430953","key":"ref9"},{"doi":"https://doi.org/10.1207/s15327965pli0102_1","key":"ref10"},{"doi":"https://doi.org/10.31234/osf.io/wxn58","key":"ref11"},{"doi":"https://doi.org/10.1016/j.tics.2019.07.009","key":"ref12"},{"doi":"https://doi.org/10.1525/collabra.24091","key":"ref13"},{"doi":"https://doi.org/10.1177%20/0956797611417632","key":"ref14"},{"doi":"https://doi.org/10.1177","key":"ref15"},{"doi":"https://doi.org/10.31222/osf.io/h8xjw","key":"ref16"},{"key":"ref17","url":"https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice"},{"key":"ref18","url":"https://osf.io/tqwgp"},{"key":"ref19","url":"https://en.wikipedia.org/wiki/There_are_unknown_unknowns"}],"blog_id":"ak4s224","blog":{"id":"ak4s224","title":"Critical
|
358
|
+
Metascience","description":"Critical perspectives in metascience and the philosophy
|
359
|
+
of science relating to open science, science reform, and the replication crisis.","language":"en","favicon":"https://substackcdn.com/image/fetch/w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00187507-ddc5-4baa-9dae-79f9487d8bce_325x325.png","feed_url":"https://markrubin.substack.com/feed","home_page_url":"https://markrubin.substack.com","user_id":"8498eaf6-8c58-4b58-bc15-27eda292b1aa","created_at":"2023-05-31T13:17:43+00:00","indexed_at":"2023-04-20","feed_format":"application/rss+xml","license":"https://creativecommons.org/licenses/by/4.0/legalcode","generator":"Substack","category":"Social
|
360
|
+
Sciences","prefix":"10.59350","modified_at":"2023-06-13T00:05:19+00:00","version":"https://jsonfeed.org/version/1.1","backlog":false,"authors":null}}'
|
361
|
+
recorded_at: Sat, 17 Jun 2023 18:41:34 GMT
|
362
|
+
- request:
|
363
|
+
method: head
|
364
|
+
uri: https://doi.org/10.3386/w27250
|
365
|
+
body:
|
366
|
+
encoding: UTF-8
|
367
|
+
string: ''
|
368
|
+
headers:
|
369
|
+
Connection:
|
370
|
+
- close
|
371
|
+
Host:
|
372
|
+
- doi.org
|
373
|
+
User-Agent:
|
374
|
+
- http.rb/5.1.1
|
375
|
+
response:
|
376
|
+
status:
|
377
|
+
code: 302
|
378
|
+
message: Found
|
379
|
+
headers:
|
380
|
+
Date:
|
381
|
+
- Sat, 17 Jun 2023 18:41:34 GMT
|
382
|
+
Content-Type:
|
383
|
+
- text/html;charset=utf-8
|
384
|
+
Content-Length:
|
385
|
+
- '159'
|
386
|
+
Connection:
|
387
|
+
- close
|
388
|
+
Location:
|
389
|
+
- http://www.nber.org/papers/w27250.pdf
|
390
|
+
Vary:
|
391
|
+
- Accept
|
392
|
+
Expires:
|
393
|
+
- Sat, 17 Jun 2023 19:25:38 GMT
|
394
|
+
Permissions-Policy:
|
395
|
+
- interest-cohort=(),browsing-topics=()
|
396
|
+
Cf-Cache-Status:
|
397
|
+
- DYNAMIC
|
398
|
+
Report-To:
|
399
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=EWhxgjgJl3PUerDbRr7OO1qhOZSueg79GjMlgwKu1drNRSLaGttbPEYEAvJn4MnSz4wQCbVYj%2FUmOSfHRFv1y4HWLvgoWT%2FX9pR76rXomymQsDrfbyr5vwiDAhTq1zicjtT1EMY%3D"}],"group":"cf-nel","max_age":604800}'
|
400
|
+
Nel:
|
401
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
402
|
+
Strict-Transport-Security:
|
403
|
+
- max-age=31536000; includeSubDomains; preload
|
404
|
+
Server:
|
405
|
+
- cloudflare
|
406
|
+
Cf-Ray:
|
407
|
+
- 7d8d6d70fcadb766-AMS
|
408
|
+
Alt-Svc:
|
409
|
+
- h3=":443"; ma=86400
|
410
|
+
body:
|
411
|
+
encoding: UTF-8
|
412
|
+
string: ''
|
413
|
+
recorded_at: Sat, 17 Jun 2023 18:41:34 GMT
|
414
|
+
- request:
|
415
|
+
method: head
|
416
|
+
uri: https://doi.org/10.1371/journal.pbio.3000937
|
417
|
+
body:
|
418
|
+
encoding: UTF-8
|
419
|
+
string: ''
|
420
|
+
headers:
|
421
|
+
Connection:
|
422
|
+
- close
|
423
|
+
Host:
|
424
|
+
- doi.org
|
425
|
+
User-Agent:
|
426
|
+
- http.rb/5.1.1
|
427
|
+
response:
|
428
|
+
status:
|
429
|
+
code: 302
|
430
|
+
message: Found
|
431
|
+
headers:
|
432
|
+
Date:
|
433
|
+
- Sat, 17 Jun 2023 18:41:35 GMT
|
434
|
+
Content-Type:
|
435
|
+
- text/html;charset=utf-8
|
436
|
+
Content-Length:
|
437
|
+
- '181'
|
438
|
+
Connection:
|
439
|
+
- close
|
440
|
+
Location:
|
441
|
+
- https://dx.plos.org/10.1371/journal.pbio.3000937
|
442
|
+
Vary:
|
443
|
+
- Accept
|
444
|
+
Expires:
|
445
|
+
- Sat, 17 Jun 2023 19:02:36 GMT
|
446
|
+
Permissions-Policy:
|
447
|
+
- interest-cohort=(),browsing-topics=()
|
448
|
+
Cf-Cache-Status:
|
449
|
+
- DYNAMIC
|
450
|
+
Report-To:
|
451
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=iRru2zcd7UhPJ06yLPqA38qUn9nYjVed%2Bq%2FFMunpYH1Z%2FmW35T3pNvHICiLZxfO9AxDUicMm9b82ntx3T3M2GCOR1%2Bz00LiQBpK4hVaFCqBt53UCWZpm9ZiyTiiAzoNtrm1zuvk%3D"}],"group":"cf-nel","max_age":604800}'
|
452
|
+
Nel:
|
453
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
454
|
+
Strict-Transport-Security:
|
455
|
+
- max-age=31536000; includeSubDomains; preload
|
456
|
+
Server:
|
457
|
+
- cloudflare
|
458
|
+
Cf-Ray:
|
459
|
+
- 7d8d6d73ffe20b66-AMS
|
460
|
+
Alt-Svc:
|
461
|
+
- h3=":443"; ma=86400
|
462
|
+
body:
|
463
|
+
encoding: UTF-8
|
464
|
+
string: ''
|
465
|
+
recorded_at: Sat, 17 Jun 2023 18:41:35 GMT
|
466
|
+
- request:
|
467
|
+
method: head
|
468
|
+
uri: https://doi.org/10.1098/rsos.211037
|
469
|
+
body:
|
470
|
+
encoding: UTF-8
|
471
|
+
string: ''
|
472
|
+
headers:
|
473
|
+
Connection:
|
474
|
+
- close
|
475
|
+
Host:
|
476
|
+
- doi.org
|
477
|
+
User-Agent:
|
478
|
+
- http.rb/5.1.1
|
479
|
+
response:
|
480
|
+
status:
|
481
|
+
code: 302
|
482
|
+
message: Found
|
483
|
+
headers:
|
484
|
+
Date:
|
485
|
+
- Sat, 17 Jun 2023 18:41:35 GMT
|
486
|
+
Content-Type:
|
487
|
+
- text/html;charset=utf-8
|
488
|
+
Content-Length:
|
489
|
+
- '201'
|
490
|
+
Connection:
|
491
|
+
- close
|
492
|
+
Location:
|
493
|
+
- https://royalsocietypublishing.org/doi/10.1098/rsos.211037
|
494
|
+
Vary:
|
495
|
+
- Accept
|
496
|
+
Expires:
|
497
|
+
- Sat, 17 Jun 2023 19:02:36 GMT
|
498
|
+
Permissions-Policy:
|
499
|
+
- interest-cohort=(),browsing-topics=()
|
500
|
+
Cf-Cache-Status:
|
501
|
+
- DYNAMIC
|
502
|
+
Report-To:
|
503
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=RWbgxwj2A2bcwtg4uy7Fy27CUHKGZY27tGqQ9qhAhm8c85yCJUpdIeyEYTokvnLfE%2BWkBznZyXVBzKQjOqfpSEN8%2B%2FOiKHbHDR7l7dZtR9qZ6uKBvTuPD2DlIZ8zaBA6MUC6HS0%3D"}],"group":"cf-nel","max_age":604800}'
|
504
|
+
Nel:
|
505
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
506
|
+
Strict-Transport-Security:
|
507
|
+
- max-age=31536000; includeSubDomains; preload
|
508
|
+
Server:
|
509
|
+
- cloudflare
|
510
|
+
Cf-Ray:
|
511
|
+
- 7d8d6d761bd0b70a-AMS
|
512
|
+
Alt-Svc:
|
513
|
+
- h3=":443"; ma=86400
|
514
|
+
body:
|
515
|
+
encoding: UTF-8
|
516
|
+
string: ''
|
517
|
+
recorded_at: Sat, 17 Jun 2023 18:41:35 GMT
|
518
|
+
- request:
|
519
|
+
method: head
|
520
|
+
uri: https://doi.org/10.7554/elife.67995
|
521
|
+
body:
|
522
|
+
encoding: UTF-8
|
523
|
+
string: ''
|
524
|
+
headers:
|
525
|
+
Connection:
|
526
|
+
- close
|
527
|
+
Host:
|
528
|
+
- doi.org
|
529
|
+
User-Agent:
|
530
|
+
- http.rb/5.1.1
|
531
|
+
response:
|
532
|
+
status:
|
533
|
+
code: 302
|
534
|
+
message: Found
|
535
|
+
headers:
|
536
|
+
Date:
|
537
|
+
- Sat, 17 Jun 2023 18:41:36 GMT
|
538
|
+
Content-Type:
|
539
|
+
- text/html;charset=utf-8
|
540
|
+
Content-Length:
|
541
|
+
- '165'
|
542
|
+
Connection:
|
543
|
+
- close
|
544
|
+
Location:
|
545
|
+
- https://elifesciences.org/articles/67995
|
546
|
+
Vary:
|
547
|
+
- Accept
|
548
|
+
Expires:
|
549
|
+
- Sat, 17 Jun 2023 19:08:58 GMT
|
550
|
+
Permissions-Policy:
|
551
|
+
- interest-cohort=(),browsing-topics=()
|
552
|
+
Cf-Cache-Status:
|
553
|
+
- DYNAMIC
|
554
|
+
Report-To:
|
555
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=N5zQd96CBBD%2BWPlDaStK%2Fqn19SdAqs9OoBmFMS%2F%2FjcMq9XpXORJJ6RhvnK22Mnd2lQipWVf%2F8zOUdKpCHKps8d4GZ9iXjD%2BkCp1WqtOzyOQ9kJAffP0NLpRbH4jqtHmRE%2BCiD1M%3D"}],"group":"cf-nel","max_age":604800}'
|
556
|
+
Nel:
|
557
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
558
|
+
Strict-Transport-Security:
|
559
|
+
- max-age=31536000; includeSubDomains; preload
|
560
|
+
Server:
|
561
|
+
- cloudflare
|
562
|
+
Cf-Ray:
|
563
|
+
- 7d8d6d781c55b712-AMS
|
564
|
+
Alt-Svc:
|
565
|
+
- h3=":443"; ma=86400
|
566
|
+
body:
|
567
|
+
encoding: UTF-8
|
568
|
+
string: ''
|
569
|
+
recorded_at: Sat, 17 Jun 2023 18:41:36 GMT
|
570
|
+
- request:
|
571
|
+
method: head
|
572
|
+
uri: https://doi.org/10.31234/osf.io/5vfq6
|
573
|
+
body:
|
574
|
+
encoding: UTF-8
|
575
|
+
string: ''
|
576
|
+
headers:
|
577
|
+
Connection:
|
578
|
+
- close
|
579
|
+
Host:
|
580
|
+
- doi.org
|
581
|
+
User-Agent:
|
582
|
+
- http.rb/5.1.1
|
583
|
+
response:
|
584
|
+
status:
|
585
|
+
code: 302
|
586
|
+
message: Found
|
587
|
+
headers:
|
588
|
+
Date:
|
589
|
+
- Sat, 17 Jun 2023 18:41:36 GMT
|
590
|
+
Content-Type:
|
591
|
+
- text/html;charset=utf-8
|
592
|
+
Content-Length:
|
593
|
+
- '125'
|
594
|
+
Connection:
|
595
|
+
- close
|
596
|
+
Location:
|
597
|
+
- https://osf.io/5vfq6
|
598
|
+
Vary:
|
599
|
+
- Accept
|
600
|
+
Expires:
|
601
|
+
- Sat, 17 Jun 2023 19:02:36 GMT
|
602
|
+
Permissions-Policy:
|
603
|
+
- interest-cohort=(),browsing-topics=()
|
604
|
+
Cf-Cache-Status:
|
605
|
+
- DYNAMIC
|
606
|
+
Report-To:
|
607
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=gAqOFrW7hd%2ByuiagvmlIuh7QAIHIlTv5o7k6J%2FuFoTjYCnwDUVwXSwPssDDcLoGXcms%2BhVcNvvZvrhbmz4MluP5Fg2UA5OpA83LqTZZHcmhEP3G2PP0lPk3xVPVQhUOSdIPA5eI%3D"}],"group":"cf-nel","max_age":604800}'
|
608
|
+
Nel:
|
609
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
610
|
+
Strict-Transport-Security:
|
611
|
+
- max-age=31536000; includeSubDomains; preload
|
612
|
+
Server:
|
613
|
+
- cloudflare
|
614
|
+
Cf-Ray:
|
615
|
+
- 7d8d6d79deb1b98c-AMS
|
616
|
+
Alt-Svc:
|
617
|
+
- h3=":443"; ma=86400
|
618
|
+
body:
|
619
|
+
encoding: UTF-8
|
620
|
+
string: ''
|
621
|
+
recorded_at: Sat, 17 Jun 2023 18:41:36 GMT
|
622
|
+
- request:
|
623
|
+
method: head
|
624
|
+
uri: https://latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html
|
625
|
+
body:
|
626
|
+
encoding: UTF-8
|
627
|
+
string: ''
|
628
|
+
headers:
|
629
|
+
Connection:
|
630
|
+
- close
|
631
|
+
Host:
|
632
|
+
- latimes.com
|
633
|
+
User-Agent:
|
634
|
+
- http.rb/5.1.1
|
635
|
+
response:
|
636
|
+
status:
|
637
|
+
code: 301
|
638
|
+
message: Moved Permanently
|
639
|
+
headers:
|
640
|
+
Server:
|
641
|
+
- awselb/2.0
|
642
|
+
Date:
|
643
|
+
- Sat, 17 Jun 2023 18:41:37 GMT
|
644
|
+
Content-Type:
|
645
|
+
- text/html
|
646
|
+
Content-Length:
|
647
|
+
- '134'
|
648
|
+
Connection:
|
649
|
+
- close
|
650
|
+
Location:
|
651
|
+
- https://www.latimes.com:443/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html
|
652
|
+
body:
|
653
|
+
encoding: ASCII-8BIT
|
654
|
+
string: ''
|
655
|
+
recorded_at: Sat, 17 Jun 2023 18:41:37 GMT
|
656
|
+
- request:
|
657
|
+
method: head
|
658
|
+
uri: https://psyarxiv.com/nj4es
|
659
|
+
body:
|
660
|
+
encoding: UTF-8
|
661
|
+
string: ''
|
662
|
+
headers:
|
663
|
+
Connection:
|
664
|
+
- close
|
665
|
+
Host:
|
666
|
+
- psyarxiv.com
|
667
|
+
User-Agent:
|
668
|
+
- http.rb/5.1.1
|
669
|
+
response:
|
670
|
+
status:
|
671
|
+
code: 200
|
672
|
+
message: OK
|
673
|
+
headers:
|
674
|
+
Date:
|
675
|
+
- Sat, 17 Jun 2023 18:41:37 GMT
|
676
|
+
Content-Type:
|
677
|
+
- text/html
|
678
|
+
Content-Length:
|
679
|
+
- '7071'
|
680
|
+
Connection:
|
681
|
+
- close
|
682
|
+
Vary:
|
683
|
+
- Accept-Encoding
|
684
|
+
- Accept-Encoding
|
685
|
+
Last-Modified:
|
686
|
+
- Tue, 31 Jan 2023 02:21:53 GMT
|
687
|
+
Etag:
|
688
|
+
- '"63d87b41-1b9f"'
|
689
|
+
Cache-Control:
|
690
|
+
- no-cache, no-store, max-age=0, must-revalidate
|
691
|
+
Expires:
|
692
|
+
- Mon, 01 Jan 1990 00:00:00 GMT
|
693
|
+
Pragma:
|
694
|
+
- no-cache
|
695
|
+
Accept-Ranges:
|
696
|
+
- bytes
|
697
|
+
Strict-Transport-Security:
|
698
|
+
- max-age=15724800
|
699
|
+
body:
|
700
|
+
encoding: ASCII-8BIT
|
701
|
+
string: ''
|
702
|
+
recorded_at: Sat, 17 Jun 2023 18:41:37 GMT
|
703
|
+
- request:
|
704
|
+
method: head
|
705
|
+
uri: https://doi.org/10.1038/s41562-021-01203-8
|
706
|
+
body:
|
707
|
+
encoding: UTF-8
|
708
|
+
string: ''
|
709
|
+
headers:
|
710
|
+
Connection:
|
711
|
+
- close
|
712
|
+
Host:
|
713
|
+
- doi.org
|
714
|
+
User-Agent:
|
715
|
+
- http.rb/5.1.1
|
716
|
+
response:
|
717
|
+
status:
|
718
|
+
code: 302
|
719
|
+
message: Found
|
720
|
+
headers:
|
721
|
+
Date:
|
722
|
+
- Sat, 17 Jun 2023 18:41:37 GMT
|
723
|
+
Content-Type:
|
724
|
+
- text/html;charset=utf-8
|
725
|
+
Content-Length:
|
726
|
+
- '185'
|
727
|
+
Connection:
|
728
|
+
- close
|
729
|
+
Location:
|
730
|
+
- https://www.nature.com/articles/s41562-021-01203-8
|
731
|
+
Vary:
|
732
|
+
- Accept
|
733
|
+
Expires:
|
734
|
+
- Sat, 17 Jun 2023 19:02:36 GMT
|
735
|
+
Permissions-Policy:
|
736
|
+
- interest-cohort=(),browsing-topics=()
|
737
|
+
Cf-Cache-Status:
|
738
|
+
- DYNAMIC
|
739
|
+
Report-To:
|
740
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Xe%2F8CHm1yrlPfvNFFzg0iYtFU3JHTHh5k%2BAt%2BICEzVPzNx3oEMjrc3a%2BqHt57QZYKTpwPwxmISjbfIoBPtC5FMGl0vES8dvZ5oOa%2BY69ZZkQXhu3vc%2Fj0R866xdJkKQKYl%2BvtKU%3D"}],"group":"cf-nel","max_age":604800}'
|
741
|
+
Nel:
|
742
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
743
|
+
Strict-Transport-Security:
|
744
|
+
- max-age=31536000; includeSubDomains; preload
|
745
|
+
Server:
|
746
|
+
- cloudflare
|
747
|
+
Cf-Ray:
|
748
|
+
- 7d8d6d82d9edb948-AMS
|
749
|
+
Alt-Svc:
|
750
|
+
- h3=":443"; ma=86400
|
751
|
+
body:
|
752
|
+
encoding: UTF-8
|
753
|
+
string: ''
|
754
|
+
recorded_at: Sat, 17 Jun 2023 18:41:37 GMT
|
755
|
+
- request:
|
756
|
+
method: head
|
757
|
+
uri: https://doi.org/10.1177/0956797611430953
|
758
|
+
body:
|
759
|
+
encoding: UTF-8
|
760
|
+
string: ''
|
761
|
+
headers:
|
762
|
+
Connection:
|
763
|
+
- close
|
764
|
+
Host:
|
765
|
+
- doi.org
|
766
|
+
User-Agent:
|
767
|
+
- http.rb/5.1.1
|
768
|
+
response:
|
769
|
+
status:
|
770
|
+
code: 302
|
771
|
+
message: Found
|
772
|
+
headers:
|
773
|
+
Date:
|
774
|
+
- Sat, 17 Jun 2023 18:41:38 GMT
|
775
|
+
Content-Type:
|
776
|
+
- text/html;charset=utf-8
|
777
|
+
Content-Length:
|
778
|
+
- '197'
|
779
|
+
Connection:
|
780
|
+
- close
|
781
|
+
Location:
|
782
|
+
- http://journals.sagepub.com/doi/10.1177/0956797611430953
|
783
|
+
Vary:
|
784
|
+
- Accept
|
785
|
+
Expires:
|
786
|
+
- Sat, 17 Jun 2023 19:25:38 GMT
|
787
|
+
Permissions-Policy:
|
788
|
+
- interest-cohort=(),browsing-topics=()
|
789
|
+
Cf-Cache-Status:
|
790
|
+
- DYNAMIC
|
791
|
+
Report-To:
|
792
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ixCVGKcIezj219ooTq0tlROIPH9oFHNcqw2aSqI7NNO8g2wuZTtnET8MbNOtnd7dgG9IpDLpdpFQwtCd16FBGSrCp3dq2peBo5p%2F5hUyoVCW4o5F%2FbxaCiOP0%2Ft5TIXh7MNDf7g%3D"}],"group":"cf-nel","max_age":604800}'
|
793
|
+
Nel:
|
794
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
795
|
+
Strict-Transport-Security:
|
796
|
+
- max-age=31536000; includeSubDomains; preload
|
797
|
+
Server:
|
798
|
+
- cloudflare
|
799
|
+
Cf-Ray:
|
800
|
+
- 7d8d6d847b8e0a69-AMS
|
801
|
+
Alt-Svc:
|
802
|
+
- h3=":443"; ma=86400
|
803
|
+
body:
|
804
|
+
encoding: UTF-8
|
805
|
+
string: ''
|
806
|
+
recorded_at: Sat, 17 Jun 2023 18:41:38 GMT
|
807
|
+
- request:
|
808
|
+
method: head
|
809
|
+
uri: https://doi.org/10.1207/s15327965pli0102_1
|
810
|
+
body:
|
811
|
+
encoding: UTF-8
|
812
|
+
string: ''
|
813
|
+
headers:
|
814
|
+
Connection:
|
815
|
+
- close
|
816
|
+
Host:
|
817
|
+
- doi.org
|
818
|
+
User-Agent:
|
819
|
+
- http.rb/5.1.1
|
820
|
+
response:
|
821
|
+
status:
|
822
|
+
code: 302
|
823
|
+
message: Found
|
824
|
+
headers:
|
825
|
+
Date:
|
826
|
+
- Sat, 17 Jun 2023 18:41:38 GMT
|
827
|
+
Content-Type:
|
828
|
+
- text/html;charset=utf-8
|
829
|
+
Content-Length:
|
830
|
+
- '207'
|
831
|
+
Connection:
|
832
|
+
- close
|
833
|
+
Location:
|
834
|
+
- http://www.tandfonline.com/doi/abs/10.1207/s15327965pli0102_1
|
835
|
+
Vary:
|
836
|
+
- Accept
|
837
|
+
Expires:
|
838
|
+
- Sat, 17 Jun 2023 19:01:56 GMT
|
839
|
+
Permissions-Policy:
|
840
|
+
- interest-cohort=(),browsing-topics=()
|
841
|
+
Cf-Cache-Status:
|
842
|
+
- DYNAMIC
|
843
|
+
Report-To:
|
844
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=3bWtC%2B%2BNnqDew7flDKXihyxWrhizHSy24uwrRoYbJ7duL%2BTG1Fr8qLY%2BurJalCccYCUsr%2BJSJ9Y2zCxtDP9374W1teiSAOI07FoOmVTzDF3L9Uvu4xEvcdw%2FGz5KtWLGnPISBmY%3D"}],"group":"cf-nel","max_age":604800}'
|
845
|
+
Nel:
|
846
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
847
|
+
Strict-Transport-Security:
|
848
|
+
- max-age=31536000; includeSubDomains; preload
|
849
|
+
Server:
|
850
|
+
- cloudflare
|
851
|
+
Cf-Ray:
|
852
|
+
- 7d8d6d861bba0bb0-AMS
|
853
|
+
Alt-Svc:
|
854
|
+
- h3=":443"; ma=86400
|
855
|
+
body:
|
856
|
+
encoding: UTF-8
|
857
|
+
string: ''
|
858
|
+
recorded_at: Sat, 17 Jun 2023 18:41:38 GMT
|
859
|
+
- request:
|
860
|
+
method: head
|
861
|
+
uri: https://doi.org/10.31234/osf.io/wxn58
|
862
|
+
body:
|
863
|
+
encoding: UTF-8
|
864
|
+
string: ''
|
865
|
+
headers:
|
866
|
+
Connection:
|
867
|
+
- close
|
868
|
+
Host:
|
869
|
+
- doi.org
|
870
|
+
User-Agent:
|
871
|
+
- http.rb/5.1.1
|
872
|
+
response:
|
873
|
+
status:
|
874
|
+
code: 302
|
875
|
+
message: Found
|
876
|
+
headers:
|
877
|
+
Date:
|
878
|
+
- Sat, 17 Jun 2023 18:41:38 GMT
|
879
|
+
Content-Type:
|
880
|
+
- text/html;charset=utf-8
|
881
|
+
Content-Length:
|
882
|
+
- '125'
|
883
|
+
Connection:
|
884
|
+
- close
|
885
|
+
Location:
|
886
|
+
- https://osf.io/wxn58
|
887
|
+
Vary:
|
888
|
+
- Accept
|
889
|
+
Expires:
|
890
|
+
- Sat, 17 Jun 2023 19:02:36 GMT
|
891
|
+
Permissions-Policy:
|
892
|
+
- interest-cohort=(),browsing-topics=()
|
893
|
+
Cf-Cache-Status:
|
894
|
+
- DYNAMIC
|
895
|
+
Report-To:
|
896
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2TPXD2bsYU1UjopKx%2F2TXH%2FmL8D%2FK2Dodvdq5U8OM0MC7wXgezbo%2FuQsE5kvyQXmNrqFkIBJ9%2Br%2ByjG0GbzoI%2FYv98alebFE7I9jIlC3vVnQ0VD%2FioalkmdBqTzWa6uZgGoKkMc%3D"}],"group":"cf-nel","max_age":604800}'
|
897
|
+
Nel:
|
898
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
899
|
+
Strict-Transport-Security:
|
900
|
+
- max-age=31536000; includeSubDomains; preload
|
901
|
+
Server:
|
902
|
+
- cloudflare
|
903
|
+
Cf-Ray:
|
904
|
+
- 7d8d6d884d5a419c-AMS
|
905
|
+
Alt-Svc:
|
906
|
+
- h3=":443"; ma=86400
|
907
|
+
body:
|
908
|
+
encoding: UTF-8
|
909
|
+
string: ''
|
910
|
+
recorded_at: Sat, 17 Jun 2023 18:41:38 GMT
|
911
|
+
- request:
|
912
|
+
method: head
|
913
|
+
uri: https://doi.org/10.1016/j.tics.2019.07.009
|
914
|
+
body:
|
915
|
+
encoding: UTF-8
|
916
|
+
string: ''
|
917
|
+
headers:
|
918
|
+
Connection:
|
919
|
+
- close
|
920
|
+
Host:
|
921
|
+
- doi.org
|
922
|
+
User-Agent:
|
923
|
+
- http.rb/5.1.1
|
924
|
+
response:
|
925
|
+
status:
|
926
|
+
code: 302
|
927
|
+
message: Found
|
928
|
+
headers:
|
929
|
+
Date:
|
930
|
+
- Sat, 17 Jun 2023 18:41:39 GMT
|
931
|
+
Content-Type:
|
932
|
+
- text/html;charset=utf-8
|
933
|
+
Content-Length:
|
934
|
+
- '209'
|
935
|
+
Connection:
|
936
|
+
- close
|
937
|
+
Location:
|
938
|
+
- https://linkinghub.elsevier.com/retrieve/pii/S1364661319301846
|
939
|
+
Vary:
|
940
|
+
- Accept
|
941
|
+
Expires:
|
942
|
+
- Sat, 17 Jun 2023 19:25:38 GMT
|
943
|
+
Link:
|
944
|
+
- "<https://dul.usage.elsevier.com/doi/>; rel=dul"
|
945
|
+
- "<https://dul.usage.elsevier.com/doi/>; rel=dul"
|
946
|
+
Permissions-Policy:
|
947
|
+
- interest-cohort=(),browsing-topics=()
|
948
|
+
Cf-Cache-Status:
|
949
|
+
- DYNAMIC
|
950
|
+
Report-To:
|
951
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=qyyBLDFT8X4AbHCnD%2Bi6AR0VlQYEVg%2BTMypMOTu%2FZqV8GZNkL4BOzk%2F61b5i%2FdEBChItSVT2EX1dt4ShUKc5WGRisxVt9SIW6SHJ5I2bN%2BolhjoRa6OCJspBefidTvSHxRYEoEU%3D"}],"group":"cf-nel","max_age":604800}'
|
952
|
+
Nel:
|
953
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
954
|
+
Strict-Transport-Security:
|
955
|
+
- max-age=31536000; includeSubDomains; preload
|
956
|
+
Server:
|
957
|
+
- cloudflare
|
958
|
+
Cf-Ray:
|
959
|
+
- 7d8d6d8a5d200bb4-AMS
|
960
|
+
Alt-Svc:
|
961
|
+
- h3=":443"; ma=86400
|
962
|
+
body:
|
963
|
+
encoding: UTF-8
|
964
|
+
string: ''
|
965
|
+
recorded_at: Sat, 17 Jun 2023 18:41:38 GMT
|
966
|
+
- request:
|
967
|
+
method: head
|
968
|
+
uri: https://doi.org/10.1525/collabra.24091
|
969
|
+
body:
|
970
|
+
encoding: UTF-8
|
971
|
+
string: ''
|
972
|
+
headers:
|
973
|
+
Connection:
|
974
|
+
- close
|
975
|
+
Host:
|
976
|
+
- doi.org
|
977
|
+
User-Agent:
|
978
|
+
- http.rb/5.1.1
|
979
|
+
response:
|
980
|
+
status:
|
981
|
+
code: 302
|
982
|
+
message: Found
|
983
|
+
headers:
|
984
|
+
Date:
|
985
|
+
- Sat, 17 Jun 2023 18:41:39 GMT
|
986
|
+
Content-Type:
|
987
|
+
- text/html;charset=utf-8
|
988
|
+
Content-Length:
|
989
|
+
- '295'
|
990
|
+
Connection:
|
991
|
+
- close
|
992
|
+
Location:
|
993
|
+
- https://online.ucpress.edu/collabra/article/7/1/24091/117222/A-Call-for-Greater-Modesty-in-Psychology-and
|
994
|
+
Vary:
|
995
|
+
- Accept
|
996
|
+
Expires:
|
997
|
+
- Sat, 17 Jun 2023 19:02:36 GMT
|
998
|
+
Permissions-Policy:
|
999
|
+
- interest-cohort=(),browsing-topics=()
|
1000
|
+
Cf-Cache-Status:
|
1001
|
+
- DYNAMIC
|
1002
|
+
Report-To:
|
1003
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=M2wqLtFVXNf0X%2FqVPqCsfQLQfw8Vkybl9rKOh5tKFDmo6DmS1BlsntJbiAq%2F2sj1MHZbinwTdoulx4j7HpVaB7o7yYuNo9Yp%2Fm4AITNsZ%2FEDHCHVJdW9mB35DNJ4oKVQydIuNxQ%3D"}],"group":"cf-nel","max_age":604800}'
|
1004
|
+
Nel:
|
1005
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
1006
|
+
Strict-Transport-Security:
|
1007
|
+
- max-age=31536000; includeSubDomains; preload
|
1008
|
+
Server:
|
1009
|
+
- cloudflare
|
1010
|
+
Cf-Ray:
|
1011
|
+
- 7d8d6d8bed2ab8eb-AMS
|
1012
|
+
Alt-Svc:
|
1013
|
+
- h3=":443"; ma=86400
|
1014
|
+
body:
|
1015
|
+
encoding: UTF-8
|
1016
|
+
string: ''
|
1017
|
+
recorded_at: Sat, 17 Jun 2023 18:41:39 GMT
|
1018
|
+
- request:
|
1019
|
+
method: head
|
1020
|
+
uri: https://doi.org/10.1177%20/0956797611417632
|
1021
|
+
body:
|
1022
|
+
encoding: UTF-8
|
1023
|
+
string: ''
|
1024
|
+
headers:
|
1025
|
+
Connection:
|
1026
|
+
- close
|
1027
|
+
Host:
|
1028
|
+
- doi.org
|
1029
|
+
User-Agent:
|
1030
|
+
- http.rb/5.1.1
|
1031
|
+
response:
|
1032
|
+
status:
|
1033
|
+
code: 404
|
1034
|
+
message: Not Found
|
1035
|
+
headers:
|
1036
|
+
Date:
|
1037
|
+
- Sat, 17 Jun 2023 18:41:39 GMT
|
1038
|
+
Content-Type:
|
1039
|
+
- text/html;charset=utf-8
|
1040
|
+
Connection:
|
1041
|
+
- close
|
1042
|
+
Cf-Cache-Status:
|
1043
|
+
- DYNAMIC
|
1044
|
+
Report-To:
|
1045
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Mn5%2FHChRZezC4CefVOmgiPm4t4KTPaAdK9BmYQ0MXcOwhFkQs5LJEdDsDOZJMTilfqgnAjXv%2B53aHtbD80qu8ZcF%2Bkp3wZ3NkUVm%2BiwupwKztXJpfauDrJbcQt9B5ZAWUVckRrI%3D"}],"group":"cf-nel","max_age":604800}'
|
1046
|
+
Nel:
|
1047
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
1048
|
+
Strict-Transport-Security:
|
1049
|
+
- max-age=31536000; includeSubDomains; preload
|
1050
|
+
Server:
|
1051
|
+
- cloudflare
|
1052
|
+
Cf-Ray:
|
1053
|
+
- 7d8d6d8daf49b7f2-AMS
|
1054
|
+
Alt-Svc:
|
1055
|
+
- h3=":443"; ma=86400
|
1056
|
+
body:
|
1057
|
+
encoding: UTF-8
|
1058
|
+
string: ''
|
1059
|
+
recorded_at: Sat, 17 Jun 2023 18:41:39 GMT
|
1060
|
+
- request:
|
1061
|
+
method: head
|
1062
|
+
uri: https://doi.org/10.1177
|
1063
|
+
body:
|
1064
|
+
encoding: UTF-8
|
1065
|
+
string: ''
|
1066
|
+
headers:
|
1067
|
+
Connection:
|
1068
|
+
- close
|
1069
|
+
Host:
|
1070
|
+
- doi.org
|
1071
|
+
User-Agent:
|
1072
|
+
- http.rb/5.1.1
|
1073
|
+
response:
|
1074
|
+
status:
|
1075
|
+
code: 200
|
1076
|
+
message: OK
|
1077
|
+
headers:
|
1078
|
+
Date:
|
1079
|
+
- Sat, 17 Jun 2023 18:41:39 GMT
|
1080
|
+
Content-Type:
|
1081
|
+
- text/html;charset=utf-8
|
1082
|
+
Connection:
|
1083
|
+
- close
|
1084
|
+
Permissions-Policy:
|
1085
|
+
- interest-cohort=(),browsing-topics=()
|
1086
|
+
Cf-Cache-Status:
|
1087
|
+
- DYNAMIC
|
1088
|
+
Report-To:
|
1089
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=xdpZQV9oyHMU0AcgxEb7%2Bo3tZmHCXGK7mbl5pOVuLwOnHCbgg%2FbJDNqvFtMNVcryoqOdb3D1LNEzDRQD9uxG45tpjfgOkn2I1tYU8gv23U7EXB6TkKVjqnFrC5eqfbDbFVb91zo%3D"}],"group":"cf-nel","max_age":604800}'
|
1090
|
+
Nel:
|
1091
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
1092
|
+
Strict-Transport-Security:
|
1093
|
+
- max-age=31536000; includeSubDomains; preload
|
1094
|
+
Server:
|
1095
|
+
- cloudflare
|
1096
|
+
Cf-Ray:
|
1097
|
+
- 7d8d6d8fff28b950-AMS
|
1098
|
+
Alt-Svc:
|
1099
|
+
- h3=":443"; ma=86400
|
1100
|
+
body:
|
1101
|
+
encoding: UTF-8
|
1102
|
+
string: ''
|
1103
|
+
recorded_at: Sat, 17 Jun 2023 18:41:39 GMT
|
1104
|
+
- request:
|
1105
|
+
method: head
|
1106
|
+
uri: https://doi.org/10.31222/osf.io/h8xjw
|
1107
|
+
body:
|
1108
|
+
encoding: UTF-8
|
1109
|
+
string: ''
|
1110
|
+
headers:
|
1111
|
+
Connection:
|
1112
|
+
- close
|
1113
|
+
Host:
|
1114
|
+
- doi.org
|
1115
|
+
User-Agent:
|
1116
|
+
- http.rb/5.1.1
|
1117
|
+
response:
|
1118
|
+
status:
|
1119
|
+
code: 302
|
1120
|
+
message: Found
|
1121
|
+
headers:
|
1122
|
+
Date:
|
1123
|
+
- Sat, 17 Jun 2023 18:41:40 GMT
|
1124
|
+
Content-Type:
|
1125
|
+
- text/html;charset=utf-8
|
1126
|
+
Content-Length:
|
1127
|
+
- '125'
|
1128
|
+
Connection:
|
1129
|
+
- close
|
1130
|
+
Location:
|
1131
|
+
- https://osf.io/h8xjw
|
1132
|
+
Vary:
|
1133
|
+
- Accept
|
1134
|
+
Expires:
|
1135
|
+
- Sat, 17 Jun 2023 19:01:56 GMT
|
1136
|
+
Permissions-Policy:
|
1137
|
+
- interest-cohort=(),browsing-topics=()
|
1138
|
+
Cf-Cache-Status:
|
1139
|
+
- DYNAMIC
|
1140
|
+
Report-To:
|
1141
|
+
- '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=BpUXXI0AvLIgHqg6jEVdl3W7THNsBtbE7kOgdgMUTeEmvL9oYDNjW2kRE%2B3rVF0NGLRmmH6IHKijf3Tqq833jKWEtElD9MLfG9PudbNp6kh164sO%2FTxK0eAcLL5r6jAX4lN4ZoM%3D"}],"group":"cf-nel","max_age":604800}'
|
1142
|
+
Nel:
|
1143
|
+
- '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'
|
1144
|
+
Strict-Transport-Security:
|
1145
|
+
- max-age=31536000; includeSubDomains; preload
|
1146
|
+
Server:
|
1147
|
+
- cloudflare
|
1148
|
+
Cf-Ray:
|
1149
|
+
- 7d8d6d91dfdb0ea9-AMS
|
1150
|
+
Alt-Svc:
|
1151
|
+
- h3=":443"; ma=86400
|
1152
|
+
body:
|
1153
|
+
encoding: UTF-8
|
1154
|
+
string: ''
|
1155
|
+
recorded_at: Sat, 17 Jun 2023 18:41:40 GMT
|
1156
|
+
- request:
|
1157
|
+
method: head
|
1158
|
+
uri: https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice
|
1159
|
+
body:
|
1160
|
+
encoding: UTF-8
|
1161
|
+
string: ''
|
1162
|
+
headers:
|
1163
|
+
Connection:
|
1164
|
+
- close
|
1165
|
+
Host:
|
1166
|
+
- metascience.info
|
1167
|
+
User-Agent:
|
1168
|
+
- http.rb/5.1.1
|
1169
|
+
response:
|
1170
|
+
status:
|
1171
|
+
code: 301
|
1172
|
+
message: Moved Permanently
|
1173
|
+
headers:
|
1174
|
+
Date:
|
1175
|
+
- Sat, 17 Jun 2023 18:41:41 GMT
|
1176
|
+
Server:
|
1177
|
+
- Apache
|
1178
|
+
Wpo-Cache-Status:
|
1179
|
+
- not cached
|
1180
|
+
Wpo-Cache-Message:
|
1181
|
+
- The request method was not GET (HEAD)
|
1182
|
+
X-Redirect-By:
|
1183
|
+
- WordPress
|
1184
|
+
Upgrade:
|
1185
|
+
- h2,h2c
|
1186
|
+
Connection:
|
1187
|
+
- Upgrade
|
1188
|
+
Location:
|
1189
|
+
- https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/
|
1190
|
+
Cache-Control:
|
1191
|
+
- max-age=7200
|
1192
|
+
Expires:
|
1193
|
+
- Sat, 17 Jun 2023 20:41:41 GMT
|
1194
|
+
Host-Header:
|
1195
|
+
- c2hhcmVkLmJsdWVob3N0LmNvbQ==
|
1196
|
+
Content-Type:
|
1197
|
+
- text/html; charset=UTF-8
|
1198
|
+
body:
|
1199
|
+
encoding: UTF-8
|
1200
|
+
string: ''
|
1201
|
+
recorded_at: Sat, 17 Jun 2023 18:41:44 GMT
|
1202
|
+
- request:
|
1203
|
+
method: head
|
1204
|
+
uri: https://osf.io/tqwgp
|
1205
|
+
body:
|
1206
|
+
encoding: UTF-8
|
1207
|
+
string: ''
|
1208
|
+
headers:
|
1209
|
+
Connection:
|
1210
|
+
- close
|
1211
|
+
Host:
|
1212
|
+
- osf.io
|
1213
|
+
User-Agent:
|
1214
|
+
- http.rb/5.1.1
|
1215
|
+
response:
|
1216
|
+
status:
|
1217
|
+
code: 308
|
1218
|
+
message: Permanent Redirect
|
1219
|
+
headers:
|
1220
|
+
Date:
|
1221
|
+
- Sat, 17 Jun 2023 18:41:45 GMT
|
1222
|
+
Content-Type:
|
1223
|
+
- text/html; charset=utf-8
|
1224
|
+
Content-Length:
|
1225
|
+
- '249'
|
1226
|
+
Connection:
|
1227
|
+
- close
|
1228
|
+
Location:
|
1229
|
+
- https://osf.io/tqwgp/
|
1230
|
+
X-Frame-Options:
|
1231
|
+
- SAMEORIGIN
|
1232
|
+
X-Cache-Status:
|
1233
|
+
- MISS
|
1234
|
+
Cache-Control:
|
1235
|
+
- no-cache, no-store, max-age=0, must-revalidate
|
1236
|
+
Expires:
|
1237
|
+
- Mon, 01 Jan 1990 00:00:00 GMT
|
1238
|
+
Pragma:
|
1239
|
+
- no-cache
|
1240
|
+
Strict-Transport-Security:
|
1241
|
+
- max-age=15724800
|
1242
|
+
body:
|
1243
|
+
encoding: UTF-8
|
1244
|
+
string: ''
|
1245
|
+
recorded_at: Sat, 17 Jun 2023 18:41:44 GMT
|
1246
|
+
- request:
|
1247
|
+
method: head
|
1248
|
+
uri: https://en.wikipedia.org/wiki/There_are_unknown_unknowns
|
1249
|
+
body:
|
1250
|
+
encoding: UTF-8
|
1251
|
+
string: ''
|
1252
|
+
headers:
|
1253
|
+
Connection:
|
1254
|
+
- close
|
1255
|
+
Host:
|
1256
|
+
- en.wikipedia.org
|
1257
|
+
User-Agent:
|
1258
|
+
- http.rb/5.1.1
|
1259
|
+
response:
|
1260
|
+
status:
|
1261
|
+
code: 200
|
1262
|
+
message: OK
|
1263
|
+
headers:
|
1264
|
+
Date:
|
1265
|
+
- Sat, 17 Jun 2023 08:44:37 GMT
|
1266
|
+
Vary:
|
1267
|
+
- Accept-Encoding,Cookie,Authorization
|
1268
|
+
Server:
|
1269
|
+
- ATS/9.1.4
|
1270
|
+
X-Content-Type-Options:
|
1271
|
+
- nosniff
|
1272
|
+
Content-Language:
|
1273
|
+
- en
|
1274
|
+
Last-Modified:
|
1275
|
+
- Thu, 08 Jun 2023 00:03:49 GMT
|
1276
|
+
Content-Type:
|
1277
|
+
- text/html; charset=UTF-8
|
1278
|
+
Age:
|
1279
|
+
- '35827'
|
1280
|
+
X-Cache:
|
1281
|
+
- cp3056 hit, cp3058 hit/4
|
1282
|
+
X-Cache-Status:
|
1283
|
+
- hit-front
|
1284
|
+
Server-Timing:
|
1285
|
+
- cache;desc="hit-front", host;desc="cp3058"
|
1286
|
+
Strict-Transport-Security:
|
1287
|
+
- max-age=106384710; includeSubDomains; preload
|
1288
|
+
Report-To:
|
1289
|
+
- '{ "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0"
|
1290
|
+
}] }'
|
1291
|
+
Nel:
|
1292
|
+
- '{ "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction":
|
1293
|
+
0.0}'
|
1294
|
+
Set-Cookie:
|
1295
|
+
- GeoIP=DE:NW:Gelsenkirchen:51.51:7.09:v4; Path=/; secure; Domain=.wikipedia.org
|
1296
|
+
- NetworkProbeLimit=0.001;Path=/;Secure;Max-Age=3600
|
1297
|
+
- WMF-DP=f04;Path=/;HttpOnly;secure;Expires=Sun, 18 Jun 2023 00:00:00 GMT
|
1298
|
+
- WMF-Last-Access-Global=17-Jun-2023;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Wed,
|
1299
|
+
19 Jul 2023 12:00:00 GMT
|
1300
|
+
- WMF-Last-Access=17-Jun-2023;Path=/;HttpOnly;secure;Expires=Wed, 19 Jul 2023
|
1301
|
+
12:00:00 GMT
|
1302
|
+
X-Client-Ip:
|
1303
|
+
- 2001:16b8:10fb:d700:90e9:71e3:7ae8:32e8
|
1304
|
+
Cache-Control:
|
1305
|
+
- private, s-maxage=0, max-age=0, must-revalidate
|
1306
|
+
Accept-Ranges:
|
1307
|
+
- bytes
|
1308
|
+
Content-Length:
|
1309
|
+
- '193391'
|
1310
|
+
Connection:
|
1311
|
+
- close
|
1312
|
+
body:
|
1313
|
+
encoding: UTF-8
|
1314
|
+
string: ''
|
1315
|
+
recorded_at: Sat, 17 Jun 2023 18:41:45 GMT
|
1316
|
+
recorded_with: VCR 6.1.0
|