Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
R
Replication of null results - Absence of evidence or evidence of absence
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Issue analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Samuel Pawel
Replication of null results - Absence of evidence or evidence of absence
Commits
e9b2067c
Commit
e9b2067c
authored
1 year ago
by
SamCH93
Browse files
Options
Downloads
Patches
Plain Diff
change prior variance to prior sd parametrization in Box
parent
9e8a93c5
No related branches found
No related tags found
1 merge request
!2
Revision
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
paper/rsabsence.Rnw
+59
-52
59 additions, 52 deletions
paper/rsabsence.Rnw
rsabsence.pdf
+0
-0
0 additions, 0 deletions
rsabsence.pdf
with
59 additions
and
52 deletions
paper/rsabsence.Rnw
+
59
−
52
View file @
e9b2067c
...
@@ -844,7 +844,7 @@ pTOSTa <- function(estimate, se, null = 0, margin) {
...
@@ -844,7 +844,7 @@ pTOSTa <- function(estimate, se, null = 0, margin) {
@
@
<< "pTOST
-
more
-
educational
-
version", eval
=
FALSE, echo
=
TRUE, size
=
"small" >>
=
<< "pTOST
-
more
-
educational
-
version", eval
=
FALSE, echo
=
TRUE, size
=
"small" >>
=
## R function to compute TOST p
-
value based on effect estimate, standard error,
## R function to compute TOST p
-
value based on effect estimate, standard error,
## null value
(
default is
0
)
, equivalence margin
## null value
(
default is
0
)
,
and
equivalence margin
pTOST <
-
function
(
estimate, se, null
=
0
, margin
)
{
pTOST <
-
function
(
estimate, se, null
=
0
, margin
)
{
p
1
<
-
pnorm
(
q
=
(
estimate
-
null
-
margin
)
/
se
)
p
1
<
-
pnorm
(
q
=
(
estimate
-
null
-
margin
)
/
se
)
p
2
<
-
1
-
pnorm
(
q
=
(
estimate
-
null
+
margin
)
/
se
)
p
2
<
-
1
-
pnorm
(
q
=
(
estimate
-
null
+
margin
)
/
se
)
...
@@ -866,8 +866,8 @@ pTOST <- function(estimate, se, null = 0, margin) {
...
@@ -866,8 +866,8 @@ pTOST <- function(estimate, se, null = 0, margin) {
\item
Specify a prior distribution for the effect size
$
\theta
$
that
\item
Specify a prior distribution for the effect size
$
\theta
$
that
represents plausible values under the alternative hypothesis that
represents plausible values under the alternative hypothesis that
there is an effect
(
$
H
_{
1
}
\colon
\theta
\neq
\theta
_{
0
}
)
$
. For
there is an effect
(
$
H
_{
1
}
\colon
\theta
\neq
\theta
_{
0
}
)
$
. For
example, specify the mean
$
m
$
and
variance
$
v
$
of a normal
example, specify the mean
$
m
$
and
standard deviation
$
s
$
of a normal
distribution
$
\theta
\given
H
_{
1
}
\sim
\Nor
(m
,v
)
$
.
distribution
$
\theta
\given
H
_{
1
}
\sim
\Nor
(m
, s
^{
2
}
)
$
.
\item
Compute the Bayes factors contrasting
\item
Compute the Bayes factors contrasting
$
H
_{
0
}
\colon
\theta
=
\theta
_{
0
}$
to
$
H
_{
0
}
\colon
\theta
=
\theta
_{
0
}$
to
$
H
_{
1
}
\colon
\theta
\neq
\theta
_{
0
}$
for original and replication
$
H
_{
1
}
\colon
\theta
\neq
\theta
_{
0
}$
for original and replication
...
@@ -875,25 +875,27 @@ pTOST <- function(estimate, se, null = 0, margin) {
...
@@ -875,25 +875,27 @@ pTOST <- function(estimate, se, null = 0, margin) {
% $\theta \given H_{1} \sim \Nor(m ,v)$,
% $\theta \given H_{1} \sim \Nor(m ,v)$,
the Bayes factor is
the Bayes factor is
$$
\BF
_{
01
,i
}
$$
\BF
_{
01
,i
}
=
\sqrt
{
1
+
\frac
{
v
}{
\sigma
^{
2
}_{
i
}}}
\,
\exp\left
[-
\frac
{
1
}{
2
}
\left\{\frac
{
(
\hat
{
\theta
}_{
i
}
-
=
\sqrt
{
1
+
\frac
{
s
^{
2
}
}{
\sigma
^{
2
}_{
i
}}}
\,
\exp\left
[-
\frac
{
1
}{
2
}
\left\{\frac
{
(
\hat
{
\theta
}_{
i
}
-
\theta
_{
0
}
)
^{
2
}}{
\sigma
^{
2
}_{
i
}}
-
\frac
{
(
\hat
{
\theta
}_{
i
}
-
m
)
^{
2
}}{
\sigma
^{
2
}_{
i
}
+
v
}
\theta
_{
0
}
)
^{
2
}}{
\sigma
^{
2
}_{
i
}}
-
\frac
{
(
\hat
{
\theta
}_{
i
}
-
m
)
^{
2
}}{
\sigma
^{
2
}_{
i
}
+
s
^
2
}
\right\}\right
]
, ~ i
\in
\{
o, r
\}
.
$$
\right\}\right
]
, ~ i
\in
\{
o, r
\}
.
$$
\begin
{
minipage
}
[
c
]
{
0
.
95
\linewidth
}
\begin
{
minipage
}
[
c
]
{
0
.
95
\linewidth
}
<< "BF
01
-
version
-
that
-
we
-
used", eval
=
FALSE, echo
=
FALSE, size
=
"small" >>
=
<< "BF
01
-
version
-
that
-
we
-
used", eval
=
FALSE, echo
=
FALSE, size
=
"small" >>
=
## R function to compute Bayes factor based on effect estimate, standard error,
## R function to compute Bayes factor based on effect estimate, standard error,
## null value
(
default is
0
)
, prior mean
(
default is null value
)
, prior variance
## null value
(
default is
0
)
, prior mean
(
default is null value
)
, and prior
BF
01
a <
-
function
(
estimate, se, null
=
0
, priormean
=
null, priorvar
)
{
## standard deviation
BF
01
a <
-
function
(
estimate, se, null
=
0
, priormean
=
null, priorsd
)
{
f
0
<
-
dnorm
(
x
=
estimate, mean
=
null, sd
=
se
)
f
0
<
-
dnorm
(
x
=
estimate, mean
=
null, sd
=
se
)
f
1
<
-
dnorm
(
x
=
estimate, mean
=
priormean, sd
=
sqrt
(
se
^
2
+
prior
var
))
f
1
<
-
dnorm
(
x
=
estimate, mean
=
priormean, sd
=
sqrt
(
se
^
2
+
prior
sd
^
2
))
return
(
f
0
/
f
1
)
return
(
f
0
/
f
1
)
}
}
@
@
<< "BF
01
-
more
-
educational
-
version", eval
=
FALSE, echo
=
TRUE, size
=
"small" >>
=
<< "BF
01
-
more
-
educational
-
version", eval
=
FALSE, echo
=
TRUE, size
=
"small" >>
=
## R function to compute Bayes factor based on effect estimate, standard error,
## R function to compute Bayes factor based on effect estimate, standard error,
## null value
(
default is
0
)
, prior mean
(
default is null value
)
, prior variance
## null value
(
default is
0
)
, prior mean
(
default is null value
)
, and prior
BF
01
<
-
function
(
estimate, se, null
=
0
, priormean
=
null, priorvar
)
{
## standard deviation
bf <
-
sqrt
(
1
+
priorvar
/
se
^
2
)
*
exp
(-
0
.
5
*
((
estimate
-
null
)
^
2
/
se
^
2
-
BF
01
<
-
function
(
estimate, se, null
=
0
, priormean
=
null, priorsd
)
{
(
estimate
-
priormean
)
^
2
/
(
se
^
2
+
priorvar
)))
bf <
-
sqrt
(
1
+
priorsd
^
2
/
se
^
2
)
*
exp
(-
0
.
5
*
((
estimate
-
null
)
^
2
/
se
^
2
-
(
estimate
-
priormean
)
^
2
/
(
se
^
2
+
priorsd
^
2
)))
return
(
bf
)
return
(
bf
)
}
}
@
@
...
@@ -1019,55 +1021,27 @@ data from the RPCB were obtained by downloading the files from
...
@@ -1019,55 +1021,27 @@ data from the RPCB were obtained by downloading the files from
relevant variables as indicated in the R script
\texttt
{
preprocess
-
rpcb
-
data.R
}
relevant variables as indicated in the R script
\texttt
{
preprocess
-
rpcb
-
data.R
}
which is available in our git repository.
which is available in our git repository.
\bibliography
{
bibliography
}
\section
*
{
Appendix
}
\
sub
section
*
{
Sensitivity analys
i
s
}
\section
*
{
Appendix:
Sensitivity analys
e
s
}
As discussed before, the post
-
hoc specification of equivalence margins
and the
As discussed before, the post
-
hoc specification of equivalence margins
$
\Delta
$
choice of the
prior distribution for the SMD under the alternative
$
H
_{
1
}$
is
and
prior distribution for the SMD under the alternative
$
H
_{
1
}$
is
debatable.
debatable and controversial.
Commonly used margins in clinical research are much
Commonly used margins in clinical research are much
more stringent; for
more stringent; for
instance, in oncology, a margin of
$
\Delta
=
\log
(1.3)
$
is
instance, in oncology, a margin of
$
\Delta
=
\log
(1.3)
$
is
commonly used for log
commonly used for log
odds
/
hazard ratios, whereas in bioequivalence studies a
odds
/
hazard ratios, whereas in bioequivalence studies a
margin of
margin of
\mbox
{$
\Delta
=
\log
(1.25)
% = \Sexpr{round(log(1.25), 2)}
\mbox
{$
\Delta
=
\log
(1.25)
% = \Sexpr{round(log(1.25), 2)}
$}
is the convention
\citep
[
chapter
22
]
{
Senn
2008
}
. These margins would
$}
is the convention
\citep
[
chapter
22
]
{
Senn
2008
}
. These margins would
translate into margins of
$
\Delta
translate into margins of
$
\Delta
=
% \log(1.3)\sqrt{3}/\pi =
=
% \log(1.3)\sqrt{3}/\pi =
\Sexpr
{
round(log(1.3)*sqrt(3)/pi, 2)
}$
and
$
\Delta
=
% \log(1.25)\sqrt{3}/\pi =
\Sexpr
{
round(log(1.3)*sqrt(3)/pi, 2)
}$
and
$
\Delta
=
% \log(1.25)\sqrt{3}/\pi =
\Sexpr
{
round(log(1.25)*sqrt(3)/pi, 2)
}$
on the SMD scale, respectively, using
\Sexpr
{
round(log(1.25)*sqrt(3)/pi, 2)
}$
on the SMD scale, respectively, using
the
$
\text
{
SMD
}
= (
\surd
{
3
}
/
\pi
)
\log\text
{
OR
}$
conversion
\citep
[
p.
the
$
\text
{
SMD
}
= (
\surd
{
3
}
/
\pi
)
\log\text
{
OR
}$
conversion
\citep
[
p.
233
]
{
Cooper
2019
}
.
As
for the Bayesian
hypothesis testing
we specified a normal
233
]
{
Cooper
2019
}
.
Similarly,
for the Bayesian
factor
we specified a normal
unit
-
information prior under the alternative while other normal priors with
unit
-
information prior under the alternative while other normal priors with
smaller
/
larger standard deviations could have been considered.
smaller
/
larger standard deviations could have been considered. Here, we
In the sensitivity analysis in the top plot of Figure~
\ref
{
fig:sensitivity
}
we
therefore investigate the sensitivity of our conclusions with respect to these
show the number of successful replications as a function of the margin
$
\Delta
$
parameters.
and for different TOST
\textit
{
p
}
-
value thresholds. Such an ``equivalence
curve'' approach was first proposed by
\citet
{
Hauck
1986
}
. We see that for
realistic margins between
$
0
$
and
$
1
$
, the proportion of replication successes
remains below
$
50
\%
$
for the conventional
$
\alpha
= 0.05
$
level. To achieve a
success rate of
$
11/15 =
\Sexpr
{
round(11/15*100, 0)
}
\%
$
, as was achieved with
the non
-
significance criterion from the RPCB, unrealistic margins of
$
\Delta
>
2
$
are required, highlighting the paucity of evidence provided by these studies.
Changing the success criterion to a more lenient level
(
$
\alpha
= 0.1
$
)
or a
more stringent level
(
$
\alpha
= 0.01
$
)
hardly changes this conclusion.
In the bottom plot of Figure~
\ref
{
fig:sensitivity
}
a sensitivity analysis is
reported with respect to the choice of the prior standard deviation and the
Bayes factor threshold. It is uncommon to specify prior standard
deviations larger than the unit
-
information standard deviation of
$
2
$
, as this
corresponds to the assumption of very large effect sizes under the alternatives.
However, to achieve replication success for a larger proportion of replications
than the observed
$
\Sexpr
{
bfSuccesses
}
/
\Sexpr
{
ntotal
}
=
\Sexpr
{
round(bfSuccesses/ntotal*100, 0)
}
\%
$
, unreasonably large prior standard
deviations have to be specified. For instance, a standard deviation of roughly
$
5
$
is required to achieve replication success in
$
50
\%
$
of the replications at
a lenient Bayes factor threshold of
$
\gamma
= 3
$
. The standard deviation needs
to be almost
$
20
$
so that the same success rate
$
11/15 =
\Sexpr
{
round(11/15*100,
0)
}
\%
$
as with the non
-
significance criterion is achieved. The necessary
standard deviations are even higher for stricter Bayes factor threshold,
such as
$
\gamma
= 6
$
or
$
\gamma
= 10
$
.
\begin
{
figure
}
[!
htb
]
\begin
{
figure
}
[!
htb
]
<< "sensitivity", fig.height
=
6
.
5
>>
=
<< "sensitivity", fig.height
=
6
.
5
>>
=
...
@@ -1196,6 +1170,39 @@ grid.arrange(plotA, plotB, ncol = 1)
...
@@ -1196,6 +1170,39 @@ grid.arrange(plotA, plotB, ncol = 1)
\label
{
fig:sensitivity
}
\label
{
fig:sensitivity
}
\end
{
figure
}
\end
{
figure
}
The top plot of Figure~
\ref
{
fig:sensitivity
}
shows the number of
successful replications as a function of the margin
$
\Delta
$
and for different
TOST
\textit
{
p
}
-
value thresholds. Such an ``equivalence curve'' approach was
first proposed by
\citet
{
Hauck
1986
}
. We see that for realistic margins between
$
0
$
and
$
1
$
, the proportion of replication successes remains below
$
50
\%
$
for
the conventional
$
\alpha
= 0.05
$
level. To achieve a success rate of
$
11/15 =
\Sexpr
{
round(11/15*100, 0)
}
\%
$
, as was achieved with the
non
-
significance criterion from the RPCB, unrealistic margins of
$
\Delta
> 2
$
are required, highlighting the paucity of evidence provided by these studies.
Changing the success criterion to a more lenient level
(
$
\alpha
= 0.1
$
)
or a
more stringent level
(
$
\alpha
= 0.01
$
)
hardly changes the conclusion.
The bottom plot of Figure~
\ref
{
fig:sensitivity
}
shows a sensitivity analysis
regarding the choice of the prior standard deviation and the Bayes factor
threshold. It is uncommon to specify prior standard deviations larger than the
unit
-
information standard deviation of
$
2
$
, as this corresponds to the
assumption of very large effect sizes under the alternatives. However, to
achieve replication success for a larger proportion of replications than the
observed
$
\Sexpr
{
bfSuccesses
}
/
\Sexpr
{
ntotal
}
=
\Sexpr
{
round(bfSuccesses/ntotal*100, 0)
}
\%
$
,
unreasonably large prior standard deviations have to be specified. For instance,
a standard deviation of roughly
$
5
$
is required to achieve replication success
in
$
50
\%
$
of the replications at a lenient Bayes factor threshold of
$
\gamma
= 3
$
. The standard deviation needs to be almost
$
20
$
so that the same
success rate
$
11/15 =
\Sexpr
{
round(11/15*100, 0)
}
\%
$
as with the
non
-
significance criterion is achieved. The necessary standard deviations are
even higher for stricter Bayes factor threshold, such as
$
\gamma
= 6
$
or
$
\gamma
= 10
$
.
\bibliography
{
bibliography
}
<< "sessionInfo
1
", eval
=
Reproducibility, results
=
"asis" >>
=
<< "sessionInfo
1
", eval
=
Reproducibility, results
=
"asis" >>
=
## print R sessionInfo to see system information and package versions
## print R sessionInfo to see system information and package versions
## used to compile the manuscript
(
set Reproducibility
=
FALSE, to not do that
)
## used to compile the manuscript
(
set Reproducibility
=
FALSE, to not do that
)
...
...
This diff is collapsed.
Click to expand it.
rsabsence.pdf
+
0
−
0
View file @
e9b2067c
No preview for this file type
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment