\documentclass[12pt]{article}
\usepackage{amsmath,amssymb}
%\usepackage{mydef2}
\usepackage[round]{natbib}
\usepackage{parskip,url}
% use [numbered] for numbering down to subsection level
%\lhead{}
%\input{title}
%\VignetteIndexEntry{The metap package}
\title{The \pkg{metap} package}
\author{Michael Dewey}
\newcommand{\pkg}[1]{\texttt{#1}}
\newcommand{\func}[1]{\texttt{#1}}
\newcommand{\code}[1]{\texttt{#1}}
\begin{document}
\maketitle

\section{Introduction}

\subsection{What is this document for?}

This document describes some methods for the meta--analysis of
$p$--values
(significance values)
and their implementation
in the package \pkg{metap}.
It also contains some commentary on
the performance of the various algorithms
under a small number of different
scenrios with
some hints on the choice of method.

The problem of meta--analysis of $p$--values is of course
not completely unconnected with the more general issue
of simultaneous statistical inference.

\subsection{Why and when to meta--analyse significance values}

The canonical way to meta--analyse a number of primary studies
is to combine estimates of effect sizes from each of them.
There are a large number of packages for
this purpose available from CRAN
and
described in the task view \url{http://CRAN.R-project.org/view=MetaAnalysis}.
However sometimes
the only available information may be $p$--values
especially when some of the primary
studies were published a long time ago or were published
in sources which were less rigorous about insisting on effect
sizes.
The methods outlined here are designed for this
eventuality.
The situation may also arise that some of the studies
can be combined in a conventional meta--analysis
using effect sizes but there
are many others which cannot and in that case the conventional
meta--analysis of the
subset of studies which do have effect sizes
may usefully be supplemented by an overall
analysis of the $p$--values.

Just for the avoidance of doubt I should point out that
if each study has produced a proportion and the goal is
to synthesise them to a common estimate or analyse the
differences between them then the standard methods
are appropriate not the ones outlined here.
The $p$--values in this document are significance levels.

\subsection{Notation}

The $k$ studies give rise to $p$--values,
 $p_i, i = 1, \dots, k$.
These are assumed to be independent
We shall also need the ordered $p$--values:
$p_{[1]} \le p_{[2]}, \dots, \le p_{[k]}$
and weights
$w_i, i = 1, \dots, k$.
Logarithms are natural.
A function for combining $p$--values is denoted $g$.

The methods are referred to by the name of the
function in \func{metap}.
Table \ref{funcs} shows other
descriptions of each method.

\begin{table}[htbp]
\begin{tabular}{lll}
Function name & \multicolumn{2}{c}{Description(s)} \\[1ex]
\func{logitp} & & Logistic\\
\func{meanp} \\
\func{maximump} \\
\func{minimump} & Tippett's method \\
\func{sumlog} & Fisher's method & Chi square (2 df)\\
\func{sump} & Edgington's method & Uniform\\
\func{sumz} & Stouffer's method & Normal\\
\func{votep} \\
\func{wilkinsonp} & Wilkinson's method \\
\end{tabular}
\caption{Methods considered in this document}
\label{funcs}
\end{table}

\section{Theoretical results}

There have been various attempts to clarify the
problem and to discuss optimality
of the various methods.
A detailed account was provided by
\citet{liptak58} although the readers is cautioned that
this requires a
certain familiarity with the methods
of probability theory.

\citet{birnbaum54} considered the property of
admissibility.
A method is admissible if when it rejects $H_0$ for
a set of $p_i$ it will also reject $H_0$ for
$P^*_i$ where $p^*_i \le p_i$ for all $i$.
He considered that Fisher's and Tippett's method were admissible.
See also \citet{owen09}.

He also points out the problem
is poorly specified.
This may
account for the number of methods available
and their differing behaviour.
The null hypothesis $H_0$ is well defined,
that all $p_i$ have a uniform distribution on the unit interval.
There are
two classes of alternative hypothesis
\begin{itemize}
\item
$H_A$: all $p_i$ have the same (unknown)
non--uniform, non--increasing density,
\item
$H_B$:
at least one $p_i$ has an (unknown)
non--uniform, non--increasing density.
\end{itemize}

If all the tests being combined come from
what are basically replicates then $H_A$ is appropriate
whereas if they are of different kinds
of test or different conditions
then $H_B$ is appropriate.
Note that Birnbaum specifically considers the
possibility that the tests being combined may be
very different 
for instance some tests of means, some of variances,
and so on.

\section{Preparation for meta--analysis of $p$--values}

\subsection{Preliminaries}

I assume you have installed \textsf{R} and \pkg{metap}.
You then need to load the package.
<<>>=
library(metap)
@

\subsection{Directionality}

It is usual to have a directional hypothesis, for
instance that treatment is better than control.
For the methods described here a necessary preliminary
is to ensure that all the $p$--values
refer to the same directional hypothesis.
If the value from the primary study is two--sided it needs to
be converted.
This is not simply a matter of halving the quoted $p$--value
as values in the opposite direction need to be reversed.
A convenience function \func{two2one} is provided for this.
<<>>=
pvals <- c(0.1, 0.1, 0.9, 0.9, 0.9, 0.9)
istwo <- c(TRUE,  FALSE, TRUE, FALSE, TRUE, FALSE)
toinvert <- c(FALSE, TRUE, FALSE, FALSE, TRUE, TRUE)
two2one(pvals, two = istwo, invert = toinvert)
@
Note in particular the way in which $0.9$ is
converted under the different scenarios.

<<echo = FALSE>>=
data(validity)
@

\subsection{Plotting}

<<>>=
print(validity)
@

\begin{figure}[tbp]
<<fig = TRUE>>=
par(pin = c(3, 3))
schweder(validity)
@
\caption{Simple example of plot using \func{schweder}}
\label{simple}
\end{figure}

It would be a wise precaution to examine the $p$--values
graphically or otherwise before subjecting them to further analysis.
A function \func{schweder} is provided for this purpose.
This plots the ordered $p$--values, $p_{[i]}$, against
$i$.
Although the original motivation
for the plot is \citet{schweder82}
the function uses a different choice of axes due
to
\citet{benjamini00}.
We will use an example dataset on the validity of student
ratings quoted in \citet{becker94}.
Figure \ref{simple} shows the plot from \func{schweder}.

\func{schweder} also offers the possibility of drawing one of 
a number of straight line summaries.
The three possible straight line summaries are
shown in Figure \ref{withlines} and are:
\begin{itemize}
\item
the lowest slope line of Benjaimin and Hochberg
which is drawn by default
as solid,
\item
a least squares line drawn passing through the point
$k+1, 1$
and using a specified fraction of the points
which is drawn by default as dotted,
\item
a line with user specified
intercept and slope
which is drawn by default as dashed.
\end{itemize}

\begin{figure}[tbp]
<<fig = TRUE>>=
par(pin = c(3, 3))
schweder(validity, drawline = c("bh", "ls", "ab"),
   ls.control = list(frac = 0.5), ab.control = list(a = 0, b = 0.01))
@
\caption{Example of plot with lines added}
\label{withlines}
\end{figure}

\subsection{Reporting problems in the primary studies}

Another issue is what to do with studies which have simply reported
on whether a conventional level of significance like 0.05
was achieved or not.
If the exact associated $p$ cannot be derived from the
statistics quoted in the primary source then the value
of the level achieved, in this case 0.05, can be used
although this may be conservative.
Studies which simply report not significant could be included
as having $p=1$ (or $p=0.5$
if it is known that the direction was right)
although this is very conservative.

\section{The methods}

\subsection{Comparison scenarios}

To provide a standard of comparison
we shall use the following two situations.
Some authors have also used the case of exactly two
$p_i$.

\subsubsection{What if all $p_i = p$?\label{twopisection}}

Perhaps surprisingly there are substantial differences here
as we shall see when we look at each method.
We describe how the returned value
varies with $p_i$ and $k$.

<<echo = FALSE>>=
genvec <- function(pvals, kvals, fun, name) {
   ps <- length(pvals)
   ks <- length(kvals)
   temp <- matrix(-1, nrow = ps, ncol = ks)
   for(i in 1:ps)
   for(j in 1:ks) {
      temp[i, j] <- fun(rep(pvals[i], kvals[j]))$p
   }
   temp2 <- as.vector(temp)
   res <- data.frame(method = rep(name, length(temp2)),
      p = rep(pvals, ks),
      k = rep(kvals, each = ps),
      g = temp2
   )
   res
}
@

<<echo = FALSE>>=
   kvals <- c(4, 5, 6, 8, 10, 15, 20)
   pvals <- c(0.2, 0.3, 0.3679, 0.4, 0.5, 0.6)
   dat <- rbind(
      genvec(pvals, kvals, logitp, "logitp"),
      genvec(pvals, kvals, meanp, "meanp"),
      genvec(pvals, kvals, maximump, "maximump"),
      genvec(pvals, kvals, minimump, "minimump"),
      genvec(pvals, kvals, sumlog, "sumlog"),
      genvec(pvals, kvals, sump, "sump"),
      genvec(pvals, kvals, sumz, "sumz"),
      genvec(pvals, kvals, votep, "votep")
   )
@

\begin{figure}[htbp]
<<echo = FALSE, fig = TRUE>>=
   lattice::xyplot(g ~ k | method, groups = p, type = "l", data = dat,
      auto.key = list(space = "left", lines = TRUE, title = "p"),
      ylab = "g(p)"
   )
@
\caption{Behaviour of the methods for $k$ values of $p=p_i$}
\label{equalp}
\end{figure}

\subsubsection{Cancellation}

When the collection of primary studies
contains a number of values significant in both directions
for example four studies having $p$--values
0.001, 0.001, 0.999, 0.999
the methods can give very different results.
If the intention of the synthesis is to examine a directional
hypothesis one would want a method where these cancelled out.
We shall use those four values as our
example.

\subsection{Methods using transformation of the $p$--values}

One class of methods relies on transforming the $p$--values
and then combining them.

\subsubsection{The method of summation of logs, Fisher's method}

The method relies on the fact that
\begin{equation}
\sum_{i=1}^{k} - 2 \log p_i
\end{equation}
is a chi-squared with $2 k$ df.
This works because $- 2 \log p_i$ is a $\chi^2_2$
and the sum of $\chi^2$ is itself a $\chi^2$
with degrees of freedom equal to the sum of the degrees
of freedom of the
individual $\chi^2$.
Of course the sum of the log of the $p_i$
is also the log of the product of the $p_i$.
Fisher's method is provided in \func{sumlog}.

When all the $p_i=p$
this method returns
a value which
decreases with $k$
when $p<0.32$, increases with $k$
when $p>0.37$, and in between
increases with $k$ and then
decreases.
Some detailed algebra provided in a post
to stats.stackexchange.com by Christoph Hanck
suggests that the breakpoint is $e^{-1} = 0.3679$
so that where the $p_i$ are less than that then for a sufficiently
large $k$ the result will be significant and not if above that.
Over the range of $k$ we are plotting this bound is not
yet closely approached.
Hanck's plot suggests that $k$ must be several hundred
for this to happen.

This method does not cancel significant
values in both direction and returns a significant
result for our example.
<<>>=
pvals <- c(0.001, 0.001, 0.999, 0.999)
sumlog(pvals)
@

It would of course be possible to generalise this to use
transformation to $\chi^2$ with any other number
of degrees of freedom rather than 2.
\citet{lancaster61} suggests that this is highly correlated
with \func{sumlog}.

\subsubsection{The method of summation of $z$ values, Stouffer's method}

Defined as
\begin{equation}
\frac{\sum_{i=1}^k z(p_i)}{\sqrt{k}}
\end{equation}

is a standard normal deviate where
$z$ is the quantile function of the normal
distribution.

The method of summation of $z$
values is provided in \func{sumz}.
It returns a value for our $p_i=p$
example which
decreases with $k$ when $p$ below 0.5
and increases above.

This method does cancel significant values in both
directions.

<<>>=
sumz(pvals)
@

A weighted version is available
\begin{equation}
\frac{\sum_{i=1}^k w_i z(p_i)}{\sqrt {\sum_{i=1}^k w_i ^ 2}}
\end{equation}

where $w_i$ are the weights.

By default the weights are equal.
In the absence of effect sizes (in which case a method for
combining effect sizes would be more appropriate anyway)
best results are believed to be obtained with weights
proportional to the square root of the sample sizes
\citep{zaykin11} following \citet{liptak58}.
At the moment weighting is only provided in \func{sumz} as this
is the only method for which a published example is accessible.


\subsubsection{The method of summation of logits}

Defined as
\begin{equation}
- \frac{\sum_{i=1}^k \log\frac{p}{1 - p}}{C}
\end{equation}
is distributed as Student's $t$ with $5 k + 4$ df
where
\begin{equation}
C = \sqrt\frac{k \pi^2 (5 k + 2)}{3(5 k + 4)}
\end{equation}
This method is provided in \func{logitp}.
The constant was arrived at by equating
skewness and kurtosis with that of the $t$--distribution
\citep{loughin04}.

This method returns a value for our $p_i=p$
example which
decreases with $k$ when $p$ below 0.5
and increases above.

This method does cancel significant values in both
directions.

<<>>=
logitp(pvals)
@


\subsubsection{Examples for \func{sumlog}, \func{sumz}, and \func{logitp}}
Using the same example dataset which we have already plotted

<<>>=
sumlog(validity)
sumz(validity)
logitp(validity)
@

As can be seen these are in quite good agreement.

\subsection{Methods using untransformed $p$--values}

\subsubsection{The method of minimum $p$ and Wilkinson's method}

The minimum $p$ method is usually described in terms
of a rejection at the $\alpha_*$ level of the
null hypothesis
\begin{equation}
p_{[1]} < 1 - (1 - \alpha_*)^{\frac{1}{k}}
\end{equation}

The minimum $p$ method is a special case of Wilkinson's
method which uses $p_{[r]}$ where $1 \le r \le k$
\citep{wilkinson51}.
% p is pbeta(p[r], r, k+1-r)
% critical p is qbeta(alpha, r, k+1-r)
Wilkinson's method is provided in \func{wilkinsonp} and a
convenience function \func{minimump}
with its own \code{print} method is provided for
the minimum $p$ method.
It is also possible to use the method for the
maximum $p$ (that is $r=k$) and a convenience function \func{maximump}
is provided for that purpose.


These methods return a value for our $p_i=p$
example which
always increases with $k$
which is true for
\func{minimump}
and
which always decreases with $k$
which is true for
\func{maximump}

The minimum $p$ method does not cancel significant
values in both direction and returns a significant
result for our example
but the maximum $p$ does cancel.
<<>>=
minimump(pvals)
maximump(pvals)
@

\subsubsection{The method of summation of $p$--values, Edgington's method\label{sump}}

Define
\begin{equation}
S = \sum_{i=1}^k p_i
\end{equation}
then this method is defined as
\begin{equation}
\frac{(S)^k}{k!}
- {k - 1 \choose 1}\frac{(S - 1)^k}{k!}
+ {k - 2 \choose 2}\frac{(S - 2)^k}{k!}
- \dots
\end{equation}

where there are $k$ studies and the series continues until the
term in in the numerator $(S-i)$
becomes negative \citep{edgington72a}.
This method is provided in \func{sump}.

This method returns a value for our $p_i=p$
example which
decreases with $k$ when $p$ below 0.5
and increases above.

This method does cancel significant values in both
directions.

<<>>=
sump(pvals)
@

Some authors use a simpler version,
for instance \citet{rosenthal78} in the text
although compare his Table 4.

\begin{equation}
\frac{(\sum p)^k}{k!}
\end{equation}

where there are $k$ studies
but this can be very conservative when
$\sum p > 1$
There seems no particular need to use this method but
it is returned by \func{sump}
as the value of \code{conservativep}
for use in checking published values.

Note also that there can be numerical problems for extreme values
of $S$ and in that case recourse might be made to
\func{sumz} or \func{logitp} which have similar
properties.

\subsubsection{The mean $p$ method}

This is defined as
\begin{equation}
\begin{aligned}
z &= (0.5 - \bar{p}) \sqrt{12k}\\
\bar p &= \frac{\sum_{i=1}^k p_i}{k}
\end{aligned}
\end{equation}
which is a standard normal \citep{edgington72b}
and
where .
Although this method is attributed to Edgington
when the phrase Edgington's method is used
it refers to the method of summation of $p$--values
described above in Section \ref{sump}.

This method returns a value for our $p_i=p$
example which
decreases with $k$ when $p$ below 0.5
and increases above.

This method does cancel significant values in both
directions.

<<>>=
meanp(pvals)
@



\subsubsection{Examples for \func{minimump}, \func{maximump}, \func{sump}, and \func{meanp}}
<<>>=
minimump(validity)
maximump(validity)
sump(validity)
meanp(validity)
@

Agreement here is not so good especially for the maximump method.

\subsection{Other methods}

\subsubsection{The method of vote--counting}

A simple way of looking at the problem is
vote counting.
Strictly speaking this is not a method which combines
$p$--values in the same sense as the other method.
If most of the studies have produced results in favour of the
alternative hypothesis irrespective of whether any of them is
individually significant then that might be regarded as evidence
for that alternative.
The numbers for and against may be compared with what
would be expected under the null using the binomial distribution.
A variation on this would allow for a neutral zone of studies
which are considered neither for nor against.
For instance one might only
count studies which have reached some conventional level of
statistical significance in the two different directions.

This method returns a value for our $p_i=p$
example which
is 1 above 0.5 and otherwise invariant with $p$
but decreases with $k$.

This method does cancel significant values in both
directions.

<<>>=
votep(pvals)
@

\subsection{Examples of \func{votep}}

<<>>=
votep(validity)
@

\section{Loughin's recommendations}

In his simulation study \citet{loughin04} carried out extensive comparisons.
He bases his recommendations on criteria of structure and the
arrangement of evidence against $H_0$.

Under structure he considers
three cases with the following recommendations:
emphasis on small $p$--values (\func{sumlog} and \func{minimump}),
emphasis on large $p$--values (\func{maximump} and \func{sump}),
and equal emphasis (\func{logitp} and \func{sumz}).

Under arrangement of evidence he considers
where this is concentrated.
His recommendations are summarised in Table
\ref{loughin}.

\begin{table}[htbp]
\begin{tabular}{ll}
Equal in all tests & $k < 10$ \func{sump}, \func{maximump} \\
 & Any $k$ \func{sumz}, \func{logitp} \\
Some in all tests & $k < 10$ \func{sump}, \func{maximump} \\
 & Any $k$ \func{sumz}, \func{logitp} \\
In majority of tests & \func{sumz}, \func{logitp} \\
In minority of tests & Moderate or strong evidence \func{sumlog} \\
 & Any power \func{sumz}, \func{logitp} \\
In one test only & Strong total evidence \func{minimup} \\
 & Moderate total evidence \func{sumlog} \\
 & Weak total evidence \func{sumz}, \func{logitp} \\
\end{tabular}
\caption{Loughin's recommendations for method choice}
\label{loughin}
\end{table}

\section{Other considerations}


\subsection{Directionality}

When the collection of primary studies
contains a number of values significant in both directions
we have seen that
the methods can give very different results.
If the intention of the synthesis is to examine a directional
hypothesis one would want a method where these cancelled out.
Clearly the choice should be made on scientific grounds
not on the baiss of the outcome.


\subsection{Extractor functions}

The standard \code{print} and \code{plot} methods
are provided.

\subsection{Legal values for $p_i$}

\begin{table}[htbp]
\begin{tabular}{lccl}
 & \multicolumn{2}{c}{Valid for} \\
 & $p=0$ & $p=1$ & Notes \\
\code{logitp} & N & N \\
\code{meanp} & Y & Y & Requires at least four studies\\
\code{sumlog} & N & Y \\
\code{sump} & Y & Y \\
\code{sumz} & Y & Y \\
\code{votep} & Y & Y \\
\code{wilkinson} & Y & Y \\
\end{tabular}
\caption{Restrictions on values of $p_i$}
\label{restrictions}
\end{table}
Not all methods work with $p=0$ or $p=1$.
See Table \ref{restrictions} for details.
If these values occur in your dataset
and you do not wish the functions to take
their routine action of excluding that study
then you need to decide what to do.
If you believe that injudicious rounding is to blame
you might wish to replace zero values by the
least upper bound of the
values which would still round to zero to the given number of decimal places.
So you might replace 0.00 with 0.005, 0.000 with 0.0005
and so on.
Similar action can be taken for values given as unity.

\subsection{Reading}

An annotated bibliography is provided by \citet{cousins08}

\section{Feedback}

I aim to include any method for which there exists a published
example against which I can test the code.
I welcome feedback about such sources and any other comments
about either the documentation or the code.
\bibliography{metap}
\bibliographystyle{plainnat}
\end{document}

