R: Compare the AUC of two ROC curves

roc.test {pROC} | R Documentation

## Compare the AUC of two ROC curves

### Description

This function compares the AUC or partial AUC of two correlated (or paired) or uncorrelated (unpaired) ROC curves. Several syntaxes are available: two object of class roc (which can be AUC or smoothed ROC), or either three vectors (response, predictor1, predictor2) or a response vector and a matrix or data.frame with two columns (predictors).

### Usage

    # roc.test(...)
    ## S3 method for class 'roc'
    roc.test(roc1, roc2, method=c("delong", "bootstrap",
    "venkatraman", "sensitivity", "specificity"), sensitivity = NULL,
    specificity = NULL, alternative = c("two.sided", "less", "greater"),
    paired=NULL, reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE,
    ties.method="first", progress=getOption("pROCProgress")$name,
    parallel=FALSE, ...)
    ## S3 method for class 'auc'
    roc.test(roc1, roc2, ...)
    ## S3 method for class 'smooth.roc'
    roc.test(roc1, roc2, ...)
    ## S3 method for class 'formula'
    roc.test(formula, data, ...)
    ## Default S3 method:
    roc.test(response, predictor1, predictor2=NULL,
    na.rm=TRUE, method=NULL, ...)

### Arguments

`roc1, roc2` | the two ROC curves to compare. Either “roc”, “auc” or “smooth.roc” objects (types can be mixed).
`response` | a vector or factor, as for the roc function.
`predictor1` | a numeric or ordered vector as for the roc function, or a matrix or data.frame with predictors two colums.
`predictor2` | only if predictor1 was a vector, the second predictor as a numeric vector.
`formula` | a formula of the type response~predictor1+predictor2. Additional arguments `data`, `subset` and `na.action` are supported, see `model.frame` for more details.
`data` | a matrix or data.frame containing the variables in the formula. See `model.frame` for more details.
`na.rm` | if `TRUE`, the observations with `NA` values will be removed.
`method` | the method to use, either “delong”, “bootstrap” or “venkatraman”. The first letter is sufficient. If omitted, the appropriate method is selected as explained in details.
`sensitivity, specificity` | if `method="sensitivity"` or `method="specificity"`, the respective level where the test must be assessed as a numeric of length 1.
`alternative` | specifies the alternative hypothesis. Either of “two.sided”, “less” or “greater”. The first letter is sufficient. Default: “two.sided”. Only “two.sided” is available with `method="venkatraman"`.
`paired` | a logical indicating whether you want a paired roc.test. If `NULL`, the paired status will be auto-detected by `are.paired`. If `TRUE` but the paired status cannot be assessed by `are.paired` will produce an error.
`reuse.auc` | if `TRUE` (default) and the “roc” objects contain an “auc” field, re-use these specifications for the test. See the AUC specification section for more details.
`boot.n` | for `method="bootstrap"` and `method="venkatraman"` only: the number of bootstrap replicates or permutations. Default: 2000.
`boot.stratified` | for `method="bootstrap"` only: should the bootstrap be stratified (same number of cases/controls in each replicate than in the original sample) or not. Ignored with `method="venkatraman"`. Default: TRUE.
`ties.method` | for `method="venkatraman"` only: argument for `rank` specifying how ties are handled. Defaults to “first” as described in the paper.
`progress` | the name of progress bar to display. Typically “none”, “win”, “tk” or “text” (see the `name` argument to `create_progress_bar` for more information), but a list as returned by `create_progress_bar` is also accepted. See also the “Progress bars” section of this package's documentation.
`parallel` | if TRUE, the bootstrap is processed in parallel, using parallel backend provided by plyr (foreach).
`...` | further arguments passed to or from other methods, especially arguments for `roc` and `roc.test.roc` when calling `roc.test.default` or `roc.test.formula`. Arguments for `auc`, and `txtProgressBar` (only `char` and `style`) if applicable.

### Details

This function compares two ROC curves. It is typically called with the two roc objects to compare. `roc.test.default` is provided as a convenience method and creates two roc objects before calling `roc.test.roc`.

Three methods are available: “delong”, “bootstrap” and “venkatraman” (see “Computational details” section below). “delong” and “bootstrap” are tests over the AUC whereas “venkatraman” compares the the ROC curves themselves.

Default is to use “delong” method except for comparison of partial AUC, smoothed curves and curves with different `direction`, where `bootstrap` is used. Using “delong” for partial AUC and smoothed ROCs is not supported in pROC (a warning is produced and “bootstrap” is employed instead). It is spurious to use “delong” for `roc` with different `direction` (a warning is issued but the spurious comparison is enforced). “venkatraman”'s test cannot be employed to compare smoothed ROC curves. Additionally, partial AUC specifications are ignored (with a warning), and comparison of ROC curves with different `direction` should be used with care (a warning is produced as well).

If `alternative="two.sided"`, a two-sided test for difference in AUC is performed. If `alternative="less"`, the alternative is that the AUC of roc1 is smaller than the AUC of roc2. For `method="venkatraman"`, only “two.sided” test is available.

If the `paired` argument is not provided, the `are.paired` function is employed to detect the paired status of the ROC curves. It will test if the original `response` is identical between the two ROC curves (this is always the case if the call is made with `roc.test.default`). This detection is unlikely to raise false positives, but this possibility cannot be excluded entierly. It would require equal sample sizes and `response` values and order in both ROC curves. If it happens to you, use `paired=FALSE`. If you know the ROC curves are paired you can pass `paired=TRUE`. However this is useless as it will be tested anyway.

For smoothed ROC curves, smoothing is performed again at each bootstrap replicate with the parameters originally provided. If a density smoothing was performed with user-provided `density.cases` or `density.controls` the bootstrap cannot be performed and an error is issued.

### Value

A list of class "htest" with following content:

`p.value` | the p-value of the test.
`statistic` | the value of the Z (`method="delong"`) or D (`method="bootstrap"`) statistics.
`alternative` | the alternative hypothesis.
`method` | the character string “DeLong's test for two correlated ROC curves” (if `method="delong"`) or “Bootstrap test for two correlated ROC curves” (if `method="bootstrap"`).
`null.value` | the expected value of the statistic under the null hypothesis, that is 0.
`estimate` | the AUC in the two ROC curves.
`data.name` | the names of the data that was used.
`parameter` | for `method="bootstrap"` only: the values of the `boot.n` and `boot.stratified` arguments.

### AUC specification

The comparison of the AUC of the ROC curves needs a specification of the AUC. The specification is defined by:

1. the “auc” field in the “roc” objects if `reuse.auc` is set to `TRUE` (default)

2. passing the specification to `auc` with ... (arguments `partial.auc`, `partial.auc.correct` and `partial.auc.focus`). In this case, you must ensure either that the `roc` object do not contain an `auc` field (if you called `roc` with `auc=FALSE`), or set `reuse.auc=FALSE`.

If `reuse.auc=FALSE` the `auc` function will always be called with `...` to determine the specification, even if the “roc” objects do contain an `auc` field.

As well if the “roc” objects do not contain an `auc` field, the `auc` function will always be called with `...` to determine the specification.

The AUC specification is ignored in the Venkatraman test.

Warning: if the roc object passed to roc.test contains an `auc` field and `reuse.auc=TRUE`, auc is not called and arguments such as `partial.auc` are silently ignored.

### Computation details

With `method="bootstrap"`, the processing is done as follow:

1. `boot.n` bootstrap replicates are drawn from the data. If `boot.stratified` is TRUE, each replicate contains exactly the same number of controls and cases than the original sample, otherwise if FALSE the numbers can vary.

2. for each bootstrap replicate, the AUC of the two ROC curves are computed and the difference is stored.

3. The following formula is used:

   D=(AUC1-AUC2)/s

   where s is the standard deviation of the bootstrap differences and AUC1 and AUC2 the AUC of the two (original) ROC curves.

4. D is then compared to the normal distribution, according to the value of `alternative`.

See also the Bootstrap section in this package's documentation.

With `method="delong"`, the processing is done as described in DeLong et al. (1988) for paired ROC curves, using the algorithm of Sun and Xu (2014). Only comparison of two ROC curves is implemented. The method has been extended for unpaired ROC curves where the p-value is computed with an unpaired t-test with unequal sample size and unequal variance, with

D=(V^r(θ^r) - V^s(θ^s)) / sqrt(S^r + S^s)

With `method="venkatraman"`, the processing is done as described in Venkatraman and Begg (1996) (for paired ROC curves) and Venkatraman (2000) (for unpaired ROC curves) with `boot.n` permutation of sample ranks (with ties breaking). For consistency reasons, the same argument `boot.n` as in bootstrap defines the number of permutations to execute, even though no bootstrap is performed.

For `method="specificity"`, the test assesses if the sensitivity of the ROC curves are different at the level of specificity given by the `specificity` argument, which must be a numeric of length 1. Bootstrap is employed as with `method="bootstrap"` and `boot.n` and `boot.stratified` are available. This is identical to the test proposed by Pepe et al. (2009). The `method="sensitivity"` is very similar, but assesses if the specificity of the ROC curves are different at the level of sensitivity given by the `sensitivity` argument.

### Warnings

If “auc” specifications are different in both roc objects, the warning “Different AUC specifications in the ROC curves. Enforcing the inconsistency, but unexpected results may be produced.” is issued. Unexpected results may be produced.

If one or both ROC curves are “smooth.roc” objects with different smoothing specifications, the warning “Different smoothing parameters in the ROC curves. Enforcing the inconsistency, but unexpected results may be produced.” is issued. This warning can be benign, especially if ROC curves were generated with `roc(..., smooth=TRUE)` with different arguments to other functions (such as plot), or if you really want to compare two ROC curves smoothed differently.

If `method="delong"` and the AUC specification specifies a partial AUC, the warning “Using DeLong's test for partial AUC is not supported. Using bootstrap test instead.” is issued. The `method` argument is ignored and “bootstrap” is used instead.

If `method="delong"` and the ROC curve is smoothed, the warning “Using DeLong's test for smoothed ROCs is not supported. Using bootstrap test instead.” is issued. The `method` argument is ignored and “bootstrap” is used instead.

If `method="venkatraman"`, and the AUC specification specifies a partial AUC, the AUC specification is ignored with the warning “Partial AUC is ignored in Venkatraman's test.”.

If `method="venkatraman"`, and `alternative` is “less” or “greater”, the warning “Only two-sided tests are available for Venkatraman. Performing two-sided test instead.” is produced and a two tailed test is performed.

Both DeLong and Venkatraman's test ignores the direction of the ROC curve so that if two ROC curves have a different differ in the value of `direction`, the warning “(DeLong|Venkatraman)'s test should not be applied to ROC curves with different directions.” is printed. However, the spurious test is enforced.

If `boot.stratified=FALSE` and the sample has a large imbalance between cases and controls, it could happen that one or more of the replicates contains no case or control observation, or that there are not enough points for smoothing, producing a `NA` area. The warning “NA value(s) produced during bootstrap were ignored.” will be issued and the observation will be ignored. If you have a large imbalance in your sample, it could be safer to keep `boot.stratified=TRUE`.

When both ROC curves have an `auc` of 1 (or 100%), their variances and covariance will always be null, and therefore the p-value will always be 1. This is true for both “delong”, “bootstrap” and “venkatraman” methods. This result is misleading, as the variances and covariance are of course not null. A `warning` will be displayed to inform of this condition, and of the misleading output.

### Errors

An error will also occur if you give a `predictor2` when `predictor1` is a `matrix` or a `data.frame`, if `predictor1` has more than two columns, or if you do not give a `predictor2` when `predictor1` is a vector.

If `density.cases` and `density.controls` were provided for smoothing, the error “Cannot compute the statistic on ROC curves smoothed with density.controls and density.cases.” is issued.

If `method="venkatraman"` and one of the ROC curves is smoothed, the error “Using Venkatraman's test for smoothed ROCs is not supported.” is produced.

With `method="specificity"`, the error “Argument 'specificity' must be numeric of length 1 for a specificity test.” is given unless the specificity argument is specified as a numeric of length 1. The “Argument 'sensitivity' must be numeric of length 1 for a sensitivity test.” message is given for `method="sensitivity"` under similar conditions.

### Acknowledgements

We would like to thank E. S. Venkatraman and Colin B. Begg for their support in the implementation of their test.

### References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845.

James A. Hanley and Barbara J. McNeil (1982) “The meaning and use of the area under a receiver operating characteristic (ROC) curve”. Radiology 143, 29–36.

Margaret Pepe, Gary Longton and Holly Janes (2009) “Estimation and Comparison of Receiver Operating Characteristic Curves”. The Stata journal 9, 1.

Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller (2009) “Combination of protein biomarkers”. useR! 2009, Rennes. https://www.r-project.org/nosvn/conferences/useR-2009/abstracts/user_author.html

Xavier Robin, Natacha Turck, Alexandre Hainard, et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 7, 77. DOI: 10.1186/1471-2105-12-77.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves”. IEEE Signal Processing Letters, 21, 1389–1393. DOI: 10.1109/LSP.2014.2337313.

E. S. Venkatraman and Colin B. Begg (1996) “A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment”. Biometrika 83, 835–848. DOI: 10.1093/biomet/83.4.835.

E. S. Venkatraman (2000) “A Permutation Test to Compare Receiver Operating Characteristic Curves”. Biometrics 56, 1134–1138. DOI: 10.1111/j.0006-341X.2000.01134.x.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”. Journal of Statistical Software, 40, 1–29. URL: www.jstatsoft.org/v40/i01.

### See Also

`roc`, `power.roc.test`

CRAN package plyr, employed in this function.

### Examples

    data(aSAH)

    # Basic example with 2 roc objects
    roc1 <- roc(aSAH$outcome, aSAH$s100b)
    roc2 <- roc(aSAH$outcome, aSAH$wfns)
    roc.test(roc1, roc2)

    ## Not run:
    # The latter used Delong's test. To use bootstrap test:
    roc.test(roc1, roc2, method="bootstrap")
    # Increase boot.n for a more precise p-value:
    roc.test(roc1, roc2, method="bootstrap", boot.n=10000)

    ## End(Not run)

    # Alternative syntaxes
    roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns)
    roc.test(aSAH$outcome, data.frame(aSAH$s100b, aSAH$wfns))

    # If we had a good a priori reason to think that wfns gives a
    # better classification than s100b (in other words, AUC of roc1
    # should be lower than AUC of roc2):
    roc.test(roc1, roc2, alternative="less")

    ## Not run:
    # Comparison can be done on smoothed ROCs
    # Smoothing is re-done at each iteration, and execution is slow
    roc.test(smooth(roc1), smooth(roc2))
    # or:
    roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns, smooth=TRUE, boot.n=100)

    ## End(Not run)
    # or from an AUC (no smoothing)
    roc.test(auc(roc1), roc2)

    ## Not run:
    # Comparison of partial AUC:
    roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")
    roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")
    roc.test(roc3, roc4)
    # This is strictly equivalent to:
    roc.test(roc3, roc4, method="bootstrap")

    # Alternatively, we could re-use roc1 and roc2 to get the same result:
    roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")

    # Comparison on specificity and sensitivity
    roc.test(roc1, roc2, method="specificity", specificity=0.9)
    roc.test(roc1, roc2, method="sensitivity", sensitivity=0.9)

    ## End(Not run)

    # Spurious use of DeLong's test with different direction:
    roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")
    roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")
    roc.test(roc5, roc6, method="delong")

    ## Not run:
    # Comparisons of the ROC curves
    roc.test(roc1, roc2, method="venkatraman")

    ## End(Not run)

    # Unpaired tests
    roc7 <- roc(aSAH$outcome, aSAH$s100b)
    # artificially create an roc8 unpaired with roc7
    roc8 <- roc(aSAH$outcome[1:100], aSAH$s100b[1:100])
    ## Not run:
    roc.test(roc7, roc8, paired=FALSE, method="delong")
    roc.test(roc7, roc8, paired=FALSE, method="bootstrap")
    roc.test(roc7, roc8, paired=FALSE, method="venkatraman")
    roc.test(roc7, roc8, paired=FALSE, method="specificity", specificity=0.9)

    ## End(Not run)

[Package pROC version 1.12.1 Index]