R: pROC

pROC-package {pROC} | R Documentation

## pROC

### Description

Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves. Sample size / power computation for one or two ROC curves are available.

### Details

The basic unit of the pROC package is the `roc` function. It will build a ROC curve, smooth it if requested (if `smooth=TRUE`), compute the AUC (if `auc=TRUE`), the confidence interval (CI) if requested (if `ci=TRUE`) and plot the curve if requested (if `plot=TRUE`).

The `roc` function will call `smooth`, `auc`, `ci` and `plot` as necessary. See these individual functions for the arguments that can be passed to them through `roc`. These function can be called separately.

Two paired (that is `roc` objects with the same `response`) or unpaired (with different `response`) ROC curves can be compared with the `roc.test` function.

### Citation

If you use pROC in published research, please cite the following paper:

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77

Type `citation("pROC")` for a BibTeX entry.

The authors would be glad to hear how pROC is employed. You are kindly encouraged to notify Xavier Robin <pROC-cran@xavier.robin.name> about any work you publish.

### Abbreviations

The following abbreviations are employed extensively in this package:

- ROC: receiver operating characteristic

- AUC: area under the ROC curve

- pAUC: partial area under the ROC curve

- CI: confidence interval

- SP: specificity

- SE: sensitivity

### Functions

`roc` | Build a ROC curve
`are.paired` | Dertermine if two ROC curves are paired
`auc` | Compute the area under the ROC curve
`ci` | Compute confidence intervals of a ROC curve
`ci.auc` | Compute the CI of the AUC
`ci.coords` | Compute the CI of arbitrary coordinates
`ci.se` | Compute the CI of sensitivities at given specificities
`ci.sp` | Compute the CI of specificities at given sensitivities
`ci.thresholds` | Compute the CI of specificity and sensitivity of thresholds
`ci.coords` | Compute the CI of arbitrary coordinates
`coords` | Coordinates of the ROC curve
`cov` | Covariance between two AUCs
`ggroc` | Plot a ROC curve with ggplot2 (Experimental)
`has.partial.auc` | Determine if the ROC curve have a partial AUC
`lines.roc` | Add a ROC line to a ROC plot
`plot.ci` | Plot CIs
`plot` | Plot a ROC curve
`print` | Print a ROC curve object
`roc.test` | Compare the AUC of two ROC curves
`smooth` | Smooth a ROC curve
`var` | Variance of the AUC

### Dataset

This package comes with a dataset of 141 patients with aneurysmal subarachnoid hemorrhage: `aSAH`.

### Installing and using

To install this package, make sure you are connected to the internet and issue the following command in the R prompt:

        install.packages("pROC")

To load the package in R:

        library(pROC)

### Bootstrap

All the bootstrap operations for significance testing, confidence interval, variance and covariance computation are performed with non-parametric stratified or non-stratified resampling (according to the `stratified` argument) and with the percentile method, as described in Carpenter and Bithell (2000) sections 2.1 and 3.3.

Stratification of bootstrap can be controlled with `boot.stratified`. In stratified bootstrap (the default), each replicate contains the same number of cases and controls than the original sample. Stratification is especially useful if one group has only little observations, or if groups are not balanced.

The number of bootstrap replicates is controlled by `boot.n`. Higher numbers will give a more precise estimate of the significance tests and confidence intervals but take more time to compute. 2000 is recommanded by Carpenter and Bithell (2000) for confidence intervals. In our experience this is sufficient for a good estimation of the first significant digit only, so we recommend the use of 10000 bootstrap replicates to obtain a good estimate of the second significant digit whenever possible.

#### Progress bars

A progressbar shows the progress of bootstrap operations. It is handled by the plyr package (Wickham, 2011), and is created by the `progress_*` family of functions. Sensible defaults are guessed during the package loading:

- In non-interactive mode, no progressbar is displayed.

- In embedded GNU Emacs “ESS”, a `txtProgressBar`

- In Windows, a `winProgressBar` bar.

- In other systems with a graphical display, a `tkProgressBar`.

- In systems without a graphical display, a `txtProgressBar`.

The default can be changed with the option “pROCProgress”. The option must be a list with a `name` item setting the type of progress bar (“none”, “win”, “tk” or “text”). Optional items of the list are “width”, “char” and “style”, corresponding to the arguments to the underlying progressbar functions. For example, to force a text progress bar:

    options(pROCProgress = list(name = "text", width = NA, char = "=", style = 3)

To inhibit the progress bars completely:

    options(pROCProgress = list(name = "none"))

### Handling large datasets

#### Algorithms

Since version 1.12, pROC can automatically select the fastest algorithm for your dataset. The choice is based on the number of thresholds of the ROC curve: until about 1500 thresholds, algorithm 3 is used. Above that limit, pROC switches to algorithm 2.

This simple selection should work in most cases. However if you are unsure or want to test it for yourself, use `algorithm=0` to run a quick benchmark between 2 and 3. Make sure microbenchmark is installed. Beware, this is very slow as it will repeat the computation 10 times to obtain a decent estimate of each algorithm speed.

    if (!requireNamespace("microbenchmark")) install.packages("microbenchmark")

    # First a ROC curve with many thresholds
    response <- rbinom(5E4, 1, .5)
    predictor <- rnorm(5E4)
    rocobj <- roc(response, predictor, algorithm = 0) # Very slow!

    # Next a ROC curve with few thresholds but more data points
    response <- rbinom(1E6, 1, .5)
    predictor <- rpois(1E6, 1)
    rocobj <- roc(response, predictor, algorithm = 0) # Very slow!

#### Boostrap

Bootstrap is typically slow because it involves repeatedly computing the ROC curve (or a part of it).

Some bootstrap functions are faster than others. Typically, `ci.thresholds` is the fastest, and `ci.coords` the slowest. Use `ci.coords` only if the CI you need cannot be computed by the specialized CI functions `ci.thresholds`, `ci.se` and `ci.sp`. Note that `ci.auc` cannot be replaced anyway.

A naive way to speed-up the boostrap is by removing the progress bar:

    rocobj <- roc(response, round(predictor))
    system.time(ci(rocobj))
    system.time(ci(rocobj, progress = "none"))

It is of course possible to reduce the number of boostrap iterations. See the `boot.n` argument to `ci`. This will reduce the precision of the bootstrap estimate.

##### Parallel processing

Bootstrap operations can be performed in parallel. The backend provided by the plyr package is used, which in turn relies on the foreach package.

To enable parallell processing, you first need to load an adaptor for the foreach package (doMC, doMPI, doParallel, doRedis, doRNG or doSNOW)), register the backend, and set `parallel=TRUE`.

    library(doParallel)
    registerDoParallel(cl <- makeCluster(getOption("mc.cores", 2)))
    ci(rocobj, method="bootstrap", parallel=TRUE)
    stopCluster(cl)

Progress bars are not available when parallel processing is enabled.

#### Using DeLong instead of boostrap

DeLong is an asymptotically exact method to evaluate the uncertainty of an AUC (DeLong et al. (1988)). Since version 1.9, pROC uses the algorithm proposed by Sun and Xu (2014) which has an O(N log N) complexity and is always faster than bootstrapping. By default, pROC will choose the DeLong method whenever possible.

    rocobj <- roc(response, round(predictor), algorithm=3)
    system.time(ci(rocobj, method="delong"))
    system.time(ci(rocobj, method="bootstrap", parallel = TRUE))

### Author(s)

Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller

Maintainer: Xavier Robin <pROC-cran@xavier.robin.name>

### References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians”. Statistics in Medicine 19, 1141–1164. DOI: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845.

Tom Fawcett (2006) “An introduction to ROC analysis”. Pattern Recognition Letters 27, 861–874. DOI: 10.1016/j.patrec.2005.10.010.

Xavier Robin, Natacha Turck, Alexandre Hainard, et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 7, 77. DOI: 10.1186/1471-2105-12-77.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves”. IEEE Signal Processing Letters, 21, 1389–1393. DOI: 10.1109/LSP.2014.2337313.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”. Journal of Statistical Software, 40, 1–29. URL: www.jstatsoft.org/v40/i01.

### See Also

CRAN packages ROCR, verification or Bioconductor's roc for ROC curves.

CRAN packages plyr, MASS and logcondens employed in this package.

### Examples

    data(aSAH)

    # Build a ROC object and compute the AUC
    roc(aSAH$outcome, aSAH$s100b)
    roc(outcome ~ s100b, aSAH)

    # Smooth ROC curve
    roc(outcome ~ s100b, aSAH, smooth=TRUE)

    # more options, CI and plotting
    roc1 <- roc(aSAH$outcome,
                aSAH$s100b, percent=TRUE,
                # arguments for auc
                partial.auc=c(100, 90), partial.auc.correct=TRUE,
                partial.auc.focus="sens",
                # arguments for ci
                ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
                # arguments for plot
                plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
                print.auc=TRUE, show.thres=TRUE)

    # Add to an existing plot. Beware of 'percent' specification!
    roc2 <- roc(aSAH$outcome, aSAH$wfns,
                plot=TRUE, add=TRUE, percent=roc1$percent)

    ## Coordinates of the curve ##
    coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))
    coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))

    ## Confidence intervals ##

    # CI of the AUC
    ci(roc2)

    ## Not run:
    # CI of the curve
    sens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))
    plot(sens.ci, type="shape", col="lightblue")
    plot(sens.ci, type="bars")

    ## End(Not run)

    # need to re-add roc2 over the shape
    plot(roc2, add=TRUE)

    ## Not run:
    # CI of thresholds
    plot(ci.thresholds(roc2))

    ## End(Not run)

    # In parallel
    if (require(doParallel)) {
        registerDoParallel(cl <- makeCluster(getOption("mc.cores", 2L)))
        ## Not run: ci(roc2, method="bootstrap", parallel=TRUE)

        stopCluster(cl)
    }

    ## Comparisons ##

    # Test on the whole AUC
    roc.test(roc1, roc2, reuse.auc=FALSE)

    ## Not run:
    # Test on a portion of the whole AUC
    roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
             partial.auc.focus="se", partial.auc.correct=TRUE)

    # With modified bootstrap parameters
    roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
             partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)

    ## End(Not run)

[Package pROC version 1.12.1 Index]