R: Estimate relative overfitting.

estimateRelativeOverfitting {mlr} | R Documentation

## Estimate relative overfitting.

### Description

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

### Usage

    estimateRelativeOverfitting(predish, measures, task, learner = NULL,
      pred.train = NULL, iter = 1)

### Arguments

`predish` | (ResampleDesc | ResamplePrediction | Prediction) Resampling strategy or resampling prediction or test predictions.
`measures` | (Measure | list of Measure) Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure.
`task` | (Task) The task.
`learner` | (Learner | `character(1)`) The learner. If you pass a string the learner will be created via makeLearner.
`pred.train` | (Prediction) Training predictions. Only needed if test predictions are passed.
`iter` | (integer) Iteration number. Default 1, usually you don't need to specify this. Only needed if test predictions are passed.

### Details

Currently only support for classification and regression tasks is implemented.

### Value

(data.frame). Relative overfitting estimate(s), named by measure(s), for each resampling iteration.

### References

Bradley Efron and Robert Tibshirani; Improvements on Cross-Validation: The .632+ Bootstrap Method, Journal of the American Statistical Association, Vol. 92, No. 438. (Jun., 1997), pp. 548-560.

### See Also

Other performance: `ConfusionMatrix`, `calculateConfusionMatrix`, `calculateROCMeasures`, `makeCostMeasure`, `makeCustomResampledMeasure`, `makeMeasure`, `measures`, `performance`, `setAggregation`, `setMeasurePars`

### Examples

    task = makeClassifTask(data = iris, target = "Species")
    rdesc = makeResampleDesc("CV", iters = 2)
    estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.knn"))
    estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.lda"))
    rpred = resample("classif.knn", task, rdesc)$pred
    estimateRelativeOverfitting(rpred, acc, task)

[Package mlr version 2.16.0 Index]