R: Generate data for critical-differences plot.

generateCritDifferencesData {mlr} | R Documentation

## Generate data for critical-differences plot.

### Description

Generates data that can be used to plot a critical differences plot. Computes the critical differences according to either the `"Bonferroni-Dunn"` test or the `"Nemenyi"` test.
`"Bonferroni-Dunn"` usually yields higher power as it does not compare all algorithms to each other, but all algorithms to a `baseline` instead.
Learners are drawn on the y-axis according to their average rank.
For `test = "nemenyi"` a bar is drawn, connecting all groups of not significantly different learners.
For `test = "bd"` an interval is drawn arround the algorithm selected as a baseline. All learners within this interval are not signifcantly different from the baseline.
Calculation:

CD = q_alpha sqrt(k(k+1)/(6N))

Where q_α is based on the studentized range statistic. See references for details.

### Usage

    generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05,
      baseline = NULL, test = "bd")

### Arguments

`bmr` | (BenchmarkResult) Benchmark result.
`measure` | (Measure) Performance measure. Default is the first measure used in the benchmark experiment.
`p.value` | (`numeric(1)`) P-value for the critical difference. Default: 0.05
`baseline` | (`character(1)`): (`learner.id`) Select a `learner.id` as baseline for the `test = "bd"` ("Bonferroni-Dunn") critical differences diagram. The critical difference interval will then be positioned arround this learner. Defaults to best performing algorithm. For `test = "nemenyi"`, no baseline is needed as it performs all pairwise comparisons.
`test` | (`character(1)`) Test for which the critical differences are computed. “bd” for the Bonferroni-Dunn Test, which is comparing all classifiers to a `baseline`, thus performing a comparison of one classifier to all others. Algorithms not connected by a single line are statistically different from the baseline. “nemenyi” for the PMCMR::posthoc.friedman.nemenyi.test which is comparing all classifiers to each other. The null hypothesis that there is a difference between the classifiers can not be rejected for all classifiers that have a single grey bar connecting them.

### Value

(`critDifferencesData`). List containing:

`data` | (data.frame) containing the info for the descriptive part of the plot
`friedman.nemenyi.test` | (list) of class `pairwise.htest` contains the calculated PMCMR::posthoc.friedman.nemenyi.test
`cd.info` | (list) containing info on the critical difference and its positioning
`baseline` | `baseline` chosen for plotting
`p.value` | p.value used for the PMCMR::posthoc.friedman.nemenyi.test and for computation of the critical difference

### See Also

Other generate_plot_data: `generateCalibrationData`, `generateFeatureImportanceData`, `generateFilterValuesData`, `generateLearningCurveData`, `generatePartialDependenceData`, `generateThreshVsPerfData`, `plotFilterValues`

Other benchmark: `BenchmarkResult`, `batchmark`, `benchmark`, `convertBMRToRankMatrix`, `friedmanPostHocTestBMR`, `friedmanTestBMR`, `getBMRAggrPerformances`, `getBMRFeatSelResults`, `getBMRFilteredFeatures`, `getBMRLearnerIds`, `getBMRLearnerShortNames`, `getBMRLearners`, `getBMRMeasureIds`, `getBMRMeasures`, `getBMRModels`, `getBMRPerformances`, `getBMRPredictions`, `getBMRTaskDescs`, `getBMRTaskIds`, `getBMRTuneResults`, `plotBMRBoxplots`, `plotBMRRanksAsBarChart`, `plotBMRSummary`, `plotCritDifferences`, `reduceBatchmarkResults`

[Package mlr version 2.14.0 Index]