generateCritDifferencesData {mlr}R Documentation

Generate data for critical-differences plot.

Description

Generates data that can be used to plot a critical differences plot. Computes the critical differences according to either the '"Bonferroni-Dunn"' test or the '"Nemenyi"' test.
'"Bonferroni-Dunn"' usually yields higher power as it does not compare all algorithms to each other, but all algorithms to a 'baseline' instead.
Learners are drawn on the y-axis according to their average rank.
For 'test = "nemenyi"' a bar is drawn, connecting all groups of not significantly different learners.
For 'test = "bd"' an interval is drawn arround the algorithm selected as baseline. All learners within this interval are not signifcantly different from the baseline.
Calculation:

CD = q_alpha sqrt(k(k+1)/(6N))


Where q_α is based on the studentized range statistic. See references for details.

Usage

generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05,
  baseline = NULL, test = "bd")

Arguments

bmr

(BenchmarkResult)
Benchmark result.

measure

(Measure)
Performance measure. Default is the first measure used in the benchmark experiment.

p.value

([numeric'(1)]
P-value for the critical difference. Default: 0.05

baseline

('character(1)'): ([learner.id])
Select a 'learner.id' as baseline for the 'test = "bd"' ("Bonferroni-Dunn") critical differences diagram.The critical difference Interval will then be positioned arround this learner. Defaults to best performing algorithm.
For 'test = "nemenyi"', no baseline is needed as it performs 'all pairwise comparisons.'

test

('character(1)')
Test for which the critical differences are computed.
“bd” for the Bonferroni-Dunn Test, which is comparing all classifiers to a 'baseline', thus performing a comparison of one classifier to all others.
Algorithms not connected by a single line are statistically different. then the baseline.
“nemenyi” for the [PMCMR::posthoc.friedman.nemenyi.test] which is comparing all classifiers to each other. The null hypothesis that there is a difference between the classifiers can not be rejected for all classifiers that have a single grey bar connecting them.

Value

([critDifferencesData]). List containing:

data

([data.frame]) containing the info for the descriptive part of the plot

friedman.nemenyi.test

([list]) of class 'pairwise.htest'
contains the calculated posthoc.friedman.nemenyi.test

cd.info

([list]) containing info on the critical difference and its positioning

baseline

'baseline' chosen for plotting

p.value

p.value used for the posthoc.friedman.nemenyi.test and for computation of the critical difference

See Also

Other generate_plot_data: generateCalibrationData, generateFeatureImportanceData, generateFilterValuesData, generateLearningCurveData, generatePartialDependenceData, generateThreshVsPerfData, getFilterValues, plotFilterValues

Other benchmark: BenchmarkResult, batchmark, benchmark, convertBMRToRankMatrix, friedmanPostHocTestBMR, friedmanTestBMR, getBMRAggrPerformances, getBMRFeatSelResults, getBMRFilteredFeatures, getBMRLearnerIds, getBMRLearnerShortNames, getBMRLearners, getBMRMeasureIds, getBMRMeasures, getBMRModels, getBMRPerformances, getBMRPredictions, getBMRTaskDescs, getBMRTaskIds, getBMRTuneResults, plotBMRBoxplots, plotBMRRanksAsBarChart, plotBMRSummary, plotCritDifferences, reduceBatchmarkResults

[Package mlr version 2.12.1 Index]