Generates data that can be used to plot a critical differences plot. Computes the critical differences according to either the `"Bonferroni-Dunn"` test or the `"Nemenyi"` test. `"Bonferroni-Dunn"` usually yields higher power as it does not compare all algorithms to each other, but all algorithms to a `baseline` instead. Learners are drawn on the y-axis according to their average rank. For `test = "nemenyi"` a bar is drawn, connecting all groups of not significantly different learners. For `test = "bd"` an interval is drawn arround the algorithm selected as baseline. All learners within this interval are not signifcantly different from the baseline. Calculation: $$ CD = q_{\alpha} \sqrt{(\frac{k(k+1)}{6N})}$$ Where \(q_\alpha\) is based on the studentized range statistic. See references for details.
generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05,
baseline = NULL, test = "bd")(BenchmarkResult) Benchmark result.
(Measure) Performance measure. Default is the first measure used in the benchmark experiment.
([numeric`(1)] P-value for the critical difference. Default: 0.05
(`character(1)`): ([learner.id]) Select a `learner.id` as baseline for the `test = "bd"` ("Bonferroni-Dunn") critical differences diagram.The critical difference Interval will then be positioned arround this learner. Defaults to best performing algorithm. For `test = "nemenyi"`, no baseline is needed as it performs `all pairwise comparisons.`
(`character(1)`) Test for which the critical differences are computed. “bd” for the Bonferroni-Dunn Test, which is comparing all classifiers to a `baseline`, thus performing a comparison of one classifier to all others. Algorithms not connected by a single line are statistically different. then the baseline. “nemenyi” for the [PMCMR::posthoc.friedman.nemenyi.test] which is comparing all classifiers to each other. The null hypothesis that there is a difference between the classifiers can not be rejected for all classifiers that have a single grey bar connecting them.
([critDifferencesData]). List containing:
([data.frame]) containing the info for the descriptive part of the plot
([list]) of class `pairwise.htest` contains the calculated posthoc.friedman.nemenyi.test
([list]) containing info on the critical difference and its positioning
`baseline` chosen for plotting
p.value used for the posthoc.friedman.nemenyi.test and for computation of the critical difference
Other generate_plot_data: generateCalibrationData,
generateFeatureImportanceData,
generateFilterValuesData,
generateLearningCurveData,
generatePartialDependenceData,
generateThreshVsPerfData,
getFilterValues,
plotFilterValues
Other benchmark: BenchmarkResult,
batchmark, benchmark,
convertBMRToRankMatrix,
friedmanPostHocTestBMR,
friedmanTestBMR,
getBMRAggrPerformances,
getBMRFeatSelResults,
getBMRFilteredFeatures,
getBMRLearnerIds,
getBMRLearnerShortNames,
getBMRLearners,
getBMRMeasureIds,
getBMRMeasures, getBMRModels,
getBMRPerformances,
getBMRPredictions,
getBMRTaskDescs,
getBMRTaskIds,
getBMRTuneResults,
plotBMRBoxplots,
plotBMRRanksAsBarChart,
plotBMRSummary,
plotCritDifferences,
reduceBatchmarkResults