Generates data that can be used to plot a
critical differences plot. Computes the critical differences according
to either the
"Bonferroni-Dunn"
test or the "Nemenyi"
test.
"Bonferroni-Dunn"
usually yields higher power as it does not
compare all algorithms to each other, but all algorithms to a
baseline
instead.
Learners are drawn on the y-axis according to their average rank.
For test = "nemenyi"
a bar is drawn, connecting all groups of not
significantly different learners.
For test = "bd"
an interval is drawn arround the algorithm selected
as a baseline. All learners within this interval are not signifcantly different
from the baseline.
Calculation:
$$CD = q_{\alpha} \sqrt{\left(\frac{k(k+1)}{6N}\right)}$$
Where \(q_\alpha\) is based on the studentized range statistic.
See references for details.
generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05,
baseline = NULL, test = "bd")
(BenchmarkResult) Benchmark result.
(Measure) Performance measure. Default is the first measure used in the benchmark experiment.
(numeric(1)
)
P-value for the critical difference. Default: 0.05
(character(1)
): (learner.id
)
Select a learner.id
as baseline for the test = "bd"
("Bonferroni-Dunn") critical differences
diagram. The critical difference interval will then be positioned arround this learner.
Defaults to best performing algorithm.
For test = "nemenyi"
, no baseline is needed as it performs all pairwise
comparisons.
(character(1)
)
Test for which the critical differences are computed.
“bd” for the Bonferroni-Dunn Test, which is comparing all
classifiers to a baseline
, thus performing a comparison
of one classifier to all others.
Algorithms not connected by a single line are statistically different
from the baseline.
“nemenyi” for the PMCMR::posthoc.friedman.nemenyi.test
which is comparing all classifiers to each other. The null hypothesis that
there is a difference between the classifiers can not be rejected for all
classifiers that have a single grey bar connecting them.
(critDifferencesData
). List containing:
(data.frame) containing the info for the descriptive part of the plot
(list) of class pairwise.htest
contains the calculated
PMCMR::posthoc.friedman.nemenyi.test
(list) containing info on the critical difference and its positioning
baseline
chosen for plotting
p.value used for the PMCMR::posthoc.friedman.nemenyi.test and for computation of the critical difference
Other generate_plot_data: generateCalibrationData
,
generateFeatureImportanceData
,
generateFilterValuesData
,
generateLearningCurveData
,
generatePartialDependenceData
,
generateThreshVsPerfData
,
getFilterValues
,
plotFilterValues
Other benchmark: BenchmarkResult
,
batchmark
, benchmark
,
convertBMRToRankMatrix
,
friedmanPostHocTestBMR
,
friedmanTestBMR
,
getBMRAggrPerformances
,
getBMRFeatSelResults
,
getBMRFilteredFeatures
,
getBMRLearnerIds
,
getBMRLearnerShortNames
,
getBMRLearners
,
getBMRMeasureIds
,
getBMRMeasures
, getBMRModels
,
getBMRPerformances
,
getBMRPredictions
,
getBMRTaskDescs
,
getBMRTaskIds
,
getBMRTuneResults
,
plotBMRBoxplots
,
plotBMRRanksAsBarChart
,
plotBMRSummary
,
plotCritDifferences
,
reduceBatchmarkResults