This function compares two comparators based on the subset of forecasts for which
both comparators have made a prediction. It gets called
from pairwise_comparison_one_group(), which handles the
comparison of multiple comparators on a single set of forecasts (there are no
subsets of forecasts to be distinguished). pairwise_comparison_one_group()
in turn gets called from from get_pairwise_comparisons() which can handle
pairwise comparisons for a set of forecasts with multiple subsets, e.g.
pairwise comparisons for one set of forecasts, but done separately for two
different forecast targets.
compare_forecasts(
scores,
compare = "model",
name_comparator1,
name_comparator2,
metric,
one_sided = FALSE,
test_type = c("non_parametric", "permutation"),
n_permutations = 999
)A list with mean score ratios and p-values for the comparison between two comparators
An object of class scores (a data.table with
scores and an additional attribute metrics as produced by score()).
Character vector with a single colum name that defines the elements for the pairwise comparison. For example, if this is set to "model" (the default), then elements of the "model" column will be compared.
Character, name of the first comparator
Character, name of the comparator to compare against
A string with the name of the metric for which a relative skill shall be computed. By default this is either "crps", "wis" or "brier_score" if any of these are available.
Boolean, default is FALSE, whether two conduct a one-sided
instead of a two-sided test to determine significance in a pairwise
comparison.
Character, either "non_parametric" (the default) or "permutation". This determines which kind of test shall be conducted to determine p-values.
Numeric, the number of permutations for a permutation test. Default is 999.
Johannes Bracher, johannes.bracher@kit.edu
Nikos Bosse nikosbosse@gmail.com