parscores: Penalized average runtime score

Description

Calculates the penalized average runtime score which is commonly used for evaluating satisfiability solvers on a set of problems.

Usage

parscores(data, model, factor = 10, timeout, addCosts = NULL)

Arguments

data

the data used to induce the model. The same as given to classify, classifyPairs, cluster or regression.

model

the algorithm selection model. Can be either a model returned by one of the model-building functions or a function that returns predictions such as vbs or the predictor function of a trained model.

factor

the penalization factor to use for non-successful choices. Default 10.

timeout

the timeout value to be multiplied by the penalization factor. If not specified, the maximum performance value of all algorithms on the entire data is used.

addCosts

whether to add feature costs. You should not need to set this manually, the default of NULL will have LLAMA figure out automatically depending on the model whether to add costs or not. This should always be true (the default) except for comparison algorithms (i.e. single best and virtual best).

Value

A list of the penalized average runtimes.

Details

Returns the penalized average runtime performances of the respective chosen algorithm on each problem instance.

If feature costs have been given and addCosts is TRUE, the cost of the used features or feature groups is added to the performance of the chosen algorithm. The used features are determined by examining the the features member of data, not the model. If after that the performance value is above the timeout value, the timeout value multiplied by the factor is assumed.

If the model returns NA (e.g. because no algorithm solved the instance), timeout * factor is returned as PAR score.

data may contain a train/test partition or not. This makes a difference when computing the PAR scores for the single best algorithm. If no train/test split is present, the single best algorithm is determined on the entire data. If it is present, the single best algorithm is determined on each test partition. That is, the single best is local to the partition and may vary across partitions.

Examples

Run this code

# NOT RUN {
if(Sys.getenv("RUN_EXPENSIVE") == "true") {
data(satsolvers)
folds = cvFolds(satsolvers)

model = classify(classifier=makeLearner("classif.J48"), data=folds)
sum(parscores(folds, model))

# use factor of 5 instead of 10.
sum(parscores(folds, model, 5))

# explicitly specify timeout.
sum(parscores(folds, model, timeout = 3600))
}
# }