tdmBigLoop: Tuning and unbiased evaluation in a big loop.

Description

For each configuration object .conf in tdm$runList call all tuning algorithms (SPOT, CMA-ES or other) specified in tdm$tuneMethod (via function tdmDispatchTuner). After each tuning process perform a run of tdm$unbiasedFunc (usually unbiasedRun). Each of these experiments is repeated tdm$nExperim times. Thus we have for each tripel

(confName,nExp,theTuner)

a tuning result. The ranges of the triple elements are:

	`confName in tdm$runList`
	`nExp in 1,...,tdm$nExperim`
	`theTuner in tdm$tuneMethod`

Usage

tdmBigLoop(envT, dataObj = NULL)

Arguments

envT

an environment containing on input at least the element tdm (a list with general settings for TDMR, see tdmDefaultsFill), which has at least the elements

tdm$runList: vector of configuration names .conf

dataObj

[NULL] optional object of class TDMdata (the same for all runs in big loop). If it is NULL, it will be constructed here with the help of tdmReadAndSplit. Then it can be different for each configuration object in the big loop.

Value

environment envT, containing the results

res

data frame with results from last tuning (one line for each call of tdmStart*)

bst

data frame with the best-so-far results from last tuning (one line collected after each (SPO) step)

resGrid

list with data frames res from all tuning runs. Use envT$getRes(envT,confFile,nExp,theTuner) to retrieve a specific res.

bstGrid

list with data frames bst from all tuning runs. Use envT$getBst(envT,confFile,nExp,theTuner) to retrieve a specific bst.

theFinals

data frame with one line for each triple (confFile,nExp,tuner), each line contains summary information about the tuning run in the form: confFile tuner nExp [params] NRUN NEVAL RGain.bst RGain.* sdR.* where [params] is written depending on tdm$withParams. NRUN is the number of unbiased evaluation runs. NEVAL is the number of function evaluations (model builds) during tuning. RGain denotes the relative gain on a certain data set: the actual gain achieved with the model divided by the maximum gain possible for the current cost matrix and the current data set. This is for classification tasks, in the case of regression each RGain.* is replaced by RMAE.*, the relative mean absolute error. Each 'sdR.' denotes the standard deviation of the preceeding RGain or RMAE. RGain.bst is the best result during tuning obtained on the training-validation data. RGain.avg is the average result during tuning. The following pairs RGain.* sdR.* are the results of one or several unbiased evaluations on the test data where '*' takes as many values as there are elements in tdm$umode (the possible values are explained in unbiasedRun).

result

object of class TDMclassifier or TDMregressor. This is a list with results from tdm$mainFunc as called in the last unbiased evaluation using the best parameters found during tuning. Use print(envT$result) to get more info on such an object of class TDMclassifier.

tunerVal

an object with the return value from the last tuning process. For every tuner, this is the list spotConfig, containing the SPOT settings plus the TDMR settings in elements opts and tdm. Every tuner extends this list by tunerVal$alg.currentResult and tunerVal$alg.currentBest, see tdmDispatchTuner. In addition, each tuning method might add specific elements to the list, see the description of each tuner.

Environment envT contains further elements, but they are only relevant for the internal operation of tdmBigLoop and its subfunctions.

Details

tdm refers to envT$tdm.

The available tuning algorithms (tuners) are

spotTuner: Call spot.
lhdTuner: Perform a parameter tuning using a Latin hypercube design (LHD) for obtaining best design points. LHD is performed by configuring SPOT in such a way that all the budget is used for the initial design (usually LHD).
cma_jTuner: Perform a parameter tuning by CMA-ES, using the *Java* implementation by Niko Hansen through the interface package rCMA-package.
cmaesTuner: Perform a parameter tuning by CMA-ES, using the *R*-implementation (package cma_es by Olaf Mersmann) (deprecated, use cma_jTuner instead).
bfgsTuner: Perform a parameter tuning by Broyden, Fletcher, Goldfarb and Shanno (BFGS) method. The L-BFGS-B version allowing box constraints is used.

Examples

Run this code

# NOT RUN {
#*# This demo shows a complete tuned data mining process (level 3 of TDMR) where 
#*# the data mining task is the classification task SONAR (from UCI repository, 
#*# http://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+%28Sonar,+Mines+vs.+Rocks%29).
#*# The data mining process is in main_sonar.r, which calls tdmClassifyLoop and tdmClassify
#*# with Random Forest as the prediction model. 
#*# The three parameter to be tuned are CUTOFF1, CLASSWT2 and XPERC, as specified 
#*# in controlSC() (control_sonar.r). The tuner used here is LHD.  
#*# Tuning runs are rather short, to make the example run quickly. 
#*# Do not expect good numeric results. 
#*# See demo/demo03sonar_B.r for a somewhat longer tuning run, with two tuners SPOT and LHD.

  ## path is the dir with data and main_*.r file:
  path <- paste(find.package("TDMR"), "demo02sonar",sep="/");
  #path <- paste("../../inst", "demo02sonar",sep="/");

  ## control settings for TDMR
  tdm <- list( mainFunc="main_sonar"
             , runList = c("sonar_04.conf")
             , umode="CV"              # { "CV" | "RSUB" | "TST" | "SP_T" }
             , tuneMethod = c("lhd")
             , filenameEnvT="exBigLoop.RData"   # file to save environment envT 
             , nrun=1, nfold=2         # repeats and CV-folds for the unbiased runs
             , nExperim=1
             , optsVerbosity = 0       # the verbosity for the unbiased runs
             );
  source(paste(path,"main_sonar.r",sep="/"));    # main_sonar, readTrnSonar
# }
# NOT RUN {
  #*# This demo is for example and help (more meaningful, a bit higher budget)
  source(paste(path,"control_sonar.r",sep="/")); # controlDM, controlSC
# }
# NOT RUN {
  
  
  ctrlSC <- controlSC();
  ctrlSC$opts <- controlDM();
  
  # construct envT from settings given in tdm & sCList
  envT <- tdmEnvTMakeNew(tdm,sCList=list(ctrlSC));
  dataObj <- tdmReadTaskData(envT,envT$tdm);
  envT <- tdmBigLoop(envT,dataObj=dataObj);     # start the big tuning loop 
# }