Learn R Programming

irace: Iterated Racing for Automatic Algorithm Configuration

[ Homepage ] [ User Guide (PDF) ]

Maintainers: Manuel López-Ibáñez, Leslie Pérez Cáceres

Creators: Manuel López-Ibáñez, Jérémie Dubois-Lacoste

Contributors: Jérémie Dubois-Lacoste, Thomas Stützle, Mauro Birattari, Eric Yuan and Prasanna Balaprakash.

Contact: https://groups.google.com/d/forum/irace-package


Introduction

The irace package implements the Iterated Race method, which is a generalization of the Iterated F-race method for the automatic configuration of optimization algorithms, that is, the tuning of their parameters by finding the most appropriate settings given a set of instances of an optimization problem. It builds upon the race package by Birattari and it is implemented in R.

You may also find the iraceplot package useful for analyzing the output of irace.

Keywords: automatic configuration, offline tuning, parameter tuning, racing, F-race.

Relevant literature:

  1. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and M. Birattari. The irace package: Iterated Racing for Automatic Algorithm Configuration.

Operations Research Perspectives, 3:43–58, 2016. [ bibtex | doi: 10.1016/j.orp.2016.09.002 ]

  1. Leslie Perez Cáceres, Manuel López-Ibáñez, Holger Hoos, and Thomas Stützle. An Experimental Study of Adaptive Capping in irace. Learning and Intelligent Optimization. LION 2017. Lecture Notes in Computer Science, vol 10556. Springer, Cham. [ bibtex | doi: 10.1007/978-3-319-69404-7_17 ]

  2. Thomas Stützle and Manuel López-Ibáñez. Tutorial: Automated algorithm configuration and design. GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference Companion, July 2021. doi:10.1145/3449726.3461404

Requisites

User guide

A complete user guide comes with the package. You can access it online or, after installing the irace package, invoking from the R console the following command:

        vignette("irace-package")

The following is a quick-start guide. The user guide gives more detailed instructions.

Quick Start

  1. Install R (with your favourite package manager, and see more details below).
  2. Install irace. This command works on Bash shell (Linux and MacOS) and Powershell (Windows) with R added to PATH (see detailed instructions below).
    Rscript -e "install.packages('irace', repos='https://cloud.r-project.org')"
  1. Add irace to the PATH environment variable. This command works in Bash shell (Linux and MacOS). For Windows user, this step is unfortunately more involved, so please see more detailed instructions below.
    export PATH="$(Rscript -e "cat(paste0(system.file(package='irace', 'bin', mustWork=TRUE), ':'))" 2> /dev/null)${PATH}"

Consider adding this line to your ~/.bashrc, ~/.zshrc, or ~/.profile for it to persist between sessions.

  1. You can open the user guide with the following command. This command works on Bash shell (Linux and MacOS) and and Powershell (Windows) with R added to PATH (see detailed instructions below).
    Rscript -e "vignette('irace-package')"

Installing R

The official instructions are available at https://cran.r-project.org/doc/manuals/r-release/R-admin.html. We give below a quick R installation guide that will work in most cases.

GNU/Linux

You should install R from your package manager. On a Debian/Ubuntu system, you will run in the Bash shell:

    sudo apt-get install r-base

Once R is installed, you can launch R from the Terminal and from the R prompt install the irace package. See instructions below.

OS X

You can install R directly from a CRAN mirror (https://cran.r-project.org/bin/macosx/).

Alternatively, if you use homebrew, you can just run from the Terminal (Bash shell):

    brew install --cask r

(Using brew install r is not recommended because that will build R from source and you will not be able to use any CRAN binary, possibly resulting in annoying build failures).

Once R is installed, you can launch R from the Terminal (or from your Applications), and from the R prompt install the irace package. See instructions below.

Windows

You can install R from a CRAN mirror (https://cran.r-project.org/bin/windows/). Once R is installed, you can launch the R console and install the irace package from it. See instructions below.

In addition to using the R console, it might be very useful to add R to PATH so you can run R commands in CMD or Powershell. Usually, R is installed in C:\Program Files\R\R-4.1.3 (the version number depends on your installation).

You should add the following line to PATH (if you want to use the 64-bit version)

C:\Program Files\R\R-4.1.3\bin\x64

Or, if you are on a 32-bit version

C:\Program Files\R\R-4.1.3\bin\i386

Installing the irace package

There are two methods for installing the irace R package on your computer:

  1. Install within R (automatic download). Run the following from the R console:
        install.packages("irace", repos = "https://cloud.r-project.org")

and test the installation with:

        library(irace)
        irace_cmdline("--version")
        q()
  1. Manually download the package from CRAN and invoke at the Bash shell, Terminal or Powershell:
        R CMD INSTALL <package>

where <package> is one of the three versions available: .tar.gz (Unix/BSD/GNU/Linux), .tgz (MacOS X), or .zip (Windows).

If the package fails to install because of insufficient permissions, you need to force a local installation by typing in the Bash shell:

    mkdir ~/R
    R CMD INSTALL --library=~/R irace.tar.gz
    export R_LIBS=~/R:${R_LIBS}

Once installed, test that it is working by typing in the R console (not in the bash shell):

    library(irace)
    irace_cmdline("--version")
    cat(system.file(package="irace", "bin", mustWork=TRUE), "\n")

The last command gives you the installation folder of irace, for example, /home/user/R/irace/bin.

GNU/Linux and OS X

Save the installation directory of irace to a variable, and add it to your .bash_profile, .bashrc or .profile:

    export IRACE_HOME=/home/user/R/irace/bin/ # Path given by system.file(package="irace", "bin", mustWork=TRUE)
    export PATH=${IRACE_HOME}:$PATH
    # export R_LIBS=~/R:${R_LIBS} # Only if local installation was forced

After adding this and opening a new terminal, you should be able to invoke irace as follows:

    irace --help

Windows

You can find out where the irace binary is installed by running the following in Powershell or CMD:

    C:\> Rscript -e "cat(gsub('/', '\\\\', system.file(package='irace', 'bin', 'x64', mustWork=TRUE)))"

It will output a path, such as C:\Program Files\R\R-4.1.3\library\irace\bin\x64 (replace x64 with i386 if you are on a 32-bit system), which can you add to PATH.

Then running the following should work:

    C:\> irace --help

You can also launch irace by opening the R console and executing:

    library(irace)
    irace_cmdline("--help")

GitHub (Development version)

If you wish to try the development version, you can install it by executing the following command within the R console:

    install.packages('irace', repos = c('https://mlopez-ibanez.r-universe.dev', 'https://cloud.r-project.org'))

Python

You can use the irace R package from Python using rpy2. There are actually two different implementations of this idea: iracepy and iracepy-tiny.

Rust

You can use irace from Rust by using irace-rs.

Usage

  1. Create a directory for storing the tuning scenario setup (Bash shell):
        mkdir ./tuning
        cd ./tuning
  1. Initialize your tuning directory with template config files (Bash shell):
        $IRACE_HOME/bin/irace --init
  1. Modify the generated files following the instructions found within each file. In particular,

    • The scripts target-runner and target-evaluator (if you need it at all) should be executable. The output of target-runner (or target-evaluator if you use a separate evaluation step) is minimized by default. If you wish to maximize it, just multiply the value by -1 within the script.
    • In scenario.txt, uncomment and assign only the parameters for which you need a value different than the default one. For example, you may need to set trainInstancesDir="./Instances/".

    There are examples in $IRACE_HOME/examples/.

  2. Put the instances in ./tuning/Instances/. In addition, you can create a file that specifies which instances from that directory should be run and which instance-specific parameters to use. See scenario.txt and instances-list.txt for examples. The command irace will not attempt to create the execution directory (execDir), so it must exist before calling irace. The default execDir is the current directory.

  3. Calling the command in the Bash shell:

        cd ./tuning/ && $IRACE_HOME/bin/irace

performs one run of Iterated Race. See the output of irace --help for additional irace parameters. Command-line parameters override the scenario setup specified in the scenario.txt file.

Many tuning runs in parallel

For executing several repetitions of irace in parallel, call the program parallel-irace from the Bash shell:

    cd ./tuning/ && $IRACE_HOME/bin/parallel-irace N

where N is the number of repetitions. By default, the execution directory of each run of irace will be set to ./execdir-dd, where dd is a number padded with zeroes.

Be careful, parallel-irace will create these directories from scratch, deleting them first if they already exist.

Check the help of parallel-irace by running it without parameters.

Parallelize one tuning

A single run of irace can be done much faster by executing the calls to targetRunner (the runs of the algorithm being tuned) in parallel. See the user guide for the details.

License

irace is Copyright (C) 2010-2020 Manuel López-Ibáñez, Jérémie Dubois-Lacoste and Leslie Pérez-Cáceres.

This program is free software (software libre); you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

IMPORTANT NOTE: Please be aware that the fact that this program is released as Free Software does not excuse you from scientific propriety, which obligates you to give appropriate credit! If you write a scientific paper describing research that made substantive use of this program, it is your obligation as a scientist to (a) mention the fashion in which this software was used in the Methods section; (b) mention the algorithm in the References section. The appropriate citation is:

  • Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas

Stützle, and Mauro Birattari. The irace package: Iterated Racing for Automatic Algorithm Configuration. Operations Research Perspectives, 2016. doi: 10.1016/j.orp.2016.09.002

The irace package uses code under the GPL from the race package is Copyright (C) 2003 Mauro Birattari.

Building an irace standalone container

Thanks to Singularity, you can build a standalone container of irace using the file irace.sindef which is available in the directory inst/ in the source tarball and github repository or, after installing the irace R package, in the installation directory given by the R expression system.file(package="irace"). After installing SingularityCE, the container may be build using:

sudo singularity build irace.sindef irace.sif

and run with:

singularity run irace.sif <arguments>

Frequently Asked Questions

The user guide contains a list of frequently asked questions.

Copy Link

Version

Install

install.packages('irace')

Monthly Downloads

2,002

Version

4.1

License

GPL (>= 2)

Issues

Pull Requests

Stars

Forks

Last Published

December 17th, 2024

Functions in irace (4.1)

irace_main

Higher-level interface to launch irace.
has_testing_data

Check if the results object generated by irace has data about the testing phase.
getFinalElites

Return the elite configurations of the final iteration.
get_instanceID_seed_pairs

Returns the pairs of instance IDs and seeds used as instances in the race (and optionally the actual instances).
irace_cmdline

Launch irace with command-line options.
getConfigurationByIteration

Returns the configurations by the iteration in which they were executed.
irace_license

irace_license
irace

Execute one run of the Iterated Racing algorithm.
getConfigurationById

Returns the configurations selected by ID.
irace-package

The irace package: tools:::Rd_package_title("irace")
multi_irace

Execute irace() multiple times with the same or different scenarios and parameter space definitions.
psRace

Postselection race
printParameters

Print parameter space in the textual format accepted by irace.
irace_summarise

Summarise the results of a run of irace
path_rel2abs

Converts a relative path to an absolute path.
random_seed

Get, set and restore the state of the random number generator state.
plotAblation

Create plot from an ablation log
irace_version

A character string containing the version of irace including git SHA.
read_ablogfile

Read the log file (log-ablation.Rdata) produced by ablation().
readScenario

Reads from a file the scenario settings to be used by irace.
readConfigurationsFile

Read parameter configurations from a file
target_runner_default

Default targetRunner function.
testConfigurations

Execute the given configurations on the testing instances specified in the scenario
testing_fromlog

Test configurations given in .Rdata file
target_evaluator_default

target_evaluator_default
testing_fromfile

Test configurations given an explicit table of configurations and a scenario file
readParameters

Reads the parameters to be tuned by irace from a file or from a character string.
read_logfile

Read the log file produced by irace (irace.Rdata).
read_pcs_file

Read parameters in PCS (AClib) format and write them in irace format.
parameters

Create a parameter space to be tuned.
removeConfigurationsMetaData

removeConfigurationsMetaData
printScenario

Prints the given scenario
scenario_update_paths

Update filesystem paths of a scenario consistently.
checkScenario

Check and correct the given scenario
check_output_target_runner

Check the output of the target runner and repair it if possible. If the output is incorrect, this function will throw an error.
checkIraceScenario

Test that the given irace scenario can be run.
configurations_print_command

Print configurations as command-line strings.
defaultScenario

Default scenario settings
checkParameters

checkParameters
buildCommandLine

Generate a command-line representation of a configuration
ablation_cmdline

Launch ablation with command-line options.
configurations_print

Print configurations as a data frame
ablation

Performs ablation between two configurations (from source to target).