Functions with the nCores, i, and j parameters provide
capabilities for both parallel and distributed computing.
For parallel computing, nCores determines the number of cores the
code is run on. Memory usage can be an issue for higher values of
nCores as R is not particularly memory-efficient. As a rule of
thumb, at least around (nCores * object_size(chunk)) +
object_size(result) MB of total memory will be needed for operations
on file-backed matrices, not including potential copies of your data that
might be created (for example lsfit runs cbind(1, X)).
i and j can be used to include or exclude certain rows or
columns. Internally, the mclapply function is used and therefore
parallel computing will not work on Windows machines.
For distributed computing, i and j determine the subset of
the input matrix that the code runs on. In an HPC environment, this can be
used not just to include or exclude certain rows or columns, but also to
partition the task among many nodes rather than cores. Scheduler-specific
code and code to aggregate the results need to be written by the user. It
is recommended to set nCores to 1 as nodes are often cheaper
than cores.
mclapply to learn more about the function used to
implement parallel computing. detectCores to detect
the number of available cores.