secr (version 3.0.1)

Parallel: Multi-core Processing

Description

From version 2.4.0 secr makes limited use of multiple cores (CPUs) through the package parallel. Only the few secr functions listed below make any use of parallel processing. Increased speed can be expected with sim.secr (e.g., x3 with 4 cores), but gains in secr.fit are much smaller and may be negative.

The function par.secr.fit added in 2.7.1 is useful if you have a set of different models you wish to fit: if you have as many cores as models then all will complete in about the time required to run the longest.

Function	Unit	Benefit	Notes
secr.fit	session likelihood	small-moderate	multi-session models only
score.test	model	moderate	multi-model comparisons only
derived	session	moderate	SE by parameter if one session
mask.check	spacing x buffer	moderate-large	no file output, suppresses messages
sim.secr	replicate	large	all models, suppresses messages
ip.secr	replicate	large
LLsurface.secr	parameter combination	large
par.secr.fit	model	large
par.derived	fitted model	large
par.region.N	fitted model	large
fxi.contour	individual	moderate-large	multiple individuals only
fxi.secr	individual	moderate-large	multiple individuals only
fx.total	individual	moderate-large
pmixProfileLL	level of pmix	large

`Unit' refers to the unit of work sent to each worker process. As a guide, a `large' benefit means >60% reduction in process time with 4 CPUs.

parallel offers several different mechanisms, bringing together the functionality of multicore and snow. The mechanism used by secr is the simplest available, and is expected to work across all operating systems. Technically, it relies on Rscript and communication between the master and worker processes is via sockets. As stated in the parallel documentation "Users of Windows and Mac OS X may expect pop-up dialog boxes from the firewall asking if an R process should accept incoming connections".

To use multiple cores, install parallel from CRAN and set ncores > 1 in the function call. Use detectCores() to get an idea of how many cores are available on your machine; this may (in Windows) include virtual cores over and above the number of physical cores. See RShowDoc("parallel", package = "parallel") in core R for explanation.

You may possibly get warnings from R about closing unused connections. These can safely be ignored.

In sim.secr, new datasets are generated in the master process, so there is no need to manage the random number streams in the worker processes.

In secr.fit the output component `proctime' misrepresents the elapsed processing time when multiple cores are used.

Worker processes are created in secr.fit with makeCluster and the options methods = FALSE, useXDR = .Platform$endian=='big' (consistently from 2.10.0). methods = TRUE is required for S4 methods, but as these are not used directly in secr it seems worth saving the time needed to load methods. useXDR = FALSE saves time ``where large amounts of data are to be transferred and all the nodes are little-endian'', which applies on Windows systems and some linux systems. Please report any issues.

Warning

It appears that multicore operations in secr using parallel may fail if the packages snow and snowfall are also loaded. The error message is obscure:

``Error in UseMethod("sendData") : no applicable method for 'sendData' applied to an object of class "SOCK0node"''

Examples

## Not run: ------------------------------------
# 
# sessionInfo()
# # R version 3.2.0 (2015-04-16)
# # Platform: x86_64-w64-mingw32/x64 (64-bit)
# # Running under: Windows 7 x64 (build 7601) Service Pack 1
# # quad-core i7 CPU, 16 Gb RAM
# # ...
# 
# ## almost no benefit with secr.fit
# 
# for (i in 1:5) 
#     print(system.time(secr.fit(ovenCH, buffer = 400, trace = FALSE, ncores = i)))
# 
# #   user  system elapsed 
# #  52.26    0.76   53.02 
# #   user  system elapsed 
# #   3.38    2.77   48.36 
# #   user  system elapsed 
# #   5.69    5.36   47.18 
# #   user  system elapsed 
# #   6.50    6.90   54.51 
# #   user  system elapsed 
# #   7.74    8.25   50.23 
# 
# ## better for simulation, up to a point
#  
# for (i in 1:8)
#     print(system.time(sim.secr(secrdemo.0, nsim = 20, tracelevel = 0, ncores = i)))
# 
# #   user  system elapsed 
# # 262.93    3.74  266.71 
# #   user  system elapsed 
# #   0.68    0.02  139.61 
# #   user  system elapsed 
# #   0.69    0.01  110.05 
# #   user  system elapsed 
# #   0.70    0.00   84.24 
# #   user  system elapsed 
# #   0.70    0.02   69.45 
# #   user  system elapsed 
# #   0.67    0.06   75.67 
# #   user  system elapsed 
# #   0.75    0.00   70.00 
# #   user  system elapsed 
# #   0.73    0.05   75.71 
#  
# for (i in 1:8) 
#     print(system.time(ip.secr (captdata, trace = FALSE, ncores = i)))
# 
# #   user  system elapsed 
# # 132.59    0.00  132.58 
# #   user  system elapsed 
# #   0.25    0.05   71.08 
# #   user  system elapsed 
# #   0.20    0.04   50.76 
# #   user  system elapsed 
# #   0.32    0.07   42.13 
# #   user  system elapsed 
# #   0.26    0.09   41.74 
# #   user  system elapsed 
# #   0.39    0.08   39.01 
# #   user  system elapsed 
# #   0.25    0.09   36.52 
# #   user  system elapsed 
# #   0.46    0.08   35.91 
#  
# for (i in 1:8)
#     print(system.time(LLsurface.secr(secrdemo.0, ncores = i)))
#     
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #  48.00    0.40   48.41 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.00    0.03   27.94 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.00    0.03   20.60 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.03    0.02   17.63 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.01    0.05   16.41 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.02    0.04   16.10 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.05    0.03   16.03 
# #Evaluating log likelihood across grid of 121 points...
# #   user  system elapsed 
# #   0.02    0.07   15.99 
# 
## ---------------------------------------------