psych (version 2.2.3)

bigCor: Find large correlation matrices by stitching together smaller ones found more rapidly

Description

When analyzing many subjects (ie. 100,000 or more) with many variables (i.e. 1000 or more) core R can take a long time and sometime exceed memory limits (i.e. with 600K subjects and 6K variables). bigCor runs (in parallel if multicores are available) by breaking the variables into subsets (of size=size), finding all subset correlations, and then stitches the resulting matrices into one large matrix. Noticeable improvements in speed compared to cor.

Usage

bigCor(x, size = NULL, use = "pairwise",cor="pearson")

Arguments

x

A data set of numeric variables

size

What should the size of the subsets be? Defaults to NCOL (x)/4

use

The standard correlation option. "pairwise" allows for missing data

cor

Defaults to Pearson correlations, alterative is polychoric.

Value

The correlation matrix

Details

The data are divided into subsets of size=size. Correlations are then found for each subset and pairs of subsets. The basic loop loops over the subsets. When the size is a integer subset of the number of variables and is a muiltiple of the number of cores, the multiple cores will be used more. Notice the benefit of 660/80 versus 660/100. But this breaks down if we try 660/165. Further notice the benefit when using a smaller subset (55) which led to the 4 cores being used more.

Timings (in seconds) for various problems with 645K subjects on an 8 core Mac Book Pro with a 2.4 GHZ Intell core i9.

options(mc.cores=4) (Because we have 8 we can work at the same time as we test this.)

First test it with 644,495 subjects and 1/10 of the number of possible variables. Then test it for somewhat fewer variables.

Variables	size	2 cores	4 cores	compared to normal cor function
660	100	430	434	430
660	80	600	348	notice the improvement with 8ths
660	165		666	(Stitching seems to have been very slow)
660	55		303	Even better if we break it into 12ths!
	500	100		332
322 secs		480	120	408
365	315 Better to change the size	480	60	358
206	This leads to 8 splits		Variables	size

We also test it with fewer subjects. Time is roughly linear with number of subjects.

Variables	size	2 cores	4 cores	compared to normal cor function Further comparisons with fewer subjects (100K)
	480	60	57	31
47 with normal cor. Note the effect of n subjects!	200	50	19.9	13.6
27.13	100	25	4.6	3.5
5.85		Variables	size	2 cores

One last comparison, 10,000 subjects, showing the effect of getting the proper size value. You can tune on these smaller sets of subjects before trying large problems.

Variables	size	2 cores	4 cores	compared to normal cor function
480	120	5.2	5.1	4.51
480	60	2.9	2.88	4.51
480	30	2.65	2.691	480
20	2.73	2.77		480
10	2.82	2.97	too many splits?	200
50	2.18	1.39	2.47 for normal cor (1.44 with 8 cores 2.99 with 1 core)	200
25	1.2	1.17	2.47 for normal cor (1.16 with 8 cores, 1.17 with 1 core)	100
25	.64	.52	.56	Variables

Time is roughly linear with the number of cases and increases by the square of the number of variables. The benefit of more cores is noticeable. It seems as if with 4 cores, we should use sizes to split it into 8 or 12 sets. Otherwise we don't actually use all cores efficiently.

There is some overhead in using multicores. So for smaller problems (e.g. the 4,000 cases of the 145 items of the psychTools::spi data set, the timings are roughly .44 seconds for bigCor (default size) and .36 for normal cor. But, fiddling with size gets us to .37 for size = 40. The cross over point seems to be at roughly 5K subjects.

References

Examples of large data sets with massively missing data are taken from the SAPA project. e.g.,

William Revelle, Elizabeth M. Dworak, and David M. Condon (2021) Exploring the persome: The power of the item in understanding personality structure. Personality and Individual Differences, 169, 10.1016/j.paid.2020.109905

David Condon (2018)The SAPA Personality Inventory: an empirically-derived, hierarchically-organized self-report personality assessment model. PsyArXiv /sc4p9/ 10.31234/osf.io/sc4p9

See Also

pairwiseCountBig which will do the same, but find the count of observations per cell.

Examples

# NOT RUN {
R <- bigCor(psychTools::bfi,10)
#compare the results with 
r.bfi <- cor(psychTools::bfi,use="pairwise")
all.equal(R,r.bfi)
# }