bestScales forms scales from the items/scales most correlated with a particular criterion and then cross validates on a hold out sample using unit weighted scales. This may be repeated n.iter times using either basic bootstrap aggregation (bagging) techniques or K-fold cross validation. Thus, the technique is known as BISCUIT (Best Items Scales that are Cross validated, Unit weighted, Informative, and Transparent). Given a dictionary of item content, bestScales will sort by criteria correlations and display the item content. Options for bagging (bootstrap aggregation) are included. An alternative to unit weighting is to weight items by their zero order correlations (cross validated) with the criteria. This weighted version is called BISCWIT and is an optional output.
bestScales(x,criteria,min.item=NULL,max.item=NULL, delta = 0,
cut=.1, n.item =10, wtd.cut=0, wtd.n=10,
n.iter =1, folds=1, p.keyed=.9,
overlap=FALSE, dictionary=NULL, check=TRUE, impute="none",log.p=FALSE,digits=2)bestItems(x,criteria=1,cut=.1, n.item=10,raw=TRUE, abs=TRUE,
dictionary=NULL,check=FALSE,digits=2)
A data matrix or data frame depending upon the function.
Which variables (by name or location) should be the empirical target for bestScales and bestItems. May be a separate object.
Find unit weighted and correlation weighted scales from min.item to max.item
These are all summarized in the final.multi.valid object
Return items where the predicted r + delta * se of r < max value
Return all values in abs(x[,c1]) > cut.
When finding the weighted scales, use all items with zero order correlations > wtd.cut
When finding the weighted scales, use the wtd.n items that are > than wtd.cut
Raw data (find the correlations) or a correlation matrix (raw=FALSE)
if TRUE, sort by absolute value in bestItems
a data.frame with rownames corresponding to rownames in the f$loadings matrix or colnames of the data matrix or correlation matrix, and entries (may be multiple columns) of item content.
if TRUE, delete items with no variance
How many items make up an empirical scale, or (bestItems, show the best n.items)
Are the correlations with other criteria fair game for bestScales
When finding the best scales, and thus the correlations with the criteria, how should we handle missing data? The default is to drop missing items. (That is to say, to use pairwise complete correlations.)
How many times to perform a bootstrap estimate. Replicate the best item function n.iter times, sampling roughly 1-1/e of the cases each time, and validating on the remaining 1/e of the cases for each iteration.
If folds > 1, this is k-folds validation. Note, set n.iter > 1 to do bootstrap aggregation, or set folds > 1 to do k-folds.
The proportion of replications needed to include items in the final best keys.
Select items based upon the log of the probability of the correlations. This will only have an effect if the number of pairwise cases differs drastically from pair to pair.
round to digits when showing output.
bestScales returns the correlation of the empirically constructed scale with each criteria and the items used in the scale. If a dictionary is specified, it also returns a list (value) that shows the item content. Also returns the keys.list so that scales can be found using cluster.cor or scoreItems. If using replications (bagging or kfold) then it also returns the best.keys, a list suitable for scoring.
There are actually four keys lists reported.
best.keys are all the items used to form unit weighted scales with the restriction of n.item.
weights may be used in the scoreWtd function to find scales based upon the raw correlation weights.
If the min.item and max.item options are used, then two more sets of weights are provided.
optimal.keys are a subset of the best.keys, taking just those items that increase the cross validation values up to the delta * se of the maximum. This is a slightly more parsimonious set.
optimal.weights is analogous to optimal keys, but for supplies weights for just those items that are used to predict cross validation values up to delta * se of the maximum.
The best.keys object is a list of items (with keying information) that may be used in subsequent analyses. These ``best.keys" are formed into scale scores for the ``final.valid" object which reports how well the best.keys work on the entire sample. This is, of course, not cross validated. Further cross validation can be done using the predict.psych function.
Are the unit weighted scores from the original items
A key list of those items that were used in the unit weighting.
Are the zero-order correlation based scores.
the scoring weights used
An object with the unit weighted and correlation weighted correlations from low.step to high.step
The print and summary output list a number of summary statistics for each criteria. This is given for the default case (number of items fixed) and then if requested, the optimal values chosen from min.item to max.item:
The default statistics:
The optimal number of items statistics:
bestItems returns a sorted list of factor loadings or correlations with the labels as provided in the dictionary.
The stats object can be used to create error.dots plots to show the mean estimate and the standard error of the estimates. See the examples.
The resulting best scales can be cross validated on a different hold out sample using crossValidation. See the last example.
There are a number of procedures that can be used for predicting criteria from a set of predictors. The generic term for this is "machine learning" or "statistical learning". The basic logic of these procedures is to find a set of items that best predict a criteria according to some fit statistic and then cross validate these items numerous times. "lasso" regression (least absolute shrinkage and selection) is one such example. bestScales differs from these procedures by unit weighting items chosen from their zero order correlations with the criteria rather than weighting the partial correlations ala regression. This is an admittedly simple procedure that takes into account the well known finding (Wilks, 1938; Wainer, 1976; Dawes, 1979; Waller, 2008) that although regression weights are optimal for any particular data set, unit weights are almost as good (fungible) and more robust across sample variation.
Following some suggestions, we have added the ability to find scales where items are weighted by their zero order correlations with the criteria. This is effectively a comprimise between unit weighting and regression weights (where the weights are the zero order correlations times the inverse of the correlation matrix). This weighted version may be thought of as BISCWIT in contrast to the unit weighted version BISCUIT.
To be comparable to other ML algorithms, we now consider multiple solutions (for number of items >= min.item to max.item). The final scale consists of the number items which maximize the validity or at least are within delta * standard error of r of the maximum.
Thus, bestScales will find up to n.items per criterion that have absolute correlations with a criterion greater than cut. If the overlap option is FALSE (default) the other criteria are not used. This is an example of ``dust bowl empiricism" in that there is no latent construct being measured, just those items that most correlate with a set of criteria. The empirically identified items are then formed into scales (ignoring concepts of internal consistency) which are then correlated with the criteria.
Clearly, bestScales is capitalizing on chance associations. Thus, we should validate the empirical scales by deriving them on a fraction of the total number of subjects, and cross validating on the remaining subjects. (This is known both as K-fold cross validation and bagging. Both may be done). If folds > 1, then a k-fold cross validation is done. This removes 1/k (a fold) from the sample for the derivation sample and validates on that remaining fold. This is done k-folds times. Traditional cross validation would thus be a k-fold with k =2. More modern applications seem to prefer k =10 to have 90% derivation sample and a 10% cross validation sample.
The alternative, known as 'bagging' is to do a bootstrap sample (which because it is sampling with replacement will typically extract 1- 1/e = 63.2% of the sample) for the derivation sample (the bag) and then validate it on the remaining 1/e of the sample (the out of bag). This is done n.iter times. This should be repeated multiple times (n.iter > 1, e.g. n.iter=1000) to get stable cross validations.
One can compare the validity of these two approaches by trying each. The average predictability of the n.iter samples are shown as are the average validity of the cross validations. This can only be done if x is a data matrix/ data.frame, not a correlation matrix. For very large data sets (e.g., those from SAPA) these scales seem very stable.
bestScales is effectively a straight forward application of 'bagging' (bootstrap aggregation) and machine learning as well as k-fold validation.
The criteria can be the colnames of elements of x, or can be a separate data.frame.
bestItems and lookup are simple helper functions to summarize correlation matrices or factor loading matrices. bestItems will sort the specified column (criteria) of x on the basis of the (absolute) value of the column. The return as a default is just the rowname of the variable with those absolute values > cut. If there is a dictionary of item content and item names, then include the contents as a two column matrix with rownames corresponding to the item name and then as many fields as desired for item content. (See the example dictionary bfi.dictionary).
The derived model can be further validated against yet another hold out sample using the predict.psych function if given the best scale object and the new data set.
Dawes, R.M. (1979) The robust beauty of improper linear models in decision making, American Psychologist, 34, 571-582.
Elleman, L. G., McDougald, S. K., Condon, D. M., & Revelle, W. 2020 (in press). That takes the BISCUIT: A comparative study of predictive accuracy and parsimony of four statistical learning techniques in personality data, with data missingness conditions. European Journal of Psychological Assessment. (Preprint available at https://psyarxiv.com/tuqap/)
Revelle, W. (in preparation) An introduction to psychometric theory with applications in R. Springer. (Available online at https://personality-project.org/r/book/).
Wainer, H. (1979) Estimating coefficients in linear models: It don't make no nevermind. Psychological Buletin, 83, 213-217.
Waller, N.G. (2008), Fungible weights in multiple regression. Psychometrica, 73, 691-703.
Wilks, S. S. (1938), Weighting systems for linear functions of correlated variables when there is no dependent variable. Psychometrika. 3. 23-40.
# NOT RUN {
#This is an example of 'bagging' (bootstrap aggregation)
#not run in order to pass the timing tests for Debian at CRAN
#bestboot <- bestScales(psychTools::bfi,criteria=cs(gender,age,education),
# n.iter=10,dictionary=psychTools::bfi.dictionary[1:3])
#bestboot
#compare with 10 fold cross validation
# }
# NOT RUN {
#don't test for purposes of speed in installation to pass the debian CRAN test
tenfold <- bestScales(psychTools::bfi,criteria=cs(gender,age,education),fold=10,
dictionary= psychTools::bfi.dictionary[1:3])
tenfold
#Then, to display the results graphically
#Note that we scale the two graphs with the same x.lim values
error.dots(tenfold,eyes=TRUE,pch=16,xlim=c(0,.4))
# error.dots(bestboot,add=TRUE,xlim=c(0,.4))
#do this again, but this time display the scale fits from 1 to 15 Items
# tenfold <- bestScales(psychTools::bfi,criteria=cs(gender,age,education),fold=10,
# dictionary= psychTools::bfi.dictionary[1:3],min.item=1,max.item=15)
# matplot(tenfold$multi.validities$wtd.deriv,typ="b",
# xlab="Number of Items",main="Fit as a function of number of items") #the wtd weights
# matpoints(tenfold$multi.validities$unit.deriv,typ="b",lwd=2) #the unit weights
#an example of using crossValidation with bestScales
set.seed(42)
ss <- sample(1:2800,1400)
model <- bestScales(bfi[ss,],criteria=cs(gender,education,age))
original.fit <- crossValidation(model,bfi[ss,]) #the derivation set
cross.fit <- crossValidation(model,bfi[-ss,]) #the cross validation set
summary(original.fit)
summary(cross.fit)
# }
Run the code above in your browser using DataLab