Iteratively calculate disproportionate impact via the percentage point gap (PPG), proportionality index, and 80% index methods for many success variables, disaggregation variables, and scenarios, using SQL (for data stored in a database or in a parquet data file).
di_iterate_sql(
db_conn,
db_table_name,
success_vars,
group_vars,
cohort_vars = NULL,
scenario_repeat_by_vars = NULL,
exclude_scenario_df = NULL,
weight_var = NULL,
include_non_disagg_results = TRUE,
ppg_reference_groups = "overall",
min_moe = 0.03,
use_prop_in_moe = FALSE,
prop_sub_0 = 0.5,
prop_sub_1 = 0.5,
di_prop_index_cutoff = 0.8,
di_80_index_cutoff = 0.8,
di_80_index_reference_groups = "hpg",
check_valid_reference = TRUE,
parallel = FALSE,
parallel_n_cores = parallel::detectCores()/2,
mssql_flag = FALSE,
return_what = "data",
staging_table = paste0("DisImpact_Staging_", paste0(sample(1:9, size = 5, replace =
TRUE), collapse = "")),
drop_staging_table = TRUE
)When return_what='data' (default), a long data frame is returned (see the return value for di_iterate). When return_what='SQL' (default), a list object where each element is a query (character value) is returned.
A database connection object, returned by dbConnect.
A character value specifying a database table name.
A character vector of success variable names to iterate across.
A character vector of group (disaggregation) variable names to iterate across.
(Optional) A character vector of the same length as success_vars to indicate the cohort variable to be used for each variable specified in success_vars. A vector of length 1 could be specified, in which case the same cohort variable is used for each success variable. If not specified, then a single cohort is assumed for all success variables (defaults to NULL).
(Optional) A character vector of variables to repeat DI calculations for across all combination of these variables. For example, the following variables could be specified:
Ed Goal: Degree/Transfer, Shot-term Career, Non-credit
First time college student: Yes, No
Full-time status: Yes, No
Each combination of these variables (eg, full time, first time college students with an ed goal of degree/transfer as one combination) would constitute an iteration / sample for which to calculate disproportionate impact for outcomes listed in success_vars and for the disaggregation variables listed in group_vars. The overall rate of success for full time, first time college students with an ed goal of degree/transfer would just include these students and not others. Each variable specified is also collapsed to an '- All' group so that the combinations also reflect all students of a particular category. The total number of combinations for the previous example would be (+1 representing the all category): (3 + 1) x (2 + 1) x (2 + 1) = 36.
(Optional) A data frame with variables that match scenario_repeat_by_vars for specifying the combinations to exclude from DI calculations. Following the example specified above, one could choose to exclude part-time non-credit students from consideration.
(Optional) A character variable specifying the weight variable if the input data set is summarized (i.e., the the success variables specified in success_vars contain count of successes). Weight here corresponds to the denominator when calculating the success rate. Defaults to NULL for an input data set where each row describes an individual.
A logical variable specifying whether or not the non-disaggregated results should be returned; defaults to TRUE. When TRUE, a new variable `- None` is added to the data set with a single data value '- All', and this variable is added to group_vars as a disaggregation/group variable. The user would want these results returned to review non-disaggregated results.
Either 'overall', 'hpg', 'all but current', or a character vector of the same length as group_vars that indicates the reference group value for each group variable in group_vars when determining disproportionate impact using the percentage point gap method.
The minimum margin of error to be used in the PPG calculation; see di_ppg.
(TRUE or FALSE) Whether the estimated proportions should be used in the margin of error calculation by the PPG; see di_ppg.
Default is 0.50; see di_ppg.
Default is 0.50; see di_ppg.
Threshold used for determining disproportionate impact using the proportionality index; see di_prop_index; defaults to 0.80.
Threshold used for determining disproportionate impact using the 80% index; see di_80_index; defaults to 0.80.
Either 'overall', 'hpg', 'all but current', or a character vector of the same length as group_vars that indicates the reference group value for each group variable in group_vars when determining disproportionate impact using the 80% index.
(TRUE or FALSE) Check whether ppg_reference_groups and di_80_index_reference_groups contain valid values; defaults to TRUE.
If TRUE, then perform calculations in parallel. The parallel feature is only supported when db_table_name is a path to a parquet file ('/path/to/data.parquet') and that db_conn is a connection to a duckdb database (e.g., dbConnect(duckdb(), dbdir=':memory:')). Defaults to FALSE.
The number of CPU cores to use if parallel=TRUE. Defaults to half of the maximum number of CPU cores on the system.
User-specified logical flag (TRUE or FALSE) that indicates if the MS SQL Server variant of the SQL language should be used.
A character value specifying the return value for the function call. For 'data', the function will return a long data frame with the disproportionate calculations and relevant statistics, after the calculations are performed on the SQL database engine. For 'SQL', a list object of individual queries will be returned for the user to execute elsewhere. Defaults to 'data'.
A character value indicating the name of the staging or results table in the database for storing the disproportionate impact calculations.
TRUE/FALSE A logical flag indicating whether or not the staging table specified in staging_table should be dropped in the database after the results are returned to R; defaults to TRUE.
Iteratively calculate disproportionate impact via the percentage point gap (PPG), proportionality index, and 80% index methods for all combinations of success_vars, group_vars, and cohort_vars, for each combination of subgroups specified by scenario_repeat_by_vars, using SQL (calculations done on the database engine or duckdb for parquet files).