A classical but still popular approach to detect univariate outlier is the boxplot method introduced by Tukey 1977. The boxplot is a simple graphical tool to display information about continuous univariate data (e.g., median, lower and upper quartile). Outliers are defined as values deviating more than \(1.5 \times IQR\) from the 1st (Q25) or 3rd (Q75) quartile. The strength of Tukey's method is that it makes no distributional assumptions and thus is also applicable to skewed or non mound-shaped data Marsh and Seo, 2006. Nevertheless, this method tends to identify frequent measurements which are falsely interpreted as true outliers.
A somewhat more conservative approach in terms of symmetric and/or normal distributions is the 3SD approach, i.e. any measurement not in the interval of \(mean(x) +/- 3 * \sigma\) is considered an outlier.
Both methods mentioned above are not ideally suited to skewed distributions.
As many biomarkers such as laboratory measurements represent in skewed
distributions the methods above may be insufficient. The approach of Hubert
and Vandervieren 2008 adjusts the boxplot for the skewness of the
distribution. This approach is implemented in several R packages such as
robustbase::mc which is used in this implementation of dataquieR.
Another completely heuristic approach is also included to identify outliers. The approach is based on the assumption that the distances between measurements of the same underlying distribution should homogeneous. For comprehension of this approach:
consider an ordered sequence of all measurements.
between these measurements all distances are calculated.
the occurrence of larger distances between two neighboring measurements may than indicate a distortion of the data. For the heuristic definition of a large distance \(1 * \sigma\) has been been chosen.
Note, that the plots are not deterministic, because they use ggplot2::geom_jitter.
Indicator
acc_univariate_outlier(
resp_vars = NULL,
label_col,
study_data,
meta_data,
exclude_roles,
n_rules = length(unique(criteria)),
max_non_outliers_plot = 10000,
criteria = c("tukey", "3sd", "hubert", "sigmagap")
)a list with:
SummaryTable: data.frame with the columns
Variables, Mean, SD, Median, Skewness, Tukey (N),
3SD (N), Hubert (N), Sigma-gap (N), NUM_acc_ud_outlu,
Outliers, low (N), Outliers, high (N) Grading
SummaryData: data.frame with the columns
Variables, Mean, SD, Median, Skewness, Tukey (N),
3SD (N), Hubert (N), Sigma-gap (N), Outliers (N),
Outliers, low (N), Outliers, high (N) Grading
SummaryPlotList: ggplot univariate outlier plots
variable list the name of the continuous measurement variable
variable attribute the name of the column in the metadata with labels of variables
data.frame the data frame that contains the measurements
data.frame the data frame that contains metadata attributes of study data
variable roles a character (vector) of variable roles not included
integer from=1 to=4. the no. rules that must be violated to flag a variable as containing outliers. The default is 4, i.e. all.
integer from=0. Maximum number of non-outlier points to be plot. If more points exist, a subsample will be plotted only. Note, that sampling is not deterministic.
set tukey | 3SD | hubert | sigmagap. a vector with methods to be used for detecting outliers.
Select all variables of type float in the study data
Remove missing codes from the study data (if defined in the metadata)
Remove measurements deviating from limits defined in the metadata
Identify outliers according to the approaches of Tukey (Tukey 1977), 3SD (Saleem et al. 2021), Hubert (Hubert and Vandervieren 2008), and SigmaGap (heuristic)
An output data frame is generated which indicates the no. possible outliers, the direction of deviations (Outliers, low; Outliers, high) for all methods and a summary score which sums up the deviations of the different rules
A scatter plot is generated for all examined variables, flagging observations according to the no. violated rules (step 5).
Hint: The function is designed for unimodal data only.
acc_robust_univariate_outlier