drQuantile: Sample Quantiles for 'ddf' Objects

Description

Compute sample quantiles for 'ddf' objects

Usage

drQuantile(x, var, by = NULL, probs = seq(0, 1, 0.005), preTransFn = NULL,
  varTransFn = identity, varRange = NULL, nBins = 10000, tails = 100,
  params = NULL, packages = NULL, control = NULL, ...)

Arguments

a 'ddf' object

var

the name of the variable to compute quantiles for

an optional variable name or vector of variable names by which to group quantile computations

probs

numeric vector of probabilities with values in [0-1]

preTransFn

a transformation function (if desired) to applied to each subset prior to computing quantiles (here it may be useful for adding a "by" variable that is not present) - note: this transformation should not modify var (use varTransFn for that) - also note: this is deprecated - instead use addTransform prior to calling divide

varTransFn

transformation to apply to variable prior to computing quantiles

varRange

range of x (can be left blank if summaries have been computed)

nBins

how many bins should the range of the variable be split into?

tails

how many exact values at each tail should be retained?

params

a named list of objects external to the input data that are needed in the distributed computing (most should be taken care of automatically such that this is rarely necessary to specify)

packages

a vector of R package names that contain functions used in fn (most should be taken care of automatically such that this is rarely necessary to specify)

control

parameters specifying how the backend should handle things (most-likely parameters to rhwatch in RHIPE) - see rhipeControl and localDiskControl

…

additional arguments

Value

data frame of quantiles q and their associated f-value fval. If by is specified, then also a variable group.

Details

This division-agnostic quantile calculation algorithm takes the range of the variable of interest and splits it into nBins bins, tabulates counts for those bins, and reconstructs a quantile approximation from them. nBins should not get too large, but larger nBins gives more accuracy. If tails is positive, the first and last tails ordered values are attached to the quantile estimate - this is useful for long-tailed distributions or distributions with outliers for which you would like more detail in the tails.

Examples

Run this code

# NOT RUN {
# break the iris data into k/v pairs
irisSplit <- list(
  list("1", iris[1:10,]), list("2", iris[11:110,]), list("3", iris[111:150,])
)
# represent it as ddf
irisSplit <- ddf(irisSplit, update = TRUE)

# approximate quantiles over the divided data set
probs <- seq(0, 1, 0.005)
iq <- drQuantile(irisSplit, var = "Sepal.Length", tails = 0, probs = probs)
plot(iq$fval, iq$q)

# compare to the all-data quantile "type 1" result
plot(probs, quantile(iris$Sepal.Length, probs = probs, type = 1))

# }