Learn R Programming

datadr: Divide and Recombine in R

datadr is an R package that leverages RHIPE to provide a simple interface to division and recombination (D&R) methods for large complex data.

To get started, see the package documentation and function reference located here.

Visualization tools based on D&R can be found here.

Installation

# from CRAN:
install.packages("datadr")

# from github:
devtools::install_github("delta-rho/datadr")

License

This software is currently under the BSD license. Please read the license document.

Acknowledgement

datadr development is sponsored by:

U.S. Department of Defense Advanced Research Projects Agency, XDATA program
U.S. Department of Homeland Security, Science and Technology Directorate, Homeland Security Advanced Research Projects Agency (HSARPA)
Pacific Northwest National Laboratory, operated by Battelle for the U.S. Department of Energy, LDRD Program, Signature Discovery and Future Power Grid Initiatives

Copy Link

Version

Install

install.packages('datadr')

Monthly Downloads

36

Version

0.8.6.1

License

BSD_3_clause + file LICENSE

Maintainer

Last Published

August 19th, 2018

Functions in datadr (0.8.6.1)

as.data.frame.ddf

Turn 'ddf' Object into Data Frame

Instantiate a Distributed Data Frame ('ddf')

divide-internals

Functions used in divide()

ddo-ddf-accessors

Accessor Functions

Subsetting Distributed Data Frames

Divide a Distributed Data Object

"Flatten" a ddf Subset

Apply Function to Key-Value Pair

Division-Agnostic Aggregation

Specify a Key-Value Pair

Specify Control Parameters for RHIPE Job

Take a Sample of Key-Value Pairs Take a sample of key-value Pairs

Random Replicate Division

Convert dplyr grouped_df to ddf

updateAttributes

Update Attributes of a 'ddo' or 'ddf' Object

Remove Key-Value Pairs from a Data Connection

"Collect" Recombination

"DDF" Recombination

Construct Between Subset Variable (BSV)

Conditioning Variable Division

Convert 'ddo' / 'ddf' Objects

Character File Hash Function

Bag of Little Bootstraps Transformation Method

Mean Coefficient Recombination

Filter a 'ddo' or 'ddf' Object

"rbind" Recombination

GLM Transformation Method

Persist a Transformed 'ddo' or 'ddf' Object

Sample Quantiles for 'ddf' Objects

Specify a Collection of Key-Value Pairs

"Census Income" Dataset

Print a key-value pair

Connect to Data Source on Local Disk

Get Global Variables and Package Dependencies

LM Transformation Method

Apply a function to all key-value pairs of a ddo/ddf object

Apply transformation function(s)

Print value of a key-value pair

ddo-ddf-attributes

Managing attributes of 'ddo' or 'ddf' objects

Instantiate a Distributed Data Object ('ddo')

Add Key-Value Pairs to a Data Connection

localDiskControl

Specify Control Parameters for MapReduce on a Local Disk Connection

Add a Transformation Function to a Distributed Data Object

Accessor methods for 'ddf' objects

Get names of the conditioning variable cuts

Connect to Data Source on HDFS

makeExtractable

Take a ddo/ddf HDFS data object and turn it into a mapfile

HexBin Aggregation for Distributed Data Frames

Join Data Sources by Key

mr-summary-stats

Functions to Compute Summary Statistics in MapReduce

setupTransformEnv

Set up transformation environment

Print a "ddo" or "ddf" Object

Execute a MapReduce Job

Extract "Split" Variable(s)

readHDFStextFile

Experimental HDFS text reader helper function

readTextFileByChunk

Experimental sequential text reader helper function

Turn 'ddo' / 'ddf' Object into a list

Digest File Hash Function

"DDO" Recombination

Mean Recombination