CFUN

ccbind

crbind

cfun

cquantile

csummary

cmedian

clength

csum

cmean

These are used in aggregating the chunks resulting from batch processing. They are usually called via <code class="command"><a href="/link/do.call?package=ff&version=4.5.0" data-mini-rdoc="ff::do.call">do.call</a></code>

 manip 

 list 

The ff package provides data structures that are stored on
disk but behave (almost) as if they were in RAM by transparently
mapping only a section (pagesize) in main memory - the effective
virtual memory consumption per ff object. ff supports R's standard
atomic data types 'double', 'logical', 'raw' and 'integer' and
non-standard atomic types boolean (1 bit), quad (2 bit unsigned),
nibble (4 bit unsigned), byte (1 byte signed with NAs), ubyte (1 byte
unsigned), short (2 byte signed with NAs), ushort (2 byte unsigned),
single (4 byte float with NAs). For example 'quad' allows efficient
storage of genomic data as an 'A','T','G','C' factor. The unsigned
types support 'circular' arithmetic. There is also support for
close-to-atomic types 'factor', 'ordered', 'POSIXct', 'Date' and
custom close-to-atomic types.
ff not only has native C-support for vectors, matrices and arrays
with flexible dimorder (major column-order, major row-order and
generalizations for arrays). There is also a ffdf class not unlike
data.frames and import/export filters for csv files.
ff objects store raw data in binary flat files in native encoding,
and complement this with metadata stored in R as physical and virtual
attributes. ff objects have well-defined hybrid copying semantics,
which gives rise to certain performance improvements through
virtualization. ff objects can be stored and reopened across R
sessions. ff files can be shared by multiple ff R objects
(using different data en/de-coding schemes) in the same process
or from multiple R processes to exploit parallelism. A wide choice of
finalizer options allows to work with 'permanent' files as well as
creating/removing 'temporary' ff files completely transparent to the
user. On certain OS/Filesystem combinations, creating the ff files
works without notable delay thanks to using sparse file allocation.
Several access optimization techniques such as Hybrid Index
Preprocessing and Virtualization are implemented to achieve good
performance even with large datasets, for example virtual matrix
transpose without touching a single byte on disk. Further, to reduce
disk I/O, 'logicals' and non-standard data types get stored native and
compact on binary flat files i.e. logicals take up exactly 2 bits to
represent TRUE, FALSE and NA.
Beyond basic access functions, the ff package also provides
compatibility functions that facilitate writing code for ff and ram
objects and support for batch processing on ff objects (e.g. as.ram,
as.ff, ffapply). ff interfaces closely with functionality from package
'bit': chunked looping, fast bit operations and coercions between
different objects that can store subscript information ('bit',
'bitwhich', ff 'boolean', ri range index, hi hybrid index). This allows
to work interactively with selections of large datasets and quickly
modify selection criteria.
Further high-performance enhancements can be made available upon request.

Jens Oehlschl�gel

Memory-Efficient Storage of Large Data on Disk and Fast Access
Functions

Daniel Adler

Christian Gläser

Oleg Nenadic

Jens Oehlschlägel

Martijn Schuemie

Walter Zucchini

CFUN function

<dl> <dt>...</dt>
<dd><code>...</code></dd> <dt>FUN</dt>
<dd>a aggregating function</dd> <dt>FUNARGS</dt>
<dd>further arguments to the aggregating function</dd> <dt>na.rm</dt>
<dd>TRUE to remove NAs</dd> <dt>probs</dt>
<dd>see <code><a href="/link/quantile?package=ff&version=4.5.0" data-mini-rdoc="ff::quantile">quantile</a></code></dd> <dt>names</dt>
<dd>see <code><a href="/link/quantile?package=ff&version=4.5.0" data-mini-rdoc="ff::quantile">quantile</a></code></dd> <dt>type</dt>
<dd>see <code><a href="/link/quantile?package=ff&version=4.5.0" data-mini-rdoc="ff::quantile">quantile</a></code></dd></dl>

Arguments

xx TODO: extend this for weighted means, weighted median etc., 
 google "Re: [R] Weighted median"

ff options

Author

These are used in aggregating the chunks resulting from batch processing. They are usually called via <code class='command'><a href='https://rdrr.io/r/base/do.call.html'>do.call</a></code>

Collapsing functions for batch processing — CFUN

<dl>

 <dt>...</dt>
<dd><code>...</code></dd>

 <dt>FUN</dt>
<dd>a aggregating function</dd>

 <dt>FUNARGS</dt>
<dd>further arguments to the aggregating function</dd>

 <dt>na.rm</dt>
<dd>TRUE to remove NAs</dd>

 <dt>probs</dt>
<dd>see <code><a href='https://rdrr.io/r/stats/quantile.html'>quantile</a></code></dd>

 <dt>names</dt>
<dd>see <code><a href='https://rdrr.io/r/stats/quantile.html'>quantile</a></code></dd>

 <dt>type</dt>
<dd>see <code><a href='https://rdrr.io/r/stats/quantile.html'>quantile</a></code></dd>

</dl>

xx TODO: extend this for weighted means, weighted median etc., 
 google "Re: [R] Weighted median"

CFUN	FUN	comment
`ccbind`	`cbind`	like `cbind` but respecting names
`crbind`	`rbind`	like `rbind` but respecting names
`cfun`		`crbind` the input chunks and then apply 'FUN' to each column
`cquantile`	`quantile`	`crbind` the input chunks and then apply 'quantile' to each column
`csummary`	`summary`	`crbind` the input chunks and then apply 'summary' to each column
`cmedian`	`median`	`crbind` the input chunks and then apply 'median' to each column
`clength`	`length`	`crbind` the input chunks and then determine the number of values in each column
`csum`	`sum`	`crbind` the input chunks and then determine the sum values in each column
`cmean`	`mean`	`crbind` the input chunks and then determine the (unweighted) mean in each column

CFUN: Collapsing functions for batch processing

Description

Usage

Value

Arguments

ff options

Author

Details

See Also

Examples