storr: Object cache

Description

Create an object cache; a "storr". A storr is a simple key-value store where the actual content is stored in a content-addressible way (so that duplicate objects are only stored once) and with a caching layer so that repeated lookups are fast even if the underlying storage driver is slow.

Usage

storr(driver, default_namespace = "objects")

Arguments

driver

A driver object

default_namespace

Default namespace to store objects in. By default "objects" is used, but this might be useful to have two diffent storr objects pointing at the same underlying storage, but storing things in different namespaces.

Methods

destroy

Totally destroys the storr by telling the driver to destroy all the data and then deleting the driver. This will remove all data and cannot be undone.

Usage: destroy()

flush_cache

Flush the temporary cache of objects that accumulates as the storr is used. Should not need to be called often.

Usage: flush_cache()

set

Set a key to a value.

Usage: set(key, value, namespace = self$default_namespace, use_cache = TRUE)

Arguments:

key: The key name. Can be any string.
value: Any R object to store. The object will generally be serialized (this is not actually true for the environment storr) so only objects that would usually be expected to survive a saveRDS/readRDS roundtrip will work. This excludes Rcpp modules objects, external pointers, etc. But any "normal" R object will work fine.
namespace: An optional namespace. By default the default namespace that the storr was created with will be used (by default that is "objects"). Different namespaces allow different types of objects to be stored without risk of names colliding. Use of namespaces is optional, but if used they must be a string.
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

Value: Invisibly, the hash of the saved object.

set_by_value

Like set but saves the object with a key that is the same as the hash of the object. Equivalent to $set(digest::digest(value), value).

Usage: set_by_value(value, namespace = self$default_namespace, use_cache = TRUE)

Arguments:

value: An R object to save, with the same limitations as set.
namespace: Optional namespace to save the key into.
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

get

Retrieve an object from the storr. If the requested value is not found then a KeyError will be raised (an R error, but can be caught with tryCatch; see the "storr" vignette).

Usage: get(key, namespace = self$default_namespace, use_cache = TRUE)

Arguments:

key: The name of the key to get.
namespace: Optional namespace to look for the key within.
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

get_hash

Retrieve the hash of an object stored in the storr (rather than the object itself).

Usage: get_hash(key, namespace = self$default_namespace)

Arguments:

key: The name of the key to get.
namespace: Optional namespace to look for the key within.

del

Delete an object fom the storr.

Usage: del(key, namespace = self$default_namespace)

Arguments:

key: A vector of names of keys
namespace: The namespace of the key.

Value: A logical vector the same length as the recycled length of key/namespace, with each element being TRUE if an object was deleted, FALSE otherwise.

duplicate

Duplicate the value of a set of keys into a second set of keys. Because the value stored against a key is just the hash of its content, this operation is very efficient - it does not make a copy of the data, just the pointer to the data (for more details see the storr vignette which explains the storage model in more detail). Multiple keys (and/or namespaces) can be provided, with keys and nmespaces recycled as needed. However, the number of source and destination keys must be the same. The order of operation is not defined, so if the sets of keys are overlapping it is undefined behaviour.

Usage: duplicate(key_src, key_dest, namespace = self$default_namespace, namespace_src = namespace, namespace_dest = namespace)

Arguments:

key_src: The source key (or vector of keys)
key_dest: The destination key
namespace: The namespace to copy keys within (used only of namespace_src and namespace_dest are not provided
namespace_src: The source namespace - use this where keys are duplicated across namespaces.
namespace_dest: The destination namespace - use this where keys are duplicated across namespaces.

fill

Set one or more keys (potentially across namespaces) to the same value, without duplication effort serialisation, or duplicating data.

Usage: fill(key, value, namespace = self$default_namespace, use_cache = TRUE)

Arguments:

key: A vector of keys to get; zero to many valid keys
value: A single value to set all keys to
namespace: A vector of namespaces (either a single namespace or a vector)
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

clear

Clear a storr. This function might be slow as it will iterate over each key. Future versions of storr might allow drivers to implement a bulk clear method that will allow faster clearing.

Usage: clear(namespace = self$default_namespace)

Arguments:

namespace: A namespace, to clear a single namespace, or NULL to clear all namespaces.

exists

Test if a key exists within a namespace

Usage: exists(key, namespace = self$default_namespace)

Arguments:

key: A vector of names of keys
namespace: The namespace of the key.

Value: A logical vector the same length as the recycled length of key/namespace, with each element being TRUE if the object exists and FALSE otherwise.

exists_object

Test if an object with a given hash exists within the storr

Usage: exists_object(hash)

Arguments:

hash: Hash to test

mset

Set multiple elements at once

Usage: mset(key, value, namespace = self$default_namespace, use_cache = TRUE)

Arguments:

key: A vector of keys to set; zero to many valid keys
value: A vector of values
namespace: A vector of namespaces (either a single namespace or a vector)
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

Details: The arguments key and namespace are recycled such that either can be given as a scalar if the other is a vector. Other recycling is not allowed.

mget

Get multiple elements at once

Usage: mget(key, namespace = self$default_namespace, use_cache = TRUE, missing = NULL)

Arguments:

key: A vector of keys to get; zero to many valid keys
namespace: A vector of namespaces (either a single namespace or a vector)
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).
missing: Value to use for missing elements; by default NULL will be used. IF NULL is a value that you might have stored in the storr you might want to use a different value here to distinguish "missing" from "set to NULL". In addition, the missing attribute will indicate which values were missing.

Details: The arguments key and namespace are recycled such that either can be given as a scalar if the other is a vector. Other recycling is not allowed.

Value: A list with a length of the recycled length of key and namespace. If any elements are missing, then an attribute missing will indicate the elements that are missing (this will be an integer vector with the indices of values were not found in the storr).

mset_by_value

Set multiple elements at once, by value. A cross between mset and set_by_value.

Usage: mset_by_value(value, namespace = self$default_namespace, use_cache = TRUE)

Arguments:

value: A list or vector of values to set into the storr.
namespace: A vector of namespaces
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

gc

Garbage collect the storr. Because keys do not directly map to objects, but instead map to hashes which map to objects, it is possible that hash/object pairs can persist with nothing pointing at them. Running gc will remove these objects from the storr.

Usage: gc()

get_value

Get the content of an object given its hash.

Usage: get_value(hash, use_cache = TRUE)

Arguments:

hash: The hash of the object to retrieve.
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

Value: The object if it is present, otherwise throw a HashError.

set_value

Add an object value, but don't add a key. You will not need to use this very often, but it is used internally.

Usage: set_value(value, use_cache = TRUE)

Arguments:

value: An R object to set.
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

Value: Invisibly, the hash of the object.

mset_value

Add a vector of object values, but don't add keys. You will not need to use this very often, but it is used internally.

Usage: mset_value(values, use_cache = TRUE)

Arguments:

values: A list of R objects to set
use_cache: Use the internal cache to avoid reading or writing to the underlying storage if the data has already been seen (i.e., we have seen the hash of the object before).

list

List all keys stored in a namespace.

Usage: list(namespace = self$default_namespace)

Arguments:

namespace: The namespace to list keys within.

Value: A sorted character vector (possibly zero-length).

list_hashes

List all hashes stored in the storr

Usage: list_hashes()

Value: A sorted character vector (possibly zero-length).

list_namespaces

List all namespaces known to the database

Usage: list_namespaces()

Value: A sorted character vector (possibly zero-length).

import

Import R objects from an environment.

Usage: import(src, list = NULL, namespace = self$default_namespace, skip_missing = FALSE)

Arguments:

src: Object to import objects from; can be a list, environment or another storr.
list: Names of of objects to import (or NULL to import all objects in envir. If given it must be a character vector. If named, the names of the character vector will be the names of the objects as created in the storr.
namespace: Namespace to get objects from, and to put objects into. If NULL, all namespaces from src will be imported. If named, then the same rule is followed as list; namespace = c(a = b) will import the contents of namespace b as namespace a.
skip_missing: Logical, indicating if missing keys (specified in list) should be skipped over, rather than being treated as an error (the default).

export

Export objects from the storr into something else.

Usage: export(dest, list = NULL, namespace = self$default_namespace, skip_missing = FALSE)

Arguments:

dest: A target destination to export objects to; can be a list, environment, or another storr. Use list() to export to a brand new list, or use as.list(object) for a shorthand.
list: Names of objects to export, with the same rules as list in $import.
namespace: Namespace to get objects from, and to put objects into. If NULL, then this will export namespaces from this (source) storr into the destination; if there is more than one namespace,this is only possible if dest is a storr (otherwise there will be an error).
skip_missing: Logical, indicating if missing keys (specified in list) should be skipped over, rather than being treated as an error (the default).

Value: Invisibly, dest, which allows use of e <- st$export(new.env()) and x <- st$export(list()).

archive_export

Export objects from the storr into a special "archive" storr, which is an storr_rds with name mangling turned on (which encodes keys with base64 so that they do not voilate filesystem naming conventions).

Usage: archive_export(path, names = NULL, namespace = NULL)

Arguments:

path: Path to create the storr at; can exist already.
names: As for $export
namespace: Namespace to get objects from. If NULL, then exports all namespaces found in this (source) storr.

archive_import

Inverse of archive_export; import objects from a storr that was created by archive_export.

Usage: archive_import(path, names = NULL, namespace = NULL)

Arguments:

path: Path of the exported storr.
names: As for $import
namespace: Namespace to import objects into. If NULL, then imports all namespaces from the source storr.

index_export

Generate a data.frame with an index of objects present in a storr. This can be saved (for an rds storr) in lieu of the keys/ directory and re-imported with index_import. It will provide a more version control friendly export of the data in a storr.

Usage: index_export(namespace = NULL)

Arguments:

namespace: Optional character vector of namespaces to export. The default is to export all namespaces.

index_import

Import an index.

Usage: index_import(index)

Arguments:

index: Must be a data.frame with columns 'namespace', 'key' and 'hash' (in any order). It is an error if not all hashes are present in the storr.

Details

To create a storr you need to provide a "driver" object. There are three in this package: driver_environment for ephemeral in-memory storage, driver_rds for serialized storage to disk, and driver_dbi for use with DBI-compliant database interfaces. The redux package (on CRAN) provides a storr driver that uses Redis.

There are convenience functions (e.g., storr_environment and storr_rds) that may be more convenient to use than this function.

Once a storr has been made it provides a number of methods. Because storr uses R6 (R6Class) objects, each method is accessed by using $ on a storr object (see the examples). The methods are described below in the "Methods" section.

The default_namespace affects all methods of the storr object that refer to namespaces; if a namespace is not given, then the action (get, set, del, list, import, export) will affect the default_namespace. By default this is "objects".

Examples

Run this code

# NOT RUN {
st <- storr(driver_environment())
## Set "mykey" to hold the mtcars dataset:
st$set("mykey", mtcars)
## and get the object:
st$get("mykey")
## List known keys:
st$list()
## List hashes
st$list_hashes()
## List keys in another namespace:
st$list("namespace2")
## We can store things in other namespaces:
st$set("x", mtcars, "namespace2")
st$set("y", mtcars, "namespace2")
st$list("namespace2")
## Duplicate data do not cause duplicate storage: despite having three
## keys we only have one bit of data:
st$list_hashes()
st$del("mykey")

## Storr objects can be created that have a default namespace that is
## not "objects" by using the \code{default_namespace} argument (this
## one also points at the same memory as the first storr).
st2 <- storr(driver_environment(st$driver$envir),
             default_namespace = "namespace2")
## All functions now use "namespace2" as the default namespace:
st2$list()
st2$del("x")
st2$del("y")
# }

Run the code above in your browser using DataLab