gar_batch_walk: Walk data through batches

Description

Convenience function for walking through data in batches

Usage

gar_batch_walk(
  f,
  walk_vector,
  gar_pars = NULL,
  gar_paths = NULL,
  the_body = NULL,
  pars_walk = NULL,
  path_walk = NULL,
  body_walk = NULL,
  batch_size = 10,
  batch_function = NULL,
  data_frame_output = TRUE,
  ...,
  batch_endpoint = getOption("googleAuthR.batch_endpoint", default =
    "https://www.googleapis.com/batch")
)

Value

if data_frame_output is FALSE: A list of lists. Outer list the length of number of batches required, inner lists the results from the calls

if data_frame_output is TRUE: The list of lists will attempt to rbind all the results

Arguments

f: a function from gar_api_generator
walk_vector: a vector of the parameter or path to change
gar_pars: a list of parameter arguments for f
gar_paths: a list of path arguments for f
the_body: a list of body arguments for f
pars_walk: a character vector of the parameter(s) to modify for each walk of f
path_walk: a character vector of the path(s) to modify for each walk of f
body_walk: a character vector of the body(s) to modify for each walk of f
batch_size: size of each request to Google /batch API
batch_function: a function that will act on the result list of each batch API call
data_frame_output: if the list of lists are dataframes, you can bind them all by setting to TRUE
...: further arguments passed to the data parse function of f
batch_endpoint: the batch API endpoint to send

Details

You can modify more than one parameter or path arg, but it must be the same walked vector e.g. start = end = x

Many Google APIs have batch_size limits greater than 10, 1000 is common.

The `f` function needs to be a `gar_api_generator()` function that uses one of `path_args`, `pars_args` or `body_args` to construct the URL (rather than say using `sprintf()` to create the API URL).

You don't need to set the headers in the Google docs for batching API functions - those are done for you.

The argument `walk_vector` needs to be a vector of the values of the arguments to walk over, which you indicate will walk over the pars/path or body arguments on the function via on of the `*_walk` arguments e.g. if walking over id=1, id=2, for a path argument then it would be `path_walk="id"` and `walk_vector=c(1,2,3,4)`

The `gar_*` parameter is required to pass intended for other arguments to the function `f` you may need to pass through.

`gar_batch_walk()` only supports changing one value at a time, for one or multiple arguments (I think only changing the `start-date`, `end-date` example would be the case when you walk through more than one per call)

`batch_size` should be over 1 for batching to be of any benefit at all

The `batch_function` argument gives you a way to operate on the parsed output of each call

Examples

Run this code


if (FALSE) {
                               

# get a webproperty per account 
getAccountInfo <- gar_api_generator(
  "https://www.googleapis.com/analytics/v3/management/accounts",
  "GET", data_parse_function = function(x) unique(x$items$id))

getWebpropertyInfo <- gar_api_generator(
  "https://www.googleapis.com/analytics/v3/management/", # don't use sprintf to construct this
  "GET",
  path_args = list(accounts = "default", webproperties = ""),
  data_parse_function = function(x) x$items)

walkData <- function(){

  # here due to R lazy evaluation  
  accs <- getAccountInfo()
  gar_batch_walk(getWebpropertyInfo, 
                 walk_vector = accs,
                 gar_paths = list("webproperties" = ""),
                 path_walk = "accounts",
                 batch_size = 100, data_frame_output = FALSE)
                 }
                 
# do the walk
walkData()

# to walk body data, be careful to modify a top level body name:
changed_emails <- lapply(email, function(x){userRef = list(email = x)})

batched <- gar_batch_walk(users, 
                       walk_vector = changed_emails, 
                       the_body = list(
                           permissions = list(
                             local = list(permissions)
                           ),
                           userRef = list(
                             email = email[[1]]
                         )
                        ),
                        body_walk = "userRef",
                        batch_size = 300,
                        data_frame_output = FALSE)
}