rgbif (version 1.3.0)

occ_download_queue: Download requests in a queue

Description

Download requests in a queue

Usage

occ_download_queue(..., .list = list(), status_ping = 10)

Arguments

...

any number of occ_download() requests

.list

any number of occ_download_prep() requests

status_ping

(integer) seconds between pings checking status of the download request. generally larger numbers for larger requests. default: 10 (i.e., 10 seconds). must be 10 or greater

Value

a list of occ_download class objects, see occ_download_get() to fetch data

How it works

It works by using lazy evaluation to collect your requests into a queue. Then it kicks of the first 3 requests. Then in a while loop, we check status of those requests, and when any request finishes, we kick off the next, and so on. So in theory, there may not always strictly be 3 running concurrently, but the function will usually provide for 3 running concurrently.

Beware

This function is still in development. There's a lot of complexity to this problem. We'll be rolling out fixes and improvements in future versions of the package, so expect to have to adjust your code with new versions.

Details

This function is a convenience wrapper around occ_download(), allowing the user to kick off any number of requests, while abiding by GBIF rules of 3 concurrent requests per user.

Examples

# NOT RUN {
# passing occ_download() requests via ...
out <- occ_download_queue(
  occ_download('taxonKey = 3119195', "year = 1976"),
  occ_download('taxonKey = 3119195', "year = 2001"),
  occ_download('taxonKey = 3119195', "year = 2001", "month <= 8"),
  occ_download('taxonKey = 5229208', "year = 2011"),
  occ_download('taxonKey = 2480946', "year = 2015"),
  occ_download("country = NZ", "year = 1999", "month = 3"),
  occ_download("catalogNumber = Bird.27847588", "year = 1998", "month = 2")
)

# supports <= 3 requests too
out <- occ_download_queue(
  occ_download("country = NZ", "year = 1999", "month = 3"),
  occ_download("catalogNumber = Bird.27847588", "year = 1998", "month = 2")
)

# using pre-prepared requests via .list
keys <- c(7905507, 5384395, 8911082)
queries <- list()
for (i in seq_along(keys)) {
  queries[[i]] <- occ_download_prep(
    paste0("taxonKey = ", keys[i]),
    "basisOfRecord = HUMAN_OBSERVATION,OBSERVATION",
    "hasCoordinate = true",
    "hasGeospatialIssue = false",
    "year = 1993"
  )
}
out <- occ_download_queue(.list = queries)
out

# another pre-prepared example
yrs <- 1930:1934
length(yrs)
queries <- list()
for (i in seq_along(yrs)) {
  queries[[i]] <- occ_download_prep(
    "taxonKey = 2877951",
    "basisOfRecord = HUMAN_OBSERVATION,OBSERVATION",
    "hasCoordinate = true",
    "hasGeospatialIssue = false",
    paste0("year = ", yrs[i])
  )
}
out <- occ_download_queue(.list = queries)
out
# }