powered by
Give your web-scraping function good manners polite
politely( fun, user_agent = paste0("polite ", getOption("HTTPUserAgent"), " bot"), robots = TRUE, force = FALSE, delay = 5, verbose = FALSE, cache = memoise::cache_memory() )
polite function
function to be turned "polite". Must contain an argument named url, which contains url to be queried.
url
optional, user agent string to be used. Defaults to paste("polite", getOption("HTTPUserAgent"), "bot")
paste("polite", getOption("HTTPUserAgent"), "bot")
optional, should robots.txt be consulted for permissions. Default is TRUE
whether or not tp force fresh download of robots.txt
minimum delay in seconds, not less than 1. Default is 5.
output more information about querying process
memoise cache function for storing results. Default memoise::cache_memory()
memoise::cache_memory()
polite_GET <- politely(httr::GET)
Run the code above in your browser using DataLab