request_handler_handler

Helper function to handle robotstxt handlers.

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

request_handler_handler function

<dl><dt>request</dt>
<dd>the request object returned by call to httr::GET()</dd>
<dt>handler</dt>
<dd>the handler either a character string entailing various options or a function producing a specific list, see return.</dd>
<dt>res</dt>
<dd>a list a list with elements '[handler names], ...', 'rtxt', and 'cache'</dd>
<dt>info</dt>
<dd>info to add to problems list</dd>
<dt>warn</dt>
<dd>if FALSE warnings and messages are suppressed</dd></dl>

Arguments

request_handler_handler — request_handler_handler

<dl>

<dt>request</dt>
<dd>the request object returned by call to httr::GET()</dd>


<dt>handler</dt>
<dd>the handler either a character string entailing various options or a function producing a specific list, see return.</dd>


<dt>res</dt>
<dd>a list a list with elements '[handler names], ...', 'rtxt', and 'cache'</dd>


<dt>info</dt>
<dd>info to add to problems list</dd>


<dt>warn</dt>
<dd>if FALSE warnings and messages are suppressed</dd>

</dl>

request_handler_handler: request_handler_handler

Description

Usage

Value

Arguments