rt_last_http

get_robotstxt_http_get

storage for http request response objects
get_robotstxt() worker function to execute HTTP request

datasets

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

rt_last_http function

An object of class <code>environment</code> of length 1.

Format

<dl><dt>domain</dt>
<dd>the domain to get tobots.txt. file for</dd>
<dt>user_agent</dt>
<dd>the user agent to use for HTTP request header</dd>
<dt>ssl_verifypeer</dt>
<dd>either 1 (default) or 0, if 0 it disables SSL peer verification, which
might help with robots.txt file retrieval</dd></dl>

Arguments

storage for http request response objects — rt_last_http

<dl>

<dt>domain</dt>
<dd>the domain to get tobots.txt. file for</dd>


<dt>user_agent</dt>
<dd>the user agent to use for HTTP request header</dd>


<dt>ssl_verifypeer</dt>
<dd>either 1 (default) or 0, if 0 it disables SSL peer verification, which
might help with robots.txt file retrieval</dd>

</dl>

rt_last_http: storage for http request response objects

Description

Usage

Format

Arguments