is_valid_robotstxt

function that checks if file is valid / parsable robots.txt file

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

is_valid_robotstxt function

<dl><dt>text</dt>
<dd>content of a robots.txt file provides as character vector</dd>
<dt>check_strickt_ascii</dt>
<dd>whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII</dd></dl>

Arguments

function that checks if file is valid / parsable robots.txt file — is_valid_robotstxt

<dl>

<dt>text</dt>
<dd>content of a robots.txt file provides as character vector</dd>


<dt>check_strickt_ascii</dt>
<dd>whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII</dd>

</dl>

is_valid_robotstxt: function that checks if file is valid / parsable robots.txt file

Description

Usage

Arguments