rt_get_fields

extracting permissions from robots.txt

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

rt_get_fields function

<dl><dt>txt</dt>
<dd>content of the robots.txt file</dd>
<dt>regex</dt>
<dd>regular expression specify field</dd>
<dt>invert</dt>
<dd>invert selection made via regex?</dd></dl>

Arguments

extracting permissions from robots.txt — rt_get_fields

<dl>

<dt>txt</dt>
<dd>content of the robots.txt file</dd>


<dt>regex</dt>
<dd>regular expression specify field</dd>


<dt>invert</dt>
<dd>invert selection made via regex?</dd>

</dl>

rt_get_fields: extracting permissions from robots.txt

Description

Usage

Arguments