rt_get_fields_worker

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

rt_get_fields_worker function

<dl><dt>txt</dt>
<dd>content of the robots.txt file</dd>
<dt>type</dt>
<dd>name or names of the fields to be returned, defaults to all
fields</dd>
<dt>regex</dt>
<dd>subsetting field names via regular expressions</dd>
<dt>invert</dt>
<dd>field selection</dd></dl>

Arguments

extracting robotstxt fields — rt_get_fields_worker

<dl>

<dt>txt</dt>
<dd>content of the robots.txt file</dd>


<dt>type</dt>
<dd>name or names of the fields to be returned, defaults to all
fields</dd>


<dt>regex</dt>
<dd>subsetting field names via regular expressions</dd>


<dt>invert</dt>
<dd>field selection</dd>

</dl>

extracting robotstxt fields

rt_get_fields_worker: extracting robotstxt fields

Description

Usage

Arguments