rt_get_rtxt

load robots.txt files saved along with the package:
these functions are very handy for testing (not used otherwise)

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

rt_get_rtxt function

<dl><dt>name</dt>
<dd>name of the robots.txt files, defaults to a random drawn file ;-)</dd></dl>

Arguments

load robots.txt files saved along with the package — rt_get_rtxt

<dl>

<dt>name</dt>
<dd>name of the robots.txt files, defaults to a random drawn file ;-)</dd>

</dl>

rt_get_rtxt: load robots.txt files saved along with the package

Description

Usage

Arguments