rt_list_rtxt

<p>list robots.txt files saved along with the package:
these functions ar very handy for testing (not used otherwise)</p>

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

rt_list_rtxt function

list robots.txt files saved along with the package — rt_list_rtxt

list robots.txt files saved along with the package

rt_list_rtxt: list robots.txt files saved along with the package

Description

Usage

Arguments