rt_cache

datasets

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

rt_cache function

<p>An object of class <code>environment</code> of length 0.</p>

Format

get_robotstxt() cache — rt_cache

get_robotstxt() cache

rt_cache: get_robotstxt() cache

Description

Usage

Arguments

Format