as.list.robotstxt_text

Method as.list() for class robotstxt_text

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

as.list.robotstxt_text function

<dl><dt>x</dt>
<dd>class robotstxt_text object to be transformed into list</dd>
<dt>...</dt>
<dd>further arguments (inherited from <code><a href="/link/base%3A%3Aas.list()?package=robotstxt&version=0.7.15" data-mini-rdoc="robotstxt::base::as.list()">base::as.list()</a></code>)</dd></dl>

Arguments

Method as.list() for class robotstxt_text — as.list.robotstxt_text

<dl>

<dt>x</dt>
<dd>class robotstxt_text object to be transformed into list</dd>


<dt>...</dt>
<dd>further arguments (inherited from <code><a href='https://rdrr.io/r/base/list.html'>base::as.list()</a></code>)</dd>

</dl>

as.list.robotstxt_text: Method as.list() for class robotstxt_text

Description

Usage

Arguments