null_to_defeault

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

null_to_defeault function

<dl><dt>x</dt>
<dd>value to check and return</dd>
<dt>d</dt>
<dd>value to return in case x is NULL</dd></dl>

Arguments

null_to_defeault — null_to_defeault

<dl>

<dt>x</dt>
<dd>value to check and return</dd>


<dt>d</dt>
<dd>value to return in case x is NULL</dd>

</dl>

null_to_defeault: null_to_defeault

Description

Usage

Arguments