remove_domain

function to remove domain from path

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

remove_domain function

<dl><dt>x</dt>
<dd>path aka URL from which to first infer domain and then remove it</dd></dl>

Arguments

function to remove domain from path — remove_domain

<dl>

<dt>x</dt>
<dd>path aka URL from which to first infer domain and then remove it</dd>

</dl>

remove_domain: function to remove domain from path

Description

Usage

Arguments