sanitize_path

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

sanitize_path function

<dl><dt>path</dt>
<dd>path to be sanitized</dd></dl>

Arguments

making paths uniform — sanitize_path

<dl>

<dt>path</dt>
<dd>path to be sanitized</dd>

</dl>

making paths uniform

sanitize_path: making paths uniform

Description

Usage

Value

Arguments