parse_url

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Kun Ren

Oliver Keys

Rich Fitz John

parse_url function

<dl><dt>url</dt>
<dd>url to parse into its components</dd></dl>

Arguments

parse_url — parse_url

<dl>

<dt>url</dt>
<dd>url to parse into its components</dd>

</dl>

parse_url: parse_url

Description

Usage

Value

Arguments