Module robots_txt 

Structs

RobotsClientRules

The effective rules for a specific user-agent.

RobotsDirectiveParseError

Parse error returned for invalid supported robots.txt directives.

RobotsGroup

A user-agent group in a robots.txt document.

RobotsRule

A single allow/disallow rule from a robots.txt document.

RobotsTxt

A typed robots.txt payload that can be parsed, inspected, serialized, and returned as a response.

Enums

RobotsRuleKind

The kind of a robots rule.