Struct RobotsTxt 

pub struct RobotsTxt {
    pub groups: Vec<RobotsGroup>,
    pub sitemaps: Vec<String>,
}

A typed robots.txt payload that can be parsed, inspected, serialized, and returned as a response.

Fields

groups: Vec<RobotsGroup>

sitemaps: Vec<String>

Implementations

impl RobotsTxt

pub const fn new() -> RobotsTxt

Create an empty RobotsTxt document.

pub fn parse(input: &str) -> RobotsTxt

Parse a robots.txt payload, ignoring malformed or invalid directives.

pub fn parse_strict(input: &str) -> Result<RobotsTxt, RobotsDirectiveParseError>

Parse a robots.txt payload strictly.

Unlike Self::parse, this returns an error when a supported directive is malformed or has an invalid value.

pub fn with_group(self, group: RobotsGroup) -> RobotsTxt

Add a group to this robots.txt document.

pub fn set_group(&mut self, group: RobotsGroup) -> &mut RobotsTxt

Add a group to this robots.txt document.

pub fn with_sitemap(self, sitemap: impl Into<String>) -> RobotsTxt

Add a sitemap entry to this robots.txt document.

pub fn set_sitemap(&mut self, sitemap: impl Into<String>) -> &mut RobotsTxt

Add a sitemap entry to this robots.txt document.

pub fn rules_for(&self, user_agent: &str) -> RobotsClientRules<'_>

Resolve the effective rules for the given user-agent.

pub fn is_url_allowed(&self, user_agent: &str, uri: &Uri) -> bool

Returns true if the given URL is allowed for the user-agent.

Trait Implementations

impl Clone for RobotsTxt

fn clone(&self) -> RobotsTxt

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotsTxt

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for RobotsTxt

fn default() -> RobotsTxt

Returns the “default value” for a type.

impl Display for RobotsTxt

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl FromStr for RobotsTxt

type Err = Infallible

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<RobotsTxt, <RobotsTxt as FromStr>::Err>

Parses a string s to return a value of this type.

impl IntoResponse for RobotsTxt

fn into_response(self) -> Response

Create a response.

impl PartialEq for RobotsTxt

fn eq(&self, other: &RobotsTxt) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for RobotsTxt

impl StructuralPartialEq for RobotsTxt