Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Available on crate features html and http only.

A byte-faithful, low-allocation, resumable HTML tokenizer.

Implementations

impl Tokenizer

pub fn new() -> Tokenizer

Creates a new tokenizer.

Non-strict by default (rama’s convention): ambiguous text-mode contexts are tokenized best-effort. Enable with_strict to abort on such ambiguity instead.

pub fn with_strict(self, strict: bool) -> Tokenizer

Sets strict mode. When enabled, ambiguous text-mode contexts abort with a ParsingAmbiguityError instead of tokenizing best-effort.

pub fn set_strict(&mut self, strict: bool) -> &mut Tokenizer

Sets strict mode. When enabled, ambiguous text-mode contexts abort with a ParsingAmbiguityError instead of tokenizing best-effort.

pub fn write<S>( &mut self, chunk: &[u8], sink: &mut S, ) -> Result<(), ParsingAmbiguityError>

where S: TokenSink,

Feeds a chunk of input, emitting every token that is now complete.

Errors

See tokenize.

pub fn end<S>(&mut self, sink: &mut S) -> Result<(), ParsingAmbiguityError>

where S: TokenSink,

Finalizes the stream, emitting any remaining (possibly unterminated) tokens, then resets for reuse.

Errors

See tokenize.

pub fn tokenize<S>( &mut self, input: &[u8], sink: &mut S, ) -> Result<(), ParsingAmbiguityError>

where S: TokenSink,

Tokenizes a complete input in one shot (write then end).

Errors

See tokenize.

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Tokenizer

Returns the “default value” for a type.