Struct Tokenizer
pub struct Tokenizer { /* private fields */ }html and http and std only.Expand description
A byte-faithful, low-allocation, resumable HTML tokenizer.
Implementations§
§impl Tokenizer
impl Tokenizer
pub fn new() -> Tokenizer
pub fn new() -> Tokenizer
Creates a new tokenizer.
Non-strict by default (rama’s convention): ambiguous text-mode contexts
are tokenized best-effort. Enable with_strict to
abort on such ambiguity instead.
pub fn with_strict(self, strict: bool) -> Tokenizer
pub fn with_strict(self, strict: bool) -> Tokenizer
Sets strict mode. When enabled, ambiguous text-mode contexts abort
with a ParsingAmbiguityError instead of tokenizing best-effort.
pub fn set_strict(&mut self, strict: bool) -> &mut Tokenizer
pub fn set_strict(&mut self, strict: bool) -> &mut Tokenizer
Sets strict mode. When enabled, ambiguous text-mode contexts abort
with a ParsingAmbiguityError instead of tokenizing best-effort.
pub fn with_max_buffered_bytes(self, max_buffered_bytes: usize) -> Tokenizer
pub fn with_max_buffered_bytes(self, max_buffered_bytes: usize) -> Tokenizer
Sets the budget of one construct (a tag with its attributes, a
comment, a doctype, …): its input bytes plus 16 bytes per
attribute. A larger one fails with a MemoryLimitExceededError.
Defaults to DEFAULT_MAX_BUFFERED_BYTES; budgets below 9 bytes
(the longest markup opener, <![CDATA[) act as 9.
The tokenizer’s input and attribute scratch buffers stay below about twice the budget (they are capped separately); this is not a ceiling for the whole heap (e.g. nesting context, a rewriter’s output). Once a construct is gone, buffers above 64 KiB shrink back to 4 KiB, and ending or failing the stream shrinks them to 4 KiB.
pub fn set_max_buffered_bytes(
&mut self,
max_buffered_bytes: usize,
) -> &mut Tokenizer
pub fn set_max_buffered_bytes( &mut self, max_buffered_bytes: usize, ) -> &mut Tokenizer
Sets the budget of one construct (a tag with its attributes, a
comment, a doctype, …): its input bytes plus 16 bytes per
attribute. A larger one fails with a MemoryLimitExceededError.
Defaults to DEFAULT_MAX_BUFFERED_BYTES; budgets below 9 bytes
(the longest markup opener, <![CDATA[) act as 9.
The tokenizer’s input and attribute scratch buffers stay below about twice the budget (they are capped separately); this is not a ceiling for the whole heap (e.g. nesting context, a rewriter’s output). Once a construct is gone, buffers above 64 KiB shrink back to 4 KiB, and ending or failing the stream shrinks them to 4 KiB.
pub const fn max_buffered_bytes(&self) -> usize
pub const fn max_buffered_bytes(&self) -> usize
The budget of one construct, see
with_max_buffered_bytes.
pub fn write<S>(
&mut self,
chunk: &[u8],
sink: &mut S,
) -> Result<(), TokenizerError>where
S: TokenSink,
pub fn write<S>(
&mut self,
chunk: &[u8],
sink: &mut S,
) -> Result<(), TokenizerError>where
S: TokenSink,
Trait Implementations§
Auto Trait Implementations§
impl Forget for Tokenizer
impl Freeze for Tokenizer
impl Move for Tokenizer
impl RefUnwindSafe for Tokenizer
impl Send for Tokenizer
impl Sync for Tokenizer
impl Unpin for Tokenizer
impl UnsafeUnpin for Tokenizer
impl UnwindSafe for Tokenizer
Blanket Implementations§
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ
fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ
§fn with_current_context(self) -> WithContext<Self> ⓘ
fn with_current_context(self) -> WithContext<Self> ⓘ
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a rama_grpc::Request§impl<T> Pointable for T
impl<T> Pointable for T
§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
§fn and<P, B, E>(self, other: P) -> And<T, P>
fn and<P, B, E>(self, other: P) -> And<T, P>
Policy that returns Action::Follow only if self and other return
Action::Follow. Read more