html-to-markdown 2.24.6 → 2.25.1

data/rust-vendor/xml5ever/examples/README.md

@@ -1,223 +0,0 @@
-# Examples
-
-The examples have been designed with [`cargo-script`](https://github.com/DanielKeep/cargo-script) in mind.
-
-Here I'll just give broad overview how to install [`cargo script`] for Rust 1.5. For more details, check out [cargo-script repository](https://github.com/DanielKeep/cargo-script).
-
-    cargo install cargo-script
-
-
-# Token printer
-
-The basis of xml5ever is its tokenizer and tree builder. Roughly speaking tokenizer
-takes input and returns a set of tokens like comment, processing instruction, start
-tag, end tag, etc.
-
-First let's define our dependencies:
-
-```toml
-    [dependencies]
-    xml5ever = "0.2.0"
-    tendril = "0.1.3"
-```
-
-With dependencies declared, we can now make a simple tokenizer sink. First step is to
-define a [`TokenSink`](https://docs.rs/xml5ever/latest/xml5ever/tokenizer/trait.TokenSink.html). [`TokenSink`](https://docs.rs/xml5ever/latest/xml5ever/tokenizer/trait.TokenSink.html) are traits that received stream of [`Tokens`](https://docs.rs/xml5ever/latest/xml5ever/tokenizer/enum.Token.html).
-
-In our case we'll define a unit struct (i.e. a struct  without any fields).
-
-```rust
-    struct SimpleTokenPrinter;
-```
-
-To make `SimpleTokenPrinter` a [`TokenSink`](https://docs.rs/xml5ever/latest/xml5ever/tokenizer/trait.TokenSink.html), we need to implement [process_token](https://docs.rs/xml5ever/latest/xml5ever/tokenizer/trait.TokenSink.html#tymethod.process_token) method.
-
-```rust
-    impl TokenSink for SimpleTokenPrinter {
-        fn process_token(&mut self, token: Token) {
-            match token {
-                CharacterTokens(b) => {
-                    println!("TEXT: {}", &*b);
-                },
-                NullCharacterToken => print!("NULL"),
-                TagToken(tag) => {
-                    println!("{:?} {} ", tag.kind, &*tag.name.local);
-                },
-                ParseError(err) => {
-                    println!("ERROR: {}", err);
-                },
-                PIToken(Pi{ref target, ref data}) => {
-                    println!("PI : <?{} {}?>", &*target, &*data);
-                },
-                CommentToken(ref comment) => {
-                    println!("<!--{:?}-->", &*comment);
-                },
-                EOFToken => {
-                    println!("EOF");
-                },
-                DoctypeToken(Doctype{ref name, ref public_id, ..}) => {
-                    println!("<!DOCTYPE {:?} {:?}>", &*name, &*public_id);
-                }
-            }
-        }
-    }
-```
-
-Now, we need some input to process. For input we'll use `stdin`. However, xml5ever `tokenize_to` method only takes `StrTendril`. So we need to construct a
-[`ByteTendril`](https://docs.rs/tendril/latest/tendril/type.ByteTendril.html) using `ByteTendril::new()`, then read the `stdin` using [`read_to_tendril`](https://docs.rs/tendril/latest/tendril/trait.ReadExt.html#tymethod.read_to_tendril) extension.
-
-Once that is set, to make `SimpleTokenPrinter` parse the input, call,
-`tokenize_to` with it as the first parameter, input wrapped in Option for second parameter and XmlToke.
-
-```rust
-    fn main() {
-        let sink = SimpleTokenPrinter;
-
-        // We need a ByteTendril to read a file
-        let mut input = ByteTendril::new();
-        // Using SliceExt.read_to_tendril we read stdin
-        io::stdin().read_to_tendril(&mut input).unwrap();
-        // For xml5ever we need StrTendril, so we reinterpret it
-        // into StrTendril.
-        //
-        // You might wonder, how does `try_reinterpret` know we
-        // need StrTendril and the answer is type inference based
-        // on `tokenize_xml_to` signature.
-        let input = input.try_reinterpret().unwrap();
-        // Here we create and run tokenizer
-        let mut tok = XmlTokenizer::new(sink, Default::default());
-        // We pass input to parsed.
-        tok.feed(input);
-
-        // tok.end must be invoked for final bytes to be processed.
-        tok.end();
-    }
-```
-
-NOTE: `unwrap` causes panic, it's only OK to use in simple examples.
-
-For full source code check out: [`examples/simple_xml_tokenizer.rs`](https://github.com/servo/html5ever/blob/main/xml5ever/examples/simple_xml_tokenizer.rs)
-
-Once we have successfully compiled the example we run the example with inline
-xml
-
-```bash
-    cargo script simple_xml_tokenizer.rs <<< "<xml>Text with <b>bold words</b>!</xml>"
-```
-
-or by sending an [`examples/example.xml`](https://github.com/servo/html5ever/blob/main/xml5ever/examples/example.xml) located in same folder as examples.
-
-```bash
-    cargo script simple_xml_tokenizer.rs < example.xml
-```
-
-# Tree printer
-
-To actually get an XML document tree from the xml5ever, you need to use a `TreeSink`.
-`TreeSink` is in many way similar to the TokenSink. Basically, TokenSink takes data
-and returns list of tokens, while TreeSink takes tokens and returns a tree of parsed
-XML document. Do note, that this is a simplified explanation and consult
-documentation for more info.
-
-Ok, with that in mind, let's build us a TreePrinter. For example if we get an XML
-file like:
-
-```xml
-    <student>
-        <first-name>Bobby</first-name>
-        <last-name>Tables</last-name>
-    </student>
-```
-
-We'd want a structure similar to this:
-
-```
-#document
-    student
-        first-name
-            #text Bobby
-        last-name
-            #text Tables
-
-```
-We won't print anything other than element names and text fields. So comments,
-doctypes and other such elements are ignored.
-
-First part is similar to making SimpleTokenPrinter:
-
-```rust
-    // We need to allocate an input tendril for xml5ever
-    let mut input = ByteTendril::new();
-    // Using SliceExt.read_to_tendril functions we can read stdin
-    io::stdin().read_to_tendril(&mut input).unwrap();
-    let input = input.try_reinterpret().unwrap();
-```
-
-This time, we need an implementation of [`TreeSink`](https://docs.rs/xml5ever/latest/xml5ever/tree_builder/trait.TreeSink.html). xml5ever comes with a
-built-in `TreeSink` implementation called [`RcDom`](https://docs.rs/markup5ever_rcdom/latest/markup5ever_rcdom/struct.RcDom.html). To process input into
-a `TreeSink` we use the following line:
-
-```rust
-    let dom: RcDom = parse(one_input(input), Default::default());
-```
-
-Let's analyze it a bit. First there is `let dom: RcDom`. We need this part,
-because the type inferencer can't infer which TreeSink implementation we mean
-in this scenario.
-
-Function [`one_input`](https://ygg01.github.io/docs/xml5ever/xml5ever/fn.one_input.html) is a convenience function that turns any value into an iterator. In this case
-it converts a StrTendril into an Iterator over itself.
-
-Ok, so now that we parsed our tree what with it? Well, for that we might need some
-kind of function that will help us traverse it. We shall call that function `walk`.
-
-```rust
-    fn walk(prefix: &str, handle: Handle) {
-        let node = handle.borrow();
-
-        // We print out the prefix before we start
-        print!("{}", prefix);
-        // We are only interested in following nodes:
-        // Document, Text and Element, so our match
-        // reflects that.
-        match node.node {
-            Document
-                => println!("#document"),
-
-            Text(ref text)  => {
-                println!("#text {}", text.escape_default())
-            },
-
-            Element(ref name, _) => {
-                println!("{}", name.local);
-            },
-
-            _ => {},
-
-        }
-
-        // We increase indent in child nodes
-        let new_indent = {
-            let mut temp = String::new();
-            temp.push_str(prefix);
-            temp.push_str("    ");
-            temp
-        };
-
-        for child in node.children.iter()
-            // In order to avoid weird indentation, we filter
-            // only Text/Element nodes.
-            // We don't need to filter Document since its guaranteed
-            // child elements don't contain documents
-            .filter(|child| match child.borrow().node {
-                Text(_) | Element (_, _) => true,
-                _ => false,
-            }
-        ) {
-            // Recursion - Yay!
-            walk(&new_indent, child.clone());
-        }
-    }
-```
-
-For full source code check out: [`examples/xml_tree_printer.rs`](https://github.com/servo/html5ever/blob/main/rcdom/examples/xml_tree_printer.rs)

data/rust-vendor/xml5ever/examples/example.xml

@@ -1,3 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE xml PUBLIC "http://www.wc3.org/">
-<student><first-name>Bobby</first-name><last-name>Tables</last-name></student>

data/rust-vendor/xml5ever/examples/simple_xml_tokenizer.rs

@@ -1,81 +0,0 @@
-#!/usr/bin/env run-cargo-script
-//! This is a regular crate doc comment, but it also contains a partial
-//! Cargo manifest.  Note the use of a *fenced* code block, and the
-//! `cargo` "language".
-//!
-//! ```cargo
-//! [dependencies]
-//! xml5ever = "0.1.1"
-//! tendril = "0.1.3"
-//! markup5ever = "0.7.4"
-//! ```
-extern crate markup5ever;
-extern crate xml5ever;
-
-use std::io;
-
-use markup5ever::buffer_queue::BufferQueue;
-use xml5ever::tendril::{ByteTendril, ReadExt};
-use xml5ever::tokenizer::{Doctype, Pi, ProcessResult, Token, TokenSink, XmlTokenizer};
-
-struct SimpleTokenPrinter;
-
-impl TokenSink for SimpleTokenPrinter {
-    type Handle = ();
-
-    fn process_token(&self, token: Token) -> ProcessResult<()> {
-        match token {
-            Token::Characters(b) => {
-                println!("TEXT: {}", &*b);
-            },
-            Token::NullCharacter => print!("NULL"),
-            Token::Tag(tag) => {
-                println!("{:?} {} ", tag.kind, &*tag.name.local);
-            },
-            Token::ParseError(err) => {
-                println!("ERROR: {err}");
-            },
-            Token::ProcessingInstruction(Pi {
-                ref target,
-                ref data,
-            }) => {
-                println!("PI : <?{target} {data}?>");
-            },
-            Token::Comment(ref comment) => {
-                println!("<!--{comment:?}-->");
-            },
-            Token::EndOfFile => {
-                println!("EOF");
-            },
-            Token::Doctype(Doctype {
-                ref name,
-                ref public_id,
-                ..
-            }) => {
-                println!("<!DOCTYPE {name:?} {public_id:?}>");
-            },
-        };
-        ProcessResult::Continue
-    }
-}
-
-fn main() {
-    // Our implementation of TokenSink
-    let sink = SimpleTokenPrinter;
-
-    // We need a ByteTendril to read a file
-    let mut input = ByteTendril::new();
-
-    // Using SliceExt.read_to_tendril we can read stdin
-    io::stdin().read_to_tendril(&mut input).unwrap();
-    // For xml5ever we need StrTendril, so we reinterpret it
-    // into StrTendril.
-
-    // Load input into BufferQueue
-    let input_buffer = BufferQueue::default();
-    input_buffer.push_back(input.try_reinterpret().unwrap());
-    // Here we create and run tokenizer
-    let tok = XmlTokenizer::new(sink, Default::default());
-    let _ = tok.feed(&input_buffer);
-    tok.end();
-}

data/rust-vendor/xml5ever/examples/xml_tokenizer.rs

@@ -1,115 +0,0 @@
-#!/usr/bin/env run-cargo-script
-//! This is a regular crate doc comment, but it also contains a partial
-//! Cargo manifest.  Note the use of a *fenced* code block, and the
-//! `cargo` "language".
-//!
-//! ```cargo
-//! [dependencies]
-//! xml5ever = "0.2.0"
-//! tendril = "0.1.3"
-//! markup5ever = "0.7.4"
-//! ```
-extern crate markup5ever;
-extern crate xml5ever;
-
-use std::cell::Cell;
-use std::io;
-
-use markup5ever::buffer_queue::BufferQueue;
-use xml5ever::tendril::{ByteTendril, ReadExt};
-use xml5ever::tokenizer::{
-    EmptyTag, EndTag, Pi, ProcessResult, ShortTag, StartTag, Token, TokenSink, XmlTokenizer,
-    XmlTokenizerOpts,
-};
-
-#[derive(Clone)]
-struct TokenPrinter {
-    in_char_run: Cell<bool>,
-}
-
-impl TokenPrinter {
-    fn is_char(&self, is_char: bool) {
-        match (self.in_char_run.get(), is_char) {
-            (false, true) => print!("CHAR : \""),
-            (true, false) => println!("\""),
-            _ => (),
-        }
-        self.in_char_run.set(is_char);
-    }
-
-    fn do_char(&self, c: char) {
-        self.is_char(true);
-        print!("{}", c.escape_default().collect::<String>());
-    }
-}
-
-impl TokenSink for TokenPrinter {
-    type Handle = ();
-
-    fn process_token(&self, token: Token) -> ProcessResult<()> {
-        match token {
-            Token::Characters(b) => {
-                for c in b.chars() {
-                    self.do_char(c);
-                }
-            },
-            Token::NullCharacter => self.do_char('\0'),
-            Token::Tag(tag) => {
-                self.is_char(false);
-                // This is not proper HTML serialization, of course.
-                match tag.kind {
-                    StartTag => print!("TAG  : <\x1b[32m{}\x1b[0m", tag.name.local),
-                    EndTag => print!("END TAG  : <\x1b[31m/{}\x1b[0m", tag.name.local),
-                    ShortTag => print!("Short TAG  : <\x1b[31m/{}\x1b[0m", tag.name.local),
-                    EmptyTag => print!("Empty TAG  : <\x1b[31m{}\x1b[0m", tag.name.local),
-                }
-                for attr in tag.attrs.iter() {
-                    print!(
-                        " \x1b[36m{}\x1b[0m='\x1b[34m{}\x1b[0m'",
-                        attr.name.local, attr.value
-                    );
-                }
-                if tag.kind == EmptyTag {
-                    print!("/");
-                }
-                println!(">");
-            },
-            Token::ParseError(err) => {
-                self.is_char(false);
-                println!("ERROR: {err}");
-            },
-            Token::ProcessingInstruction(Pi { target, data }) => {
-                self.is_char(false);
-                println!("PI : <?{target:?} {data:?}?>");
-            },
-            _ => {
-                self.is_char(false);
-                println!("OTHER: {token:?}");
-            },
-        };
-
-        ProcessResult::Continue
-    }
-}
-
-fn main() {
-    let sink = TokenPrinter {
-        in_char_run: Cell::new(false),
-    };
-    let mut input = ByteTendril::new();
-    io::stdin().read_to_tendril(&mut input).unwrap();
-    let input_buffer = BufferQueue::default();
-    input_buffer.push_back(input.try_reinterpret().unwrap());
-
-    let tok = XmlTokenizer::new(
-        sink,
-        XmlTokenizerOpts {
-            profile: true,
-            exact_errors: true,
-            ..Default::default()
-        },
-    );
-    let _ = tok.feed(&input_buffer);
-    tok.end();
-    tok.sink.is_char(false);
-}

data/rust-vendor/xml5ever/src/driver.rs

@@ -1,90 +0,0 @@
-// Copyright 2014-2017 The html5ever Project Developers. See the
-// COPYRIGHT file at the top-level directory of this distribution.
-//
-// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
-// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
-// option. This file may not be copied, modified, or distributed
-// except according to those terms.
-
-use crate::tokenizer::{XmlTokenizer, XmlTokenizerOpts};
-use crate::tree_builder::{TreeSink, XmlTreeBuilder, XmlTreeBuilderOpts};
-
-use std::borrow::Cow;
-
-use crate::tendril;
-use crate::tendril::stream::{TendrilSink, Utf8LossyDecoder};
-use crate::tendril::StrTendril;
-use markup5ever::buffer_queue::BufferQueue;
-
-/// All-encompasing parser setting structure.
-#[derive(Clone, Default)]
-pub struct XmlParseOpts {
-    /// Xml tokenizer options.
-    pub tokenizer: XmlTokenizerOpts,
-    /// Xml tree builder .
-    pub tree_builder: XmlTreeBuilderOpts,
-}
-
-/// Parse and send results to a `TreeSink`.
-///
-/// ## Example
-///
-/// ```ignore
-/// let mut sink = MySink;
-/// parse_document(&mut sink, iter::once(my_str), Default::default());
-/// ```
-pub fn parse_document<Sink>(sink: Sink, opts: XmlParseOpts) -> XmlParser<Sink>
-where
-    Sink: TreeSink,
-{
-    let tb = XmlTreeBuilder::new(sink, opts.tree_builder);
-    let tok = XmlTokenizer::new(tb, opts.tokenizer);
-    XmlParser {
-        tokenizer: tok,
-        input_buffer: BufferQueue::default(),
-    }
-}
-
-/// An XML parser,
-/// ready to receive Unicode input through the `tendril::TendrilSink` trait’s methods.
-pub struct XmlParser<Sink>
-where
-    Sink: TreeSink,
-{
-    /// Tokenizer used by XmlParser.
-    pub tokenizer: XmlTokenizer<XmlTreeBuilder<Sink::Handle, Sink>>,
-    /// Input used by XmlParser.
-    pub input_buffer: BufferQueue,
-}
-
-impl<Sink: TreeSink> TendrilSink<tendril::fmt::UTF8> for XmlParser<Sink> {
-    type Output = Sink::Output;
-
-    fn process(&mut self, t: StrTendril) {
-        self.input_buffer.push_back(t);
-        // FIXME: Properly support </script> somehow.
-        let _ = self.tokenizer.feed(&self.input_buffer);
-    }
-
-    // FIXME: Is it too noisy to report every character decoding error?
-    fn error(&mut self, desc: Cow<'static, str>) {
-        self.tokenizer.sink.sink.parse_error(desc)
-    }
-
-    fn finish(self) -> Self::Output {
-        self.tokenizer.end();
-        self.tokenizer.sink.sink.finish()
-    }
-}
-
-impl<Sink: TreeSink> XmlParser<Sink> {
-    /// Wrap this parser into a `TendrilSink` that accepts UTF-8 bytes.
-    ///
-    /// Use this when your input is bytes that are known to be in the UTF-8 encoding.
-    /// Decoding is lossy, like `String::from_utf8_lossy`.
-    #[allow(clippy::wrong_self_convention)]
-    pub fn from_utf8(self) -> Utf8LossyDecoder<Self> {
-        Utf8LossyDecoder::new(self)
-    }
-}

data/rust-vendor/xml5ever/src/lib.rs

@@ -1,47 +0,0 @@
-// Copyright 2014-2017 The html5ever Project Developers. See the
-// COPYRIGHT file at the top-level directory of this distribution.
-//
-// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
-// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
-// option. This file may not be copied, modified, or distributed
-// except according to those terms.
-
-//! This crate provides a push based XML parser library that
-//! adheres to XML5 specification. In other words this library
-//! trades well-formedness for error recovery.
-//!
-//! The idea behind this, was to minimize number of errors from
-//! tools that generate XML (e.g. `&#83` won't just return `&#83`
-//! as text, but will parse it into `S` ).
-//! You can check out full specification [here](https://ygg01.github.io/xml5_draft/).
-//!
-//! What this library provides is a solid XML parser that can:
-//!
-//!   * Parse somewhat erroneous XML input
-//!   * Provide support for [Numeric character references](https://en.wikipedia.org/wiki/Numeric_character_reference).
-//!   * Provide partial [XML namespace](http://www.w3.org/TR/xml-names11/) support.
-//!   * Provide full set of SVG/MathML entities
-//!
-//! What isn't in scope for this library:
-//!
-//!   * Document Type Definition parsing - this is pretty hard to do right and nowadays, its used
-//!
-
-#![crate_name = "xml5ever"]
-#![crate_type = "dylib"]
-#![allow(unexpected_cfgs)]
-#![deny(missing_docs)]
-
-pub use markup5ever::*;
-
-pub(crate) mod macros;
-
-/// Driver
-pub mod driver;
-/// Serializer for XML5.
-pub mod serialize;
-/// XML5 tokenizer - converts input into tokens
-pub mod tokenizer;
-/// XML5 tree builder - converts tokens into a tree like structure
-pub mod tree_builder;

data/rust-vendor/xml5ever/src/macros.rs

@@ -1,18 +0,0 @@
-// Copyright 2014-2017 The html5ever Project Developers. See the
-// COPYRIGHT file at the top-level directory of this distribution.
-//
-// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
-// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
-// option. This file may not be copied, modified, or distributed
-// except according to those terms.
-
-macro_rules! time {
-    ($e:expr) => {{
-        let t0 = ::std::time::Instant::now();
-        let result = $e;
-        let dt = t0.elapsed().as_nanos() as u64;
-        (result, dt)
-    }};
-}
-pub(crate) use time;