Extract information from text and HTML.
Useful for processing data from web scrapers.
HTML content extraction: extract visible text, hyperlinks, or image URLs
from a parsed HTML tree. Key function: HTMLGetText.
HTML parsing and querying: parse HTML from string, []byte, or io.Reader;
query nodes with XPath. Key functions: HTMLXPath.
Text cleaning: normalize Unicode representations, remove redundant whitespace, strip Vietnamese diacritics.
Text analysis: split text into words, build n-gram frequency maps, hash text to a 64-bit integer.
A typical web scraping workflow:
- Download the page
- Parse a downloaded page with
HTMLParseToNode - Extract visible text with
HTMLGetText - Collect outgoing links with
HTMLGetHREFs - Retrieve image URLs with
HTMLGetImgSrc
See html_test.go TestHtmlUtils for a full worked example.
For text processing (n-grams, Vietnamese diacritic removal, word splitting), see text_test.go.