U
lu
fK: @ s d Z dZdgZddlmZ ddlZddlZddlmZm Z m
Z
m Z m Z ddl
mZmZ ddlmZmZmZmZmZ d ZG d
d d eeZG d d deZdS )
zCUse the HTMLParser library to parse HTML files that aren't too bad.MITHTMLParserTreeBuilder )
HTMLParserN)CDataComment DeclarationDoctypeProcessingInstruction)EntitySubstitution
UnicodeDammit)DetectsXMLParsedAsHTMLParserRejectedMarkupHTMLHTMLTreeBuilderSTRICTz html.parserc @ s| e Zd ZdZdZdZdd Zdd Zdd Zd d d Z d d
dZ
dd Z dd Z dd Z
dd Zdd Zdd Zdd Zd S ) BeautifulSoupHTMLParserzA subclass of the Python standard library's HTMLParser class, which
listens for HTMLParser events and translates them into calls
to Beautiful Soup's tree construction API.
ignorereplacec O s4 | d| j| _tj| f|| g | _| dS )a Constructor.
:param on_duplicate_attribute: A strategy for what to do if a
tag includes the same attribute more than once. Accepted
values are: REPLACE (replace earlier values with later
ones, the default), IGNORE (keep the earliest value
encountered), or a callable. A callable must take three
arguments: the dictionary of attributes already processed,
the name of the duplicate attribute, and the most recent value
encountered.
on_duplicate_attributeN)popREPLACEr r __init__ already_closed_empty_elementZ_initialize_xml_detector)selfargskwargs r ;/tmp/pip-unpacked-wheel-pg4vfqbd/bs4/builder/_htmlparser.pyr . s z BeautifulSoupHTMLParser.__init__c C s t |d S )N)r
)r messager r r errorJ s z BeautifulSoupHTMLParser.errorc C s | j ||dd}| | dS )zHandle an incoming empty-element tag.
This is only called when the markup looks like .
:param name: Name of the tag.
:param attrs: Dictionary of the tag's attributes.
F)handle_empty_elementN)handle_starttag
handle_endtag)r nameattrstagr r r handle_startendtagZ s z*BeautifulSoupHTMLParser.handle_startendtagTc C s i }|D ]`\}}|dkr d}||kr\| j }|| jkr6qd|d| jfkrN|||< qd|||| n|||< d}q| \} }
| jj|dd|| |
d} | r| jr|r| j|dd | j | | j
dkr| | dS )a3 Handle an opening tag, e.g. ''
:param name: Name of the tag.
:param attrs: Dictionary of the tag's attributes.
:param handle_empty_element: True if this tag is known to be
an empty-element tag (i.e. there is not expected to be any
closing tag).
N z"")
sourceline sourceposF)check_already_closed) r IGNOREr getpossoupr! Zis_empty_elementr" r appendZ _root_tagZ_root_tag_encountered) r r# r$ r Z attr_dictkeyvalueZon_dupe attrvaluer( r) r% r r r r! i s6
z'BeautifulSoupHTMLParser.handle_starttagc C s, |r || j kr | j | n | j| dS )zHandle a closing tag, e.g. ''
:param name: A tag name.
:param check_already_closed: True if this tag is expected to
be the closing portion of an empty-element tag,
e.g. ''.
N)r remover- r" )r r# r* r r r r" s z%BeautifulSoupHTMLParser.handle_endtagc C s | j | dS )z4Handle some textual data that shows up between tags.N)r- handle_datar datar r r r3 s z#BeautifulSoupHTMLParser.handle_datac C s | dr t|dd}n$| dr8t|dd}nt|}d}|dk r| jjdfD ]B}|sbqXzt|g|}W qX tk
r } zW 5 d}~X Y qXX qX|sz t|}W n& t t
fk
r } zW 5 d}~X Y nX |pd}| | dS )zHandle a numeric character reference by converting it to the
corresponding Unicode character and treating it as textual
data.
:param name: Character number, possibly in hexadecimal.
x XN z windows-1252u �)
startswithintlstripr- original_encoding bytearraydecodeUnicodeDecodeErrorchr
ValueError
OverflowErrorr3 )r r# Z real_namer5 encodinger r r handle_charref s*
z&BeautifulSoupHTMLParser.handle_charrefc C s0 t j|}|dk r|}nd| }| | dS )zHandle a named entity reference by converting it to the
corresponding Unicode character(s) and treating it as textual
data.
:param name: Name of the entity reference.
Nz&%s)r
ZHTML_ENTITY_TO_CHARACTERgetr3 )r r# characterr5 r r r handle_entityref s
z(BeautifulSoupHTMLParser.handle_entityrefc C s& | j | j | | j t dS )zOHandle an HTML comment.
:param data: The text of the comment.
N)r- endDatar3 r r4 r r r handle_comment s
z&BeautifulSoupHTMLParser.handle_commentc C s6 | j |tdd }| j | | j t dS )zYHandle a DOCTYPE declaration.
:param data: The text of the declaration.
zDOCTYPE N)r- rJ lenr3 r r4 r r r handle_decl s
z#BeautifulSoupHTMLParser.handle_declc C sN | dr$t}|tdd }nt}| j | j| | j| dS )z{Handle a declaration of unknown type -- probably a CDATA block.
:param data: The text of the declaration.
zCDATA[N)upperr: r rL r r- rJ r3 )r r5 clsr r r unknown_decl s
z$BeautifulSoupHTMLParser.unknown_declc C s0 | j | j | | | | j t dS )z\Handle a processing instruction.
:param data: The text of the instruction.
N)r- rJ r3 Z_document_might_be_xmlr r4 r r r handle_pi s
z!BeautifulSoupHTMLParser.handle_piN)T)T)__name__
__module__ __qualname____doc__r+ r r r r& r! r" r3 rF rI rK rM rP rQ r r r r r $ s
7
(
r c sN e Zd ZdZdZdZeZeee gZ
dZ d fdd Z d ddZ
d d
Z ZS )
r zpA Beautiful soup `TreeBuilder` that uses the `HTMLParser` parser,
found in the Python standard library.
FTNc sl t }dD ] }||kr
||}|||< q
tt| jf | |pBg }|pJi }|| d|d< ||f| _dS )a Constructor.
:param parser_args: Positional arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param parser_kwargs: Keyword arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param kwargs: Keyword arguments for the superclass constructor.
)r Fconvert_charrefsN)dictr superr r update parser_args)r rZ Z
parser_kwargsr Zextra_parser_kwargsargr0 __class__r r r * s
z HTMLParserTreeBuilder.__init__c c s\ t |tr |dddfV dS |g}|g}||g}t|||d|d}|j|j|j|jfV dS )a Run any preliminary steps necessary to make incoming markup
acceptable to the parser.
:param markup: Some markup -- probably a bytestring.
:param user_specified_encoding: The user asked to try this encoding.
:param document_declared_encoding: The markup itself claims to be
in this encoding.
:param exclude_encodings: The user asked _not_ to try any of
these encodings.
:yield: A series of 4-tuples:
(markup, encoding, declared encoding,
has undergone character replacement)
Each 4-tuple represents a strategy for converting the
document to Unicode and parsing it. Each strategy will be tried
in turn.
NFT)known_definite_encodingsuser_encodingsZis_htmlexclude_encodings)
isinstancestrr markupr= Zdeclared_html_encodingZ contains_replacement_characters) r rc Zuser_specified_encodingZdocument_declared_encodingr` r^ r_ Z
try_encodingsZdammitr r r prepare_markupC s"
z$HTMLParserTreeBuilder.prepare_markupc
C sh | j \}}t||}| j|_z|| | W n* tk
r\ } z t|W 5 d}~X Y nX g |_dS )z{Run some incoming markup through some parsing process,
populating the `BeautifulSoup` object in self.soup.
N)rZ r r- feedcloseAssertionErrorr
r )r rc r r parserrE r r r re t s
zHTMLParserTreeBuilder.feed)NN)NNN)rR rS rT rU Zis_xmlZ picklable
HTMLPARSERNAMEr r featuresZTRACKS_LINE_NUMBERSr rd re
__classcell__r r r\ r r s
1)rU __license____all__ html.parserr syswarningsZ bs4.elementr r r r r Z
bs4.dammitr
r Z bs4.builderr r
r r r ri r r r r r r s z