U nu fE@sddlZddlZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZddlmZerddlmZGdd d e ZeeefZejdd kreeeeeefeeefefZ nej Z d d Z!Gd dde"ddZ#ejdkr(ej$Z%nGdddZ%dSddZ&dTddZ'ejdkr`ddZ(nddZ(ddZ)ddZ*dUd d!Z+d"d#eej,DZ-dVd$d%Z.d&d#eej,DZ/d'd#eej,Dd(d)d*d+d,hBZ0d-d.Z1d/d0Z2d1d2Z3d3d4Z4d5d6Z5d7d8Z6d9d:Z7e8Z9d;d<Z:dWd=d>Z;d?d@ZndEdDZ>dFdGZ?dHdIZ@ejdd dJkrddKlAmBZBeBddLdMdNZCdOdPZDndQdNZCdRdPZDdS)XN)ABCMeta)ModuleexprAST) CallableDictIterableIteratorListOptionalTupleUnioncastAny TYPE_CHECKING) iteritems)NodeNGc@seZdZdZdZdZdS) EnhancedASTNr)__name__ __module__ __qualname__Z first_tokenZ last_tokenlinenorr2/tmp/pip-unpacked-wheel-0ukdbujv/asttokens/util.pyr!srcCsdtj|t|dfS)zRReturns a human-friendly representation of a token with the given type and string.z%s:%su)tokentok_namereprlstrip)tok_typestringrrr token_repr/sr#c@seZdZdZddZdS)Tokena TokenInfo is an 8-tuple containing the same 5 fields as the tokens produced by the tokenize module, and 3 additional ones useful for this module: - [0] .type Token type (see token.py) - [1] .string Token (a string) - [2] .start Starting (row, column) indices of the token (a 2-tuple of ints) - [3] .end Ending (row, column) indices of the token (a 2-tuple of ints) - [4] .line Original line (string) - [5] .index Index of the token in the list of tokens that it belongs to. - [6] .startpos Starting character offset into the input text. - [7] .endpos Ending character offset into the input text. cCst|j|jSN)r#typer"selfrrr__str__Dsz Token.__str__N)rrr__doc__r)rrrrr$6s r$z0type string start end line index startpos endpos)c@seZdZeZdS) AstConstantN)rrrobjectvaluerrrrr-Lsr-cCs|j|ko|dkp|j|kS)zVReturns true if token is of the given type and, if a string is given, has that string.N)r&r"rr!Ztok_strrrr match_tokenPsr1cCs>t|||s:tdt||t||jd|jddfdS)z Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. z+Expected token %s, got %s on line %s col %srrN)r1 ValueErrorr#strstartr0rrr expect_tokenVs  r5)r+cCs|tjtjtjfkSzW These are considered non-coding tokens, as they don't affect the syntax tree. )rNLCOMMENTENCODING token_typerrris_non_coding_tokendsr=cCs |tjkSr7)rN_TOKENSr;rrrr=kscCs tttgtft|jS)z; Generates standard library tokens for the given code. )tokenizegenerate_tokensrrr3ioStringIOreadline)textrrrr@ssr@cCst|drtStS)z Returns a function which yields all direct children of a AST node, skipping children that are singleton nodes. The function depends on whether ``node`` is from ``ast`` or from the ``astroid`` module. get_children)hasattriter_children_astroiditer_children_astnoderrriter_children_func~srKFcCs|st|rgS|Sr%) is_joined_strrE)rJinclude_joined_strrrrrGs rGc Cs<h|]4\}}t|trt|tjtjtjtjtjfr|qSr) isinstancer& issubclassastZ expr_contextZboolopoperatorZunaryopZcmpop.0ncrrr s rVccsp|st|rdSt|tjrLt|j|jD]\}}|dk r@|V|Vq*dSt|D]}|jt krV|VqVdSr%) rLrNrPrzipkeysvaluesZiter_child_nodes __class__ SINGLETONS)rJrMkeyr/childrrrrHs   rHcCs*h|]"\}}t|trt|tjr|qSr)rNr&rOrPZstmtrRrrrrVs cCs*h|]"\}}t|trt|tjr|qSr)rNr&rOrPrrRrrrrVs Z AssignNameZDelNameZConstZ AssignAttrZDelAttrcCs |jjtkS)z+Returns whether node is an expression node.)rZrexpr_class_namesrIrrris_exprsr_cCs |jjtkS)z)Returns whether node is a statement node.)rZrstmt_class_namesrIrrris_stmtsracCs |jjdkS)z&Returns whether node is a module node.rrZrrIrrr is_modulesrccCs |jjdkS)zFReturns whether node is a JoinedStr node, used to represent f-strings. JoinedStrrbrIrrrrLsrLcCs |jjdkS)z2Returns whether node is a starred expression node.ZStarredrbrIrrr is_starredsrecCs0|jjdkp.|jjdko.tttttj|jS)z?Returns whether node represents a slice, e.g. `1:2` in `x[1:2]`)SliceZExtSlicer ) rZranymapis_slicerrPr ZeltsrIrrrris  ricCsB|jjdko@t|tj o@|j|jko<|jkozvisit_tree..N) rKset _PREVISITpopAssertionErroraddappendleninsertrr r$)rJZprevisitZ postvisit iter_childrendoneretstackcurrentZ par_valuer/rnZ post_valueinsrTrrr visit_trees$    rccsft|}t}|g}|rb|}||ks,t|||Vt|}|||D]}|||qNqdS)a1 Recursively yield all descendant nodes in the tree starting at ``node`` (including ``node`` itself), using depth-first pre-order traversal (yieling parents before their children). This is similar to ``ast.walk()``, but with a different order, and it works for both ``ast`` and ``astroid`` trees. Also, as ``iter_children()``, it skips singleton nodes generated by ``ast``. By default, ``JoinedStr`` (f-string) nodes and their contents are skipped because they previously couldn't be handled. Set ``include_joined_str`` to True to include them. N)rKrqrsrtrurwrx)rJrMryrzr|r}r~rUrrrwalks   rcCsXd}g}t|D]*\}}}|||||||}q|||dd|S)ah Replaces multiple slices of text with new values. This is a convenience method for making code modifications of ranges e.g. as identified by ``ASTTokens.get_text_range(node)``. Replacements is an iterable of ``(start, end, new_text)`` tuples. For example, ``replace("this is a test", [(0, 4, "X"), (8, 9, "THE")])`` produces ``"X is THE test"``. rN)sortedrvjoin)rD replacementsppartsr4endZnew_textrrrreplace1s  rc@s eZdZdZddZddZdS) NodeMethodsz[ Helper to get `visit_{node_type}` methods given a node's class and cache the results. cCs i|_dSr%)_cacher'rrr__init__IszNodeMethods.__init__cCs:|j|}|s6d|j}t|||j}||j|<|S)z Using the lowercase name of the class as node_type, returns `obj.visit_{node_type}`, or `obj.visit_default` if the type-specific method is not found. Zvisit_)rgetrrjgetattrZ visit_default)r(objclsmethodnamerrrrMs   zNodeMethods.getN)rrrr*rrrrrrrEsrcCst|Sr%)iter)original_tokensrrrpatched_generate_tokens^srccszg}|D]X}|jtjtjtjfkrB|r6|dj|jkrB||qt|D] }|VqJg}|Vqt|D] }|VqjdS)a Fixes tokens yielded by `tokenize.generate_tokens` to handle more non-ASCII characters in identifiers. Workaround for https://github.com/python/cpython/issues/68382. Should only be used when tokenizing a string that is known to be valid syntax, because it assumes that error tokens are not actually errors. Combines groups of consecutive NAME, NUMBER, and/or ERRORTOKEN tokens into a single NAME token. N) r&r?NAME ERRORTOKENNUMBERrr4rvcombine_tokens)rgrouptokZcombined_tokenrrrrbs    cCsftdd|Dr(tdd|Ddkr,|Stjtjddd|D|dj|d j|djd gS) Ncss|]}|jtjkVqdSr%)r&r?rrSrrrr }sz!combine_tokens..cSsh|] }|jqSr)linerrrrrV}sz!combine_tokens..rrcss|] }|jVqdSr%)r")rStrrrrsrr)r&r"r4rr) rgrwr? TokenInforrr4rr)rrrrr{s(rcCs*ddt||D}|r&t|dS|S)zq If the given AST node contains multiple statements, return the last one. Otherwise, just return the node. cSs&g|]}t|st|jdkr|qS))Z excepthandlerZ ExceptHandlerZ match_caseZ MatchCaseZ TryExceptZ TryFinally)rar&r)rSr]rrr szlast_stmt..r)rK last_stmt)rJZ child_stmtsrrrrs    r)r+) lru_cache)maxsizecsbdt}ddt|D}dd|D}tt|t|k}tfdd|D}|o`|S)a The positions attached to nodes inside f-string FormattedValues have some bugs that were fixed in Python 3.9.7 in https://github.com/python/cpython/pull/27729. This checks for those bugs more concretely without relying on the Python version. Specifically this checks: - Values with a format spec or conversion - Repeated (i.e. identical-looking) expressions - f-strings implicitly concatenated over multiple lines. - Multiline, triple-quoted f-strings. z( f"a {b}{b} c {d!r} e {f:g} h {i:{j}} k {l:{m:n}}" f"a {b}{b} c {d!r} e {f:g} h {i:{j}} k {l:{m:n}}" f"{x + y + z} {x} {y} {z} {z} {z!a} {z:z}" f''' {s} {t} {u} {v} ''' )cSsg|]}t|tjr|qSr)rNrPNamerSrJrrrrs z*fstring_positions_work..cSsg|]}|j|jfqSr)r col_offsetrrrrrsc3s |]}t||jkVqdSr%)rPZget_source_segmentidrsourcerrrsz)fstring_positions_work..)rPparserrwrqall)treeZ name_nodesZname_positionsZpositions_are_uniqueZcorrect_source_segmentsrrrfstring_positions_works   rcCstjdkrdSt|ddD]l}t|tjs,q|jD]R}t|ddt|tjr2t spt|j D]}t|ddq^|j r2t|j ddq2qdS)z Add a special attribute `_broken_positions` to nodes inside f-strings if the lineno/col_offset cannot be trusted. )r+ NT)rMZ_broken_positions) sys version_inforrNrPrdrYsetattrZFormattedValuerr/ format_spec)rZ joinedstrpartr]rrrannotate_fstring_nodess     rcCsdS)NFrrrrrrscCsdSr%r)Z_treerrrrs)N)N)F)F)F)ErP collectionsrArrr?abcrrrrtypingrrrr r r r r rrrsixrZastroid_compatrrZAstNoderintr3rr# namedtupler$Constantr-r1r5r=r@rKrG__dict__r[rHr`r^r_rarcrLrerirmr.rrrrrrrrr functoolsrrrrrrrsv 4   $         +